Hi all,
As some of you have seen (at least Stephen), I filed a ticket this morning about increasing testing coverage. The other night Anthony and I met up in NYC and he had something of an "intervention" about the sufficiency of answer testing for yt; it didn't take too much work on his part to convince me that we should be testing not just against a gold standard, but also performing unit tests. In the past I had eschewed unit testing simply because the task of mocking data was quite tricky, and by adding tests that use smaller bits we could cover unit testable areas with answer testing.
But, this isn't really a good strategy. Let's move to having both. The testing infrastructure he recommends is the nearly-omnipresent nose:
http://nose.readthedocs.org/en/latest/
The ticket to track this is here:
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
There are a couple sub-items here:
1) NumPy's nose test plugins provide a lot of necessary functionality that we have reimplemented in the answer testing utilities. I'd like to start using the numpy plugins, which include things like conditional test execution, array comparisons, "slow" tests, etc etc. 2) We can evaluate, using conditional test execution, moving to nose for answer testing. But that's not on the agenda now. 3) Writing tests for nose is super easy, and running them is too. Just do:
nosetest -w yt/
when in your source directory.
4) I've written a simple sample here:
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
5) I'll handle writing up some mock data that doesn't require shipping lots of binary files, which can then be used for checking things that absolutely require hierarchies.
--
The way to organize tests is easy. Inside each directory with testable items create a new directory called "tests", and in here toss some scripts. You can stick a bunch of functions in those scripts.
Anyway, I'm going to start writing more of these (in the main yt repo, and this change will be grafted there as well) and I'll write back once the data mocking is ready. I'd like it if we started encouraging or even mandating simple tests (and/or answer tests) for functionality that gets added, but that's a discussion that should be held separately.
The items on the ticket:
* kD-tree for nearest neighbor * Geometric selection routines * Profiles * Projections -- underlying quadtree * Data object selection of data containers * Data object selection of points * Orientation class * Pixelization * Color maps * PNG writing
Is anyone willing to claim any additional items that they will help write unit tests for?
-Matt
Hi Matt.
I would like to help with this, but it's difficult to figure out where to start.
Say I want to test projections. I make a fake 3D density field, maybe something as simple as np.arange(4**3).reshape((4, 4, 4)). I write down the answer to the x-projection. Now all I need to do is call assert_allclose(yt_result, answer, rtol=1e-15), but I don't know what pieces of low-level yt stuff to call to get to `yt_result`. Hopefully that's clear...
Maybe this comes down to creating a fake frontend we can attach fields to?
- Casey
On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
As some of you have seen (at least Stephen), I filed a ticket this morning about increasing testing coverage. The other night Anthony and I met up in NYC and he had something of an "intervention" about the sufficiency of answer testing for yt; it didn't take too much work on his part to convince me that we should be testing not just against a gold standard, but also performing unit tests. In the past I had eschewed unit testing simply because the task of mocking data was quite tricky, and by adding tests that use smaller bits we could cover unit testable areas with answer testing.
But, this isn't really a good strategy. Let's move to having both. The testing infrastructure he recommends is the nearly-omnipresent nose:
http://nose.readthedocs.org/en/latest/
The ticket to track this is here:
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
There are a couple sub-items here:
- NumPy's nose test plugins provide a lot of necessary functionality
that we have reimplemented in the answer testing utilities. I'd like to start using the numpy plugins, which include things like conditional test execution, array comparisons, "slow" tests, etc etc. 2) We can evaluate, using conditional test execution, moving to nose for answer testing. But that's not on the agenda now. 3) Writing tests for nose is super easy, and running them is too. Just do:
nosetest -w yt/
when in your source directory.
- I've written a simple sample here:
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
- I'll handle writing up some mock data that doesn't require shipping
lots of binary files, which can then be used for checking things that absolutely require hierarchies.
--
The way to organize tests is easy. Inside each directory with testable items create a new directory called "tests", and in here toss some scripts. You can stick a bunch of functions in those scripts.
Anyway, I'm going to start writing more of these (in the main yt repo, and this change will be grafted there as well) and I'll write back once the data mocking is ready. I'd like it if we started encouraging or even mandating simple tests (and/or answer tests) for functionality that gets added, but that's a discussion that should be held separately.
The items on the ticket:
- kD-tree for nearest neighbor
- Geometric selection routines
- Profiles
- Projections -- underlying quadtree
- Data object selection of data containers
- Data object selection of points
- Orientation class
- Pixelization
- Color maps
- PNG writing
Is anyone willing to claim any additional items that they will help write unit tests for?
-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Why do I always get these emails right before I have to get on a plane?! I'll write a more detailed response soon.... On Sep 21, 2012 6:16 PM, "Casey W. Stark" caseywstark@gmail.com wrote:
Hi Matt.
I would like to help with this, but it's difficult to figure out where to start.
Say I want to test projections. I make a fake 3D density field, maybe something as simple as np.arange(4**3).reshape((4, 4, 4)). I write down the answer to the x-projection. Now all I need to do is call assert_allclose(yt_result, answer, rtol=1e-15), but I don't know what pieces of low-level yt stuff to call to get to `yt_result`. Hopefully that's clear...
Maybe this comes down to creating a fake frontend we can attach fields to?
- Casey
On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
As some of you have seen (at least Stephen), I filed a ticket this morning about increasing testing coverage. The other night Anthony and I met up in NYC and he had something of an "intervention" about the sufficiency of answer testing for yt; it didn't take too much work on his part to convince me that we should be testing not just against a gold standard, but also performing unit tests. In the past I had eschewed unit testing simply because the task of mocking data was quite tricky, and by adding tests that use smaller bits we could cover unit testable areas with answer testing.
But, this isn't really a good strategy. Let's move to having both. The testing infrastructure he recommends is the nearly-omnipresent nose:
http://nose.readthedocs.org/en/latest/
The ticket to track this is here:
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
There are a couple sub-items here:
- NumPy's nose test plugins provide a lot of necessary functionality
that we have reimplemented in the answer testing utilities. I'd like to start using the numpy plugins, which include things like conditional test execution, array comparisons, "slow" tests, etc etc. 2) We can evaluate, using conditional test execution, moving to nose for answer testing. But that's not on the agenda now. 3) Writing tests for nose is super easy, and running them is too. Just do:
nosetest -w yt/
when in your source directory.
- I've written a simple sample here:
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
- I'll handle writing up some mock data that doesn't require shipping
lots of binary files, which can then be used for checking things that absolutely require hierarchies.
--
The way to organize tests is easy. Inside each directory with testable items create a new directory called "tests", and in here toss some scripts. You can stick a bunch of functions in those scripts.
Anyway, I'm going to start writing more of these (in the main yt repo, and this change will be grafted there as well) and I'll write back once the data mocking is ready. I'd like it if we started encouraging or even mandating simple tests (and/or answer tests) for functionality that gets added, but that's a discussion that should be held separately.
The items on the ticket:
- kD-tree for nearest neighbor
- Geometric selection routines
- Profiles
- Projections -- underlying quadtree
- Data object selection of data containers
- Data object selection of points
- Orientation class
- Pixelization
- Color maps
- PNG writing
Is anyone willing to claim any additional items that they will help write unit tests for?
-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Helo Casey,
Sorry for taking the whole weekend to respond.
I would like to help with this, but it's difficult to figure out where to
start.
Not to worry. I think that any of the items listed at the bottom of Matt's original email would be a great place to start.
Say I want to test projections. I make a fake 3D density field, maybe something as simple as np.arange(4**3).reshape((4, 4, 4)). I write down the answer to the x-projection. Now all I need to do is call assert_allclose(yt_result, answer, rtol=1e-15), but I don't know what pieces of low-level yt stuff to call to get to `yt_result`. Hopefully that's clear...
Maybe this comes down to creating a fake frontend we can attach fields to?
Actually, I disagree with this strategy, as I told Matt when we spoke last week. *What is important is that we test the science and math parts of the code * *before, **if ever, dealing with the software architecture that surrounds them. * * * Let's taking your example of projections. What we need to test is the actual function or method which actually slogs through the projection calculation. In many cases in yt these functions are not directly attached to the front end but live in analysis, visualization or utilities subpackages. It is these such packages that we should worry about testing. We can easily create routines to feed them sample data.
On the other hand, testing or mocking things like frontends should be a very low priority. At the end of the day what you are testing here is pulling in data from disk or other sources. Effectively, this is just re-testing functionality present in h5py, etc. That is not really our job. Yes, in a perfect world, front ends would be tested too. But I think that the priority should be placed on things like the KDTree.
Be Well Anthony
- Casey
On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
As some of you have seen (at least Stephen), I filed a ticket this morning about increasing testing coverage. The other night Anthony and I met up in NYC and he had something of an "intervention" about the sufficiency of answer testing for yt; it didn't take too much work on his part to convince me that we should be testing not just against a gold standard, but also performing unit tests. In the past I had eschewed unit testing simply because the task of mocking data was quite tricky, and by adding tests that use smaller bits we could cover unit testable areas with answer testing.
But, this isn't really a good strategy. Let's move to having both. The testing infrastructure he recommends is the nearly-omnipresent nose:
http://nose.readthedocs.org/en/latest/
The ticket to track this is here:
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
There are a couple sub-items here:
- NumPy's nose test plugins provide a lot of necessary functionality
that we have reimplemented in the answer testing utilities. I'd like to start using the numpy plugins, which include things like conditional test execution, array comparisons, "slow" tests, etc etc. 2) We can evaluate, using conditional test execution, moving to nose for answer testing. But that's not on the agenda now. 3) Writing tests for nose is super easy, and running them is too. Just do:
nosetest -w yt/
when in your source directory.
- I've written a simple sample here:
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
- I'll handle writing up some mock data that doesn't require shipping
lots of binary files, which can then be used for checking things that absolutely require hierarchies.
--
The way to organize tests is easy. Inside each directory with testable items create a new directory called "tests", and in here toss some scripts. You can stick a bunch of functions in those scripts.
Anyway, I'm going to start writing more of these (in the main yt repo, and this change will be grafted there as well) and I'll write back once the data mocking is ready. I'd like it if we started encouraging or even mandating simple tests (and/or answer tests) for functionality that gets added, but that's a discussion that should be held separately.
The items on the ticket:
- kD-tree for nearest neighbor
- Geometric selection routines
- Profiles
- Projections -- underlying quadtree
- Data object selection of data containers
- Data object selection of points
- Orientation class
- Pixelization
- Color maps
- PNG writing
Is anyone willing to claim any additional items that they will help write unit tests for?
-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Anthony.
I completely agree that we should target the level of functions actually performing the projection rather than yt's organization. The mock frontend suggestion was just a hack to get there. I don't know if there's a way around it though...
Here's an example of what I sorted through to get to projections: - Load a test plotfile, check pf.h.proj to find it's source. - Read through data_objects/hierarchy.py and utilities/parallel_tools/parallel_analysis_interface.py to find where proj is attached, can't find it. - The proj docstring says it is a reference to AMRQuadProj. Can't find a class by that name. - Search data_objects sources for "proj", find AMRProjBase.
So it looks like the functionality is wrapped up in the __project_level and _project_grid methods. I can't think of a way to test those without creating an AMRProjBase, and that requires a staticoutput object.
So unfortunately, I think it would still come down to having a fake frontend. It's not ideal, but it seems like any more isolation would require big rewrites to yt.
Of course, I could be missing something. Matt, can you think of a better way?
- Casey
On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz scopatz@gmail.com wrote:
Helo Casey,
Sorry for taking the whole weekend to respond.
I would like to help with this, but it's difficult to figure out where to
start.
Not to worry. I think that any of the items listed at the bottom of Matt's original email would be a great place to start.
Say I want to test projections. I make a fake 3D density field, maybe something as simple as np.arange(4**3).reshape((4, 4, 4)). I write down the answer to the x-projection. Now all I need to do is call assert_allclose(yt_result, answer, rtol=1e-15), but I don't know what pieces of low-level yt stuff to call to get to `yt_result`. Hopefully that's clear...
Maybe this comes down to creating a fake frontend we can attach fields to?
Actually, I disagree with this strategy, as I told Matt when we spoke last week. *What is important is that we test the science and math parts of the code
*before, **if ever, dealing with the software architecture that surrounds them. *
Let's taking your example of projections. What we need to test is the actual function or method which actually slogs through the projection calculation. In many cases in yt these functions are not directly attached to the front end but live in analysis, visualization or utilities subpackages. It is these such packages that we should worry about testing. We can easily create routines to feed them sample data.
On the other hand, testing or mocking things like frontends should be a very low priority. At the end of the day what you are testing here is pulling in data from disk or other sources. Effectively, this is just re-testing functionality present in h5py, etc. That is not really our job. Yes, in a perfect world, front ends would be tested too. But I think that the priority should be placed on things like the KDTree.
Be Well Anthony
- Casey
On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
As some of you have seen (at least Stephen), I filed a ticket this morning about increasing testing coverage. The other night Anthony and I met up in NYC and he had something of an "intervention" about the sufficiency of answer testing for yt; it didn't take too much work on his part to convince me that we should be testing not just against a gold standard, but also performing unit tests. In the past I had eschewed unit testing simply because the task of mocking data was quite tricky, and by adding tests that use smaller bits we could cover unit testable areas with answer testing.
But, this isn't really a good strategy. Let's move to having both. The testing infrastructure he recommends is the nearly-omnipresent nose:
http://nose.readthedocs.org/en/latest/
The ticket to track this is here:
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
There are a couple sub-items here:
- NumPy's nose test plugins provide a lot of necessary functionality
that we have reimplemented in the answer testing utilities. I'd like to start using the numpy plugins, which include things like conditional test execution, array comparisons, "slow" tests, etc etc. 2) We can evaluate, using conditional test execution, moving to nose for answer testing. But that's not on the agenda now. 3) Writing tests for nose is super easy, and running them is too. Just do:
nosetest -w yt/
when in your source directory.
- I've written a simple sample here:
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
- I'll handle writing up some mock data that doesn't require shipping
lots of binary files, which can then be used for checking things that absolutely require hierarchies.
--
The way to organize tests is easy. Inside each directory with testable items create a new directory called "tests", and in here toss some scripts. You can stick a bunch of functions in those scripts.
Anyway, I'm going to start writing more of these (in the main yt repo, and this change will be grafted there as well) and I'll write back once the data mocking is ready. I'd like it if we started encouraging or even mandating simple tests (and/or answer tests) for functionality that gets added, but that's a discussion that should be held separately.
The items on the ticket:
- kD-tree for nearest neighbor
- Geometric selection routines
- Profiles
- Projections -- underlying quadtree
- Data object selection of data containers
- Data object selection of points
- Orientation class
- Pixelization
- Color maps
- PNG writing
Is anyone willing to claim any additional items that they will help write unit tests for?
-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hey Casey and Anthony,
On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark caseywstark@gmail.com wrote:
Hi Anthony.
I completely agree that we should target the level of functions actually performing the projection rather than yt's organization. The mock frontend suggestion was just a hack to get there. I don't know if there's a way around it though...
Here's an example of what I sorted through to get to projections:
- Load a test plotfile, check pf.h.proj to find it's source.
- Read through data_objects/hierarchy.py and
utilities/parallel_tools/parallel_analysis_interface.py to find where proj is attached, can't find it.
- The proj docstring says it is a reference to AMRQuadProj. Can't find a
class by that name.
- Search data_objects sources for "proj", find AMRProjBase.
So it looks like the functionality is wrapped up in the __project_level and _project_grid methods. I can't think of a way to test those without creating an AMRProjBase, and that requires a staticoutput object.
You're right, the projection stuff as *projections* is not easy to test. But in terms of testing the underlying code, which is wrapped up in a Cython class called QuadTree, I think it could be done. The steps you're describing are actually all part of the existing answer testing machinery, which performs a couple things and verifies that they don't change over time:
1) Project some fields from the disk 2) Project a couple derived fields 3) Project a derived field that requires spatial derivatives 4) Project the "Ones" field, which should be 1.0 everywhere.
So these things are done, but it is also possible that the specific quadtree functionality could be tested, in isolation from the projection. I think this may be oneo f the things Anthony is talking about -- answer testing can handle the big, complex items, and by breaking down to the fundamentals we can address isolated items from a unit testing perspective.
So unfortunately, I think it would still come down to having a fake frontend. It's not ideal, but it seems like any more isolation would require big rewrites to yt.
One fun thing that is not usually known is that we have a fake frontend already, it just doesn't get used much. It's called the "Stream" frontend and it was designed originally to be used in ParaView, but now gets used by the (new, not-yet-documented/released) load_uniform_grid function as well as by Hyperion, the RT code by Tom R. It can set up AMR as well as static mesh. It's not terribly well documented, but there are examples on the wiki.
One thing I've been thinking about is actually creating a couple fake outputs, which could be defined analytically with spheres of overdensity inside them. In principle, if we added refinement criteria, we could make this relatively complex data that was defined with only a few lines of code, but spun up a big in-memory dataset.
(This exact thing is on my list of things to do and then to output in GDF, by the way...)
That I think could come, down the road a bit. The refinement criteria wouldn't be too bad to implement, especially since we already have the grid splitting routines. I just don't think we should focus on it at the moment. But the uniform grid creation and loading works already -- I used it this morning. You can do it with:
from yt.frontends.stream.api import load_uniform_grid ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2}, [359, 359, 359], 1.0)
the list is the dimensions of the data and the value is the to-cm conversion.
Of course, I could be missing something. Matt, can you think of a better way?
I think for this specific example (and your damningly complex tracing of things through the source ...) the easiest thing to do is isolate the Cython routine, which it seems I was able to do only because I wrote it and which seems quite buried in the code, and to also provide high-level machinery for faking a frontend.
-Matt
- Casey
On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz scopatz@gmail.com wrote:
Helo Casey,
Sorry for taking the whole weekend to respond.
I would like to help with this, but it's difficult to figure out where to start.
Not to worry. I think that any of the items listed at the bottom of Matt's original email would be a great place to start.
Say I want to test projections. I make a fake 3D density field, maybe something as simple as np.arange(4**3).reshape((4, 4, 4)). I write down the answer to the x-projection. Now all I need to do is call assert_allclose(yt_result, answer, rtol=1e-15), but I don't know what pieces of low-level yt stuff to call to get to `yt_result`. Hopefully that's clear...
Maybe this comes down to creating a fake frontend we can attach fields to?
Actually, I disagree with this strategy, as I told Matt when we spoke last week. What is important is that we test the science and math parts of the code before, if ever, dealing with the software architecture that surrounds them.
Let's taking your example of projections. What we need to test is the actual function or method which actually slogs through the projection calculation. In many cases in yt these functions are not directly attached to the front end but live in analysis, visualization or utilities subpackages. It is these such packages that we should worry about testing. We can easily create routines to feed them sample data.
On the other hand, testing or mocking things like frontends should be a very low priority. At the end of the day what you are testing here is pulling in data from disk or other sources. Effectively, this is just re-testing functionality present in h5py, etc. That is not really our job. Yes, in a perfect world, front ends would be tested too. But I think that the priority should be placed on things like the KDTree.
Be Well Anthony
- Casey
On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
As some of you have seen (at least Stephen), I filed a ticket this morning about increasing testing coverage. The other night Anthony and I met up in NYC and he had something of an "intervention" about the sufficiency of answer testing for yt; it didn't take too much work on his part to convince me that we should be testing not just against a gold standard, but also performing unit tests. In the past I had eschewed unit testing simply because the task of mocking data was quite tricky, and by adding tests that use smaller bits we could cover unit testable areas with answer testing.
But, this isn't really a good strategy. Let's move to having both. The testing infrastructure he recommends is the nearly-omnipresent nose:
http://nose.readthedocs.org/en/latest/
The ticket to track this is here:
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
There are a couple sub-items here:
- NumPy's nose test plugins provide a lot of necessary functionality
that we have reimplemented in the answer testing utilities. I'd like to start using the numpy plugins, which include things like conditional test execution, array comparisons, "slow" tests, etc etc. 2) We can evaluate, using conditional test execution, moving to nose for answer testing. But that's not on the agenda now. 3) Writing tests for nose is super easy, and running them is too. Just do:
nosetest -w yt/
when in your source directory.
- I've written a simple sample here:
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
- I'll handle writing up some mock data that doesn't require shipping
lots of binary files, which can then be used for checking things that absolutely require hierarchies.
--
The way to organize tests is easy. Inside each directory with testable items create a new directory called "tests", and in here toss some scripts. You can stick a bunch of functions in those scripts.
Anyway, I'm going to start writing more of these (in the main yt repo, and this change will be grafted there as well) and I'll write back once the data mocking is ready. I'd like it if we started encouraging or even mandating simple tests (and/or answer tests) for functionality that gets added, but that's a discussion that should be held separately.
The items on the ticket:
- kD-tree for nearest neighbor
- Geometric selection routines
- Profiles
- Projections -- underlying quadtree
- Data object selection of data containers
- Data object selection of points
- Orientation class
- Pixelization
- Color maps
- PNG writing
Is anyone willing to claim any additional items that they will help write unit tests for?
-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Matt.
Glad my example was useful in some way. I guess knowing exactly which Cython routines to test for what is what I meant about where to start.
Thanks for the tip about the stream frontend.
- Casey
On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk matthewturk@gmail.com wrote:
Hey Casey and Anthony,
On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark caseywstark@gmail.com wrote:
Hi Anthony.
I completely agree that we should target the level of functions actually performing the projection rather than yt's organization. The mock
frontend
suggestion was just a hack to get there. I don't know if there's a way around it though...
Here's an example of what I sorted through to get to projections:
- Load a test plotfile, check pf.h.proj to find it's source.
- Read through data_objects/hierarchy.py and
utilities/parallel_tools/parallel_analysis_interface.py to find where
proj
is attached, can't find it.
- The proj docstring says it is a reference to AMRQuadProj. Can't find a
class by that name.
- Search data_objects sources for "proj", find AMRProjBase.
So it looks like the functionality is wrapped up in the __project_level
and
_project_grid methods. I can't think of a way to test those without
creating
an AMRProjBase, and that requires a staticoutput object.
You're right, the projection stuff as *projections* is not easy to test. But in terms of testing the underlying code, which is wrapped up in a Cython class called QuadTree, I think it could be done. The steps you're describing are actually all part of the existing answer testing machinery, which performs a couple things and verifies that they don't change over time:
- Project some fields from the disk
- Project a couple derived fields
- Project a derived field that requires spatial derivatives
- Project the "Ones" field, which should be 1.0 everywhere.
So these things are done, but it is also possible that the specific quadtree functionality could be tested, in isolation from the projection. I think this may be oneo f the things Anthony is talking about -- answer testing can handle the big, complex items, and by breaking down to the fundamentals we can address isolated items from a unit testing perspective.
So unfortunately, I think it would still come down to having a fake frontend. It's not ideal, but it seems like any more isolation would
require
big rewrites to yt.
One fun thing that is not usually known is that we have a fake frontend already, it just doesn't get used much. It's called the "Stream" frontend and it was designed originally to be used in ParaView, but now gets used by the (new, not-yet-documented/released) load_uniform_grid function as well as by Hyperion, the RT code by Tom R. It can set up AMR as well as static mesh. It's not terribly well documented, but there are examples on the wiki.
One thing I've been thinking about is actually creating a couple fake outputs, which could be defined analytically with spheres of overdensity inside them. In principle, if we added refinement criteria, we could make this relatively complex data that was defined with only a few lines of code, but spun up a big in-memory dataset.
(This exact thing is on my list of things to do and then to output in GDF, by the way...)
That I think could come, down the road a bit. The refinement criteria wouldn't be too bad to implement, especially since we already have the grid splitting routines. I just don't think we should focus on it at the moment. But the uniform grid creation and loading works already -- I used it this morning. You can do it with:
from yt.frontends.stream.api import load_uniform_grid ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2}, [359, 359, 359], 1.0)
the list is the dimensions of the data and the value is the to-cm conversion.
Of course, I could be missing something. Matt, can you think of a better way?
I think for this specific example (and your damningly complex tracing of things through the source ...) the easiest thing to do is isolate the Cython routine, which it seems I was able to do only because I wrote it and which seems quite buried in the code, and to also provide high-level machinery for faking a frontend.
-Matt
- Casey
On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz scopatz@gmail.com
wrote:
Helo Casey,
Sorry for taking the whole weekend to respond.
I would like to help with this, but it's difficult to figure out where to start.
Not to worry. I think that any of the items listed at the bottom of
Matt's
original email would be a great place to start.
Say I want to test projections. I make a fake 3D density field, maybe something as simple as np.arange(4**3).reshape((4, 4, 4)). I write
down the
answer to the x-projection. Now all I need to do is call assert_allclose(yt_result, answer, rtol=1e-15), but I don't know what
pieces
of low-level yt stuff to call to get to `yt_result`. Hopefully that's clear...
Maybe this comes down to creating a fake frontend we can attach fields to?
Actually, I disagree with this strategy, as I told Matt when we spoke
last
week. What is important is that we test the science and math parts of the code before, if ever, dealing with the software architecture that surrounds them.
Let's taking your example of projections. What we need to test is the actual function or method which actually slogs through the projection calculation. In many cases in yt these functions are not directly attached to the front end but live
in
analysis, visualization or utilities subpackages. It is these such packages that we should
worry
about testing. We can easily create routines to feed them sample data.
On the other hand, testing or mocking things like frontends should be a very low priority. At the end of the day what you are testing here is pulling in data from disk or other sources. Effectively, this is just re-testing functionality present in h5py, etc. That is not really our job. Yes, in a perfect world, front ends would be tested
too.
But I think that the priority should be placed on things like the KDTree.
Be Well Anthony
- Casey
On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
As some of you have seen (at least Stephen), I filed a ticket this morning about increasing testing coverage. The other night Anthony and I met up in NYC and he had something of an "intervention" about the sufficiency of answer testing for yt; it didn't take too much
work
on his part to convince me that we should be testing not just against a gold standard, but also performing unit tests. In the past I had eschewed unit testing simply because the task of mocking data was quite tricky, and by adding tests that use smaller bits we could
cover
unit testable areas with answer testing.
But, this isn't really a good strategy. Let's move to having both. The testing infrastructure he recommends is the nearly-omnipresent nose:
http://nose.readthedocs.org/en/latest/
The ticket to track this is here:
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
There are a couple sub-items here:
- NumPy's nose test plugins provide a lot of necessary functionality
that we have reimplemented in the answer testing utilities. I'd like to start using the numpy plugins, which include things like conditional test execution, array comparisons, "slow" tests, etc etc. 2) We can evaluate, using conditional test execution, moving to nose for answer testing. But that's not on the agenda now. 3) Writing tests for nose is super easy, and running them is too.
Just
do:
nosetest -w yt/
when in your source directory.
- I've written a simple sample here:
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
- I'll handle writing up some mock data that doesn't require
shipping
lots of binary files, which can then be used for checking things that absolutely require hierarchies.
--
The way to organize tests is easy. Inside each directory with testable items create a new directory called "tests", and in here
toss
some scripts. You can stick a bunch of functions in those scripts.
Anyway, I'm going to start writing more of these (in the main yt
repo,
and this change will be grafted there as well) and I'll write back once the data mocking is ready. I'd like it if we started
encouraging
or even mandating simple tests (and/or answer tests) for
functionality
that gets added, but that's a discussion that should be held separately.
The items on the ticket:
- kD-tree for nearest neighbor
- Geometric selection routines
- Profiles
- Projections -- underlying quadtree
- Data object selection of data containers
- Data object selection of points
- Orientation class
- Pixelization
- Color maps
- PNG writing
Is anyone willing to claim any additional items that they will help write unit tests for?
-Matt _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hey Casey,
On Mon, Sep 24, 2012 at 5:39 PM, Casey W. Stark caseywstark@gmail.com wrote:
Hi Matt.
Glad my example was useful in some way. I guess knowing exactly which Cython routines to test for what is what I meant about where to start.
I completely understand -- and that's a failure of my part in the code. So I think what might help is if we start looking at the routines that could be tested, and then I can jump in when it looks like they're buried. I will also volunteer to try to refactor this code, once it's tested, to make it clearer what is where and why.
Thanks for the tip about the stream frontend.
No prob, it should be documented better.
In terms of generating data, here's an example of what I was meaning:
https://hub.yt-project.org/nb/ompxtg
This sets up a random (but small-valued) background density and then applies a couple top hat spheres on top of it. I think this could be a good starting point for unit testing. I looked at the code that exists for flagging cells and refining them, and it's currently a bit too specific for this, as it was designed to take RAMSES data and partition it, which operates slightly differently. I'll take a pass at extracting it and making it work in this case.
The way I see it working is that one would set up the operators, as is done here, and then a progressive routine would be applied, something like this:
while grids.flag() > 0: grids.refine() for operator in operators: operator.apply(grids) pf = grids.convert_to_dataset()
The missing step in the code base is mostly just the refinement, as for the smoothed covering grids we have the necessary machinery to interpolate from one level to the next. Adding more types of operators to this library would be beneficial as well. The finalization step at the end would then convert the collection of grids into a Stream dataset. So with only a couple lines of code we could in all likelihood be able to generate in-memory datasets.
(This underscores in my mind why it would be awesome to have GDF in more places, as with this kind of machinery we've just written initial conditions generation, especially since we have a GDF writer already...)
Anyway, what might be really helpful is if interested people would volunteer to address a few of the testable areas? Then we can start pushing and identifying problem areas. I'll volunteer to handle at least the data container stuff, and whatever else slips through the cracks; although my time will be somewhat limited in the very near future, I will try to make this a priority.
-Matt
- Casey
On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk matthewturk@gmail.com wrote:
Hey Casey and Anthony,
On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark caseywstark@gmail.com wrote:
Hi Anthony.
I completely agree that we should target the level of functions actually performing the projection rather than yt's organization. The mock frontend suggestion was just a hack to get there. I don't know if there's a way around it though...
Here's an example of what I sorted through to get to projections:
- Load a test plotfile, check pf.h.proj to find it's source.
- Read through data_objects/hierarchy.py and
utilities/parallel_tools/parallel_analysis_interface.py to find where proj is attached, can't find it.
- The proj docstring says it is a reference to AMRQuadProj. Can't find a
class by that name.
- Search data_objects sources for "proj", find AMRProjBase.
So it looks like the functionality is wrapped up in the __project_level and _project_grid methods. I can't think of a way to test those without creating an AMRProjBase, and that requires a staticoutput object.
You're right, the projection stuff as *projections* is not easy to test. But in terms of testing the underlying code, which is wrapped up in a Cython class called QuadTree, I think it could be done. The steps you're describing are actually all part of the existing answer testing machinery, which performs a couple things and verifies that they don't change over time:
- Project some fields from the disk
- Project a couple derived fields
- Project a derived field that requires spatial derivatives
- Project the "Ones" field, which should be 1.0 everywhere.
So these things are done, but it is also possible that the specific quadtree functionality could be tested, in isolation from the projection. I think this may be oneo f the things Anthony is talking about -- answer testing can handle the big, complex items, and by breaking down to the fundamentals we can address isolated items from a unit testing perspective.
So unfortunately, I think it would still come down to having a fake frontend. It's not ideal, but it seems like any more isolation would require big rewrites to yt.
One fun thing that is not usually known is that we have a fake frontend already, it just doesn't get used much. It's called the "Stream" frontend and it was designed originally to be used in ParaView, but now gets used by the (new, not-yet-documented/released) load_uniform_grid function as well as by Hyperion, the RT code by Tom R. It can set up AMR as well as static mesh. It's not terribly well documented, but there are examples on the wiki.
One thing I've been thinking about is actually creating a couple fake outputs, which could be defined analytically with spheres of overdensity inside them. In principle, if we added refinement criteria, we could make this relatively complex data that was defined with only a few lines of code, but spun up a big in-memory dataset.
(This exact thing is on my list of things to do and then to output in GDF, by the way...)
That I think could come, down the road a bit. The refinement criteria wouldn't be too bad to implement, especially since we already have the grid splitting routines. I just don't think we should focus on it at the moment. But the uniform grid creation and loading works already -- I used it this morning. You can do it with:
from yt.frontends.stream.api import load_uniform_grid ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2}, [359, 359, 359], 1.0)
the list is the dimensions of the data and the value is the to-cm conversion.
Of course, I could be missing something. Matt, can you think of a better way?
I think for this specific example (and your damningly complex tracing of things through the source ...) the easiest thing to do is isolate the Cython routine, which it seems I was able to do only because I wrote it and which seems quite buried in the code, and to also provide high-level machinery for faking a frontend.
-Matt
- Casey
On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz scopatz@gmail.com wrote:
Helo Casey,
Sorry for taking the whole weekend to respond.
I would like to help with this, but it's difficult to figure out where to start.
Not to worry. I think that any of the items listed at the bottom of Matt's original email would be a great place to start.
Say I want to test projections. I make a fake 3D density field, maybe something as simple as np.arange(4**3).reshape((4, 4, 4)). I write down the answer to the x-projection. Now all I need to do is call assert_allclose(yt_result, answer, rtol=1e-15), but I don't know what pieces of low-level yt stuff to call to get to `yt_result`. Hopefully that's clear...
Maybe this comes down to creating a fake frontend we can attach fields to?
Actually, I disagree with this strategy, as I told Matt when we spoke last week. What is important is that we test the science and math parts of the code before, if ever, dealing with the software architecture that surrounds them.
Let's taking your example of projections. What we need to test is the actual function or method which actually slogs through the projection calculation. In many cases in yt these functions are not directly attached to the front end but live in analysis, visualization or utilities subpackages. It is these such packages that we should worry about testing. We can easily create routines to feed them sample data.
On the other hand, testing or mocking things like frontends should be a very low priority. At the end of the day what you are testing here is pulling in data from disk or other sources. Effectively, this is just re-testing functionality present in h5py, etc. That is not really our job. Yes, in a perfect world, front ends would be tested too. But I think that the priority should be placed on things like the KDTree.
Be Well Anthony
- Casey
On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk matthewturk@gmail.com wrote: > > Hi all, > > As some of you have seen (at least Stephen), I filed a ticket this > morning about increasing testing coverage. The other night Anthony > and I met up in NYC and he had something of an "intervention" about > the sufficiency of answer testing for yt; it didn't take too much > work > on his part to convince me that we should be testing not just > against > a gold standard, but also performing unit tests. In the past I had > eschewed unit testing simply because the task of mocking data was > quite tricky, and by adding tests that use smaller bits we could > cover > unit testable areas with answer testing. > > But, this isn't really a good strategy. Let's move to having both. > The testing infrastructure he recommends is the nearly-omnipresent > nose: > > http://nose.readthedocs.org/en/latest/ > > The ticket to track this is here: > > > > https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage > > There are a couple sub-items here: > > 1) NumPy's nose test plugins provide a lot of necessary > functionality > that we have reimplemented in the answer testing utilities. I'd > like > to start using the numpy plugins, which include things like > conditional test execution, array comparisons, "slow" tests, etc > etc. > 2) We can evaluate, using conditional test execution, moving to nose > for answer testing. But that's not on the agenda now. > 3) Writing tests for nose is super easy, and running them is too. > Just > do: > > nosetest -w yt/ > > when in your source directory. > > 4) I've written a simple sample here: > > > > https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests... > > 5) I'll handle writing up some mock data that doesn't require > shipping > lots of binary files, which can then be used for checking things > that > absolutely require hierarchies. > > -- > > The way to organize tests is easy. Inside each directory with > testable items create a new directory called "tests", and in here > toss > some scripts. You can stick a bunch of functions in those scripts. > > Anyway, I'm going to start writing more of these (in the main yt > repo, > and this change will be grafted there as well) and I'll write back > once the data mocking is ready. I'd like it if we started > encouraging > or even mandating simple tests (and/or answer tests) for > functionality > that gets added, but that's a discussion that should be held > separately. > > The items on the ticket: > > * kD-tree for nearest neighbor > * Geometric selection routines > * Profiles > * Projections -- underlying quadtree > * Data object selection of data containers > * Data object selection of points > * Orientation class > * Pixelization > * Color maps > * PNG writing > > Is anyone willing to claim any additional items that they will help > write unit tests for? > > -Matt > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all,
As a note, Anthony just added this to the 3.0 base. Here's a set of tests:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/utilities/lib/t...
and here's a utilities section he added:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/testing.py
-Matt
On Mon, Sep 24, 2012 at 10:01 PM, Matthew Turk matthewturk@gmail.com wrote:
Hey Casey,
On Mon, Sep 24, 2012 at 5:39 PM, Casey W. Stark caseywstark@gmail.com wrote:
Hi Matt.
Glad my example was useful in some way. I guess knowing exactly which Cython routines to test for what is what I meant about where to start.
I completely understand -- and that's a failure of my part in the code. So I think what might help is if we start looking at the routines that could be tested, and then I can jump in when it looks like they're buried. I will also volunteer to try to refactor this code, once it's tested, to make it clearer what is where and why.
Thanks for the tip about the stream frontend.
No prob, it should be documented better.
In terms of generating data, here's an example of what I was meaning:
https://hub.yt-project.org/nb/ompxtg
This sets up a random (but small-valued) background density and then applies a couple top hat spheres on top of it. I think this could be a good starting point for unit testing. I looked at the code that exists for flagging cells and refining them, and it's currently a bit too specific for this, as it was designed to take RAMSES data and partition it, which operates slightly differently. I'll take a pass at extracting it and making it work in this case.
The way I see it working is that one would set up the operators, as is done here, and then a progressive routine would be applied, something like this:
while grids.flag() > 0: grids.refine() for operator in operators: operator.apply(grids) pf = grids.convert_to_dataset()
The missing step in the code base is mostly just the refinement, as for the smoothed covering grids we have the necessary machinery to interpolate from one level to the next. Adding more types of operators to this library would be beneficial as well. The finalization step at the end would then convert the collection of grids into a Stream dataset. So with only a couple lines of code we could in all likelihood be able to generate in-memory datasets.
(This underscores in my mind why it would be awesome to have GDF in more places, as with this kind of machinery we've just written initial conditions generation, especially since we have a GDF writer already...)
Anyway, what might be really helpful is if interested people would volunteer to address a few of the testable areas? Then we can start pushing and identifying problem areas. I'll volunteer to handle at least the data container stuff, and whatever else slips through the cracks; although my time will be somewhat limited in the very near future, I will try to make this a priority.
-Matt
- Casey
On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk matthewturk@gmail.com wrote:
Hey Casey and Anthony,
On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark caseywstark@gmail.com wrote:
Hi Anthony.
I completely agree that we should target the level of functions actually performing the projection rather than yt's organization. The mock frontend suggestion was just a hack to get there. I don't know if there's a way around it though...
Here's an example of what I sorted through to get to projections:
- Load a test plotfile, check pf.h.proj to find it's source.
- Read through data_objects/hierarchy.py and
utilities/parallel_tools/parallel_analysis_interface.py to find where proj is attached, can't find it.
- The proj docstring says it is a reference to AMRQuadProj. Can't find a
class by that name.
- Search data_objects sources for "proj", find AMRProjBase.
So it looks like the functionality is wrapped up in the __project_level and _project_grid methods. I can't think of a way to test those without creating an AMRProjBase, and that requires a staticoutput object.
You're right, the projection stuff as *projections* is not easy to test. But in terms of testing the underlying code, which is wrapped up in a Cython class called QuadTree, I think it could be done. The steps you're describing are actually all part of the existing answer testing machinery, which performs a couple things and verifies that they don't change over time:
- Project some fields from the disk
- Project a couple derived fields
- Project a derived field that requires spatial derivatives
- Project the "Ones" field, which should be 1.0 everywhere.
So these things are done, but it is also possible that the specific quadtree functionality could be tested, in isolation from the projection. I think this may be oneo f the things Anthony is talking about -- answer testing can handle the big, complex items, and by breaking down to the fundamentals we can address isolated items from a unit testing perspective.
So unfortunately, I think it would still come down to having a fake frontend. It's not ideal, but it seems like any more isolation would require big rewrites to yt.
One fun thing that is not usually known is that we have a fake frontend already, it just doesn't get used much. It's called the "Stream" frontend and it was designed originally to be used in ParaView, but now gets used by the (new, not-yet-documented/released) load_uniform_grid function as well as by Hyperion, the RT code by Tom R. It can set up AMR as well as static mesh. It's not terribly well documented, but there are examples on the wiki.
One thing I've been thinking about is actually creating a couple fake outputs, which could be defined analytically with spheres of overdensity inside them. In principle, if we added refinement criteria, we could make this relatively complex data that was defined with only a few lines of code, but spun up a big in-memory dataset.
(This exact thing is on my list of things to do and then to output in GDF, by the way...)
That I think could come, down the road a bit. The refinement criteria wouldn't be too bad to implement, especially since we already have the grid splitting routines. I just don't think we should focus on it at the moment. But the uniform grid creation and loading works already -- I used it this morning. You can do it with:
from yt.frontends.stream.api import load_uniform_grid ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2}, [359, 359, 359], 1.0)
the list is the dimensions of the data and the value is the to-cm conversion.
Of course, I could be missing something. Matt, can you think of a better way?
I think for this specific example (and your damningly complex tracing of things through the source ...) the easiest thing to do is isolate the Cython routine, which it seems I was able to do only because I wrote it and which seems quite buried in the code, and to also provide high-level machinery for faking a frontend.
-Matt
- Casey
On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz scopatz@gmail.com wrote:
Helo Casey,
Sorry for taking the whole weekend to respond.
> I would like to help with this, but it's difficult to figure out > where > to start.
Not to worry. I think that any of the items listed at the bottom of Matt's original email would be a great place to start.
> > > Say I want to test projections. I make a fake 3D density field, maybe > something as simple as np.arange(4**3).reshape((4, 4, 4)). I write > down the > answer to the x-projection. Now all I need to do is call > assert_allclose(yt_result, answer, rtol=1e-15), but I don't know what > pieces > of low-level yt stuff to call to get to `yt_result`. Hopefully that's > clear... > > Maybe this comes down to creating a fake frontend we can attach > fields > to?
Actually, I disagree with this strategy, as I told Matt when we spoke last week. What is important is that we test the science and math parts of the code before, if ever, dealing with the software architecture that surrounds them.
Let's taking your example of projections. What we need to test is the actual function or method which actually slogs through the projection calculation. In many cases in yt these functions are not directly attached to the front end but live in analysis, visualization or utilities subpackages. It is these such packages that we should worry about testing. We can easily create routines to feed them sample data.
On the other hand, testing or mocking things like frontends should be a very low priority. At the end of the day what you are testing here is pulling in data from disk or other sources. Effectively, this is just re-testing functionality present in h5py, etc. That is not really our job. Yes, in a perfect world, front ends would be tested too. But I think that the priority should be placed on things like the KDTree.
Be Well Anthony
> > > - Casey > > > On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk matthewturk@gmail.com > wrote: >> >> Hi all, >> >> As some of you have seen (at least Stephen), I filed a ticket this >> morning about increasing testing coverage. The other night Anthony >> and I met up in NYC and he had something of an "intervention" about >> the sufficiency of answer testing for yt; it didn't take too much >> work >> on his part to convince me that we should be testing not just >> against >> a gold standard, but also performing unit tests. In the past I had >> eschewed unit testing simply because the task of mocking data was >> quite tricky, and by adding tests that use smaller bits we could >> cover >> unit testable areas with answer testing. >> >> But, this isn't really a good strategy. Let's move to having both. >> The testing infrastructure he recommends is the nearly-omnipresent >> nose: >> >> http://nose.readthedocs.org/en/latest/ >> >> The ticket to track this is here: >> >> >> >> https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage >> >> There are a couple sub-items here: >> >> 1) NumPy's nose test plugins provide a lot of necessary >> functionality >> that we have reimplemented in the answer testing utilities. I'd >> like >> to start using the numpy plugins, which include things like >> conditional test execution, array comparisons, "slow" tests, etc >> etc. >> 2) We can evaluate, using conditional test execution, moving to nose >> for answer testing. But that's not on the agenda now. >> 3) Writing tests for nose is super easy, and running them is too. >> Just >> do: >> >> nosetest -w yt/ >> >> when in your source directory. >> >> 4) I've written a simple sample here: >> >> >> >> https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests... >> >> 5) I'll handle writing up some mock data that doesn't require >> shipping >> lots of binary files, which can then be used for checking things >> that >> absolutely require hierarchies. >> >> -- >> >> The way to organize tests is easy. Inside each directory with >> testable items create a new directory called "tests", and in here >> toss >> some scripts. You can stick a bunch of functions in those scripts. >> >> Anyway, I'm going to start writing more of these (in the main yt >> repo, >> and this change will be grafted there as well) and I'll write back >> once the data mocking is ready. I'd like it if we started >> encouraging >> or even mandating simple tests (and/or answer tests) for >> functionality >> that gets added, but that's a discussion that should be held >> separately. >> >> The items on the ticket: >> >> * kD-tree for nearest neighbor >> * Geometric selection routines >> * Profiles >> * Projections -- underlying quadtree >> * Data object selection of data containers >> * Data object selection of points >> * Orientation class >> * Pixelization >> * Color maps >> * PNG writing >> >> Is anyone willing to claim any additional items that they will help >> write unit tests for? >> >> -Matt >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hey Matt, Anthony.
Looks like a good start to nosetests. One small thing -- would anyone be opposed to moving these out of the yt directory and into a root "test" directory?
I can take on CIC deposition and interpolation wherever those are.
- Casey
On Tue, Sep 25, 2012 at 11:20 AM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
As a note, Anthony just added this to the 3.0 base. Here's a set of tests:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/utilities/lib/t...
and here's a utilities section he added:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/testing.py
-Matt
On Mon, Sep 24, 2012 at 10:01 PM, Matthew Turk matthewturk@gmail.com wrote:
Hey Casey,
On Mon, Sep 24, 2012 at 5:39 PM, Casey W. Stark caseywstark@gmail.com
wrote:
Hi Matt.
Glad my example was useful in some way. I guess knowing exactly which
Cython
routines to test for what is what I meant about where to start.
I completely understand -- and that's a failure of my part in the code. So I think what might help is if we start looking at the routines that could be tested, and then I can jump in when it looks like they're buried. I will also volunteer to try to refactor this code, once it's tested, to make it clearer what is where and why.
Thanks for the tip about the stream frontend.
No prob, it should be documented better.
In terms of generating data, here's an example of what I was meaning:
https://hub.yt-project.org/nb/ompxtg
This sets up a random (but small-valued) background density and then applies a couple top hat spheres on top of it. I think this could be a good starting point for unit testing. I looked at the code that exists for flagging cells and refining them, and it's currently a bit too specific for this, as it was designed to take RAMSES data and partition it, which operates slightly differently. I'll take a pass at extracting it and making it work in this case.
The way I see it working is that one would set up the operators, as is done here, and then a progressive routine would be applied, something like this:
while grids.flag() > 0: grids.refine() for operator in operators: operator.apply(grids) pf = grids.convert_to_dataset()
The missing step in the code base is mostly just the refinement, as for the smoothed covering grids we have the necessary machinery to interpolate from one level to the next. Adding more types of operators to this library would be beneficial as well. The finalization step at the end would then convert the collection of grids into a Stream dataset. So with only a couple lines of code we could in all likelihood be able to generate in-memory datasets.
(This underscores in my mind why it would be awesome to have GDF in more places, as with this kind of machinery we've just written initial conditions generation, especially since we have a GDF writer already...)
Anyway, what might be really helpful is if interested people would volunteer to address a few of the testable areas? Then we can start pushing and identifying problem areas. I'll volunteer to handle at least the data container stuff, and whatever else slips through the cracks; although my time will be somewhat limited in the very near future, I will try to make this a priority.
-Matt
- Casey
On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk matthewturk@gmail.com
wrote:
Hey Casey and Anthony,
On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark <caseywstark@gmail.com
wrote:
Hi Anthony.
I completely agree that we should target the level of functions
actually
performing the projection rather than yt's organization. The mock frontend suggestion was just a hack to get there. I don't know if there's a
way
around it though...
Here's an example of what I sorted through to get to projections:
- Load a test plotfile, check pf.h.proj to find it's source.
- Read through data_objects/hierarchy.py and
utilities/parallel_tools/parallel_analysis_interface.py to find where proj is attached, can't find it.
- The proj docstring says it is a reference to AMRQuadProj. Can't
find a
class by that name.
- Search data_objects sources for "proj", find AMRProjBase.
So it looks like the functionality is wrapped up in the
__project_level
and _project_grid methods. I can't think of a way to test those without creating an AMRProjBase, and that requires a staticoutput object.
You're right, the projection stuff as *projections* is not easy to test. But in terms of testing the underlying code, which is wrapped up in a Cython class called QuadTree, I think it could be done. The steps you're describing are actually all part of the existing answer testing machinery, which performs a couple things and verifies that they don't change over time:
- Project some fields from the disk
- Project a couple derived fields
- Project a derived field that requires spatial derivatives
- Project the "Ones" field, which should be 1.0 everywhere.
So these things are done, but it is also possible that the specific quadtree functionality could be tested, in isolation from the projection. I think this may be oneo f the things Anthony is talking about -- answer testing can handle the big, complex items, and by breaking down to the fundamentals we can address isolated items from a unit testing perspective.
So unfortunately, I think it would still come down to having a fake frontend. It's not ideal, but it seems like any more isolation would require big rewrites to yt.
One fun thing that is not usually known is that we have a fake frontend already, it just doesn't get used much. It's called the "Stream" frontend and it was designed originally to be used in ParaView, but now gets used by the (new, not-yet-documented/released) load_uniform_grid function as well as by Hyperion, the RT code by Tom R. It can set up AMR as well as static mesh. It's not terribly well documented, but there are examples on the wiki.
One thing I've been thinking about is actually creating a couple fake outputs, which could be defined analytically with spheres of overdensity inside them. In principle, if we added refinement criteria, we could make this relatively complex data that was defined with only a few lines of code, but spun up a big in-memory dataset.
(This exact thing is on my list of things to do and then to output in GDF, by the way...)
That I think could come, down the road a bit. The refinement criteria wouldn't be too bad to implement, especially since we already have the grid splitting routines. I just don't think we should focus on it at the moment. But the uniform grid creation and loading works already -- I used it this morning. You can do it with:
from yt.frontends.stream.api import load_uniform_grid ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2}, [359, 359, 359], 1.0)
the list is the dimensions of the data and the value is the to-cm conversion.
Of course, I could be missing something. Matt, can you think of a
better
way?
I think for this specific example (and your damningly complex tracing of things through the source ...) the easiest thing to do is isolate the Cython routine, which it seems I was able to do only because I wrote it and which seems quite buried in the code, and to also provide high-level machinery for faking a frontend.
-Matt
- Casey
On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz <scopatz@gmail.com
wrote:
Helo Casey,
Sorry for taking the whole weekend to respond.
>> I would like to help with this, but it's difficult to figure out >> where >> to start.
Not to worry. I think that any of the items listed at the bottom of Matt's original email would be a great place to start.
>> >> >> Say I want to test projections. I make a fake 3D density field,
maybe
>> something as simple as np.arange(4**3).reshape((4, 4, 4)). I write >> down the >> answer to the x-projection. Now all I need to do is call >> assert_allclose(yt_result, answer, rtol=1e-15), but I don't know
what
>> pieces >> of low-level yt stuff to call to get to `yt_result`. Hopefully
that's
>> clear... >> >> Maybe this comes down to creating a fake frontend we can attach >> fields >> to?
Actually, I disagree with this strategy, as I told Matt when we
spoke
last week. What is important is that we test the science and math parts of the code before, if ever, dealing with the software architecture that
surrounds
them.
Let's taking your example of projections. What we need to test is
the
actual function or method which actually slogs through the projection calculation.
In
many cases in yt these functions are not directly attached to the front end but
live
in analysis, visualization or utilities subpackages. It is these such packages that we should worry about testing. We can easily create routines to feed them sample data.
On the other hand, testing or mocking things like frontends should
be a
very low priority. At the end of the day what you are testing here is pulling in data
from
disk or other sources. Effectively, this is just re-testing functionality
present in
h5py, etc. That is not really our job. Yes, in a perfect world, front ends would be tested too. But I think that the priority should be placed on things like the KDTree.
Be Well Anthony
>> >> >> - Casey >> >> >> On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk <
matthewturk@gmail.com>
>> wrote: >>> >>> Hi all, >>> >>> As some of you have seen (at least Stephen), I filed a ticket
this
>>> morning about increasing testing coverage. The other night
Anthony
>>> and I met up in NYC and he had something of an "intervention"
about
>>> the sufficiency of answer testing for yt; it didn't take too much >>> work >>> on his part to convince me that we should be testing not just >>> against >>> a gold standard, but also performing unit tests. In the past I
had
>>> eschewed unit testing simply because the task of mocking data was >>> quite tricky, and by adding tests that use smaller bits we could >>> cover >>> unit testable areas with answer testing. >>> >>> But, this isn't really a good strategy. Let's move to having
both.
>>> The testing infrastructure he recommends is the
nearly-omnipresent
>>> nose: >>> >>> http://nose.readthedocs.org/en/latest/ >>> >>> The ticket to track this is here: >>> >>> >>> >>>
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
>>> >>> There are a couple sub-items here: >>> >>> 1) NumPy's nose test plugins provide a lot of necessary >>> functionality >>> that we have reimplemented in the answer testing utilities. I'd >>> like >>> to start using the numpy plugins, which include things like >>> conditional test execution, array comparisons, "slow" tests, etc >>> etc. >>> 2) We can evaluate, using conditional test execution, moving to
nose
>>> for answer testing. But that's not on the agenda now. >>> 3) Writing tests for nose is super easy, and running them is too. >>> Just >>> do: >>> >>> nosetest -w yt/ >>> >>> when in your source directory. >>> >>> 4) I've written a simple sample here: >>> >>> >>> >>>
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
>>> >>> 5) I'll handle writing up some mock data that doesn't require >>> shipping >>> lots of binary files, which can then be used for checking things >>> that >>> absolutely require hierarchies. >>> >>> -- >>> >>> The way to organize tests is easy. Inside each directory with >>> testable items create a new directory called "tests", and in here >>> toss >>> some scripts. You can stick a bunch of functions in those
scripts.
>>> >>> Anyway, I'm going to start writing more of these (in the main yt >>> repo, >>> and this change will be grafted there as well) and I'll write
back
>>> once the data mocking is ready. I'd like it if we started >>> encouraging >>> or even mandating simple tests (and/or answer tests) for >>> functionality >>> that gets added, but that's a discussion that should be held >>> separately. >>> >>> The items on the ticket: >>> >>> * kD-tree for nearest neighbor >>> * Geometric selection routines >>> * Profiles >>> * Projections -- underlying quadtree >>> * Data object selection of data containers >>> * Data object selection of points >>> * Orientation class >>> * Pixelization >>> * Color maps >>> * PNG writing >>> >>> Is anyone willing to claim any additional items that they will
help
>>> write unit tests for? >>> >>> -Matt >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi Casey,
The reason I suggested that we do it this way originally was because this is the way that most other packages in the scientific python ecosystem are set up. If you look at numpy, scipy, cython, pytables, ipython, etc. they all have the following file system structure:
root/ module1.py module2.py tests/ test_module2.py test_module1.py subpack1/ module3.py tests/ test_module3.py
I think that it is done this way to make it easy/possible for UnitTest and nose to autodetect the tests. I am not saying that we have to follow this de facto standard. However, I don't really see a reason why we shouldn't either. If you do, please let me know ;)
Notably, matplotlib does not follow this standard. But then again, since they are testing GUI and rendering code their tests are a level of complexity above what we need.
Be Well Anthony
On Tue, Sep 25, 2012 at 5:50 PM, Casey W. Stark caseywstark@gmail.comwrote:
Hey Matt, Anthony.
Looks like a good start to nosetests. One small thing -- would anyone be opposed to moving these out of the yt directory and into a root "test" directory?
I can take on CIC deposition and interpolation wherever those are.
- Casey
On Tue, Sep 25, 2012 at 11:20 AM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
As a note, Anthony just added this to the 3.0 base. Here's a set of tests:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/utilities/lib/t...
and here's a utilities section he added:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/testing.py
-Matt
On Mon, Sep 24, 2012 at 10:01 PM, Matthew Turk matthewturk@gmail.com wrote:
Hey Casey,
On Mon, Sep 24, 2012 at 5:39 PM, Casey W. Stark caseywstark@gmail.com
wrote:
Hi Matt.
Glad my example was useful in some way. I guess knowing exactly which
Cython
routines to test for what is what I meant about where to start.
I completely understand -- and that's a failure of my part in the code. So I think what might help is if we start looking at the routines that could be tested, and then I can jump in when it looks like they're buried. I will also volunteer to try to refactor this code, once it's tested, to make it clearer what is where and why.
Thanks for the tip about the stream frontend.
No prob, it should be documented better.
In terms of generating data, here's an example of what I was meaning:
https://hub.yt-project.org/nb/ompxtg
This sets up a random (but small-valued) background density and then applies a couple top hat spheres on top of it. I think this could be a good starting point for unit testing. I looked at the code that exists for flagging cells and refining them, and it's currently a bit too specific for this, as it was designed to take RAMSES data and partition it, which operates slightly differently. I'll take a pass at extracting it and making it work in this case.
The way I see it working is that one would set up the operators, as is done here, and then a progressive routine would be applied, something like this:
while grids.flag() > 0: grids.refine() for operator in operators: operator.apply(grids) pf = grids.convert_to_dataset()
The missing step in the code base is mostly just the refinement, as for the smoothed covering grids we have the necessary machinery to interpolate from one level to the next. Adding more types of operators to this library would be beneficial as well. The finalization step at the end would then convert the collection of grids into a Stream dataset. So with only a couple lines of code we could in all likelihood be able to generate in-memory datasets.
(This underscores in my mind why it would be awesome to have GDF in more places, as with this kind of machinery we've just written initial conditions generation, especially since we have a GDF writer already...)
Anyway, what might be really helpful is if interested people would volunteer to address a few of the testable areas? Then we can start pushing and identifying problem areas. I'll volunteer to handle at least the data container stuff, and whatever else slips through the cracks; although my time will be somewhat limited in the very near future, I will try to make this a priority.
-Matt
- Casey
On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk matthewturk@gmail.com
wrote:
Hey Casey and Anthony,
On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark <
caseywstark@gmail.com>
wrote:
Hi Anthony.
I completely agree that we should target the level of functions
actually
performing the projection rather than yt's organization. The mock frontend suggestion was just a hack to get there. I don't know if there's a
way
around it though...
Here's an example of what I sorted through to get to projections:
- Load a test plotfile, check pf.h.proj to find it's source.
- Read through data_objects/hierarchy.py and
utilities/parallel_tools/parallel_analysis_interface.py to find
where
proj is attached, can't find it.
- The proj docstring says it is a reference to AMRQuadProj. Can't
find a
class by that name.
- Search data_objects sources for "proj", find AMRProjBase.
So it looks like the functionality is wrapped up in the
__project_level
and _project_grid methods. I can't think of a way to test those without creating an AMRProjBase, and that requires a staticoutput object.
You're right, the projection stuff as *projections* is not easy to test. But in terms of testing the underlying code, which is wrapped up in a Cython class called QuadTree, I think it could be done. The steps you're describing are actually all part of the existing answer testing machinery, which performs a couple things and verifies that they don't change over time:
- Project some fields from the disk
- Project a couple derived fields
- Project a derived field that requires spatial derivatives
- Project the "Ones" field, which should be 1.0 everywhere.
So these things are done, but it is also possible that the specific quadtree functionality could be tested, in isolation from the projection. I think this may be oneo f the things Anthony is talking about -- answer testing can handle the big, complex items, and by breaking down to the fundamentals we can address isolated items from a unit testing perspective.
So unfortunately, I think it would still come down to having a fake frontend. It's not ideal, but it seems like any more isolation would require big rewrites to yt.
One fun thing that is not usually known is that we have a fake frontend already, it just doesn't get used much. It's called the "Stream" frontend and it was designed originally to be used in ParaView, but now gets used by the (new, not-yet-documented/released) load_uniform_grid function as well as by Hyperion, the RT code by Tom R. It can set up AMR as well as static mesh. It's not terribly well documented, but there are examples on the wiki.
One thing I've been thinking about is actually creating a couple fake outputs, which could be defined analytically with spheres of overdensity inside them. In principle, if we added refinement criteria, we could make this relatively complex data that was defined with only a few lines of code, but spun up a big in-memory dataset.
(This exact thing is on my list of things to do and then to output in GDF, by the way...)
That I think could come, down the road a bit. The refinement criteria wouldn't be too bad to implement, especially since we already have the grid splitting routines. I just don't think we should focus on it at the moment. But the uniform grid creation and loading works already -- I used it this morning. You can do it with:
from yt.frontends.stream.api import load_uniform_grid ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2}, [359, 359, 359], 1.0)
the list is the dimensions of the data and the value is the to-cm conversion.
Of course, I could be missing something. Matt, can you think of a
better
way?
I think for this specific example (and your damningly complex tracing of things through the source ...) the easiest thing to do is isolate the Cython routine, which it seems I was able to do only because I wrote it and which seems quite buried in the code, and to also provide high-level machinery for faking a frontend.
-Matt
- Casey
On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz <
scopatz@gmail.com>
wrote: > > Helo Casey, > > Sorry for taking the whole weekend to respond. > >>> I would like to help with this, but it's difficult to figure out >>> where >>> to start. > > > Not to worry. I think that any of the items listed at the bottom of > Matt's > original email > would be a great place to start. > >>> >>> >>> Say I want to test projections. I make a fake 3D density field,
maybe
>>> something as simple as np.arange(4**3).reshape((4, 4, 4)). I
write
>>> down the >>> answer to the x-projection. Now all I need to do is call >>> assert_allclose(yt_result, answer, rtol=1e-15), but I don't know
what
>>> pieces >>> of low-level yt stuff to call to get to `yt_result`. Hopefully
that's
>>> clear... >>> >>> Maybe this comes down to creating a fake frontend we can attach >>> fields >>> to? > > > Actually, I disagree with this strategy, as I told Matt when we
spoke
> last > week. > What is important is that we test the science and math parts of the > code > before, if ever, dealing with the software architecture that
surrounds
> them. > > Let's taking your example of projections. What we need to test is
the
> actual function > or method which actually slogs through the projection calculation.
In
> many cases in > yt these functions are not directly attached to the front end but
live
> in > analysis, visualization > or utilities subpackages. It is these such packages that we
should
> worry > about testing. > We can easily create routines to feed them sample data. > > On the other hand, testing or mocking things like frontends should
be a
> very low priority. > At the end of the day what you are testing here is pulling in data
from
> disk or other > sources. Effectively, this is just re-testing functionality
present in
> h5py, etc. That is not > really our job. Yes, in a perfect world, front ends would be
tested
> too. > But I think that the > priority should be placed on things like the KDTree. > > Be Well > Anthony > >>> >>> >>> - Casey >>> >>> >>> On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk <
matthewturk@gmail.com>
>>> wrote: >>>> >>>> Hi all, >>>> >>>> As some of you have seen (at least Stephen), I filed a ticket
this
>>>> morning about increasing testing coverage. The other night
Anthony
>>>> and I met up in NYC and he had something of an "intervention"
about
>>>> the sufficiency of answer testing for yt; it didn't take too
much
>>>> work >>>> on his part to convince me that we should be testing not just >>>> against >>>> a gold standard, but also performing unit tests. In the past I
had
>>>> eschewed unit testing simply because the task of mocking data
was
>>>> quite tricky, and by adding tests that use smaller bits we could >>>> cover >>>> unit testable areas with answer testing. >>>> >>>> But, this isn't really a good strategy. Let's move to having
both.
>>>> The testing infrastructure he recommends is the
nearly-omnipresent
>>>> nose: >>>> >>>> http://nose.readthedocs.org/en/latest/ >>>> >>>> The ticket to track this is here: >>>> >>>> >>>> >>>>
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
>>>> >>>> There are a couple sub-items here: >>>> >>>> 1) NumPy's nose test plugins provide a lot of necessary >>>> functionality >>>> that we have reimplemented in the answer testing utilities. I'd >>>> like >>>> to start using the numpy plugins, which include things like >>>> conditional test execution, array comparisons, "slow" tests, etc >>>> etc. >>>> 2) We can evaluate, using conditional test execution, moving to
nose
>>>> for answer testing. But that's not on the agenda now. >>>> 3) Writing tests for nose is super easy, and running them is
too.
>>>> Just >>>> do: >>>> >>>> nosetest -w yt/ >>>> >>>> when in your source directory. >>>> >>>> 4) I've written a simple sample here: >>>> >>>> >>>> >>>>
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
>>>> >>>> 5) I'll handle writing up some mock data that doesn't require >>>> shipping >>>> lots of binary files, which can then be used for checking things >>>> that >>>> absolutely require hierarchies. >>>> >>>> -- >>>> >>>> The way to organize tests is easy. Inside each directory with >>>> testable items create a new directory called "tests", and in
here
>>>> toss >>>> some scripts. You can stick a bunch of functions in those
scripts.
>>>> >>>> Anyway, I'm going to start writing more of these (in the main yt >>>> repo, >>>> and this change will be grafted there as well) and I'll write
back
>>>> once the data mocking is ready. I'd like it if we started >>>> encouraging >>>> or even mandating simple tests (and/or answer tests) for >>>> functionality >>>> that gets added, but that's a discussion that should be held >>>> separately. >>>> >>>> The items on the ticket: >>>> >>>> * kD-tree for nearest neighbor >>>> * Geometric selection routines >>>> * Profiles >>>> * Projections -- underlying quadtree >>>> * Data object selection of data containers >>>> * Data object selection of points >>>> * Orientation class >>>> * Pixelization >>>> * Color maps >>>> * PNG writing >>>> >>>> Is anyone willing to claim any additional items that they will
help
>>>> write unit tests for? >>>> >>>> -Matt >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >>> >>> >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Would root in our case be yt-hg/yt, in which case the tests would live at yt-hg/yt/tests/ ?
Sorry if I misunderstood. Sam
On Tue, Sep 25, 2012 at 5:00 PM, Anthony Scopatz scopatz@gmail.com wrote:
Hi Casey,
The reason I suggested that we do it this way originally was because this is the way that most other packages in the scientific python ecosystem are set up. If you look at numpy, scipy, cython, pytables, ipython, etc. they all have the following file system structure:
root/ module1.py module2.py tests/ test_module2.py test_module1.py subpack1/ module3.py tests/ test_module3.py
I think that it is done this way to make it easy/possible for UnitTest and nose to autodetect the tests. I am not saying that we have to follow this de facto standard. However, I don't really see a reason why we shouldn't either. If you do, please let me know ;)
Notably, matplotlib does not follow this standard. But then again, since they are testing GUI and rendering code their tests are a level of complexity above what we need.
Be Well Anthony
On Tue, Sep 25, 2012 at 5:50 PM, Casey W. Stark caseywstark@gmail.comwrote:
Hey Matt, Anthony.
Looks like a good start to nosetests. One small thing -- would anyone be opposed to moving these out of the yt directory and into a root "test" directory?
I can take on CIC deposition and interpolation wherever those are.
- Casey
On Tue, Sep 25, 2012 at 11:20 AM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
As a note, Anthony just added this to the 3.0 base. Here's a set of tests:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/utilities/lib/t...
and here's a utilities section he added:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/testing.py
-Matt
On Mon, Sep 24, 2012 at 10:01 PM, Matthew Turk matthewturk@gmail.com wrote:
Hey Casey,
On Mon, Sep 24, 2012 at 5:39 PM, Casey W. Stark caseywstark@gmail.com
wrote:
Hi Matt.
Glad my example was useful in some way. I guess knowing exactly which
Cython
routines to test for what is what I meant about where to start.
I completely understand -- and that's a failure of my part in the code. So I think what might help is if we start looking at the routines that could be tested, and then I can jump in when it looks like they're buried. I will also volunteer to try to refactor this code, once it's tested, to make it clearer what is where and why.
Thanks for the tip about the stream frontend.
No prob, it should be documented better.
In terms of generating data, here's an example of what I was meaning:
https://hub.yt-project.org/nb/ompxtg
This sets up a random (but small-valued) background density and then applies a couple top hat spheres on top of it. I think this could be a good starting point for unit testing. I looked at the code that exists for flagging cells and refining them, and it's currently a bit too specific for this, as it was designed to take RAMSES data and partition it, which operates slightly differently. I'll take a pass at extracting it and making it work in this case.
The way I see it working is that one would set up the operators, as is done here, and then a progressive routine would be applied, something like this:
while grids.flag() > 0: grids.refine() for operator in operators: operator.apply(grids) pf = grids.convert_to_dataset()
The missing step in the code base is mostly just the refinement, as for the smoothed covering grids we have the necessary machinery to interpolate from one level to the next. Adding more types of operators to this library would be beneficial as well. The finalization step at the end would then convert the collection of grids into a Stream dataset. So with only a couple lines of code we could in all likelihood be able to generate in-memory datasets.
(This underscores in my mind why it would be awesome to have GDF in more places, as with this kind of machinery we've just written initial conditions generation, especially since we have a GDF writer already...)
Anyway, what might be really helpful is if interested people would volunteer to address a few of the testable areas? Then we can start pushing and identifying problem areas. I'll volunteer to handle at least the data container stuff, and whatever else slips through the cracks; although my time will be somewhat limited in the very near future, I will try to make this a priority.
-Matt
- Casey
On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk matthewturk@gmail.com
wrote:
Hey Casey and Anthony,
On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark <
caseywstark@gmail.com>
wrote: > Hi Anthony. > > I completely agree that we should target the level of functions
actually
> performing the projection rather than yt's organization. The mock > frontend > suggestion was just a hack to get there. I don't know if there's a
way
> around it though... > > Here's an example of what I sorted through to get to projections: > - Load a test plotfile, check pf.h.proj to find it's source. > - Read through data_objects/hierarchy.py and > utilities/parallel_tools/parallel_analysis_interface.py to find
where
> proj > is attached, can't find it. > - The proj docstring says it is a reference to AMRQuadProj. Can't
find a
> class by that name. > - Search data_objects sources for "proj", find AMRProjBase. > > So it looks like the functionality is wrapped up in the
__project_level
> and > _project_grid methods. I can't think of a way to test those without > creating > an AMRProjBase, and that requires a staticoutput object.
You're right, the projection stuff as *projections* is not easy to test. But in terms of testing the underlying code, which is wrapped up in a Cython class called QuadTree, I think it could be done. The steps you're describing are actually all part of the existing answer testing machinery, which performs a couple things and verifies that they don't change over time:
- Project some fields from the disk
- Project a couple derived fields
- Project a derived field that requires spatial derivatives
- Project the "Ones" field, which should be 1.0 everywhere.
So these things are done, but it is also possible that the specific quadtree functionality could be tested, in isolation from the projection. I think this may be oneo f the things Anthony is talking about -- answer testing can handle the big, complex items, and by breaking down to the fundamentals we can address isolated items from
a
unit testing perspective.
> > So unfortunately, I think it would still come down to having a fake > frontend. It's not ideal, but it seems like any more isolation
would
> require > big rewrites to yt.
One fun thing that is not usually known is that we have a fake frontend already, it just doesn't get used much. It's called the "Stream" frontend and it was designed originally to be used in ParaView, but now gets used by the (new, not-yet-documented/released) load_uniform_grid function as well as by Hyperion, the RT code by Tom R. It can set up AMR as well as static mesh. It's not terribly well documented, but there are examples on the wiki.
One thing I've been thinking about is actually creating a couple fake outputs, which could be defined analytically with spheres of overdensity inside them. In principle, if we added refinement criteria, we could make this relatively complex data that was defined with only a few lines of code, but spun up a big in-memory dataset.
(This exact thing is on my list of things to do and then to output in GDF, by the way...)
That I think could come, down the road a bit. The refinement
criteria
wouldn't be too bad to implement, especially since we already have
the
grid splitting routines. I just don't think we should focus on it at the moment. But the uniform grid creation and loading works already -- I used it this morning. You can do it with:
from yt.frontends.stream.api import load_uniform_grid ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2}, [359, 359, 359], 1.0)
the list is the dimensions of the data and the value is the to-cm conversion.
> > Of course, I could be missing something. Matt, can you think of a
better
> way?
I think for this specific example (and your damningly complex tracing of things through the source ...) the easiest thing to do is isolate the Cython routine, which it seems I was able to do only because I wrote it and which seems quite buried in the code, and to also
provide
high-level machinery for faking a frontend.
-Matt
> > - Casey > > > On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz <
scopatz@gmail.com>
> wrote: >> >> Helo Casey, >> >> Sorry for taking the whole weekend to respond. >> >>>> I would like to help with this, but it's difficult to figure out >>>> where >>>> to start. >> >> >> Not to worry. I think that any of the items listed at the bottom
of
>> Matt's >> original email >> would be a great place to start. >> >>>> >>>> >>>> Say I want to test projections. I make a fake 3D density field,
maybe
>>>> something as simple as np.arange(4**3).reshape((4, 4, 4)). I
write
>>>> down the >>>> answer to the x-projection. Now all I need to do is call >>>> assert_allclose(yt_result, answer, rtol=1e-15), but I don't
know what
>>>> pieces >>>> of low-level yt stuff to call to get to `yt_result`. Hopefully
that's
>>>> clear... >>>> >>>> Maybe this comes down to creating a fake frontend we can attach >>>> fields >>>> to? >> >> >> Actually, I disagree with this strategy, as I told Matt when we
spoke
>> last >> week. >> What is important is that we test the science and math parts of
the
>> code >> before, if ever, dealing with the software architecture that
surrounds
>> them. >> >> Let's taking your example of projections. What we need to test
is the
>> actual function >> or method which actually slogs through the projection
calculation. In
>> many cases in >> yt these functions are not directly attached to the front end but
live
>> in >> analysis, visualization >> or utilities subpackages. It is these such packages that we
should
>> worry >> about testing. >> We can easily create routines to feed them sample data. >> >> On the other hand, testing or mocking things like frontends
should be a
>> very low priority. >> At the end of the day what you are testing here is pulling in
data from
>> disk or other >> sources. Effectively, this is just re-testing functionality
present in
>> h5py, etc. That is not >> really our job. Yes, in a perfect world, front ends would be
tested
>> too. >> But I think that the >> priority should be placed on things like the KDTree. >> >> Be Well >> Anthony >> >>>> >>>> >>>> - Casey >>>> >>>> >>>> On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk <
matthewturk@gmail.com>
>>>> wrote: >>>>> >>>>> Hi all, >>>>> >>>>> As some of you have seen (at least Stephen), I filed a ticket
this
>>>>> morning about increasing testing coverage. The other night
Anthony
>>>>> and I met up in NYC and he had something of an "intervention"
about
>>>>> the sufficiency of answer testing for yt; it didn't take too
much
>>>>> work >>>>> on his part to convince me that we should be testing not just >>>>> against >>>>> a gold standard, but also performing unit tests. In the past
I had
>>>>> eschewed unit testing simply because the task of mocking data
was
>>>>> quite tricky, and by adding tests that use smaller bits we
could
>>>>> cover >>>>> unit testable areas with answer testing. >>>>> >>>>> But, this isn't really a good strategy. Let's move to having
both.
>>>>> The testing infrastructure he recommends is the
nearly-omnipresent
>>>>> nose: >>>>> >>>>> http://nose.readthedocs.org/en/latest/ >>>>> >>>>> The ticket to track this is here: >>>>> >>>>> >>>>> >>>>>
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
>>>>> >>>>> There are a couple sub-items here: >>>>> >>>>> 1) NumPy's nose test plugins provide a lot of necessary >>>>> functionality >>>>> that we have reimplemented in the answer testing utilities.
I'd
>>>>> like >>>>> to start using the numpy plugins, which include things like >>>>> conditional test execution, array comparisons, "slow" tests,
etc
>>>>> etc. >>>>> 2) We can evaluate, using conditional test execution, moving
to nose
>>>>> for answer testing. But that's not on the agenda now. >>>>> 3) Writing tests for nose is super easy, and running them is
too.
>>>>> Just >>>>> do: >>>>> >>>>> nosetest -w yt/ >>>>> >>>>> when in your source directory. >>>>> >>>>> 4) I've written a simple sample here: >>>>> >>>>> >>>>> >>>>>
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
>>>>> >>>>> 5) I'll handle writing up some mock data that doesn't require >>>>> shipping >>>>> lots of binary files, which can then be used for checking
things
>>>>> that >>>>> absolutely require hierarchies. >>>>> >>>>> -- >>>>> >>>>> The way to organize tests is easy. Inside each directory with >>>>> testable items create a new directory called "tests", and in
here
>>>>> toss >>>>> some scripts. You can stick a bunch of functions in those
scripts.
>>>>> >>>>> Anyway, I'm going to start writing more of these (in the main
yt
>>>>> repo, >>>>> and this change will be grafted there as well) and I'll write
back
>>>>> once the data mocking is ready. I'd like it if we started >>>>> encouraging >>>>> or even mandating simple tests (and/or answer tests) for >>>>> functionality >>>>> that gets added, but that's a discussion that should be held >>>>> separately. >>>>> >>>>> The items on the ticket: >>>>> >>>>> * kD-tree for nearest neighbor >>>>> * Geometric selection routines >>>>> * Profiles >>>>> * Projections -- underlying quadtree >>>>> * Data object selection of data containers >>>>> * Data object selection of points >>>>> * Orientation class >>>>> * Pixelization >>>>> * Color maps >>>>> * PNG writing >>>>> >>>>> Is anyone willing to claim any additional items that they will
help
>>>>> write unit tests for? >>>>> >>>>> -Matt >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hey Sam.
I was suggesting...
yt-hg/ test/ yt/
So that it is separate from the yt package.
- Casey
On Tue, Sep 25, 2012 at 4:05 PM, Sam Skillman samskillman@gmail.com wrote:
Would root in our case be yt-hg/yt, in which case the tests would live at yt-hg/yt/tests/ ?
Sorry if I misunderstood. Sam
On Tue, Sep 25, 2012 at 5:00 PM, Anthony Scopatz scopatz@gmail.comwrote:
Hi Casey,
The reason I suggested that we do it this way originally was because this is the way that most other packages in the scientific python ecosystem are set up. If you look at numpy, scipy, cython, pytables, ipython, etc. they all have the following file system structure:
root/ module1.py module2.py tests/ test_module2.py test_module1.py subpack1/ module3.py tests/ test_module3.py
I think that it is done this way to make it easy/possible for UnitTest and nose to autodetect the tests. I am not saying that we have to follow this de facto standard. However, I don't really see a reason why we shouldn't either. If you do, please let me know ;)
Notably, matplotlib does not follow this standard. But then again, since they are testing GUI and rendering code their tests are a level of complexity above what we need.
Be Well Anthony
On Tue, Sep 25, 2012 at 5:50 PM, Casey W. Stark caseywstark@gmail.comwrote:
Hey Matt, Anthony.
Looks like a good start to nosetests. One small thing -- would anyone be opposed to moving these out of the yt directory and into a root "test" directory?
I can take on CIC deposition and interpolation wherever those are.
- Casey
On Tue, Sep 25, 2012 at 11:20 AM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
As a note, Anthony just added this to the 3.0 base. Here's a set of tests:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/utilities/lib/t...
and here's a utilities section he added:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/testing.py
-Matt
On Mon, Sep 24, 2012 at 10:01 PM, Matthew Turk matthewturk@gmail.com wrote:
Hey Casey,
On Mon, Sep 24, 2012 at 5:39 PM, Casey W. Stark <
caseywstark@gmail.com> wrote:
Hi Matt.
Glad my example was useful in some way. I guess knowing exactly
which Cython
routines to test for what is what I meant about where to start.
I completely understand -- and that's a failure of my part in the code. So I think what might help is if we start looking at the routines that could be tested, and then I can jump in when it looks like they're buried. I will also volunteer to try to refactor this code, once it's tested, to make it clearer what is where and why.
Thanks for the tip about the stream frontend.
No prob, it should be documented better.
In terms of generating data, here's an example of what I was meaning:
https://hub.yt-project.org/nb/ompxtg
This sets up a random (but small-valued) background density and then applies a couple top hat spheres on top of it. I think this could be a good starting point for unit testing. I looked at the code that exists for flagging cells and refining them, and it's currently a bit too specific for this, as it was designed to take RAMSES data and partition it, which operates slightly differently. I'll take a pass at extracting it and making it work in this case.
The way I see it working is that one would set up the operators, as is done here, and then a progressive routine would be applied, something like this:
while grids.flag() > 0: grids.refine() for operator in operators: operator.apply(grids) pf = grids.convert_to_dataset()
The missing step in the code base is mostly just the refinement, as for the smoothed covering grids we have the necessary machinery to interpolate from one level to the next. Adding more types of operators to this library would be beneficial as well. The finalization step at the end would then convert the collection of grids into a Stream dataset. So with only a couple lines of code we could in all likelihood be able to generate in-memory datasets.
(This underscores in my mind why it would be awesome to have GDF in more places, as with this kind of machinery we've just written initial conditions generation, especially since we have a GDF writer already...)
Anyway, what might be really helpful is if interested people would volunteer to address a few of the testable areas? Then we can start pushing and identifying problem areas. I'll volunteer to handle at least the data container stuff, and whatever else slips through the cracks; although my time will be somewhat limited in the very near future, I will try to make this a priority.
-Matt
- Casey
On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk matthewturk@gmail.com
wrote:
> > Hey Casey and Anthony, > > On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark <
caseywstark@gmail.com>
> wrote: > > Hi Anthony. > > > > I completely agree that we should target the level of functions
actually
> > performing the projection rather than yt's organization. The mock > > frontend > > suggestion was just a hack to get there. I don't know if there's
a way
> > around it though... > > > > Here's an example of what I sorted through to get to projections: > > - Load a test plotfile, check pf.h.proj to find it's source. > > - Read through data_objects/hierarchy.py and > > utilities/parallel_tools/parallel_analysis_interface.py to find
where
> > proj > > is attached, can't find it. > > - The proj docstring says it is a reference to AMRQuadProj. Can't
find a
> > class by that name. > > - Search data_objects sources for "proj", find AMRProjBase. > > > > So it looks like the functionality is wrapped up in the
__project_level
> > and > > _project_grid methods. I can't think of a way to test those
without
> > creating > > an AMRProjBase, and that requires a staticoutput object. > > You're right, the projection stuff as *projections* is not easy to > test. But in terms of testing the underlying code, which is wrapped > up in a Cython class called QuadTree, I think it could be done. The > steps you're describing are actually all part of the existing answer > testing machinery, which performs a couple things and verifies that > they don't change over time: > > 1) Project some fields from the disk > 2) Project a couple derived fields > 3) Project a derived field that requires spatial derivatives > 4) Project the "Ones" field, which should be 1.0 everywhere. > > So these things are done, but it is also possible that the specific > quadtree functionality could be tested, in isolation from the > projection. I think this may be oneo f the things Anthony is
talking
> about -- answer testing can handle the big, complex items, and by > breaking down to the fundamentals we can address isolated items
from a
> unit testing perspective. > > > > > So unfortunately, I think it would still come down to having a
fake
> > frontend. It's not ideal, but it seems like any more isolation
would
> > require > > big rewrites to yt. > > One fun thing that is not usually known is that we have a fake > frontend already, it just doesn't get used much. It's called the > "Stream" frontend and it was designed originally to be used in > ParaView, but now gets used by the (new,
not-yet-documented/released)
> load_uniform_grid function as well as by Hyperion, the RT code by
Tom
> R. It can set up AMR as well as static mesh. It's not terribly
well
> documented, but there are examples on the wiki. > > One thing I've been thinking about is actually creating a couple
fake
> outputs, which could be defined analytically with spheres of > overdensity inside them. In principle, if we added refinement > criteria, we could make this relatively complex data that was
defined
> with only a few lines of code, but spun up a big in-memory dataset. > > (This exact thing is on my list of things to do and then to output
in
> GDF, by the way...) > > That I think could come, down the road a bit. The refinement
criteria
> wouldn't be too bad to implement, especially since we already have
the
> grid splitting routines. I just don't think we should focus on it
at
> the moment. But the uniform grid creation and loading works already > -- I used it this morning. You can do it with: > > from yt.frontends.stream.api import load_uniform_grid > ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2},
[359,
> 359, 359], 1.0) > > the list is the dimensions of the data and the value is the to-cm > conversion. > > > > > Of course, I could be missing something. Matt, can you think of a
better
> > way? > > I think for this specific example (and your damningly complex
tracing
> of things through the source ...) the easiest thing to do is isolate > the Cython routine, which it seems I was able to do only because I > wrote it and which seems quite buried in the code, and to also
provide
> high-level machinery for faking a frontend. > > -Matt > > > > > - Casey > > > > > > On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz <
scopatz@gmail.com>
> > wrote: > >> > >> Helo Casey, > >> > >> Sorry for taking the whole weekend to respond. > >> > >>>> I would like to help with this, but it's difficult to figure
out
> >>>> where > >>>> to start. > >> > >> > >> Not to worry. I think that any of the items listed at the bottom
of
> >> Matt's > >> original email > >> would be a great place to start. > >> > >>>> > >>>> > >>>> Say I want to test projections. I make a fake 3D density
field, maybe
> >>>> something as simple as np.arange(4**3).reshape((4, 4, 4)). I
write
> >>>> down the > >>>> answer to the x-projection. Now all I need to do is call > >>>> assert_allclose(yt_result, answer, rtol=1e-15), but I don't
know what
> >>>> pieces > >>>> of low-level yt stuff to call to get to `yt_result`. Hopefully
that's
> >>>> clear... > >>>> > >>>> Maybe this comes down to creating a fake frontend we can attach > >>>> fields > >>>> to? > >> > >> > >> Actually, I disagree with this strategy, as I told Matt when we
spoke
> >> last > >> week. > >> What is important is that we test the science and math parts of
the
> >> code > >> before, if ever, dealing with the software architecture that
surrounds
> >> them. > >> > >> Let's taking your example of projections. What we need to test
is the
> >> actual function > >> or method which actually slogs through the projection
calculation. In
> >> many cases in > >> yt these functions are not directly attached to the front end
but live
> >> in > >> analysis, visualization > >> or utilities subpackages. It is these such packages that we
should
> >> worry > >> about testing. > >> We can easily create routines to feed them sample data. > >> > >> On the other hand, testing or mocking things like frontends
should be a
> >> very low priority. > >> At the end of the day what you are testing here is pulling in
data from
> >> disk or other > >> sources. Effectively, this is just re-testing functionality
present in
> >> h5py, etc. That is not > >> really our job. Yes, in a perfect world, front ends would be
tested
> >> too. > >> But I think that the > >> priority should be placed on things like the KDTree. > >> > >> Be Well > >> Anthony > >> > >>>> > >>>> > >>>> - Casey > >>>> > >>>> > >>>> On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk <
matthewturk@gmail.com>
> >>>> wrote: > >>>>> > >>>>> Hi all, > >>>>> > >>>>> As some of you have seen (at least Stephen), I filed a ticket
this
> >>>>> morning about increasing testing coverage. The other night
Anthony
> >>>>> and I met up in NYC and he had something of an "intervention"
about
> >>>>> the sufficiency of answer testing for yt; it didn't take too
much
> >>>>> work > >>>>> on his part to convince me that we should be testing not just > >>>>> against > >>>>> a gold standard, but also performing unit tests. In the past
I had
> >>>>> eschewed unit testing simply because the task of mocking data
was
> >>>>> quite tricky, and by adding tests that use smaller bits we
could
> >>>>> cover > >>>>> unit testable areas with answer testing. > >>>>> > >>>>> But, this isn't really a good strategy. Let's move to having
both.
> >>>>> The testing infrastructure he recommends is the
nearly-omnipresent
> >>>>> nose: > >>>>> > >>>>> http://nose.readthedocs.org/en/latest/ > >>>>> > >>>>> The ticket to track this is here: > >>>>> > >>>>> > >>>>> > >>>>>
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
> >>>>> > >>>>> There are a couple sub-items here: > >>>>> > >>>>> 1) NumPy's nose test plugins provide a lot of necessary > >>>>> functionality > >>>>> that we have reimplemented in the answer testing utilities.
I'd
> >>>>> like > >>>>> to start using the numpy plugins, which include things like > >>>>> conditional test execution, array comparisons, "slow" tests,
etc
> >>>>> etc. > >>>>> 2) We can evaluate, using conditional test execution, moving
to nose
> >>>>> for answer testing. But that's not on the agenda now. > >>>>> 3) Writing tests for nose is super easy, and running them is
too.
> >>>>> Just > >>>>> do: > >>>>> > >>>>> nosetest -w yt/ > >>>>> > >>>>> when in your source directory. > >>>>> > >>>>> 4) I've written a simple sample here: > >>>>> > >>>>> > >>>>> > >>>>>
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
> >>>>> > >>>>> 5) I'll handle writing up some mock data that doesn't require > >>>>> shipping > >>>>> lots of binary files, which can then be used for checking
things
> >>>>> that > >>>>> absolutely require hierarchies. > >>>>> > >>>>> -- > >>>>> > >>>>> The way to organize tests is easy. Inside each directory with > >>>>> testable items create a new directory called "tests", and in
here
> >>>>> toss > >>>>> some scripts. You can stick a bunch of functions in those
scripts.
> >>>>> > >>>>> Anyway, I'm going to start writing more of these (in the main
yt
> >>>>> repo, > >>>>> and this change will be grafted there as well) and I'll write
back
> >>>>> once the data mocking is ready. I'd like it if we started > >>>>> encouraging > >>>>> or even mandating simple tests (and/or answer tests) for > >>>>> functionality > >>>>> that gets added, but that's a discussion that should be held > >>>>> separately. > >>>>> > >>>>> The items on the ticket: > >>>>> > >>>>> * kD-tree for nearest neighbor > >>>>> * Geometric selection routines > >>>>> * Profiles > >>>>> * Projections -- underlying quadtree > >>>>> * Data object selection of data containers > >>>>> * Data object selection of points > >>>>> * Orientation class > >>>>> * Pixelization > >>>>> * Color maps > >>>>> * PNG writing > >>>>> > >>>>> Is anyone willing to claim any additional items that they
will help
> >>>>> write unit tests for? > >>>>> > >>>>> -Matt > >>>>> _______________________________________________ > >>>>> yt-dev mailing list > >>>>> yt-dev@lists.spacepope.org > >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> yt-dev mailing list > >>>> yt-dev@lists.spacepope.org > >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >>>> > >> > >> > >> _______________________________________________ > >> yt-dev mailing list > >> yt-dev@lists.spacepope.org > >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > >> > > > > > > _______________________________________________ > > yt-dev mailing list > > yt-dev@lists.spacepope.org > > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Gotcha, thanks for the clarification. I'm just trying to stay caught up on all the movement.
Having one tests/ folder per dir seems reasonable.
Sam
On Tue, Sep 25, 2012 at 5:12 PM, Casey W. Stark caseywstark@gmail.comwrote:
Hey Sam.
I was suggesting...
yt-hg/ test/ yt/
So that it is separate from the yt package.
- Casey
On Tue, Sep 25, 2012 at 4:05 PM, Sam Skillman samskillman@gmail.comwrote:
Would root in our case be yt-hg/yt, in which case the tests would live at yt-hg/yt/tests/ ?
Sorry if I misunderstood. Sam
On Tue, Sep 25, 2012 at 5:00 PM, Anthony Scopatz scopatz@gmail.comwrote:
Hi Casey,
The reason I suggested that we do it this way originally was because this is the way that most other packages in the scientific python ecosystem are set up. If you look at numpy, scipy, cython, pytables, ipython, etc. they all have the following file system structure:
root/ module1.py module2.py tests/ test_module2.py test_module1.py subpack1/ module3.py tests/ test_module3.py
I think that it is done this way to make it easy/possible for UnitTest and nose to autodetect the tests. I am not saying that we have to follow this de facto standard. However, I don't really see a reason why we shouldn't either. If you do, please let me know ;)
Notably, matplotlib does not follow this standard. But then again, since they are testing GUI and rendering code their tests are a level of complexity above what we need.
Be Well Anthony
On Tue, Sep 25, 2012 at 5:50 PM, Casey W. Stark caseywstark@gmail.comwrote:
Hey Matt, Anthony.
Looks like a good start to nosetests. One small thing -- would anyone be opposed to moving these out of the yt directory and into a root "test" directory?
I can take on CIC deposition and interpolation wherever those are.
- Casey
On Tue, Sep 25, 2012 at 11:20 AM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
As a note, Anthony just added this to the 3.0 base. Here's a set of tests:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/utilities/lib/t...
and here's a utilities section he added:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/testing.py
-Matt
On Mon, Sep 24, 2012 at 10:01 PM, Matthew Turk matthewturk@gmail.com wrote:
Hey Casey,
On Mon, Sep 24, 2012 at 5:39 PM, Casey W. Stark <
caseywstark@gmail.com> wrote:
> Hi Matt. > > Glad my example was useful in some way. I guess knowing exactly
which Cython
> routines to test for what is what I meant about where to start.
I completely understand -- and that's a failure of my part in the code. So I think what might help is if we start looking at the routines that could be tested, and then I can jump in when it looks like they're buried. I will also volunteer to try to refactor this code, once it's tested, to make it clearer what is where and why.
> > Thanks for the tip about the stream frontend.
No prob, it should be documented better.
In terms of generating data, here's an example of what I was meaning:
https://hub.yt-project.org/nb/ompxtg
This sets up a random (but small-valued) background density and then applies a couple top hat spheres on top of it. I think this could be a good starting point for unit testing. I looked at the code that exists for flagging cells and refining them, and it's currently a bit too specific for this, as it was designed to take RAMSES data and partition it, which operates slightly differently. I'll take a pass at extracting it and making it work in this case.
The way I see it working is that one would set up the operators, as
is
done here, and then a progressive routine would be applied, something like this:
while grids.flag() > 0: grids.refine() for operator in operators: operator.apply(grids) pf = grids.convert_to_dataset()
The missing step in the code base is mostly just the refinement, as for the smoothed covering grids we have the necessary machinery to interpolate from one level to the next. Adding more types of operators to this library would be beneficial as well. The finalization step at the end would then convert the collection of grids into a Stream dataset. So with only a couple lines of code we could in all likelihood be able to generate in-memory datasets.
(This underscores in my mind why it would be awesome to have GDF in more places, as with this kind of machinery we've just written
initial
conditions generation, especially since we have a GDF writer already...)
Anyway, what might be really helpful is if interested people would volunteer to address a few of the testable areas? Then we can start pushing and identifying problem areas. I'll volunteer to handle at least the data container stuff, and whatever else slips through the cracks; although my time will be somewhat limited in the very near future, I will try to make this a priority.
-Matt
> > - Casey > > > On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk <
matthewturk@gmail.com> wrote:
>> >> Hey Casey and Anthony, >> >> On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark <
caseywstark@gmail.com>
>> wrote: >> > Hi Anthony. >> > >> > I completely agree that we should target the level of functions
actually
>> > performing the projection rather than yt's organization. The mock >> > frontend >> > suggestion was just a hack to get there. I don't know if there's
a way
>> > around it though... >> > >> > Here's an example of what I sorted through to get to projections: >> > - Load a test plotfile, check pf.h.proj to find it's source. >> > - Read through data_objects/hierarchy.py and >> > utilities/parallel_tools/parallel_analysis_interface.py to find
where
>> > proj >> > is attached, can't find it. >> > - The proj docstring says it is a reference to AMRQuadProj.
Can't find a
>> > class by that name. >> > - Search data_objects sources for "proj", find AMRProjBase. >> > >> > So it looks like the functionality is wrapped up in the
__project_level
>> > and >> > _project_grid methods. I can't think of a way to test those
without
>> > creating >> > an AMRProjBase, and that requires a staticoutput object. >> >> You're right, the projection stuff as *projections* is not easy to >> test. But in terms of testing the underlying code, which is
wrapped
>> up in a Cython class called QuadTree, I think it could be done.
The
>> steps you're describing are actually all part of the existing
answer
>> testing machinery, which performs a couple things and verifies that >> they don't change over time: >> >> 1) Project some fields from the disk >> 2) Project a couple derived fields >> 3) Project a derived field that requires spatial derivatives >> 4) Project the "Ones" field, which should be 1.0 everywhere. >> >> So these things are done, but it is also possible that the specific >> quadtree functionality could be tested, in isolation from the >> projection. I think this may be oneo f the things Anthony is
talking
>> about -- answer testing can handle the big, complex items, and by >> breaking down to the fundamentals we can address isolated items
from a
>> unit testing perspective. >> >> > >> > So unfortunately, I think it would still come down to having a
fake
>> > frontend. It's not ideal, but it seems like any more isolation
would
>> > require >> > big rewrites to yt. >> >> One fun thing that is not usually known is that we have a fake >> frontend already, it just doesn't get used much. It's called the >> "Stream" frontend and it was designed originally to be used in >> ParaView, but now gets used by the (new,
not-yet-documented/released)
>> load_uniform_grid function as well as by Hyperion, the RT code by
Tom
>> R. It can set up AMR as well as static mesh. It's not terribly
well
>> documented, but there are examples on the wiki. >> >> One thing I've been thinking about is actually creating a couple
fake
>> outputs, which could be defined analytically with spheres of >> overdensity inside them. In principle, if we added refinement >> criteria, we could make this relatively complex data that was
defined
>> with only a few lines of code, but spun up a big in-memory dataset. >> >> (This exact thing is on my list of things to do and then to output
in
>> GDF, by the way...) >> >> That I think could come, down the road a bit. The refinement
criteria
>> wouldn't be too bad to implement, especially since we already have
the
>> grid splitting routines. I just don't think we should focus on it
at
>> the moment. But the uniform grid creation and loading works
already
>> -- I used it this morning. You can do it with: >> >> from yt.frontends.stream.api import load_uniform_grid >> ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2},
[359,
>> 359, 359], 1.0) >> >> the list is the dimensions of the data and the value is the to-cm >> conversion. >> >> > >> > Of course, I could be missing something. Matt, can you think of
a better
>> > way? >> >> I think for this specific example (and your damningly complex
tracing
>> of things through the source ...) the easiest thing to do is
isolate
>> the Cython routine, which it seems I was able to do only because I >> wrote it and which seems quite buried in the code, and to also
provide
>> high-level machinery for faking a frontend. >> >> -Matt >> >> > >> > - Casey >> > >> > >> > On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz <
scopatz@gmail.com>
>> > wrote: >> >> >> >> Helo Casey, >> >> >> >> Sorry for taking the whole weekend to respond. >> >> >> >>>> I would like to help with this, but it's difficult to figure
out
>> >>>> where >> >>>> to start. >> >> >> >> >> >> Not to worry. I think that any of the items listed at the
bottom of
>> >> Matt's >> >> original email >> >> would be a great place to start. >> >> >> >>>> >> >>>> >> >>>> Say I want to test projections. I make a fake 3D density
field, maybe
>> >>>> something as simple as np.arange(4**3).reshape((4, 4, 4)). I
write
>> >>>> down the >> >>>> answer to the x-projection. Now all I need to do is call >> >>>> assert_allclose(yt_result, answer, rtol=1e-15), but I don't
know what
>> >>>> pieces >> >>>> of low-level yt stuff to call to get to `yt_result`.
Hopefully that's
>> >>>> clear... >> >>>> >> >>>> Maybe this comes down to creating a fake frontend we can
attach
>> >>>> fields >> >>>> to? >> >> >> >> >> >> Actually, I disagree with this strategy, as I told Matt when we
spoke
>> >> last >> >> week. >> >> What is important is that we test the science and math parts of
the
>> >> code >> >> before, if ever, dealing with the software architecture that
surrounds
>> >> them. >> >> >> >> Let's taking your example of projections. What we need to test
is the
>> >> actual function >> >> or method which actually slogs through the projection
calculation. In
>> >> many cases in >> >> yt these functions are not directly attached to the front end
but live
>> >> in >> >> analysis, visualization >> >> or utilities subpackages. It is these such packages that we
should
>> >> worry >> >> about testing. >> >> We can easily create routines to feed them sample data. >> >> >> >> On the other hand, testing or mocking things like frontends
should be a
>> >> very low priority. >> >> At the end of the day what you are testing here is pulling in
data from
>> >> disk or other >> >> sources. Effectively, this is just re-testing functionality
present in
>> >> h5py, etc. That is not >> >> really our job. Yes, in a perfect world, front ends would be
tested
>> >> too. >> >> But I think that the >> >> priority should be placed on things like the KDTree. >> >> >> >> Be Well >> >> Anthony >> >> >> >>>> >> >>>> >> >>>> - Casey >> >>>> >> >>>> >> >>>> On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk <
matthewturk@gmail.com>
>> >>>> wrote: >> >>>>> >> >>>>> Hi all, >> >>>>> >> >>>>> As some of you have seen (at least Stephen), I filed a
ticket this
>> >>>>> morning about increasing testing coverage. The other night
Anthony
>> >>>>> and I met up in NYC and he had something of an
"intervention" about
>> >>>>> the sufficiency of answer testing for yt; it didn't take too
much
>> >>>>> work >> >>>>> on his part to convince me that we should be testing not just >> >>>>> against >> >>>>> a gold standard, but also performing unit tests. In the
past I had
>> >>>>> eschewed unit testing simply because the task of mocking
data was
>> >>>>> quite tricky, and by adding tests that use smaller bits we
could
>> >>>>> cover >> >>>>> unit testable areas with answer testing. >> >>>>> >> >>>>> But, this isn't really a good strategy. Let's move to
having both.
>> >>>>> The testing infrastructure he recommends is the
nearly-omnipresent
>> >>>>> nose: >> >>>>> >> >>>>> http://nose.readthedocs.org/en/latest/ >> >>>>> >> >>>>> The ticket to track this is here: >> >>>>> >> >>>>> >> >>>>> >> >>>>>
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
>> >>>>> >> >>>>> There are a couple sub-items here: >> >>>>> >> >>>>> 1) NumPy's nose test plugins provide a lot of necessary >> >>>>> functionality >> >>>>> that we have reimplemented in the answer testing utilities.
I'd
>> >>>>> like >> >>>>> to start using the numpy plugins, which include things like >> >>>>> conditional test execution, array comparisons, "slow" tests,
etc
>> >>>>> etc. >> >>>>> 2) We can evaluate, using conditional test execution, moving
to nose
>> >>>>> for answer testing. But that's not on the agenda now. >> >>>>> 3) Writing tests for nose is super easy, and running them is
too.
>> >>>>> Just >> >>>>> do: >> >>>>> >> >>>>> nosetest -w yt/ >> >>>>> >> >>>>> when in your source directory. >> >>>>> >> >>>>> 4) I've written a simple sample here: >> >>>>> >> >>>>> >> >>>>> >> >>>>>
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
>> >>>>> >> >>>>> 5) I'll handle writing up some mock data that doesn't require >> >>>>> shipping >> >>>>> lots of binary files, which can then be used for checking
things
>> >>>>> that >> >>>>> absolutely require hierarchies. >> >>>>> >> >>>>> -- >> >>>>> >> >>>>> The way to organize tests is easy. Inside each directory
with
>> >>>>> testable items create a new directory called "tests", and in
here
>> >>>>> toss >> >>>>> some scripts. You can stick a bunch of functions in those
scripts.
>> >>>>> >> >>>>> Anyway, I'm going to start writing more of these (in the
main yt
>> >>>>> repo, >> >>>>> and this change will be grafted there as well) and I'll
write back
>> >>>>> once the data mocking is ready. I'd like it if we started >> >>>>> encouraging >> >>>>> or even mandating simple tests (and/or answer tests) for >> >>>>> functionality >> >>>>> that gets added, but that's a discussion that should be held >> >>>>> separately. >> >>>>> >> >>>>> The items on the ticket: >> >>>>> >> >>>>> * kD-tree for nearest neighbor >> >>>>> * Geometric selection routines >> >>>>> * Profiles >> >>>>> * Projections -- underlying quadtree >> >>>>> * Data object selection of data containers >> >>>>> * Data object selection of points >> >>>>> * Orientation class >> >>>>> * Pixelization >> >>>>> * Color maps >> >>>>> * PNG writing >> >>>>> >> >>>>> Is anyone willing to claim any additional items that they
will help
>> >>>>> write unit tests for? >> >>>>> >> >>>>> -Matt >> >>>>> _______________________________________________ >> >>>>> yt-dev mailing list >> >>>>> yt-dev@lists.spacepope.org >> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> >>>> >> >>>> >> >>>> >> >>>> _______________________________________________ >> >>>> yt-dev mailing list >> >>>> yt-dev@lists.spacepope.org >> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> >>>> >> >> >> >> >> >> _______________________________________________ >> >> yt-dev mailing list >> >> yt-dev@lists.spacepope.org >> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> >> >> > >> > >> > _______________________________________________ >> > yt-dev mailing list >> > yt-dev@lists.spacepope.org >> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Hi all,
So I think a refresher on what we have currently is worthwhile, as well as some other suggestions/discussion.
Currently, yt has answer testing. What this does has been covered elsewhere, but this lives in yt/utilities/answer_testing.py. Plus, the mechanisms in there get used in Enzo, so this needs to be modified only very carefully. The reason this comes up here is because it will show up (currently) when running a raw nosetest discovery. There's probably a way around this.
Anthony added an amrspace function to yt/testing.py. I'd like to see our testing modules all import from yt/testing.py, the module they're to test, and hopefully not much else. I've copied Anthony's module into the main yt branch and also added a "fake_random_pf" function to it, which creates a unigrid parameter file with affiliated data in any number of fields. To show this off, I've also implemented a refinement criteria module which currently contains an OverDensity criterion.
Here's the testing.py module from Anthony, with my (short) addition:
https://bitbucket.org/yt_analysis/yt/src/db90f5fd1742/yt/testing.py
Flagging methods:
https://bitbucket.org/yt_analysis/yt/src/db90f5fd1742/yt/utilities/flagging_...
Tests, both of the interpolators and the flagging methods:
https://bitbucket.org/yt_analysis/yt/src/db90f5fd1742/yt/utilities/tests/tes... https://bitbucket.org/yt_analysis/yt/src/db90f5fd1742/yt/utilities/tests/tes...
(These are still being expanded.) I'm optimistic we could have full fake data generation in memory down the road, but that's mostly orthogonal to what Anthony has been proposing, which is to reduce the complexity of the tests, and to avoid testing on data. I think this is reasonable, and I think if we start isolating things it's workable.
So while I'm +0 on putting everything into the tests/ directory (I can see both sides, but I epsilon prefer the single-directory) that directory already has a bunch of answer testing stuff in it, which could confuse nose, since they work differently. Down the road I will decorate them to make them more clearly different.
What does everybody think? Are we ready to start divvying up tests?
To run nose with the tests the way they're set up now, and to avoid the broken-for-nose tests inside the kdree module and the answer testing stuff, I ran this:
nosetests --pdb-failures -v -w yt -e "answer_testing" -e "kdtree"
In the main branch, while we have ~500 answer tests, we're now up to an impressive 4 unit tests. Using the generator test-creation mechanisms Anthony used in the 3.0 branch, this can spiral up quite quickly. Is anybody willing to volunteer to take on a couple sections of the code? I'm going to be trying to add tests over the next little while, but starting tomorrow I'm offline through Monday night.
-Matt
On Tue, Sep 25, 2012 at 7:18 PM, Sam Skillman samskillman@gmail.com wrote:
Gotcha, thanks for the clarification. I'm just trying to stay caught up on all the movement.
Having one tests/ folder per dir seems reasonable.
Sam
On Tue, Sep 25, 2012 at 5:12 PM, Casey W. Stark caseywstark@gmail.com wrote:
Hey Sam.
I was suggesting...
yt-hg/ test/ yt/
So that it is separate from the yt package.
- Casey
On Tue, Sep 25, 2012 at 4:05 PM, Sam Skillman samskillman@gmail.com wrote:
Would root in our case be yt-hg/yt, in which case the tests would live at yt-hg/yt/tests/ ?
Sorry if I misunderstood. Sam
On Tue, Sep 25, 2012 at 5:00 PM, Anthony Scopatz scopatz@gmail.com wrote:
Hi Casey,
The reason I suggested that we do it this way originally was because this is the way that most other packages in the scientific python ecosystem are set up. If you look at numpy, scipy, cython, pytables, ipython, etc. they all have the following file system structure:
root/ module1.py module2.py tests/ test_module2.py test_module1.py subpack1/ module3.py tests/ test_module3.py
I think that it is done this way to make it easy/possible for UnitTest and nose to autodetect the tests. I am not saying that we have to follow this de facto standard. However, I don't really see a reason why we shouldn't either. If you do, please let me know ;)
Notably, matplotlib does not follow this standard. But then again, since they are testing GUI and rendering code their tests are a level of complexity above what we need.
Be Well Anthony
On Tue, Sep 25, 2012 at 5:50 PM, Casey W. Stark caseywstark@gmail.com wrote:
Hey Matt, Anthony.
Looks like a good start to nosetests. One small thing -- would anyone be opposed to moving these out of the yt directory and into a root "test" directory?
I can take on CIC deposition and interpolation wherever those are.
- Casey
On Tue, Sep 25, 2012 at 11:20 AM, Matthew Turk matthewturk@gmail.com wrote:
Hi all,
As a note, Anthony just added this to the 3.0 base. Here's a set of tests:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/utilities/lib/t...
and here's a utilities section he added:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/testing.py
-Matt
On Mon, Sep 24, 2012 at 10:01 PM, Matthew Turk matthewturk@gmail.com wrote: > Hey Casey, > > On Mon, Sep 24, 2012 at 5:39 PM, Casey W. Stark > caseywstark@gmail.com wrote: >> Hi Matt. >> >> Glad my example was useful in some way. I guess knowing exactly >> which Cython >> routines to test for what is what I meant about where to start. > > I completely understand -- and that's a failure of my part in the > code. So I think what might help is if we start looking at the > routines that could be tested, and then I can jump in when it looks > like they're buried. I will also volunteer to try to refactor this > code, once it's tested, to make it clearer what is where and why. > >> >> Thanks for the tip about the stream frontend. > > No prob, it should be documented better. > > In terms of generating data, here's an example of what I was > meaning: > > https://hub.yt-project.org/nb/ompxtg > > This sets up a random (but small-valued) background density and then > applies a couple top hat spheres on top of it. I think this could > be > a good starting point for unit testing. I looked at the code that > exists for flagging cells and refining them, and it's currently a > bit > too specific for this, as it was designed to take RAMSES data and > partition it, which operates slightly differently. I'll take a pass > at extracting it and making it work in this case. > > The way I see it working is that one would set up the operators, as > is > done here, and then a progressive routine would be applied, > something > like this: > > while grids.flag() > 0: > grids.refine() > for operator in operators: operator.apply(grids) > pf = grids.convert_to_dataset() > > The missing step in the code base is mostly just the refinement, as > for the smoothed covering grids we have the necessary machinery to > interpolate from one level to the next. Adding more types of > operators to this library would be beneficial as well. The > finalization step at the end would then convert the collection of > grids into a Stream dataset. So with only a couple lines of code we > could in all likelihood be able to generate in-memory datasets. > > (This underscores in my mind why it would be awesome to have GDF in > more places, as with this kind of machinery we've just written > initial > conditions generation, especially since we have a GDF writer > already...) > > Anyway, what might be really helpful is if interested people would > volunteer to address a few of the testable areas? Then we can start > pushing and identifying problem areas. I'll volunteer to handle at > least the data container stuff, and whatever else slips through the > cracks; although my time will be somewhat limited in the very near > future, I will try to make this a priority. > > -Matt > >> >> - Casey >> >> >> On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk >> matthewturk@gmail.com wrote: >>> >>> Hey Casey and Anthony, >>> >>> On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark >>> caseywstark@gmail.com >>> wrote: >>> > Hi Anthony. >>> > >>> > I completely agree that we should target the level of functions >>> > actually >>> > performing the projection rather than yt's organization. The >>> > mock >>> > frontend >>> > suggestion was just a hack to get there. I don't know if there's >>> > a way >>> > around it though... >>> > >>> > Here's an example of what I sorted through to get to >>> > projections: >>> > - Load a test plotfile, check pf.h.proj to find it's source. >>> > - Read through data_objects/hierarchy.py and >>> > utilities/parallel_tools/parallel_analysis_interface.py to find >>> > where >>> > proj >>> > is attached, can't find it. >>> > - The proj docstring says it is a reference to AMRQuadProj. >>> > Can't find a >>> > class by that name. >>> > - Search data_objects sources for "proj", find AMRProjBase. >>> > >>> > So it looks like the functionality is wrapped up in the >>> > __project_level >>> > and >>> > _project_grid methods. I can't think of a way to test those >>> > without >>> > creating >>> > an AMRProjBase, and that requires a staticoutput object. >>> >>> You're right, the projection stuff as *projections* is not easy to >>> test. But in terms of testing the underlying code, which is >>> wrapped >>> up in a Cython class called QuadTree, I think it could be done. >>> The >>> steps you're describing are actually all part of the existing >>> answer >>> testing machinery, which performs a couple things and verifies >>> that >>> they don't change over time: >>> >>> 1) Project some fields from the disk >>> 2) Project a couple derived fields >>> 3) Project a derived field that requires spatial derivatives >>> 4) Project the "Ones" field, which should be 1.0 everywhere. >>> >>> So these things are done, but it is also possible that the >>> specific >>> quadtree functionality could be tested, in isolation from the >>> projection. I think this may be oneo f the things Anthony is >>> talking >>> about -- answer testing can handle the big, complex items, and by >>> breaking down to the fundamentals we can address isolated items >>> from a >>> unit testing perspective. >>> >>> > >>> > So unfortunately, I think it would still come down to having a >>> > fake >>> > frontend. It's not ideal, but it seems like any more isolation >>> > would >>> > require >>> > big rewrites to yt. >>> >>> One fun thing that is not usually known is that we have a fake >>> frontend already, it just doesn't get used much. It's called the >>> "Stream" frontend and it was designed originally to be used in >>> ParaView, but now gets used by the (new, >>> not-yet-documented/released) >>> load_uniform_grid function as well as by Hyperion, the RT code by >>> Tom >>> R. It can set up AMR as well as static mesh. It's not terribly >>> well >>> documented, but there are examples on the wiki. >>> >>> One thing I've been thinking about is actually creating a couple >>> fake >>> outputs, which could be defined analytically with spheres of >>> overdensity inside them. In principle, if we added refinement >>> criteria, we could make this relatively complex data that was >>> defined >>> with only a few lines of code, but spun up a big in-memory >>> dataset. >>> >>> (This exact thing is on my list of things to do and then to output >>> in >>> GDF, by the way...) >>> >>> That I think could come, down the road a bit. The refinement >>> criteria >>> wouldn't be too bad to implement, especially since we already have >>> the >>> grid splitting routines. I just don't think we should focus on it >>> at >>> the moment. But the uniform grid creation and loading works >>> already >>> -- I used it this morning. You can do it with: >>> >>> from yt.frontends.stream.api import load_uniform_grid >>> ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2}, >>> [359, >>> 359, 359], 1.0) >>> >>> the list is the dimensions of the data and the value is the to-cm >>> conversion. >>> >>> > >>> > Of course, I could be missing something. Matt, can you think of >>> > a better >>> > way? >>> >>> I think for this specific example (and your damningly complex >>> tracing >>> of things through the source ...) the easiest thing to do is >>> isolate >>> the Cython routine, which it seems I was able to do only because I >>> wrote it and which seems quite buried in the code, and to also >>> provide >>> high-level machinery for faking a frontend. >>> >>> -Matt >>> >>> > >>> > - Casey >>> > >>> > >>> > On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz >>> > scopatz@gmail.com >>> > wrote: >>> >> >>> >> Helo Casey, >>> >> >>> >> Sorry for taking the whole weekend to respond. >>> >> >>> >>>> I would like to help with this, but it's difficult to figure >>> >>>> out >>> >>>> where >>> >>>> to start. >>> >> >>> >> >>> >> Not to worry. I think that any of the items listed at the >>> >> bottom of >>> >> Matt's >>> >> original email >>> >> would be a great place to start. >>> >> >>> >>>> >>> >>>> >>> >>>> Say I want to test projections. I make a fake 3D density >>> >>>> field, maybe >>> >>>> something as simple as np.arange(4**3).reshape((4, 4, 4)). I >>> >>>> write >>> >>>> down the >>> >>>> answer to the x-projection. Now all I need to do is call >>> >>>> assert_allclose(yt_result, answer, rtol=1e-15), but I don't >>> >>>> know what >>> >>>> pieces >>> >>>> of low-level yt stuff to call to get to `yt_result`. >>> >>>> Hopefully that's >>> >>>> clear... >>> >>>> >>> >>>> Maybe this comes down to creating a fake frontend we can >>> >>>> attach >>> >>>> fields >>> >>>> to? >>> >> >>> >> >>> >> Actually, I disagree with this strategy, as I told Matt when we >>> >> spoke >>> >> last >>> >> week. >>> >> What is important is that we test the science and math parts of >>> >> the >>> >> code >>> >> before, if ever, dealing with the software architecture that >>> >> surrounds >>> >> them. >>> >> >>> >> Let's taking your example of projections. What we need to test >>> >> is the >>> >> actual function >>> >> or method which actually slogs through the projection >>> >> calculation. In >>> >> many cases in >>> >> yt these functions are not directly attached to the front end >>> >> but live >>> >> in >>> >> analysis, visualization >>> >> or utilities subpackages. It is these such packages that we >>> >> should >>> >> worry >>> >> about testing. >>> >> We can easily create routines to feed them sample data. >>> >> >>> >> On the other hand, testing or mocking things like frontends >>> >> should be a >>> >> very low priority. >>> >> At the end of the day what you are testing here is pulling in >>> >> data from >>> >> disk or other >>> >> sources. Effectively, this is just re-testing functionality >>> >> present in >>> >> h5py, etc. That is not >>> >> really our job. Yes, in a perfect world, front ends would be >>> >> tested >>> >> too. >>> >> But I think that the >>> >> priority should be placed on things like the KDTree. >>> >> >>> >> Be Well >>> >> Anthony >>> >> >>> >>>> >>> >>>> >>> >>>> - Casey >>> >>>> >>> >>>> >>> >>>> On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk >>> >>>> matthewturk@gmail.com >>> >>>> wrote: >>> >>>>> >>> >>>>> Hi all, >>> >>>>> >>> >>>>> As some of you have seen (at least Stephen), I filed a >>> >>>>> ticket this >>> >>>>> morning about increasing testing coverage. The other night >>> >>>>> Anthony >>> >>>>> and I met up in NYC and he had something of an >>> >>>>> "intervention" about >>> >>>>> the sufficiency of answer testing for yt; it didn't take too >>> >>>>> much >>> >>>>> work >>> >>>>> on his part to convince me that we should be testing not >>> >>>>> just >>> >>>>> against >>> >>>>> a gold standard, but also performing unit tests. In the >>> >>>>> past I had >>> >>>>> eschewed unit testing simply because the task of mocking >>> >>>>> data was >>> >>>>> quite tricky, and by adding tests that use smaller bits we >>> >>>>> could >>> >>>>> cover >>> >>>>> unit testable areas with answer testing. >>> >>>>> >>> >>>>> But, this isn't really a good strategy. Let's move to >>> >>>>> having both. >>> >>>>> The testing infrastructure he recommends is the >>> >>>>> nearly-omnipresent >>> >>>>> nose: >>> >>>>> >>> >>>>> http://nose.readthedocs.org/en/latest/ >>> >>>>> >>> >>>>> The ticket to track this is here: >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage >>> >>>>> >>> >>>>> There are a couple sub-items here: >>> >>>>> >>> >>>>> 1) NumPy's nose test plugins provide a lot of necessary >>> >>>>> functionality >>> >>>>> that we have reimplemented in the answer testing utilities. >>> >>>>> I'd >>> >>>>> like >>> >>>>> to start using the numpy plugins, which include things like >>> >>>>> conditional test execution, array comparisons, "slow" tests, >>> >>>>> etc >>> >>>>> etc. >>> >>>>> 2) We can evaluate, using conditional test execution, moving >>> >>>>> to nose >>> >>>>> for answer testing. But that's not on the agenda now. >>> >>>>> 3) Writing tests for nose is super easy, and running them is >>> >>>>> too. >>> >>>>> Just >>> >>>>> do: >>> >>>>> >>> >>>>> nosetest -w yt/ >>> >>>>> >>> >>>>> when in your source directory. >>> >>>>> >>> >>>>> 4) I've written a simple sample here: >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests... >>> >>>>> >>> >>>>> 5) I'll handle writing up some mock data that doesn't >>> >>>>> require >>> >>>>> shipping >>> >>>>> lots of binary files, which can then be used for checking >>> >>>>> things >>> >>>>> that >>> >>>>> absolutely require hierarchies. >>> >>>>> >>> >>>>> -- >>> >>>>> >>> >>>>> The way to organize tests is easy. Inside each directory >>> >>>>> with >>> >>>>> testable items create a new directory called "tests", and in >>> >>>>> here >>> >>>>> toss >>> >>>>> some scripts. You can stick a bunch of functions in those >>> >>>>> scripts. >>> >>>>> >>> >>>>> Anyway, I'm going to start writing more of these (in the >>> >>>>> main yt >>> >>>>> repo, >>> >>>>> and this change will be grafted there as well) and I'll >>> >>>>> write back >>> >>>>> once the data mocking is ready. I'd like it if we started >>> >>>>> encouraging >>> >>>>> or even mandating simple tests (and/or answer tests) for >>> >>>>> functionality >>> >>>>> that gets added, but that's a discussion that should be held >>> >>>>> separately. >>> >>>>> >>> >>>>> The items on the ticket: >>> >>>>> >>> >>>>> * kD-tree for nearest neighbor >>> >>>>> * Geometric selection routines >>> >>>>> * Profiles >>> >>>>> * Projections -- underlying quadtree >>> >>>>> * Data object selection of data containers >>> >>>>> * Data object selection of points >>> >>>>> * Orientation class >>> >>>>> * Pixelization >>> >>>>> * Color maps >>> >>>>> * PNG writing >>> >>>>> >>> >>>>> Is anyone willing to claim any additional items that they >>> >>>>> will help >>> >>>>> write unit tests for? >>> >>>>> >>> >>>>> -Matt >>> >>>>> _______________________________________________ >>> >>>>> yt-dev mailing list >>> >>>>> yt-dev@lists.spacepope.org >>> >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >>>> >>> >>>> >>> >>>> >>> >>>> _______________________________________________ >>> >>>> yt-dev mailing list >>> >>>> yt-dev@lists.spacepope.org >>> >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >>>> >>> >> >>> >> >>> >> _______________________________________________ >>> >> yt-dev mailing list >>> >> yt-dev@lists.spacepope.org >>> >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> >> >>> > >>> > >>> > _______________________________________________ >>> > yt-dev mailing list >>> > yt-dev@lists.spacepope.org >>> > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>> > >>> _______________________________________________ >>> yt-dev mailing list >>> yt-dev@lists.spacepope.org >>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
Ah okay. I hadn't looked at test organization in numpy and scipy. Thanks for pointing that out. Looks like either way is good. I like the root level test dir because then the tests are separate from source. However, one directory with all test scripts could get out of hand quickly.
- Casey
On Tue, Sep 25, 2012 at 4:00 PM, Anthony Scopatz scopatz@gmail.com wrote:
Hi Casey,
The reason I suggested that we do it this way originally was because this is the way that most other packages in the scientific python ecosystem are set up. If you look at numpy, scipy, cython, pytables, ipython, etc. they all have the following file system structure:
root/ module1.py module2.py tests/ test_module2.py test_module1.py subpack1/ module3.py tests/ test_module3.py
I think that it is done this way to make it easy/possible for UnitTest and nose to autodetect the tests. I am not saying that we have to follow this de facto standard. However, I don't really see a reason why we shouldn't either. If you do, please let me know ;)
Notably, matplotlib does not follow this standard. But then again, since they are testing GUI and rendering code their tests are a level of complexity above what we need.
Be Well Anthony
On Tue, Sep 25, 2012 at 5:50 PM, Casey W. Stark caseywstark@gmail.comwrote:
Hey Matt, Anthony.
Looks like a good start to nosetests. One small thing -- would anyone be opposed to moving these out of the yt directory and into a root "test" directory?
I can take on CIC deposition and interpolation wherever those are.
- Casey
On Tue, Sep 25, 2012 at 11:20 AM, Matthew Turk matthewturk@gmail.comwrote:
Hi all,
As a note, Anthony just added this to the 3.0 base. Here's a set of tests:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/utilities/lib/t...
and here's a utilities section he added:
https://bitbucket.org/yt_analysis/yt-3.0/src/52a1b2ecea94/yt/testing.py
-Matt
On Mon, Sep 24, 2012 at 10:01 PM, Matthew Turk matthewturk@gmail.com wrote:
Hey Casey,
On Mon, Sep 24, 2012 at 5:39 PM, Casey W. Stark caseywstark@gmail.com
wrote:
Hi Matt.
Glad my example was useful in some way. I guess knowing exactly which
Cython
routines to test for what is what I meant about where to start.
I completely understand -- and that's a failure of my part in the code. So I think what might help is if we start looking at the routines that could be tested, and then I can jump in when it looks like they're buried. I will also volunteer to try to refactor this code, once it's tested, to make it clearer what is where and why.
Thanks for the tip about the stream frontend.
No prob, it should be documented better.
In terms of generating data, here's an example of what I was meaning:
https://hub.yt-project.org/nb/ompxtg
This sets up a random (but small-valued) background density and then applies a couple top hat spheres on top of it. I think this could be a good starting point for unit testing. I looked at the code that exists for flagging cells and refining them, and it's currently a bit too specific for this, as it was designed to take RAMSES data and partition it, which operates slightly differently. I'll take a pass at extracting it and making it work in this case.
The way I see it working is that one would set up the operators, as is done here, and then a progressive routine would be applied, something like this:
while grids.flag() > 0: grids.refine() for operator in operators: operator.apply(grids) pf = grids.convert_to_dataset()
The missing step in the code base is mostly just the refinement, as for the smoothed covering grids we have the necessary machinery to interpolate from one level to the next. Adding more types of operators to this library would be beneficial as well. The finalization step at the end would then convert the collection of grids into a Stream dataset. So with only a couple lines of code we could in all likelihood be able to generate in-memory datasets.
(This underscores in my mind why it would be awesome to have GDF in more places, as with this kind of machinery we've just written initial conditions generation, especially since we have a GDF writer already...)
Anyway, what might be really helpful is if interested people would volunteer to address a few of the testable areas? Then we can start pushing and identifying problem areas. I'll volunteer to handle at least the data container stuff, and whatever else slips through the cracks; although my time will be somewhat limited in the very near future, I will try to make this a priority.
-Matt
- Casey
On Mon, Sep 24, 2012 at 1:52 PM, Matthew Turk matthewturk@gmail.com
wrote:
Hey Casey and Anthony,
On Mon, Sep 24, 2012 at 4:20 PM, Casey W. Stark <
caseywstark@gmail.com>
wrote: > Hi Anthony. > > I completely agree that we should target the level of functions
actually
> performing the projection rather than yt's organization. The mock > frontend > suggestion was just a hack to get there. I don't know if there's a
way
> around it though... > > Here's an example of what I sorted through to get to projections: > - Load a test plotfile, check pf.h.proj to find it's source. > - Read through data_objects/hierarchy.py and > utilities/parallel_tools/parallel_analysis_interface.py to find
where
> proj > is attached, can't find it. > - The proj docstring says it is a reference to AMRQuadProj. Can't
find a
> class by that name. > - Search data_objects sources for "proj", find AMRProjBase. > > So it looks like the functionality is wrapped up in the
__project_level
> and > _project_grid methods. I can't think of a way to test those without > creating > an AMRProjBase, and that requires a staticoutput object.
You're right, the projection stuff as *projections* is not easy to test. But in terms of testing the underlying code, which is wrapped up in a Cython class called QuadTree, I think it could be done. The steps you're describing are actually all part of the existing answer testing machinery, which performs a couple things and verifies that they don't change over time:
- Project some fields from the disk
- Project a couple derived fields
- Project a derived field that requires spatial derivatives
- Project the "Ones" field, which should be 1.0 everywhere.
So these things are done, but it is also possible that the specific quadtree functionality could be tested, in isolation from the projection. I think this may be oneo f the things Anthony is talking about -- answer testing can handle the big, complex items, and by breaking down to the fundamentals we can address isolated items from
a
unit testing perspective.
> > So unfortunately, I think it would still come down to having a fake > frontend. It's not ideal, but it seems like any more isolation
would
> require > big rewrites to yt.
One fun thing that is not usually known is that we have a fake frontend already, it just doesn't get used much. It's called the "Stream" frontend and it was designed originally to be used in ParaView, but now gets used by the (new, not-yet-documented/released) load_uniform_grid function as well as by Hyperion, the RT code by Tom R. It can set up AMR as well as static mesh. It's not terribly well documented, but there are examples on the wiki.
One thing I've been thinking about is actually creating a couple fake outputs, which could be defined analytically with spheres of overdensity inside them. In principle, if we added refinement criteria, we could make this relatively complex data that was defined with only a few lines of code, but spun up a big in-memory dataset.
(This exact thing is on my list of things to do and then to output in GDF, by the way...)
That I think could come, down the road a bit. The refinement
criteria
wouldn't be too bad to implement, especially since we already have
the
grid splitting routines. I just don't think we should focus on it at the moment. But the uniform grid creation and loading works already -- I used it this morning. You can do it with:
from yt.frontends.stream.api import load_uniform_grid ug = load_uniform_grid({"VelocityNorm":data1, "Density":data2}, [359, 359, 359], 1.0)
the list is the dimensions of the data and the value is the to-cm conversion.
> > Of course, I could be missing something. Matt, can you think of a
better
> way?
I think for this specific example (and your damningly complex tracing of things through the source ...) the easiest thing to do is isolate the Cython routine, which it seems I was able to do only because I wrote it and which seems quite buried in the code, and to also
provide
high-level machinery for faking a frontend.
-Matt
> > - Casey > > > On Mon, Sep 24, 2012 at 11:02 AM, Anthony Scopatz <
scopatz@gmail.com>
> wrote: >> >> Helo Casey, >> >> Sorry for taking the whole weekend to respond. >> >>>> I would like to help with this, but it's difficult to figure out >>>> where >>>> to start. >> >> >> Not to worry. I think that any of the items listed at the bottom
of
>> Matt's >> original email >> would be a great place to start. >> >>>> >>>> >>>> Say I want to test projections. I make a fake 3D density field,
maybe
>>>> something as simple as np.arange(4**3).reshape((4, 4, 4)). I
write
>>>> down the >>>> answer to the x-projection. Now all I need to do is call >>>> assert_allclose(yt_result, answer, rtol=1e-15), but I don't
know what
>>>> pieces >>>> of low-level yt stuff to call to get to `yt_result`. Hopefully
that's
>>>> clear... >>>> >>>> Maybe this comes down to creating a fake frontend we can attach >>>> fields >>>> to? >> >> >> Actually, I disagree with this strategy, as I told Matt when we
spoke
>> last >> week. >> What is important is that we test the science and math parts of
the
>> code >> before, if ever, dealing with the software architecture that
surrounds
>> them. >> >> Let's taking your example of projections. What we need to test
is the
>> actual function >> or method which actually slogs through the projection
calculation. In
>> many cases in >> yt these functions are not directly attached to the front end but
live
>> in >> analysis, visualization >> or utilities subpackages. It is these such packages that we
should
>> worry >> about testing. >> We can easily create routines to feed them sample data. >> >> On the other hand, testing or mocking things like frontends
should be a
>> very low priority. >> At the end of the day what you are testing here is pulling in
data from
>> disk or other >> sources. Effectively, this is just re-testing functionality
present in
>> h5py, etc. That is not >> really our job. Yes, in a perfect world, front ends would be
tested
>> too. >> But I think that the >> priority should be placed on things like the KDTree. >> >> Be Well >> Anthony >> >>>> >>>> >>>> - Casey >>>> >>>> >>>> On Fri, Sep 21, 2012 at 2:42 PM, Matthew Turk <
matthewturk@gmail.com>
>>>> wrote: >>>>> >>>>> Hi all, >>>>> >>>>> As some of you have seen (at least Stephen), I filed a ticket
this
>>>>> morning about increasing testing coverage. The other night
Anthony
>>>>> and I met up in NYC and he had something of an "intervention"
about
>>>>> the sufficiency of answer testing for yt; it didn't take too
much
>>>>> work >>>>> on his part to convince me that we should be testing not just >>>>> against >>>>> a gold standard, but also performing unit tests. In the past
I had
>>>>> eschewed unit testing simply because the task of mocking data
was
>>>>> quite tricky, and by adding tests that use smaller bits we
could
>>>>> cover >>>>> unit testable areas with answer testing. >>>>> >>>>> But, this isn't really a good strategy. Let's move to having
both.
>>>>> The testing infrastructure he recommends is the
nearly-omnipresent
>>>>> nose: >>>>> >>>>> http://nose.readthedocs.org/en/latest/ >>>>> >>>>> The ticket to track this is here: >>>>> >>>>> >>>>> >>>>>
https://bitbucket.org/yt_analysis/yt/issue/426/increase-unit-test-coverage
>>>>> >>>>> There are a couple sub-items here: >>>>> >>>>> 1) NumPy's nose test plugins provide a lot of necessary >>>>> functionality >>>>> that we have reimplemented in the answer testing utilities.
I'd
>>>>> like >>>>> to start using the numpy plugins, which include things like >>>>> conditional test execution, array comparisons, "slow" tests,
etc
>>>>> etc. >>>>> 2) We can evaluate, using conditional test execution, moving
to nose
>>>>> for answer testing. But that's not on the agenda now. >>>>> 3) Writing tests for nose is super easy, and running them is
too.
>>>>> Just >>>>> do: >>>>> >>>>> nosetest -w yt/ >>>>> >>>>> when in your source directory. >>>>> >>>>> 4) I've written a simple sample here: >>>>> >>>>> >>>>> >>>>>
https://bitbucket.org/yt_analysis/yt-3.0/src/da10ffc17f6d/yt/utilities/tests...
>>>>> >>>>> 5) I'll handle writing up some mock data that doesn't require >>>>> shipping >>>>> lots of binary files, which can then be used for checking
things
>>>>> that >>>>> absolutely require hierarchies. >>>>> >>>>> -- >>>>> >>>>> The way to organize tests is easy. Inside each directory with >>>>> testable items create a new directory called "tests", and in
here
>>>>> toss >>>>> some scripts. You can stick a bunch of functions in those
scripts.
>>>>> >>>>> Anyway, I'm going to start writing more of these (in the main
yt
>>>>> repo, >>>>> and this change will be grafted there as well) and I'll write
back
>>>>> once the data mocking is ready. I'd like it if we started >>>>> encouraging >>>>> or even mandating simple tests (and/or answer tests) for >>>>> functionality >>>>> that gets added, but that's a discussion that should be held >>>>> separately. >>>>> >>>>> The items on the ticket: >>>>> >>>>> * kD-tree for nearest neighbor >>>>> * Geometric selection routines >>>>> * Profiles >>>>> * Projections -- underlying quadtree >>>>> * Data object selection of data containers >>>>> * Data object selection of points >>>>> * Orientation class >>>>> * Pixelization >>>>> * Color maps >>>>> * PNG writing >>>>> >>>>> Is anyone willing to claim any additional items that they will
help
>>>>> write unit tests for? >>>>> >>>>> -Matt >>>>> _______________________________________________ >>>>> yt-dev mailing list >>>>> yt-dev@lists.spacepope.org >>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >>>> >>>> >>>> _______________________________________________ >>>> yt-dev mailing list >>>> yt-dev@lists.spacepope.org >>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >>>> >> >> >> _______________________________________________ >> yt-dev mailing list >> yt-dev@lists.spacepope.org >> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org >> > > > _______________________________________________ > yt-dev mailing list > yt-dev@lists.spacepope.org > http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org > _______________________________________________ yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
yt-dev mailing list yt-dev@lists.spacepope.org http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
participants (4)
-
Anthony Scopatz -
Casey W. Stark -
Matthew Turk -
Sam Skillman