On Mon, Jun 23, 2008 at 1:03 AM, Stéfan van der Walt wrote:

2008/6/23 Alan McIntyre :

...

Some docstrings have examples of how to use the function that aren't executable code (see numpy.core.defmatrix.bmat for an example) in their current form. Should these examples have the ">>>" removed from them to avoid them being picked up as doctests?

The examples written for the random module warrants the same question. First and foremost, the docstrings are there to illustrate to users how to use the code; second, to serve as tests.

Example codes should run, but I'm not sure whether they should always be valid doctests.

In the `bmat` example, I would remove the '>>>' like you suggested.

There's also the option of marking them so doctest skips them via #doctest: +SKIP http://docs.python.org/lib/doctest-options.html Cheers, f

Mon, 23 Jun 2008 14:17:09 -0400, Alan McIntyre wrote:

On Mon, Jun 23, 2008 at 2:02 PM, Fernando Perez wrote:

...

There's also the option of marking them so doctest skips them via

#doctest: +SKIP

http://docs.python.org/lib/doctest-options.html

For short examples, that seems like a good option, but it seems like you have to have that comment on every line that you want skipped. There are some long examples (like the one in lib/function_base.py:bartlett) that (to me) would look pretty ugly having that comment tacked on to every line.

Either way is fine with me in the end, though, so long as it doesn't produce test failures. :)

Can you make the convention chosen for the examples (currently only in the doc wiki, not yet in SVN) to work: assuming "import numpy as np" in examples? This would remove the need for those "from numpy import *" lines in the examples that I see were added in r5311. -- Pauli Virtanen

On Mon, Jun 23, 2008 at 3:03 PM, Alan McIntyre wrote:

On Mon, Jun 23, 2008 at 2:37 PM, Pauli Virtanen wrote:

...

Can you make the convention chosen for the examples (currently only in the doc wiki, not yet in SVN) to work: assuming "import numpy as np" in examples?

This would remove the need for those "from numpy import *" lines in the examples that I see were added in r5311.

Sure, I'll look at that. It seems like every possible option for importing stuff from numpy is used in doctests (sometimes even in the same module), so having them standardized with that implicit import is much better.

It turns out it's possible to give all the doctests an implicit "import numpy as np" (and probably any other arbitrary tweaks to their execution context, if need be). Once I can include some of the other doctest tweaks discussed in this thread, and it's checked in, I'll go back and remove all those "import numpy" statements I inserted into the docstrings. Alan

On Mon, Jun 23, 2008 at 11:17 AM, Alan McIntyre wrote:

On Mon, Jun 23, 2008 at 2:02 PM, Fernando Perez wrote:

...

There's also the option of marking them so doctest skips them via

#doctest: +SKIP

http://docs.python.org/lib/doctest-options.html

For short examples, that seems like a good option, but it seems like you have to have that comment on every line that you want skipped. There are some long examples (like the one in lib/function_base.py:bartlett) that (to me) would look pretty ugly having that comment tacked on to every line.

Ugh. Definitely too ugly if it has to go in every line. From reading the docs I interpreted it as affecting the whole example, which would be far more sensible...

Either way is fine with me in the end, though, so long as it doesn't produce test failures. :)

Yes, but we also want to make these really easy for users to cleanly paste in with minimal effort. I wonder if a decorator could be applied to those functions so that nose would then skip the doctests: @skip_doctest def foo().... but the nose doctest plugin isn't exactly a model of modularity, so I'm not sure you want to go down that particular road... The no-prompts option seems to be the cleanest right now. You may want then to instead just use a reST block for those, so that sphinx eventually renders them nicely at least as blocks: Plot the window and its frequency response:: ### <<< Added double colon here from numpy import clip, log10, array, bartlett ### <<< Extra indent so this is a reST block from scipy.fftpack import fft etc... Just some ideas... Cheers, f

On Mon, Jun 23, 2008 at 3:21 PM, Stéfan van der Walt wrote:

Another alternative is to replace +SKIP with something like +IGNORE. That way, the statement is still executed, we just don't care about its outcome. If we skip the line entirely, it often affects the rest of the tests later on.

Ugh. That just seems like a lot of unreadable ugliness to me. If this comment magic is the only way to make that stuff execute properly under doctest, I think I'd rather just skip it in favor of clean, uncluttered, non-doctestable code samples in the docstrings. If the code that's currently in docstrings needs to be retained as test code, I'll gladly take the time to put it into a test_ module where it doesn't get in the way of documentation. I'll defer to the consensus, though.

Mon, 23 Jun 2008 15:53:55 -0400, Anne Archibald wrote:

2008/6/23 Alan McIntyre :

...

On Mon, Jun 23, 2008 at 3:21 PM, Stéfan van der Walt wrote:

...

Another alternative is to replace +SKIP with something like +IGNORE. That way, the statement is still executed, we just don't care about its outcome. If we skip the line entirely, it often affects the rest of the tests later on.

Ugh. That just seems like a lot of unreadable ugliness to me. If this comment magic is the only way to make that stuff execute properly under doctest, I think I'd rather just skip it in favor of clean, uncluttered, non-doctestable code samples in the docstrings. If the code that's currently in docstrings needs to be retained as test code, I'll gladly take the time to put it into a test_ module where it doesn't get in the way of documentation. I'll defer to the consensus, though.

I think doctests are valuable: it's very hard for the documentation to get out of sync with the code, and it makes it very easy to write tests, particularly in light of the wiki documentation framework. But I think encrusting examples with weird comments will be a pain for documentors and off-putting to users. Perhaps doctests can be positively marked, in some relatively unobtrusive way?

I also think being able to test that the examples in docstrings run correctly could be valuable, but I'm not sure if it makes sense to have this enabled in the default test set. Another idea (in addition to whitelisting): how easy would it be to subclass doctest.DocTestParser so that it would eg. automatically +IGNORE any doctest lines containing ">>> plt."? -- Pauli Virtanen

On 23 Jun 2008, at 12:37 PM, Alan McIntyre wrote:

...

Ugh. That just seems like a lot of unreadable ugliness to me. If this comment magic is the only way to make that stuff execute properly under doctest, I think I'd rather just skip it in favor of clean, uncluttered, non-doctestable code samples in the docstrings.

Another perspective: doctests ensure that documentation stays up to date (if the behaviour or interface changes, then tests will fail indicating that the documentation also needs to be updated.) Thus, one can argue that all examples should also be doctests. This generally makes things a little more ugly, but much less ambiguous. ... Examples: --------- If A, B, C, and D are appropriately shaped 2-d arrays, then one can produce [ A B ] [ C D ] using any of these methods:

...

...

A, B, C, D = [[1,1]], [[2,2]], [[3,3]], [[4,4]] np.bmat('A, B; C, D') # From a string matrix([[ 1, 1, 2, 2], [ 3, 3, 4, 4]]) np.bmat([[A,B],[C,D]]) # From a nested sequence matrix([[ 1, 1, 2, 2], [ 3, 3, 4, 4]]) np.bmat(np.r_[np.c_[A,B],np.c_[C,D]]) # From an array matrix([[ 1, 1, 2, 2], [ 3, 3, 4, 4]])

Michael.

On 23 Jun 2008, at 1:28 PM, Anne Archibald wrote:

2008/6/23 Michael McNeil Forbes :

...

Thus, one can argue that all examples should also be doctests. This generally makes things a little more ugly, but much less ambiguous.

This is a bit awkward. How do you give an example for a random-number generator? Even if you are willing to include a seed in each statement, misleading users into thinking it's necessary, the value returned for a given seed is not necessarily part of the interface a random-number generator agrees to support.

I agree that this can be awkward sometimes, and should certainly not be policy, but one can usually get around this. Instead of printing the result, you can use it, or demonstrate porperties:

...

...

random_array = np.random.rand(3,4) random_array.shape (3,4) random_array.max() < 1 True random_array.min() > 0 True

etc. Michael.

On Mon, Jun 23, 2008 at 4:51 PM, Robert Kern wrote:

...

I agree that this can be awkward sometimes, and should certainly not be policy, but one can usually get around this. Instead of printing the result, you can use it, or demonstrate porperties:

...

...

...

random_array = np.random.rand(3,4) random_array.shape (3,4) random_array.max() < 1 True random_array.min() > 0 True

Yes, this makes it doctestable, but you've destroyed the exampleness. It should be policy *not* to do this.

So it seems we have: 1. Example code that is doctestable 2. Example code that probably can't ever be doctestable (random number stuff, etc.), but is still executable 3. Schematic examples that aren't executable Personally, I'm in favor of filling out examples of type #3 to make them at least #2, but maybe that's not always practical. I don't think #3 should ever have ">>>" prompts, so it shouldn't ever be picked up by doctest. I suppose I could go for a decorator option to flag #2. If we execute them, but not look at the results, then at least we find out about examples that are broken enough to raise exceptions.

On Mon, Jun 23, 2008 at 16:51, Michael McNeil Forbes wrote:

...

On Mon, Jun 23, 2008 at 4:51 PM, Robert Kern wrote:

...

...

...

...

> random_array = np.random.rand(3,4) > random_array.shape (3,4) > random_array.max() < 1 True > random_array.min() > 0 True

Yes, this makes it doctestable, but you've destroyed the exampleness. It should be policy *not* to do this.

Well perhaps... but do you think that

rand(d0, d1, ..., dn) -> random values

is more exampley than

...

...

...

r = np.random.rand(3,2,4) r.shape (3,2,4)

?

No. It wasn't an example. It was a specification of the call signature because it is in an extension module, so the call signature is not available like it is for pure Python functions. Thus, it needs to be given in the docstring. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

2008/6/24 Stéfan van der Walt :

It should be fairly easy to execute the example code, just to make sure it runs. We can always work out a scheme to test its validity later.

Mike Hansen just explained to me that the Sage doctest system sets the random seed before executing each test. If we address a) Random variables b) Plotting representations and c) Endianness we're probably halfway there. Regards Stéfan

On Mon, Jun 23, 2008 at 4:58 PM, Michael Abshoff wrote:

...

a) Random variables

we have some small extensions to the doctesting framework that allow us to mark doctests as "#random" so that the result it not checked. Carl Witty wrote some code that makes the random number generator in a lot of the Sage components behave consistently on all supported platforms.

Care to share? (BSD, we can't even look at the Sage code). Cheers, f

On Mon, Jun 23, 2008 at 5:26 PM, Michael Abshoff wrote:

I am not the author, so I need to find out who wrote the code, but I am sure it can be made BSD. We are also working on "doctest+timeit" to hunt for performance regressions, but that one is not ready for prime time yet.

Great, thanks. Cheers, f

Charles R Harris wrote:

On Mon, Jun 23, 2008 at 5:58 PM, Michael Abshoff mailto:michael.abshoff@googlemail.com> wrote:

Stéfan van der Walt wrote: > 2008/6/24 Stéfan van der Walt mailto:stefan@sun.ac.za>: >> It should be fairly easy to execute the example code, just to make >> sure it runs. We can always work out a scheme to test its validity >> later.

Hi,

> Mike Hansen just explained to me that the Sage doctest system sets the > random seed before executing each test. If we address > > a) Random variables

we have some small extensions to the doctesting framework that allow us to mark doctests as "#random" so that the result it not checked. Carl Witty wrote some code that makes the random number generator in a lot of the Sage components behave consistently on all supported platforms.

Hi,

But there is more than one possible random number generator. If you do that you are tied into one kind of generator and one kind of initialization implementation.

Chuck

Correct, but so far Carl has hooked into six out of the many random number generators in the various components of Sage. This way we can set a global seed and also more easily reproduce issues with algorithms where randomness plays a role without being forced to be on the same platform. There are still doctests in Sage where the randomness comes from sources not in randgen (Carl's code), but sooner or later we will get around to all of them. Cheers, Michael

------------------------------------------------------------------------

_______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

On Mon, Jun 23, 2008 at 22:53, Anne Archibald wrote:

2008/6/23 Michael Abshoff :

...

Correct, but so far Carl has hooked into six out of the many random number generators in the various components of Sage. This way we can set a global seed and also more easily reproduce issues with algorithms where randomness plays a role without being forced to be on the same platform. There are still doctests in Sage where the randomness comes from sources not in randgen (Carl's code), but sooner or later we will get around to all of them.

Doesn't this mean you can't change your implementation of random number generators (for example choosing a different implementation of generation of normally-distributed random numbers, or replacing the Mersenne Twister) without causing countless doctests to fail meaninglessly?

It's not that bad. After you've verified that your new code works, you regenerate the examples. You check in both at the same time. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

Mon, 23 Jun 2008 10:03:28 +0200, Stéfan van der Walt wrote:

2008/6/23 Alan McIntyre :

...

Some docstrings have examples of how to use the function that aren't executable code (see numpy.core.defmatrix.bmat for an example) in their current form. Should these examples have the ">>>" removed from them to avoid them being picked up as doctests?

The examples written for the random module warrants the same question. First and foremost, the docstrings are there to illustrate to users how to use the code; second, to serve as tests.

Example codes should run, but I'm not sure whether they should always be valid doctests.

In the `bmat` example, I would remove the '>>>' like you suggested.

"Schematic" code (such as that currently in numpy.bmat) that doesn't run probably shouldn't be written with >>>, and for it the ReST block quote syntax is also looks OK. But I'm personally not in favor of a distinction between a "doctest" and a "code sample", as the difference is not of interest to the main target audience who reads the docstrings (or the reference documentation generated based on them). As I see it, Numpy has a test architecture that is separate from doctests, so that most of the bonus doctests gives us is ensuring that all of our examples run without errors and produce expected results. It's a bit unfortunate though that the doctest directives are as obtrusive as they are and only apply to a single line. One problem that I see now is quite annoying in the sample codes using matplotlib: matplotlib functions tend to return some objects whose <repr> contains a memory address, which causes the code to fit badly in a doctest. This can be worked around (ELLIPSIS, assigning to a variable), but I don't see a clean way. (I'm not so worried here about plot windows popping up as they can be worked around by monkey-patching matplotlib.show and choosing a non-graphical backend.) Another point related to numpy are blank lines often appearing in array printouts (the text <BLANKLINE> is not a pretty sight in documentation). Also, NORMALIZE_WHITESPACE is useful for reducing the whitespace in array printout. -- Pauli Virtanen