[Numpy-discussion] I/O documentation and code

[Ralf Gommers]

On Sat, Jun 20, 2009 at 6:08 PM, Robert Kern <robert.kern at gmail.com> wrote:

> On Sat, Jun 20, 2009 at 16:33, Ralf Gommers <ralf.gommers at googlemail.com>
> wrote:
> >
> > Hi,
> >
> > I'm working on the I/O documentation, and have a bunch of questions.
> >
> > 1. The npy/npz formats are documented in lib.format and in the NEP (
> http://svn.scipy.org/svn/numpy/trunk/doc/neps/npy-format.txt). Is
> lib.format the right place to add relevant parts of the NEP, or would
> doc.io be better?
>
> What parts?
>

- abstract (i.e. what is this, what's it good for)
- comparison with pickle, memmep
- most of the items in "Requirements"
- extension info (.npy/.npz, not enforced)

Right now lib.format does not contain the word "binary", or ".npy". We need
a complete description in the reference guide that functions like `save` and
`load` can reference.


>
> > Or create a separate page (maybe doc.npy_format)?
>
> Probably all of the implemented NEPs should have their own place in
> the documentation and other parts should reference the NEPs for
> technical detail.
>

Good point, NEPs should be somewhere in the docs. However, they do not seem
appropriate to refer users to, paragraphs like Rationale, Use Cases,
Implementation are not aimed directly at users.


>
> > And is the .npz format fixed or still in flux?
>
> It's not as formalized as the .npy format, but I expect it to be at
> least as solid as other code in numpy.
>
> > 2. Is the .npy format version number (now at 1.0) independent of the
> numpy version numbering, when is it incremented, and will it be backwards
> compatible?
>
> It is independent of numpy version numbering. If we do upgrade the
> format, the code in numpy.io will still be able to read and write 1.0
> files.
>
> > 4. This page http://www.scipy.org/Data_sets_and_examples talks about
> including data sets with scipy, has this happened? Would it be possible to
> include a single small dataset in numpy for use in examples?
>
> I think the dataset convention is entirely independent of numpy per
> se. The current version of this stuff is in the scikits.learn package:
>
> http://svn.scipy.org/svn/scikits/trunk/learn/scikits/learn/datasets/
>
> The proposal could be turned into an "informative" NEP, of course. It
> needs to be updated, though (e.g. it talks about not needing to
> combine masked arrays and record arrays, but this has already been
> done with the numpy.ma rewrite).
>

Interesting, might be useful for all sorts of examples in docstrings and
especially tutorial-style docs. David, do you still plan to put this forward
for inclusion in numpy or scipy?

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090620/7f5bd5ac/attachment.html>