[ANN] PyTables release 0.2

[Francesc Alted]

Announcing PyTables 0.2
-----------------------

What's new
-----------

- Numerical Python arrays supported!
- Much improved documentation
- Programming API almost stable
- Improved navegability across the object tree
- Added more unit tests (there are almost 50)
- Dropped HDF5_HL dependency (a tailored version is included in sources now)
- License changed from LGPL to BSD

What is
-------

The goal of PyTables is to enable the end user to manipulate easily
scientific data tables and Numerical Python objects (new in 0.2!)  in
a persistent hierarchical structure. The foundation of the underlying
hierachical data organization is the excellent HDF5 library
(http://hdf.ncsa.uiuc.edu/HDF5). Right now, PyTables provides limited
support of all the HDF5 functions, but I hope to add the more
interesting ones (for PyTables needs) in the near future.
Nonetheless, this package is not intended to serve as a complete
wrapper for the entire HDF5 API.

A table is defined as a collection of records whose values are stored
in fixed-length fields. All records have the same structure and all
values in each field have the same data type.  The terms
"fixed-length" and strict "data types" seems to be quite a strange
requirement for an interpreted language like Python, but they serve a
useful function if the goal is to save very large quantities of data
(such as is generated by many scientifc applications, for example) in
an efficient manner that reduces demand on CPU time and I/O.

In order to emulate records (C structs in HDF5) in Python, PyTables
implements a special metaclass that detects errors in field
assignments as well as range overflows. PyTables also provides a
powerful interface to process table data.

Quite a bit effort has been invested to make browsing the hierarchical
data structure a pleasant experience. PyTables implements just three
(orthogonal) easy-to-use methods for browsing.

What is HDF5?
-------------

For those people who know nothing about HDF5, it is is a general
purpose library and file format for storing scientific data made at
NCSA. HDF5 can store two primary objects: datasets and groups. A
dataset is essentially a multidimensional array of data elements, and
a group is a structure for organizing objects in an HDF5 file. Using
these two basic constructs, one can create and store almost any kind of
scientific data structure, such as images, arrays of vectors, and
structured and unstructured grids. You can also mix and match them in
HDF5 files according to your needs.

How fast is it?
---------------

Despite to be an alpha version and that there is lot of room for
improvements (it's still CPU bounded!), PyTables can read and write
tables quite fast. But, if you want some (very preliminary) figures
(just to know orders of magnitude), in a AMD Athlon@900 it can
currently read from 40000 up to 60000 records/s and write from 5000 up
to 13000 records/s. Raw data speed in read mode ranges from 1 MB/s up
to 2 MB/s, and it drops to the 200 KB/s - 600 KB/s range for writes.

Go to http://pytables.sf.net/bench.html for a somewhat more detailed
description of this small (and synthetic) benchmark.

Anyway, this is only the beginning (premature optimization is the root
of all evils, you know ;-).

Platforms
---------

I'm using Linux as the main development platform, but PyTables should
be easy to compile/install on other UNIX machines. Thanks to Scott
Prater, this package has passed all the tests on a UltraSparc platform
with Solaris 7. It also compiles and passes all the tests on a SGI
Origin2000 with MIPS R12000 processors and running IRIX 6.5.

If you are using Windows and you get the library to work, please let
me know.

An example?
-----------

At the bottom of this message there is some code (less that 100 lines
and only less than half being real code) that shows basic capabilities
of PyTables.

Web site
--------

Go to the PyTables web site for more details:

http://pytables.sf.net/

Final note
----------

This is second alpha release, and probably last alpha, so it is
still time if you want to suggest some API addition/change or
addition/change of any useful missing capability. Let me know of any
bugs, suggestions, gripes, kudos, etc. you may have.

-- Francesc Alted
falted@openlc.org


*-*-*-**-*-*-**-*-*-**-*-*- Small code example  *-*-*-**-*-*-**-*-*-**-*-*-*

"""Small but almost complete example showing the PyTables mode of use.

As a result of execution, a 'tutorial1.h5' file is created. You can
look at it with whatever HDF5 generic utility, like h5ls, h5dump or
h5view.

"""


import sys
from Numeric import *
from tables import *


	#'-**-**-**-**-**-**- user record definition  -**-**-**-**-**-**-**-'

# Define a user record to characterize some kind of particles
class Particle(IsRecord):
    name        = '16s'  # 16-character String
    idnumber    = 'Q'    # unsigned long long (i.e. 64-bit integer)
    TDCcount    = 'B'    # unsigned byte
    ADCcount    = 'H'    # unsigned short integer
    grid_i      = 'i'    # integer
    grid_j      = 'i'    # integer
    pressure    = 'f'    # float  (single-precision)
    energy      = 'd'    # double (double-precision)

print
print	'-**-**-**-**-**-**- file creation  -**-**-**-**-**-**-**-'

# The name of our HDF5 filename
filename = "tutorial1.h5"
    
print "Creating file:", filename

# Open a file in "w"rite mode
h5file = openFile(filename, mode = "w", title = "Test file")

print
print	'-**-**-**-**-**-**- group an table creation  -**-**-**-**-**-**-**-'

# Create a new group under "/" (root)
group = h5file.createGroup("/", 'detector', 'Detector information')
print "Group '/detector' created"

# Create one table on it
table = h5file.createTable(group, 'readout', Particle(), "Readout example")
print "Table '/detector/readout' created"

# Get a shortcut to the record object in table
particle = table.record

# Fill the table with 10 particles
for i in xrange(10):
    # First, assign the values to the Particle record
    particle.name  = 'Particle: %6d' % (i)
    particle.TDCcount = i % 256    
    particle.ADCcount = (i * 256) % (1 << 16)
    particle.grid_i = i 
    particle.grid_j = 10 - i
    particle.pressure = float(i*i)
    particle.energy = float(particle.pressure ** 4)
    particle.idnumber = i * (2 ** 34)  # This exceeds long integer range
    # Insert a new particle record
    table.appendAsRecord(particle)      

# Flush the buffers for table
table.flush()

print
print	'-**-**-**-**-**-**- table data reading & selection  -**-**-**-**-**-'

# Read actual data from table. We are interested in collecting pressure values
# on entries where TDCcount field is greater than 3 and pressure less than 50
pressure = [ x.pressure for x in table.readAsRecords()
	         if x.TDCcount > 3 and x.pressure < 50 ]
print "Last record read:"
print x
print "Field pressure elements satisfying the cuts ==>", pressure

# Read also the names with the same cuts
names = [ x.name for x in table.readAsRecords()
	      if x.TDCcount > 3 and x.pressure < 50 ]

print
print	'-**-**-**-**-**-**- array object creation  -**-**-**-**-**-**-**-'

print "Creating a new group called '/columns' to hold new arrays"
gcolumns = h5file.createGroup(h5file.root, "columns", "Pressure and Name")

print "Creating a Numeric array called 'pressure' under '/columns' group"
h5file.createArray(gcolumns, 'pressure', array(pressure), 
                   "Pressure column selection")

print "Creating another Numeric array called 'name' under '/columns' group"
h5file.createArray('/columns', 'name', array(names),
                   "Name column selection")

# Close the file
h5file.close()
print "File '"+filename+"' created"