[Numpy-discussion] ANN: HDF5 for Python 1.1

[Stephen Simmons]

Hi Andrew,

Do you have any plans to support LZO compression in h5py?
 
I have lots of LZO-compressed datasets created with PyTables.
There's a real barrier to using both h5py and PyTables if the fast
decompressor options are just LZF on h5py and LZO on PyTables.

Many thanks
Stephen


Andrew Collette wrote:
> =====================================
> Announcing HDF5 for Python (h5py) 1.1
> =====================================
>
> What is h5py?
> -------------
>
> HDF5 for Python (h5py) is a general-purpose Python interface to the
> Hierarchical Data Format library, version 5.  HDF5 is a versatile,
> mature scientific software library designed for the fast, flexible
> storage of enormous amounts of data.
>
> >From a Python programmer's perspective, HDF5 provides a robust way to
> store data, organized by name in a tree-like fashion.  You can create
> datasets (arrays on disk) hundreds of gigabytes in size, and perform
> random-access I/O on desired sections.  Datasets are organized in a
> filesystem-like hierarchy using containers called "groups", and
> accesed using the tradional POSIX /path/to/resource syntax.
>
> In addition to providing interoperability with existing HDF5 datasets
> and platforms, h5py is a convienient way to store and retrieve
> arbitrary NumPy data and metadata.
>
>
> New features in 1.1
> -------------------
>
>   - A new compression filter based on the LZF library, which provides
>     transparent compression many times faster than the standard HDF5
>     GZIP filter.
>
>   - Efficient broadcasting using HDF5 hyperslab selections; for example,
>     you can write to a (2000 x 100 x 50) selection from a (100 x 50)
>     source array.
>
>   - Now supports the NumPy boolean type
>
>   - Auto-completion for IPython 0.9.X (contributed by Darren Dale)
>
>   - Installable via easy_install
>
>
> Standard features
> -----------------
>
>   - Supports storage of NumPy data of the following types:
>
>     * Integer/Unsigned Integer
>     * Float/Double
>     * Complex/Double Complex
>     * Compound ("recarray")
>     * Strings
>     * Boolean
>     * Array (as members of a compound type only)
>     * Void
>
>   - Random access to datasets using the standard NumPy slicing syntax,
>     including fancy indexing and point-based selection
>
>   - Transparent compression of datasets using GZIP, LZF or SZIP,
>     and error-detection using Fletcher32
>
>   - "Pythonic" interface supporting dictionary and NumPy-array metaphors
>     for the high-level HDF5 abstrations like groups and datasets
>
>   - A comprehensive, object-oriented wrapping of the HDF5 low-level C API
>     via Cython, in addition to the NumPy-like high-level interface.
>
>   - Supports many new features of HDF5 1.8, including recursive iteration
>     over entire files and in-library copy operations on the file tree
>
>   - Thread-safe
>
>
> Where to get it
> ---------------
>
> * Main website, documentation:  http://h5py.alfven.org
>
> * Downloads, bug tracker:       http://h5py.googlecode.com
>
>
> Requires
> --------
>
> * Linux, Mac OS-X or Windows
>
> * Python 2.5 (Windows), Python 2.5 or 2.6 (Linux/Mac OS-X)
>
> * NumPy 1.0.3 or later
>
> * HDF5 1.6.5 or later (including 1.8); HDF5 is included with
>   the Windows version.
>
>
> Thanks
> ------
>
> Thanks to D. Dale, E. Lawrence and other for their continued support
> and comments.  Also thanks to the Francesc Alted and the PyTables project,
> for inspiration and generously providing their code to the community. Thanks
> to everyone at the HDF Group for creating such a useful piece of software.
> _______________________________________________
> Numpy-discussion mailing list
> Numpy-discussion at scipy.org
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>
>