Announcing carray 0.5 ===================== What's new ---------- carray 0.5 supports completely transparent storage on-disk in addition to memory. That means that everything that can be done with an in-memory container can be done using the disk instead. The advantages of a disk-based container is that your addressable space is much larger than just your available memory. Also, as carray is based on a chunked and compressed data layout based on the super-fast Blosc compression library, and the different cache levels existing in both modern operating systems and the internal carray machinery, the data access speed is very good. The format chosen for the persistence layer is based on the 'bloscpack' library (thanks to Valentin Haenel for his inspiration) and described in 'persistence.rst', although not everything has been implemented yet. You may want to contribute by proposing enhancements to it. See: https://github.com/FrancescAlted/carray/wiki/PersistenceProposal CAVEAT: The bloscpack format is still evolving, so don't trust on forward compatibility of the format, at least until 1.0, where the internal format will be declared frozen. For more detailed info, see the release notes in: https://github.com/FrancescAlted/carray/wiki/Release-0.5 What it is ---------- carray is a chunked container for numerical data. Chunking allows for efficient enlarging/shrinking of data container. In addition, it can also be compressed for reducing memory/disk needs. The compression process is carried out internally by Blosc, a high-performance compressor that is optimized for binary data. carray can use numexpr internally so as to accelerate many vector and query operations (although it can use pure NumPy for doing so too). numexpr can use optimize the memory usage and use several cores for doing the computations, so it is blazing fast. Moreover, with the introduction of a carray/ctable disk-based container (in version 0.5), it can be used for seamlessly performing out-of-core computations. carray comes with an exhaustive test suite and fully supports both 32-bit and 64-bit platforms. Also, it is typically tested on both UNIX and Windows operating systems. Resources --------- Visit the main carray site repository at: http://github.com/FrancescAlted/carray You can download a source package from: http://carray.pytables.org/download Manual: http://carray.pytables.org/docs/manual Home of Blosc compressor: http://blosc.pytables.org User's mail list: carray@googlegroups.com http://groups.google.com/group/carray ---- Enjoy! -- Francesc Alted