Mailman 3 April 2007 - NumPy-Discussion

take(..., out=something) does not seem to work
by Jouni K. Seppänen 27 Apr '07

27 Apr '07

Hi, Either I have completely misunderstood what the out= arg to the take function is supposed to do, or there is a bug in numpy 1.0.2, as demonstrated below. First, some initialization: In [1]: from numpy import * In [2]: a=arange(12).reshape((3,4)) In [3]: a Out[3]: array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) Let's try take without an output argument: In [4]: take(a, [0,2], axis=1) Out[4]: array([[ 0, 2], [ 4, 6], [ 8, 10]]) Good. Now let's create an array of the right size and dtype, and use it for output: In [5]: b = zeros_like(_) In [6]: take(a, [0,2], axis=1, out=b) Out[6]: array([[ 0, 2], [ 4, 6], [ 8, 10]]) In [7]: b Out[7]: array([[0, 0], [0, 0], [0, 0]]) It seems that b was not affected at all. -- Jouni K. Seppänen http://www.iki.fi/jks

2 1

Re: [Numpy-discussion] Numerics vs. numpy
by Bruce Southey 27 Apr '07

27 Apr '07

Hi, Ed Schofield has created a patch for NumPy to work with BioPython (see below). Bruce ---------- Forwarded message ---------- From: bugzilla-daemon(a)portal.open-bio.org <bugzilla-daemon(a)portal.open-bio.org> Date: Mar 27, 2007 8:08 AM Subject: [Biopython-dev] [Bug 2251] New: [PATCH] NumPy support for BioPython To: biopython-dev(a)biopython.org http://bugzilla.open-bio.org/show_bug.cgi?id=2251 Summary: [PATCH] NumPy support for BioPython Product: Biopython Version: Not Applicable Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: P2 Component: Main Distribution AssignedTo: biopython-dev(a)biopython.org ReportedBy: edschofield(a)gmail.com The following patch adds support for NumPy in addition to Numeric. It does so with a thin wrapper layer at the C level and Python level, similar in purpose to (but less ambitious than) the numerix wrapper layer used by matplotlib. The patch is designed to prevent imports of incompatible compiled C modules in the case one installs Numeric after installing BioPython with NumPy support. It also changes the following: - C #include statements - Python import statements - references to obsoleted function names - the width of the dimension data types in cluster.c from int to intp (for 64-bit architectures) - the distutils setup.py file to supply the correct NumPy header locations. - the documentation (updating references to NumPy) It also fixes array boolean operators in MarkovModel.py, which were silently broken before. It applies to BioPython 1.43, as follows: $ tar xzvf biopython-1.43.tar.gz $ cd biopython-1.43 $ patch -p1 < /path/to/biopython-1.43-numpy-support-v5.patch -- Configure bugmail: http://bugzilla.open-bio.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee. _______________________________________________ Biopython-dev mailing list Biopython-dev(a)lists.open-bio.org http://lists.open-bio.org/mailman/listinfo/biopython-dev

1 0

Re: [Numpy-discussion] [Python-3000] PEP 31XX: A Type Hierarchy for Numbers (and other algebraic entities)
by David Goldsmith 27 Apr '07

27 Apr '07

Bill Janssen wrote: >> Jeffrey, is there any way you can drop the top of the tree and going >> straight from Number to Complex -> Real -> Rational -> Integer? These >> are the things that everyone with high school math will know. >> > > I think knowledge of the concepts of group, ring, and field is > supposed to be standard knowledge for any high-school senior -- isn't > this what the "new math" was all about?. Nah, AFAIK, not really. Of course, for those who remember Tom Lehrer, New Math was about learning that one could base number representation on bases other than 10 (even more fundamental to CS, no?), but I think it was rather more about broadening the standard curriculum to include more than just arithmetic drill, i.e., to more prominently include things like problem solving skills, (somewhat) more abstraction, etc. Along the latter lines, I do remember learning (and tutoring), by name, things like the symmetric, reflexive, transitive, associative, commutative, and distributive properties, but I didn't hear/see the words group, ring, or field (in a math context) 'til I got to college and was formally introduced to the subject of Linear Algebra (and lest you think my HS math curriculum was deficient and/or non-standard, I don't want to go in that direction, but trust me, it was more than sufficient and it was non-standard, but in the positive direction, not in the negative, and I graduated 1st in my class in math). As far as the scientific disciplines which one might reasonably expect a college major therein to have at least a "nodding acquaintance" w/ the classes of Abstract Algebra, certainly Math (and hopefully Appl. Math and Stats), probably Physics, perhaps Chemistry, CS if you say so, but even Engineering I'd say you're beginning to tread in uncertain territory, and Biology, etc., unless taken with a large dose of formal mathematics, fuhget-about-it. DG > But they're pretty > fundamental to computer science; anyone trying to do serious work in > the field (that is, anyone with a reason to read the PEP) should have > a nodding acquaintance with them. > > Bill > > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion(a)scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion >

1 0

Numerics vs. numpy
by O.Doehring＠cs.ucl.ac.uk 27 Apr '07

27 Apr '07

Dear community, I have a question regarding the installation process of the Biopython toolbox ( http://biopython.org/DIST/docs/install/Installation.html#htoc14 ): I want to make sure that Numerical Python is correctly installed under Windows XP which it is not in my case: >>> from Numeric import * Traceback (most recent call last): File "<pyshell#0>", line 1, in <module> from Numeric import * ImportError: No module named Numeric But if I am doing: >>> from numpy import * >>> it works. This is not too surprised as it gets installed under numpy in the folder: C:\Python25\Lib\site-packages But the problem is that if I am making various library calls to Biopython I get an error, e.g. for handling PDB-stuff: File "C:\Python25\Lib\site-packages\Bio\PDB\PDBParser.py", line 9, in <module> from Numeric import array, Float0 ImportError: No module named Numeric For help I would be very grateful. Thanks. Yours, Orlando

2 1

TypeError when writing .data with python 2.5.1
by Glen W. Mabey 27 Apr '07

27 Apr '07

The following works fine with a python 2.4.3 installation, but with 2.5.1 I get: In [1]:import numpy In [2]:fid = file( '/tmp/tfile', 'w' ) In [3]:fid.write( numpy.zeros( 55 ).data ) --------------------------------------------------------------------------- <type 'exceptions.TypeError'> Traceback (most recent call last) /home/gmabey/src/R9619_dev_acqlibweb/Projects/R9619_NChannelDetection/NED/<ipython console> in <module>() <type 'exceptions.TypeError'>: non-character (or 8-bit) array cannot be interpreted as character buffer > <ipython console>(1)<module>() I haven't tried it with python 2.5 Glen

2 3

PEP 31XX: A Type Hierarchy for Numbers (and other algebraic entities)
by Jeffrey Yasskin 27 Apr '07

27 Apr '07

Here's a draft of the numbers ABCs PEP. The most up to date version will live in the darcs repository at http://jeffrey.yasskin.info/darcs/PEPs/pep-3141.txt (unless the number changes) for now. Naming a PEP about numbers 3.141 seems cute, but of course, I don't get to pick the number. :) This is my first PEP, so apologies for any obvious mistakes. I'd particularly like the numpy people's input on whether I've gotten floating-point support right. Thanks, Jeffrey Yasskin ------------------------------- PEP: 3141 Title: A Type Hierarchy for Numbers (and other algebraic entities) Version: $Revision: 54928 $ Last-Modified: $Date: 2007-04-23 16:37:29 -0700 (Mon, 23 Apr 2007) $ Author: Jeffrey Yasskin <jyasskin(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 23-Apr-2007 Post-History: Not yet posted Abstract ======== This proposal defines a hierarchy of Abstract Base Classes (ABCs) [#pep3119] to represent numbers and other algebraic entities similar to numbers. It proposes: * A hierarchy of algebraic concepts, including monoids, groups, rings, and fields with successively more operators and constraints on their operators. This will be added as a new library module named "algebra". * A hierarchy of specifically numeric types, which can be converted to and from the native Python types. This will be added as a new library module named "numbers". Rationale ========= Functions that take numbers as arguments should be able to determine the properties of those numbers, and if and when overloading based on types is added to the language, should be overloadable based on the types of the arguments. This PEP defines some abstract base classes that are useful in numerical calculations. A function can check that variable is an instance of one of these classes and then rely on the properties specified for them. Of course, the language cannot check these properties, so where I say something is "guaranteed", I really just mean that it's one of those properties a user should be able to rely on. This PEP tries to find a balance between providing fine-grained distinctions and specifying types that few people will ever use. Specification ============= Although this PEP uses terminology from PEP3119, the hierarchy is meaningful for any systematic method of defining sets of classes. **Todo:** link to the Interfaces PEP when it's ready. I'm also using the extra notation from [#pep3107] (annotations) to specify some types. Object oriented systems have a general problem in constraining functions that take two arguments. To take addition as an example, ``int(3) + int(4)`` is defined, and ``vector(1,2,3) + vector(3,4,5)`` is defined, but ``int(3) + vector(3,4,5)`` doesn't make much sense. So ``a + b`` is not guaranteed to be defined for any two instances of ``AdditiveGroup``, but it is guaranteed to be defined when ``type(a) == type(b)``. On the other hand, ``+`` does make sense for any sorts of numbers, so the ``Complex`` ABC refines the properties for plus so that ``a + b`` is defined whenever ``isinstance(a,Complex) and isinstance(b,Complex)``, even if ``type(a) != type(b)``. Monoids (http://en.wikipedia.org/wiki/Monoid) consist of a set with an associative operation, and an identity element under that operation. **Open issue**: Is a @classmethod the best way to define constants that depend only on the type?:: class MonoidUnderPlus(Abstract): """+ is associative but not necessarily commutative and has an identity given by plus_identity(). Subclasses follow the laws: a + (b + c) === (a + b) + c a.plus_identity() + a === a === a + a.plus_identity() Sequences are monoids under plus (in Python) but are not AdditiveGroups. """ @abstractmethod def __add__(self, other): raise NotImplementedError @classmethod @abstractmethod def plus_identity(cls): raise NotImplementedError I skip ordinary non-commutative groups here because I don't have any common examples of groups that use ``+`` as their operator but aren't commutative. If we find some, the class can be added later.:: class AdditiveGroup(MonoidUnderPlus): """Defines a commutative group whose operator is +, and whose inverses are produced by -x. See http://en.wikipedia.org/wiki/Abelian_group. Where a, b, and c are instances of the same subclass of AdditiveGroup, the operations should follow these laws, where 'zero' is a.__class__.zero(). a + b === b + a (a + b) + c === a + (b + c) zero + a === a a + (-a) === zero a - b === a + -b Some abstract subclasses, such as Complex, may extend the definition of + to heterogenous subclasses, but AdditiveGroup only guarantees it's defined on arguments of exactly the same types. Vectors are AdditiveGroups but are not Rings. """ @abstractmethod def __add__(self, other): """Associative commutative operation, whose inverse is negation.""" raise NotImplementedError **Open issue:** Do we want to give people a choice of which of the following to define, or should we pick one arbitrarily?:: def __neg__(self): """Must define this or __sub__().""" return self.zero() - self def __sub__(self, other): """Must define this or __neg__().""" return self + -other @classmethod @abstractmethod def zero(cls): """A better name for +'s identity as we move into more mathematical domains.""" raise NotImplementedError @classmethod def plus_identity(cls): return cls.zero() Including Semiring (http://en.wikipedia.org/wiki/Semiring) would help a little with defining a type for the natural numbers. That can be split out once someone needs it (see ``IntegralDomain`` for how).:: class Ring(AdditiveGroup): """A mathematical ring over the operations + and *. See http://en.wikipedia.org/wiki/Ring_%28mathematics%29. In addition to the requirements of the AdditiveGroup superclass, a Ring has an associative but not necessarily commutative multiplication operation with identity (one) that distributes over addition. A Ring can be constructed from any integer 'i' by adding 'one' to itself 'i' times. When R is a subclass of Ring, the additive identity is R(0), and the multiplicative identity is R(1). Matrices are Rings but not Commutative Rings or Division Rings. The quaternions are a Division Ring but not a Field. The integers are a Commutative Ring but not a Field. """ @abstractmethod def __init__(self, i:int): """An instance of a Ring may be constructed from an integer. This may be a lossy conversion, as in the case of the integers modulo N.""" pass @abstractmethod def __mul__(self, other): """Satisfies: a * (b * c) === (a * b) * c one * a === a a * one === a a * (b + c) === a * b + a * c where one == a.__class__(1) """ raise NotImplementedError @classmethod def zero(cls): return cls(0) @classmethod def one(cls): return cls(1) I'm skipping both CommutativeRing and DivisionRing here. class Field(Ring): """The class Field adds to Ring the requirement that * be a commutative group operation except that zero does not have an inverse. See http://en.wikipedia.org/wiki/Field_%28mathematics%29. Practically, that means we can define division on a Field. The additional laws are: a * b === b * a a / a === a.__class_(1) # when a != a.__class__(0) Division lets us construct a Field from any Python float, although the conversion is likely to be lossy. Some Fields include the real numbers, rationals, and integers mod a prime. Python's ``float`` resembles a Field closely. """ def __init__(self, f:float): """A Field should be constructible from any rational number, which includes Python floats.""" pass @abstractmethod def __div__(self, divisor): raise NotImplementedError Division is somewhat complicated in Python. You have both __floordiv__ and __div__, and ints produce floats when they're divided. For the purposes of this hierarchy, ``__floordiv__(a, b)`` is defined by ``floor(__div__(a, b))``, and, since int is not a subclass of Field, it's allowed to do whatever it wants with __div__. There are four more reasonable classes that I'm skipping here in the interest of keeping the initial library simple. They are: ``Algebraic`` Rational powers of its elements are defined (and maybe a few other operations) (http://en.wikipedia.org/wiki/Algebraic_number). Complex numbers are the most well-known algebraic set. Real numbers are _not_ algebraic, but Python does define these operations on floats, which makes defining this class somewhat difficult. ``Trancendental`` The elementary functions (http://en.wikipedia.org/wiki/Elementary_function) are defined. These are basically arbitrary powers, trig functions, and logs, the contents of ``cmath``. The following two classes can be reasonably combined with ``Integral`` for now. ``IntegralDomain`` Defines __divmod__. (http://darcs.haskell.org/numericprelude/docs/html/Algebra-IntegralDomain.ht…) ``PrincipalIdealDomain`` Defines gcd and lcm. (http://darcs.haskell.org/numericprelude/docs/html/Algebra-PrincipalIdealDom…) If someone needs to split them later, they can use code like:: import numbers class IntegralDomain(Ring): ... numbers.Integral.__bases__ = (IntegralDomain,) + numbers.Integral.__bases__ Finally, we get to numbers. This is where we switch from the "algebra" module to the "numbers" module.:: class Complex(Ring, Hashable): """The ``Complex`` ABC indicates that the value lies somewhere on the complex plane, not that it in fact has a complex component: ``int`` is a subclass of ``Complex``. Because these actually represent complex numbers, they can be converted to the ``complex`` type. ``Complex`` finally gets around to requiring its subtypes to be immutable so they can be hashed in a standard way. ``Complex`` also requires its operations to accept heterogenous arguments. Subclasses should override the operators to be more accurate when they can, but should fall back on the default definitions to handle arguments of different (Complex) types. **Open issue:** __abs__ doesn't fit here because it doesn't exist for the Gaussian integers (http://en.wikipedia.org/wiki/Gaussian_integer). In fact, it only exists for algebraic complex numbers and real numbers. We could define it in both places, or leave it out of the ``Complex`` classes entirely and let it be a custom extention of the ``complex`` type. The Gaussian integers are ``Complex`` but not a ``Field``. """ @abstractmethod def __complex__(self): """Any Complex can be converted to a native complex object.""" raise NotImplementedError def __hash__(self): return hash(complex(self)) @abstractmethod def real(self) => Real: raise NotImplementedError @abstractmethod def imag(self) => Real: raise NotImplementedError @abstractmethod def __add__(self, other): """The other Ring operations should be implemented similarly.""" if isinstance(other, Complex): return complex(self) + complex(other) else: return NotImplemented ``FractionalComplex(Complex, Field)`` might fit here, except that it wouldn't give us any new operations. class Real(Complex, TotallyOrdered): """Numbers along the real line. Some subclasses of this class may contain NaNs that are not ordered with the rest of the instances of that type. Oh well. **Open issue:** what problems will that cause? Is it worth it in order to get a straightforward type hierarchy? """ @abstractmethod def __float__(self): raise NotImplementedError def __complex__(self): return complex(float(self)) def real(self) => self.__class__: return self def imag(self) => self.__class__: return self.__class__(0) def __abs__(self) => self.__class__: if self < 0: return -self else: return self class FractionalReal(Real, Field): """Rationals and floats. This class provides concrete definitions of the other four methods from properfraction and allows you to convert fractional reals to integers in a disciplined way. """ @abstractmethod def properfraction(self) => (int, self.__class__): """Returns a pair (n,f) such that self == n+f, and: * n is an integral number with the same sign as self; and * f is a fraction with the same type and sign as self, and with absolute value less than 1. """ raise NotImplementedError def floor(self) => int: n, r = self.properfraction() if r < 0 then n - 1 else n def ceiling(self) => int: ... def __trunc__(self) => int: ... def round(self) => int: ... **Open issue:** What's the best name for this class? RealIntegral? Integer?:: class Integral(Real): """Integers!""" @abstractmethod def __int__(self): raise NotImplementedError def __float__(self): return float(int(self)) @abstractmethod def __or__(self, other): raise NotImplementedError @abstractmethod def __xor__(self, other): raise NotImplementedError @abstractmethod def __and__(self, other): raise NotImplementedError @abstractmethod def __lshift__(self, other): raise NotImplementedError @abstractmethod def __rshift__(self, other): raise NotImplementedError @abstractmethod def __invert__(self): raise NotImplementedError Floating point values may not exactly obey several of the properties you would expect from their superclasses. For example, it is possible for ``(large_val + -large_val) + 3 == 3``, but ``large_val + (-large_val + 3) == 0``. On the values most functions deal with this isn't a problem, but it is something to be aware of. Types like this inherit from ``FloatingReal`` so that functions that care can know to use a numerically stable algorithm on them. **Open issue:** Is this the proper way to handle floating types?:: class FloatingReal: """A "floating" number is one that is represented as ``mantissa * radix**exponent`` where mantissa, radix, and exponent are all integers. Subclasses of FloatingReal don't follow all the rules you'd expect numbers to follow. If you really care about the answer, you have to use numerically stable algorithms, whatever those are. **Open issue:** What other operations would be useful here? These include floats and Decimals. """ @classmethod @abstractmethod def radix(cls) => int: raise NotImplementedError @classmethod @abstractmethod def digits(cls) => int: """The number of significant digits of base cls.radix().""" raise NotImplementedError @classmethod @abstractmethod def exponentRange(cls) => (int, int): """A pair of the (lowest,highest) values possible in the exponent.""" raise NotImplementedError @abstractmethod def decode(self) => (int, int): """Returns a pair (mantissa, exponent) such that mantissa*self.radix()**exponent == self.""" raise NotImplementedError Inspiration =========== http://hackage.haskell.org/trac/haskell-prime/wiki/StandardClasses http://repetae.net/john/recent/out/classalias.html References ========== .. [#pep3119] Introducing Abstract Base Classes (http://www.python.org/dev/peps/pep-3119/) .. [#pep3107] Function Annotations (http://www.python.org/dev/peps/pep-3107/) .. [3] Possible Python 3K Class Tree?, wiki page created by Bill Janssen (http://wiki.python.org/moin/AbstractBaseClasses) .. [#numericprelude] NumericPrelude: An experimental alternative hierarchy of numeric type classes (http://darcs.haskell.org/numericprelude/docs/html/index.html) Acknowledgements ---------------- Thanks to Neil Norwitz for helping me through the PEP process. The Haskell Numeric Prelude [#numericprelude] nicely condensed a lot of experience with the Haskell numeric hierarchy into a form that was relatively easily adaptable to Python. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

9 13

Questions about converting to numpy
by Russell E. Owen 26 Apr '07

26 Apr '07

So I finally bit the bullet and converted most of my code from Numeric and numarray to numpy. (I haven't yet tried to convert one package that makes heavy use of nd_image and has C extensions). But it left me with a few questions: - What exception does numpy throw if it runs out of memory? (I can try harder to make it do that, but trying to chew up all memory tends to slow the machine down and my first tests weren't successful) -- the equivalent to numarray.memory.error. The numpy book is silent on the issue of what exceptions numpy can throw (at least the index was). - Is there a list of the data types that we can expect to be available on all regular platforms (including 32-bit linux, MacOS X and Windows) and of usual speed for computations (instead of some kind of slow emulation)? - Even after reading the book I'm not really clear on why one would use numpy.float_ instead of numpy.float or float for day-to-day programming (where the size doesn't matter enough to use float64 or whatever). Any hints? Overall it went fine. I'm not entirely sure I caught everything yet, but the new code seems to be working. -- Russell

3 5

[ANN] PyTables 2.0rc1 released
by Ivan Vilata i Balaguer 26 Apr '07

26 Apr '07

Hi all, The development of the second major release of PyTables is steadily approaching its goal! Today we want to announce the *first release candidate version of PyTables 2.0*, i.e. PyTables 2.0rc1. This version settles the API and file format for the following PyTables 2.0 series. No more features will be added until the final version of 2.0 is released, so we do enter a time for exhaustive platform testing and fixing the last remaining bugs, as well as updating any outdated documentation. Your collaboration is very important in this stage of development, so we encourage you to download PyTables, test it, and report any problems you find or any suggestions you have. Thank you! And now, the official announcement: ============================ Announcing PyTables 2.0rc1 ============================ PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. You can download a source package of the version 2.0rc1 with generated PDF and HTML docs and binaries for Windows from http://www.pytables.org/download/preliminary/ For an on-line version of the manual, visit: http://www.pytables.org/docs/manual-2.0rc1 Please have in mind that some sections in the manual can be obsolete (specially the "Optimization tips" chapter). Other chapters should be fairly up-to-date though (although still a bit in state of flux). In case you want to know more in detail what has changed in this version, have a look at ``RELEASE_NOTES.txt``. Find the HTML version for this document at: http://www.pytables.org/moin/ReleaseNotes/Release_2.0rc1 If you are a user of PyTables 1.x, probably it is worth for you to look at ``MIGRATING_TO_2.x.txt`` file where you will find directions on how to migrate your existing PyTables 1.x apps to the 2.0 version. You can find an HTML version of this document at http://www.pytables.org/moin/ReleaseNotes/Migrating_To_2.x Keep reading for an overview of the most prominent improvements in PyTables 2.0 series. New features of PyTables 2.0 ============================ - NumPy is finally at the core! That means that PyTables no longer needs numarray in order to operate, although it continues to be supported (as well as Numeric). This also means that you should be able to run PyTables in scenarios combining Python 2.5 and 64-bit platforms (these are a source of problems with numarray/Numeric because they don't support this combination as of this writing). - Most of the operations in PyTables have experimented noticeable speed-ups (sometimes up to 2x, like in regular Python table selections). This is a consequence of both using NumPy internally and a considerable effort in terms of refactorization and optimization of the new code. - Combined conditions are finally supported for in-kernel selections. So, now it is possible to perform complex selections like:: result = [ row['var3'] for row in table.where('(var2 < 20) | (var1 == "sas")') ] or:: complex_cond = '((%s <= col5) & (col2 <= %s)) ' \ '| (sqrt(col1 + 3.1*col2 + col3*col4) > 3)' result = [ row['var3'] for row in table.where(complex_cond % (inf, sup)) ] and run them at full C-speed (or perhaps more, due to the cache-tuned computing kernel of Numexpr, which has been integrated into PyTables). - Now, it is possible to get fields of the ``Row`` iterator by specifying their position, or even ranges of positions (extended slicing is supported). For example, you can do:: result = [ row[4] for row in table # fetch field #4 if row[1] < 20 ] result = [ row[:] for row in table # fetch all fields if row['var2'] < 20 ] result = [ row[1::2] for row in # fetch odd fields table.iterrows(2, 3000, 3) ] in addition to the classical:: result = [row['var3'] for row in table.where('var2 < 20')] - ``Row`` has received a new method called ``fetch_all_fields()`` in order to easily retrieve all the fields of a row in situations like:: [row.fetch_all_fields() for row in table.where('column1 < 0.3')] The difference between ``row[:]`` and ``row.fetch_all_fields()`` is that the former will return all the fields as a tuple, while the latter will return the fields in a NumPy void type and should be faster. Choose whatever fits better to your needs. - Now, all data that is read from disk is converted, if necessary, to the native byteorder of the hosting machine (before, this only happened with ``Table`` objects). This should help to accelerate applications that have to do computations with data generated in platforms with a byteorder different than the user machine. - The modification of values in ``*Array`` objects (through __setitem__) now doesn't make a copy of the value in the case that the shape of the value passed is the same as the slice to be overwritten. This results in considerable memory savings when you are modifying disk objects with big array values. - All leaf constructors (except for ``Array``) have received a new ``chunkshape`` argument that lets the user explicitly select the chunksizes for the underlying HDF5 datasets (only for advanced users). - All leaf constructors have received a new parameter called ``byteorder`` that lets the user specify the byteorder of their data *on disk*. This effectively allows to create datasets in other byteorders than the native platform. - Native HDF5 datasets with ``H5T_ARRAY`` datatypes are fully supported for reading now. - The test suites for the different packages are installed now, so you don't need a copy of the PyTables sources to run the tests. Besides, you can run the test suite from the Python console by using:: >>> tables.tests() Resources ========= Go to the PyTables web site for more details: http://www.pytables.org About the HDF5 library: http://hdf.ncsa.uiuc.edu/HDF5/ About NumPy: http://numpy.scipy.org/ To know more about the company behind the development of PyTables, see: http://www.carabos.com/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Many thanks also to SourceForge who have helped to make and distribute this package! And last, but not least thanks a lot to the HDF5 and NumPy (and numarray!) makers. Without them PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Team :: Ivan Vilata i Balaguer >qo< http://www.carabos.com/ Cárabos Coop. V. V V Enjoy Data ""

1 0

Re: [Numpy-discussion] matlab vs. python question
by Nadav Horesh 26 Apr '07

26 Apr '07

Beside proper programing paradigm Python easily scales to large-scale number crunching: You can run large-matrices calculations with about 1/2 to 1/4 of memory consumption comparing to Matlab. It is not difficult to construct a program that run over several computers (independent of the hardware and OS they have). If you are willing to invest a little time in software integration you'll have an access to powerful computing packages such as VTK, OpenCV, etc. Nadav. -----Original Message----- From: numpy-discussion-bounces(a)scipy.org on behalf of Christopher Barker Sent: Wed 25-Apr-07 20:24 To: Discussion of Numerical Python Cc: Subject: Re: [Numpy-discussion] matlab vs. python question Neal Becker wrote: > I'm interested in this comparison There have got to be comparison's on the web -- google away! My few comments: > I happened to look on the matlab vendor's > website, and found that it does have classes. Matlab added classes in a fairly recent version, so technically, yes, it does support OO. However, OO aside, Python is, in many ways, are far more sophisticated and capable language. It is better suited to larger projects, and well suited to wide variety of software development, rather than just numerical work. Indeed, python+numpy supports more sophisticated numerical work too (more data types, etc). So, my reasons for using python+numpy (I did my entire dissertation with Matlab -- I loved it at the time): * Better, more flexible language. * I can use it for other things I do: web programming, sophisticated GUIs, etc. * It integrates well with lots of other libraries * It's free: both $$ and libre What's better about Matlab: * A wider selection of out-of-the-box numerical routines. * Excellent integration of command-line and plotting. In short, $$ and freedom aside, I think Matlab provides a slightly more productive environment for interactive experimentation and quickie prototypes and scripts, but much less productive for larger projects, or for people that need to do non-numerical work too. just my $0.2 -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov _______________________________________________ Numpy-discussion mailing list Numpy-discussion(a)scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion

10 14

numexpr efficency depends on the size of the computing kernel
by Francesc Altet 26 Apr '07

26 Apr '07

Hi, Now that I'm commanding my old AMD Duron machine, I've made some benchmarks just to prove that the numexpr computing is not influenced by the size of the CPU cache, but I failed miserably (and Tim was right: there is a dependency of the numexpr efficency on CPU cache size). Provided that the pytables instance of the computing kernel of numexpr is quite larger (it supports more datatypes) than the original, comparing the performance of both versions can be a good way to check the influence of CPU cache on the computing efficency. The attached benchmark is a small modification of the timing.py that comes with the numexpr package (the modification was needed to allow the numexpr version of pytables to run all the cases). Basically, the expressions tested operations with arrays of 1 million of elements, with a mix of contiguous and strided arrays (no unaligned arrays are present here). See the code in benchmark for the details. The speed-ups of numexpr over plain numpy on a AMD Duron machine (64 + 64 KB L1 cache, 64 KB L2 cache) are: For the original numexpr package: 2.14, 2.21, 2.21 (these represent averages for 3 complete runs) For the modified pytables version (enlarged computing kernel): 1.32, 1.34, 1.37 So, with a CPU with a very small cache, the original numexpr kernel is 1.6x faster than the pytables one. However, using a AMD Opteron which has a much bigger L2 cache (64 + 64 KB L1 cache, 1 MB L2 cache), the speed-ups are quite similar: For the original numexpr package: 3.10, 3.35, 3.35 For the modified pytables version (enlarged computing kernel): 3.37, 3.50, 3.45 So, there is effectively a dependency on the CPU cache size. It would be nice to run the benchmark with other CPUs with a L2 cache in the range between 64 KB and 1 MB so as to find the point where the performance starts to be similar (this should be a good guess on the size of the computing kernel). Meanwhile, the lesson learned is that Tim worries were correct: one should be very careful on adding more opcodes (at least, until CPUs with a very small L2 cache are in use). With this, perhaps we will have to reduce the opcodes in the numexpr version for pytables to a bare minimum :-/ Cheers, -- Francesc Altet | Be careful about using the following code -- Carabos Coop. V. | I've only proven that it works, www.carabos.com | I haven't tested it. -- Donald Knuth

1 1