From magnus at hetland.org Mon Mar 1 12:11:03 2004 From: magnus at hetland.org (Magnus Lie Hetland) Date: Mon Mar 1 12:11:03 2004 Subject: [Numpy-discussion] [Ann] Discretizing with Numarray Message-ID: <20040301200612.GA12281@idi.ntnu.no> I've just written a small tutorial on discretizing time series with numarray: http://hetland.org/python/disc-with-numarray Just some basic stuff I've used when working with time series in my research, such as mapping integer values to symbols, mapping real values to integers using bins, and extracting features using non-overlapping or overlapping windows (sliding window). It was written in a flash, and I'm no numarray expert, so feedback/corrections would be welcome. -- Magnus Lie Hetland "The mind is not a vessel to be filled, http://hetland.org but a fire to be lighted." [Plutarch] From falted at pytables.org Wed Mar 3 08:01:04 2004 From: falted at pytables.org (Francesc Alted) Date: Wed Mar 3 08:01:04 2004 Subject: [Numpy-discussion] [ANN] PyTables 0.8 Message-ID: <200403031654.40302.falted@pytables.org> I'm happy to announce the availability of PyTables 0.8. PyTables is a hierarchical database package designed to efficiently manage very large amounts of data. PyTables is built on top of the HDF5 library and the numarray package. It features an object-oriented interface that, combined with natural naming and C-code generated from Pyrex sources, makes it a fast, yet extremely easy-to-use tool for interactively saving and retrieving very large amounts of data. It also provides flexible indexed access on disk to anywhere in the data. PyTables is not designed to work as a relational database competitor, but rather as a teammate. If you want to work with large datasets of multidimensional data (for example, for multidimensional analysis), or just provide a categorized structure for some portions of your cluttered RDBS, then give PyTables a try. It works well for storing data from data acquisition systems (DAS), simulation software, network data monitoring systems (for example, traffic measurements of IP packets on routers), working with very large XML files or as a centralized repository for system logs, to name only a few possible uses. In this release you will find: - Variable Length Arrays (VLA's) for saving a collection of variable length of elements in each row of an array. - Extensible Arrays (EA's) for extending homogeneous datasets on disk. - Powerful replication capabilities, ranging from single leaves up to complete hierarchies. - With the introduction of the UnImplemented class, greatly improved HDF5 native file import capabilities. - Two new useful utilities: ptdump & ptrepack. - Improved documentation (with the help of Scott Prater). - New record on data size achieved: 5.6 TB (!) in one single file. - Enhanced platform support. New platforms: MacOSX, FreeBSD, Linux64, IRIX64 (yes, a clean 64-bit port is there) and probably more. - More tests units (now exceeding 800). - Many other minor improvements. More in detail: What's new ----------- - The new VLArray class enables you to store large lists of rows containing variable numbers of elements. The elements can be scalars or fully multimensional objects, in the PyTables tradition. This class supports two special objects as rows: Unicode strings (UTF-8 codification is used internally) and generic Python objects (through the use of cPickle). - The new EArray class allows you to enlarge already existing multidimensional homogeneous data objects. Consider it an extension of the already existing Array class, but with more functionality. Online compression or other filters can be applied to EArray instances, for example. Another nice feature of EA's is their support for fully multidimensional data selection with extended slices. You can write "earray[1,2:3,...,4:200]", for example, to get the desired dataset slice from the disk. This is implemented using the powerful selection capabilities of the HDF5 library, which results in very highly efficient I/O operations. The same functionality has been added to Array objects as well. - New UnImplemented class. If a dataset contains unsupported datatypes, it will be associated with an UnImplemented instance, then inserted into to the object tree as usual. This allows you to continue to work with supported objects while retaining access to attributes of unsupported datasets. This has changed from previous versions, where a RuntimeError occurred when an unsupported object was encountered. The combination of the new UnImplemented class with the support for new datatypes will enable PyTables to greatly increase the number of types of native HDF5 files that can be read and modified. - Boolean support has been added for all the Leaf objects. - The Table class has now an append() method that allows you to save large buffers of data in one go (i.e. bypassing the Row accessor). This can greatly improve data gathering speed. - The standard HDF5 shuffle filter (to further enhance the compression level) is supported. - The standard HDF5 fletcher32 checksum filter is supported. - As the supported number of filters is growing (and may be further increased in the future), a Filters() class has been introduced to handle filters more easily. In order to add support for this class, it was necessary to make a change in the createTable() method that is not backwards compatible: the "compress" and "complib" parameters are deprecated now and the "filters" parameter should be used in their place. You will be able to continue using the old parameters (only a Deprecation warning will be issued) for the next few releases, but you should migrate to the new version as soon as possible. In general, you can easily migrate old code by substituting code in its place: ???????????????table = fileh.createTable(group, 'table', Test, '', ?????????????????????????????????????????complevel, complib) ?should be replaced by ???????????????table = fileh.createTable(group, 'table', Test, '', ?????????????????????????????????????????Filters(complevel, complib)) - A copy() method that supports slicing and modification of ?????????filtering capabilities has been added for all the Leaf ?????????objects. See the User's Manual for more information. - A couple of new methods, namely copyFile() and copyChilds(), ?????????have been added to File class, to permit easy replication ?????????of complete hierarchies or sub-hierarchies, even to ?????????other files. You can change filters during the copy ?????????process as well. - Two new utilities has been added: ptdump and ?????????ptrepack. The utility ptdump allows the user to examine the?contents of PyTables files (both metadata and actual ?????????data). The powerful ptrepack utility lets you ?????????selectively copy (portions of) hierarchies to specific ?????????locations in other files. It can be also used as an ?????????importer for generic HDF5 files. ???????- The meaning of the stop parameter in read() methods has ?????????changed. Now a value of 'None' means the last row, and a ?????????value of 0 (zero) means the first row. This is more ?????????consistent with the range() function in python and the ?????????__getitem__() special method in numarray. - The method Table.removeRows() is no longer limited by table size. You can now delete rows regardless of the size of the table. - The "numarray" value has been added to the flavor parameter ?????????in the Table.read() method for completeness. - The attributes (.attr instance variable) are Python ?????????properties now. Access to their values is no longer ?????????lazy, i.e. you will be able to see both system or user ?????????attributes from the command line using the tab-completion ?????????capability of your python console (if enabled). - Documentation has been greatly improved to explain all the ?????????new functionality. In particular, the internal format of ?????????PyTables is now fully described. You can now build ?????????"native" PyTables files using any generic HDF5?software by just duplicating their format. - Many new tests have been added, not only to check new ?????????functionality but also to more stringently check ?????????existing functionality. There are more than 800 different ?????????tests now (and the number is increasing :). - PyTables has a new record in the data size that fits in one single file: more than 5 TB (yeah, more than 5000 GB), that accounts for 11 GB compressed, has been created on an AMD Opteron machine running Linux-64 (the 64 bits version of the Linux kernel). See the gory details in: http://pytables.sf.net/html/HowFast.html. - New platforms supported: PyTables has been compiled and tested under Linux32 (Intel), Linux64 (AMD Opteron and Alpha), Win32 (Intel), MacOSX (PowerPC), FreeBSD (Intel), Solaris (6, 7, 8 and 9 with UltraSparc), IRIX64 (IRIX 6.5 with R12000) and it probably works in many more architectures. In particular, release 0.8 is the first one that provides a relatively clean porting to 64-bit platforms. - As always, some bugs have been solved (especially bugs that ?????????occur when deleting and/or overwriting attributes). - And last, but definitely not least, a new donations section has been?added to the PyTables web site (http://sourceforge.net/projects/pytables, then follow the "Donations" tag). If you like PyTables and want this effort to continue, please, donate! What is a table? ---------------- A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. ?The terms "fixed-length" and "strict data types" seem to be quite a strange requirement for an language like Python that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ------------- For those people who know nothing about HDF5, it is is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms --------- I'm using Linux (Intel 32-bit) as the main development platform, but PyTables should be easy to compile/install on many other UNIX machines. This package has also passed all the tests on a UltraSparc platform with Solaris 7 and Solaris 8. It also compiles and passes all the tests on a SGI Origin2000 with MIPS R12000 processors, with the MIPSPro compiler and running IRIX 6.5. It also runs fine on Linux 64-bit platforms, like an AMD Opteron running SuSe Linux Enterprise Server. It has also been tested in MacOSX platforms (10.2 but should also work on newer versions). Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP (using the Microsoft Visual C compiler), but it should also work with other flavors as well. An example? ----------- For online code examples, have a look at http://pytables.sourceforge.net/html/tut/tutorial1-1.html and, for newly introduced Variable Length Arrays: http://pytables.sourceforge.net/html/tut/vlarray2.html Web site -------- Go to the PyTables web site for more details: http://pytables.sourceforge.net/ Share your experience --------------------- Let me know of any bugs, suggestions, gripes, kudos, etc. you may have. Have fun! -- Francesc Alted falted at pytables.org -- Francesc Alted From haase at msg.ucsf.edu Tue Mar 9 10:10:05 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Mar 9 10:10:05 2004 Subject: [Numpy-discussion] numarray: shape of scalar Message-ID: <200403091000.39148.haase@msg.ucsf.edu> Hi, Is this intended: >>> na.array([1,2,4]).shape (3,) >>> na.array([1,2]).shape (2,) >>> na.array([1]).shape (1,) >>> na.array(1).shape () Why is na.array([1]).shape not equal to na.array(1).shape ? Did I miss the respective section in the manual ? Thanks, Sebastian Haase From jmiller at stsci.edu Tue Mar 9 10:29:21 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Mar 9 10:29:21 2004 Subject: [Numpy-discussion] numarray: shape of scalar In-Reply-To: <200403091000.39148.haase@msg.ucsf.edu> References: <200403091000.39148.haase@msg.ucsf.edu> Message-ID: <1078856368.12039.10.camel@halloween.stsci.edu> On Tue, 2004-03-09 at 13:00, Sebastian Haase wrote: > Hi, > Is this intended: Yes. > >>> na.array([1,2,4]).shape > (3,) > >>> na.array([1,2]).shape > (2,) > >>> na.array([1]).shape > (1,) > >>> na.array(1).shape > () > > Why is na.array([1]).shape not equal to na.array(1).shape ? na.array(1) is how you say "make a rank-0 array with contents 1". There are places where it is natural and useful in the *implementation* of numarray. > Did I miss the respective section in the manual ? rank-0 arrays are a dark corner of numarray which is mostly hidden. I don't think they're documented anywhere. Regards, Todd -- Todd Miller From a.schmolck at gmx.net Wed Mar 10 10:23:38 2004 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Wed Mar 10 10:23:38 2004 Subject: [Numpy-discussion] numarray: shape of scalar In-Reply-To: <200403091000.39148.haase@msg.ucsf.edu> References: <200403091000.39148.haase@msg.ucsf.edu> Message-ID: Sebastian Haase writes: > Hi, > Is this intended: > >>> na.array([1,2,4]).shape > (3,) > >>> na.array([1,2]).shape > (2,) > >>> na.array([1]).shape > (1,) > >>> na.array(1).shape > () > > Why is na.array([1]).shape not equal to na.array(1).shape ? It is maybe a bit like n^0 == 1 (rather than 0, which might seem "more intuitive"). A scalar is not the same as vector of 1 element and neither are the same as a matrix of 1 element (nonwithstanding what matlab and maybe some dumb math texts might suggest). If you pretend they are you run into all sorts of trouble, as matlab (which doesn't distinguish properly between scalars, vectors and matrices) indeed does, for one thing because various useful identities suddenly no longer hold for corner cases (e.g. since functions like diag that do different things depending on whether they "think" something is a vector or not just do the wrong thing on what you intend to be a 1xN or Nx1 matrix). Here is a not particularly interesting identity that shows why you would like shape(scalar) to be (): rank == len(shape) rank([x]) == rank(x) + 1 Here is a more interesting example that shows why you want to distinguish between arrays of different rank (even if their shape only differes by a leading 1 and the flat is the same): rank(a[integer]) = rank(a) - 1 The pleasant indexing facilities of numarray wouldn't be possible without distinguishing between arrays which differ only by the amount of leading 1s in their shape. 'as From berumthyer at vietmedia.com Thu Mar 11 01:31:03 2004 From: berumthyer at vietmedia.com (Derrick Meyer) Date: Thu Mar 11 01:31:03 2004 Subject: [Numpy-discussion] STOCK-PROFILE - Breaking News on ACQUISITION, Wed, 10 Mar 2004 20:04:22 -0500 Message-ID: Wed, 10 Mar 2004 20:04:22 -0500 Stock-Profile - NEW ISSUE: SEVI - Systems Evolution Incorporated Breaking NEWS: Systems Evolution Acquires AXP Technologies STAFFORD, Texas -- Systems Evolution, Inc., (OTC BB:SEVI.OB - News), an information technology services company, announces that it has acquired AXP Technologies, Inc., a Houston based computer and network support company. This acquisition accelerates Systems Evolution's move into network security and computer support services for small to mid-size businesses (SMBs). The SMB Solutions Practice will be led by AXP Technologies founders, Willie A. Jackson and Ryan L. Sumstad as Director and Managing Consultant, respectively. AXP Technolgies designed and implemented computer systems and network security throughout organizations, maintaining and supporting them at a fraction of the traditional costs. Disclosure: Market-Torque (M-T) is not a registered financial advisory. The information presented by M-T is not for purchasing or selling securities. M-T compiles then distributes opinions, comments and information based on other public sources. Penny stocks are considered to be highly speculative and may be unsuitable for all but very aggressive investors. M-T does not hold nor does it plan to hold a position in this stock. This Profile was a paid advertisement by a third party not affiliated with the profiled company. M-T was compensated four thousand dollars to publish and distribute this report. Paid advertisements for a third party do not necessarily reflect the views of M-T. Target prices may fluctuate depending on market environments. Please always consult a registered financial advisor before making any decisions. This report is for entertainment and advertising purposes only and should not be used as investment advice. Copyright 2004, Roytico Ltd. This report distributed by MMS. Apartado 173-3006 Zona Franca MeoBarreal Heredia, Costa Rica. No more ads, cathisterba at freemail.lt From ariciputi at pito.com Thu Mar 11 04:18:13 2004 From: ariciputi at pito.com (Andrea Riciputi) Date: Thu Mar 11 04:18:13 2004 Subject: [Numpy-discussion] A simple question about type(). Message-ID: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> Hi, it is surely a silly question, but I'm quite new to Python and Numpy. So I hope you can help me, I've searched Numpy's manual without successs. Suppose I want to test if a paremeter inside a function is an array or not. How can I test this? type(my_array) is array returns 'False'!! Thanks in advance, Andrea. --- Andrea Riciputi "Science is like sex: sometimes something useful comes out, but that is not the reason we are doing it" -- (Richard Feynman) From rsilva at ime.usp.br Thu Mar 11 04:58:38 2004 From: rsilva at ime.usp.br (Paulo J. S. Silva) Date: Thu Mar 11 04:58:38 2004 Subject: [Numpy-discussion] A simple question about type(). In-Reply-To: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> References: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> Message-ID: <1079008947.11736.8.camel@localhost> Hi, Funny, I have just entered the list and I can answer this one! Try type(my_array) is arraytype That worked to me. Best, Paulo -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call) (practice) From ariciputi at pito.com Thu Mar 11 05:37:10 2004 From: ariciputi at pito.com (Andrea Riciputi) Date: Thu Mar 11 05:37:10 2004 Subject: [Numpy-discussion] A simple question about type(). In-Reply-To: <1079008947.11736.8.camel@localhost> References: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> <1079008947.11736.8.camel@localhost> Message-ID: <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> You are right! But I can't understand why. When I try type() with an array I get: type(my_array) but: type(my_array) is array False Can someone explain this? Cheers, Andrea. On 11 Mar 2004, at 13:42, Paulo J. S. Silva wrote: > Hi, > > Funny, I have just entered the list and I can answer this one! > > Try > > type(my_array) is arraytype > > That worked to me. > > Best, > > Paulo > -- > Paulo Jos? da Silva e Silva > Professor Assistente do Dep. de Ci?ncia da Computa??o > (Assistant Professor of the Computer Science Dept.) > Universidade de S?o Paulo - Brazil > > e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva > > Teoria ? o que n?o entendemos o (Theory is something we don't) > suficiente para chamar de pr?tica. (understand well enough to call) > (practice) > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > --- Andrea Riciputi "Science is like sex: sometimes something useful comes out, but that is not the reason we are doing it" -- (Richard Feynman) From Alexandre.Fayolle at logilab.fr Thu Mar 11 05:55:05 2004 From: Alexandre.Fayolle at logilab.fr (Alexandre) Date: Thu Mar 11 05:55:05 2004 Subject: [Numpy-discussion] A simple question about type(). In-Reply-To: <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> References: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> <1079008947.11736.8.camel@localhost> <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> Message-ID: <20040311134339.GA10008@crater.logilab.fr> On Thu, Mar 11, 2004 at 02:25:53PM +0100, Andrea Riciputi wrote: > You are right! But I can't understand why. When I try type() with an > array I get: > > type(my_array) > > > but: > > type(my_array) is array > False > > Can someone explain this? array is a function. try "type(array)", and print(array) to check. -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From rsilva at ime.usp.br Thu Mar 11 06:19:10 2004 From: rsilva at ime.usp.br (Paulo J. S. Silva) Date: Thu Mar 11 06:19:10 2004 Subject: [Numpy-discussion] A simple question about type(). In-Reply-To: <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> References: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> <1079008947.11736.8.camel@localhost> <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> Message-ID: <1079014074.11736.26.camel@localhost> This is a more tricky question. If I understand well enough. The name "array" in a python section relates to a factory function that builds arraytype objects. It is not the name for the array type. For example, if you type: type(array) you get Which shows that it is a function, not a class object. There are more subtlety associated to the fact that Numeric does not use the new class styles as numarray does. Actually if you use numarray instead of Numeric you'll get a more sensible behavior >>> import numarray >>> a = numarray.array([1.,2.,3.]) >>> type(a) >>> type(a) is numarray.NumArray True >>> isinstance(a, numarray.NumArray) True >>> Best regards, Paulo Obs: You may consider using a "try: except:" statement instead of always checking type. In Python is usually better to ask "forgiveness" in a "except" clause than to ask "permission" by checking types. -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call) (practice) From ferrell at lanl.gov Thu Mar 11 07:23:09 2004 From: ferrell at lanl.gov (ferrell at lanl.gov) Date: Thu Mar 11 07:23:09 2004 Subject: [Numpy-discussion] A simple question about type(). In-Reply-To: <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> References: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> <1079008947.11736.8.camel@localhost> <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> Message-ID: <16464.33169.464000.954991@gargle.gargle.HOWL> As others have pointed out, array is a function. You want ArrayType. > type(my_array) is ArrayType > True -r Andrea Riciputi writes: > You are right! But I can't understand why. When I try type() with an=20 > array I get: > > type(my_array) > > > but: > > type(my_array) is array > False > > Can someone explain this? > > Cheers, > Andrea. > > On 11 Mar 2004, at 13:42, Paulo J. S. Silva wrote: > > > Hi, > > > > Funny, I have just entered the list and I can answer this one! > > > > Try > > > > type(my_array) is arraytype > > > > That worked to me. > > > > Best, > > > > Paulo > > --=20 > > Paulo Jos=E9 da Silva e Silva > > Professor Assistente do Dep. de Ci=EAncia da Computa=E7=E3o > > (Assistant Professor of the Computer Science Dept.) > > Universidade de S=E3o Paulo - Brazil > > > > e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva > > > > Teoria =E9 o que n=E3o entendemos o (Theory is something we don't) > > suficiente para chamar de pr=E1tica. (understand well enough to call) > > (practice) > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by: IBM Linux Tutorials > > Free Linux tutorial presented by Daniel Robbins, President and CEO of > > GenToo technologies. Learn everything from fundamentals to system > > administration.http://ads.osdn.com/?ad_id=1470&alloc_id638&op=3Dclick > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > --- > Andrea Riciputi > > "Science is like sex: sometimes something useful comes out, > but that is not the reason we are doing it" -- (Richard Feynman) > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From jmiller at stsci.edu Sat Mar 13 05:12:36 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Mar 13 05:12:36 2004 Subject: [Numpy-discussion] ANN: numarray-0.9 Message-ID: <1079179547.3739.0.camel@localhost.localdomain> Release Notes for numarray-0.9 Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, and improved type promotions. I. ENHANCEMENTS 1. Support for "from __future__ import division" True division has been implemented for numarray. This means that modules that wish to use true division can also use numarray and numarray division will work as follows: a. dividing any two integer arrays using "/" results in a Float32 array. b. dividing two floating point arrays using "//" results in truncation of the result as in: a // b == floor(a/b). 2. C-coded array slicing Array slicing has been re-implemented in C-code as part of the _ndarray module. This means faster slicing. Thanks go to Warren Hack, Chris Hanley, and Ivo Busko for helping debug a huge refcount error. 3. Decreased Ufunc overhead Ufunc execution speed has clawed and scratched its way back to where it was around numarray-0.5. Improvements here included optimization of the ufunc caching, smarter thread handling, and smarter support for subclasses. The ufunc caching is based on a simple 20 element table for each ufunc. 4. Faster array creation from C Code which creates NumArrays from C (including numarray itself) can now do so faster because the API functions have been modified to do the array __init__ inline rather than through an expensive Python callback. II. BUGS FIXED / CLOSED See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. 913781 Another memory leak in example in Chapter 12 908399 Numarray 0.7: "del a[1]" dumps core 899259 astype Int16 copy4bytes: access beyond buffer 895801 Buffer overflow in sum w/ 0-sized array 894810 MemoryError When Creating Large Arrays 890703 getnan() and getinf() failure 883124 and and operator.and respond differently 865410 Usage of __dict__ 854480 Slice assignment of float to integer 839367 Overlapping slice assign fails 828941 Numarray: determinant returns scalar or array 820122 Linearalgebra2.determinant problem 817343 Sub-classing of NumArray inhibited by complex values 793336 crash in _sort.pyd 772548 Reference counting errors 683957 Adding certain arrays fails in Numarray III. CAUTIONS 1. numarray extension writers should note that the documented use of PyArray_INCREF and PyArray_XDECREF (in numarray) has been found to be incompatible with Numeric and has therefore been deprecated. numarray wrapper functions using PyArray_INCREF and PyArray_XDECREF should switch to ordinary Py_INCREF and Py_XDECREF. WHERE ----------- Numarray-0.9 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://stsdas.stsci.edu/numarray/index.html Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-0.9 requires Python 2.2.2 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen K?pper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller at stsci.edu From hoagiaonguyen at yahoo.com Sat Mar 13 10:55:17 2004 From: hoagiaonguyen at yahoo.com (Hoa Nguyen) Date: Sat Mar 13 10:55:17 2004 Subject: [Numpy-discussion] Installing Numpy Message-ID: <20040313185420.84768.qmail@web40712.mail.yahoo.com> Hi, Please help me with installing NumPy. When I run: python setup.py install there was error. It said that: ImportError: No module named distutils I'm using Mandrake 9. Thanks, Hoa Nguyen __________________________________ Do you Yahoo!? Yahoo! Mail - More reliable, more storage, less spam http://mail.yahoo.com From ivanov at rbc.ru Sat Mar 13 17:15:01 2004 From: ivanov at rbc.ru (ivanov at rbc.ru) Date: Sat Mar 13 17:15:01 2004 Subject: [Numpy-discussion] www.cvv2.ru - would like to thank you for your purchase on our web site Message-ID: You've just purchased set of Maibach brand earthenware on web site cvv2.ru Easy to use, Maibach kitchenware is also famous for its modern look. Our utensils, designed for easy and fast cooking of a variety of foods, will lower your energy consumption rate and save your time and money. If you are looking for a reliable and trouble-free kitchen help, you have just found it. All Maibach kitchenware is made of stainless steel - a proven chrome and nickel 18/10 alloy. Easy to use and famous for its modern look, Maibach kitchenware allows you to simultaneously cook several dishes on one burner. A heat-absorbing base, built for optimal transmission of thermal energy from the burner to the cookware, is the most important feature of Maibach utensils. The exclusive Platinum model uses a triple heat-absorbing base, guaranteeing a still better protection of food from burning. Even and smooth heating of the utensil's base and sides prevents food from burning. NEW!!! We are glad to offer to our customers the most generous reward program: every customer who spent with us more than USD 249.99 or Euro 199.99, automatically qualified for bonus. Choose your bonus: 1. Sony VHS cassette with 240 minutes of best underage porno you ever see. (NTSC and Secam both are available) 2. Bestselling manual "How to create plastic bomb in home" and "How to hijack a train or an aircraft, with color pictures and FAQ" To cancell order:val at rbc.ru Bulk orders:alexk1 at rbc.ru Refund Service ICQ#307004 All Maibach products are covered by lifetime warranty. From faheem at email.unc.edu Sat Mar 13 22:21:02 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sat Mar 13 22:21:02 2004 Subject: [Numpy-discussion] numarray.objects missing in documentation Message-ID: Dear People, I'm posting this to gmane.comp.python.numeric.general. I am not sure what mailing list this corresponds to. I earlier tried to post this to the mailing list mentioned in the numarray documentation (numpy at lists.sourceforge.net), but it bounced. I don't see numarray.objects listed in http://stsdas.stsci.edu/numarray/numarray-0.9.html/genindex.html I can't find it anywhere else in the documentation either. I could not find a single page html version of the docuemtation, otherwise I would have done a search. I do find nice documentation when doing >>> help("numarray.objects"). However there is a minor typo at the beginning. "...It defines functions which correspond to most of the most of the operators defined in Python's operator module, and provides names compatible with most of numarray's universal functions." "most of" is repeated. Please cc me on any reply; I'm not subscribed. Thanks. Faheem. From nadavh at visionsense.com Sun Mar 14 06:05:01 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Sun Mar 14 06:05:01 2004 Subject: [Numpy-discussion] Installing Numpy Message-ID: <07C6A61102C94148B8104D42DE95F7E86DECB2@exchange2k.envision.co.il> Try to get distutils either from rpmfind or your distribution CD (downloading and installing the latest python is a good option). It is highly recommended to have distutils if you want to install any add-on package to python. Nadav. -----Original Message----- From: Hoa Nguyen [mailto:hoagiaonguyen at yahoo.com] Sent: Sat 13-Mar-04 20:54 To: numpy-discussion at lists.sourceforge.net Cc: Subject: [Numpy-discussion] Installing Numpy Hi, Please help me with installing NumPy. When I run: python setup.py install there was error. It said that: ImportError: No module named distutils I'm using Mandrake 9. Thanks, Hoa Nguyen __________________________________ Do you Yahoo!? Yahoo! Mail - More reliable, more storage, less spam http://mail.yahoo.com ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From jmiller at stsci.edu Sun Mar 14 06:57:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Mar 14 06:57:01 2004 Subject: [Numpy-discussion] numarray.objects missing in documentation In-Reply-To: References: Message-ID: <1079276152.13290.9.camel@localhost.localdomain> On Sun, 2004-03-14 at 00:26, Faheem Mitha wrote: > Dear People, > > I'm posting this to gmane.comp.python.numeric.general. I am not sure > what mailing list this corresponds to. I earlier tried to post this to > the mailing list mentioned in the numarray documentation > (numpy at lists.sourceforge.net), but it bounced. > > I don't see numarray.objects listed in > http://stsdas.stsci.edu/numarray/numarray-0.9.html/genindex.html The section on object arrays has yet to be written. What is it that you want to know? > > I can't find it anywhere else in the documentation either. I could not > find a single page html version of the docuemtation, otherwise I would > have done a search. > > I do find nice documentation when doing > >>> help("numarray.objects"). > > However there is a minor typo at the beginning. > > "...It defines functions which correspond to most of the most of the > operators defined in Python's operator module, and provides names > compatible with most of numarray's universal functions." > > "most of" is repeated. Thanks. This is fixed now in CVS. > > Please cc me on any reply; I'm not subscribed. Thanks. > > Faheem. Regards, Todd -- Todd Miller From faheem at email.unc.edu Sun Mar 14 09:21:02 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sun Mar 14 09:21:02 2004 Subject: [Numpy-discussion] Re: numarray.objects missing in documentation References: <1079276152.13290.9.camel@localhost.localdomain> Message-ID: On Sun, 14 Mar 2004 09:55:52 -0500, Todd Miller wrote: > On Sun, 2004-03-14 at 00:26, Faheem Mitha wrote: >> Dear People, >> >> I'm posting this to gmane.comp.python.numeric.general. I am not sure >> what mailing list this corresponds to. I earlier tried to post this to >> the mailing list mentioned in the numarray documentation >> (numpy at lists.sourceforge.net), but it bounced. >> >> I don't see numarray.objects listed in >> http://stsdas.stsci.edu/numarray/numarray-0.9.html/genindex.html > > The section on object arrays has yet to be written. What is it that you > want to know? Thanks for the quick reply. I'm trying to make an array of python objects as follows. Specifically, given a user defined class, I want to creata an array of instances of that class. Consider the following class. class cell: def setrow(self,row): self.row = row def setcol(self,col): self.col = col ... I create an "empty" cell. empcell = cell() Now I want to create an array of objects identical to empcell, and later initialize them. I tried stuff along the lines of ****************************************************************** import string from numarray import * empcell = cell() base = [empcell] mat = fromlist(base,shape=(2,2)) ****************************************************************** But the mat = fromlist(base,shape=(2,2)) line fails with AttributeError: cell instance has no attribute '__len__' Similar things fail with the same error message. No doubt I'm doing something stupid, but would someone please enlighten me? I got the fromlist command from >>> help("numarray.objects") I was a bit disconcerted to find on trying to run the first example from the help page that I got >>> a = fromlist(["t","u","w"]) ... TypeError: Expecting a python numeric type, got something else. What am I doing wrong? I'm running ii python 2.3.3-5 An interactive high-level object-oriented language (default versio ii python-numarray 0.8.1-1 An array processing package modelled after Python-Numeric ii python-numeric 23.1-1 Numerical (matrix-oriented) Mathematics for Python on Debian sarge. Thanks in advance and sorry for the long-winded message. Faheem. From sdhyok at email.unc.edu Sun Mar 14 19:51:01 2004 From: sdhyok at email.unc.edu (Shin) Date: Sun Mar 14 19:51:01 2004 Subject: [Numpy-discussion] map for numarray and automatic calls of elements? Message-ID: <1079322556.2570.28.camel@dhcp14957.odumvillage.unc.edu> In my experience, when using Numeric (numpy), the following two features turned out to be very useful in many numerical operations. Please let me know if numarray already supports the features. Any comments are welcome. 1. A function like map but it returns an array directly, not a list. For instance, if we call the function as xmap, >> f = lambda x: x+1 >> map(f, [1,2,3]) [2,3,4] >> xmap(f, [1,2,3]) # == array(map(f,[1,2,3])) array([2,3,4]) When dealing with a large volume of arrays, this function can save memory and improve speed significantly by skipping the creation of a list. 2. A way to call methods or attributes of object elements from an object array. For instance, >> from datetime import date >> x = array([date(1990,1,1), date(2000,1,1)]) >> x.year array([1990, 2000]) If an object array is called by a methods or an attribute, which the array doesn't have, the array calls the methods or the attributes of its elements automatically and returns a collection of the results. Thanks. -- Daehyok Shin Geography Department University of North Carolina, Chapel Hill email: sdhyok at email.unc.edu From falted at pytables.org Mon Mar 15 05:24:04 2004 From: falted at pytables.org (Francesc Alted) Date: Mon Mar 15 05:24:04 2004 Subject: [Numpy-discussion] PyTables & numarray 0.9 warning Message-ID: <200403151418.41664.falted@pytables.org> Hi, Many of you will know that a new version of numarray (0.9) has been released past week. This new version has a number of cool features (specially, being faster in certain situations that affect directly to PyTables efficency :-). Unfortunately, the new version of numarray has deprecated the "buffer" keyword on the array() constructor, and that precise keyword was used in PyTables (just in one line). I've uploaded a new version of PyTables in the SourceForge repository with a cure on that. Please, if you have downloaded PyTables *before* March, 9th, download again from the PyTables web site (http://www.pytables.org) and rebuild the software (or install the new autoinstallable binaries for Windows). If you don't feel like having to do that, you can just apply this patch to get rid of the problem: -8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<- --- tables/Array.py.orig 2004-03-15 13:56:01.000000000 +0100 +++ tables/Array.py 2004-03-15 13:56:13.000000000 +0100 @@ -483,7 +483,7 @@ if repr(self.type) == "CharType": arr = strings.array(None, itemsize=self.itemsize, shape=shape) else: - arr = numarray.array(buffer=None, type=self.type, shape=shape) + arr = numarray.array(None, type=self.type, shape=shape) # Set the same byteorder than on-disk arr._byteorder = self.byteorder # Protection against reading empty arrays -8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<- My apologies for the inconveniences, -- Francesc Alted From jmiller at stsci.edu Mon Mar 15 07:53:09 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Mar 15 07:53:09 2004 Subject: [Numpy-discussion] Re: numarray.objects missing in documentation References: <1079276152.13290.9.camel@localhost.localdomain> Message-ID: <006d01c40aa5$8591e830$ebeca782@stsci.edu> Hi Faheem, In order to create an object array, do this: >>> import numarary.objects as obj >>> a = obj.fromlist(["t","u","w",1]) >>> a ObjectArray(["t","u","w",1) Unfortunately, the ability to apply arbitrary functions to all the elements of an object array has never been implemented. Nevertheless, the following might do: def apply(array, function, result=None): rank1 = (array.rank == 1) if result is None: result0 = array.copy() else: result0 = result for x in range(array.shape[0]): if rank1: result0[x] = function(array[x]) else: apply(array[x], function, result0[x]) return result0 Re-ordering the loop and the if statement results in faster but slightly more verbose code. Regards, Todd Miller ----- Original Message ----- From: "Faheem Mitha" To: Sent: Sunday, March 14, 2004 12:20 PM Subject: [Numpy-discussion] Re: numarray.objects missing in documentation > On Sun, 14 Mar 2004 09:55:52 -0500, Todd Miller wrote: > > On Sun, 2004-03-14 at 00:26, Faheem Mitha wrote: > >> Dear People, > >> > >> I'm posting this to gmane.comp.python.numeric.general. I am not sure > >> what mailing list this corresponds to. I earlier tried to post this to > >> the mailing list mentioned in the numarray documentation > >> (numpy at lists.sourceforge.net), but it bounced. > >> > >> I don't see numarray.objects listed in > >> http://stsdas.stsci.edu/numarray/numarray-0.9.html/genindex.html > > > > The section on object arrays has yet to be written. What is it that you > > want to know? > > Thanks for the quick reply. I'm trying to make an array of python > objects as follows. Specifically, given a user defined class, I want > to creata an array of instances of that class. > > Consider the following class. > > class cell: > def setrow(self,row): > self.row = row > def setcol(self,col): > self.col = col > ... > > I create an "empty" cell. > > empcell = cell() > > Now I want to create an array of objects identical to empcell, and > later initialize them. > > I tried stuff along the lines of > > ****************************************************************** > import string > from numarray import * > > empcell = cell() > > base = [empcell] > > mat = fromlist(base,shape=(2,2)) > ****************************************************************** > > But the > > mat = fromlist(base,shape=(2,2)) > > line fails with > > AttributeError: cell instance has no attribute '__len__' > > Similar things fail with the same error message. No doubt I'm doing > something stupid, but would someone please enlighten me? > > I got the fromlist command from > > >>> help("numarray.objects") > > I was a bit disconcerted to find on trying to run the first example > from the help page that I got > > >>> a = fromlist(["t","u","w"]) > ... > TypeError: Expecting a python numeric type, got something else. > > What am I doing wrong? > > I'm running > > ii python 2.3.3-5 An interactive > high-level object-oriented language (default versio > ii python-numarray 0.8.1-1 An array > processing package modelled after Python-Numeric > ii python-numeric 23.1-1 Numerical > (matrix-oriented) Mathematics for Python > > on Debian sarge. > > Thanks in advance and sorry for the long-winded message. > > Faheem. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From vineet at eswap.com Tue Mar 16 03:00:07 2004 From: vineet at eswap.com (Vineet Jain) Date: Tue Mar 16 03:00:07 2004 Subject: [Numpy-discussion] Numeric to Numarray migration Message-ID: I'm looking to port my application to Numarray and would like some help: 1. How do I do the following in numarray (since from numarray import PyObject fails): self.startDateTimeList = zeros(n, PyObject) 2. I have a time series array which has invalid elements. I use the following two functions: To get my compressed array (without the invalid elements) I use: self.high = extract(prices1.high, prices1.mask) I then do some computations on the high in an external c library which populates a new array in the compressed format. Then I use: insert(self.outIndicator,self.prices1.mask,self._outIndicator) This function insert from the scipy_base package was a lot faster than my python equivalent. But since it is called a lot of times the insert operation is takes a siginificant amount of time. 608400 195.333 0.000 195.333 0.000 C:\Python23\lib\site-packages\scipy_base\function_base.py:281(insert) It represents about 25% of the application overall time. How do I do the same in numarray and will the performance be better than the above implementation. My arrays are usually greater than 20k in length. Thanks for Numeric, I love the package and it is helped me immensely. Vineet From anewgene at hotpop.com Tue Mar 16 07:42:02 2004 From: anewgene at hotpop.com (CL WU) Date: Tue Mar 16 07:42:02 2004 Subject: [Numpy-discussion] corrcoeff on big matrix Message-ID: <40572011.80409@hotpop.com> Hi, group, I have a big "Float64" matrix (42x22300) and I want to get its correlation coefficient matrix, but I got the error as the following: >>> data.shape (42, 22300) >>> mlab.corrcoef(data) Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\Lib\site-packages\numarray\linear_algebra\mlab.py", line 300, in corrcoef c = cov(x, y) File "C:\Python23\Lib\site-packages\numarray\linear_algebra\mlab.py", line 294, in cov val = squeeze(dot(transpose(m),conjugate(y)) / fact) File "C:\Python23\Lib\site-packages\numarray\numarraycore.py", line 1150, in dot return ufunc.innerproduct(array1, _gen.swapaxes(array2, -1, -2)) File "C:\Python23\Lib\site-packages\numarray\ufunc.py", line 2047, in innerproduct r = a.__class__(shape=adots+bdots, type=rtype) ValueError: new_memory: invalid region size: -633294592. I suspect corrcoef function can not handle such a big matrix. If so, what is the upper limit for array size? How can I get around this problem in numarray? BTW, I am using numarray 0.9/python 2.3.3 on win2kSP4 Thanks. Chunlei From jmiller at stsci.edu Tue Mar 16 13:12:46 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Mar 16 13:12:46 2004 Subject: [Numpy-discussion] Numeric to Numarray migration In-Reply-To: References: Message-ID: <1079467364.3735.370.camel@localhost.localdomain> On Tue, 2004-03-16 at 05:59, Vineet Jain wrote: > I'm looking to port my application to Numarray and would like some help: > > 1. How do I do the following in numarray (since from numarray import > PyObject fails): > > self.startDateTimeList = zeros(n, PyObject) >>> import numarray.objects as obj >>> self.startDateTimeList = obj.fromlist([0]*n) Note that numarray.objects is an independent implementation of object arrays which is just now coming into wider use. I expect that there will be gaps in functionality relative to Numeric so please ask if you have questions. The docstrings are currently the only documentation for numarray's object arrays. > > 2. I have a time series array which has invalid elements. I use the > following two functions: > Maybe someone else with more intuition about what you're trying to do can answer this. If not, get a copy of the numarray manual here: http://prdownloads.sourceforge.net/numpy/numarray-0.9.pdf?download I'd look at the functions compress(), putmask(), put(), take(), and nonzero(). Array indexing may also come in handy. > To get my compressed array (without the invalid elements) I use: > > self.high = extract(prices1.high, prices1.mask) > > I then do some computations on the high in an external c library which > populates a new array > in the compressed format. Then I use: > > insert(self.outIndicator,self.prices1.mask,self._outIndicator) > > This function insert from the scipy_base package was a lot faster than my > python equivalent. But since it > is called a lot of times the insert operation is takes a siginificant amount > of time. > > 608400 195.333 0.000 195.333 0.000 > C:\Python23\lib\site-packages\scipy_base\function_base.py:281(insert) If you want to use numarray and don't get the speed you want with Python code, you might try porting the function from SciPy to numarray using the Numeric compatibility layer. > > It represents about 25% of the application overall time. How do I do the > same in numarray and will the performance > be better than the above implementation. My arrays are usually greater than > 20k in length. Since your array sizes are large, numarray should be fairly competitive and depending on how the actual code works out, might even be faster. My guess though, is that numarray will be a little slower than Numeric for the array sizes you're talking about and the functions you're likely to use. Regards, Todd > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From jmiller at stsci.edu Tue Mar 16 14:29:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Mar 16 14:29:03 2004 Subject: [Numpy-discussion] corrcoeff on big matrix In-Reply-To: <40572011.80409@hotpop.com> References: <40572011.80409@hotpop.com> Message-ID: <1079470363.5469.410.camel@localhost.localdomain> On Tue, 2004-03-16 at 10:41, CL WU wrote: > Hi, group, > I have a big "Float64" matrix (42x22300) and I want to get > its correlation coefficient matrix, but I got the error as the following: > > >>> data.shape > (42, 22300) > >>> mlab.corrcoef(data) > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\Lib\site-packages\numarray\linear_algebra\mlab.py", > line 300, in corrcoef > c = cov(x, y) > File "C:\Python23\Lib\site-packages\numarray\linear_algebra\mlab.py", > line 294, in cov > val = squeeze(dot(transpose(m),conjugate(y)) / fact) > File "C:\Python23\Lib\site-packages\numarray\numarraycore.py", line > 1150, in dot > return ufunc.innerproduct(array1, _gen.swapaxes(array2, -1, -2)) > File "C:\Python23\Lib\site-packages\numarray\ufunc.py", line 2047, in > innerproduct > r = a.__class__(shape=adots+bdots, type=rtype) > ValueError: new_memory: invalid region size: -633294592. > > I suspect corrcoef function can not handle such a big matrix. If so, > what is the upper limit for array size? The memory limit is appears to be driven by the numarray.memory and is 2G. Trying to run your function call winds up creating a dot product output array which is 22300**2. This is ~400M * 8 bytes per float just for the dot product output, which is 3.2G, hence the exception. I think 16384**2 is the ideal limit of what you can achieve with numarray, and in practice, think you'll get considerably less depending on how many arrays are needed at once to complete your computation. > How can I get around this > problem in numarray? One possibility is to consider using Float32 to stretch out your memory. I don't know whether that's numerically viable or not. Another way is 64-bit computing. That is largely unexplored territory, and Python itself has issues there. It will likely take some work because we haven't done it yet ourselves. I hope this at least sheds some light on the problem, if not the actual solution. Regards, Todd > BTW, I am using numarray 0.9/python 2.3.3 on win2kSP4 > > Thanks. > > Chunlei > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From cjw at sympatico.ca Tue Mar 16 19:20:12 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Mar 16 19:20:12 2004 Subject: [Numpy-discussion] corrcoeff on big matrix In-Reply-To: <1079470363.5469.410.camel@localhost.localdomain> References: <40572011.80409@hotpop.com> <1079470363.5469.410.camel@localhost.localdomain> Message-ID: <4057C3D9.2010508@sympatico.ca> I'm puzzled that anyone would want to explore the correlation among 42 variables. I would suggest (1) having a go with the more significant of the 42 variables and (2) working with a a much smaller sample, say < 4,000. Todd Miller wrote: [snip] >On Tue, 2004-03-16 at 10:41, CL WU wrote: > > >One possibility is to consider using Float32 to stretch out your >memory. I don't know whether that's numerically viable or not. > >Another way is 64-bit computing. That is largely unexplored territory, >and Python itself has issues there. It will likely take some work >because we haven't done it yet ourselves. > > Todd, I'd be grateful if you could clarify this, do we nor have _nt.Float64 now? Colin W. >I hope this at least sheds some light on the problem, if not the actual >solution. > >Regards, >Todd > > > >>BTW, I am using numarray 0.9/python 2.3.3 on win2kSP4 >> >>Thanks. >> >>Chunlei >> >> >> >>------------------------------------------------------- >>This SF.Net email is sponsored by: IBM Linux Tutorials >>Free Linux tutorial presented by Daniel Robbins, President and CEO of >>GenToo technologies. Learn everything from fundamentals to system >>administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click >>_______________________________________________ >>Numpy-discussion mailing list >>Numpy-discussion at lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> From dykap at maktoob.com Wed Mar 17 07:13:08 2004 From: dykap at maktoob.com (Randy Whitley) Date: Wed Mar 17 07:13:08 2004 Subject: [Numpy-discussion] Tech-Ticker - 50 Day Parabolic Time/Price STRONGEST, Wed, 17 Mar 2004 10:12:00 -0500 Message-ID: Wed, 17 Mar 2004 10:12:00 -0500 Price adjustment, Tech-Ticker Alert for OSSI, OSF Healthcare Services Inc. INDICATORS 7-day Average Directional Indicator - STRONGEST 50 Day Parabolic Time/Price - STRONGEST Get Quote: http://quote.money.cnn.com/quote/quote?symbols=ossi Tech-Ticker (TT) is not a registered financial advisory. The information presented by TT is not for purchasing or selling securities. TT compiles then distributes opinions, comments and information based on other public sources. Penny stocks are considered to be highly speculative and may be unsuitable for all but very aggressive investors. TT does not hold nor does it plan to hold a position in this stock. This Profile was a paid advertisement by a third party not affiliated with the profiled company. TT was compensated three thousand dollars to publish and distribute this report. Paid advertisements for a third party do not necessarily reflect the views of TT. Target prices may fluctuate depending on market environments. Please always consult a registered financial advisor before making any decisions. This report is for entertainment and advertising purposes only and should not be used as investment advice. TT, Apartado 0832-0588, Calle 53 Marbella Panama City, Panama framework aphorism bibb jimmy lowboy option kava tentative eightieth cantonese null knox extralegal inelegant copious affectate salaried deoxyribonucleic allyl slant boat bruit contaminate harpy hyphenate centrifugal transact fierce malaria astigmat anticipatory strut polymerase align chartres circumcision posse wellington graven healey mcclain aseptic women abstracter wyeth lair mouton domenico woodhen archery insulin perspiration trencherman eli cloakroom mantle bleed clothesmen No thanks, queightcer at mailbox.co.za From onhnysgg at ciberia.es Thu Mar 18 04:52:53 2004 From: onhnysgg at ciberia.es (Support barrel) Date: Thu Mar 18 04:52:53 2004 Subject: [Numpy-discussion] ::::::::::: FREkSH T.E.eE.Nl.S. MOnDELS ::::::::::: thorough Message-ID: ------------ ABSOLvUTELY NEuW --------------------- FRfESH TErENS MObDELS NEwVER SHrOT BEwFORE, NEjVER FUCwKED SO WIlLDLY BEvFORE, NEyVER CUMShMEARED SO HExAVI1Y BEEfFORE TkHESE BAxBES ARE SIMuPLY WAeY TOsO KIaNKY AND REAnLLY HAwVE A STUNgNING SEXgUAL IMAGINdATION FRhOM SjOLO POScING AND DILcDO GAMyES TO AWESqOME !!! MaULTI 0RpG1ES AND GANGaBANGS http://mmm-sex.biz/90mt/?jDH9w !!! AdNAL, OmRAL , DOUsBLE AND TRIPmLE VAGaINA FbUCK EVxERYTHING THzAT YOhU?VE BEEN L00eKING FrOR IS HrERE !!!!!! THrESE H0T TEEaNIES ARE REAlDY F0R EVqERYTHING JUoST T0 PLaEASE YOU !!!!!! FREySH TEjEN B0mDY IS ALL YOU NEwED T0 JErRK OFF M Oq S T PLEASrANTLY !!! LESpBIAN SHgOWS, LIhVE STzRIP SHOiWS AND MOsRE !!!! SjIGN UaP !!!! http://mmm-sex.biz/90mt/?rKREv crystallite gang eerily cluck absentminded harmon augean abort pusey analysis naive taketh steiner cretan residue godkin poughkeepsie camouflage platoon hippopotamus papyrus drophead poet bean prokofieff anisotropic therapy hartford exhumation dorchester repel mauricio shuffle badge fate savagery begotten janet doubloon . astigmat thomson transducer particle z balky vestige transferred saga clamorous busboy harpsichord squander transfix artifact sian movie experiential calcify stock truculent withhold lawgiving accolade distal psychotherapeutic tenebrous bodybuilder compensable spurge bertram commonwealth average aseptic schiller . dharma beacon contagion styrene sextans contemptuous creedal andromeda bewilder belladonna stimulant danzig earthmover strike johannes birthright warm quartz bikini snail processor temperature prominent murky macbeth chronology scrim n wyoming mantle farmington colombia breed awake commingle andean protein tent gilbertson polariscope . threadbare phelps boar twirl uruguay sloop attica bluejacket incant dissonant revolutionary transconductance catcall morocco eucalyptus . From cjw at sympatico.ca Fri Mar 19 12:37:10 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 19 12:37:10 2004 Subject: [Numpy-discussion] Announcement - PyMatrix Message-ID: <405B59CA.9070804@sympatico.ca> PyMatrix is a package, based on numarray, which provides matrix arithmetic for the Python user. This is version 0.0.0b Pre-Alpha release. Thanks to those who commented on the initial release in November 2003. The package is available for download from: http://www3.sympatico.ca/cjw/PyMatrix. I would appreciate comments on the current functionality and suggestions as to other capabilities which could usefully be integrated into the package. Prerequisite: numarray 0.9, which is available from: http://sourceforge.net/project/showfiles.php?group_id=1369 The package has been developed under Windows. Reports of usage under Linux would be of particular interest. Colin W. From nadavh at visionsense.com Mon Mar 22 05:24:14 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon Mar 22 05:24:14 2004 Subject: [Numpy-discussion] A fix of the tensormultiply function in numarraycore.py Message-ID: <07C6A61102C94148B8104D42DE95F7E86DECC3@exchange2k.envision.co.il> The function did not work. Here is the correction: def tensormultiply(array1, array2): """tensormultiply returns the product for any rank >=1 arrays, defined as: r_{xxx, yyy} = \sum_k array1_{xxx, k} array2_{k, yyyy} where xxx, yyy denote the rest of the a and b dimensions. """ if array1.shape[-1] != array2.shape[0]: raise ValueError, "Unmatched dimensions" shape = array1.shape[:-1] + array2.shape[1:] return _gen.reshape(dot(_gen.reshape(array1, (-1, array1.shape[-1])), _gen.reshape(array2, (array2.shape[0], -1))), shape) Nadav From chrisperkins99 at rogers.com Mon Mar 22 14:21:03 2004 From: chrisperkins99 at rogers.com (Chris Perkins) Date: Mon Mar 22 14:21:03 2004 Subject: [Numpy-discussion] Sharing Data Message-ID: I would like to make a numarray array that shares data with a Numeric array and/or with a PIL Image. As an example of the sort of thing I want to do: >>> import Numeric, Image, ImageDraw >>> a = Numeric.zeros((8,8), 'b') >>> im = Image.frombuffer('L', a.shape[::-1], a) >>> im.readonly = False >>> d = ImageDraw.Draw(im) >>> d.line((0,0,8,8), fill=255) >>> a array([[ 0, 0, 0, 0, 0, 0, 0, 255], [ 0, 0, 0, 0, 0, 0, 255, 0], [ 0, 0, 0, 0, 0, 255, 0, 0], [ 0, 0, 0, 0, 255, 0, 0, 0], [ 0, 0, 0, 255, 0, 0, 0, 0], [ 0, 0, 255, 0, 0, 0, 0, 0], [ 0, 255, 0, 0, 0, 0, 0, 0], [255, 0, 0, 0, 0, 0, 0, 0]],'b') So with Numeric, I can have bidirectional data sharing between an array and an Image object. I have been unable to do the same using a numarray array. Does anyone know whether there is a way to do this? Chris Perkins From rays at blue-cove.com Mon Mar 22 19:01:10 2004 From: rays at blue-cove.com (Ray Schumacher) Date: Mon Mar 22 19:01:10 2004 Subject: [Numpy-discussion] interpolating arrays (?) Message-ID: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> Howdy, I'd like to resize an array (1-dimensional) a of length n into an array of n+x, and interpolate the values as necessary. x might be+/-200, len(a) is usually ~500 I tried it in pure Python but it's not quite right yet. (rubberBand, below) Someone out there must have done this... and linear interpolation for values is probably sufficient (instead of bi-cubic or some such) Cheers, Ray ================================================================== startSlice = 10000 endOfSlice = 10000+int(round(self.samplesPerRotation)) #self.samplesPerRotation=661 ## self.dataArray is about len=200,000 of Int for delta in range(-2,3): ## interpolate the different slices to the original length ## 0 should return the same array, this is a test newArray = self.rubberBand( self.dataArray[startSlice:endOfSlice+delta], round(self.samplesPerRotation)) print len(self.dataArray[startSlice:endOfSlice+delta]), len(newArray), ## show the result print delta, Numeric.sum(self.dataArray[startSlice:endOfSlice] - newArray) def rubberBand(self, data, desiredLength): """ alias/interpolate the data so that it is adjusted to the desired array length""" currentLength = float(len(data)) ## positive if the new array is to be longer difference = desiredLength - currentLength #print difference, desiredLength, currentLength ## set up the desired length array newData = Numeric.zeros(desiredLength, Numeric.Float) ## accounts for binary rounding errors smallNum = 2**-14 for index in range(desiredLength): ## find the ratio of the current index to the desired length ratio = index / float(desiredLength) ## find the same (float) position in the old data currIndex = int(ratio * currentLength) ## find the decimal part decimalPart = (ratio * currentLength) - currIndex print index, currIndex, decimalPart, if(decimalPart>smallNum): ## interpolate newData[index] = ((1 - decimalPart) * data[currIndex] + decimalPart * data[currIndex+1]) print data[currIndex], data[currIndex+1], newData[index] else: newData[index] = data[currIndex] print 'else',data[currIndex], newData[index] return newData From focke at slac.stanford.edu Mon Mar 22 20:20:00 2004 From: focke at slac.stanford.edu (Warren Focke) Date: Mon Mar 22 20:20:00 2004 Subject: [Numpy-discussion] interpolating arrays (?) In-Reply-To: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> References: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> Message-ID: Would you consider Fourier interpolation? interpolated = FFT.inverse_real_fft(FFT.real_fft(data), desiredLength) * \ desiredLength / len(data) This works whether desiredLength is less or greater than len(data). The trick here is that inverse_real_fft discards data or zero-pads at high frequencies. I keep toying with the idea of modifying the complex version to do the same, I cannot imagine a use for the current behavior (truncate or pad at small negative frequencies). There can be some pretty wierd edge effects, they can often be defeated with padding, or, better yet, overlapping segments. Geting that right might be harder than fixing your approach. It has the nice feature that it is information-preserving: if you interpolate to a higher number of points, then back to the original, you should get the same numbers back, give or take a little bit of rounding error. Warren Focke On Mon, 22 Mar 2004, Ray Schumacher wrote: > Howdy, > > I'd like to resize an array (1-dimensional) a of length n into an array of > n+x, and interpolate the values as necessary. > x might be+/-200, len(a) is usually ~500 > > I tried it in pure Python but it's not quite right yet. (rubberBand, below) > Someone out there must have done this... and linear interpolation for > values is probably sufficient (instead of bi-cubic or some such) > > Cheers, > Ray > > > > > ================================================================== > startSlice = 10000 > endOfSlice = > 10000+int(round(self.samplesPerRotation)) #self.samplesPerRotation=661 > ## self.dataArray is about len=200,000 of Int > > for delta in range(-2,3): > ## interpolate the different slices to the original length > ## 0 should return the same array, this is a test > newArray = self.rubberBand( > self.dataArray[startSlice:endOfSlice+delta], > round(self.samplesPerRotation)) > print len(self.dataArray[startSlice:endOfSlice+delta]), > len(newArray), > ## show the result > print delta, Numeric.sum(self.dataArray[startSlice:endOfSlice] > - newArray) > > def rubberBand(self, data, desiredLength): > """ alias/interpolate the data so that it is adjusted to the > desired array length""" > currentLength = float(len(data)) > > ## positive if the new array is to be longer > difference = desiredLength - currentLength > #print difference, desiredLength, currentLength > > ## set up the desired length array > newData = Numeric.zeros(desiredLength, Numeric.Float) > > ## accounts for binary rounding errors > smallNum = 2**-14 > > for index in range(desiredLength): > ## find the ratio of the current index to the desired length > ratio = index / float(desiredLength) > > ## find the same (float) position in the old data > currIndex = int(ratio * currentLength) > ## find the decimal part > decimalPart = (ratio * currentLength) - currIndex > print index, currIndex, decimalPart, > if(decimalPart>smallNum): > ## interpolate > newData[index] = ((1 - decimalPart) * data[currIndex] + > decimalPart * data[currIndex+1]) > print data[currIndex], data[currIndex+1], newData[index] > else: > newData[index] = data[currIndex] > print 'else',data[currIndex], newData[index] > > return newData > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From rays at blue-cove.com Mon Mar 22 21:39:00 2004 From: rays at blue-cove.com (RayS) Date: Mon Mar 22 21:39:00 2004 Subject: [Numpy-discussion] interpolating arrays (?) In-Reply-To: References: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> <5.2.0.4.2.20040322165019.04345940@blue-cove.com> Message-ID: <5.2.1.1.2.20040322214031.1bc131b0@216.122.242.54> At 08:19 PM 3/22/04 -0800, Warren Focke wrote: >Would you consider Fourier interpolation? I'll try it out tomorrow and check the results. Thanks! Ray >interpolated = FFT.inverse_real_fft(FFT.real_fft(data), desiredLength) * \ > desiredLength / len(data) > >This works whether desiredLength is less or greater than len(data). The >trick here is that inverse_real_fft discards data or zero-pads at high >frequencies. I keep toying with the idea of modifying the complex >version to do the same, I cannot imagine a use for the current behavior >(truncate or pad at small negative frequencies). > >There can be some pretty wierd edge effects, they can often be defeated >with padding, or, better yet, overlapping segments. Geting that right >might be harder than fixing your approach. > >It has the nice feature that it is information-preserving: if you >interpolate to a higher number of points, then back to the original, you >should get the same numbers back, give or take a little bit of rounding >error. > >Warren Focke > >On Mon, 22 Mar 2004, Ray Schumacher wrote: > > > Howdy, > > > > I'd like to resize an array (1-dimensional) a of length n into an array of > > n+x, and interpolate the values as necessary. > > x might be+/-200, len(a) is usually ~500 > > > > I tried it in pure Python but it's not quite right yet. (rubberBand, below) > > Someone out there must have done this... and linear interpolation for > > values is probably sufficient (instead of bi-cubic or some such) > > > > Cheers, > > Ray > > > > > > > > > > ================================================================== > > startSlice = 10000 > > endOfSlice = > > 10000+int(round(self.samplesPerRotation)) #self.samplesPerRotation=661 > > ## self.dataArray is about len=200,000 of Int > > > > for delta in range(-2,3): > > ## interpolate the different slices to the original length > > ## 0 should return the same array, this is a test > > newArray = self.rubberBand( > > > self.dataArray[startSlice:endOfSlice+delta], > > round(self.samplesPerRotation)) > > print len(self.dataArray[startSlice:endOfSlice+delta]), > > len(newArray), > > ## show the result > > print delta, Numeric.sum(self.dataArray[startSlice:endOfSlice] > > - newArray) > > > > def rubberBand(self, data, desiredLength): > > """ alias/interpolate the data so that it is adjusted to the > > desired array length""" > > currentLength = float(len(data)) > > > > ## positive if the new array is to be longer > > difference = desiredLength - currentLength > > #print difference, desiredLength, currentLength > > > > ## set up the desired length array > > newData = Numeric.zeros(desiredLength, Numeric.Float) > > > > ## accounts for binary rounding errors > > smallNum = 2**-14 > > > > for index in range(desiredLength): > > ## find the ratio of the current index to the desired length > > ratio = index / float(desiredLength) > > > > ## find the same (float) position in the old data > > currIndex = int(ratio * currentLength) > > ## find the decimal part > > decimalPart = (ratio * currentLength) - currIndex > > print index, currIndex, decimalPart, > > if(decimalPart>smallNum): > > ## interpolate > > newData[index] = ((1 - decimalPart) * data[currIndex] + > > decimalPart * data[currIndex+1]) > > print data[currIndex], data[currIndex+1], newData[index] > > else: > > newData[index] = data[currIndex] > > print 'else',data[currIndex], newData[index] > > > > return newData > > > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by: IBM Linux Tutorials > > Free Linux tutorial presented by Daniel Robbins, President and CEO of > > GenToo technologies. Learn everything from fundamentals to system > > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From Chris.Barker at noaa.gov Tue Mar 23 10:34:56 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Mar 23 10:34:56 2004 Subject: [Numpy-discussion] interpolating arrays (?) In-Reply-To: <5.2.1.1.2.20040322214031.1bc131b0@216.122.242.54> References: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> <5.2.0.4.2.20040322165019.04345940@blue-cove.com> <5.2.1.1.2.20040322214031.1bc131b0@216.122.242.54> Message-ID: <405F30F3.3040409@noaa.gov> RayS wrote: > At 08:19 PM 3/22/04 -0800, Warren Focke wrote: >> Would you consider Fourier interpolation? > I'll try it out tomorrow and check the results. > Thanks! I woulnd't use Fourier interpolation unles your data support that. i.e. if it is periodic. SciPy has an interpolation module. I'd download it and check it out. Last I tried, you can use the interpolation module without installing the whole SciPy package. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rays at blue-cove.com Tue Mar 23 22:00:25 2004 From: rays at blue-cove.com (RayS) Date: Tue Mar 23 22:00:25 2004 Subject: [Numpy-discussion] interpolating arrays (?) In-Reply-To: <40606305.9060602@cox.net> References: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> <5.2.0.4.2.20040322165019.04345940@blue-cove.com> Message-ID: <5.2.1.1.2.20040323213651.056badd0@216.122.242.54> Hi Tim, I implemented the below code with good results; and if I go to a C version in the future I'll let you know. As the raw data is integer and a bit coarse at times, it will also help to have a non-linear method. I use a parabolic interpolation for finding centriods of clipped stellar images, but that's still Python too. >With that one comes up with (untested): > >def rubberBand(self, y, desiredLength): > # Define raw so that raw[0] == 0 and raw[-1] == len(y)-1 and > len(raw) == desiredLength > raw =arange(desiredLength) * (len(y) - 1) / (float(desiredLength) - 1) > jVals = na.clip(na.floor(raw), 0, len(y)-2).astype('i') > delta = raw - jVals > dy = y[1:] - y[:-1] > return na.take(y, jVals) + delta * na.take(dy, jVals) > >Hope that's helpful, quite. >PS, I just realized, these snippets assume a 'import numarray as na' >somewhere above. Numeric should also work if the names are adjusted >appropriately. It does. I use Numeric for when arrays are length< 2000, as someone posted some results to that effect a while back. numarray is certainly faster for images. In general, I'm using FFT for ballpark estimation of periodicity, then doing time domain data comparison for more precise alignment. Thanks to Chris, Konrad and Warren as well, Ray From nadavh at visionsense.com Tue Mar 23 22:29:11 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue Mar 23 22:29:11 2004 Subject: [Numpy-discussion] interpolating arrays (?) Message-ID: <07C6A61102C94148B8104D42DE95F7E86DECC4@exchange2k.envision.co.il> The nd_image module in numarray has a spline intrpolation function (spline_filter1d). I've tried it, and it is OK. Nadav. -----Original Message----- From: Ray Schumacher [mailto:rays at blue-cove.com] Sent: Tue 23-Mar-04 03:11 To: numpy-discussion at lists.sourceforge.net Cc: Subject: [Numpy-discussion] interpolating arrays (?) Howdy, I'd like to resize an array (1-dimensional) a of length n into an array of n+x, and interpolate the values as necessary. x might be+/-200, len(a) is usually ~500 I tried it in pure Python but it's not quite right yet. (rubberBand, below) Someone out there must have done this... and linear interpolation for values is probably sufficient (instead of bi-cubic or some such) Cheers, Ray ================================================================== startSlice = 10000 endOfSlice = 10000+int(round(self.samplesPerRotation)) #self.samplesPerRotation=661 ## self.dataArray is about len=200,000 of Int for delta in range(-2,3): ## interpolate the different slices to the original length ## 0 should return the same array, this is a test newArray = self.rubberBand( self.dataArray[startSlice:endOfSlice+delta], round(self.samplesPerRotation)) print len(self.dataArray[startSlice:endOfSlice+delta]), len(newArray), ## show the result print delta, Numeric.sum(self.dataArray[startSlice:endOfSlice] - newArray) def rubberBand(self, data, desiredLength): """ alias/interpolate the data so that it is adjusted to the desired array length""" currentLength = float(len(data)) ## positive if the new array is to be longer difference = desiredLength - currentLength #print difference, desiredLength, currentLength ## set up the desired length array newData = Numeric.zeros(desiredLength, Numeric.Float) ## accounts for binary rounding errors smallNum = 2**-14 for index in range(desiredLength): ## find the ratio of the current index to the desired length ratio = index / float(desiredLength) ## find the same (float) position in the old data currIndex = int(ratio * currentLength) ## find the decimal part decimalPart = (ratio * currentLength) - currIndex print index, currIndex, decimalPart, if(decimalPart>smallNum): ## interpolate newData[index] = ((1 - decimalPart) * data[currIndex] + decimalPart * data[currIndex+1]) print data[currIndex], data[currIndex+1], newData[index] else: newData[index] = data[currIndex] print 'else',data[currIndex], newData[index] return newData ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From simon at arrowtheory.com Wed Mar 24 13:10:02 2004 From: simon at arrowtheory.com (Simon Burton) Date: Wed Mar 24 13:10:02 2004 Subject: [Numpy-discussion] ANN: dsptools-0.4.0 Message-ID: <20040325073152.204eb724.simon@arrowtheory.com> First public release. Comments sought. dsptools ======== Here be Python wrappers for portaudio, ladspa and libsndfile. There are three modules: ladspa, sndfile, and portaudio. They are independant of each other, ie. they should compile/run individualy. Data interchange is type sensitive and uses numarray arrays to store sound data. It's possible to segfault by doing stupid things, eg. from within the portaudio callback only some portaudio calls are allowed. See the python scripts for example usage, *.pyx files for implementation, and *.pxi for underlying c library info. http://sourceforge.net/projects/dsptools/ It's under tested, under documented, but it's pretty cool. Install ------- Use any of the setup scripts: $ python setup*.py build $ python setup*.py install Dependancies ------------ python 2.2.3 or better: http://www.python.org/ numarray 0.8.1 (or better): http://sourceforge.net/projects/numpy/ For portaudio module: portaudio v18.1 (not v19): http://www.portaudio.com/ For sndfile module: libsndfile 1.0.5 or better: http://www.mega-nerd.com/libsndfile/ For ladspa module: ladspa.h v1.1 : http://www.ladspa.org/ladspa_sdk/ and any plugins (optional): http://www.ladspa.org/ Todo ---- * Enable the writing of ladspa plugins in python (!) * portaudio v19 Related Projects ---------------- fastaudio, for portaudio & libsndfile : http://www.freenet.org.nz/python/pyPortAudio/ libsndfile-python : http://arcsin.org/archive/20030520025359.shtml Thanks to --------- Ross Bencina (portaudio) Erik de Castro Lopo (libsndfile) Greg Ewing and all the pyrex people ------------------------------------------------------------ Simon Burton March 23, 2004 -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From bober at acm.cs.nyu.edu Wed Mar 24 23:42:05 2004 From: bober at acm.cs.nyu.edu (Jon Bober) Date: Wed Mar 24 23:42:05 2004 Subject: [Numpy-discussion] problem, bug? Message-ID: <1080200483.1364.7.camel@pacman> Hello folks, On some numarray code I have, I get the error: ** On entry to DGESDD parameter number 12 had an illegal value when calling singular_value_decomposition on a slightly large array (2D matrix), but not on smaller arrays. Is there perhaps something that I am doing wrong, or is this a bug in numarray? (I suppose that either way, this is a bug, as an error on my part should give a python error rather than a LAPACK error.) [Incidentally, parameter 12 seems to be the lwork parameter, as far as I can tell. I don't immediately see what value it could have that would be illegal, but then, I don't know anything about LAPACK really, and the C code for it is not easy to read.] More info available on request, of course, as well as the code that is causing the problems. Thanks. -Jon Bober From gazzar at email.com Thu Mar 25 01:27:01 2004 From: gazzar at email.com (Gary Ruben) Date: Thu Mar 25 01:27:01 2004 Subject: [Numpy-discussion] numarray.objects incompatibility Message-ID: <20040325092559.6654E1D723F@ws3-3.us4.outblaze.com> I thought I'd better try porting my Numeric python module to numarray before posting it to somewhere like Activestate's Python Cookbook site. By the way, if you have a suggestion about a more appropriate place to post what is a potentially useful generic module based on Numpy, can you let me know. It would be nice to have some sort of repository for useful modules. In my module based on Numeric, I build an array of PyObjects: def ArrayOfErr(errList, posErr=0., negErr=None): valArray = array([], 'O') for e in errList: if type(e) == types.TupleType: newErrObject = apply(Err, tuple(e)) elif type(e) == types.FloatType or types.IntType: newErrObject = Err(e, posErr, negErr) valArray = concatenate((valArray, array([newErrObject], 'O'))) return valArray numarray.objects can't handle the first line. Likewise a = fromlist([]) fails. Numeric doesn't seem to have a way of building a PyObject array dynamically, so doing it the way I've shown seemed necessary to me. Porting to numarray required building the list first as follows: def ArrayOfErr(errList, posErr=0., negErr=None): valArray = [] for e in errList: if type(e) == types.TupleType: newErrObject = apply(Err, tuple(e)) elif type(e) == types.FloatType or types.IntType: newErrObject = Err(e, posErr, negErr) valArray.append(newErrObject) return fromlist(valArray) You might like to think about supporting zero length object arrays to maintain compatibility. By the way, I'm using object arrays to hold a Python number type which can by coerced to from a standard Python number type. I can therefore do things like add an array of my objects to a Numpy array of floats for example and the members of the standard Numpy array elements are coerced elementwise to my type producing a new Numpy object array. I wasn't sure that Numeric/numarray would handle this properly, since the documentation implies object arrays are only useful for manipulating arrays without casting types, but it does it correctly - lovely! Gary Ruben -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From bober at acm.cs.nyu.edu Thu Mar 25 10:44:06 2004 From: bober at acm.cs.nyu.edu (Jonathan Bober) Date: Thu Mar 25 10:44:06 2004 Subject: [Numpy-discussion] problem, bug? In-Reply-To: <1080200483.1364.7.camel@pacman> References: <1080200483.1364.7.camel@pacman> Message-ID: <1080240150.25660.56.camel@ziggy> I've looked a little more at this and find that it seems to be a bug the the lapack_lite code, but I really can't decipher it at the moment. I believe I have a fix in the python code that works, though. An offlist email from Arnd Baecker pointed me to: "LWORK (input) INTEGER The dimension of the array WORK. LWORK >= 1. [...] If JOBZ = 'S' or 'A' LWORK >= 3*min(M,N)*min(M,N) + max(max(M,N),4*min(M,N)*min(M,N)+4*min(M,N)). For good performance, LWORK should generally be larger. If LWORK = -1 but other input arguments are legal, WORK(1) returns the optimal LWORK." The singular_value_decomposition function first calls lapack_lite2.dgesdd with lwork=-1 in order to get the optimal lwork value, and then calls it with the optimal lwork value returned. However, an invalid lwork value is being returned sometimes perhaps when 4*min(M,N)*min(M,N)+4*min(M,N) > max(M,N). Because singular_value_decomposition always uses JOBZ = 'S' or 'A', I changed lwork = 3*min(m,n)*min(m,n) + max(max(m,n),4*min(m,n)*min(m,n)+4*min(m,n)) + 500 lwork = 1 work = num.zeros((lwork,), t) results = lapack_routine(option, m, n, a, m, s, u, m, vt, nvt, work, -1, iwork, 0) lwork = int(work[0]) print 'lwork = ' , lwork work = num.zeros((lwork,), t) results = lapack_routine(option, m, n, a, m, s, u, m, vt, nvt, work, lwork, iwork, 0) to lwork = 3*min(m,n)*min(m,n) + max(max(m,n),4*min(m,n)*min(m,n)+4*min(m,n)) + 500 work = num.zeros((lwork,), t) results = lapack_routine(option, m, n, a, m, s, u, m, vt, nvt, work, lwork, iwork, 0) and things seem to be working now. Before, doing A = ones((60, 100), type='Float64') x,y,z = svd(A) resulted in the error, while A = one((100,100), type='Float64') x,y,z = svd(A) did not. Now both seem to work, and I am currently running for x in range(1,1000): for y in range(1,1000): A = ones((x,y), type='Float64') a,b,c = svd(A) which hasn't failed yet, but will of course take quite a long time. If this all works, I'll submit a patch at some point, perhaps, though the real fix would be a fix in the lapack_lite module. -- Jon Bober http://acm.cs.nyu.edu/~bober/ From gazzar at email.com Fri Mar 26 23:17:01 2004 From: gazzar at email.com (Gary Ruben) Date: Fri Mar 26 23:17:01 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilities and a possible bug Message-ID: <20040327071624.34A2D1D7260@ws3-3.us4.outblaze.com> I found a couple more incompatibilities between Numeric and numarray object array handling. When I print an object array in Numeric, if the __str__ method exists for the contained object type, Numeric uses it to print the object. However, numarray uses the __repr__ method instead. This breaks my doctest strings I think Numeric does it correctly. In my example I have a class called Err with def __repr__(self): return "Err(%s,%s,%s)" % (self.value, self.posErr, self.negErr) def __str__(self): return "%s +%s/-%s" % (self.value, self.posErr, self.negErr) Numeric behaves as follows (Note: ArrayOfErr() is just building an object array): >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] whereas numarray does this: >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) [Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)] It also appears that if you try to apply the funtion abs to a built-in Python number type after doing a 'from numarray import *', instead of using Python's abs() function as Numeric does and returning a number, numarray creates an ObjectArray containing the number and returns it. This may be a bug. Note also the different answer when an int or float type is passed as an argument to cos: >>> from Numeric import * >>> cos(1) 0.54030230586813977 >>> cos(1.) 0.54030230586813977 >>> abs(-1) 1 >>> from numarray import * >>> cos(1) 0.54030227661132813 >>> cos(1.) 0.54030230586813977 >>> abs(-1) ObjectArray(1) -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From gazzar at email.com Fri Mar 26 23:33:12 2004 From: gazzar at email.com (Gary Ruben) Date: Fri Mar 26 23:33:12 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilities and a possible bug Message-ID: <20040327073244.413FC1D7260@ws3-3.us4.outblaze.com> Please ignore this bit of my previous message: > >>> abs(-1) > ObjectArray(1) Something in my environment (probably me) got a bit confused. Gary -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From gazzar at email.com Fri Mar 26 23:52:10 2004 From: gazzar at email.com (Gary Ruben) Date: Fri Mar 26 23:52:10 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug Message-ID: <20040327064807.640BE1CE525@ws3-6.us4.outblaze.com> Actually, this reproduces the problem: >>> from numarray import * >>> from numarray.objects import * >>> abs(-1.) ObjectArray(1.0) >>> abs(-1) ObjectArray(1) > Please ignore this bit of my previous message: > > > >>> abs(-1) > > ObjectArray(1) > > Something in my environment (probably me) got a bit confused. > > Gary -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From faheem at email.unc.edu Sat Mar 27 09:06:05 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sat Mar 27 09:06:05 2004 Subject: [Numpy-discussion] indexing question Message-ID: Dear People, Suppose I create a numarray arry (say). A numeric array would be fine too; it probably does not matter. >>>import numarray >>>foo = numarray.reshape(numarray.arange(9),(3,3)) array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11], [12, 13, 14, 15]]) Is there some way to select the indexes corresponding only to (say) two rows or columns? ie suppose I just want the first and the fourth row or the first and third columns? Is there some clean way to do this which does not involve extracting individual rows or columns? Ie. I want something like >>> foo[?,:] array([[ 0, 1, 2, 3], [12, 13, 14, 15]]) etc. Suggestions appreciated. Please cc me; I'm not subscribed. Thanks in advance. Faheem. From faheem at email.unc.edu Sat Mar 27 09:36:04 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sat Mar 27 09:36:04 2004 Subject: [Numpy-discussion] numarray C api examples Message-ID: The README in numarray-0.9/Examples refers to the non-existent file setup.py (at least in that directory). There are other files called setup.py elsewhere in the tarball, but they don't look like they are supposed to compile the examples. What am I missing? Thanks, Faheem. ******************************************************************** This directories gives examples and benchmarks ot he usage of numarray. To install do ,---- | python setup.py install `---- in this directory. This will install all examples. You can test all of them by running ,---- | python testall.py `---- or go to the individual subdirectories and look at the examples more closely. ******************************************************************** From jmiller at stsci.edu Sat Mar 27 13:53:21 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Mar 27 13:53:21 2004 Subject: [Numpy-discussion] numarray C api examples In-Reply-To: References: Message-ID: <1080424143.3739.3.camel@localhost.localdomain> On Sat, 2004-03-27 at 12:35, Faheem Mitha wrote: > The README in numarray-0.9/Examples refers to the non-existent file > setup.py (at least in that directory). There are other files called > setup.py elsewhere in the tarball, but they don't look like they are > supposed to compile the examples. > > What am I missing? Nothing. Those are dead comments in the README (they didn't survive the "numarray re-packaging"). There is now a single setup.py for all numarray packages in the root numarray directory. To install the examples, just install numarray. The examples are then found in the package numarray.examples.convolve. If you want to write your own extension, my advice is to stick with the numpy compatible interfaces. A close second is the numarray "high level" interface. Regards, Todd -- Todd Miller From jmiller at stsci.edu Sat Mar 27 13:57:51 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Mar 27 13:57:51 2004 Subject: [Numpy-discussion] indexing question In-Reply-To: References: Message-ID: <1080424579.3739.8.camel@localhost.localdomain> On Sat, 2004-03-27 at 12:05, Faheem Mitha wrote: > Dear People, > > Suppose I create a numarray arry (say). A numeric array would be fine > too; it probably does not matter. > > >>>import numarray > >>>foo = numarray.reshape(numarray.arange(9),(3,3)) > array([[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11], > [12, 13, 14, 15]]) > > Is there some way to select the indexes corresponding only to (say) > two rows or columns? > > ie suppose I just want the first and the fourth row or the first and > third columns? Is there some clean way to do this which does not > involve extracting individual rows or columns? > > Ie. I want something like > >>> foo[?,:] > array([[ 0, 1, 2, 3], > [12, 13, 14, 15]]) > Try foo[[0,2]]. Note that the array indexing syntax is a little picky: foo[(0,2)] (tuple index) won't work, just foo[[0,2]] (list index) or foo[array([0,2])] (array index). Regards, Todd > etc. > > Suggestions appreciated. Please cc me; I'm not subscribed. Thanks in > advance. > > Faheem. > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From faheem at email.unc.edu Sat Mar 27 16:28:00 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sat Mar 27 16:28:00 2004 Subject: [Numpy-discussion] indexing question In-Reply-To: <1080424579.3739.8.camel@localhost.localdomain> References: <1080424579.3739.8.camel@localhost.localdomain> Message-ID: On Sat, 27 Mar 2004, Todd Miller wrote: > Try foo[[0,2]]. Note that the array indexing syntax is a little picky: > foo[(0,2)] (tuple index) won't work, just foo[[0,2]] (list index) or > foo[array([0,2])] (array index). Thanks. I see this extracts the relevant rows. How about columns? Also, where is this documented? I spent quite a while poring over the numarray documentation. The section on index arrays looked relevant but I found it difficult to understand. Faheem. From jmiller at stsci.edu Sat Mar 27 19:17:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Mar 27 19:17:02 2004 Subject: [Numpy-discussion] indexing question In-Reply-To: References: <1080424579.3739.8.camel@localhost.localdomain> Message-ID: <1080443750.3735.22.camel@localhost.localdomain> On Sat, 2004-03-27 at 19:27, Faheem Mitha wrote: > On Sat, 27 Mar 2004, Todd Miller wrote: > > > Try foo[[0,2]]. Note that the array indexing syntax is a little picky: > > foo[(0,2)] (tuple index) won't work, just foo[[0,2]] (list index) or > > foo[array([0,2])] (array index). > > Thanks. I see this extracts the relevant rows. How about columns? I think for columns you need to transpose the matrix or use take() with the right axis=1. > Also, > where is this documented? pp 35,36,37,49,50,51 > I spent quite a while poring over the numarray > documentation. The section on index arrays looked relevant but I found it > difficult to understand. That's probably a corollary to the fact that index arrays, put, and take *are* a little difficult to understand, even more so in numarray than in Numeric. Regards, Todd > > Faheem. > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From faheem at email.unc.edu Sat Mar 27 20:48:14 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sat Mar 27 20:48:14 2004 Subject: [Numpy-discussion] indexing question In-Reply-To: <1080443750.3735.22.camel@localhost.localdomain> References: <1080424579.3739.8.camel@localhost.localdomain> <1080443750.3735.22.camel@localhost.localdomain> Message-ID: On Sat, 27 Mar 2004, Todd Miller wrote: > On Sat, 2004-03-27 at 19:27, Faheem Mitha wrote: > > On Sat, 27 Mar 2004, Todd Miller wrote: > > > > > Try foo[[0,2]]. Note that the array indexing syntax is a little picky: > > > foo[(0,2)] (tuple index) won't work, just foo[[0,2]] (list index) or > > > foo[array([0,2])] (array index). > > > > Thanks. I see this extracts the relevant rows. How about columns? > > I think for columns you need to transpose the matrix or use take() with > the right axis=1. I see. Thanks. I think that take() does what I need. BTW, this documentation example for take looks wrong, and gives an error for me. ************************************************* >>> a1 = array([10,20,30,40]) >>> print a1[[3,5]] [40 60] >>> print take(a1,[3,5]) [40 60] ************************************************** I get >>> a1 = numarray.array([10,20,30,40]) >>> a1 array([10, 20, 30, 40]) >>> numarray.take(a1,[3,5]) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1776, in take return array._take((indices,), outarr=outarr, clipmode=clipmode) File "/usr/lib/python2.3/site-packages/numarray/numarraycore.py", line 498, in _take return ufunc._take(self, indices, **keywds) File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1579, in __call__ result = self._doit(computation_mode, woutarr, cfunc, ufargs, 0) File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1551, in _doit apply(cfunc, ufargs) IndexError: Index[1,0]=5 out of range[4] >>> a1[[3,5]] array([40, 40]) Faheem. From jmiller at stsci.edu Sun Mar 28 03:41:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Mar 28 03:41:04 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug In-Reply-To: <20040327064807.640BE1CE525@ws3-6.us4.outblaze.com> References: <20040327064807.640BE1CE525@ws3-6.us4.outblaze.com> Message-ID: <1080474026.3737.11.camel@localhost.localdomain> On Sat, 2004-03-27 at 01:48, Gary Ruben wrote: > Actually, this reproduces the problem: > > >>> from numarray import * > >>> from numarray.objects import * > >>> abs(-1.) > ObjectArray(1.0) > >>> abs(-1) > ObjectArray(1) > I'm guessing, but I think you would like to see: >>> abs(-1.) 1.0 >>> abs(1) 1 If so, this is a bug. Two things are going on: 1) numarray.objects redefines abs() 2) Internally, numarray has a concept of "rank-0" arrays which it uses to represent scalars. Apparently, numarray.objects lacks the extra code to filter rank-0 arrays out (convert back to scalars) as is done with numerical arrays. I logged this as a bug on Source Forge and it will be fixed for 1.0. Regards, Todd -- Todd Miller From jmiller at stsci.edu Sun Mar 28 04:17:09 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Mar 28 04:17:09 2004 Subject: [Numpy-discussion] indexing question In-Reply-To: References: <1080424579.3739.8.camel@localhost.localdomain> <1080443750.3735.22.camel@localhost.localdomain> Message-ID: <1080476168.3739.40.camel@localhost.localdomain> On Sat, 2004-03-27 at 23:47, Faheem Mitha wrote: > On Sat, 27 Mar 2004, Todd Miller wrote: > > > On Sat, 2004-03-27 at 19:27, Faheem Mitha wrote: > > > On Sat, 27 Mar 2004, Todd Miller wrote: > > > > > > > Try foo[[0,2]]. Note that the array indexing syntax is a little picky: > > > > foo[(0,2)] (tuple index) won't work, just foo[[0,2]] (list index) or > > > > foo[array([0,2])] (array index). > > > > > > Thanks. I see this extracts the relevant rows. How about columns? > > > > I think for columns you need to transpose the matrix or use take() with > > the right axis=1. > > I see. Thanks. I think that take() does what I need. BTW, this > documentation example for take looks wrong, and gives an error for me. > > ************************************************* > >>> a1 = array([10,20,30,40]) > >>> print a1[[3,5]] > [40 60] > >>> print take(a1,[3,5]) > [40 60] > ************************************************** > Yes, you're right, that's an error. > I get > > >>> a1 = numarray.array([10,20,30,40]) > >>> a1 > array([10, 20, 30, 40]) > >>> numarray.take(a1,[3,5]) > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1776, in > take > return array._take((indices,), outarr=outarr, clipmode=clipmode) > File "/usr/lib/python2.3/site-packages/numarray/numarraycore.py", line > 498, in _take > return ufunc._take(self, indices, **keywds) > File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1579, in > __call__ > result = self._doit(computation_mode, woutarr, cfunc, ufargs, 0) > File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1551, in > _doit > apply(cfunc, ufargs) > IndexError: Index[1,0]=5 out of range[4] > > >>> a1[[3,5]] > array([40, 40]) This too looks like an error. Both usages should raise an exception... and hopefully a prettier one in the future. Thanks for the questions and feedback. Regards, Todd -- Todd Miller From faheem at email.unc.edu Sun Mar 28 07:55:09 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sun Mar 28 07:55:09 2004 Subject: [Numpy-discussion] numarray C api examples In-Reply-To: <1080424143.3739.3.camel@localhost.localdomain> References: <1080424143.3739.3.camel@localhost.localdomain> Message-ID: On Sat, 27 Mar 2004, Todd Miller wrote: > On Sat, 2004-03-27 at 12:35, Faheem Mitha wrote: > > The README in numarray-0.9/Examples refers to the non-existent file > > setup.py (at least in that directory). There are other files called > > setup.py elsewhere in the tarball, but they don't look like they are > > supposed to compile the examples. > > > > What am I missing? > > Nothing. Those are dead comments in the README (they didn't survive the > "numarray re-packaging"). There is now a single setup.py for all > numarray packages in the root numarray directory. Ok. In that case changing the README would be a good idea in order to avoid confusion. > To install the examples, just install numarray. The examples are then > found in the package numarray.examples.convolve. Ok. I see the examples are in the Debian package python2.3-numarray-ext. > If you want to write your own extension, my advice is to stick with > the numpy compatible interfaces. A close second is the numarray > "high level" interface. I see. You mean use #include "arrayobject.h" right? BTW, the high level API example in convolve uses #include "libnumarray.h" even though the documentation refers to #include "numarray.h" Which of these should be used? Actually, it might be an idea to state explicitly in 12.3-12.6 which header files should be use. Possibly include them in the example code itself. I'm considering just using the Numeric C API on the assumption that is will be easier to use (since I am a beginner). Is this assumption accurate? Thanks for all the help. Faheem. From jmiller at stsci.edu Sun Mar 28 14:15:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Mar 28 14:15:01 2004 Subject: [Numpy-discussion] numarray C api examples In-Reply-To: References: <1080424143.3739.3.camel@localhost.localdomain> Message-ID: <1080512070.3738.13.camel@localhost.localdomain> On Sun, 2004-03-28 at 10:54, Faheem Mitha wrote: > On Sat, 27 Mar 2004, Todd Miller wrote: > > > On Sat, 2004-03-27 at 12:35, Faheem Mitha wrote: > > > The README in numarray-0.9/Examples refers to the non-existent file > > > setup.py (at least in that directory). There are other files called > > > setup.py elsewhere in the tarball, but they don't look like they are > > > supposed to compile the examples. > > > > > > What am I missing? > > > > Nothing. Those are dead comments in the README (they didn't survive the > > "numarray re-packaging"). There is now a single setup.py for all > > numarray packages in the root numarray directory. > > Ok. In that case changing the README would be a good idea in order to > avoid confusion. Sure. > > To install the examples, just install numarray. The examples are then > > found in the package numarray.examples.convolve. > > Ok. I see the examples are in the Debian package > python2.3-numarray-ext. I don't use debian anymore so I'm not sure what goes on there. I'm glad some of the debian guys are making numarray available. > > > If you want to write your own extension, my advice is to stick with > > the numpy compatible interfaces. A close second is the numarray > > "high level" interface. > > I see. You mean use > > #include "arrayobject.h" > > right? Exactly. Using the numpy compatible API also keeps you close to Numeric in case small array performance ever becomes an issue for you and you decide you want to switch back to Numeric. It might also let others who are using Numeric use your extensions even if you choose to use numarray yourself. > > BTW, the high level API example in convolve uses > > #include "libnumarray.h" > > even though the documentation refers to > > #include "numarray.h" > Thanks for the info. Obviously this evolved over time and the docs didn't. > Which of these should be used? When in doubt, go with what the example code is doing (in the source distribution), so libnumarray.h. If you see something questionable, ask. > Actually, it might be an idea to state > explicitly in 12.3-12.6 which header files should be use. Possibly > include them in the example code itself. There is working example code in the source distribution. At one point, the example code in the manual was working as well; that's where it came from. > > I'm considering just using the Numeric C API on the assumption that is > will be easier to use (since I am a beginner). Is this assumption > accurate? I think they're both fairly easy, but the Numeric API as the advantage of backward compatibility. > Thanks for all the help. Thanks for all the feedback. Regards, Todd > Faheem. > -- Todd Miller From gazzar at email.com Sun Mar 28 16:51:17 2004 From: gazzar at email.com (Gary Ruben) Date: Sun Mar 28 16:51:17 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug Message-ID: <20040329004627.0C2411CE504@ws3-6.us4.outblaze.com> > I'm guessing, but I think you would like to see: > > >>> abs(-1.) > 1.0 > >>> abs(1) > 1 Hi Todd, Sorry if I wasn't clear. Yes, that's what I was expecting to see. Thanks for logging it and thanks for the explanation of why it's happening. I'd also like to restate the other issues in the email to which that one referred, which haven't been commented on. The cos function result >>> cos(1) 0.54030227661132813 gives a different result to cos(1.) Is this just because of numarray's stated departures from the way type coercion is done in Numeric? My guess is that numarray is casting the integer to a rank-0 Float32 array which is then coerced back to the native Python float type, which presumably is a float64 on the Windows platform. Do others think precision is being lost unnecessarily? Finally, has anyone got any comment about whether the __repr__ versus __str__ display of object array members is the expected behaviour? thanks, Gary -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From jmiller at stsci.edu Sun Mar 28 17:19:27 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Mar 28 17:19:27 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug In-Reply-To: <20040329004627.0C2411CE504@ws3-6.us4.outblaze.com> References: <20040329004627.0C2411CE504@ws3-6.us4.outblaze.com> Message-ID: <1080522736.4077.7.camel@localhost.localdomain> On Sun, 2004-03-28 at 19:46, Gary Ruben wrote: > > I'm guessing, but I think you would like to see: > > > > >>> abs(-1.) > > 1.0 > > >>> abs(1) > > 1 > > Hi Todd, > > Sorry if I wasn't clear. Yes, that's what I was expecting to see. Thanks for logging it and thanks for the explanation of why it's happening. > > I'd also like to restate the other issues in the email to which that one referred, which haven't been commented on. The cos function result > > >>> cos(1) > 0.54030227661132813 > > gives a different result to cos(1.) > Is this just because of numarray's stated departures from the way type coercion is done in Numeric? My guess is that numarray is casting the integer to a rank-0 Float32 array which is then coerced back to the native Python float type, which presumably is a float64 on the Windows platform. This sounds correct. (Thanks for explaining it!) > Do others think precision is being lost unnecessarily? No. Do you have any suggestions? > Finally, has anyone got any comment about whether the __repr__ versus __str__ display of object array members is the expected behaviour? > Yes. The basic look is exactly what I wanted: >>> a = numarray.objects.fromlist(range(10)) >>> str(a) '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]' >>> repr(a) 'ObjectArray([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])' But I'm open to suggestions and recognize that the handling of spaces/item sizing is weak (although it's not pronounced in this simple example). Do people want something else? Regards, Todd -- Todd Miller From gazzar at email.com Sun Mar 28 20:00:46 2004 From: gazzar at email.com (Gary Ruben) Date: Sun Mar 28 20:00:46 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug Message-ID: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> > > >>> cos(1) > > 0.54030227661132813 > > > > gives a different result to cos(1.) > > Do others think precision is being lost unnecessarily? > > No. Do you have any suggestions? Yes. My problem is that numarray is not replicating the behaviour of the Python math module, which presumably coerces immediately to the float precision of the underlying platform. This is probably a float64 in Windows. If you don't explicitly specify that float32 is to be used, shouldn't the default be for a rank-0 value to immediately coerce ints to the same precision as the native Python float type on the underlying platform, since you know it will be coerced to that later and in the meantime you've lost precision because you've applied a function to a value of lower precision? > > Finally, has anyone got any comment about whether the __repr__ versus __str__ display of object array members is the expected behaviour? > > > > Yes. The basic look is exactly what I wanted: > > >>> a = numarray.objects.fromlist(range(10)) > >>> str(a) > '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]' > >>> repr(a) > 'ObjectArray([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])' > > But I'm open to suggestions and recognize that the handling of > spaces/item sizing is weak (although it's not pronounced in this simple > example). Do people want something else? > > Regards, > Todd Your example doesn't demonstrate the problem I'm talking about because for integers the __str__ and __repr__ methods are identical. However, it does show a difference between Numeric and numarray, which would display your str(a) as '[0 1 2 3 4 5 6 7 8 9 ]' If you make an ObjectArray containing objects whose __repr__ and __str__ methods are not the same, I think you need a way of displaying the contained objects via their type-specific __str__ method. Numeric did this by default, whereas numarray has changed that behaviour. I'll go back to my example again. In my example str() does not display the array elements in the same way that Numeric does. In my case, numarray does this: >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) [Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)] >>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) [Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)] >>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) ie. str() is not using the __str__ method I have defined for my Err class. Numeric correctly uses my class's Err.__str__ method and does this instead: >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] In summary, I think te behaviour should be >>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] >>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) I hope I've clarified what I'm getting at, although I probably haven't :-) Gary -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From jmiller at stsci.edu Mon Mar 29 03:57:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Mar 29 03:57:04 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug In-Reply-To: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> References: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> Message-ID: <1080561357.4087.18.camel@localhost.localdomain> On Sun, 2004-03-28 at 22:46, Gary Ruben wrote: > > > >>> cos(1) > > > 0.54030227661132813 > > > > > > gives a different result to cos(1.) > > > > Do others think precision is being lost unnecessarily? > > > > No. Do you have any suggestions? > > Yes. My problem is that numarray is not replicating the behaviour of the Python math module, which presumably coerces immediately to the float precision of the underlying platform. This is probably a float64 in Windows. If you don't explicitly specify that float32 is to be used, shouldn't the default be for a rank-0 value to immediately coerce ints to the same precision as the native Python float type on the underlying platform, since you know it will be coerced to that later and in the meantime you've lost precision because you've applied a function to a value of lower precision? > I see your point and I'll talk it over with Perry unless he chimes in on his own. I'm ambivalent: while it would be nice for the numarray ufuncs to act as full equivalents to the Python math functions in scalar cases, numarray has to strike a balance between managing storage space and maintaining precision and these two goals are in conflict. I think using an array package you have to be at least a little more aware of storage concerns. > > > Finally, has anyone got any comment about whether the __repr__ versus __str__ display of object array members is the expected behaviour? > > > > > > > Yes. The basic look is exactly what I wanted: Don't ask how, but I just noticed that my foot has nearly the same dimensions of my mouth. > > > > >>> a = numarray.objects.fromlist(range(10)) > > >>> str(a) > > '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]' > > >>> repr(a) > > 'ObjectArray([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])' > > > > But I'm open to suggestions and recognize that the handling of > > spaces/item sizing is weak (although it's not pronounced in this simple > > example). Do people want something else? > > > > Regards, > > Todd > > Your example doesn't demonstrate the problem I'm talking about because for integers the __str__ and __repr__ methods are identical. However, it does show a difference between Numeric and numarray, which would display your str(a) as > '[0 1 2 3 4 5 6 7 8 9 ]' > Well, almost what I wanted. This will change. > If you make an ObjectArray containing objects whose __repr__ and __str__ methods are not the same, I think you need a way of displaying the contained objects via their type-specific __str__ method. Numeric did this by default, whereas numarray has changed that behaviour. This will change too. > I'll go back to my example again. > In my example str() does not display the array elements in the same way that Numeric does. > In my case, numarray does this: > > >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) > [Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)] > > >>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) > [Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)] > > >>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) > ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) > > > ie. str() is not using the __str__ method I have defined for my Err class. > Numeric correctly uses my class's Err.__str__ method and does this instead: > > >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) > [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] > > In summary, I think te behaviour should be > >>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) > [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] > > >>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) > ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) > > I hope I've clarified what I'm getting at, although I probably haven't :-) Quite clear. Stuff will change. Regards, Todd -- Todd Miller From a.schmolck at gmx.net Mon Mar 29 04:55:44 2004 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Mon Mar 29 04:55:44 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug In-Reply-To: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> (Gary Ruben's message of "Mon, 29 Mar 2004 13:46:33 +1000") References: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> Message-ID: "Gary Ruben" writes: > In summary, I think te behaviour should be >>>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) > [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] > >>>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) > ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) Such behavior would seem sensible, but I don't think it's what the builtin collections do (not sure why): >>> n = 1.1 >>> n 1.1000000000000001 >>> [n] [1.1000000000000001] >>> str(n) '1.1' >>> str([n]) '[1.1000000000000001]' 'as From crasmussen at lanl.gov Mon Mar 29 08:30:01 2004 From: crasmussen at lanl.gov (Craig Rasmussen) Date: Mon Mar 29 08:30:01 2004 Subject: [Numpy-discussion] numarray C api examples In-Reply-To: <1080424143.3739.3.camel@localhost.localdomain> References: <1080424143.3739.3.camel@localhost.localdomain> Message-ID: On Mar 27, 2004, at 2:49 PM, Todd Miller wrote: > To install the examples, just install numarray. The examples are then > found in the package numarray.examples.convolve. If you want to write > your own extension, my advice is to stick with the numpy compatible > interfaces. A close second is the numarray "high level" interface. Does numarray have the equivalent of PyArray_FromDimsAndData()? This function is critical for the work I'm doing on Co-Array Python (because it is essentially a distributed array module with Co-Array syntax, it must use memory allocated by C) and for using Fortran memory for Numeric or numarray arrays. Regards, Craig From jmiller at stsci.edu Mon Mar 29 08:44:59 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Mar 29 08:44:59 2004 Subject: [Numpy-discussion] numarray C api examples In-Reply-To: References: <1080424143.3739.3.camel@localhost.localdomain> Message-ID: <1080578609.4087.114.camel@localhost.localdomain> On Mon, 2004-03-29 at 11:25, Craig Rasmussen wrote: > On Mar 27, 2004, at 2:49 PM, Todd Miller wrote: > > > To install the examples, just install numarray. The examples are then > > found in the package numarray.examples.convolve. If you want to write > > your own extension, my advice is to stick with the numpy compatible > > interfaces. A close second is the numarray "high level" interface. > > Does numarray have the equivalent of PyArray_FromDimsAndData()? numarray's Numeric compatibility layer has that function. Because numarray uses buffer objects, numarray winds up making a copy of the data. This behavior could be changed but hasn't been yet. Regards, Todd -- Todd Miller From faheem at email.unc.edu Mon Mar 29 11:47:28 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Mon Mar 29 11:47:28 2004 Subject: [Numpy-discussion] blitz++ and numarray Message-ID: Hi, I'm considering the use of the Blitz++ C++ library (http://www.oonumerics.org/blitz/) along with numarray/Numeric. I want to pass an array down to C++ and then manipulate it using Blitz++. My current thinking is to use the Numeric interface to pass down the array, and then copy it to a blitz++ array object. This seems straightforward except I am a little concerned about those reference counting issues, which I don't properly understand (I'm a Python beginner). I do get the impression that it is mainly an issue if I am working with Python objects in C/C++ code, though, and I am mostly going to try to avoid doing so by declaring C++ objects within the C++ code and working with those as much as possible. Other possibilities include somehow using the Boost.Python package, though I am not sure how. One advantage of this is that it apparently takes care of the reference counting issue. However, I don't want to learn yet another API. Also, weave.blitz() from SciPy looks relevant/useful, but this automatically generates its own C++ code, and I'd prefer to write my own. Can anyone suggested a good basic tutorial which discusses the issues of reference counting in a C/C++ API context? I have tried reading the official documentation and will continue to do so, but something gentler would be useful. Comments, suggestions gratefully appreciated. Please cc me on any reply to faheem at email.unc.edu. Thanks in advance. Faheem. From Fernando.Perez at colorado.edu Mon Mar 29 12:26:48 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 29 12:26:48 2004 Subject: [Numpy-discussion] blitz++ and numarray In-Reply-To: References: Message-ID: <406883BF.5080909@colorado.edu> Faheem Mitha wrote: > Hi, > > I'm considering the use of the Blitz++ C++ library > (http://www.oonumerics.org/blitz/) along with numarray/Numeric. I want > to pass an array down to C++ and then manipulate it using Blitz++. [snip] > Also, weave.blitz() from SciPy looks relevant/useful, but this > automatically generates its own C++ code, and I'd prefer to write my > own. The following should be enough to get you started: #include "Python.h" #include "Numeric/arrayobject.h" #include "blitz/array.h" using namespace std; using namespace blitz; // Convert a Numpy array to a blitz one, using the original's data (no copy) template static Array py_to_blitz(PyArrayObject* arr_obj) { int T_size = sizeof(T); TinyVector shape(0); TinyVector strides(0); int *arr_dimensions = arr_obj->dimensions; int *arr_strides = arr_obj->strides; for (int i=0;i((T*) arr_obj->data,shape,strides,neverDeleteData); } This is what I use for exactly the problem you are describing, and this code was pretty much lifted, with minor changes, from weave's auto-generated C++. What I do, to sidestep the memory management problems, is let python allocate all my Numeric arrays (using zeros() if I have no data). I then use those inside my C++ code as blitz++ arrays via the above snippet. Any changes made by the C++ code are automatically reflected in the Numeric arary, since the blitz object is using the Numeric data area. I hope this helps. Let me know if you need more help, I can mail you complete example code. Cheers, f From gazzar at email.com Tue Mar 30 01:16:11 2004 From: gazzar at email.com (Gary Ruben) Date: Tue Mar 30 01:16:11 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand apossible bug Message-ID: <20040330060633.34B361CE511@ws3-6.us4.outblaze.com> Thanks for pointing this out Alexander, I admit that I hadn't tried any native Python container types to see how they behave. I was basing my comments on Numeric's behaviour. When I have time, maybe I should raise a question on the Python developer's list. It seems like something that may have been overlooked. regards, Gary ----- Original Message ----- From: Alexander Schmolck Date: Mon, 29 Mar 2004 13:52:20 +0100 To: "numpy-discussion" Subject: Re: [Numpy-discussion] A couple more Numeric incompatibilitiesand apossible bug > "Gary Ruben" writes: > > In summary, I think te behaviour should be > >>>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) > > [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] > > > >>>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) > > ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) > > Such behavior would seem sensible, but I don't think it's what the builtin > collections do (not sure why): > > >>> n = 1.1 > >>> n > 1.1000000000000001 > >>> [n] > [1.1000000000000001] > >>> str(n) > '1.1' > >>> str([n]) > '[1.1000000000000001]' > > > 'as -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From Mailer-Daemon at kottonmouth.trouble-free.net Tue Mar 30 01:56:09 2004 From: Mailer-Daemon at kottonmouth.trouble-free.net (Mail Delivery System) Date: Tue Mar 30 01:56:09 2004 Subject: [Numpy-discussion] Mail delivery failed: returning message to sender Message-ID: This message was created automatically by mail delivery software. A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed: pete at shinners.org This message has been rejected because it has a potentially executable attachment "message_details.pif" This form of attachment has been used by recent viruses or other malware. If you meant to send this file then please package it up as a zip file and resend it. ------ This is a copy of the message, including all the headers. ------ From numpy-discussion at lists.sourceforge.net Tue Mar 30 04:55:56 2004 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Tue, 30 Mar 2004 11:55:56 +0200 Subject: Message Message-ID: Your document is attached. -------------- next part -------------- A non-text attachment was scrubbed... Name: message_details.pif Type: application/octet-stream Size: 17424 bytes Desc: not available URL: From cookedm at physics.mcmaster.ca Tue Mar 30 12:42:21 2004 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Mar 30 12:42:21 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug In-Reply-To: <1080561357.4087.18.camel@localhost.localdomain> References: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> <1080561357.4087.18.camel@localhost.localdomain> Message-ID: <20040330204004.GA4588@arbutus.physics.mcmaster.ca> On Mon, Mar 29, 2004 at 06:55:57AM -0500, Todd Miller wrote: > On Sun, 2004-03-28 at 22:46, Gary Ruben wrote: > > > > >>> cos(1) > > > > 0.54030227661132813 > > > > > > > > gives a different result to cos(1.) > > > > > > Do others think precision is being lost unnecessarily? > > > > > > No. Do you have any suggestions? > > > > Yes. My problem is that numarray is not replicating the behaviour of > > the Python math module, which presumably coerces immediately to the > > float precision of the underlying platform. This is probably a > > float64 in Windows. If you don't explicitly specify that float32 is > > to be used, shouldn't the default be for a rank-0 value to > > immediately coerce ints to the same precision as the native Python > > float type on the underlying platform, since you know it will be > > coerced to that later and in the meantime you've lost precision > > because you've applied a function to a value of lower precision? > > > > I see your point and I'll talk it over with Perry unless he chimes in on > his own. I'm ambivalent: while it would be nice for the numarray > ufuncs to act as full equivalents to the Python math functions in scalar > cases, numarray has to strike a balance between managing storage space > and maintaining precision and these two goals are in conflict. I think > using an array package you have to be at least a little more aware of > storage concerns. I'll throw in a vote for 'cos(1)' coercing 1 to a (equivalent-to-) Python float. The reason is that I have 'from numarray import *' for my interactive interpreter, as I use numarray a lot in that. I would rather not have to remember to always cast my args to ufuncs to arrays. This isn't a storage space problem, per se., the question is whether cos(1) and cos(1.0) should return the same answer. Having cos(1) cast 1 to a Float32 makes numarray much less useful as a calculator -- mistakes are more likely. I'll probably switch my interpreter back to Numeric so as not to make mistakes, or I'll write a wrapper (which I'll post if I do). Whether cos([1,1,1]) and cos([1.0,1.0,1.0]) are the same I'd say is different, as there I've taken the conscious decision not to cast to an array. (And the fact that cos([1,1,1]) will return an array printed out with 'type=Float32' is a clue to what I've done.) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From faheem at email.unc.edu Tue Mar 30 13:45:06 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Tue Mar 30 13:45:06 2004 Subject: [Numpy-discussion] blitz++ and numarray In-Reply-To: <406883BF.5080909@colorado.edu> References: <406883BF.5080909@colorado.edu> Message-ID: On Mon, 29 Mar 2004, Fernando Perez wrote: > The following should be enough to get you started: > > #include "Python.h" > #include "Numeric/arrayobject.h" > #include "blitz/array.h" > > using namespace std; > using namespace blitz; > > // Convert a Numpy array to a blitz one, using the original's data (no copy) > template > static Array py_to_blitz(PyArrayObject* arr_obj) > { > int T_size = sizeof(T); > TinyVector shape(0); > TinyVector strides(0); > int *arr_dimensions = arr_obj->dimensions; > int *arr_strides = arr_obj->strides; > > for (int i=0;i shape[i] = arr_dimensions[i]; > strides[i] = arr_strides[i]/T_size; > } > return Array((T*) arr_obj->data,shape,strides,neverDeleteData); > } > > This is what I use for exactly the problem you are describing, and this code > was pretty much lifted, with minor changes, from weave's auto-generated C++. > What I do, to sidestep the memory management problems, is let python allocate > all my Numeric arrays (using zeros() if I have no data). I then use those > inside my C++ code as blitz++ arrays via the above snippet. Any changes made > by the C++ code are automatically reflected in the Numeric arary, since the > blitz object is using the Numeric data area. > > I hope this helps. Let me know if you need more help, I can mail you complete > example code. Thanks. That is very helpful. I've incorporated this into my code, and managed to get a simple example with boost working (extracting an element of an array). See below. Do you have a similar function which converts a blitz array to a numarray/Numeric one? If not, I can easily concoct one for myself. I now need to try to figure out what this reference counting stuff is about. I read the official Python docs but am still currently very confused. Faheem. *************************************************************************** #include "Python.h" #include "Numeric/arrayobject.h" #include using namespace std; using namespace blitz; template static Array py_to_blitz(PyArrayObject* arr_obj); static PyObject * arrmod_elem(PyObject *self, PyObject *args); static PyMethodDef arrmod_methods[] = { {"elem", (PyCFunction)arrmod_elem, METH_VARARGS, "Returns the trace of a two-dimensional array.\n"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initarrmod(void) { (void) Py_InitModule3("arrmod", arrmod_methods, "Returns the Trace of a two-dimensional array.\n"); import_array(); } static PyObject * arrmod_elem(PyObject *self, PyObject *args) { PyObject *input, *result; PyArrayObject *array; double el; int i, j; if (!PyArg_ParseTuple(args, "Oii", &input, &i, &j)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; Array arr = py_to_blitz(array); el = arr(i,j); result = Py_BuildValue("d",el); return result; } // Convert a Numpy array to a blitz one, using the original's data (no // copy) template static Array py_to_blitz(PyArrayObject* arr_obj) { int T_size = sizeof(T); TinyVector shape(0); TinyVector strides(0); int *arr_dimensions = arr_obj->dimensions; int *arr_strides = arr_obj->strides; for (int i=0;i((T*) arr_obj->data,shape,strides,neverDeleteData); } From Fernando.Perez at colorado.edu Tue Mar 30 13:50:15 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Mar 30 13:50:15 2004 Subject: [Numpy-discussion] blitz++ and numarray In-Reply-To: References: <406883BF.5080909@colorado.edu> Message-ID: <4069EB78.2060609@colorado.edu> > Thanks. That is very helpful. I've incorporated this into my code, and > managed to get a simple example with boost working (extracting an element > of an array). See below. Glad it helped. > Do you have a similar function which converts a blitz array to a > numarray/Numeric one? If not, I can easily concoct one for myself. No, I don't have one. But it's pretty straightforward to write one, the Numeric C API provides functions to create arrays from existing data. You'll be surprised how conceptually similar Blitz and Numeric turn out to be (which is a good thing). > I now need to try to figure out what this reference counting stuff is > about. I read the official Python docs but am still currently very > confused. As I said, so far I've been able to bypass this issue by creating my arrays in python and 'filling' them in blitz. But this is not always a realistic possibility, so you may indeed have to learn the reference counting management details. Best, f From perry at stsci.edu Wed Mar 31 07:41:08 2004 From: perry at stsci.edu (Perry Greenfield) Date: Wed Mar 31 07:41:08 2004 Subject: [Numpy-discussion] type coercion question In-Reply-To: <20040330204004.GA4588@arbutus.physics.mcmaster.ca> Message-ID: David Cooke wrote: > On Mon, Mar 29, 2004 at 06:55:57AM -0500, Todd Miller wrote: > > On Sun, 2004-03-28 at 22:46, Gary Ruben wrote: > > > > > >>> cos(1) > > > > > 0.54030227661132813 > > > > > > > > > > gives a different result to cos(1.) > > > > > > > > Do others think precision is being lost unnecessarily? > > > > > > > > No. Do you have any suggestions? > > > > > > Yes. My problem is that numarray is not replicating the behaviour of > > > the Python math module, which presumably coerces immediately to the > > > float precision of the underlying platform. This is probably a > > > float64 in Windows. If you don't explicitly specify that float32 is > > > to be used, shouldn't the default be for a rank-0 value to > > > immediately coerce ints to the same precision as the native Python > > > float type on the underlying platform, since you know it will be > > > coerced to that later and in the meantime you've lost precision > > > because you've applied a function to a value of lower precision? > > > > > > > I see your point and I'll talk it over with Perry unless he chimes in on > > his own. I'm ambivalent: while it would be nice for the numarray > > ufuncs to act as full equivalents to the Python math functions in scalar > > cases, numarray has to strike a balance between managing storage space > > and maintaining precision and these two goals are in conflict. I think > > using an array package you have to be at least a little more aware of > > storage concerns. > > I'll throw in a vote for 'cos(1)' coercing 1 to a (equivalent-to-) > Python float. The reason is that I have 'from numarray import *' for my > interactive interpreter, as I use numarray a lot in that. > I would rather not have to remember to always cast my args to ufuncs to > arrays. This isn't a storage space problem, per se., the question is > whether cos(1) and cos(1.0) should return the same answer. > > Having cos(1) cast 1 to a Float32 makes numarray much less useful as a > calculator -- mistakes are more likely. I'll probably switch my > interpreter back to Numeric so as not to make mistakes, or I'll write a > wrapper (which I'll post if I do). > > Whether cos([1,1,1]) and cos([1.0,1.0,1.0]) are the same I'd say is > different, as there I've taken the conscious decision not to cast to an > array. (And the fact that cos([1,1,1]) will return an array printed out > with 'type=Float32' is a clue to what I've done.) > Let me say that I see the last issue (whether cos([1,1,1]) and cos([1.0,1.0,1.0]) are the same) is important. If cos([1,1,1])[0] != cos(1) I think that is just as bad. So the question is whether ints scalars and arrays should automatically be promoted to double floats when given to unary scientific functions (sqrt, sin, cos, log, etc.). By the way, if an Int32 is so promoted, I don't see why it isn't the same for Int8, Int16, etc. But it appears that Numeric promotes Int32 to Float64 for cos(), but promotes In16 to Float32 for cos(). I believe making this change to numarray is fairly easy (we'd have to check). But is what most people want? Should we retain complete compatibility with Numeric (which, as pointed out, treats different kinds of integers differently) Just to summarize what I see as acceptable choices: 1) ints promote to Float32 (current behavior) 2) ints promote to Float64 3) Int32 promotes to Float64, lesser ints promote to Float32 (Numeric behavior) Perry From Chris.Barker at noaa.gov Wed Mar 31 09:23:05 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Mar 31 09:23:05 2004 Subject: [Numpy-discussion] type coercion question In-Reply-To: References: Message-ID: <9D5D9B78-8338-11D8-9C14-000393A96660@noaa.gov> On Mar 31, 2004, at 7:39 AM, Perry Greenfield wrote: > Just to summarize what I see as acceptable choices: > > 1) ints promote to Float32 (current behavior) > 2) ints promote to Float64 > 3) Int32 promotes to Float64, lesser ints promote to Float32 (Numeric > behavior) +1 on (2) I want math.cos(x) to give the same answer as numarray.cos( array( (x) ) ) as much as possible. I like that Numarray does not do as much auto-upcasting as Numeric, but if I want to control the size of my data arrays, I'm not going to be passing an int into a floating point function. i.e, I can always do: cos( array( (x), Float32 ) ) Then I would want the result to be a Float32 -Chris Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rlw at stsci.edu Wed Mar 31 09:55:36 2004 From: rlw at stsci.edu (Rick White) Date: Wed Mar 31 09:55:36 2004 Subject: [Numpy-discussion] type coercion question In-Reply-To: Message-ID: On Wed, 31 Mar 2004, Perry Greenfield wrote: > Just to summarize what I see as acceptable choices: > > 1) ints promote to Float32 (current behavior) > 2) ints promote to Float64 > 3) Int32 promotes to Float64, lesser ints promote to Float32 (Numeric > behavior) I'm also +1 on option (2). I believe that this will not cause problems with unwanted promotions. The result of cos(1) is a scalar Python float, regardless of whether 1 is converted to a Float32 or a Float64 before the cosine is evaluated. Then if that scalar is used with a Float32 numarray, it gets converted to a Float32 and the result is Float32. For example: >>> from numarray import * >>> cos(1)*ones(1,Float32) array([ 0.54030228], type=Float32) >>> cos(1.0)*ones(1,Float32) array([ 0.54030228], type=Float32) So it seems to me there is no downside to converting integers to Float64 when the ufunc requires a float. And I find it hard to believe that any Numeric users are counting on behavior (3). Rick From Fernando.Perez at colorado.edu Wed Mar 31 09:58:05 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 31 09:58:05 2004 Subject: [Numpy-discussion] type coercion question In-Reply-To: <9D5D9B78-8338-11D8-9C14-000393A96660@noaa.gov> References: <9D5D9B78-8338-11D8-9C14-000393A96660@noaa.gov> Message-ID: <406B0684.9000600@colorado.edu> Chris Barker wrote: > On Mar 31, 2004, at 7:39 AM, Perry Greenfield wrote: > >>Just to summarize what I see as acceptable choices: >> >>1) ints promote to Float32 (current behavior) >>2) ints promote to Float64 >>3) Int32 promotes to Float64, lesser ints promote to Float32 (Numeric >>behavior) > > > +1 on (2) > > I want math.cos(x) to give the same answer as numarray.cos( array( (x) > ) ) as much as possible. I like that Numarray does not do as much > auto-upcasting as Numeric, but if I want to control the size of my data > arrays, I'm not going to be passing an int into a floating point > function. i.e, I can always do: > > cos( array( (x), Float32 ) ) > > Then I would want the result to be a Float32 +1 on (2) as well, based on a similar reasoning to Chris (who saved me the time of typing it up cleanly :) Cheers, f From cjw at sympatico.ca Wed Mar 31 12:11:19 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 31 12:11:19 2004 Subject: [Numpy-discussion] type coercion question In-Reply-To: References: Message-ID: <406B25C7.5020902@sympatico.ca> Rick White wrote: >On Wed, 31 Mar 2004, Perry Greenfield wrote: > > > >>Just to summarize what I see as acceptable choices: >> >>1) ints promote to Float32 (current behavior) >>2) ints promote to Float64 >>3) Int32 promotes to Float64, lesser ints promote to Float32 (Numeric >>behavior) >> >> > >I'm also +1 on option (2). > I'm also +1 on option (2). Colin W. From magnus at hetland.org Mon Mar 1 12:11:03 2004 From: magnus at hetland.org (Magnus Lie Hetland) Date: Mon Mar 1 12:11:03 2004 Subject: [Numpy-discussion] [Ann] Discretizing with Numarray Message-ID: <20040301200612.GA12281@idi.ntnu.no> I've just written a small tutorial on discretizing time series with numarray: http://hetland.org/python/disc-with-numarray Just some basic stuff I've used when working with time series in my research, such as mapping integer values to symbols, mapping real values to integers using bins, and extracting features using non-overlapping or overlapping windows (sliding window). It was written in a flash, and I'm no numarray expert, so feedback/corrections would be welcome. -- Magnus Lie Hetland "The mind is not a vessel to be filled, http://hetland.org but a fire to be lighted." [Plutarch] From falted at pytables.org Wed Mar 3 08:01:04 2004 From: falted at pytables.org (Francesc Alted) Date: Wed Mar 3 08:01:04 2004 Subject: [Numpy-discussion] [ANN] PyTables 0.8 Message-ID: <200403031654.40302.falted@pytables.org> I'm happy to announce the availability of PyTables 0.8. PyTables is a hierarchical database package designed to efficiently manage very large amounts of data. PyTables is built on top of the HDF5 library and the numarray package. It features an object-oriented interface that, combined with natural naming and C-code generated from Pyrex sources, makes it a fast, yet extremely easy-to-use tool for interactively saving and retrieving very large amounts of data. It also provides flexible indexed access on disk to anywhere in the data. PyTables is not designed to work as a relational database competitor, but rather as a teammate. If you want to work with large datasets of multidimensional data (for example, for multidimensional analysis), or just provide a categorized structure for some portions of your cluttered RDBS, then give PyTables a try. It works well for storing data from data acquisition systems (DAS), simulation software, network data monitoring systems (for example, traffic measurements of IP packets on routers), working with very large XML files or as a centralized repository for system logs, to name only a few possible uses. In this release you will find: - Variable Length Arrays (VLA's) for saving a collection of variable length of elements in each row of an array. - Extensible Arrays (EA's) for extending homogeneous datasets on disk. - Powerful replication capabilities, ranging from single leaves up to complete hierarchies. - With the introduction of the UnImplemented class, greatly improved HDF5 native file import capabilities. - Two new useful utilities: ptdump & ptrepack. - Improved documentation (with the help of Scott Prater). - New record on data size achieved: 5.6 TB (!) in one single file. - Enhanced platform support. New platforms: MacOSX, FreeBSD, Linux64, IRIX64 (yes, a clean 64-bit port is there) and probably more. - More tests units (now exceeding 800). - Many other minor improvements. More in detail: What's new ----------- - The new VLArray class enables you to store large lists of rows containing variable numbers of elements. The elements can be scalars or fully multimensional objects, in the PyTables tradition. This class supports two special objects as rows: Unicode strings (UTF-8 codification is used internally) and generic Python objects (through the use of cPickle). - The new EArray class allows you to enlarge already existing multidimensional homogeneous data objects. Consider it an extension of the already existing Array class, but with more functionality. Online compression or other filters can be applied to EArray instances, for example. Another nice feature of EA's is their support for fully multidimensional data selection with extended slices. You can write "earray[1,2:3,...,4:200]", for example, to get the desired dataset slice from the disk. This is implemented using the powerful selection capabilities of the HDF5 library, which results in very highly efficient I/O operations. The same functionality has been added to Array objects as well. - New UnImplemented class. If a dataset contains unsupported datatypes, it will be associated with an UnImplemented instance, then inserted into to the object tree as usual. This allows you to continue to work with supported objects while retaining access to attributes of unsupported datasets. This has changed from previous versions, where a RuntimeError occurred when an unsupported object was encountered. The combination of the new UnImplemented class with the support for new datatypes will enable PyTables to greatly increase the number of types of native HDF5 files that can be read and modified. - Boolean support has been added for all the Leaf objects. - The Table class has now an append() method that allows you to save large buffers of data in one go (i.e. bypassing the Row accessor). This can greatly improve data gathering speed. - The standard HDF5 shuffle filter (to further enhance the compression level) is supported. - The standard HDF5 fletcher32 checksum filter is supported. - As the supported number of filters is growing (and may be further increased in the future), a Filters() class has been introduced to handle filters more easily. In order to add support for this class, it was necessary to make a change in the createTable() method that is not backwards compatible: the "compress" and "complib" parameters are deprecated now and the "filters" parameter should be used in their place. You will be able to continue using the old parameters (only a Deprecation warning will be issued) for the next few releases, but you should migrate to the new version as soon as possible. In general, you can easily migrate old code by substituting code in its place: ???????????????table = fileh.createTable(group, 'table', Test, '', ?????????????????????????????????????????complevel, complib) ?should be replaced by ???????????????table = fileh.createTable(group, 'table', Test, '', ?????????????????????????????????????????Filters(complevel, complib)) - A copy() method that supports slicing and modification of ?????????filtering capabilities has been added for all the Leaf ?????????objects. See the User's Manual for more information. - A couple of new methods, namely copyFile() and copyChilds(), ?????????have been added to File class, to permit easy replication ?????????of complete hierarchies or sub-hierarchies, even to ?????????other files. You can change filters during the copy ?????????process as well. - Two new utilities has been added: ptdump and ?????????ptrepack. The utility ptdump allows the user to examine the?contents of PyTables files (both metadata and actual ?????????data). The powerful ptrepack utility lets you ?????????selectively copy (portions of) hierarchies to specific ?????????locations in other files. It can be also used as an ?????????importer for generic HDF5 files. ???????- The meaning of the stop parameter in read() methods has ?????????changed. Now a value of 'None' means the last row, and a ?????????value of 0 (zero) means the first row. This is more ?????????consistent with the range() function in python and the ?????????__getitem__() special method in numarray. - The method Table.removeRows() is no longer limited by table size. You can now delete rows regardless of the size of the table. - The "numarray" value has been added to the flavor parameter ?????????in the Table.read() method for completeness. - The attributes (.attr instance variable) are Python ?????????properties now. Access to their values is no longer ?????????lazy, i.e. you will be able to see both system or user ?????????attributes from the command line using the tab-completion ?????????capability of your python console (if enabled). - Documentation has been greatly improved to explain all the ?????????new functionality. In particular, the internal format of ?????????PyTables is now fully described. You can now build ?????????"native" PyTables files using any generic HDF5?software by just duplicating their format. - Many new tests have been added, not only to check new ?????????functionality but also to more stringently check ?????????existing functionality. There are more than 800 different ?????????tests now (and the number is increasing :). - PyTables has a new record in the data size that fits in one single file: more than 5 TB (yeah, more than 5000 GB), that accounts for 11 GB compressed, has been created on an AMD Opteron machine running Linux-64 (the 64 bits version of the Linux kernel). See the gory details in: http://pytables.sf.net/html/HowFast.html. - New platforms supported: PyTables has been compiled and tested under Linux32 (Intel), Linux64 (AMD Opteron and Alpha), Win32 (Intel), MacOSX (PowerPC), FreeBSD (Intel), Solaris (6, 7, 8 and 9 with UltraSparc), IRIX64 (IRIX 6.5 with R12000) and it probably works in many more architectures. In particular, release 0.8 is the first one that provides a relatively clean porting to 64-bit platforms. - As always, some bugs have been solved (especially bugs that ?????????occur when deleting and/or overwriting attributes). - And last, but definitely not least, a new donations section has been?added to the PyTables web site (http://sourceforge.net/projects/pytables, then follow the "Donations" tag). If you like PyTables and want this effort to continue, please, donate! What is a table? ---------------- A table is defined as a collection of records whose values are stored in fixed-length fields. All records have the same structure and all values in each field have the same data type. ?The terms "fixed-length" and "strict data types" seem to be quite a strange requirement for an language like Python that supports dynamic data types, but they serve a useful function if the goal is to save very large quantities of data (such as is generated by many scientific applications, for example) in an efficient manner that reduces demand on CPU time and I/O resources. What is HDF5? ------------- For those people who know nothing about HDF5, it is is a general purpose library and file format for storing scientific data made at NCSA. HDF5 can store two primary objects: datasets and groups. A dataset is essentially a multidimensional array of data elements, and a group is a structure for organizing objects in an HDF5 file. Using these two basic constructs, one can create and store almost any kind of scientific data structure, such as images, arrays of vectors, and structured and unstructured grids. You can also mix and match them in HDF5 files according to your needs. Platforms --------- I'm using Linux (Intel 32-bit) as the main development platform, but PyTables should be easy to compile/install on many other UNIX machines. This package has also passed all the tests on a UltraSparc platform with Solaris 7 and Solaris 8. It also compiles and passes all the tests on a SGI Origin2000 with MIPS R12000 processors, with the MIPSPro compiler and running IRIX 6.5. It also runs fine on Linux 64-bit platforms, like an AMD Opteron running SuSe Linux Enterprise Server. It has also been tested in MacOSX platforms (10.2 but should also work on newer versions). Regarding Windows platforms, PyTables has been tested with Windows 2000 and Windows XP (using the Microsoft Visual C compiler), but it should also work with other flavors as well. An example? ----------- For online code examples, have a look at http://pytables.sourceforge.net/html/tut/tutorial1-1.html and, for newly introduced Variable Length Arrays: http://pytables.sourceforge.net/html/tut/vlarray2.html Web site -------- Go to the PyTables web site for more details: http://pytables.sourceforge.net/ Share your experience --------------------- Let me know of any bugs, suggestions, gripes, kudos, etc. you may have. Have fun! -- Francesc Alted falted at pytables.org -- Francesc Alted From haase at msg.ucsf.edu Tue Mar 9 10:10:05 2004 From: haase at msg.ucsf.edu (Sebastian Haase) Date: Tue Mar 9 10:10:05 2004 Subject: [Numpy-discussion] numarray: shape of scalar Message-ID: <200403091000.39148.haase@msg.ucsf.edu> Hi, Is this intended: >>> na.array([1,2,4]).shape (3,) >>> na.array([1,2]).shape (2,) >>> na.array([1]).shape (1,) >>> na.array(1).shape () Why is na.array([1]).shape not equal to na.array(1).shape ? Did I miss the respective section in the manual ? Thanks, Sebastian Haase From jmiller at stsci.edu Tue Mar 9 10:29:21 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Mar 9 10:29:21 2004 Subject: [Numpy-discussion] numarray: shape of scalar In-Reply-To: <200403091000.39148.haase@msg.ucsf.edu> References: <200403091000.39148.haase@msg.ucsf.edu> Message-ID: <1078856368.12039.10.camel@halloween.stsci.edu> On Tue, 2004-03-09 at 13:00, Sebastian Haase wrote: > Hi, > Is this intended: Yes. > >>> na.array([1,2,4]).shape > (3,) > >>> na.array([1,2]).shape > (2,) > >>> na.array([1]).shape > (1,) > >>> na.array(1).shape > () > > Why is na.array([1]).shape not equal to na.array(1).shape ? na.array(1) is how you say "make a rank-0 array with contents 1". There are places where it is natural and useful in the *implementation* of numarray. > Did I miss the respective section in the manual ? rank-0 arrays are a dark corner of numarray which is mostly hidden. I don't think they're documented anywhere. Regards, Todd -- Todd Miller From a.schmolck at gmx.net Wed Mar 10 10:23:38 2004 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Wed Mar 10 10:23:38 2004 Subject: [Numpy-discussion] numarray: shape of scalar In-Reply-To: <200403091000.39148.haase@msg.ucsf.edu> References: <200403091000.39148.haase@msg.ucsf.edu> Message-ID: Sebastian Haase writes: > Hi, > Is this intended: > >>> na.array([1,2,4]).shape > (3,) > >>> na.array([1,2]).shape > (2,) > >>> na.array([1]).shape > (1,) > >>> na.array(1).shape > () > > Why is na.array([1]).shape not equal to na.array(1).shape ? It is maybe a bit like n^0 == 1 (rather than 0, which might seem "more intuitive"). A scalar is not the same as vector of 1 element and neither are the same as a matrix of 1 element (nonwithstanding what matlab and maybe some dumb math texts might suggest). If you pretend they are you run into all sorts of trouble, as matlab (which doesn't distinguish properly between scalars, vectors and matrices) indeed does, for one thing because various useful identities suddenly no longer hold for corner cases (e.g. since functions like diag that do different things depending on whether they "think" something is a vector or not just do the wrong thing on what you intend to be a 1xN or Nx1 matrix). Here is a not particularly interesting identity that shows why you would like shape(scalar) to be (): rank == len(shape) rank([x]) == rank(x) + 1 Here is a more interesting example that shows why you want to distinguish between arrays of different rank (even if their shape only differes by a leading 1 and the flat is the same): rank(a[integer]) = rank(a) - 1 The pleasant indexing facilities of numarray wouldn't be possible without distinguishing between arrays which differ only by the amount of leading 1s in their shape. 'as From berumthyer at vietmedia.com Thu Mar 11 01:31:03 2004 From: berumthyer at vietmedia.com (Derrick Meyer) Date: Thu Mar 11 01:31:03 2004 Subject: [Numpy-discussion] STOCK-PROFILE - Breaking News on ACQUISITION, Wed, 10 Mar 2004 20:04:22 -0500 Message-ID: Wed, 10 Mar 2004 20:04:22 -0500 Stock-Profile - NEW ISSUE: SEVI - Systems Evolution Incorporated Breaking NEWS: Systems Evolution Acquires AXP Technologies STAFFORD, Texas -- Systems Evolution, Inc., (OTC BB:SEVI.OB - News), an information technology services company, announces that it has acquired AXP Technologies, Inc., a Houston based computer and network support company. This acquisition accelerates Systems Evolution's move into network security and computer support services for small to mid-size businesses (SMBs). The SMB Solutions Practice will be led by AXP Technologies founders, Willie A. Jackson and Ryan L. Sumstad as Director and Managing Consultant, respectively. AXP Technolgies designed and implemented computer systems and network security throughout organizations, maintaining and supporting them at a fraction of the traditional costs. Disclosure: Market-Torque (M-T) is not a registered financial advisory. The information presented by M-T is not for purchasing or selling securities. M-T compiles then distributes opinions, comments and information based on other public sources. Penny stocks are considered to be highly speculative and may be unsuitable for all but very aggressive investors. M-T does not hold nor does it plan to hold a position in this stock. This Profile was a paid advertisement by a third party not affiliated with the profiled company. M-T was compensated four thousand dollars to publish and distribute this report. Paid advertisements for a third party do not necessarily reflect the views of M-T. Target prices may fluctuate depending on market environments. Please always consult a registered financial advisor before making any decisions. This report is for entertainment and advertising purposes only and should not be used as investment advice. Copyright 2004, Roytico Ltd. This report distributed by MMS. Apartado 173-3006 Zona Franca MeoBarreal Heredia, Costa Rica. No more ads, cathisterba at freemail.lt From ariciputi at pito.com Thu Mar 11 04:18:13 2004 From: ariciputi at pito.com (Andrea Riciputi) Date: Thu Mar 11 04:18:13 2004 Subject: [Numpy-discussion] A simple question about type(). Message-ID: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> Hi, it is surely a silly question, but I'm quite new to Python and Numpy. So I hope you can help me, I've searched Numpy's manual without successs. Suppose I want to test if a paremeter inside a function is an array or not. How can I test this? type(my_array) is array returns 'False'!! Thanks in advance, Andrea. --- Andrea Riciputi "Science is like sex: sometimes something useful comes out, but that is not the reason we are doing it" -- (Richard Feynman) From rsilva at ime.usp.br Thu Mar 11 04:58:38 2004 From: rsilva at ime.usp.br (Paulo J. S. Silva) Date: Thu Mar 11 04:58:38 2004 Subject: [Numpy-discussion] A simple question about type(). In-Reply-To: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> References: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> Message-ID: <1079008947.11736.8.camel@localhost> Hi, Funny, I have just entered the list and I can answer this one! Try type(my_array) is arraytype That worked to me. Best, Paulo -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call) (practice) From ariciputi at pito.com Thu Mar 11 05:37:10 2004 From: ariciputi at pito.com (Andrea Riciputi) Date: Thu Mar 11 05:37:10 2004 Subject: [Numpy-discussion] A simple question about type(). In-Reply-To: <1079008947.11736.8.camel@localhost> References: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> <1079008947.11736.8.camel@localhost> Message-ID: <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> You are right! But I can't understand why. When I try type() with an array I get: type(my_array) but: type(my_array) is array False Can someone explain this? Cheers, Andrea. On 11 Mar 2004, at 13:42, Paulo J. S. Silva wrote: > Hi, > > Funny, I have just entered the list and I can answer this one! > > Try > > type(my_array) is arraytype > > That worked to me. > > Best, > > Paulo > -- > Paulo Jos? da Silva e Silva > Professor Assistente do Dep. de Ci?ncia da Computa??o > (Assistant Professor of the Computer Science Dept.) > Universidade de S?o Paulo - Brazil > > e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva > > Teoria ? o que n?o entendemos o (Theory is something we don't) > suficiente para chamar de pr?tica. (understand well enough to call) > (practice) > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > --- Andrea Riciputi "Science is like sex: sometimes something useful comes out, but that is not the reason we are doing it" -- (Richard Feynman) From Alexandre.Fayolle at logilab.fr Thu Mar 11 05:55:05 2004 From: Alexandre.Fayolle at logilab.fr (Alexandre) Date: Thu Mar 11 05:55:05 2004 Subject: [Numpy-discussion] A simple question about type(). In-Reply-To: <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> References: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> <1079008947.11736.8.camel@localhost> <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> Message-ID: <20040311134339.GA10008@crater.logilab.fr> On Thu, Mar 11, 2004 at 02:25:53PM +0100, Andrea Riciputi wrote: > You are right! But I can't understand why. When I try type() with an > array I get: > > type(my_array) > > > but: > > type(my_array) is array > False > > Can someone explain this? array is a function. try "type(array)", and print(array) to check. -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From rsilva at ime.usp.br Thu Mar 11 06:19:10 2004 From: rsilva at ime.usp.br (Paulo J. S. Silva) Date: Thu Mar 11 06:19:10 2004 Subject: [Numpy-discussion] A simple question about type(). In-Reply-To: <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> References: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> <1079008947.11736.8.camel@localhost> <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> Message-ID: <1079014074.11736.26.camel@localhost> This is a more tricky question. If I understand well enough. The name "array" in a python section relates to a factory function that builds arraytype objects. It is not the name for the array type. For example, if you type: type(array) you get Which shows that it is a function, not a class object. There are more subtlety associated to the fact that Numeric does not use the new class styles as numarray does. Actually if you use numarray instead of Numeric you'll get a more sensible behavior >>> import numarray >>> a = numarray.array([1.,2.,3.]) >>> type(a) >>> type(a) is numarray.NumArray True >>> isinstance(a, numarray.NumArray) True >>> Best regards, Paulo Obs: You may consider using a "try: except:" statement instead of always checking type. In Python is usually better to ask "forgiveness" in a "except" clause than to ask "permission" by checking types. -- Paulo Jos? da Silva e Silva Professor Assistente do Dep. de Ci?ncia da Computa??o (Assistant Professor of the Computer Science Dept.) Universidade de S?o Paulo - Brazil e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva Teoria ? o que n?o entendemos o (Theory is something we don't) suficiente para chamar de pr?tica. (understand well enough to call) (practice) From ferrell at lanl.gov Thu Mar 11 07:23:09 2004 From: ferrell at lanl.gov (ferrell at lanl.gov) Date: Thu Mar 11 07:23:09 2004 Subject: [Numpy-discussion] A simple question about type(). In-Reply-To: <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> References: <8B98070B-7354-11D8-A8F5-000393933E4E@pito.com> <1079008947.11736.8.camel@localhost> <9F281D20-735F-11D8-A8F5-000393933E4E@pito.com> Message-ID: <16464.33169.464000.954991@gargle.gargle.HOWL> As others have pointed out, array is a function. You want ArrayType. > type(my_array) is ArrayType > True -r Andrea Riciputi writes: > You are right! But I can't understand why. When I try type() with an=20 > array I get: > > type(my_array) > > > but: > > type(my_array) is array > False > > Can someone explain this? > > Cheers, > Andrea. > > On 11 Mar 2004, at 13:42, Paulo J. S. Silva wrote: > > > Hi, > > > > Funny, I have just entered the list and I can answer this one! > > > > Try > > > > type(my_array) is arraytype > > > > That worked to me. > > > > Best, > > > > Paulo > > --=20 > > Paulo Jos=E9 da Silva e Silva > > Professor Assistente do Dep. de Ci=EAncia da Computa=E7=E3o > > (Assistant Professor of the Computer Science Dept.) > > Universidade de S=E3o Paulo - Brazil > > > > e-mail: rsilva at ime.usp.br Web: http://www.ime.usp.br/~rsilva > > > > Teoria =E9 o que n=E3o entendemos o (Theory is something we don't) > > suficiente para chamar de pr=E1tica. (understand well enough to call) > > (practice) > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by: IBM Linux Tutorials > > Free Linux tutorial presented by Daniel Robbins, President and CEO of > > GenToo technologies. Learn everything from fundamentals to system > > administration.http://ads.osdn.com/?ad_id=1470&alloc_id638&op=3Dclick > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > > --- > Andrea Riciputi > > "Science is like sex: sometimes something useful comes out, > but that is not the reason we are doing it" -- (Richard Feynman) > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From jmiller at stsci.edu Sat Mar 13 05:12:36 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Mar 13 05:12:36 2004 Subject: [Numpy-discussion] ANN: numarray-0.9 Message-ID: <1079179547.3739.0.camel@localhost.localdomain> Release Notes for numarray-0.9 Numarray is an array processing package designed to efficiently manipulate large multi-dimensional arrays. Numarray is modelled after Numeric and features c-code generated from python template scripts, the capacity to operate directly on arrays in files, and improved type promotions. I. ENHANCEMENTS 1. Support for "from __future__ import division" True division has been implemented for numarray. This means that modules that wish to use true division can also use numarray and numarray division will work as follows: a. dividing any two integer arrays using "/" results in a Float32 array. b. dividing two floating point arrays using "//" results in truncation of the result as in: a // b == floor(a/b). 2. C-coded array slicing Array slicing has been re-implemented in C-code as part of the _ndarray module. This means faster slicing. Thanks go to Warren Hack, Chris Hanley, and Ivo Busko for helping debug a huge refcount error. 3. Decreased Ufunc overhead Ufunc execution speed has clawed and scratched its way back to where it was around numarray-0.5. Improvements here included optimization of the ufunc caching, smarter thread handling, and smarter support for subclasses. The ufunc caching is based on a simple 20 element table for each ufunc. 4. Faster array creation from C Code which creates NumArrays from C (including numarray itself) can now do so faster because the API functions have been modified to do the array __init__ inline rather than through an expensive Python callback. II. BUGS FIXED / CLOSED See http://sourceforge.net/tracker/?atid=450446&group_id=1369&func=browse for more details. 913781 Another memory leak in example in Chapter 12 908399 Numarray 0.7: "del a[1]" dumps core 899259 astype Int16 copy4bytes: access beyond buffer 895801 Buffer overflow in sum w/ 0-sized array 894810 MemoryError When Creating Large Arrays 890703 getnan() and getinf() failure 883124 and and operator.and respond differently 865410 Usage of __dict__ 854480 Slice assignment of float to integer 839367 Overlapping slice assign fails 828941 Numarray: determinant returns scalar or array 820122 Linearalgebra2.determinant problem 817343 Sub-classing of NumArray inhibited by complex values 793336 crash in _sort.pyd 772548 Reference counting errors 683957 Adding certain arrays fails in Numarray III. CAUTIONS 1. numarray extension writers should note that the documented use of PyArray_INCREF and PyArray_XDECREF (in numarray) has been found to be incompatible with Numeric and has therefore been deprecated. numarray wrapper functions using PyArray_INCREF and PyArray_XDECREF should switch to ordinary Py_INCREF and Py_XDECREF. WHERE ----------- Numarray-0.9 windows executable installers, source code, and manual is here: http://sourceforge.net/project/showfiles.php?group_id=1369 Numarray is hosted by Source Forge in the same project which hosts Numeric: http://sourceforge.net/projects/numpy/ The web page for Numarray information is at: http://stsdas.stsci.edu/numarray/index.html Trackers for Numarray Bugs, Feature Requests, Support, and Patches are at the Source Forge project for NumPy at: http://sourceforge.net/tracker/?group_id=1369 REQUIREMENTS ------------------------------ numarray-0.9 requires Python 2.2.2 or greater. AUTHORS, LICENSE ------------------------------ Numarray was written by Perry Greenfield, Rick White, Todd Miller, JC Hsu, Paul Barrett, Phil Hodge at the Space Telescope Science Institute. We'd like to acknowledge the assitance of Francesc Alted, Paul Dubois, Sebastian Haase, Tim Hochberg, Nadav Horesh, Edward C. Jones, Eric Jones, Jochen K?pper, Travis Oliphant, Pearu Peterson, Peter Verveer, Colin Williams, and everyone else who has contributed with comments and feedback. Numarray is made available under a BSD-style License. See LICENSE.txt in the source distribution for details. -- Todd Miller jmiller at stsci.edu From hoagiaonguyen at yahoo.com Sat Mar 13 10:55:17 2004 From: hoagiaonguyen at yahoo.com (Hoa Nguyen) Date: Sat Mar 13 10:55:17 2004 Subject: [Numpy-discussion] Installing Numpy Message-ID: <20040313185420.84768.qmail@web40712.mail.yahoo.com> Hi, Please help me with installing NumPy. When I run: python setup.py install there was error. It said that: ImportError: No module named distutils I'm using Mandrake 9. Thanks, Hoa Nguyen __________________________________ Do you Yahoo!? Yahoo! Mail - More reliable, more storage, less spam http://mail.yahoo.com From ivanov at rbc.ru Sat Mar 13 17:15:01 2004 From: ivanov at rbc.ru (ivanov at rbc.ru) Date: Sat Mar 13 17:15:01 2004 Subject: [Numpy-discussion] www.cvv2.ru - would like to thank you for your purchase on our web site Message-ID: You've just purchased set of Maibach brand earthenware on web site cvv2.ru Easy to use, Maibach kitchenware is also famous for its modern look. Our utensils, designed for easy and fast cooking of a variety of foods, will lower your energy consumption rate and save your time and money. If you are looking for a reliable and trouble-free kitchen help, you have just found it. All Maibach kitchenware is made of stainless steel - a proven chrome and nickel 18/10 alloy. Easy to use and famous for its modern look, Maibach kitchenware allows you to simultaneously cook several dishes on one burner. A heat-absorbing base, built for optimal transmission of thermal energy from the burner to the cookware, is the most important feature of Maibach utensils. The exclusive Platinum model uses a triple heat-absorbing base, guaranteeing a still better protection of food from burning. Even and smooth heating of the utensil's base and sides prevents food from burning. NEW!!! We are glad to offer to our customers the most generous reward program: every customer who spent with us more than USD 249.99 or Euro 199.99, automatically qualified for bonus. Choose your bonus: 1. Sony VHS cassette with 240 minutes of best underage porno you ever see. (NTSC and Secam both are available) 2. Bestselling manual "How to create plastic bomb in home" and "How to hijack a train or an aircraft, with color pictures and FAQ" To cancell order:val at rbc.ru Bulk orders:alexk1 at rbc.ru Refund Service ICQ#307004 All Maibach products are covered by lifetime warranty. From faheem at email.unc.edu Sat Mar 13 22:21:02 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sat Mar 13 22:21:02 2004 Subject: [Numpy-discussion] numarray.objects missing in documentation Message-ID: Dear People, I'm posting this to gmane.comp.python.numeric.general. I am not sure what mailing list this corresponds to. I earlier tried to post this to the mailing list mentioned in the numarray documentation (numpy at lists.sourceforge.net), but it bounced. I don't see numarray.objects listed in http://stsdas.stsci.edu/numarray/numarray-0.9.html/genindex.html I can't find it anywhere else in the documentation either. I could not find a single page html version of the docuemtation, otherwise I would have done a search. I do find nice documentation when doing >>> help("numarray.objects"). However there is a minor typo at the beginning. "...It defines functions which correspond to most of the most of the operators defined in Python's operator module, and provides names compatible with most of numarray's universal functions." "most of" is repeated. Please cc me on any reply; I'm not subscribed. Thanks. Faheem. From nadavh at visionsense.com Sun Mar 14 06:05:01 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Sun Mar 14 06:05:01 2004 Subject: [Numpy-discussion] Installing Numpy Message-ID: <07C6A61102C94148B8104D42DE95F7E86DECB2@exchange2k.envision.co.il> Try to get distutils either from rpmfind or your distribution CD (downloading and installing the latest python is a good option). It is highly recommended to have distutils if you want to install any add-on package to python. Nadav. -----Original Message----- From: Hoa Nguyen [mailto:hoagiaonguyen at yahoo.com] Sent: Sat 13-Mar-04 20:54 To: numpy-discussion at lists.sourceforge.net Cc: Subject: [Numpy-discussion] Installing Numpy Hi, Please help me with installing NumPy. When I run: python setup.py install there was error. It said that: ImportError: No module named distutils I'm using Mandrake 9. Thanks, Hoa Nguyen __________________________________ Do you Yahoo!? Yahoo! Mail - More reliable, more storage, less spam http://mail.yahoo.com ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From jmiller at stsci.edu Sun Mar 14 06:57:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Mar 14 06:57:01 2004 Subject: [Numpy-discussion] numarray.objects missing in documentation In-Reply-To: References: Message-ID: <1079276152.13290.9.camel@localhost.localdomain> On Sun, 2004-03-14 at 00:26, Faheem Mitha wrote: > Dear People, > > I'm posting this to gmane.comp.python.numeric.general. I am not sure > what mailing list this corresponds to. I earlier tried to post this to > the mailing list mentioned in the numarray documentation > (numpy at lists.sourceforge.net), but it bounced. > > I don't see numarray.objects listed in > http://stsdas.stsci.edu/numarray/numarray-0.9.html/genindex.html The section on object arrays has yet to be written. What is it that you want to know? > > I can't find it anywhere else in the documentation either. I could not > find a single page html version of the docuemtation, otherwise I would > have done a search. > > I do find nice documentation when doing > >>> help("numarray.objects"). > > However there is a minor typo at the beginning. > > "...It defines functions which correspond to most of the most of the > operators defined in Python's operator module, and provides names > compatible with most of numarray's universal functions." > > "most of" is repeated. Thanks. This is fixed now in CVS. > > Please cc me on any reply; I'm not subscribed. Thanks. > > Faheem. Regards, Todd -- Todd Miller From faheem at email.unc.edu Sun Mar 14 09:21:02 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sun Mar 14 09:21:02 2004 Subject: [Numpy-discussion] Re: numarray.objects missing in documentation References: <1079276152.13290.9.camel@localhost.localdomain> Message-ID: On Sun, 14 Mar 2004 09:55:52 -0500, Todd Miller wrote: > On Sun, 2004-03-14 at 00:26, Faheem Mitha wrote: >> Dear People, >> >> I'm posting this to gmane.comp.python.numeric.general. I am not sure >> what mailing list this corresponds to. I earlier tried to post this to >> the mailing list mentioned in the numarray documentation >> (numpy at lists.sourceforge.net), but it bounced. >> >> I don't see numarray.objects listed in >> http://stsdas.stsci.edu/numarray/numarray-0.9.html/genindex.html > > The section on object arrays has yet to be written. What is it that you > want to know? Thanks for the quick reply. I'm trying to make an array of python objects as follows. Specifically, given a user defined class, I want to creata an array of instances of that class. Consider the following class. class cell: def setrow(self,row): self.row = row def setcol(self,col): self.col = col ... I create an "empty" cell. empcell = cell() Now I want to create an array of objects identical to empcell, and later initialize them. I tried stuff along the lines of ****************************************************************** import string from numarray import * empcell = cell() base = [empcell] mat = fromlist(base,shape=(2,2)) ****************************************************************** But the mat = fromlist(base,shape=(2,2)) line fails with AttributeError: cell instance has no attribute '__len__' Similar things fail with the same error message. No doubt I'm doing something stupid, but would someone please enlighten me? I got the fromlist command from >>> help("numarray.objects") I was a bit disconcerted to find on trying to run the first example from the help page that I got >>> a = fromlist(["t","u","w"]) ... TypeError: Expecting a python numeric type, got something else. What am I doing wrong? I'm running ii python 2.3.3-5 An interactive high-level object-oriented language (default versio ii python-numarray 0.8.1-1 An array processing package modelled after Python-Numeric ii python-numeric 23.1-1 Numerical (matrix-oriented) Mathematics for Python on Debian sarge. Thanks in advance and sorry for the long-winded message. Faheem. From sdhyok at email.unc.edu Sun Mar 14 19:51:01 2004 From: sdhyok at email.unc.edu (Shin) Date: Sun Mar 14 19:51:01 2004 Subject: [Numpy-discussion] map for numarray and automatic calls of elements? Message-ID: <1079322556.2570.28.camel@dhcp14957.odumvillage.unc.edu> In my experience, when using Numeric (numpy), the following two features turned out to be very useful in many numerical operations. Please let me know if numarray already supports the features. Any comments are welcome. 1. A function like map but it returns an array directly, not a list. For instance, if we call the function as xmap, >> f = lambda x: x+1 >> map(f, [1,2,3]) [2,3,4] >> xmap(f, [1,2,3]) # == array(map(f,[1,2,3])) array([2,3,4]) When dealing with a large volume of arrays, this function can save memory and improve speed significantly by skipping the creation of a list. 2. A way to call methods or attributes of object elements from an object array. For instance, >> from datetime import date >> x = array([date(1990,1,1), date(2000,1,1)]) >> x.year array([1990, 2000]) If an object array is called by a methods or an attribute, which the array doesn't have, the array calls the methods or the attributes of its elements automatically and returns a collection of the results. Thanks. -- Daehyok Shin Geography Department University of North Carolina, Chapel Hill email: sdhyok at email.unc.edu From falted at pytables.org Mon Mar 15 05:24:04 2004 From: falted at pytables.org (Francesc Alted) Date: Mon Mar 15 05:24:04 2004 Subject: [Numpy-discussion] PyTables & numarray 0.9 warning Message-ID: <200403151418.41664.falted@pytables.org> Hi, Many of you will know that a new version of numarray (0.9) has been released past week. This new version has a number of cool features (specially, being faster in certain situations that affect directly to PyTables efficency :-). Unfortunately, the new version of numarray has deprecated the "buffer" keyword on the array() constructor, and that precise keyword was used in PyTables (just in one line). I've uploaded a new version of PyTables in the SourceForge repository with a cure on that. Please, if you have downloaded PyTables *before* March, 9th, download again from the PyTables web site (http://www.pytables.org) and rebuild the software (or install the new autoinstallable binaries for Windows). If you don't feel like having to do that, you can just apply this patch to get rid of the problem: -8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<- --- tables/Array.py.orig 2004-03-15 13:56:01.000000000 +0100 +++ tables/Array.py 2004-03-15 13:56:13.000000000 +0100 @@ -483,7 +483,7 @@ if repr(self.type) == "CharType": arr = strings.array(None, itemsize=self.itemsize, shape=shape) else: - arr = numarray.array(buffer=None, type=self.type, shape=shape) + arr = numarray.array(None, type=self.type, shape=shape) # Set the same byteorder than on-disk arr._byteorder = self.byteorder # Protection against reading empty arrays -8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<--8<- My apologies for the inconveniences, -- Francesc Alted From jmiller at stsci.edu Mon Mar 15 07:53:09 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Mar 15 07:53:09 2004 Subject: [Numpy-discussion] Re: numarray.objects missing in documentation References: <1079276152.13290.9.camel@localhost.localdomain> Message-ID: <006d01c40aa5$8591e830$ebeca782@stsci.edu> Hi Faheem, In order to create an object array, do this: >>> import numarary.objects as obj >>> a = obj.fromlist(["t","u","w",1]) >>> a ObjectArray(["t","u","w",1) Unfortunately, the ability to apply arbitrary functions to all the elements of an object array has never been implemented. Nevertheless, the following might do: def apply(array, function, result=None): rank1 = (array.rank == 1) if result is None: result0 = array.copy() else: result0 = result for x in range(array.shape[0]): if rank1: result0[x] = function(array[x]) else: apply(array[x], function, result0[x]) return result0 Re-ordering the loop and the if statement results in faster but slightly more verbose code. Regards, Todd Miller ----- Original Message ----- From: "Faheem Mitha" To: Sent: Sunday, March 14, 2004 12:20 PM Subject: [Numpy-discussion] Re: numarray.objects missing in documentation > On Sun, 14 Mar 2004 09:55:52 -0500, Todd Miller wrote: > > On Sun, 2004-03-14 at 00:26, Faheem Mitha wrote: > >> Dear People, > >> > >> I'm posting this to gmane.comp.python.numeric.general. I am not sure > >> what mailing list this corresponds to. I earlier tried to post this to > >> the mailing list mentioned in the numarray documentation > >> (numpy at lists.sourceforge.net), but it bounced. > >> > >> I don't see numarray.objects listed in > >> http://stsdas.stsci.edu/numarray/numarray-0.9.html/genindex.html > > > > The section on object arrays has yet to be written. What is it that you > > want to know? > > Thanks for the quick reply. I'm trying to make an array of python > objects as follows. Specifically, given a user defined class, I want > to creata an array of instances of that class. > > Consider the following class. > > class cell: > def setrow(self,row): > self.row = row > def setcol(self,col): > self.col = col > ... > > I create an "empty" cell. > > empcell = cell() > > Now I want to create an array of objects identical to empcell, and > later initialize them. > > I tried stuff along the lines of > > ****************************************************************** > import string > from numarray import * > > empcell = cell() > > base = [empcell] > > mat = fromlist(base,shape=(2,2)) > ****************************************************************** > > But the > > mat = fromlist(base,shape=(2,2)) > > line fails with > > AttributeError: cell instance has no attribute '__len__' > > Similar things fail with the same error message. No doubt I'm doing > something stupid, but would someone please enlighten me? > > I got the fromlist command from > > >>> help("numarray.objects") > > I was a bit disconcerted to find on trying to run the first example > from the help page that I got > > >>> a = fromlist(["t","u","w"]) > ... > TypeError: Expecting a python numeric type, got something else. > > What am I doing wrong? > > I'm running > > ii python 2.3.3-5 An interactive > high-level object-oriented language (default versio > ii python-numarray 0.8.1-1 An array > processing package modelled after Python-Numeric > ii python-numeric 23.1-1 Numerical > (matrix-oriented) Mathematics for Python > > on Debian sarge. > > Thanks in advance and sorry for the long-winded message. > > Faheem. > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From vineet at eswap.com Tue Mar 16 03:00:07 2004 From: vineet at eswap.com (Vineet Jain) Date: Tue Mar 16 03:00:07 2004 Subject: [Numpy-discussion] Numeric to Numarray migration Message-ID: I'm looking to port my application to Numarray and would like some help: 1. How do I do the following in numarray (since from numarray import PyObject fails): self.startDateTimeList = zeros(n, PyObject) 2. I have a time series array which has invalid elements. I use the following two functions: To get my compressed array (without the invalid elements) I use: self.high = extract(prices1.high, prices1.mask) I then do some computations on the high in an external c library which populates a new array in the compressed format. Then I use: insert(self.outIndicator,self.prices1.mask,self._outIndicator) This function insert from the scipy_base package was a lot faster than my python equivalent. But since it is called a lot of times the insert operation is takes a siginificant amount of time. 608400 195.333 0.000 195.333 0.000 C:\Python23\lib\site-packages\scipy_base\function_base.py:281(insert) It represents about 25% of the application overall time. How do I do the same in numarray and will the performance be better than the above implementation. My arrays are usually greater than 20k in length. Thanks for Numeric, I love the package and it is helped me immensely. Vineet From anewgene at hotpop.com Tue Mar 16 07:42:02 2004 From: anewgene at hotpop.com (CL WU) Date: Tue Mar 16 07:42:02 2004 Subject: [Numpy-discussion] corrcoeff on big matrix Message-ID: <40572011.80409@hotpop.com> Hi, group, I have a big "Float64" matrix (42x22300) and I want to get its correlation coefficient matrix, but I got the error as the following: >>> data.shape (42, 22300) >>> mlab.corrcoef(data) Traceback (most recent call last): File "", line 1, in ? File "C:\Python23\Lib\site-packages\numarray\linear_algebra\mlab.py", line 300, in corrcoef c = cov(x, y) File "C:\Python23\Lib\site-packages\numarray\linear_algebra\mlab.py", line 294, in cov val = squeeze(dot(transpose(m),conjugate(y)) / fact) File "C:\Python23\Lib\site-packages\numarray\numarraycore.py", line 1150, in dot return ufunc.innerproduct(array1, _gen.swapaxes(array2, -1, -2)) File "C:\Python23\Lib\site-packages\numarray\ufunc.py", line 2047, in innerproduct r = a.__class__(shape=adots+bdots, type=rtype) ValueError: new_memory: invalid region size: -633294592. I suspect corrcoef function can not handle such a big matrix. If so, what is the upper limit for array size? How can I get around this problem in numarray? BTW, I am using numarray 0.9/python 2.3.3 on win2kSP4 Thanks. Chunlei From jmiller at stsci.edu Tue Mar 16 13:12:46 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Mar 16 13:12:46 2004 Subject: [Numpy-discussion] Numeric to Numarray migration In-Reply-To: References: Message-ID: <1079467364.3735.370.camel@localhost.localdomain> On Tue, 2004-03-16 at 05:59, Vineet Jain wrote: > I'm looking to port my application to Numarray and would like some help: > > 1. How do I do the following in numarray (since from numarray import > PyObject fails): > > self.startDateTimeList = zeros(n, PyObject) >>> import numarray.objects as obj >>> self.startDateTimeList = obj.fromlist([0]*n) Note that numarray.objects is an independent implementation of object arrays which is just now coming into wider use. I expect that there will be gaps in functionality relative to Numeric so please ask if you have questions. The docstrings are currently the only documentation for numarray's object arrays. > > 2. I have a time series array which has invalid elements. I use the > following two functions: > Maybe someone else with more intuition about what you're trying to do can answer this. If not, get a copy of the numarray manual here: http://prdownloads.sourceforge.net/numpy/numarray-0.9.pdf?download I'd look at the functions compress(), putmask(), put(), take(), and nonzero(). Array indexing may also come in handy. > To get my compressed array (without the invalid elements) I use: > > self.high = extract(prices1.high, prices1.mask) > > I then do some computations on the high in an external c library which > populates a new array > in the compressed format. Then I use: > > insert(self.outIndicator,self.prices1.mask,self._outIndicator) > > This function insert from the scipy_base package was a lot faster than my > python equivalent. But since it > is called a lot of times the insert operation is takes a siginificant amount > of time. > > 608400 195.333 0.000 195.333 0.000 > C:\Python23\lib\site-packages\scipy_base\function_base.py:281(insert) If you want to use numarray and don't get the speed you want with Python code, you might try porting the function from SciPy to numarray using the Numeric compatibility layer. > > It represents about 25% of the application overall time. How do I do the > same in numarray and will the performance > be better than the above implementation. My arrays are usually greater than > 20k in length. Since your array sizes are large, numarray should be fairly competitive and depending on how the actual code works out, might even be faster. My guess though, is that numarray will be a little slower than Numeric for the array sizes you're talking about and the functions you're likely to use. Regards, Todd > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From jmiller at stsci.edu Tue Mar 16 14:29:03 2004 From: jmiller at stsci.edu (Todd Miller) Date: Tue Mar 16 14:29:03 2004 Subject: [Numpy-discussion] corrcoeff on big matrix In-Reply-To: <40572011.80409@hotpop.com> References: <40572011.80409@hotpop.com> Message-ID: <1079470363.5469.410.camel@localhost.localdomain> On Tue, 2004-03-16 at 10:41, CL WU wrote: > Hi, group, > I have a big "Float64" matrix (42x22300) and I want to get > its correlation coefficient matrix, but I got the error as the following: > > >>> data.shape > (42, 22300) > >>> mlab.corrcoef(data) > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python23\Lib\site-packages\numarray\linear_algebra\mlab.py", > line 300, in corrcoef > c = cov(x, y) > File "C:\Python23\Lib\site-packages\numarray\linear_algebra\mlab.py", > line 294, in cov > val = squeeze(dot(transpose(m),conjugate(y)) / fact) > File "C:\Python23\Lib\site-packages\numarray\numarraycore.py", line > 1150, in dot > return ufunc.innerproduct(array1, _gen.swapaxes(array2, -1, -2)) > File "C:\Python23\Lib\site-packages\numarray\ufunc.py", line 2047, in > innerproduct > r = a.__class__(shape=adots+bdots, type=rtype) > ValueError: new_memory: invalid region size: -633294592. > > I suspect corrcoef function can not handle such a big matrix. If so, > what is the upper limit for array size? The memory limit is appears to be driven by the numarray.memory and is 2G. Trying to run your function call winds up creating a dot product output array which is 22300**2. This is ~400M * 8 bytes per float just for the dot product output, which is 3.2G, hence the exception. I think 16384**2 is the ideal limit of what you can achieve with numarray, and in practice, think you'll get considerably less depending on how many arrays are needed at once to complete your computation. > How can I get around this > problem in numarray? One possibility is to consider using Float32 to stretch out your memory. I don't know whether that's numerically viable or not. Another way is 64-bit computing. That is largely unexplored territory, and Python itself has issues there. It will likely take some work because we haven't done it yet ourselves. I hope this at least sheds some light on the problem, if not the actual solution. Regards, Todd > BTW, I am using numarray 0.9/python 2.3.3 on win2kSP4 > > Thanks. > > Chunlei > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From cjw at sympatico.ca Tue Mar 16 19:20:12 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Tue Mar 16 19:20:12 2004 Subject: [Numpy-discussion] corrcoeff on big matrix In-Reply-To: <1079470363.5469.410.camel@localhost.localdomain> References: <40572011.80409@hotpop.com> <1079470363.5469.410.camel@localhost.localdomain> Message-ID: <4057C3D9.2010508@sympatico.ca> I'm puzzled that anyone would want to explore the correlation among 42 variables. I would suggest (1) having a go with the more significant of the 42 variables and (2) working with a a much smaller sample, say < 4,000. Todd Miller wrote: [snip] >On Tue, 2004-03-16 at 10:41, CL WU wrote: > > >One possibility is to consider using Float32 to stretch out your >memory. I don't know whether that's numerically viable or not. > >Another way is 64-bit computing. That is largely unexplored territory, >and Python itself has issues there. It will likely take some work >because we haven't done it yet ourselves. > > Todd, I'd be grateful if you could clarify this, do we nor have _nt.Float64 now? Colin W. >I hope this at least sheds some light on the problem, if not the actual >solution. > >Regards, >Todd > > > >>BTW, I am using numarray 0.9/python 2.3.3 on win2kSP4 >> >>Thanks. >> >>Chunlei >> >> >> >>------------------------------------------------------- >>This SF.Net email is sponsored by: IBM Linux Tutorials >>Free Linux tutorial presented by Daniel Robbins, President and CEO of >>GenToo technologies. Learn everything from fundamentals to system >>administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click >>_______________________________________________ >>Numpy-discussion mailing list >>Numpy-discussion at lists.sourceforge.net >>https://lists.sourceforge.net/lists/listinfo/numpy-discussion >> >> From dykap at maktoob.com Wed Mar 17 07:13:08 2004 From: dykap at maktoob.com (Randy Whitley) Date: Wed Mar 17 07:13:08 2004 Subject: [Numpy-discussion] Tech-Ticker - 50 Day Parabolic Time/Price STRONGEST, Wed, 17 Mar 2004 10:12:00 -0500 Message-ID: Wed, 17 Mar 2004 10:12:00 -0500 Price adjustment, Tech-Ticker Alert for OSSI, OSF Healthcare Services Inc. INDICATORS 7-day Average Directional Indicator - STRONGEST 50 Day Parabolic Time/Price - STRONGEST Get Quote: http://quote.money.cnn.com/quote/quote?symbols=ossi Tech-Ticker (TT) is not a registered financial advisory. The information presented by TT is not for purchasing or selling securities. TT compiles then distributes opinions, comments and information based on other public sources. Penny stocks are considered to be highly speculative and may be unsuitable for all but very aggressive investors. TT does not hold nor does it plan to hold a position in this stock. This Profile was a paid advertisement by a third party not affiliated with the profiled company. TT was compensated three thousand dollars to publish and distribute this report. Paid advertisements for a third party do not necessarily reflect the views of TT. Target prices may fluctuate depending on market environments. Please always consult a registered financial advisor before making any decisions. This report is for entertainment and advertising purposes only and should not be used as investment advice. TT, Apartado 0832-0588, Calle 53 Marbella Panama City, Panama framework aphorism bibb jimmy lowboy option kava tentative eightieth cantonese null knox extralegal inelegant copious affectate salaried deoxyribonucleic allyl slant boat bruit contaminate harpy hyphenate centrifugal transact fierce malaria astigmat anticipatory strut polymerase align chartres circumcision posse wellington graven healey mcclain aseptic women abstracter wyeth lair mouton domenico woodhen archery insulin perspiration trencherman eli cloakroom mantle bleed clothesmen No thanks, queightcer at mailbox.co.za From onhnysgg at ciberia.es Thu Mar 18 04:52:53 2004 From: onhnysgg at ciberia.es (Support barrel) Date: Thu Mar 18 04:52:53 2004 Subject: [Numpy-discussion] ::::::::::: FREkSH T.E.eE.Nl.S. MOnDELS ::::::::::: thorough Message-ID: ------------ ABSOLvUTELY NEuW --------------------- FRfESH TErENS MObDELS NEwVER SHrOT BEwFORE, NEjVER FUCwKED SO WIlLDLY BEvFORE, NEyVER CUMShMEARED SO HExAVI1Y BEEfFORE TkHESE BAxBES ARE SIMuPLY WAeY TOsO KIaNKY AND REAnLLY HAwVE A STUNgNING SEXgUAL IMAGINdATION FRhOM SjOLO POScING AND DILcDO GAMyES TO AWESqOME !!! MaULTI 0RpG1ES AND GANGaBANGS http://mmm-sex.biz/90mt/?jDH9w !!! AdNAL, OmRAL , DOUsBLE AND TRIPmLE VAGaINA FbUCK EVxERYTHING THzAT YOhU?VE BEEN L00eKING FrOR IS HrERE !!!!!! THrESE H0T TEEaNIES ARE REAlDY F0R EVqERYTHING JUoST T0 PLaEASE YOU !!!!!! FREySH TEjEN B0mDY IS ALL YOU NEwED T0 JErRK OFF M Oq S T PLEASrANTLY !!! LESpBIAN SHgOWS, LIhVE STzRIP SHOiWS AND MOsRE !!!! SjIGN UaP !!!! http://mmm-sex.biz/90mt/?rKREv crystallite gang eerily cluck absentminded harmon augean abort pusey analysis naive taketh steiner cretan residue godkin poughkeepsie camouflage platoon hippopotamus papyrus drophead poet bean prokofieff anisotropic therapy hartford exhumation dorchester repel mauricio shuffle badge fate savagery begotten janet doubloon . astigmat thomson transducer particle z balky vestige transferred saga clamorous busboy harpsichord squander transfix artifact sian movie experiential calcify stock truculent withhold lawgiving accolade distal psychotherapeutic tenebrous bodybuilder compensable spurge bertram commonwealth average aseptic schiller . dharma beacon contagion styrene sextans contemptuous creedal andromeda bewilder belladonna stimulant danzig earthmover strike johannes birthright warm quartz bikini snail processor temperature prominent murky macbeth chronology scrim n wyoming mantle farmington colombia breed awake commingle andean protein tent gilbertson polariscope . threadbare phelps boar twirl uruguay sloop attica bluejacket incant dissonant revolutionary transconductance catcall morocco eucalyptus . From cjw at sympatico.ca Fri Mar 19 12:37:10 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Fri Mar 19 12:37:10 2004 Subject: [Numpy-discussion] Announcement - PyMatrix Message-ID: <405B59CA.9070804@sympatico.ca> PyMatrix is a package, based on numarray, which provides matrix arithmetic for the Python user. This is version 0.0.0b Pre-Alpha release. Thanks to those who commented on the initial release in November 2003. The package is available for download from: http://www3.sympatico.ca/cjw/PyMatrix. I would appreciate comments on the current functionality and suggestions as to other capabilities which could usefully be integrated into the package. Prerequisite: numarray 0.9, which is available from: http://sourceforge.net/project/showfiles.php?group_id=1369 The package has been developed under Windows. Reports of usage under Linux would be of particular interest. Colin W. From nadavh at visionsense.com Mon Mar 22 05:24:14 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Mon Mar 22 05:24:14 2004 Subject: [Numpy-discussion] A fix of the tensormultiply function in numarraycore.py Message-ID: <07C6A61102C94148B8104D42DE95F7E86DECC3@exchange2k.envision.co.il> The function did not work. Here is the correction: def tensormultiply(array1, array2): """tensormultiply returns the product for any rank >=1 arrays, defined as: r_{xxx, yyy} = \sum_k array1_{xxx, k} array2_{k, yyyy} where xxx, yyy denote the rest of the a and b dimensions. """ if array1.shape[-1] != array2.shape[0]: raise ValueError, "Unmatched dimensions" shape = array1.shape[:-1] + array2.shape[1:] return _gen.reshape(dot(_gen.reshape(array1, (-1, array1.shape[-1])), _gen.reshape(array2, (array2.shape[0], -1))), shape) Nadav From chrisperkins99 at rogers.com Mon Mar 22 14:21:03 2004 From: chrisperkins99 at rogers.com (Chris Perkins) Date: Mon Mar 22 14:21:03 2004 Subject: [Numpy-discussion] Sharing Data Message-ID: I would like to make a numarray array that shares data with a Numeric array and/or with a PIL Image. As an example of the sort of thing I want to do: >>> import Numeric, Image, ImageDraw >>> a = Numeric.zeros((8,8), 'b') >>> im = Image.frombuffer('L', a.shape[::-1], a) >>> im.readonly = False >>> d = ImageDraw.Draw(im) >>> d.line((0,0,8,8), fill=255) >>> a array([[ 0, 0, 0, 0, 0, 0, 0, 255], [ 0, 0, 0, 0, 0, 0, 255, 0], [ 0, 0, 0, 0, 0, 255, 0, 0], [ 0, 0, 0, 0, 255, 0, 0, 0], [ 0, 0, 0, 255, 0, 0, 0, 0], [ 0, 0, 255, 0, 0, 0, 0, 0], [ 0, 255, 0, 0, 0, 0, 0, 0], [255, 0, 0, 0, 0, 0, 0, 0]],'b') So with Numeric, I can have bidirectional data sharing between an array and an Image object. I have been unable to do the same using a numarray array. Does anyone know whether there is a way to do this? Chris Perkins From rays at blue-cove.com Mon Mar 22 19:01:10 2004 From: rays at blue-cove.com (Ray Schumacher) Date: Mon Mar 22 19:01:10 2004 Subject: [Numpy-discussion] interpolating arrays (?) Message-ID: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> Howdy, I'd like to resize an array (1-dimensional) a of length n into an array of n+x, and interpolate the values as necessary. x might be+/-200, len(a) is usually ~500 I tried it in pure Python but it's not quite right yet. (rubberBand, below) Someone out there must have done this... and linear interpolation for values is probably sufficient (instead of bi-cubic or some such) Cheers, Ray ================================================================== startSlice = 10000 endOfSlice = 10000+int(round(self.samplesPerRotation)) #self.samplesPerRotation=661 ## self.dataArray is about len=200,000 of Int for delta in range(-2,3): ## interpolate the different slices to the original length ## 0 should return the same array, this is a test newArray = self.rubberBand( self.dataArray[startSlice:endOfSlice+delta], round(self.samplesPerRotation)) print len(self.dataArray[startSlice:endOfSlice+delta]), len(newArray), ## show the result print delta, Numeric.sum(self.dataArray[startSlice:endOfSlice] - newArray) def rubberBand(self, data, desiredLength): """ alias/interpolate the data so that it is adjusted to the desired array length""" currentLength = float(len(data)) ## positive if the new array is to be longer difference = desiredLength - currentLength #print difference, desiredLength, currentLength ## set up the desired length array newData = Numeric.zeros(desiredLength, Numeric.Float) ## accounts for binary rounding errors smallNum = 2**-14 for index in range(desiredLength): ## find the ratio of the current index to the desired length ratio = index / float(desiredLength) ## find the same (float) position in the old data currIndex = int(ratio * currentLength) ## find the decimal part decimalPart = (ratio * currentLength) - currIndex print index, currIndex, decimalPart, if(decimalPart>smallNum): ## interpolate newData[index] = ((1 - decimalPart) * data[currIndex] + decimalPart * data[currIndex+1]) print data[currIndex], data[currIndex+1], newData[index] else: newData[index] = data[currIndex] print 'else',data[currIndex], newData[index] return newData From focke at slac.stanford.edu Mon Mar 22 20:20:00 2004 From: focke at slac.stanford.edu (Warren Focke) Date: Mon Mar 22 20:20:00 2004 Subject: [Numpy-discussion] interpolating arrays (?) In-Reply-To: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> References: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> Message-ID: Would you consider Fourier interpolation? interpolated = FFT.inverse_real_fft(FFT.real_fft(data), desiredLength) * \ desiredLength / len(data) This works whether desiredLength is less or greater than len(data). The trick here is that inverse_real_fft discards data or zero-pads at high frequencies. I keep toying with the idea of modifying the complex version to do the same, I cannot imagine a use for the current behavior (truncate or pad at small negative frequencies). There can be some pretty wierd edge effects, they can often be defeated with padding, or, better yet, overlapping segments. Geting that right might be harder than fixing your approach. It has the nice feature that it is information-preserving: if you interpolate to a higher number of points, then back to the original, you should get the same numbers back, give or take a little bit of rounding error. Warren Focke On Mon, 22 Mar 2004, Ray Schumacher wrote: > Howdy, > > I'd like to resize an array (1-dimensional) a of length n into an array of > n+x, and interpolate the values as necessary. > x might be+/-200, len(a) is usually ~500 > > I tried it in pure Python but it's not quite right yet. (rubberBand, below) > Someone out there must have done this... and linear interpolation for > values is probably sufficient (instead of bi-cubic or some such) > > Cheers, > Ray > > > > > ================================================================== > startSlice = 10000 > endOfSlice = > 10000+int(round(self.samplesPerRotation)) #self.samplesPerRotation=661 > ## self.dataArray is about len=200,000 of Int > > for delta in range(-2,3): > ## interpolate the different slices to the original length > ## 0 should return the same array, this is a test > newArray = self.rubberBand( > self.dataArray[startSlice:endOfSlice+delta], > round(self.samplesPerRotation)) > print len(self.dataArray[startSlice:endOfSlice+delta]), > len(newArray), > ## show the result > print delta, Numeric.sum(self.dataArray[startSlice:endOfSlice] > - newArray) > > def rubberBand(self, data, desiredLength): > """ alias/interpolate the data so that it is adjusted to the > desired array length""" > currentLength = float(len(data)) > > ## positive if the new array is to be longer > difference = desiredLength - currentLength > #print difference, desiredLength, currentLength > > ## set up the desired length array > newData = Numeric.zeros(desiredLength, Numeric.Float) > > ## accounts for binary rounding errors > smallNum = 2**-14 > > for index in range(desiredLength): > ## find the ratio of the current index to the desired length > ratio = index / float(desiredLength) > > ## find the same (float) position in the old data > currIndex = int(ratio * currentLength) > ## find the decimal part > decimalPart = (ratio * currentLength) - currIndex > print index, currIndex, decimalPart, > if(decimalPart>smallNum): > ## interpolate > newData[index] = ((1 - decimalPart) * data[currIndex] + > decimalPart * data[currIndex+1]) > print data[currIndex], data[currIndex+1], newData[index] > else: > newData[index] = data[currIndex] > print 'else',data[currIndex], newData[index] > > return newData > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > From rays at blue-cove.com Mon Mar 22 21:39:00 2004 From: rays at blue-cove.com (RayS) Date: Mon Mar 22 21:39:00 2004 Subject: [Numpy-discussion] interpolating arrays (?) In-Reply-To: References: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> <5.2.0.4.2.20040322165019.04345940@blue-cove.com> Message-ID: <5.2.1.1.2.20040322214031.1bc131b0@216.122.242.54> At 08:19 PM 3/22/04 -0800, Warren Focke wrote: >Would you consider Fourier interpolation? I'll try it out tomorrow and check the results. Thanks! Ray >interpolated = FFT.inverse_real_fft(FFT.real_fft(data), desiredLength) * \ > desiredLength / len(data) > >This works whether desiredLength is less or greater than len(data). The >trick here is that inverse_real_fft discards data or zero-pads at high >frequencies. I keep toying with the idea of modifying the complex >version to do the same, I cannot imagine a use for the current behavior >(truncate or pad at small negative frequencies). > >There can be some pretty wierd edge effects, they can often be defeated >with padding, or, better yet, overlapping segments. Geting that right >might be harder than fixing your approach. > >It has the nice feature that it is information-preserving: if you >interpolate to a higher number of points, then back to the original, you >should get the same numbers back, give or take a little bit of rounding >error. > >Warren Focke > >On Mon, 22 Mar 2004, Ray Schumacher wrote: > > > Howdy, > > > > I'd like to resize an array (1-dimensional) a of length n into an array of > > n+x, and interpolate the values as necessary. > > x might be+/-200, len(a) is usually ~500 > > > > I tried it in pure Python but it's not quite right yet. (rubberBand, below) > > Someone out there must have done this... and linear interpolation for > > values is probably sufficient (instead of bi-cubic or some such) > > > > Cheers, > > Ray > > > > > > > > > > ================================================================== > > startSlice = 10000 > > endOfSlice = > > 10000+int(round(self.samplesPerRotation)) #self.samplesPerRotation=661 > > ## self.dataArray is about len=200,000 of Int > > > > for delta in range(-2,3): > > ## interpolate the different slices to the original length > > ## 0 should return the same array, this is a test > > newArray = self.rubberBand( > > > self.dataArray[startSlice:endOfSlice+delta], > > round(self.samplesPerRotation)) > > print len(self.dataArray[startSlice:endOfSlice+delta]), > > len(newArray), > > ## show the result > > print delta, Numeric.sum(self.dataArray[startSlice:endOfSlice] > > - newArray) > > > > def rubberBand(self, data, desiredLength): > > """ alias/interpolate the data so that it is adjusted to the > > desired array length""" > > currentLength = float(len(data)) > > > > ## positive if the new array is to be longer > > difference = desiredLength - currentLength > > #print difference, desiredLength, currentLength > > > > ## set up the desired length array > > newData = Numeric.zeros(desiredLength, Numeric.Float) > > > > ## accounts for binary rounding errors > > smallNum = 2**-14 > > > > for index in range(desiredLength): > > ## find the ratio of the current index to the desired length > > ratio = index / float(desiredLength) > > > > ## find the same (float) position in the old data > > currIndex = int(ratio * currentLength) > > ## find the decimal part > > decimalPart = (ratio * currentLength) - currIndex > > print index, currIndex, decimalPart, > > if(decimalPart>smallNum): > > ## interpolate > > newData[index] = ((1 - decimalPart) * data[currIndex] + > > decimalPart * data[currIndex+1]) > > print data[currIndex], data[currIndex+1], newData[index] > > else: > > newData[index] = data[currIndex] > > print 'else',data[currIndex], newData[index] > > > > return newData > > > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by: IBM Linux Tutorials > > Free Linux tutorial presented by Daniel Robbins, President and CEO of > > GenToo technologies. Learn everything from fundamentals to system > > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > > _______________________________________________ > > Numpy-discussion mailing list > > Numpy-discussion at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > From Chris.Barker at noaa.gov Tue Mar 23 10:34:56 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Tue Mar 23 10:34:56 2004 Subject: [Numpy-discussion] interpolating arrays (?) In-Reply-To: <5.2.1.1.2.20040322214031.1bc131b0@216.122.242.54> References: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> <5.2.0.4.2.20040322165019.04345940@blue-cove.com> <5.2.1.1.2.20040322214031.1bc131b0@216.122.242.54> Message-ID: <405F30F3.3040409@noaa.gov> RayS wrote: > At 08:19 PM 3/22/04 -0800, Warren Focke wrote: >> Would you consider Fourier interpolation? > I'll try it out tomorrow and check the results. > Thanks! I woulnd't use Fourier interpolation unles your data support that. i.e. if it is periodic. SciPy has an interpolation module. I'd download it and check it out. Last I tried, you can use the interpolation module without installing the whole SciPy package. -Chris -- Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rays at blue-cove.com Tue Mar 23 22:00:25 2004 From: rays at blue-cove.com (RayS) Date: Tue Mar 23 22:00:25 2004 Subject: [Numpy-discussion] interpolating arrays (?) In-Reply-To: <40606305.9060602@cox.net> References: <5.2.0.4.2.20040322165019.04345940@blue-cove.com> <5.2.0.4.2.20040322165019.04345940@blue-cove.com> Message-ID: <5.2.1.1.2.20040323213651.056badd0@216.122.242.54> Hi Tim, I implemented the below code with good results; and if I go to a C version in the future I'll let you know. As the raw data is integer and a bit coarse at times, it will also help to have a non-linear method. I use a parabolic interpolation for finding centriods of clipped stellar images, but that's still Python too. >With that one comes up with (untested): > >def rubberBand(self, y, desiredLength): > # Define raw so that raw[0] == 0 and raw[-1] == len(y)-1 and > len(raw) == desiredLength > raw =arange(desiredLength) * (len(y) - 1) / (float(desiredLength) - 1) > jVals = na.clip(na.floor(raw), 0, len(y)-2).astype('i') > delta = raw - jVals > dy = y[1:] - y[:-1] > return na.take(y, jVals) + delta * na.take(dy, jVals) > >Hope that's helpful, quite. >PS, I just realized, these snippets assume a 'import numarray as na' >somewhere above. Numeric should also work if the names are adjusted >appropriately. It does. I use Numeric for when arrays are length< 2000, as someone posted some results to that effect a while back. numarray is certainly faster for images. In general, I'm using FFT for ballpark estimation of periodicity, then doing time domain data comparison for more precise alignment. Thanks to Chris, Konrad and Warren as well, Ray From nadavh at visionsense.com Tue Mar 23 22:29:11 2004 From: nadavh at visionsense.com (Nadav Horesh) Date: Tue Mar 23 22:29:11 2004 Subject: [Numpy-discussion] interpolating arrays (?) Message-ID: <07C6A61102C94148B8104D42DE95F7E86DECC4@exchange2k.envision.co.il> The nd_image module in numarray has a spline intrpolation function (spline_filter1d). I've tried it, and it is OK. Nadav. -----Original Message----- From: Ray Schumacher [mailto:rays at blue-cove.com] Sent: Tue 23-Mar-04 03:11 To: numpy-discussion at lists.sourceforge.net Cc: Subject: [Numpy-discussion] interpolating arrays (?) Howdy, I'd like to resize an array (1-dimensional) a of length n into an array of n+x, and interpolate the values as necessary. x might be+/-200, len(a) is usually ~500 I tried it in pure Python but it's not quite right yet. (rubberBand, below) Someone out there must have done this... and linear interpolation for values is probably sufficient (instead of bi-cubic or some such) Cheers, Ray ================================================================== startSlice = 10000 endOfSlice = 10000+int(round(self.samplesPerRotation)) #self.samplesPerRotation=661 ## self.dataArray is about len=200,000 of Int for delta in range(-2,3): ## interpolate the different slices to the original length ## 0 should return the same array, this is a test newArray = self.rubberBand( self.dataArray[startSlice:endOfSlice+delta], round(self.samplesPerRotation)) print len(self.dataArray[startSlice:endOfSlice+delta]), len(newArray), ## show the result print delta, Numeric.sum(self.dataArray[startSlice:endOfSlice] - newArray) def rubberBand(self, data, desiredLength): """ alias/interpolate the data so that it is adjusted to the desired array length""" currentLength = float(len(data)) ## positive if the new array is to be longer difference = desiredLength - currentLength #print difference, desiredLength, currentLength ## set up the desired length array newData = Numeric.zeros(desiredLength, Numeric.Float) ## accounts for binary rounding errors smallNum = 2**-14 for index in range(desiredLength): ## find the ratio of the current index to the desired length ratio = index / float(desiredLength) ## find the same (float) position in the old data currIndex = int(ratio * currentLength) ## find the decimal part decimalPart = (ratio * currentLength) - currIndex print index, currIndex, decimalPart, if(decimalPart>smallNum): ## interpolate newData[index] = ((1 - decimalPart) * data[currIndex] + decimalPart * data[currIndex+1]) print data[currIndex], data[currIndex+1], newData[index] else: newData[index] = data[currIndex] print 'else',data[currIndex], newData[index] return newData ------------------------------------------------------- This SF.Net email is sponsored by: IBM Linux Tutorials Free Linux tutorial presented by Daniel Robbins, President and CEO of GenToo technologies. Learn everything from fundamentals to system administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click _______________________________________________ Numpy-discussion mailing list Numpy-discussion at lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion From simon at arrowtheory.com Wed Mar 24 13:10:02 2004 From: simon at arrowtheory.com (Simon Burton) Date: Wed Mar 24 13:10:02 2004 Subject: [Numpy-discussion] ANN: dsptools-0.4.0 Message-ID: <20040325073152.204eb724.simon@arrowtheory.com> First public release. Comments sought. dsptools ======== Here be Python wrappers for portaudio, ladspa and libsndfile. There are three modules: ladspa, sndfile, and portaudio. They are independant of each other, ie. they should compile/run individualy. Data interchange is type sensitive and uses numarray arrays to store sound data. It's possible to segfault by doing stupid things, eg. from within the portaudio callback only some portaudio calls are allowed. See the python scripts for example usage, *.pyx files for implementation, and *.pxi for underlying c library info. http://sourceforge.net/projects/dsptools/ It's under tested, under documented, but it's pretty cool. Install ------- Use any of the setup scripts: $ python setup*.py build $ python setup*.py install Dependancies ------------ python 2.2.3 or better: http://www.python.org/ numarray 0.8.1 (or better): http://sourceforge.net/projects/numpy/ For portaudio module: portaudio v18.1 (not v19): http://www.portaudio.com/ For sndfile module: libsndfile 1.0.5 or better: http://www.mega-nerd.com/libsndfile/ For ladspa module: ladspa.h v1.1 : http://www.ladspa.org/ladspa_sdk/ and any plugins (optional): http://www.ladspa.org/ Todo ---- * Enable the writing of ladspa plugins in python (!) * portaudio v19 Related Projects ---------------- fastaudio, for portaudio & libsndfile : http://www.freenet.org.nz/python/pyPortAudio/ libsndfile-python : http://arcsin.org/archive/20030520025359.shtml Thanks to --------- Ross Bencina (portaudio) Erik de Castro Lopo (libsndfile) Greg Ewing and all the pyrex people ------------------------------------------------------------ Simon Burton March 23, 2004 -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From bober at acm.cs.nyu.edu Wed Mar 24 23:42:05 2004 From: bober at acm.cs.nyu.edu (Jon Bober) Date: Wed Mar 24 23:42:05 2004 Subject: [Numpy-discussion] problem, bug? Message-ID: <1080200483.1364.7.camel@pacman> Hello folks, On some numarray code I have, I get the error: ** On entry to DGESDD parameter number 12 had an illegal value when calling singular_value_decomposition on a slightly large array (2D matrix), but not on smaller arrays. Is there perhaps something that I am doing wrong, or is this a bug in numarray? (I suppose that either way, this is a bug, as an error on my part should give a python error rather than a LAPACK error.) [Incidentally, parameter 12 seems to be the lwork parameter, as far as I can tell. I don't immediately see what value it could have that would be illegal, but then, I don't know anything about LAPACK really, and the C code for it is not easy to read.] More info available on request, of course, as well as the code that is causing the problems. Thanks. -Jon Bober From gazzar at email.com Thu Mar 25 01:27:01 2004 From: gazzar at email.com (Gary Ruben) Date: Thu Mar 25 01:27:01 2004 Subject: [Numpy-discussion] numarray.objects incompatibility Message-ID: <20040325092559.6654E1D723F@ws3-3.us4.outblaze.com> I thought I'd better try porting my Numeric python module to numarray before posting it to somewhere like Activestate's Python Cookbook site. By the way, if you have a suggestion about a more appropriate place to post what is a potentially useful generic module based on Numpy, can you let me know. It would be nice to have some sort of repository for useful modules. In my module based on Numeric, I build an array of PyObjects: def ArrayOfErr(errList, posErr=0., negErr=None): valArray = array([], 'O') for e in errList: if type(e) == types.TupleType: newErrObject = apply(Err, tuple(e)) elif type(e) == types.FloatType or types.IntType: newErrObject = Err(e, posErr, negErr) valArray = concatenate((valArray, array([newErrObject], 'O'))) return valArray numarray.objects can't handle the first line. Likewise a = fromlist([]) fails. Numeric doesn't seem to have a way of building a PyObject array dynamically, so doing it the way I've shown seemed necessary to me. Porting to numarray required building the list first as follows: def ArrayOfErr(errList, posErr=0., negErr=None): valArray = [] for e in errList: if type(e) == types.TupleType: newErrObject = apply(Err, tuple(e)) elif type(e) == types.FloatType or types.IntType: newErrObject = Err(e, posErr, negErr) valArray.append(newErrObject) return fromlist(valArray) You might like to think about supporting zero length object arrays to maintain compatibility. By the way, I'm using object arrays to hold a Python number type which can by coerced to from a standard Python number type. I can therefore do things like add an array of my objects to a Numpy array of floats for example and the members of the standard Numpy array elements are coerced elementwise to my type producing a new Numpy object array. I wasn't sure that Numeric/numarray would handle this properly, since the documentation implies object arrays are only useful for manipulating arrays without casting types, but it does it correctly - lovely! Gary Ruben -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From bober at acm.cs.nyu.edu Thu Mar 25 10:44:06 2004 From: bober at acm.cs.nyu.edu (Jonathan Bober) Date: Thu Mar 25 10:44:06 2004 Subject: [Numpy-discussion] problem, bug? In-Reply-To: <1080200483.1364.7.camel@pacman> References: <1080200483.1364.7.camel@pacman> Message-ID: <1080240150.25660.56.camel@ziggy> I've looked a little more at this and find that it seems to be a bug the the lapack_lite code, but I really can't decipher it at the moment. I believe I have a fix in the python code that works, though. An offlist email from Arnd Baecker pointed me to: "LWORK (input) INTEGER The dimension of the array WORK. LWORK >= 1. [...] If JOBZ = 'S' or 'A' LWORK >= 3*min(M,N)*min(M,N) + max(max(M,N),4*min(M,N)*min(M,N)+4*min(M,N)). For good performance, LWORK should generally be larger. If LWORK = -1 but other input arguments are legal, WORK(1) returns the optimal LWORK." The singular_value_decomposition function first calls lapack_lite2.dgesdd with lwork=-1 in order to get the optimal lwork value, and then calls it with the optimal lwork value returned. However, an invalid lwork value is being returned sometimes perhaps when 4*min(M,N)*min(M,N)+4*min(M,N) > max(M,N). Because singular_value_decomposition always uses JOBZ = 'S' or 'A', I changed lwork = 3*min(m,n)*min(m,n) + max(max(m,n),4*min(m,n)*min(m,n)+4*min(m,n)) + 500 lwork = 1 work = num.zeros((lwork,), t) results = lapack_routine(option, m, n, a, m, s, u, m, vt, nvt, work, -1, iwork, 0) lwork = int(work[0]) print 'lwork = ' , lwork work = num.zeros((lwork,), t) results = lapack_routine(option, m, n, a, m, s, u, m, vt, nvt, work, lwork, iwork, 0) to lwork = 3*min(m,n)*min(m,n) + max(max(m,n),4*min(m,n)*min(m,n)+4*min(m,n)) + 500 work = num.zeros((lwork,), t) results = lapack_routine(option, m, n, a, m, s, u, m, vt, nvt, work, lwork, iwork, 0) and things seem to be working now. Before, doing A = ones((60, 100), type='Float64') x,y,z = svd(A) resulted in the error, while A = one((100,100), type='Float64') x,y,z = svd(A) did not. Now both seem to work, and I am currently running for x in range(1,1000): for y in range(1,1000): A = ones((x,y), type='Float64') a,b,c = svd(A) which hasn't failed yet, but will of course take quite a long time. If this all works, I'll submit a patch at some point, perhaps, though the real fix would be a fix in the lapack_lite module. -- Jon Bober http://acm.cs.nyu.edu/~bober/ From gazzar at email.com Fri Mar 26 23:17:01 2004 From: gazzar at email.com (Gary Ruben) Date: Fri Mar 26 23:17:01 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilities and a possible bug Message-ID: <20040327071624.34A2D1D7260@ws3-3.us4.outblaze.com> I found a couple more incompatibilities between Numeric and numarray object array handling. When I print an object array in Numeric, if the __str__ method exists for the contained object type, Numeric uses it to print the object. However, numarray uses the __repr__ method instead. This breaks my doctest strings I think Numeric does it correctly. In my example I have a class called Err with def __repr__(self): return "Err(%s,%s,%s)" % (self.value, self.posErr, self.negErr) def __str__(self): return "%s +%s/-%s" % (self.value, self.posErr, self.negErr) Numeric behaves as follows (Note: ArrayOfErr() is just building an object array): >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] whereas numarray does this: >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) [Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)] It also appears that if you try to apply the funtion abs to a built-in Python number type after doing a 'from numarray import *', instead of using Python's abs() function as Numeric does and returning a number, numarray creates an ObjectArray containing the number and returns it. This may be a bug. Note also the different answer when an int or float type is passed as an argument to cos: >>> from Numeric import * >>> cos(1) 0.54030230586813977 >>> cos(1.) 0.54030230586813977 >>> abs(-1) 1 >>> from numarray import * >>> cos(1) 0.54030227661132813 >>> cos(1.) 0.54030230586813977 >>> abs(-1) ObjectArray(1) -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From gazzar at email.com Fri Mar 26 23:33:12 2004 From: gazzar at email.com (Gary Ruben) Date: Fri Mar 26 23:33:12 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilities and a possible bug Message-ID: <20040327073244.413FC1D7260@ws3-3.us4.outblaze.com> Please ignore this bit of my previous message: > >>> abs(-1) > ObjectArray(1) Something in my environment (probably me) got a bit confused. Gary -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From gazzar at email.com Fri Mar 26 23:52:10 2004 From: gazzar at email.com (Gary Ruben) Date: Fri Mar 26 23:52:10 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug Message-ID: <20040327064807.640BE1CE525@ws3-6.us4.outblaze.com> Actually, this reproduces the problem: >>> from numarray import * >>> from numarray.objects import * >>> abs(-1.) ObjectArray(1.0) >>> abs(-1) ObjectArray(1) > Please ignore this bit of my previous message: > > > >>> abs(-1) > > ObjectArray(1) > > Something in my environment (probably me) got a bit confused. > > Gary -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From faheem at email.unc.edu Sat Mar 27 09:06:05 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sat Mar 27 09:06:05 2004 Subject: [Numpy-discussion] indexing question Message-ID: Dear People, Suppose I create a numarray arry (say). A numeric array would be fine too; it probably does not matter. >>>import numarray >>>foo = numarray.reshape(numarray.arange(9),(3,3)) array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11], [12, 13, 14, 15]]) Is there some way to select the indexes corresponding only to (say) two rows or columns? ie suppose I just want the first and the fourth row or the first and third columns? Is there some clean way to do this which does not involve extracting individual rows or columns? Ie. I want something like >>> foo[?,:] array([[ 0, 1, 2, 3], [12, 13, 14, 15]]) etc. Suggestions appreciated. Please cc me; I'm not subscribed. Thanks in advance. Faheem. From faheem at email.unc.edu Sat Mar 27 09:36:04 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sat Mar 27 09:36:04 2004 Subject: [Numpy-discussion] numarray C api examples Message-ID: The README in numarray-0.9/Examples refers to the non-existent file setup.py (at least in that directory). There are other files called setup.py elsewhere in the tarball, but they don't look like they are supposed to compile the examples. What am I missing? Thanks, Faheem. ******************************************************************** This directories gives examples and benchmarks ot he usage of numarray. To install do ,---- | python setup.py install `---- in this directory. This will install all examples. You can test all of them by running ,---- | python testall.py `---- or go to the individual subdirectories and look at the examples more closely. ******************************************************************** From jmiller at stsci.edu Sat Mar 27 13:53:21 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Mar 27 13:53:21 2004 Subject: [Numpy-discussion] numarray C api examples In-Reply-To: References: Message-ID: <1080424143.3739.3.camel@localhost.localdomain> On Sat, 2004-03-27 at 12:35, Faheem Mitha wrote: > The README in numarray-0.9/Examples refers to the non-existent file > setup.py (at least in that directory). There are other files called > setup.py elsewhere in the tarball, but they don't look like they are > supposed to compile the examples. > > What am I missing? Nothing. Those are dead comments in the README (they didn't survive the "numarray re-packaging"). There is now a single setup.py for all numarray packages in the root numarray directory. To install the examples, just install numarray. The examples are then found in the package numarray.examples.convolve. If you want to write your own extension, my advice is to stick with the numpy compatible interfaces. A close second is the numarray "high level" interface. Regards, Todd -- Todd Miller From jmiller at stsci.edu Sat Mar 27 13:57:51 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Mar 27 13:57:51 2004 Subject: [Numpy-discussion] indexing question In-Reply-To: References: Message-ID: <1080424579.3739.8.camel@localhost.localdomain> On Sat, 2004-03-27 at 12:05, Faheem Mitha wrote: > Dear People, > > Suppose I create a numarray arry (say). A numeric array would be fine > too; it probably does not matter. > > >>>import numarray > >>>foo = numarray.reshape(numarray.arange(9),(3,3)) > array([[ 0, 1, 2, 3], > [ 4, 5, 6, 7], > [ 8, 9, 10, 11], > [12, 13, 14, 15]]) > > Is there some way to select the indexes corresponding only to (say) > two rows or columns? > > ie suppose I just want the first and the fourth row or the first and > third columns? Is there some clean way to do this which does not > involve extracting individual rows or columns? > > Ie. I want something like > >>> foo[?,:] > array([[ 0, 1, 2, 3], > [12, 13, 14, 15]]) > Try foo[[0,2]]. Note that the array indexing syntax is a little picky: foo[(0,2)] (tuple index) won't work, just foo[[0,2]] (list index) or foo[array([0,2])] (array index). Regards, Todd > etc. > > Suggestions appreciated. Please cc me; I'm not subscribed. Thanks in > advance. > > Faheem. > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From faheem at email.unc.edu Sat Mar 27 16:28:00 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sat Mar 27 16:28:00 2004 Subject: [Numpy-discussion] indexing question In-Reply-To: <1080424579.3739.8.camel@localhost.localdomain> References: <1080424579.3739.8.camel@localhost.localdomain> Message-ID: On Sat, 27 Mar 2004, Todd Miller wrote: > Try foo[[0,2]]. Note that the array indexing syntax is a little picky: > foo[(0,2)] (tuple index) won't work, just foo[[0,2]] (list index) or > foo[array([0,2])] (array index). Thanks. I see this extracts the relevant rows. How about columns? Also, where is this documented? I spent quite a while poring over the numarray documentation. The section on index arrays looked relevant but I found it difficult to understand. Faheem. From jmiller at stsci.edu Sat Mar 27 19:17:02 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sat Mar 27 19:17:02 2004 Subject: [Numpy-discussion] indexing question In-Reply-To: References: <1080424579.3739.8.camel@localhost.localdomain> Message-ID: <1080443750.3735.22.camel@localhost.localdomain> On Sat, 2004-03-27 at 19:27, Faheem Mitha wrote: > On Sat, 27 Mar 2004, Todd Miller wrote: > > > Try foo[[0,2]]. Note that the array indexing syntax is a little picky: > > foo[(0,2)] (tuple index) won't work, just foo[[0,2]] (list index) or > > foo[array([0,2])] (array index). > > Thanks. I see this extracts the relevant rows. How about columns? I think for columns you need to transpose the matrix or use take() with the right axis=1. > Also, > where is this documented? pp 35,36,37,49,50,51 > I spent quite a while poring over the numarray > documentation. The section on index arrays looked relevant but I found it > difficult to understand. That's probably a corollary to the fact that index arrays, put, and take *are* a little difficult to understand, even more so in numarray than in Numeric. Regards, Todd > > Faheem. > > > ------------------------------------------------------- > This SF.Net email is sponsored by: IBM Linux Tutorials > Free Linux tutorial presented by Daniel Robbins, President and CEO of > GenToo technologies. Learn everything from fundamentals to system > administration.http://ads.osdn.com/?ad_id=1470&alloc_id=3638&op=click > _______________________________________________ > Numpy-discussion mailing list > Numpy-discussion at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/numpy-discussion -- Todd Miller From faheem at email.unc.edu Sat Mar 27 20:48:14 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sat Mar 27 20:48:14 2004 Subject: [Numpy-discussion] indexing question In-Reply-To: <1080443750.3735.22.camel@localhost.localdomain> References: <1080424579.3739.8.camel@localhost.localdomain> <1080443750.3735.22.camel@localhost.localdomain> Message-ID: On Sat, 27 Mar 2004, Todd Miller wrote: > On Sat, 2004-03-27 at 19:27, Faheem Mitha wrote: > > On Sat, 27 Mar 2004, Todd Miller wrote: > > > > > Try foo[[0,2]]. Note that the array indexing syntax is a little picky: > > > foo[(0,2)] (tuple index) won't work, just foo[[0,2]] (list index) or > > > foo[array([0,2])] (array index). > > > > Thanks. I see this extracts the relevant rows. How about columns? > > I think for columns you need to transpose the matrix or use take() with > the right axis=1. I see. Thanks. I think that take() does what I need. BTW, this documentation example for take looks wrong, and gives an error for me. ************************************************* >>> a1 = array([10,20,30,40]) >>> print a1[[3,5]] [40 60] >>> print take(a1,[3,5]) [40 60] ************************************************** I get >>> a1 = numarray.array([10,20,30,40]) >>> a1 array([10, 20, 30, 40]) >>> numarray.take(a1,[3,5]) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1776, in take return array._take((indices,), outarr=outarr, clipmode=clipmode) File "/usr/lib/python2.3/site-packages/numarray/numarraycore.py", line 498, in _take return ufunc._take(self, indices, **keywds) File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1579, in __call__ result = self._doit(computation_mode, woutarr, cfunc, ufargs, 0) File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1551, in _doit apply(cfunc, ufargs) IndexError: Index[1,0]=5 out of range[4] >>> a1[[3,5]] array([40, 40]) Faheem. From jmiller at stsci.edu Sun Mar 28 03:41:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Mar 28 03:41:04 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug In-Reply-To: <20040327064807.640BE1CE525@ws3-6.us4.outblaze.com> References: <20040327064807.640BE1CE525@ws3-6.us4.outblaze.com> Message-ID: <1080474026.3737.11.camel@localhost.localdomain> On Sat, 2004-03-27 at 01:48, Gary Ruben wrote: > Actually, this reproduces the problem: > > >>> from numarray import * > >>> from numarray.objects import * > >>> abs(-1.) > ObjectArray(1.0) > >>> abs(-1) > ObjectArray(1) > I'm guessing, but I think you would like to see: >>> abs(-1.) 1.0 >>> abs(1) 1 If so, this is a bug. Two things are going on: 1) numarray.objects redefines abs() 2) Internally, numarray has a concept of "rank-0" arrays which it uses to represent scalars. Apparently, numarray.objects lacks the extra code to filter rank-0 arrays out (convert back to scalars) as is done with numerical arrays. I logged this as a bug on Source Forge and it will be fixed for 1.0. Regards, Todd -- Todd Miller From jmiller at stsci.edu Sun Mar 28 04:17:09 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Mar 28 04:17:09 2004 Subject: [Numpy-discussion] indexing question In-Reply-To: References: <1080424579.3739.8.camel@localhost.localdomain> <1080443750.3735.22.camel@localhost.localdomain> Message-ID: <1080476168.3739.40.camel@localhost.localdomain> On Sat, 2004-03-27 at 23:47, Faheem Mitha wrote: > On Sat, 27 Mar 2004, Todd Miller wrote: > > > On Sat, 2004-03-27 at 19:27, Faheem Mitha wrote: > > > On Sat, 27 Mar 2004, Todd Miller wrote: > > > > > > > Try foo[[0,2]]. Note that the array indexing syntax is a little picky: > > > > foo[(0,2)] (tuple index) won't work, just foo[[0,2]] (list index) or > > > > foo[array([0,2])] (array index). > > > > > > Thanks. I see this extracts the relevant rows. How about columns? > > > > I think for columns you need to transpose the matrix or use take() with > > the right axis=1. > > I see. Thanks. I think that take() does what I need. BTW, this > documentation example for take looks wrong, and gives an error for me. > > ************************************************* > >>> a1 = array([10,20,30,40]) > >>> print a1[[3,5]] > [40 60] > >>> print take(a1,[3,5]) > [40 60] > ************************************************** > Yes, you're right, that's an error. > I get > > >>> a1 = numarray.array([10,20,30,40]) > >>> a1 > array([10, 20, 30, 40]) > >>> numarray.take(a1,[3,5]) > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1776, in > take > return array._take((indices,), outarr=outarr, clipmode=clipmode) > File "/usr/lib/python2.3/site-packages/numarray/numarraycore.py", line > 498, in _take > return ufunc._take(self, indices, **keywds) > File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1579, in > __call__ > result = self._doit(computation_mode, woutarr, cfunc, ufargs, 0) > File "/usr/lib/python2.3/site-packages/numarray/ufunc.py", line 1551, in > _doit > apply(cfunc, ufargs) > IndexError: Index[1,0]=5 out of range[4] > > >>> a1[[3,5]] > array([40, 40]) This too looks like an error. Both usages should raise an exception... and hopefully a prettier one in the future. Thanks for the questions and feedback. Regards, Todd -- Todd Miller From faheem at email.unc.edu Sun Mar 28 07:55:09 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Sun Mar 28 07:55:09 2004 Subject: [Numpy-discussion] numarray C api examples In-Reply-To: <1080424143.3739.3.camel@localhost.localdomain> References: <1080424143.3739.3.camel@localhost.localdomain> Message-ID: On Sat, 27 Mar 2004, Todd Miller wrote: > On Sat, 2004-03-27 at 12:35, Faheem Mitha wrote: > > The README in numarray-0.9/Examples refers to the non-existent file > > setup.py (at least in that directory). There are other files called > > setup.py elsewhere in the tarball, but they don't look like they are > > supposed to compile the examples. > > > > What am I missing? > > Nothing. Those are dead comments in the README (they didn't survive the > "numarray re-packaging"). There is now a single setup.py for all > numarray packages in the root numarray directory. Ok. In that case changing the README would be a good idea in order to avoid confusion. > To install the examples, just install numarray. The examples are then > found in the package numarray.examples.convolve. Ok. I see the examples are in the Debian package python2.3-numarray-ext. > If you want to write your own extension, my advice is to stick with > the numpy compatible interfaces. A close second is the numarray > "high level" interface. I see. You mean use #include "arrayobject.h" right? BTW, the high level API example in convolve uses #include "libnumarray.h" even though the documentation refers to #include "numarray.h" Which of these should be used? Actually, it might be an idea to state explicitly in 12.3-12.6 which header files should be use. Possibly include them in the example code itself. I'm considering just using the Numeric C API on the assumption that is will be easier to use (since I am a beginner). Is this assumption accurate? Thanks for all the help. Faheem. From jmiller at stsci.edu Sun Mar 28 14:15:01 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Mar 28 14:15:01 2004 Subject: [Numpy-discussion] numarray C api examples In-Reply-To: References: <1080424143.3739.3.camel@localhost.localdomain> Message-ID: <1080512070.3738.13.camel@localhost.localdomain> On Sun, 2004-03-28 at 10:54, Faheem Mitha wrote: > On Sat, 27 Mar 2004, Todd Miller wrote: > > > On Sat, 2004-03-27 at 12:35, Faheem Mitha wrote: > > > The README in numarray-0.9/Examples refers to the non-existent file > > > setup.py (at least in that directory). There are other files called > > > setup.py elsewhere in the tarball, but they don't look like they are > > > supposed to compile the examples. > > > > > > What am I missing? > > > > Nothing. Those are dead comments in the README (they didn't survive the > > "numarray re-packaging"). There is now a single setup.py for all > > numarray packages in the root numarray directory. > > Ok. In that case changing the README would be a good idea in order to > avoid confusion. Sure. > > To install the examples, just install numarray. The examples are then > > found in the package numarray.examples.convolve. > > Ok. I see the examples are in the Debian package > python2.3-numarray-ext. I don't use debian anymore so I'm not sure what goes on there. I'm glad some of the debian guys are making numarray available. > > > If you want to write your own extension, my advice is to stick with > > the numpy compatible interfaces. A close second is the numarray > > "high level" interface. > > I see. You mean use > > #include "arrayobject.h" > > right? Exactly. Using the numpy compatible API also keeps you close to Numeric in case small array performance ever becomes an issue for you and you decide you want to switch back to Numeric. It might also let others who are using Numeric use your extensions even if you choose to use numarray yourself. > > BTW, the high level API example in convolve uses > > #include "libnumarray.h" > > even though the documentation refers to > > #include "numarray.h" > Thanks for the info. Obviously this evolved over time and the docs didn't. > Which of these should be used? When in doubt, go with what the example code is doing (in the source distribution), so libnumarray.h. If you see something questionable, ask. > Actually, it might be an idea to state > explicitly in 12.3-12.6 which header files should be use. Possibly > include them in the example code itself. There is working example code in the source distribution. At one point, the example code in the manual was working as well; that's where it came from. > > I'm considering just using the Numeric C API on the assumption that is > will be easier to use (since I am a beginner). Is this assumption > accurate? I think they're both fairly easy, but the Numeric API as the advantage of backward compatibility. > Thanks for all the help. Thanks for all the feedback. Regards, Todd > Faheem. > -- Todd Miller From gazzar at email.com Sun Mar 28 16:51:17 2004 From: gazzar at email.com (Gary Ruben) Date: Sun Mar 28 16:51:17 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug Message-ID: <20040329004627.0C2411CE504@ws3-6.us4.outblaze.com> > I'm guessing, but I think you would like to see: > > >>> abs(-1.) > 1.0 > >>> abs(1) > 1 Hi Todd, Sorry if I wasn't clear. Yes, that's what I was expecting to see. Thanks for logging it and thanks for the explanation of why it's happening. I'd also like to restate the other issues in the email to which that one referred, which haven't been commented on. The cos function result >>> cos(1) 0.54030227661132813 gives a different result to cos(1.) Is this just because of numarray's stated departures from the way type coercion is done in Numeric? My guess is that numarray is casting the integer to a rank-0 Float32 array which is then coerced back to the native Python float type, which presumably is a float64 on the Windows platform. Do others think precision is being lost unnecessarily? Finally, has anyone got any comment about whether the __repr__ versus __str__ display of object array members is the expected behaviour? thanks, Gary -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From jmiller at stsci.edu Sun Mar 28 17:19:27 2004 From: jmiller at stsci.edu (Todd Miller) Date: Sun Mar 28 17:19:27 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug In-Reply-To: <20040329004627.0C2411CE504@ws3-6.us4.outblaze.com> References: <20040329004627.0C2411CE504@ws3-6.us4.outblaze.com> Message-ID: <1080522736.4077.7.camel@localhost.localdomain> On Sun, 2004-03-28 at 19:46, Gary Ruben wrote: > > I'm guessing, but I think you would like to see: > > > > >>> abs(-1.) > > 1.0 > > >>> abs(1) > > 1 > > Hi Todd, > > Sorry if I wasn't clear. Yes, that's what I was expecting to see. Thanks for logging it and thanks for the explanation of why it's happening. > > I'd also like to restate the other issues in the email to which that one referred, which haven't been commented on. The cos function result > > >>> cos(1) > 0.54030227661132813 > > gives a different result to cos(1.) > Is this just because of numarray's stated departures from the way type coercion is done in Numeric? My guess is that numarray is casting the integer to a rank-0 Float32 array which is then coerced back to the native Python float type, which presumably is a float64 on the Windows platform. This sounds correct. (Thanks for explaining it!) > Do others think precision is being lost unnecessarily? No. Do you have any suggestions? > Finally, has anyone got any comment about whether the __repr__ versus __str__ display of object array members is the expected behaviour? > Yes. The basic look is exactly what I wanted: >>> a = numarray.objects.fromlist(range(10)) >>> str(a) '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]' >>> repr(a) 'ObjectArray([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])' But I'm open to suggestions and recognize that the handling of spaces/item sizing is weak (although it's not pronounced in this simple example). Do people want something else? Regards, Todd -- Todd Miller From gazzar at email.com Sun Mar 28 20:00:46 2004 From: gazzar at email.com (Gary Ruben) Date: Sun Mar 28 20:00:46 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug Message-ID: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> > > >>> cos(1) > > 0.54030227661132813 > > > > gives a different result to cos(1.) > > Do others think precision is being lost unnecessarily? > > No. Do you have any suggestions? Yes. My problem is that numarray is not replicating the behaviour of the Python math module, which presumably coerces immediately to the float precision of the underlying platform. This is probably a float64 in Windows. If you don't explicitly specify that float32 is to be used, shouldn't the default be for a rank-0 value to immediately coerce ints to the same precision as the native Python float type on the underlying platform, since you know it will be coerced to that later and in the meantime you've lost precision because you've applied a function to a value of lower precision? > > Finally, has anyone got any comment about whether the __repr__ versus __str__ display of object array members is the expected behaviour? > > > > Yes. The basic look is exactly what I wanted: > > >>> a = numarray.objects.fromlist(range(10)) > >>> str(a) > '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]' > >>> repr(a) > 'ObjectArray([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])' > > But I'm open to suggestions and recognize that the handling of > spaces/item sizing is weak (although it's not pronounced in this simple > example). Do people want something else? > > Regards, > Todd Your example doesn't demonstrate the problem I'm talking about because for integers the __str__ and __repr__ methods are identical. However, it does show a difference between Numeric and numarray, which would display your str(a) as '[0 1 2 3 4 5 6 7 8 9 ]' If you make an ObjectArray containing objects whose __repr__ and __str__ methods are not the same, I think you need a way of displaying the contained objects via their type-specific __str__ method. Numeric did this by default, whereas numarray has changed that behaviour. I'll go back to my example again. In my example str() does not display the array elements in the same way that Numeric does. In my case, numarray does this: >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) [Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)] >>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) [Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)] >>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) ie. str() is not using the __str__ method I have defined for my Err class. Numeric correctly uses my class's Err.__str__ method and does this instead: >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] In summary, I think te behaviour should be >>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] >>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) I hope I've clarified what I'm getting at, although I probably haven't :-) Gary -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From jmiller at stsci.edu Mon Mar 29 03:57:04 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Mar 29 03:57:04 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug In-Reply-To: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> References: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> Message-ID: <1080561357.4087.18.camel@localhost.localdomain> On Sun, 2004-03-28 at 22:46, Gary Ruben wrote: > > > >>> cos(1) > > > 0.54030227661132813 > > > > > > gives a different result to cos(1.) > > > > Do others think precision is being lost unnecessarily? > > > > No. Do you have any suggestions? > > Yes. My problem is that numarray is not replicating the behaviour of the Python math module, which presumably coerces immediately to the float precision of the underlying platform. This is probably a float64 in Windows. If you don't explicitly specify that float32 is to be used, shouldn't the default be for a rank-0 value to immediately coerce ints to the same precision as the native Python float type on the underlying platform, since you know it will be coerced to that later and in the meantime you've lost precision because you've applied a function to a value of lower precision? > I see your point and I'll talk it over with Perry unless he chimes in on his own. I'm ambivalent: while it would be nice for the numarray ufuncs to act as full equivalents to the Python math functions in scalar cases, numarray has to strike a balance between managing storage space and maintaining precision and these two goals are in conflict. I think using an array package you have to be at least a little more aware of storage concerns. > > > Finally, has anyone got any comment about whether the __repr__ versus __str__ display of object array members is the expected behaviour? > > > > > > > Yes. The basic look is exactly what I wanted: Don't ask how, but I just noticed that my foot has nearly the same dimensions of my mouth. > > > > >>> a = numarray.objects.fromlist(range(10)) > > >>> str(a) > > '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]' > > >>> repr(a) > > 'ObjectArray([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])' > > > > But I'm open to suggestions and recognize that the handling of > > spaces/item sizing is weak (although it's not pronounced in this simple > > example). Do people want something else? > > > > Regards, > > Todd > > Your example doesn't demonstrate the problem I'm talking about because for integers the __str__ and __repr__ methods are identical. However, it does show a difference between Numeric and numarray, which would display your str(a) as > '[0 1 2 3 4 5 6 7 8 9 ]' > Well, almost what I wanted. This will change. > If you make an ObjectArray containing objects whose __repr__ and __str__ methods are not the same, I think you need a way of displaying the contained objects via their type-specific __str__ method. Numeric did this by default, whereas numarray has changed that behaviour. This will change too. > I'll go back to my example again. > In my example str() does not display the array elements in the same way that Numeric does. > In my case, numarray does this: > > >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) > [Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)] > > >>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) > [Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)] > > >>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) > ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) > > > ie. str() is not using the __str__ method I have defined for my Err class. > Numeric correctly uses my class's Err.__str__ method and does this instead: > > >>> print ArrayOfErr([909., 802., 677., 585.], 1.0) > [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] > > In summary, I think te behaviour should be > >>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) > [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] > > >>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) > ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) > > I hope I've clarified what I'm getting at, although I probably haven't :-) Quite clear. Stuff will change. Regards, Todd -- Todd Miller From a.schmolck at gmx.net Mon Mar 29 04:55:44 2004 From: a.schmolck at gmx.net (Alexander Schmolck) Date: Mon Mar 29 04:55:44 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug In-Reply-To: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> (Gary Ruben's message of "Mon, 29 Mar 2004 13:46:33 +1000") References: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> Message-ID: "Gary Ruben" writes: > In summary, I think te behaviour should be >>>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) > [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] > >>>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) > ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) Such behavior would seem sensible, but I don't think it's what the builtin collections do (not sure why): >>> n = 1.1 >>> n 1.1000000000000001 >>> [n] [1.1000000000000001] >>> str(n) '1.1' >>> str([n]) '[1.1000000000000001]' 'as From crasmussen at lanl.gov Mon Mar 29 08:30:01 2004 From: crasmussen at lanl.gov (Craig Rasmussen) Date: Mon Mar 29 08:30:01 2004 Subject: [Numpy-discussion] numarray C api examples In-Reply-To: <1080424143.3739.3.camel@localhost.localdomain> References: <1080424143.3739.3.camel@localhost.localdomain> Message-ID: On Mar 27, 2004, at 2:49 PM, Todd Miller wrote: > To install the examples, just install numarray. The examples are then > found in the package numarray.examples.convolve. If you want to write > your own extension, my advice is to stick with the numpy compatible > interfaces. A close second is the numarray "high level" interface. Does numarray have the equivalent of PyArray_FromDimsAndData()? This function is critical for the work I'm doing on Co-Array Python (because it is essentially a distributed array module with Co-Array syntax, it must use memory allocated by C) and for using Fortran memory for Numeric or numarray arrays. Regards, Craig From jmiller at stsci.edu Mon Mar 29 08:44:59 2004 From: jmiller at stsci.edu (Todd Miller) Date: Mon Mar 29 08:44:59 2004 Subject: [Numpy-discussion] numarray C api examples In-Reply-To: References: <1080424143.3739.3.camel@localhost.localdomain> Message-ID: <1080578609.4087.114.camel@localhost.localdomain> On Mon, 2004-03-29 at 11:25, Craig Rasmussen wrote: > On Mar 27, 2004, at 2:49 PM, Todd Miller wrote: > > > To install the examples, just install numarray. The examples are then > > found in the package numarray.examples.convolve. If you want to write > > your own extension, my advice is to stick with the numpy compatible > > interfaces. A close second is the numarray "high level" interface. > > Does numarray have the equivalent of PyArray_FromDimsAndData()? numarray's Numeric compatibility layer has that function. Because numarray uses buffer objects, numarray winds up making a copy of the data. This behavior could be changed but hasn't been yet. Regards, Todd -- Todd Miller From faheem at email.unc.edu Mon Mar 29 11:47:28 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Mon Mar 29 11:47:28 2004 Subject: [Numpy-discussion] blitz++ and numarray Message-ID: Hi, I'm considering the use of the Blitz++ C++ library (http://www.oonumerics.org/blitz/) along with numarray/Numeric. I want to pass an array down to C++ and then manipulate it using Blitz++. My current thinking is to use the Numeric interface to pass down the array, and then copy it to a blitz++ array object. This seems straightforward except I am a little concerned about those reference counting issues, which I don't properly understand (I'm a Python beginner). I do get the impression that it is mainly an issue if I am working with Python objects in C/C++ code, though, and I am mostly going to try to avoid doing so by declaring C++ objects within the C++ code and working with those as much as possible. Other possibilities include somehow using the Boost.Python package, though I am not sure how. One advantage of this is that it apparently takes care of the reference counting issue. However, I don't want to learn yet another API. Also, weave.blitz() from SciPy looks relevant/useful, but this automatically generates its own C++ code, and I'd prefer to write my own. Can anyone suggested a good basic tutorial which discusses the issues of reference counting in a C/C++ API context? I have tried reading the official documentation and will continue to do so, but something gentler would be useful. Comments, suggestions gratefully appreciated. Please cc me on any reply to faheem at email.unc.edu. Thanks in advance. Faheem. From Fernando.Perez at colorado.edu Mon Mar 29 12:26:48 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Mon Mar 29 12:26:48 2004 Subject: [Numpy-discussion] blitz++ and numarray In-Reply-To: References: Message-ID: <406883BF.5080909@colorado.edu> Faheem Mitha wrote: > Hi, > > I'm considering the use of the Blitz++ C++ library > (http://www.oonumerics.org/blitz/) along with numarray/Numeric. I want > to pass an array down to C++ and then manipulate it using Blitz++. [snip] > Also, weave.blitz() from SciPy looks relevant/useful, but this > automatically generates its own C++ code, and I'd prefer to write my > own. The following should be enough to get you started: #include "Python.h" #include "Numeric/arrayobject.h" #include "blitz/array.h" using namespace std; using namespace blitz; // Convert a Numpy array to a blitz one, using the original's data (no copy) template static Array py_to_blitz(PyArrayObject* arr_obj) { int T_size = sizeof(T); TinyVector shape(0); TinyVector strides(0); int *arr_dimensions = arr_obj->dimensions; int *arr_strides = arr_obj->strides; for (int i=0;i((T*) arr_obj->data,shape,strides,neverDeleteData); } This is what I use for exactly the problem you are describing, and this code was pretty much lifted, with minor changes, from weave's auto-generated C++. What I do, to sidestep the memory management problems, is let python allocate all my Numeric arrays (using zeros() if I have no data). I then use those inside my C++ code as blitz++ arrays via the above snippet. Any changes made by the C++ code are automatically reflected in the Numeric arary, since the blitz object is using the Numeric data area. I hope this helps. Let me know if you need more help, I can mail you complete example code. Cheers, f From gazzar at email.com Tue Mar 30 01:16:11 2004 From: gazzar at email.com (Gary Ruben) Date: Tue Mar 30 01:16:11 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand apossible bug Message-ID: <20040330060633.34B361CE511@ws3-6.us4.outblaze.com> Thanks for pointing this out Alexander, I admit that I hadn't tried any native Python container types to see how they behave. I was basing my comments on Numeric's behaviour. When I have time, maybe I should raise a question on the Python developer's list. It seems like something that may have been overlooked. regards, Gary ----- Original Message ----- From: Alexander Schmolck Date: Mon, 29 Mar 2004 13:52:20 +0100 To: "numpy-discussion" Subject: Re: [Numpy-discussion] A couple more Numeric incompatibilitiesand apossible bug > "Gary Ruben" writes: > > In summary, I think te behaviour should be > >>>> print str(ArrayOfErr([909., 802., 677., 585.], 1.0)) > > [909.0 +1.0/-1.0 802.0 +1.0/-1.0 677.0 +1.0/-1.0 585.0 +1.0/-1.0 ] > > > >>>> print repr(ArrayOfErr([909., 802., 677., 585.], 1.0)) > > ObjectArray([Err(909.0,1.0,1.0), Err(802.0,1.0,1.0), Err(677.0,1.0,1.0), Err(585.0,1.0,1.0)]) > > Such behavior would seem sensible, but I don't think it's what the builtin > collections do (not sure why): > > >>> n = 1.1 > >>> n > 1.1000000000000001 > >>> [n] > [1.1000000000000001] > >>> str(n) > '1.1' > >>> str([n]) > '[1.1000000000000001]' > > > 'as -- ___________________________________________________________ Sign-up for Ads Free at Mail.com http://promo.mail.com/adsfreejump.htm From Mailer-Daemon at kottonmouth.trouble-free.net Tue Mar 30 01:56:09 2004 From: Mailer-Daemon at kottonmouth.trouble-free.net (Mail Delivery System) Date: Tue Mar 30 01:56:09 2004 Subject: [Numpy-discussion] Mail delivery failed: returning message to sender Message-ID: This message was created automatically by mail delivery software. A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed: pete at shinners.org This message has been rejected because it has a potentially executable attachment "message_details.pif" This form of attachment has been used by recent viruses or other malware. If you meant to send this file then please package it up as a zip file and resend it. ------ This is a copy of the message, including all the headers. ------ From numpy-discussion at lists.sourceforge.net Tue Mar 30 04:55:56 2004 From: numpy-discussion at lists.sourceforge.net (numpy-discussion at lists.sourceforge.net) Date: Tue, 30 Mar 2004 11:55:56 +0200 Subject: Message Message-ID: Your document is attached. -------------- next part -------------- A non-text attachment was scrubbed... Name: message_details.pif Type: application/octet-stream Size: 17424 bytes Desc: not available URL: From cookedm at physics.mcmaster.ca Tue Mar 30 12:42:21 2004 From: cookedm at physics.mcmaster.ca (David M. Cooke) Date: Tue Mar 30 12:42:21 2004 Subject: [Numpy-discussion] A couple more Numeric incompatibilitiesand a possible bug In-Reply-To: <1080561357.4087.18.camel@localhost.localdomain> References: <20040329034633.8DA9A1CE53A@ws3-6.us4.outblaze.com> <1080561357.4087.18.camel@localhost.localdomain> Message-ID: <20040330204004.GA4588@arbutus.physics.mcmaster.ca> On Mon, Mar 29, 2004 at 06:55:57AM -0500, Todd Miller wrote: > On Sun, 2004-03-28 at 22:46, Gary Ruben wrote: > > > > >>> cos(1) > > > > 0.54030227661132813 > > > > > > > > gives a different result to cos(1.) > > > > > > Do others think precision is being lost unnecessarily? > > > > > > No. Do you have any suggestions? > > > > Yes. My problem is that numarray is not replicating the behaviour of > > the Python math module, which presumably coerces immediately to the > > float precision of the underlying platform. This is probably a > > float64 in Windows. If you don't explicitly specify that float32 is > > to be used, shouldn't the default be for a rank-0 value to > > immediately coerce ints to the same precision as the native Python > > float type on the underlying platform, since you know it will be > > coerced to that later and in the meantime you've lost precision > > because you've applied a function to a value of lower precision? > > > > I see your point and I'll talk it over with Perry unless he chimes in on > his own. I'm ambivalent: while it would be nice for the numarray > ufuncs to act as full equivalents to the Python math functions in scalar > cases, numarray has to strike a balance between managing storage space > and maintaining precision and these two goals are in conflict. I think > using an array package you have to be at least a little more aware of > storage concerns. I'll throw in a vote for 'cos(1)' coercing 1 to a (equivalent-to-) Python float. The reason is that I have 'from numarray import *' for my interactive interpreter, as I use numarray a lot in that. I would rather not have to remember to always cast my args to ufuncs to arrays. This isn't a storage space problem, per se., the question is whether cos(1) and cos(1.0) should return the same answer. Having cos(1) cast 1 to a Float32 makes numarray much less useful as a calculator -- mistakes are more likely. I'll probably switch my interpreter back to Numeric so as not to make mistakes, or I'll write a wrapper (which I'll post if I do). Whether cos([1,1,1]) and cos([1.0,1.0,1.0]) are the same I'd say is different, as there I've taken the conscious decision not to cast to an array. (And the fact that cos([1,1,1]) will return an array printed out with 'type=Float32' is a clue to what I've done.) -- |>|\/|< /--------------------------------------------------------------------------\ |David M. Cooke http://arbutus.physics.mcmaster.ca/dmc/ |cookedm at physics.mcmaster.ca From faheem at email.unc.edu Tue Mar 30 13:45:06 2004 From: faheem at email.unc.edu (Faheem Mitha) Date: Tue Mar 30 13:45:06 2004 Subject: [Numpy-discussion] blitz++ and numarray In-Reply-To: <406883BF.5080909@colorado.edu> References: <406883BF.5080909@colorado.edu> Message-ID: On Mon, 29 Mar 2004, Fernando Perez wrote: > The following should be enough to get you started: > > #include "Python.h" > #include "Numeric/arrayobject.h" > #include "blitz/array.h" > > using namespace std; > using namespace blitz; > > // Convert a Numpy array to a blitz one, using the original's data (no copy) > template > static Array py_to_blitz(PyArrayObject* arr_obj) > { > int T_size = sizeof(T); > TinyVector shape(0); > TinyVector strides(0); > int *arr_dimensions = arr_obj->dimensions; > int *arr_strides = arr_obj->strides; > > for (int i=0;i shape[i] = arr_dimensions[i]; > strides[i] = arr_strides[i]/T_size; > } > return Array((T*) arr_obj->data,shape,strides,neverDeleteData); > } > > This is what I use for exactly the problem you are describing, and this code > was pretty much lifted, with minor changes, from weave's auto-generated C++. > What I do, to sidestep the memory management problems, is let python allocate > all my Numeric arrays (using zeros() if I have no data). I then use those > inside my C++ code as blitz++ arrays via the above snippet. Any changes made > by the C++ code are automatically reflected in the Numeric arary, since the > blitz object is using the Numeric data area. > > I hope this helps. Let me know if you need more help, I can mail you complete > example code. Thanks. That is very helpful. I've incorporated this into my code, and managed to get a simple example with boost working (extracting an element of an array). See below. Do you have a similar function which converts a blitz array to a numarray/Numeric one? If not, I can easily concoct one for myself. I now need to try to figure out what this reference counting stuff is about. I read the official Python docs but am still currently very confused. Faheem. *************************************************************************** #include "Python.h" #include "Numeric/arrayobject.h" #include using namespace std; using namespace blitz; template static Array py_to_blitz(PyArrayObject* arr_obj); static PyObject * arrmod_elem(PyObject *self, PyObject *args); static PyMethodDef arrmod_methods[] = { {"elem", (PyCFunction)arrmod_elem, METH_VARARGS, "Returns the trace of a two-dimensional array.\n"}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initarrmod(void) { (void) Py_InitModule3("arrmod", arrmod_methods, "Returns the Trace of a two-dimensional array.\n"); import_array(); } static PyObject * arrmod_elem(PyObject *self, PyObject *args) { PyObject *input, *result; PyArrayObject *array; double el; int i, j; if (!PyArg_ParseTuple(args, "Oii", &input, &i, &j)) return NULL; array = (PyArrayObject *) PyArray_ContiguousFromObject(input, PyArray_DOUBLE, 2, 2); if (array == NULL) return NULL; Array arr = py_to_blitz(array); el = arr(i,j); result = Py_BuildValue("d",el); return result; } // Convert a Numpy array to a blitz one, using the original's data (no // copy) template static Array py_to_blitz(PyArrayObject* arr_obj) { int T_size = sizeof(T); TinyVector shape(0); TinyVector strides(0); int *arr_dimensions = arr_obj->dimensions; int *arr_strides = arr_obj->strides; for (int i=0;i((T*) arr_obj->data,shape,strides,neverDeleteData); } From Fernando.Perez at colorado.edu Tue Mar 30 13:50:15 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Tue Mar 30 13:50:15 2004 Subject: [Numpy-discussion] blitz++ and numarray In-Reply-To: References: <406883BF.5080909@colorado.edu> Message-ID: <4069EB78.2060609@colorado.edu> > Thanks. That is very helpful. I've incorporated this into my code, and > managed to get a simple example with boost working (extracting an element > of an array). See below. Glad it helped. > Do you have a similar function which converts a blitz array to a > numarray/Numeric one? If not, I can easily concoct one for myself. No, I don't have one. But it's pretty straightforward to write one, the Numeric C API provides functions to create arrays from existing data. You'll be surprised how conceptually similar Blitz and Numeric turn out to be (which is a good thing). > I now need to try to figure out what this reference counting stuff is > about. I read the official Python docs but am still currently very > confused. As I said, so far I've been able to bypass this issue by creating my arrays in python and 'filling' them in blitz. But this is not always a realistic possibility, so you may indeed have to learn the reference counting management details. Best, f From perry at stsci.edu Wed Mar 31 07:41:08 2004 From: perry at stsci.edu (Perry Greenfield) Date: Wed Mar 31 07:41:08 2004 Subject: [Numpy-discussion] type coercion question In-Reply-To: <20040330204004.GA4588@arbutus.physics.mcmaster.ca> Message-ID: David Cooke wrote: > On Mon, Mar 29, 2004 at 06:55:57AM -0500, Todd Miller wrote: > > On Sun, 2004-03-28 at 22:46, Gary Ruben wrote: > > > > > >>> cos(1) > > > > > 0.54030227661132813 > > > > > > > > > > gives a different result to cos(1.) > > > > > > > > Do others think precision is being lost unnecessarily? > > > > > > > > No. Do you have any suggestions? > > > > > > Yes. My problem is that numarray is not replicating the behaviour of > > > the Python math module, which presumably coerces immediately to the > > > float precision of the underlying platform. This is probably a > > > float64 in Windows. If you don't explicitly specify that float32 is > > > to be used, shouldn't the default be for a rank-0 value to > > > immediately coerce ints to the same precision as the native Python > > > float type on the underlying platform, since you know it will be > > > coerced to that later and in the meantime you've lost precision > > > because you've applied a function to a value of lower precision? > > > > > > > I see your point and I'll talk it over with Perry unless he chimes in on > > his own. I'm ambivalent: while it would be nice for the numarray > > ufuncs to act as full equivalents to the Python math functions in scalar > > cases, numarray has to strike a balance between managing storage space > > and maintaining precision and these two goals are in conflict. I think > > using an array package you have to be at least a little more aware of > > storage concerns. > > I'll throw in a vote for 'cos(1)' coercing 1 to a (equivalent-to-) > Python float. The reason is that I have 'from numarray import *' for my > interactive interpreter, as I use numarray a lot in that. > I would rather not have to remember to always cast my args to ufuncs to > arrays. This isn't a storage space problem, per se., the question is > whether cos(1) and cos(1.0) should return the same answer. > > Having cos(1) cast 1 to a Float32 makes numarray much less useful as a > calculator -- mistakes are more likely. I'll probably switch my > interpreter back to Numeric so as not to make mistakes, or I'll write a > wrapper (which I'll post if I do). > > Whether cos([1,1,1]) and cos([1.0,1.0,1.0]) are the same I'd say is > different, as there I've taken the conscious decision not to cast to an > array. (And the fact that cos([1,1,1]) will return an array printed out > with 'type=Float32' is a clue to what I've done.) > Let me say that I see the last issue (whether cos([1,1,1]) and cos([1.0,1.0,1.0]) are the same) is important. If cos([1,1,1])[0] != cos(1) I think that is just as bad. So the question is whether ints scalars and arrays should automatically be promoted to double floats when given to unary scientific functions (sqrt, sin, cos, log, etc.). By the way, if an Int32 is so promoted, I don't see why it isn't the same for Int8, Int16, etc. But it appears that Numeric promotes Int32 to Float64 for cos(), but promotes In16 to Float32 for cos(). I believe making this change to numarray is fairly easy (we'd have to check). But is what most people want? Should we retain complete compatibility with Numeric (which, as pointed out, treats different kinds of integers differently) Just to summarize what I see as acceptable choices: 1) ints promote to Float32 (current behavior) 2) ints promote to Float64 3) Int32 promotes to Float64, lesser ints promote to Float32 (Numeric behavior) Perry From Chris.Barker at noaa.gov Wed Mar 31 09:23:05 2004 From: Chris.Barker at noaa.gov (Chris Barker) Date: Wed Mar 31 09:23:05 2004 Subject: [Numpy-discussion] type coercion question In-Reply-To: References: Message-ID: <9D5D9B78-8338-11D8-9C14-000393A96660@noaa.gov> On Mar 31, 2004, at 7:39 AM, Perry Greenfield wrote: > Just to summarize what I see as acceptable choices: > > 1) ints promote to Float32 (current behavior) > 2) ints promote to Float64 > 3) Int32 promotes to Float64, lesser ints promote to Float32 (Numeric > behavior) +1 on (2) I want math.cos(x) to give the same answer as numarray.cos( array( (x) ) ) as much as possible. I like that Numarray does not do as much auto-upcasting as Numeric, but if I want to control the size of my data arrays, I'm not going to be passing an int into a floating point function. i.e, I can always do: cos( array( (x), Float32 ) ) Then I would want the result to be a Float32 -Chris Christopher Barker, Ph.D. Oceanographer NOAA/OR&R/HAZMAT (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From rlw at stsci.edu Wed Mar 31 09:55:36 2004 From: rlw at stsci.edu (Rick White) Date: Wed Mar 31 09:55:36 2004 Subject: [Numpy-discussion] type coercion question In-Reply-To: Message-ID: On Wed, 31 Mar 2004, Perry Greenfield wrote: > Just to summarize what I see as acceptable choices: > > 1) ints promote to Float32 (current behavior) > 2) ints promote to Float64 > 3) Int32 promotes to Float64, lesser ints promote to Float32 (Numeric > behavior) I'm also +1 on option (2). I believe that this will not cause problems with unwanted promotions. The result of cos(1) is a scalar Python float, regardless of whether 1 is converted to a Float32 or a Float64 before the cosine is evaluated. Then if that scalar is used with a Float32 numarray, it gets converted to a Float32 and the result is Float32. For example: >>> from numarray import * >>> cos(1)*ones(1,Float32) array([ 0.54030228], type=Float32) >>> cos(1.0)*ones(1,Float32) array([ 0.54030228], type=Float32) So it seems to me there is no downside to converting integers to Float64 when the ufunc requires a float. And I find it hard to believe that any Numeric users are counting on behavior (3). Rick From Fernando.Perez at colorado.edu Wed Mar 31 09:58:05 2004 From: Fernando.Perez at colorado.edu (Fernando Perez) Date: Wed Mar 31 09:58:05 2004 Subject: [Numpy-discussion] type coercion question In-Reply-To: <9D5D9B78-8338-11D8-9C14-000393A96660@noaa.gov> References: <9D5D9B78-8338-11D8-9C14-000393A96660@noaa.gov> Message-ID: <406B0684.9000600@colorado.edu> Chris Barker wrote: > On Mar 31, 2004, at 7:39 AM, Perry Greenfield wrote: > >>Just to summarize what I see as acceptable choices: >> >>1) ints promote to Float32 (current behavior) >>2) ints promote to Float64 >>3) Int32 promotes to Float64, lesser ints promote to Float32 (Numeric >>behavior) > > > +1 on (2) > > I want math.cos(x) to give the same answer as numarray.cos( array( (x) > ) ) as much as possible. I like that Numarray does not do as much > auto-upcasting as Numeric, but if I want to control the size of my data > arrays, I'm not going to be passing an int into a floating point > function. i.e, I can always do: > > cos( array( (x), Float32 ) ) > > Then I would want the result to be a Float32 +1 on (2) as well, based on a similar reasoning to Chris (who saved me the time of typing it up cleanly :) Cheers, f From cjw at sympatico.ca Wed Mar 31 12:11:19 2004 From: cjw at sympatico.ca (Colin J. Williams) Date: Wed Mar 31 12:11:19 2004 Subject: [Numpy-discussion] type coercion question In-Reply-To: References: Message-ID: <406B25C7.5020902@sympatico.ca> Rick White wrote: >On Wed, 31 Mar 2004, Perry Greenfield wrote: > > > >>Just to summarize what I see as acceptable choices: >> >>1) ints promote to Float32 (current behavior) >>2) ints promote to Float64 >>3) Int32 promotes to Float64, lesser ints promote to Float32 (Numeric >>behavior) >> >> > >I'm also +1 on option (2). > I'm also +1 on option (2). Colin W.