Mailman 3 May 2009 - NumPy-Discussion

List/location of consecutive integers (2)
by Andrea Gavana 27 May '09

27 May '09

Hi All, I have tried the solutions proposed in the previous thread and it looks like Chris' one is the fastest for my purposes. Now, I have a question which is probably more conceptual than implementation-related. I started this little thread as my task is to read medium to (relatively) big unformatted binary files written by another (black-box) software (which is written in Fortran). These files can range from 10 MB to 200 MB, more or less, and I read them using a f2py-wrapped Fortran subroutine. I got a stupendous speed improvement when I switched from Compaq Visual Fortran to G95 with "STREAM" access (from 8% to 90% faster, depending on the infamous "indices" I was talking about). Now, I was thinking about using the multiprocessing module in Python, as we have 4-cpus PCs at work and I could try to call my subroutine using multiple Python processes. I *really* should do this in Fortran directly but I haven't found any reference on how to do file I/O in parallel in Fortran and I haven't got any help from comp.lang.fortran in that sense (only a warning that I may slow down everything by using multiple processes). Splitting the reading process between 4 processes will require the exchange of 5-20 MB from the child processes to the main one: do you think my script will benefit from using multiprocessing? Is there any drawback in using Numpy arrays in multiple processes? If using multiprocessing in Python will create too much overhead, does anyone have any suggestion/reference/link/code on how to handle parallel I/O in Fortran directly? Should I try another approach? Thank you a lot for your suggestions. Andrea. "Imagination Is The Only Weapon In The War Against Reality." http://xoomer.alice.it/infinity77/ http://thedoomedcity.blogspot.com/

3 2

Best way to inherit from PyArrayIterObject at the C level ?
by David Cournapeau 27 May '09

27 May '09

Hi, I have been scratching my head on the following problem. I am designing a new array iterator, in C, to walk into a neighborhood of an array. I would like this iterator to 'inherit' from PyArrayIterObject, so that I can design some API which accept both PyArrayIterObject and PyArrayNeighIterObject (through pointer casting). I have tried something like: typedef struct { /* first item is the base class, so that casting a PyArrayNeighIterObject* to PyArrayIterObject* works */ PyArrayIterObject base; /* PyArrayNeighIterObject specific members */ .... } PyArrayNeighIterObject; But this forces me to cast a PyArrayNeighIterObject* to PyArrayIterObject whenever I want to access members of the base instance. The alternative is to copy the PyArrayIterObject members by hand, as is currently done in numpy itself (for broadcasting iterator PyArrayMapIterObject). But since my iterator lives outside numpy, this is really error-prone IMHO - there will be crashes whenever the PyArrayIterObject struct changes in an ABI incompatible way, and this may be quite hard to debug Is there a better way ? David

1 0

help Can you teach me how to used array api in C/C+
by chaow porkaew 27 May '09

27 May '09

Can you teach me how to used array api in C/C++? 1.How to get a data in co-ordinate i,j , example a = array([[1,2,3],[4,5,6]]) how do i get the value of 5 in c/c++ or 2.How i sum all of data in arrays in c/c++ Best regards.

1 0

Can you teach me how to used array api in C/C++?
by chaow porkaew 27 May '09

27 May '09

Can you teach me how to used array api in C/C++? 1.How to get a data in co-ordinate i,j , example a = array([[1,2,3],[4,5,6]]) how do i get the value of 5 in c/c++ or 2.How i sum all of data in arrays in c/c++

1 0

Re: [Numpy-discussion] Home for pyhdf5io?
by Francesc Alted 27 May '09

27 May '09

Hello Albert, A Thursday 21 May 2009 22:32:10 escriguéreu: > Hi, > First of all thanks for your work on PyTables! I think it is excellent > and it has been really nice working with it. > > I'm writing this because i have developed a small python module that > uses pyTables: > > http://code.google.com/p/pyhdf5io/ > > It basically implements load/save functions that mimic the behaviour > of those found in Matlab, i.e. with them you can store your variables > from within the interactive shell (IPython, python) or from within a > function, and then load them back in again. I've been having a look at your module and seems pretty cute. Incidentally, there is another module module that does similar things: http://www.elisanet.fi/ptvirtan/software/hdf5pickle/index.html However, I do like your package better in the sense that it adds more 'magic' to the load/save routines. But maybe you want to have a look at the above: it can give you more ideas, like for example, using CArrays and compression for very large arrays, or Tables for structured arrays. > And now to the question: > > I think that this module is to small to be developed and maintained on > its own. I think It would be better if it could be part of some larger > project, maybe pyTables, I don't know. Sure. I think it could fit perfectly as a module inside PyTables, in the same wave than 'filenode' and 'netcdf3'. Most of your module can be dropped as-is into the PyTables module hierarchy. However, it would be nice if you can write some documentation following the format of User's Guide chapters like the ones about 'filenode' or 'netcdf3' modules. Please, let's continue the discussion in the PyTables list in case we need to. Thanks for contributing! -- Francesc Alted

9 28

Re: [Numpy-discussion] add xirr to numpy financial functions?
by Joe Harrington 26 May '09

26 May '09

Let's keep this thread focussed on the original issue: just add a floating array of times to irr or a new xirr continuous interest no more Anyone can use the timeseries package to produce a floating array of times from normal dates, if those are the dates they want. If they want some specialized financial date, they may want a different conversion, however. All we should provide in NumPy would be the simplest tool. Specialized dates and date-time conversion belong elsewhere. If we're *not* skipping dates, there is no need for xirr, just use irr, which exists. scikits.financial seems like a great idea, and then knock yourselves out for date conversions and definitions of compounding. Just think big and design it first. But let's keep this thread on the simple question for NumPy. --jh--

4 4

Re: [Numpy-discussion] add xirr to numpy financial functions?
by Joe Harrington 26 May '09

26 May '09

On Mon, 25 May 2009 13:51:38 -0400, josef.pktd(a)gmail.com wrote: > On Mon, May 25, 2009 at 11:50 AM, Joe Harrington <jh(a)physics.ucf.edu> wrote: > > On Sun, 24 May 2009 18:14:42 -0400 josef.pktd(a)gmail.com wrote: > >> On Sun, May 24, 2009 at 4:33 PM, Joe Harrington <jh(a)physics.ucf.edu> wrote: > >> > I hate to ask for another function in numpy, but there's an obvious > >> > one missing in the financial group: xirr. ?It could be done as a new > >> > function or as an extension to the existing np.irr. > >> > > >> > The internal rate of return (np.irr) is defined as the growth rate > >> > that would give you a zero balance at the end of a period of > >> > investment given a series of cash flows into or out of the investment > >> > at regular intervals (the first and last cash flows are usually an > >> > initial deposit and a withdrawal of the current balance). > >> > > >> > This is useful in academics, but if you're tracking a real investment, > >> > you don't just withdraw or add money on a perfectly annual basis, nor > >> > do you want a calc with thousands of days of zero entries just so you > >> > can handle the uneven intervals by evening them out. ?Both excel and > >> > openoffice define a "xirr" function that pairs each cash flow with a > >> > date. ?Would there be an objection to either a xirr or adding an > >> > optional second arg (or a keyword arg) to np.irr in numpy? ?Who writes > >> > the code is a different question, but that part isn't hard. > >> > > >> > >> > >> > >> 3 comments: > >> > >> * open office has also the other function in an x??? version, so it > >> might be good to add it consistently to all functions > >> > >> * date type: scikits.timeseries and the gsoc for implementing a date > >> type would be useful to have a clear date type, or would you want to > >> base it only on python standard library > >> > >> * real life accuracy: given that there are large differences in the > >> definition of a year for financial calculations, any simple > >> implementation would be only approximately accurate. for example in > >> the open office help, oddlyield list the following option > >> > >> Basis is chosen from a list of options and indicates how the year is > >> to be calculated. > >> Basis Calculation > >> 0 or missing US method (NASD), 12 months of 30 days each > >> 1 Exact number of days in months, exact number of days in year > >> 2 Exact number of days in month, year has 360 days > >> 3 Exact number of days in month, year has 365 days > >> 4 European method, 12 months of 30 days each > >> > >> So, my question: what's the purpose of the financial function in numpy? > >> Currently it provides convenient functions for (approximate) interest > >> calculations. > >> If they get expanded to a "serious" implementation of, for example, > >> the main financial functions listed in the open office help (just for > >> reference) then maybe numpy is not the right location for it. > >> > >> I started to do something similar in matlab, and once I tried to use > >> real dates instead of just counting months, the accounting rules get > >> quickly very messy. > >> > >> Using dates as you propose would be very convenient, but the users > >> shouldn't be surprised that their actual payments at the end of the > >> year don't fully match up with what numpy told them. > >> > >> my 3cents > >> > >> Josef > > > > First point: agreed. ?I wish this community had a design review > > process for numpy and scipy, so that these things could get properly > > hashed out, and not just one person (even Travis) suggesting something > > and everyone else saying yeah-sure-whatever. > > > > Does anyone on the list have the financial background to suggest what > > functions "should" be included in a basic set of financial routines? > > xirr is the only one I've ever used in a spreadsheet, myself. > > > > Other points: Yuk. ?You're right. > > > > When these first came up for discussion, I had a Han Solo moment > > ("I've got a baaad feeling about this...") but I couldn't put my > > finger on why. ?They seemed like simple and limited functions with > > high utility. ?Certainly anything as open-ended as financial-industry > > rules should go elsewhere (scikits, scipy, monpy, whatever). > > > > But, that doesn't prevent a user-supplied, floating-point time array > > from going into a function in numpy. ?The rate of return would be in > > units of that array. ?Functions that convert date/time in some format > > (or many) and following some rule (or one of many) to such a floating > > array can still go elsewhere, maintained by people who know the > > definitions, if they have interest (pun intended). ?That would make > > the functions in numpy much more useful without bloating them or > > making them a maintenance nightmare. > > > > If you think of time just as a regularly spaced, e.g. days, but with > sparse points on it, or as a continuous variable, then extending the > current functions should be relatively easy. I guess the only > questions are compounding, annual, quarterly or at each payment, and > whether the annual rate is calculated as real compounded annualized > rate or as accounting annual rate, e.g. quarterlyrate*4. > > This leaves "What is the present value, if you get 100 Dollars at the > 10th day of each month (or at the next working day if the 10th day is > a holiday or a weekend) for the next 5 years and the monthly interest > rate is 5/12%?" for another day. > > Initially I understood you wanted the date as a string or date type as > in e.g open office. What would be the units of the user-supplied, > floating-point time array? > It is still necessary to know the time units to provide an annualized > rate, unless the rate is in continuous time, exp(r*t). I don't know > whether this would apply to all functions in numpy.finance, it's a > while since I looked at the code. Maybe there are some standard > simplifications in open office or excel. > > I briefly skimmed the list of function in the open office help, and it > would be useful to have them available, e.g. as a package in scipy. > But my google searches in the past for applications in finance with a > compatible license didn't provide much useful code that could form the > basis of a finance package. > > Adding more convenience and functionality to numpy.finance is useful, > but if they get extended with slow feature creep, then another > location (scipy) might be more appropriate and would be more > expandable, even if it happens only slowly. > > That's just my opinion (obviously), I'm a relative newbie to > numpy/scipy and still working my way through all the different > subpackages. np.irr is defined on (anonymous) constant time intervals and gives you the growth per time interval. The code is very short, basically a call to np.roots(values): def irr(values): """ Return the Internal Rate of Return (IRR). This is the rate of return that gives a net present value of 0.0. Parameters ---------- values : array_like, shape(N,) Input cash flows per time period. At least the first value would be negative to represent the investment in the project. Returns ------- out : float Internal Rate of Return for periodic input values. Examples -------- >>> np.irr([-100, 39, 59, 55, 20]) 0.2809484211599611 """ res = np.roots(values[::-1]) # Find the root(s) between 0 and 1 mask = (res.imag == 0) & (res.real > 0) & (res.real <= 1) res = res[mask].real if res.size == 0: return np.nan rate = 1.0/res - 1 if rate.size == 1: rate = rate.item() return rate So, I think this is a continuous definition of growth, not some periodic compounding. I'd propose the time array would be in anonymous units, and the result would be in terms of those units. For example, if an interval of 1.0 in the time array were one fortnight, it would give interest in units of continuous growth per fortnight, etc. Anything with many more options than that does not belong in numpy (but it would be interesting to have elsewhere). --jh--

8 19

Segmentation fault on large arrays
by Nicolas Rougier 26 May '09

26 May '09

Hello, I've come across what is probably a bug in size check for large arrays: >>> import numpy >>> z1 = numpy.zeros((255*256,256*256)) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: dimensions too large. >>> z2 = numpy.zeros((256*256,256*256)) >>> z2.shape (65536, 65536) >>> z2[0] = 0 Segmentation fault Note that z1 size is smaller than z2 but z2 is not told that its dimensions are too large. This has been tested with numpy 1.3.0. Nicolas

2 1

CUDA
by Brennan Williams 26 May '09

26 May '09

Not a question really but just for discussion/pie-in-the-sky etc.... This is a news item on vizworld about getting Matlab code to run on a CUDA enabled GPU. http://www.vizworld.com/2009/05/cuda-enable-matlab-with-gpumat/ If the use of GPU's for numerical tasks takes off (has it already?) then Id be interested to know the views of the numpy experts out there. Cheers Brennan

7 7

parsing text strings/files in fromfile, fromstring
by Charles R Harris 26 May '09

26 May '09

Hi All, I am trying to put together some rule for parsing text strings/files in fromfile, fromstring so that the two are consistent. Tickets relevant to this are #1116 <http://projects.scipy.org/numpy/ticket/1116> and #883<http://projects.scipy.org/numpy/ticket/883>. The question here is the interpretation of the separators, not the parsing of the numbers themselves. Below is the current behavior of fromstring, fromfile, and python split for content of "", "1", "1 1", " " respectively. fromstring : In [5]: fromstring("", sep=" ") Out[5]: array([ 0.]) In [6]: fromstring("1", sep=" ") Out[6]: array([ 1.]) In [7]: fromstring("1 1", sep=" ") Out[7]: array([ 1., 1.]) In [8]: fromstring(" ", sep=" ") Out[8]: array([ 0.]) fromfile: In [1]: fromfile("tmp", sep=" ") Out[1]: array([], dtype=float64) In [2]: fromfile("tmp", sep=" ") Out[2]: array([ 1.]) In [3]: fromfile("tmp", sep=" ") Out[3]: array([ 1., 1.]) In [4]: fromfile("tmp", sep=" ") Out[4]: array([ 0.]) split: In [9]: "".split(" ") Out[9]: [''] In [10]: "1".split(" ") Out[10]: ['1'] In [11]: "1 1".split(" ") Out[11]: ['1', '1'] In [12]: " ".split(" ") Out[12]: ['', ''] Differences: 1) When the string/file is empty fromfile returns and empty array, split returns an empty string, and fromstring converts the empty string to a default value. Which should we use? 2) When the string/file contains only a single seperator fromfile/fromstring both return a single value, while split returns two empty strings. Which should we use? My preferences would be to return empty arrays whenever the string/file is empty, but I don't feel strongly about that. I think the single separator should definitely produce two values. Also, wouldn't a missing value be better interpreted as nan than zero in the float case? Chuck

4 4