From jr at sun.ac.za  Tue Apr  1 07:42:42 2014
From: jr at sun.ac.za (Johann Rohwer)
Date: Tue, 01 Apr 2014 13:42:42 +0200
Subject: [SciPy-User] Connect Input File Pysces
In-Reply-To: <loom.20140331T231537-721@post.gmane.org>
References: <loom.20140331T231537-721@post.gmane.org>
Message-ID: <533AA632.9030603@sun.ac.za>

I am one of the co-authors of the PySCeS package but unfortunately I am 
unclear as to what exactly your problem is. What do you mean by "connecting" 
the PySCeS input file?

Regards
Johann

On 31/03/2014 23:15, Syry wrote:
> Hi! I'm using Pysces to reproduce the cell metabolism. I want to say if is
> possible to connect the Pysces Input File. I need to connect the input file
> beacause the output of the first input file is the input of the second.
> There is anyone that can help me?
> Thanks
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>




From sole155 at tiscali.it  Tue Apr  1 12:41:06 2014
From: sole155 at tiscali.it (Syry)
Date: Tue, 1 Apr 2014 16:41:06 +0000 (UTC)
Subject: [SciPy-User] Connect Input File Pysces
References: <loom.20140331T231537-721@post.gmane.org>
	<533AA632.9030603@sun.ac.za>
Message-ID: <loom.20140401T174435-197@post.gmane.org>

Johann Rohwer <jr <at> sun.ac.za> writes:

> 
> I am one of the co-authors of the PySCeS package but unfortunately I am 
> unclear as to what exactly your problem is. What do you mean by 
"connecting" 
> the PySCeS input file?
> 
> Regards
> Johann
> 
> On 31/03/2014 23:15, Syry wrote:
> > Hi! I'm using Pysces to reproduce the cell metabolism. I want to say if 
is
> > possible to connect the Pysces Input File. I need to connect the input 
file
> > beacause the output of the first input file is the input of the second.
> > There is anyone that can help me?
> > Thanks
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User <at> scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> 
Hi, I'm sorry if I'm not clear!
I utilise pysces to simulate the hepatocyte cell metabolism. I have 
implemented glycolysis pathway with an input file. Then, I have made the 
parsing of this input file on Python. Now the results of this simulation 
(Pyruvate, ATP) are the input of the second metabolism pathway (Kreb's 
cycle). I don't know if I have to put the later pathway  into my input file 
(and doing an enormous input file with many rate equation) or I have to 
create another input file but in this case I have to get Pyruvato and ATP 
from the simulation and pass them to the second input file
I hope it is more clear now!

Regards

Syry





From sole155 at tiscali.it  Tue Apr  1 12:46:21 2014
From: sole155 at tiscali.it (Syry)
Date: Tue, 1 Apr 2014 16:46:21 +0000 (UTC)
Subject: [SciPy-User] Connect Input File Pysces
References: <loom.20140331T231537-721@post.gmane.org>
Message-ID: <loom.20140401T184537-991@post.gmane.org>

Syry <sole155 <at> tiscali.it> writes:



Hi, I'm sorry if I'm not clear!
I utilise pysces to simulate the hepatocyte cell metabolism. I have 
implemented glycolysis pathway with an input file. Then, I have made the 
parsing of this input file on Python. Now the results of this simulation 
(Pyruvate, ATP) are the input of the second metabolism pathway (Kreb's 
cycle). I don't know if I have to put the later pathway  into my input file 
(and doing an enormous input file with many rate equation) or I have to 
create another input file but in this case I have to get Pyruvato and ATP 
from the simulation and pass them to the second input file
I hope it is more clear now!

Regards

Syry




From sole155 at tiscali.it  Tue Apr  1 12:50:32 2014
From: sole155 at tiscali.it (Syry)
Date: Tue, 1 Apr 2014 16:50:32 +0000 (UTC)
Subject: [SciPy-User] Connect Input File Pysces
References: <loom.20140331T231537-721@post.gmane.org>
	<533AA632.9030603@sun.ac.za>
	<loom.20140401T174435-197@post.gmane.org>
Message-ID: <loom.20140401T184723-307@post.gmane.org>

Hi, I'm sorry if I'm not clear!
I utilise pysces to simulate the hepatocyte cell metabolism. I have 
implemented glycolysis pathway with an input file. Then, I have made the 
parsing of this input file on Python. Now the results of this simulation 
(Pyruvate, ATP) are the input of the second metabolism pathway (Kreb's 
cycle). I don't know if I have to put the later pathway  into my input file 
(and doing an enormous input file with many rate equation) or I have to 
create another input file but in this case I have to get Pyruvato and ATP 
from the simulation and pass them to the second input file
I hope it is more clear now!

 
Regards

Syry
 






From chocold12 at gmail.com  Wed Apr  2 12:24:38 2014
From: chocold12 at gmail.com (lily li)
Date: Wed, 2 Apr 2014 10:24:38 -0600
Subject: [SciPy-User] about plot
Message-ID: <CAN5afy-ssp+80sNDJ8A5w6Vmumty8+B7HOu8ocRm+Z50THSTwg@mail.gmail.com>

Hi Scipy users,

I'd like to ask that when plotting lines between two pairs of points, how
to avoid scientific notation? For example, 100,000 is show as 1 +10e5, but
I want it to show as 100,000. Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140402/f4b4ac82/attachment.html>

From lists at hilboll.de  Wed Apr  2 12:27:41 2014
From: lists at hilboll.de (Andreas Hilboll)
Date: Wed, 02 Apr 2014 18:27:41 +0200
Subject: [SciPy-User] about plot
In-Reply-To: <CAN5afy-ssp+80sNDJ8A5w6Vmumty8+B7HOu8ocRm+Z50THSTwg@mail.gmail.com>
References: <CAN5afy-ssp+80sNDJ8A5w6Vmumty8+B7HOu8ocRm+Z50THSTwg@mail.gmail.com>
Message-ID: <533C3A7D.9090402@hilboll.de>

On 02.04.2014 18:24, lily li wrote:
> Hi Scipy users,
> 
> I'd like to ask that when plotting lines between two pairs of points,
> how to avoid scientific notation? For example, 100,000 is show as 1
> +10e5, but I want it to show as 100,000. Thanks!

Hi Lily,

questions about matplotlib are best posted to the matplotlib mailing
list or to stackoverflow.

In short, you can achieve what you want using the set_xticklabels() /
set_yticklabels() functions.

-- Andreas.


From a.klein at science-applied.nl  Wed Apr  2 16:15:22 2014
From: a.klein at science-applied.nl (Almar Klein)
Date: Wed, 2 Apr 2014 22:15:22 +0200
Subject: [SciPy-User] ANN: IEP 3.4 - the Interactive Editor for Python (now
	with IPython)
Message-ID: <CAB_T6=mc8KAaj0yRSBncBxdAOHyYrbK9G0OQsRvx5880M9zR3Q@mail.gmail.com>

Dear all,

We are pleased to announce that IEP version 3.4 is now available for
download.

IEP is a cross-platform Python IDE focused on interactivity and
introspection, which makes it very suitable for scientific computing. Its
practical design is aimed at simplicity and efficiency.

This version is marked by many improvements to the shell. Most notably, we
now have *IPython integration*! Further, there are more ways to customize
the shell (e.g. define command line and environment args), there is support
for coloured text, you can click on filenames in tracebacks to open them,
and it just behaves better. Also, many improvements have been made to the
debug system.

Website: http://iep-project.org
Release notes: https://bitbucket.org/iep-project/iep/wiki/Release_notes

To install:

   - Binaries are available for Windows, OSX and Linux.
   - conda install -c pyzo iep  (in a p33 env)
   - pip install iep  (IEP depends on Python3 and PySide/PyQt4.)

Happy coding,
  Almar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140402/9b2914f3/attachment.html>

From leah at numfocus.org  Wed Apr  2 18:39:49 2014
From: leah at numfocus.org (Leah Silen)
Date: Wed, 2 Apr 2014 17:39:49 -0500
Subject: [SciPy-User] PyData Proposals
Message-ID: <85ED038C-D97B-42B8-AB20-F67215A6DF0F@numfocus.org>

Be sure and get your proposals in for PyData Silicon Valley 2014 by April 6! 

The event brings together scientists, analysts, developers, engineers, architects and others from the Python data science community to discuss new techniques and tools for management, analytics and visualization of data. Presentation content can be at a novice, intermediate or advanced level. Talks will run 30-40 min and hands-on tutorial and workshop sessions will run 90 min. or 3 hours.

If you are interested in presenting a talk, tutorial or sponsor workshop on meeting the challenges of big data using Python, please submit a SHORT abstract and bio. If you don't have time to create an abstract, feel free to submit a presentation outline. To see the type of topics presented at previous PyData events, please look at our past conference sites at pydata.org or check out the videos on vimeo.com/pydata.

Submit Your Proposal

Registration will be complimentary for speakers.
Any questions can be directed to admin at pydata.org.
Conference dates are May 2-4.
Proposal deadline has been extended through April 6, 2014.
---
Leah Silen
leah at numfocus.org
leah at pydata.org
512.222.5449











-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140402/0d1d3093/attachment.html>

From gary.ruben at gmail.com  Thu Apr  3 08:34:46 2014
From: gary.ruben at gmail.com (gary ruben)
Date: Thu, 3 Apr 2014 23:34:46 +1100
Subject: [SciPy-User] about plot
In-Reply-To: <533C3A7D.9090402@hilboll.de>
References: <CAN5afy-ssp+80sNDJ8A5w6Vmumty8+B7HOu8ocRm+Z50THSTwg@mail.gmail.com>
	<533C3A7D.9090402@hilboll.de>
Message-ID: <CAExsOdapia0uHZkNC90LLausvT7vBEZtusA3=FcFzcO5dPRh9g@mail.gmail.com>

Hi Lily,

See also this answer to a related question:
https://stackoverflow.com/questions/14775040/matplotlib-axis-label-format

Gary R


On 3 April 2014 03:27, Andreas Hilboll <lists at hilboll.de> wrote:

> On 02.04.2014 18:24, lily li wrote:
> > Hi Scipy users,
> >
> > I'd like to ask that when plotting lines between two pairs of points,
> > how to avoid scientific notation? For example, 100,000 is show as 1
> > +10e5, but I want it to show as 100,000. Thanks!
>
> Hi Lily,
>
> questions about matplotlib are best posted to the matplotlib mailing
> list or to stackoverflow.
>
> In short, you can achieve what you want using the set_xticklabels() /
> set_yticklabels() functions.
>
> -- Andreas.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140403/c1fc09f6/attachment.html>

From ndbecker2 at gmail.com  Thu Apr  3 09:16:49 2014
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 03 Apr 2014 09:16:49 -0400
Subject: [SciPy-User] ANN: IEP 3.4 - the Interactive Editor for Python
	(now with IPython)
References: <CAB_T6=mc8KAaj0yRSBncBxdAOHyYrbK9G0OQsRvx5880M9zR3Q@mail.gmail.com>
Message-ID: <lhjn01$qvg$1@ger.gmane.org>

Almar Klein wrote:

> Dear all,
> 
> We are pleased to announce that IEP version 3.4 is now available for
> download.
> 
> IEP is a cross-platform Python IDE focused on interactivity and
> introspection, which makes it very suitable for scientific computing. Its
> practical design is aimed at simplicity and efficiency.
> 
> This version is marked by many improvements to the shell. Most notably, we
> now have *IPython integration*! Further, there are more ways to customize
> the shell (e.g. define command line and environment args), there is support
> for coloured text, you can click on filenames in tracebacks to open them,
> and it just behaves better. Also, many improvements have been made to the
> debug system.
> 
> Website: http://iep-project.org
> Release notes: https://bitbucket.org/iep-project/iep/wiki/Release_notes
> 
> To install:
> 
>    - Binaries are available for Windows, OSX and Linux.
>    - conda install -c pyzo iep  (in a p33 env)
>    - pip install iep  (IEP depends on Python3 and PySide/PyQt4.)
> 
> Happy coding,
>   Almar

Are you sure pip install iep works?

It did not seem to install any iep into bin directory (only installed packages 
into site-packages).  Tried on fedora 20 linux.



From ralf.gommers at gmail.com  Thu Apr  3 16:14:24 2014
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Thu, 3 Apr 2014 22:14:24 +0200
Subject: [SciPy-User] ANN: Scipy 0.14.0 release candidate 1
Message-ID: <CABL7CQgih6dwtyEHMTc-N0Rtk5yTRMJ5Mf8iqXMYfq5657myVA@mail.gmail.com>

Hi,

I'm pleased to announce the availability of the first release candidate of
Scipy 0.14.0. Please try this RC and report any issues on the scipy-dev
mailing list. A significant number of fixes for scipy.sparse went in after
the beta release, so users of that module may want to test this release
carefully.

Source tarballs, binaries and the full release notes can be found at
https://sourceforge.net/projects/scipy/files/scipy/0.14.0rc1/. The final
release will follow in one week if no new issues are found.

A big thank you to everyone who contributed to this release!

Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140403/c10f7ae5/attachment.html>

From a.klein at science-applied.nl  Fri Apr  4 04:31:17 2014
From: a.klein at science-applied.nl (Almar Klein)
Date: Fri, 4 Apr 2014 10:31:17 +0200
Subject: [SciPy-User] ANN: IEP 3.4 - the Interactive Editor for Python
 (now with IPython)
In-Reply-To: <lhjn01$qvg$1@ger.gmane.org>
References: <CAB_T6=mc8KAaj0yRSBncBxdAOHyYrbK9G0OQsRvx5880M9zR3Q@mail.gmail.com>
	<lhjn01$qvg$1@ger.gmane.org>
Message-ID: <CAB_T6=nHWWdOrk8ZgmADmY-7pGRXrn-Zx7nVkDs=tFBndPpg=A@mail.gmail.com>

> Are you sure pip install iep works?
>
> It did not seem to install any iep into bin directory (only installed
> packages
> into site-packages).  Tried on fedora 20 linux.
>

Ah, I think you are right. pip will only install iep as a package (not a
startup script), so you'd need to run it as "python -m iep". I will make an
issue for fixing  this. Thanks!

- Almar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140404/a414a17b/attachment.html>

From wenlei.xie at gmail.com  Fri Apr  4 11:21:52 2014
From: wenlei.xie at gmail.com (Wenlei Xie)
Date: Fri, 4 Apr 2014 11:21:52 -0400
Subject: [SciPy-User] Segment fault when run svd
Message-ID: <CABTg2xrDSzAZi0GKxogWuevMmHcD3o+G2j0_n9QueFx14X7O-Q@mail.gmail.com>

Hi,

I got segment fault when I run svd on a huge matrix. The matrix contains n
lines, each line is a vector with 2 million dimension vector. I use the
following command for SVD:

s = linalg.svdvals(mat)

It works well when for n = 100, 300, 500. But it got segment fault when n =
1000.

Any suggestions?

Thank you!

Best,
Wenlei
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140404/0566f2c9/attachment.html>

From cjordan1 at uw.edu  Sat Apr  5 16:55:00 2014
From: cjordan1 at uw.edu (Christopher Jordan-Squire)
Date: Sat, 5 Apr 2014 13:55:00 -0700
Subject: [SciPy-User] scipy.sparse parallelism?
Message-ID: <CAEJxiFpd7ZU1LJVdH9yipYx2L_Skn1UNA+so7eAyTsxQx90qLQ@mail.gmail.com>

Does parallelism in the scipy.sparse solvers only come from
multi-threaded BLAS? I.e., the solvers themselves aren't inherently
parallel other than through a multi-threaded BLAS?

Follow-up question: Numpy/Scipy are using the intel mkl, will
scipy.sparse matrices use intel mkl's sparse blas?

-Chris


From josef.pktd at gmail.com  Sun Apr  6 08:34:01 2014
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 6 Apr 2014 08:34:01 -0400
Subject: [SciPy-User] sparse distance matrix ?
Message-ID: <CAMMTP+C07WQWJ0f3323xFH+5_9xUpk2hpoqAgSdvDnZ=kQydDw@mail.gmail.com>

(sunday morning puzzle)

Is there a way to get thresholded distance matrices as sparse matrices
without going through the full dense distance array or explicit
looping?

I didn't see anything in scipy.spatial or in scikit-learn.

usecase is linear algebra not nearest neighbor search

example converts 2d score levels into 1d score levels.
The example could be done with kronecker product, but not in the general case.

>>> ng, ni = 5, 4
>>> x = 100 * np.repeat(np.arange(1,ng+1), ni) + np.tile(np.arange(ni), ng)
>>> x
array([100, 101, 102, 103, 200, 201, 202, 203, 300, 301, 302, 303, 400,
       401, 402, 403, 500, 501, 502, 503])

dense absolute distance

>>> dx1 = np.abs(x[:,None] - x)
>>> dx1[:10, :10]
array([[  0,   1,   2,   3, 100, 101, 102, 103, 200, 201],
       [  1,   0,   1,   2,  99, 100, 101, 102, 199, 200],
       [  2,   1,   0,   1,  98,  99, 100, 101, 198, 199],
       [  3,   2,   1,   0,  97,  98,  99, 100, 197, 198],
       [100,  99,  98,  97,   0,   1,   2,   3, 100, 101],
       [101, 100,  99,  98,   1,   0,   1,   2,  99, 100],
       [102, 101, 100,  99,   2,   1,   0,   1,  98,  99],
       [103, 102, 101, 100,   3,   2,   1,   0,  97,  98],
       [200, 199, 198, 197, 100,  99,  98,  97,   0,   1],
       [201, 200, 199, 198, 101, 100,  99,  98,   1,   0]])

I'd like to get the following as scipy.sparse matrices:

close distances

>>> (dx1) * (dx1 < 3)
array([[0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [1, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [2, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 1, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 2, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 2, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 1, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 2],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 1],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0]])

reversed

>>> (5 - dx1) * (dx1 < 3)
array([[5, 4, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [4, 5, 4, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [3, 4, 5, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 3, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 5, 4, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 4, 5, 4, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 3, 4, 5, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 3, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 5, 4, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 4, 5, 4, 3, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 3, 4, 5, 4, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 4, 3, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 5, 4, 3, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 4, 5, 4, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 4, 5, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 4, 3, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 5, 4, 3],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 4, 5, 4],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 4, 5]])

boolean

>>> (dx1 < 3).astype(int)
array([[1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]])
>>>

Josef


From pav at iki.fi  Sun Apr  6 09:30:11 2014
From: pav at iki.fi (Pauli Virtanen)
Date: Sun, 06 Apr 2014 16:30:11 +0300
Subject: [SciPy-User] scipy.sparse parallelism?
In-Reply-To: <CAEJxiFpd7ZU1LJVdH9yipYx2L_Skn1UNA+so7eAyTsxQx90qLQ@mail.gmail.com>
References: <CAEJxiFpd7ZU1LJVdH9yipYx2L_Skn1UNA+so7eAyTsxQx90qLQ@mail.gmail.com>
Message-ID: <lhrkt3$98b$1@ger.gmane.org>

Hi,

05.04.2014 23:55, Christopher Jordan-Squire kirjoitti:
> Does parallelism in the scipy.sparse solvers only come from
> multi-threaded BLAS? I.e., the solvers themselves aren't inherently
> parallel other than through a multi-threaded BLAS?

There's no support for parallelism or sparse BLAS in scipy.sparse at the
moment.

-- 
Pauli Virtanen



From sturla.molden at gmail.com  Sun Apr  6 14:41:30 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Sun, 6 Apr 2014 18:41:30 +0000 (UTC)
Subject: [SciPy-User] sparse distance matrix ?
References: <CAMMTP+C07WQWJ0f3323xFH+5_9xUpk2hpoqAgSdvDnZ=kQydDw@mail.gmail.com>
Message-ID: <473018060418502352.294308sturla.molden-gmail.com@news.gmane.org>

<josef.pktd at gmail.com> wrote:
> (sunday morning puzzle)
> 
> Is there a way to get thresholded distance matrices as sparse matrices
> without going through the full dense distance array or explicit
> looping?

scipy.spatial.cKDTree can do this.

Sturla



From josef.pktd at gmail.com  Sun Apr  6 15:34:59 2014
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 6 Apr 2014 15:34:59 -0400
Subject: [SciPy-User] sparse distance matrix ?
In-Reply-To: <473018060418502352.294308sturla.molden-gmail.com@news.gmane.org>
References: <CAMMTP+C07WQWJ0f3323xFH+5_9xUpk2hpoqAgSdvDnZ=kQydDw@mail.gmail.com>
	<473018060418502352.294308sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CAMMTP+CpDVak19pxPn1ODnc1zLi10suRO4aV3scF3eGzBqypQg@mail.gmail.com>

On Sun, Apr 6, 2014 at 2:41 PM, Sturla Molden <sturla.molden at gmail.com> wrote:
> <josef.pktd at gmail.com> wrote:
>> (sunday morning puzzle)
>>
>> Is there a way to get thresholded distance matrices as sparse matrices
>> without going through the full dense distance array or explicit
>> looping?
>
> scipy.spatial.cKDTree can do this.

Thanks Sturla, I forgot to look there, because I don't need repeated look-up.

Josef

>
> Sturla
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From wenlei.xie at gmail.com  Sun Apr  6 16:19:11 2014
From: wenlei.xie at gmail.com (Wenlei Xie)
Date: Sun, 6 Apr 2014 13:19:11 -0700
Subject: [SciPy-User] Segment fault when run svd
In-Reply-To: <CABTg2xrDSzAZi0GKxogWuevMmHcD3o+G2j0_n9QueFx14X7O-Q@mail.gmail.com>
References: <CABTg2xrDSzAZi0GKxogWuevMmHcD3o+G2j0_n9QueFx14X7O-Q@mail.gmail.com>
Message-ID: <CABTg2xpBNOGewTzpSp9iyEKpww2qqqQ1S-U6j=YOCC9HTGg4KQ@mail.gmail.com>

Hi the problem was solved when I optimized the memory usage a bit to make
the code run on a 48G memory machine.

I guess the issue is there might be some strange problem when using scipy
on 1TB main memory machine ?

Best,
Wenlei


On Fri, Apr 4, 2014 at 8:21 AM, Wenlei Xie <wenlei.xie at gmail.com> wrote:

> Hi,
>
> I got segment fault when I run svd on a huge matrix. The matrix contains n
> lines, each line is a vector with 2 million dimension vector. I use the
> following command for SVD:
>
> s = linalg.svdvals(mat)
>
> It works well when for n = 100, 300, 500. But it got segment fault when n
> = 1000.
>
> Any suggestions?
>
> Thank you!
>
> Best,
> Wenlei
>
>


-- 
Wenlei Xie (???)

Department of Computer Science
5132 Upson Hall, Cornell University
Ithaca, NY 14853, USA
Phone: (607) 255-5577
Email: wenlei.xie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140406/b2a1b028/attachment.html>

From benjamincorr at gmail.com  Sun Apr  6 17:54:36 2014
From: benjamincorr at gmail.com (Ben Orr)
Date: Sun, 6 Apr 2014 16:54:36 -0500
Subject: [SciPy-User] simple optimization but very complex constraints
Message-ID: <CAC-oNP5ibAxhrtT3xSqD4D5W5=k7yrx=o_xO2ncEUbHUMo6VSA@mail.gmail.com>

I am trying to find the optimal student loan payoff schedule. I believe the
scipy.optimize.minimize function will do what I need but I'm not sure if
it's the best solution.

I represent the loan payment schedule as a 2D array; each column is a loan,
each row is a month. I start off by determining the initial guess (x0).
There are 'number-of-loans' columns. The number of rows is the max of the
number of months to pay off each loan if making the minimum payments. I
then fill in each column with the minimum payment for each loan until it is
payed off, followed by zeros.

I would like to pay "y" amount each month (must be greater than the sum of
the minimum payments). Therefore, I would like to adjust the amount payed
each month (aka the sum of each row) to be equal to or less than my target
monthly payment such that the total payoff amount (sum of the array) is
minimized.

Thus, my function to minimize is simply "sum(x)".

The trick is all the constraints. Each cell must be
greater-than-or-equal-to the minimum payment, unless the remaining balance
is less than the minimum payment, in which case the minimum of that cell is
the remaining balance (0 if paid off). The maximum for any given cell is
the target monthly payment "y". The maximum of the sum of each row is also
the target monthly payment "y". Each column must pay-off the loan that
column represents.

Note that the sum(column)!=initialBalance. "Balance" is only the principal.
The following formula calculates the remaining balance after a monthly
 payment:
i = interest rate (ex: 6.7%, i=6.7 not 0.067)
b = balance (after a payment)
bp = balance previous (before a payment)
p = payment
b = bp*(1+(i/1200))-((p/(i/1200))*((1+(i/1200))-1))

I hoped to simplify the problem by having an array the same shape as "x"
but with the remaining balance (formula above) after each month's payment.
This array would be used in for both constraints pertaining to remaining
balance (remaining balance < min payment and each column must pay off each
loan). See attached spreadsheet for calculating both x0 and the remaining
balance.

Given all that (sorry for the long description), any help choosing and
implementing the appropriate optimization method is greatly appreciated.
Thank you.

Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140406/1c077926/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Loan Calculator.xlsx
Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Size: 126797 bytes
Desc: not available
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140406/1c077926/attachment.xlsx>

From josef.pktd at gmail.com  Sun Apr  6 18:14:21 2014
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 6 Apr 2014 18:14:21 -0400
Subject: [SciPy-User] simple optimization but very complex constraints
In-Reply-To: <CAC-oNP5ibAxhrtT3xSqD4D5W5=k7yrx=o_xO2ncEUbHUMo6VSA@mail.gmail.com>
References: <CAC-oNP5ibAxhrtT3xSqD4D5W5=k7yrx=o_xO2ncEUbHUMo6VSA@mail.gmail.com>
Message-ID: <CAMMTP+D21ujCV3CX4yUbFMVYiXBx4ouD4HkryvL8b47NaURdPQ@mail.gmail.com>

On Sun, Apr 6, 2014 at 5:54 PM, Ben Orr <benjamincorr at gmail.com> wrote:
> I am trying to find the optimal student loan payoff schedule. I believe the
> scipy.optimize.minimize function will do what I need but I'm not sure if
> it's the best solution.
>
> I represent the loan payment schedule as a 2D array; each column is a loan,
> each row is a month. I start off by determining the initial guess (x0).
> There are 'number-of-loans' columns. The number of rows is the max of the
> number of months to pay off each loan if making the minimum payments. I then
> fill in each column with the minimum payment for each loan until it is payed
> off, followed by zeros.
>
> I would like to pay "y" amount each month (must be greater than the sum of
> the minimum payments). Therefore, I would like to adjust the amount payed
> each month (aka the sum of each row) to be equal to or less than my target
> monthly payment such that the total payoff amount (sum of the array) is
> minimized.
>
> Thus, my function to minimize is simply "sum(x)".
>
> The trick is all the constraints. Each cell must be greater-than-or-equal-to
> the minimum payment, unless the remaining balance is less than the minimum
> payment, in which case the minimum of that cell is the remaining balance (0
> if paid off). The maximum for any given cell is the target monthly payment
> "y". The maximum of the sum of each row is also the target monthly payment
> "y". Each column must pay-off the loan that column represents.
>
> Note that the sum(column)!=initialBalance. "Balance" is only the principal.
> The following formula calculates the remaining balance after a monthly
> payment:
> i = interest rate (ex: 6.7%, i=6.7 not 0.067)
> b = balance (after a payment)
> bp = balance previous (before a payment)
> p = payment
> b = bp*(1+(i/1200))-((p/(i/1200))*((1+(i/1200))-1))
>
> I hoped to simplify the problem by having an array the same shape as "x" but
> with the remaining balance (formula above) after each month's payment. This
> array would be used in for both constraints pertaining to remaining balance
> (remaining balance < min payment and each column must pay off each loan).
> See attached spreadsheet for calculating both x0 and the remaining balance.
>
> Given all that (sorry for the long description), any help choosing and
> implementing the appropriate optimization method is greatly appreciated.
> Thank you.

You don't have any curvature that I can see. This looks more like a
linear programming problem.

AFAICS: The solution will be to pay off as fast as possible, if you
don't discount in the objective, max(x), but pay interest on loans. If
all loans have the same interest rate, then it doesn't matter which
ones you pay off faster or slower. If interest rates differ, then you
pay off the highest interest rate loan as fast as possible.

With curvature it would be commonly (?) treated as a dynamic
programming/optimization problem.

Josef

>
> Ben
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From sturla.molden at gmail.com  Sun Apr  6 19:50:23 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Sun, 6 Apr 2014 23:50:23 +0000 (UTC)
Subject: [SciPy-User] sparse distance matrix ?
References: <CAMMTP+C07WQWJ0f3323xFH+5_9xUpk2hpoqAgSdvDnZ=kQydDw@mail.gmail.com>
	<473018060418502352.294308sturla.molden-gmail.com@news.gmane.org>
	<CAMMTP+CpDVak19pxPn1ODnc1zLi10suRO4aV3scF3eGzBqypQg@mail.gmail.com>
Message-ID: <1444510635418519749.598126sturla.molden-gmail.com@news.gmane.org>

<josef.pktd at gmail.com> wrote:

>> scipy.spatial.cKDTree can do this.
> 
> Thanks Sturla, I forgot to look there, because I don't need repeated look-up.

You do need repeated lookups in the case of distance matrix. There are
potentially n*(n-1)/2 distances to compute. But you don't need all of
these, if they are out of the bound you specity. So with a kd-tree you can
avoid computing distances you don't need, by rejecting a whole branch of
the tree at once. That can be considerably faster than brute-forcing all
pair-wise distances.

Sturla



From sturla.molden at gmail.com  Sun Apr  6 20:11:46 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Mon, 7 Apr 2014 00:11:46 +0000 (UTC)
Subject: [SciPy-User] Segment fault when run svd
References: <CABTg2xrDSzAZi0GKxogWuevMmHcD3o+G2j0_n9QueFx14X7O-Q@mail.gmail.com>
	<CABTg2xpBNOGewTzpSp9iyEKpww2qqqQ1S-U6j=YOCC9HTGg4KQ@mail.gmail.com>
Message-ID: <947313563418522113.055952sturla.molden-gmail.com@news.gmane.org>

Wenlei Xie <wenlei.xie at gmail.com> wrote:
> Hi the problem was solved when I optimized the memory usage a bit to make
> the code run on a 48G memory machine.
> 
> I guess the issue is there might be some strange problem when using scipy
> on 1TB main memory machine ?

Unfortunately, SciPy is not tested a lot on that kind of hardware... 

The error you saw might be in SciPy, but it could also be your BLAS or
LAPACK library. It is hard ot say which.


Sturla



From yuqingpsy at gmail.com  Mon Apr  7 15:48:17 2014
From: yuqingpsy at gmail.com (Qing Yu)
Date: Mon, 7 Apr 2014 15:48:17 -0400
Subject: [SciPy-User]  Optimization problem using fmin_slsqp
Message-ID: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>

Dear SciPy experts,

I'm using the fmin_slsqp function in scipy.optimize to fit my data, I first
defined the function f and the residual of fitting, and then used actual
data x and y to fit the curve. However, I encountered an error that I could
not explain.
Here's a short version of my script:

def f(x,a):
return a[0]*exp((a[2]**2)*cos(x-a[1])-1)+a[3]
def residual(a,x,y):
 return y-f(x,a)

x=[0,pi/6,pi/3,pi/2,2*pi/3,5*pi/6]
y=[0.17,0.36,0.61,0.38,0.17,0.16]
a0=[1,pi/3,pi/2,0]
out, fx, its, imode, smode =
fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
full_output=True)

I ended up with the error: "failed in converting 8th argument `g' of
_slsqp.slsqp to C/Fortran array." Does anyone have any idea of why this
happened?

Thanks so much in advance for any help!

Best,
Qing
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140407/92c46359/attachment.html>

From newville at cars.uchicago.edu  Mon Apr  7 16:05:11 2014
From: newville at cars.uchicago.edu (Matt Newville)
Date: Mon, 7 Apr 2014 15:05:11 -0500
Subject: [SciPy-User] Optimization problem using fmin_slsqp
In-Reply-To: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
References: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
Message-ID: <CA+7ESbrh0BU_DsPKP4Jc-HbQtbCWn92xCDi2+qWJOUq0NAsnAg@mail.gmail.com>

On Mon, Apr 7, 2014 at 2:48 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
> Dear SciPy experts,
>
> I'm using the fmin_slsqp function in scipy.optimize to fit my data, I first
> defined the function f and the residual of fitting, and then used actual
> data x and y to fit the curve. However, I encountered an error that I could
> not explain.
> Here's a short version of my script:
>
> def f(x,a):
> return a[0]*exp((a[2]**2)*cos(x-a[1])-1)+a[3]
> def residual(a,x,y):
> return y-f(x,a)
>
> x=[0,pi/6,pi/3,pi/2,2*pi/3,5*pi/6]
> y=[0.17,0.36,0.61,0.38,0.17,0.16]
> a0=[1,pi/3,pi/2,0]
> out, fx, its, imode, smode =
> fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
> full_output=True)
>
> I ended up with the error: "failed in converting 8th argument `g' of
> _slsqp.slsqp to C/Fortran array." Does anyone have any idea of why this
> happened?
>
> Thanks so much in advance for any help!
>
> Best,
> Qing
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


The objective function for fmin_slsqp(),  and many other functions in
scipy.optimize, needs to return a scalar value.  You're passing in a
list of values -- these should almost certainly be numpy arrays.   You
may want to calculate a scaled sum of squares of 'y-f(x,a)' in your
residual() function, if what you had in mind was a least-squares fit
to the data y(x).

--Matt


From yuqingpsy at gmail.com  Mon Apr  7 16:56:18 2014
From: yuqingpsy at gmail.com (Qing Yu)
Date: Mon, 7 Apr 2014 16:56:18 -0400
Subject: [SciPy-User] Optimization problem using fmin_slsqp
In-Reply-To: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
References: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
Message-ID: <CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>

Hi Matt,

Thanks so much for your help! I've now changed the residual function as
below:

def residual(a,x,y):
        return sum((y-f(x,a))**2)

the script seems to be working after the change, but I have another problem
now:

because I've set the upper and lower bounds of the parameters in fmin_slsqp
(and that is why I use fmin_slsqp instead of
leastsq): bounds=[(None,None),(0,pi),(0,None),(None,None)], if I keep the
bounds, I get the error:
            Iteration limit exceeded    (Exit mode 9)
            Current function value: nan
            Iterations: 101
            Function evaluations: 1606
            Gradient evaluations: 101
I tried changing iter to 10000, the error still existed.
If I delete the bound option, I can get the successful output. However, the
fitted values are quite different from what I get using lsqcurvefit in
matlab. Moreover, if I plot the data, the model fit is indeed worse using
fmin_slsqp. I'm not sure if the difference is caused by the missing bounds.
But the outputs from fmin_slsqp did not go out of bounds anyway. Does
anyone have any idea why I could not set the bounds in fmin_slsqp?

Thanks so much for any help!

Best,
Qing


On Mon, Apr 7, 2014 at 3:48 PM, Qing Yu <yuqingpsy at gmail.com> wrote:

> Dear SciPy experts,
>
> I'm using the fmin_slsqp function in scipy.optimize to fit my data, I
> first defined the function f and the residual of fitting, and then used
> actual data x and y to fit the curve. However, I encountered an error that
> I could not explain.
> Here's a short version of my script:
>
> def f(x,a):
> return a[0]*exp((a[2]**2)*cos(x-a[1])-1)+a[3]
> def residual(a,x,y):
>  return y-f(x,a)
>
> x=[0,pi/6,pi/3,pi/2,2*pi/3,5*pi/6]
> y=[0.17,0.36,0.61,0.38,0.17,0.16]
> a0=[1,pi/3,pi/2,0]
> out, fx, its, imode, smode =
> fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
> full_output=True)
>
> I ended up with the error: "failed in converting 8th argument `g' of
> _slsqp.slsqp to C/Fortran array." Does anyone have any idea of why this
> happened?
>
> Thanks so much in advance for any help!
>
> Best,
> Qing
>




-- 
Graduate Student
Qing Yu
Department of Psychological and Brain Sciences
Dartmouth College
Hanover, New Hampshire 03755, US
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140407/573d5a00/attachment.html>

From d_l_goldsmith at yahoo.com  Mon Apr  7 19:12:02 2014
From: d_l_goldsmith at yahoo.com (OlyDLG)
Date: Mon, 7 Apr 2014 23:12:02 +0000 (UTC)
Subject: [SciPy-User] [OT, FFT] FFT of signal larger than processor
Message-ID: <loom.20140408T005809-592@post.gmane.org>

Hi!  This Q came up at my work and I said I'd post here to see if anyone had
any reference suggestions.  We need to FFT a series that uses up to 64K RAM
to process, but we're downsizing the processing core to 8K RAM; we have 2
kinds of supplementary flash memory available: "on-chip" (limited supply,
faster read/writes) & "off-chip" (ample supply, slower read/writes).  Are
there known algorithms for FFT-ing a series piecemeal?  Such that minimize
the number of data exchanges between storage and processor?  References? 
Thanks!

OlyDLG



From newville at cars.uchicago.edu  Tue Apr  8 00:31:28 2014
From: newville at cars.uchicago.edu (Matt Newville)
Date: Mon, 7 Apr 2014 23:31:28 -0500
Subject: [SciPy-User] Optimization problem using fmin_slsqp
In-Reply-To: <CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>
References: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
	<CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>
Message-ID: <CA+7ESbp_bTFkYeMF-VUOht1++5-r=M4qT7t56avRUvPOAKYW2g@mail.gmail.com>

Hi Qing,

On Mon, Apr 7, 2014 at 3:56 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
> Hi Matt,
>
> Thanks so much for your help! I've now changed the residual function as
> below:
>
> def residual(a,x,y):
>         return sum((y-f(x,a))**2)
>
> the script seems to be working after the change, but I have another problem
> now:
>
> because I've set the upper and lower bounds of the parameters in fmin_slsqp
> (and that is why I use fmin_slsqp instead of leastsq):
> bounds=[(None,None),(0,pi),(0,None),(None,None)], if I keep the bounds, I
> get the error:
>             Iteration limit exceeded    (Exit mode 9)
>             Current function value: nan
>             Iterations: 101
>             Function evaluations: 1606
>             Gradient evaluations: 101
> I tried changing iter to 10000, the error still existed.
> If I delete the bound option, I can get the successful output. However, the
> fitted values are quite different from what I get using lsqcurvefit in
> matlab. Moreover, if I plot the data, the model fit is indeed worse using
> fmin_slsqp. I'm not sure if the difference is caused by the missing bounds.
> But the outputs from fmin_slsqp did not go out of bounds anyway. Does anyone
> have any idea why I could not set the bounds in fmin_slsqp?
>
> Thanks so much for any help!
>

I don't know enough about the internals of fmin_slsqp() to know for
sure what's going wrong.


You might find the lmfit-py (http://lmfit.github.io/lmfit-py/) package
useful.   This builds on scipy.optimize to allow upper/lower bounds to
be used with any of the scalar minimizers or with least-squares.  It
also allows you to easily change between fitting methods, and several
other bells and whistles.

Translating your problem into lmfit would look something like this:

############
import numpy as np
import pylab

from lmfit import minimize, Parameters, fit_report

pi = np.pi

def f(x, scale, phase, expon, offset):
    return scale*np.exp((expon**2)*np.cos(x-phase)-1) + offset

def resid(params, x, y):
    scale = params['scale'].value
    phase = params['phase'].value
    expon = params['expon'].value
    offset = params['offset'].value
    return f(x, scale, phase, expon, offset) - y

x = np.array([0,pi/6,pi/3,pi/2,2*pi/3,5*pi/6])
y = np.array([0.17,0.36,0.61,0.38,0.17,0.16])

pars = Parameters()
pars.add('scale', value=1, vary=True)
pars.add('phase', value=1, vary=True, min=0, max=pi)
pars.add('expon', value=1, vary=True)
pars.add('offset', value=0, vary=True)

# first fit with the Nelder-Mead method (fmin, scalar)
result = minimize(resid, pars, args=(x,y), method='Nelder')

# then use these values as initial guess to least--squares
# which provides estimate of uncertainties
result = minimize(resid, pars, args=(x,y), method='leastsq')

print fit_report(result.params)

fit = y + result.residual
pylab.plot(x, y, 'r+')
pylab.plot(x, fit, 'b-')
pylab.show()

############

which would print out the results:

[Variables]]
     expon:      2.371419 +/- 0.07606183 (3.21%) initial =  2.371424
     offset:     0.1487316 +/- 0.008012265 (5.39%) initial =  0.1487296
     phase:      1.062886 +/- 0.01148847 (1.08%) initial =  1.062866
     scale:      0.004551185 +/- 0.001651263 (36.28%) initial =  0.004551088
[[Correlations]] (unreported correlations are <  0.100)
    C(expon, scale)              = -0.998
    C(offset, scale)             = -0.719
    C(expon, offset)             =  0.684


(initial values here are those from fmin(), and leastsq() agrees with
the results).   Note that parameters can have bounds (specified by
min, max), and that one can use fmin() or leastsq() without modifying
the objective --   for scalar minimizers it automatically converts
arrays returned by the objective function to a sum of squares.

Hope that helps,

--Matt


From nouiz at nouiz.org  Tue Apr  8 09:31:45 2014
From: nouiz at nouiz.org (=?ISO-8859-1?Q?Fr=E9d=E9ric_Bastien?=)
Date: Tue, 8 Apr 2014 09:31:45 -0400
Subject: [SciPy-User] [OT, FFT] FFT of signal larger than processor
In-Reply-To: <loom.20140408T005809-592@post.gmane.org>
References: <loom.20140408T005809-592@post.gmane.org>
Message-ID: <CADKKbthQSgC082P1=H76MPARWNf-vrhdPJkjYjwxPrm-i_SzOA@mail.gmail.com>

>From memory, I saw something like this for the CELL processor, google
for CELL + fft would probably find this.

Fred

On Mon, Apr 7, 2014 at 7:12 PM, OlyDLG <d_l_goldsmith at yahoo.com> wrote:
> Hi!  This Q came up at my work and I said I'd post here to see if anyone had
> any reference suggestions.  We need to FFT a series that uses up to 64K RAM
> to process, but we're downsizing the processing core to 8K RAM; we have 2
> kinds of supplementary flash memory available: "on-chip" (limited supply,
> faster read/writes) & "off-chip" (ample supply, slower read/writes).  Are
> there known algorithms for FFT-ing a series piecemeal?  Such that minimize
> the number of data exchanges between storage and processor?  References?
> Thanks!
>
> OlyDLG
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From cournape at gmail.com  Tue Apr  8 14:22:43 2014
From: cournape at gmail.com (David Cournapeau)
Date: Tue, 8 Apr 2014 19:22:43 +0100
Subject: [SciPy-User] [OT, FFT] FFT of signal larger than processor
In-Reply-To: <CADKKbthQSgC082P1=H76MPARWNf-vrhdPJkjYjwxPrm-i_SzOA@mail.gmail.com>
References: <loom.20140408T005809-592@post.gmane.org>
	<CADKKbthQSgC082P1=H76MPARWNf-vrhdPJkjYjwxPrm-i_SzOA@mail.gmail.com>
Message-ID: <CAGY4rcU5y7RPRQy_Ez4m_2T9FN_E=rwUUZSzaqmieKmvgtFTSA@mail.gmail.com>

On Tue, Apr 8, 2014 at 2:31 PM, Fr?d?ric Bastien <nouiz at nouiz.org> wrote:

> >From memory, I saw something like this for the CELL processor, google
> for CELL + fft would probably find this.
>

There are many available algorithms, but some patented.

The keyword you are looking code for the DSP literature is Overlap Add
(OLA), and similar techniques. I don't know about special optimizations for
memory bw savings.

David

>
> Fred
>
> On Mon, Apr 7, 2014 at 7:12 PM, OlyDLG <d_l_goldsmith at yahoo.com> wrote:
> > Hi!  This Q came up at my work and I said I'd post here to see if anyone
> had
> > any reference suggestions.  We need to FFT a series that uses up to 64K
> RAM
> > to process, but we're downsizing the processing core to 8K RAM; we have 2
> > kinds of supplementary flash memory available: "on-chip" (limited supply,
> > faster read/writes) & "off-chip" (ample supply, slower read/writes).  Are
> > there known algorithms for FFT-ing a series piecemeal?  Such that
> minimize
> > the number of data exchanges between storage and processor?  References?
> > Thanks!
> >
> > OlyDLG
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140408/22261a6a/attachment.html>

From eric.moore2 at nih.gov  Tue Apr  8 14:30:21 2014
From: eric.moore2 at nih.gov (Moore, Eric (NIH/NIDDK) [F])
Date: Tue, 8 Apr 2014 18:30:21 +0000
Subject: [SciPy-User] [OT, FFT] FFT of signal larger than processor
In-Reply-To: <loom.20140408T005809-592@post.gmane.org>
References: <loom.20140408T005809-592@post.gmane.org>
Message-ID: <649847CE7F259144A0FD99AC64E7326D0BC7A8@MLBXV17.nih.gov>

> -----Original Message-----
> From: OlyDLG [mailto:d_l_goldsmith at yahoo.com]
> Sent: Monday, April 07, 2014 7:12 PM
> To: scipy-user at scipy.org
> Subject: [SciPy-User] [OT, FFT] FFT of signal larger than processor
> 
> Hi!  This Q came up at my work and I said I'd post here to see if
> anyone had
> any reference suggestions.  We need to FFT a series that uses up to 64K
> RAM
> to process, but we're downsizing the processing core to 8K RAM; we have
> 2
> kinds of supplementary flash memory available: "on-chip" (limited
> supply,
> faster read/writes) & "off-chip" (ample supply, slower read/writes).
> Are
> there known algorithms for FFT-ing a series piecemeal?  Such that
> minimize
> the number of data exchanges between storage and processor?
> References?
> Thanks!
> 
> OlyDLG
> 

You can do this by implementing one round of the FFT by hand and then doing all of the smaller transforms using the library call.  I can probably dig up some Matlab code which should be straightforward for you to translate if you're interested. 

When I was doing this, the problem was heavily IO bound as you've indicated. But all I had were spinning disks so perhaps it won't be quite as bad for you.  I could never find anything helpful about any algorithmic improvements when I looked, but I had trouble finding a search term that would bring me anything useful.

Eric


From yuqingpsy at gmail.com  Tue Apr  8 15:10:39 2014
From: yuqingpsy at gmail.com (Qing Yu)
Date: Tue, 8 Apr 2014 15:10:39 -0400
Subject: [SciPy-User] Optimization problem using fmin_slsqp
In-Reply-To: <CA+7ESbp_bTFkYeMF-VUOht1++5-r=M4qT7t56avRUvPOAKYW2g@mail.gmail.com>
References: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
	<CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>
	<CA+7ESbp_bTFkYeMF-VUOht1++5-r=M4qT7t56avRUvPOAKYW2g@mail.gmail.com>
Message-ID: <CAFEPp9=VPVK81801K16DO8wXF62q1hKxCcZYvCmXGEhxvDz19g@mail.gmail.com>

Hi Matt,

Thanks so much! I've actually tried the 'minimize' function in
scipy.optimize. There are several constrained optimization methods in
'minimize' and among them 'SLSQP' seems to be equivalent to the method used
in fmin_slsqp; I got similar bad fitting results using these two functions.
Meanwhile, using the 'L-BFGS-B' method in 'minimize', I can get similar
good fitting results as 'leastsq'. But I'm not sure how these two
algorithms differ.

I then tried the 'lmfit' method you provided, and it seems that this method
is exactly what I want, though I've no idea how it adds bounds to
'leastsq'. Thanks again for your help!

Best,
Qing


On Tue, Apr 8, 2014 at 12:31 AM, Matt Newville
<newville at cars.uchicago.edu>wrote:

> Hi Qing,
>
> On Mon, Apr 7, 2014 at 3:56 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
> > Hi Matt,
> >
> > Thanks so much for your help! I've now changed the residual function as
> > below:
> >
> > def residual(a,x,y):
> >         return sum((y-f(x,a))**2)
> >
> > the script seems to be working after the change, but I have another
> problem
> > now:
> >
> > because I've set the upper and lower bounds of the parameters in
> fmin_slsqp
> > (and that is why I use fmin_slsqp instead of leastsq):
> > bounds=[(None,None),(0,pi),(0,None),(None,None)], if I keep the bounds, I
> > get the error:
> >             Iteration limit exceeded    (Exit mode 9)
> >             Current function value: nan
> >             Iterations: 101
> >             Function evaluations: 1606
> >             Gradient evaluations: 101
> > I tried changing iter to 10000, the error still existed.
> > If I delete the bound option, I can get the successful output. However,
> the
> > fitted values are quite different from what I get using lsqcurvefit in
> > matlab. Moreover, if I plot the data, the model fit is indeed worse using
> > fmin_slsqp. I'm not sure if the difference is caused by the missing
> bounds.
> > But the outputs from fmin_slsqp did not go out of bounds anyway. Does
> anyone
> > have any idea why I could not set the bounds in fmin_slsqp?
> >
> > Thanks so much for any help!
> >
>
> I don't know enough about the internals of fmin_slsqp() to know for
> sure what's going wrong.
>
>
> You might find the lmfit-py (http://lmfit.github.io/lmfit-py/) package
> useful.   This builds on scipy.optimize to allow upper/lower bounds to
> be used with any of the scalar minimizers or with least-squares.  It
> also allows you to easily change between fitting methods, and several
> other bells and whistles.
>
> Translating your problem into lmfit would look something like this:
>
> ############
> import numpy as np
> import pylab
>
> from lmfit import minimize, Parameters, fit_report
>
> pi = np.pi
>
> def f(x, scale, phase, expon, offset):
>     return scale*np.exp((expon**2)*np.cos(x-phase)-1) + offset
>
> def resid(params, x, y):
>     scale = params['scale'].value
>     phase = params['phase'].value
>     expon = params['expon'].value
>     offset = params['offset'].value
>     return f(x, scale, phase, expon, offset) - y
>
> x = np.array([0,pi/6,pi/3,pi/2,2*pi/3,5*pi/6])
> y = np.array([0.17,0.36,0.61,0.38,0.17,0.16])
>
> pars = Parameters()
> pars.add('scale', value=1, vary=True)
> pars.add('phase', value=1, vary=True, min=0, max=pi)
> pars.add('expon', value=1, vary=True)
> pars.add('offset', value=0, vary=True)
>
> # first fit with the Nelder-Mead method (fmin, scalar)
> result = minimize(resid, pars, args=(x,y), method='Nelder')
>
> # then use these values as initial guess to least--squares
> # which provides estimate of uncertainties
> result = minimize(resid, pars, args=(x,y), method='leastsq')
>
> print fit_report(result.params)
>
> fit = y + result.residual
> pylab.plot(x, y, 'r+')
> pylab.plot(x, fit, 'b-')
> pylab.show()
>
> ############
>
> which would print out the results:
>
> [Variables]]
>      expon:      2.371419 +/- 0.07606183 (3.21%) initial =  2.371424
>      offset:     0.1487316 +/- 0.008012265 (5.39%) initial =  0.1487296
>      phase:      1.062886 +/- 0.01148847 (1.08%) initial =  1.062866
>      scale:      0.004551185 +/- 0.001651263 (36.28%) initial =
>  0.004551088
> [[Correlations]] (unreported correlations are <  0.100)
>     C(expon, scale)              = -0.998
>     C(offset, scale)             = -0.719
>     C(expon, offset)             =  0.684
>
>
> (initial values here are those from fmin(), and leastsq() agrees with
> the results).   Note that parameters can have bounds (specified by
> min, max), and that one can use fmin() or leastsq() without modifying
> the objective --   for scalar minimizers it automatically converts
> arrays returned by the objective function to a sum of squares.
>
> Hope that helps,
>
> --Matt
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140408/5dc43b7d/attachment.html>

From d_l_goldsmith at yahoo.com  Tue Apr  8 19:08:59 2014
From: d_l_goldsmith at yahoo.com (OlyDLG)
Date: Tue, 8 Apr 2014 23:08:59 +0000 (UTC)
Subject: [SciPy-User] [OT, FFT] FFT of signal larger than processor
References: <loom.20140408T005809-592@post.gmane.org>
	<649847CE7F259144A0FD99AC64E7326D0BC7A8@MLBXV17.nih.gov>
Message-ID: <loom.20140409T010756-93@post.gmane.org>

Thanks, all!

DLG





From newville at cars.uchicago.edu  Tue Apr  8 20:10:42 2014
From: newville at cars.uchicago.edu (Matt Newville)
Date: Tue, 8 Apr 2014 19:10:42 -0500
Subject: [SciPy-User] Optimization problem using fmin_slsqp
In-Reply-To: <CAFEPp9=VPVK81801K16DO8wXF62q1hKxCcZYvCmXGEhxvDz19g@mail.gmail.com>
References: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
	<CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>
	<CA+7ESbp_bTFkYeMF-VUOht1++5-r=M4qT7t56avRUvPOAKYW2g@mail.gmail.com>
	<CAFEPp9=VPVK81801K16DO8wXF62q1hKxCcZYvCmXGEhxvDz19g@mail.gmail.com>
Message-ID: <CA+7ESbrOhdiH5s2FDDb2B+W7MvjJZHyHg9CNTdx6gfB_QW4vcg@mail.gmail.com>

Hi Qing,

On Tue, Apr 8, 2014 at 2:10 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
> Hi Matt,
>
> Thanks so much! I've actually tried the 'minimize' function in
> scipy.optimize. There are several constrained optimization methods in
> 'minimize' and among them 'SLSQP' seems to be equivalent to the method used
> in fmin_slsqp; I got similar bad fitting results using these two functions.
> Meanwhile, using the 'L-BFGS-B' method in 'minimize', I can get similar good
> fitting results as 'leastsq'. But I'm not sure how these two algorithms
> differ.

I think your experience matches many others: for many real data
fitting problems, it's hard to know how well many of these algorithms
will really work.   Just the fact that there are so many methods
available gives a hint that they aren't all robust in all situations,
right?     Then again, you're asking to fit four parameters given only
six observations.... that's going to to be challenging for most of
these methods.

> I then tried the 'lmfit' method you provided, and it seems that this method
> is exactly what I want, though I've no idea how it adds bounds to 'leastsq'.
> Thanks again for your help!

The bounds method used in lmfit are described at
http://lmfit.github.io/lmfit-py/bounds.html

--Matt


From yw5aj at virginia.edu  Wed Apr  9 11:00:16 2014
From: yw5aj at virginia.edu (Yuxiang Wang)
Date: Wed, 9 Apr 2014 11:00:16 -0400
Subject: [SciPy-User] Optimization problem using fmin_slsqp
In-Reply-To: <CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>
References: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
	<CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>
Message-ID: <CANRv7QoNPi245nT3d=4n4qv4+6Wz9xv1s6aki3ZtL+BE6RuYzw@mail.gmail.com>

Hi Qing,

In your description I see a line:

Current function value: nan

Is your function outputting a np.nan for certain inputs? That might be
bad for the algorithm to work. Could you check that?

-Shawn

On Mon, Apr 7, 2014 at 4:56 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
> Hi Matt,
>
> Thanks so much for your help! I've now changed the residual function as
> below:
>
> def residual(a,x,y):
>         return sum((y-f(x,a))**2)
>
> the script seems to be working after the change, but I have another problem
> now:
>
> because I've set the upper and lower bounds of the parameters in fmin_slsqp
> (and that is why I use fmin_slsqp instead of leastsq):
> bounds=[(None,None),(0,pi),(0,None),(None,None)], if I keep the bounds, I
> get the error:
>             Iteration limit exceeded    (Exit mode 9)
>             Current function value: nan
>             Iterations: 101
>             Function evaluations: 1606
>             Gradient evaluations: 101
> I tried changing iter to 10000, the error still existed.
> If I delete the bound option, I can get the successful output. However, the
> fitted values are quite different from what I get using lsqcurvefit in
> matlab. Moreover, if I plot the data, the model fit is indeed worse using
> fmin_slsqp. I'm not sure if the difference is caused by the missing bounds.
> But the outputs from fmin_slsqp did not go out of bounds anyway. Does anyone
> have any idea why I could not set the bounds in fmin_slsqp?
>
> Thanks so much for any help!
>
> Best,
> Qing
>
>
> On Mon, Apr 7, 2014 at 3:48 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
>>
>> Dear SciPy experts,
>>
>> I'm using the fmin_slsqp function in scipy.optimize to fit my data, I
>> first defined the function f and the residual of fitting, and then used
>> actual data x and y to fit the curve. However, I encountered an error that I
>> could not explain.
>> Here's a short version of my script:
>>
>> def f(x,a):
>> return a[0]*exp((a[2]**2)*cos(x-a[1])-1)+a[3]
>> def residual(a,x,y):
>> return y-f(x,a)
>>
>> x=[0,pi/6,pi/3,pi/2,2*pi/3,5*pi/6]
>> y=[0.17,0.36,0.61,0.38,0.17,0.16]
>> a0=[1,pi/3,pi/2,0]
>> out, fx, its, imode, smode =
>> fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
>> full_output=True)
>>
>> I ended up with the error: "failed in converting 8th argument `g' of
>> _slsqp.slsqp to C/Fortran array." Does anyone have any idea of why this
>> happened?
>>
>> Thanks so much in advance for any help!
>>
>> Best,
>> Qing
>
>
>
>
>
> --
> Graduate Student
> Qing Yu
> Department of Psychological and Brain Sciences
> Dartmouth College
> Hanover, New Hampshire 03755, US
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Yuxiang "Shawn" Wang
Gerling Research Lab
University of Virginia
yw5aj at virginia.edu
+1 (434) 284-0836
https://sites.google.com/a/virginia.edu/yw5aj/


From yuqingpsy at gmail.com  Wed Apr  9 11:24:55 2014
From: yuqingpsy at gmail.com (Qing Yu)
Date: Wed, 9 Apr 2014 11:24:55 -0400
Subject: [SciPy-User] Optimization problem using fmin_slsqp
In-Reply-To: <CANRv7QoNPi245nT3d=4n4qv4+6Wz9xv1s6aki3ZtL+BE6RuYzw@mail.gmail.com>
References: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
	<CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>
	<CANRv7QoNPi245nT3d=4n4qv4+6Wz9xv1s6aki3ZtL+BE6RuYzw@mail.gmail.com>
Message-ID: <CAFEPp9=bF3g5YzktsiEjrygZRerkF30vfTxmWtTC-zgF9tmDEQ@mail.gmail.com>

Hi Shawn,

Yes I was actually using a 'try...except...' command in my script. It's
simply because I want an output when the fitting procedure fails. Here's
the part of the script:

try:
       out, fx, its, imode, smode
= fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
full_output=True)
except:
       out = [nan, nan, nan, nan]

But I suppose I won't get the nan output unless the fitting fails?

Thanks!


Best,
Qing

On Wed, Apr 9, 2014 at 11:00 AM, Yuxiang Wang <yw5aj at virginia.edu> wrote:

> Hi Qing,
>
> In your description I see a line:
>
> Current function value: nan
>
> Is your function outputting a np.nan for certain inputs? That might be
> bad for the algorithm to work. Could you check that?
>
> -Shawn
>
> On Mon, Apr 7, 2014 at 4:56 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
> > Hi Matt,
> >
> > Thanks so much for your help! I've now changed the residual function as
> > below:
> >
> > def residual(a,x,y):
> >         return sum((y-f(x,a))**2)
> >
> > the script seems to be working after the change, but I have another
> problem
> > now:
> >
> > because I've set the upper and lower bounds of the parameters in
> fmin_slsqp
> > (and that is why I use fmin_slsqp instead of leastsq):
> > bounds=[(None,None),(0,pi),(0,None),(None,None)], if I keep the bounds, I
> > get the error:
> >             Iteration limit exceeded    (Exit mode 9)
> >             Current function value: nan
> >             Iterations: 101
> >             Function evaluations: 1606
> >             Gradient evaluations: 101
> > I tried changing iter to 10000, the error still existed.
> > If I delete the bound option, I can get the successful output. However,
> the
> > fitted values are quite different from what I get using lsqcurvefit in
> > matlab. Moreover, if I plot the data, the model fit is indeed worse using
> > fmin_slsqp. I'm not sure if the difference is caused by the missing
> bounds.
> > But the outputs from fmin_slsqp did not go out of bounds anyway. Does
> anyone
> > have any idea why I could not set the bounds in fmin_slsqp?
> >
> > Thanks so much for any help!
> >
> > Best,
> > Qing
> >
> >
> > On Mon, Apr 7, 2014 at 3:48 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
> >>
> >> Dear SciPy experts,
> >>
> >> I'm using the fmin_slsqp function in scipy.optimize to fit my data, I
> >> first defined the function f and the residual of fitting, and then used
> >> actual data x and y to fit the curve. However, I encountered an error
> that I
> >> could not explain.
> >> Here's a short version of my script:
> >>
> >> def f(x,a):
> >> return a[0]*exp((a[2]**2)*cos(x-a[1])-1)+a[3]
> >> def residual(a,x,y):
> >> return y-f(x,a)
> >>
> >> x=[0,pi/6,pi/3,pi/2,2*pi/3,5*pi/6]
> >> y=[0.17,0.36,0.61,0.38,0.17,0.16]
> >> a0=[1,pi/3,pi/2,0]
> >> out, fx, its, imode, smode =
> >>
> fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
> >> full_output=True)
> >>
> >> I ended up with the error: "failed in converting 8th argument `g' of
> >> _slsqp.slsqp to C/Fortran array." Does anyone have any idea of why this
> >> happened?
> >>
> >> Thanks so much in advance for any help!
> >>
> >> Best,
> >> Qing
> >
> >
> >
> >
> >
> > --
> > Graduate Student
> > Qing Yu
> > Department of Psychological and Brain Sciences
> > Dartmouth College
> > Hanover, New Hampshire 03755, US
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
>
>
>
> --
> Yuxiang "Shawn" Wang
> Gerling Research Lab
> University of Virginia
> yw5aj at virginia.edu
> +1 (434) 284-0836
> https://sites.google.com/a/virginia.edu/yw5aj/
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Graduate Student
Qing Yu
Department of Psychological and Brain Sciences
Dartmouth College
Hanover, New Hampshire 03755, US
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140409/7332a1d3/attachment.html>

From bje at mayo.edu  Wed Apr  9 12:43:13 2014
From: bje at mayo.edu (Erickson, Bradley J., M.D., Ph.D.)
Date: Wed, 09 Apr 2014 16:43:13 +0000
Subject: [SciPy-User] [OT]  Job / Post doc position
Message-ID: <6e55ab$8knhd9@ironport10.mayo.edu>

Hi Everyone,
  I want to let anyone interested in a post-doctoral position be aware that I am currently hiring. The position is for a PhD with experience in image processing preferably using python. The exact field of your PhD is flexible. There is more than 1 position open, and the focus can range from computation of features in medical images (mostly brain MRI) to developing and applying machine learning methods to find important diagnostic and therapeutic features in the images. Our main development environment is python.
  The salary is competitive, and Mayo is very respected in medical image processing. If interested in learning more, please send me a note indicating your area of interest and a current CV to bje at mayo.edu<mailto:bje at mayo.edu>. This is a 100% FTE position with initial appointment of 1 year, but with option of renewal for more years.
  Mayo Clinic is an equal opportunity employer.
Thank you and pardon the interruption if you are not looking for a job.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140409/7496309e/attachment.html>

From yw5aj at virginia.edu  Wed Apr  9 21:16:24 2014
From: yw5aj at virginia.edu (Yuxiang Wang)
Date: Wed, 9 Apr 2014 21:16:24 -0400
Subject: [SciPy-User] Optimization problem using fmin_slsqp
In-Reply-To: <CAFEPp9=bF3g5YzktsiEjrygZRerkF30vfTxmWtTC-zgF9tmDEQ@mail.gmail.com>
References: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
	<CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>
	<CANRv7QoNPi245nT3d=4n4qv4+6Wz9xv1s6aki3ZtL+BE6RuYzw@mail.gmail.com>
	<CAFEPp9=bF3g5YzktsiEjrygZRerkF30vfTxmWtTC-zgF9tmDEQ@mail.gmail.com>
Message-ID: <CANRv7Qp9abVRZi7PJDP3CWK5NHLOrWy3guOL=8tBVEckumAeEA@mail.gmail.com>

Hi Qing,

I was thinking that your function residual() may give nan and this
would make the fitting uncontrollable.

I think that this may be the case because in your previous emails you
mentioned the error:
            Iteration limit exceeded    (Exit mode 9)
            Current function value: nan
            Iterations: 101
            Function evaluations: 1606
            Gradient evaluations: 101
The current function value is a "nan".

It might simply be that in the bounds you defined, the algorithm does
not converge due to the nan; if you get rid of the bounds, it will go
elsewhere for sure but that does not give what you want.

Could you check that to see whether residual() function is spitting out nan?


PS: I heard that minimize() should always be used and fmin_slsqp etc.
should be deprecated... I personally do prefer minimize because then I
can easily try some other algorithms like L-BFGS-B (this and SLSQP are
my favorite two!).

-Shawn

On Wed, Apr 9, 2014 at 11:24 AM, Qing Yu <yuqingpsy at gmail.com> wrote:
> Hi Shawn,
>
> Yes I was actually using a 'try...except...' command in my script. It's
> simply because I want an output when the fitting procedure fails. Here's the
> part of the script:
>
> try:
>        out, fx, its, imode, smode =
> fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
> full_output=True)
> except:
>        out = [nan, nan, nan, nan]
>
> But I suppose I won't get the nan output unless the fitting fails?
>
> Thanks!
>
>
> Best,
> Qing
>
> On Wed, Apr 9, 2014 at 11:00 AM, Yuxiang Wang <yw5aj at virginia.edu> wrote:
>>
>> Hi Qing,
>>
>> In your description I see a line:
>>
>> Current function value: nan
>>
>> Is your function outputting a np.nan for certain inputs? That might be
>> bad for the algorithm to work. Could you check that?
>>
>> -Shawn
>>
>> On Mon, Apr 7, 2014 at 4:56 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
>> > Hi Matt,
>> >
>> > Thanks so much for your help! I've now changed the residual function as
>> > below:
>> >
>> > def residual(a,x,y):
>> >         return sum((y-f(x,a))**2)
>> >
>> > the script seems to be working after the change, but I have another
>> > problem
>> > now:
>> >
>> > because I've set the upper and lower bounds of the parameters in
>> > fmin_slsqp
>> > (and that is why I use fmin_slsqp instead of leastsq):
>> > bounds=[(None,None),(0,pi),(0,None),(None,None)], if I keep the bounds,
>> > I
>> > get the error:
>> >             Iteration limit exceeded    (Exit mode 9)
>> >             Current function value: nan
>> >             Iterations: 101
>> >             Function evaluations: 1606
>> >             Gradient evaluations: 101
>> > I tried changing iter to 10000, the error still existed.
>> > If I delete the bound option, I can get the successful output. However,
>> > the
>> > fitted values are quite different from what I get using lsqcurvefit in
>> > matlab. Moreover, if I plot the data, the model fit is indeed worse
>> > using
>> > fmin_slsqp. I'm not sure if the difference is caused by the missing
>> > bounds.
>> > But the outputs from fmin_slsqp did not go out of bounds anyway. Does
>> > anyone
>> > have any idea why I could not set the bounds in fmin_slsqp?
>> >
>> > Thanks so much for any help!
>> >
>> > Best,
>> > Qing
>> >
>> >
>> > On Mon, Apr 7, 2014 at 3:48 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
>> >>
>> >> Dear SciPy experts,
>> >>
>> >> I'm using the fmin_slsqp function in scipy.optimize to fit my data, I
>> >> first defined the function f and the residual of fitting, and then used
>> >> actual data x and y to fit the curve. However, I encountered an error
>> >> that I
>> >> could not explain.
>> >> Here's a short version of my script:
>> >>
>> >> def f(x,a):
>> >> return a[0]*exp((a[2]**2)*cos(x-a[1])-1)+a[3]
>> >> def residual(a,x,y):
>> >> return y-f(x,a)
>> >>
>> >> x=[0,pi/6,pi/3,pi/2,2*pi/3,5*pi/6]
>> >> y=[0.17,0.36,0.61,0.38,0.17,0.16]
>> >> a0=[1,pi/3,pi/2,0]
>> >> out, fx, its, imode, smode =
>> >>
>> >> fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
>> >> full_output=True)
>> >>
>> >> I ended up with the error: "failed in converting 8th argument `g' of
>> >> _slsqp.slsqp to C/Fortran array." Does anyone have any idea of why this
>> >> happened?
>> >>
>> >> Thanks so much in advance for any help!
>> >>
>> >> Best,
>> >> Qing
>> >
>> >
>> >
>> >
>> >
>> > --
>> > Graduate Student
>> > Qing Yu
>> > Department of Psychological and Brain Sciences
>> > Dartmouth College
>> > Hanover, New Hampshire 03755, US
>> >
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>>
>>
>>
>> --
>> Yuxiang "Shawn" Wang
>> Gerling Research Lab
>> University of Virginia
>> yw5aj at virginia.edu
>> +1 (434) 284-0836
>> https://sites.google.com/a/virginia.edu/yw5aj/
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>
>
> --
> Graduate Student
> Qing Yu
> Department of Psychological and Brain Sciences
> Dartmouth College
> Hanover, New Hampshire 03755, US
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Yuxiang "Shawn" Wang
Gerling Research Lab
University of Virginia
yw5aj at virginia.edu
+1 (434) 284-0836
https://sites.google.com/a/virginia.edu/yw5aj/


From nelle.varoquaux at gmail.com  Thu Apr 10 02:04:44 2014
From: nelle.varoquaux at gmail.com (Nelle Varoquaux)
Date: Thu, 10 Apr 2014 08:04:44 +0200
Subject: [SciPy-User] EuroScipy Reminder: call for abstracts closes in 4 days
Message-ID: <CAE-UAvR2qHTtdw11uQubvdy4XM3DRGTNXCoOF4uNNx5vkVk68g@mail.gmail.com>

Hello everyone,

Just a quick reminder that the EuroScipy call for abstracts closes on the
14th: don't forget to submit your talk proposal! It is in four days only!

In short, EuroScipy is a cross-disciplinary gathering focused on the use
and development of the Python language in scientific research. This event
strives to bring together both users and developers of scientific tools, as
well as academic research and state of the art industry.

EuroSciPy 2014, the Seventh Annual Conference on Python in Science, takes
place in *Cambridge, UK on 27 - 30 August 2014*. The conference features
two days of tutorials followed by two days of scientific talks. The day
after the main conference, developer sprints will be organized on projects
of interest to attendees.

The topics presented at EuroSciPy are very diverse, with a focus on
advanced software engineering and original uses of Python and its
scientific libraries, either in theoretical or experimental research, from
both academia and the industry. The program includes keynotes, contributed
talks and posters.

Submissions for talks and posters are welcome on our website (
http://www.euroscipy.org/2014/). In your abstract, please provide details
on what Python tools are being employed, and how. The deadline for
submission is 14 April 2014.

Also until 14 April 2014, you can apply for a sprint session on 31 August
2014. See https://www.euroscipy.org/2014/calls/sprints/ for details.

Thanks,

N
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140410/2582d170/attachment.html>

From emanuele at relativita.com  Thu Apr 10 10:07:46 2014
From: emanuele at relativita.com (Emanuele Olivetti)
Date: Thu, 10 Apr 2014 16:07:46 +0200
Subject: [SciPy-User] sparse distance matrix ?
In-Reply-To: <CAMMTP+C07WQWJ0f3323xFH+5_9xUpk2hpoqAgSdvDnZ=kQydDw@mail.gmail.com>
References: <CAMMTP+C07WQWJ0f3323xFH+5_9xUpk2hpoqAgSdvDnZ=kQydDw@mail.gmail.com>
Message-ID: <5346A5B2.3070607@relativita.com>

On 04/06/2014 02:34 PM, josef.pktd at gmail.com wrote:
> (sunday morning puzzle)
>
> Is there a way to get thresholded distance matrices as sparse matrices
> without going through the full dense distance array or explicit
> looping?
>
>

There is something like that in scikit.learn: kneighbors_graph() and
radius_neighbors_graph(), with the argument mode='distance':

http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.kneighbors_graph.html#sklearn.neighbors.kneighbors_graph
http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.radius_neighbors_graph.html#sklearn.neighbors.radius_neighbors_graph

Best,

Emanuele




From yuqingpsy at gmail.com  Thu Apr 10 11:22:10 2014
From: yuqingpsy at gmail.com (Qing Yu)
Date: Thu, 10 Apr 2014 11:22:10 -0400
Subject: [SciPy-User] Optimization problem using fmin_slsqp
In-Reply-To: <CANRv7Qp9abVRZi7PJDP3CWK5NHLOrWy3guOL=8tBVEckumAeEA@mail.gmail.com>
References: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
	<CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>
	<CANRv7QoNPi245nT3d=4n4qv4+6Wz9xv1s6aki3ZtL+BE6RuYzw@mail.gmail.com>
	<CAFEPp9=bF3g5YzktsiEjrygZRerkF30vfTxmWtTC-zgF9tmDEQ@mail.gmail.com>
	<CANRv7Qp9abVRZi7PJDP3CWK5NHLOrWy3guOL=8tBVEckumAeEA@mail.gmail.com>
Message-ID: <CAFEPp9muvEJMXo_KxK6-1Ehqs_W_pT=iFJSA1z+3o5NcamFbcw@mail.gmail.com>

Hi Shawn,

Thanks for your detailed explanation! I'm actually not sure how to check
whether residual() is spitting out nan; and also the fitting result (a~0.1,
mu~1, sigma~1.57, b~0.06) I got without bounds still fell in the bounds I
defined [(None,None),(0,pi),(0,None),(None,None)].

On the other hand I've tried the 'minimize' function. I mentioned it in my
reply to Matt's email and here it is:
> Hi Matt,
>
> Thanks so much! I've actually tried the 'minimize' function in
> scipy.optimize. There are several constrained optimization methods in
> 'minimize' and among them 'SLSQP' seems to be equivalent to the method
used
> in fmin_slsqp; I got similar bad fitting results using these two
functions.
> Meanwhile, using the 'L-BFGS-B' method in 'minimize', I can get similar
good
> fitting results as 'leastsq'. But I'm not sure how these two algorithms
> differ.

So yes I've tried the 'L-BFGS-B' method and it is working well. Also, using
the 'lmfit' package as Matt suggested, I can get same good fitting results
as 'L-BFGS-B'. The 'SLSQP' method, however, provides similar bad fitting
results as 'fmin_slsqp'. I think that is because they are using the same
algorithm (Sequential Least SQuares Programming).

Best,
Qing
  '
""


On Wed, Apr 9, 2014 at 9:16 PM, Yuxiang Wang <yw5aj at virginia.edu> wrote:

> Hi Qing,
>
> I was thinking that your function residual() may give nan and this
> would make the fitting uncontrollable.
>
> I think that this may be the case because in your previous emails you
> mentioned the error:
>             Iteration limit exceeded    (Exit mode 9)
>             Current function value: nan
>             Iterations: 101
>             Function evaluations: 1606
>             Gradient evaluations: 101
> The current function value is a "nan".
>
> It might simply be that in the bounds you defined, the algorithm does
> not converge due to the nan; if you get rid of the bounds, it will go
> elsewhere for sure but that does not give what you want.
>
> Could you check that to see whether residual() function is spitting out
> nan?
>
>
> PS: I heard that minimize() should always be used and fmin_slsqp etc.
> should be deprecated... I personally do prefer minimize because then I
> can easily try some other algorithms like L-BFGS-B (this and SLSQP are
> my favorite two!).
>
> -Shawn
>
> On Wed, Apr 9, 2014 at 11:24 AM, Qing Yu <yuqingpsy at gmail.com> wrote:
> > Hi Shawn,
> >
> > Yes I was actually using a 'try...except...' command in my script. It's
> > simply because I want an output when the fitting procedure fails. Here's
> the
> > part of the script:
> >
> > try:
> >        out, fx, its, imode, smode =
> >
> fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
> > full_output=True)
> > except:
> >        out = [nan, nan, nan, nan]
> >
> > But I suppose I won't get the nan output unless the fitting fails?
> >
> > Thanks!
> >
> >
> > Best,
> > Qing
> >
> > On Wed, Apr 9, 2014 at 11:00 AM, Yuxiang Wang <yw5aj at virginia.edu>
> wrote:
> >>
> >> Hi Qing,
> >>
> >> In your description I see a line:
> >>
> >> Current function value: nan
> >>
> >> Is your function outputting a np.nan for certain inputs? That might be
> >> bad for the algorithm to work. Could you check that?
> >>
> >> -Shawn
> >>
> >> On Mon, Apr 7, 2014 at 4:56 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
> >> > Hi Matt,
> >> >
> >> > Thanks so much for your help! I've now changed the residual function
> as
> >> > below:
> >> >
> >> > def residual(a,x,y):
> >> >         return sum((y-f(x,a))**2)
> >> >
> >> > the script seems to be working after the change, but I have another
> >> > problem
> >> > now:
> >> >
> >> > because I've set the upper and lower bounds of the parameters in
> >> > fmin_slsqp
> >> > (and that is why I use fmin_slsqp instead of leastsq):
> >> > bounds=[(None,None),(0,pi),(0,None),(None,None)], if I keep the
> bounds,
> >> > I
> >> > get the error:
> >> >             Iteration limit exceeded    (Exit mode 9)
> >> >             Current function value: nan
> >> >             Iterations: 101
> >> >             Function evaluations: 1606
> >> >             Gradient evaluations: 101
> >> > I tried changing iter to 10000, the error still existed.
> >> > If I delete the bound option, I can get the successful output.
> However,
> >> > the
> >> > fitted values are quite different from what I get using lsqcurvefit in
> >> > matlab. Moreover, if I plot the data, the model fit is indeed worse
> >> > using
> >> > fmin_slsqp. I'm not sure if the difference is caused by the missing
> >> > bounds.
> >> > But the outputs from fmin_slsqp did not go out of bounds anyway. Does
> >> > anyone
> >> > have any idea why I could not set the bounds in fmin_slsqp?
> >> >
> >> > Thanks so much for any help!
> >> >
> >> > Best,
> >> > Qing
> >> >
> >> >
> >> > On Mon, Apr 7, 2014 at 3:48 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
> >> >>
> >> >> Dear SciPy experts,
> >> >>
> >> >> I'm using the fmin_slsqp function in scipy.optimize to fit my data, I
> >> >> first defined the function f and the residual of fitting, and then
> used
> >> >> actual data x and y to fit the curve. However, I encountered an error
> >> >> that I
> >> >> could not explain.
> >> >> Here's a short version of my script:
> >> >>
> >> >> def f(x,a):
> >> >> return a[0]*exp((a[2]**2)*cos(x-a[1])-1)+a[3]
> >> >> def residual(a,x,y):
> >> >> return y-f(x,a)
> >> >>
> >> >> x=[0,pi/6,pi/3,pi/2,2*pi/3,5*pi/6]
> >> >> y=[0.17,0.36,0.61,0.38,0.17,0.16]
> >> >> a0=[1,pi/3,pi/2,0]
> >> >> out, fx, its, imode, smode =
> >> >>
> >> >>
> fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
> >> >> full_output=True)
> >> >>
> >> >> I ended up with the error: "failed in converting 8th argument `g' of
> >> >> _slsqp.slsqp to C/Fortran array." Does anyone have any idea of why
> this
> >> >> happened?
> >> >>
> >> >> Thanks so much in advance for any help!
> >> >>
> >> >> Best,
> >> >> Qing
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Graduate Student
> >> > Qing Yu
> >> > Department of Psychological and Brain Sciences
> >> > Dartmouth College
> >> > Hanover, New Hampshire 03755, US
> >> >
> >> > _______________________________________________
> >> > SciPy-User mailing list
> >> > SciPy-User at scipy.org
> >> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >> >
> >>
> >>
> >>
> >> --
> >> Yuxiang "Shawn" Wang
> >> Gerling Research Lab
> >> University of Virginia
> >> yw5aj at virginia.edu
> >> +1 (434) 284-0836
> >> https://sites.google.com/a/virginia.edu/yw5aj/
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> >
> >
> > --
> > Graduate Student
> > Qing Yu
> > Department of Psychological and Brain Sciences
> > Dartmouth College
> > Hanover, New Hampshire 03755, US
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
>
>
>
> --
> Yuxiang "Shawn" Wang
> Gerling Research Lab
> University of Virginia
> yw5aj at virginia.edu
> +1 (434) 284-0836
> https://sites.google.com/a/virginia.edu/yw5aj/
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Graduate Student
Qing Yu
Department of Psychological and Brain Sciences
Dartmouth College
Hanover, New Hampshire 03755, US
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140410/fd556f87/attachment.html>

From yw5aj at virginia.edu  Thu Apr 10 12:31:57 2014
From: yw5aj at virginia.edu (Yuxiang Wang)
Date: Thu, 10 Apr 2014 12:31:57 -0400
Subject: [SciPy-User] Optimization problem using fmin_slsqp
In-Reply-To: <CAFEPp9muvEJMXo_KxK6-1Ehqs_W_pT=iFJSA1z+3o5NcamFbcw@mail.gmail.com>
References: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
	<CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>
	<CANRv7QoNPi245nT3d=4n4qv4+6Wz9xv1s6aki3ZtL+BE6RuYzw@mail.gmail.com>
	<CAFEPp9=bF3g5YzktsiEjrygZRerkF30vfTxmWtTC-zgF9tmDEQ@mail.gmail.com>
	<CANRv7Qp9abVRZi7PJDP3CWK5NHLOrWy3guOL=8tBVEckumAeEA@mail.gmail.com>
	<CAFEPp9muvEJMXo_KxK6-1Ehqs_W_pT=iFJSA1z+3o5NcamFbcw@mail.gmail.com>
Message-ID: <CANRv7QoiXsNVwDa4eMWn=HJfQLee__MzHiuUNUJJ4LTW=x44-g@mail.gmail.com>

Hi Qing,

Sorry that I have not read the whole thread in my previous postings
and not noticing that you have already got your full code... I rewrote
the code and it works now.

####################################################################################

# -*- coding: utf-8 -*-

"""

Created on Thu Apr 10 12:23:59 2014

temp.py

@author: Yuxiang Wang

"""


import numpy as np

import matplotlib.pyplot as plt

from scipy.optimize import minimize


# Function definitions

def f(x,a):

return a[0]*np.exp((a[2]**2)*np.cos(x-a[1])-1)+a[3]


def residual(a,x,y):

return np.linalg.norm(y-f(x,a))**2


# Assign values

x=[0,np.pi/6,np.pi/3,np.pi/2,2*np.pi/3,5*np.pi/6]

y=[0.17,0.36,0.61,0.38,0.17,0.16]

a0=[1,np.pi/3,np.pi/2,0]


# Fit!

bounds=((None, None), (0, np.pi), (0, None), (None,None))

res = minimize(residual, a0, args=(x, y), method='L-BFGS-B', bounds=bounds)


# Plot

fig, axs = plt.subplots()

axs.plot(x, y, '.k')

axs.plot(x, f(x, res.x), '-r')

####################################################################################

And notice that you can safely change method='L-BFGS-B' to 'SLSQP' and
it worked perfectly as well.

Best,
Shawn

On Thu, Apr 10, 2014 at 11:22 AM, Qing Yu <yuqingpsy at gmail.com> wrote:
> Hi Shawn,
>
> Thanks for your detailed explanation! I'm actually not sure how to check
> whether residual() is spitting out nan; and also the fitting result (a~0.1,
> mu~1, sigma~1.57, b~0.06) I got without bounds still fell in the bounds I
> defined [(None,None),(0,pi),(0,None),(None,None)].
>
> On the other hand I've tried the 'minimize' function. I mentioned it in my
> reply to Matt's email and here it is:
>> Hi Matt,
>>
>> Thanks so much! I've actually tried the 'minimize' function in
>> scipy.optimize. There are several constrained optimization methods in
>> 'minimize' and among them 'SLSQP' seems to be equivalent to the method
>> used
>> in fmin_slsqp; I got similar bad fitting results using these two
>> functions.
>> Meanwhile, using the 'L-BFGS-B' method in 'minimize', I can get similar
>> good
>> fitting results as 'leastsq'. But I'm not sure how these two algorithms
>> differ.
>
> So yes I've tried the 'L-BFGS-B' method and it is working well. Also, using
> the 'lmfit' package as Matt suggested, I can get same good fitting results
> as 'L-BFGS-B'. The 'SLSQP' method, however, provides similar bad fitting
> results as 'fmin_slsqp'. I think that is because they are using the same
> algorithm (Sequential Least SQuares Programming).
>
> Best,
> Qing
>   '
> ""
>
>
> On Wed, Apr 9, 2014 at 9:16 PM, Yuxiang Wang <yw5aj at virginia.edu> wrote:
>>
>> Hi Qing,
>>
>> I was thinking that your function residual() may give nan and this
>> would make the fitting uncontrollable.
>>
>> I think that this may be the case because in your previous emails you
>> mentioned the error:
>>             Iteration limit exceeded    (Exit mode 9)
>>             Current function value: nan
>>             Iterations: 101
>>             Function evaluations: 1606
>>             Gradient evaluations: 101
>> The current function value is a "nan".
>>
>> It might simply be that in the bounds you defined, the algorithm does
>> not converge due to the nan; if you get rid of the bounds, it will go
>> elsewhere for sure but that does not give what you want.
>>
>> Could you check that to see whether residual() function is spitting out
>> nan?
>>
>>
>> PS: I heard that minimize() should always be used and fmin_slsqp etc.
>> should be deprecated... I personally do prefer minimize because then I
>> can easily try some other algorithms like L-BFGS-B (this and SLSQP are
>> my favorite two!).
>>
>> -Shawn
>>
>> On Wed, Apr 9, 2014 at 11:24 AM, Qing Yu <yuqingpsy at gmail.com> wrote:
>> > Hi Shawn,
>> >
>> > Yes I was actually using a 'try...except...' command in my script. It's
>> > simply because I want an output when the fitting procedure fails. Here's
>> > the
>> > part of the script:
>> >
>> > try:
>> >        out, fx, its, imode, smode =
>> >
>> > fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
>> > full_output=True)
>> > except:
>> >        out = [nan, nan, nan, nan]
>> >
>> > But I suppose I won't get the nan output unless the fitting fails?
>> >
>> > Thanks!
>> >
>> >
>> > Best,
>> > Qing
>> >
>> > On Wed, Apr 9, 2014 at 11:00 AM, Yuxiang Wang <yw5aj at virginia.edu>
>> > wrote:
>> >>
>> >> Hi Qing,
>> >>
>> >> In your description I see a line:
>> >>
>> >> Current function value: nan
>> >>
>> >> Is your function outputting a np.nan for certain inputs? That might be
>> >> bad for the algorithm to work. Could you check that?
>> >>
>> >> -Shawn
>> >>
>> >> On Mon, Apr 7, 2014 at 4:56 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
>> >> > Hi Matt,
>> >> >
>> >> > Thanks so much for your help! I've now changed the residual function
>> >> > as
>> >> > below:
>> >> >
>> >> > def residual(a,x,y):
>> >> >         return sum((y-f(x,a))**2)
>> >> >
>> >> > the script seems to be working after the change, but I have another
>> >> > problem
>> >> > now:
>> >> >
>> >> > because I've set the upper and lower bounds of the parameters in
>> >> > fmin_slsqp
>> >> > (and that is why I use fmin_slsqp instead of leastsq):
>> >> > bounds=[(None,None),(0,pi),(0,None),(None,None)], if I keep the
>> >> > bounds,
>> >> > I
>> >> > get the error:
>> >> >             Iteration limit exceeded    (Exit mode 9)
>> >> >             Current function value: nan
>> >> >             Iterations: 101
>> >> >             Function evaluations: 1606
>> >> >             Gradient evaluations: 101
>> >> > I tried changing iter to 10000, the error still existed.
>> >> > If I delete the bound option, I can get the successful output.
>> >> > However,
>> >> > the
>> >> > fitted values are quite different from what I get using lsqcurvefit
>> >> > in
>> >> > matlab. Moreover, if I plot the data, the model fit is indeed worse
>> >> > using
>> >> > fmin_slsqp. I'm not sure if the difference is caused by the missing
>> >> > bounds.
>> >> > But the outputs from fmin_slsqp did not go out of bounds anyway. Does
>> >> > anyone
>> >> > have any idea why I could not set the bounds in fmin_slsqp?
>> >> >
>> >> > Thanks so much for any help!
>> >> >
>> >> > Best,
>> >> > Qing
>> >> >
>> >> >
>> >> > On Mon, Apr 7, 2014 at 3:48 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
>> >> >>
>> >> >> Dear SciPy experts,
>> >> >>
>> >> >> I'm using the fmin_slsqp function in scipy.optimize to fit my data,
>> >> >> I
>> >> >> first defined the function f and the residual of fitting, and then
>> >> >> used
>> >> >> actual data x and y to fit the curve. However, I encountered an
>> >> >> error
>> >> >> that I
>> >> >> could not explain.
>> >> >> Here's a short version of my script:
>> >> >>
>> >> >> def f(x,a):
>> >> >> return a[0]*exp((a[2]**2)*cos(x-a[1])-1)+a[3]
>> >> >> def residual(a,x,y):
>> >> >> return y-f(x,a)
>> >> >>
>> >> >> x=[0,pi/6,pi/3,pi/2,2*pi/3,5*pi/6]
>> >> >> y=[0.17,0.36,0.61,0.38,0.17,0.16]
>> >> >> a0=[1,pi/3,pi/2,0]
>> >> >> out, fx, its, imode, smode =
>> >> >>
>> >> >>
>> >> >> fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
>> >> >> full_output=True)
>> >> >>
>> >> >> I ended up with the error: "failed in converting 8th argument `g' of
>> >> >> _slsqp.slsqp to C/Fortran array." Does anyone have any idea of why
>> >> >> this
>> >> >> happened?
>> >> >>
>> >> >> Thanks so much in advance for any help!
>> >> >>
>> >> >> Best,
>> >> >> Qing
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Graduate Student
>> >> > Qing Yu
>> >> > Department of Psychological and Brain Sciences
>> >> > Dartmouth College
>> >> > Hanover, New Hampshire 03755, US
>> >> >
>> >> > _______________________________________________
>> >> > SciPy-User mailing list
>> >> > SciPy-User at scipy.org
>> >> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Yuxiang "Shawn" Wang
>> >> Gerling Research Lab
>> >> University of Virginia
>> >> yw5aj at virginia.edu
>> >> +1 (434) 284-0836
>> >> https://sites.google.com/a/virginia.edu/yw5aj/
>> >> _______________________________________________
>> >> SciPy-User mailing list
>> >> SciPy-User at scipy.org
>> >> http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>> >
>> >
>> >
>> > --
>> > Graduate Student
>> > Qing Yu
>> > Department of Psychological and Brain Sciences
>> > Dartmouth College
>> > Hanover, New Hampshire 03755, US
>> >
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>> >
>>
>>
>>
>> --
>> Yuxiang "Shawn" Wang
>> Gerling Research Lab
>> University of Virginia
>> yw5aj at virginia.edu
>> +1 (434) 284-0836
>> https://sites.google.com/a/virginia.edu/yw5aj/
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>
>
> --
> Graduate Student
> Qing Yu
> Department of Psychological and Brain Sciences
> Dartmouth College
> Hanover, New Hampshire 03755, US
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Yuxiang "Shawn" Wang
Gerling Research Lab
University of Virginia
yw5aj at virginia.edu
+1 (434) 284-0836
https://sites.google.com/a/virginia.edu/yw5aj/


From jacksilver045 at gmail.com  Thu Apr 10 16:53:46 2014
From: jacksilver045 at gmail.com (=?UTF-8?B?2KzYp9mDINin2YTZgdi22Kk=?=)
Date: Thu, 10 Apr 2014 16:53:46 -0400
Subject: [SciPy-User] Question on spectral derivative
Message-ID: <CAJkLvGUv5c-QAeXDh2Ma4aTTaaWZy9nbfbg0iWv7tVJs96n8Rg@mail.gmail.com>

I have data that form a scalar field on a 2D grid, evenly spaced. The grid
has a finite size. There is no particular periodicity patern in my data.

I want to calculate the value of the gradient at each grid point with a
spectral method. Using simple FFT does not seem to work, especially near
the borders (probably because the data are not periodic). I have ears about
DCT, but I am not sure.

I think that I could extend my grid with one row/column at each of the 4
boundaries, duplicate the real boundary on these new boundary points to get
a zero derivative boundary for the DCT algorithm. However, I am not sure
how to get it working. Another possibility would be to extend my grid in
some way that FFT will not create spurious oscillation near the boundaries,
but again, I my attempts failed so far.

I would appreciate advice on the best way to do this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140410/bd702c3c/attachment.html>

From gdmcbain at freeshell.org  Thu Apr 10 20:20:42 2014
From: gdmcbain at freeshell.org (Geordie McBain)
Date: Fri, 11 Apr 2014 10:20:42 +1000
Subject: [SciPy-User] Question on spectral derivative
In-Reply-To: <CAJkLvGUv5c-QAeXDh2Ma4aTTaaWZy9nbfbg0iWv7tVJs96n8Rg@mail.gmail.com>
References: <CAJkLvGUv5c-QAeXDh2Ma4aTTaaWZy9nbfbg0iWv7tVJs96n8Rg@mail.gmail.com>
Message-ID: <CAB0gBH33cSO_rmnAJ5eYhFmPtupBUqgM6Psydikm3pu8ibRhBA@mail.gmail.com>

2014-04-11 6:53 GMT+10:00 ??? ????? <jacksilver045 at gmail.com>:
> I have data that form a scalar field on a 2D grid, evenly spaced. The grid
> has a finite size. There is no particular periodicity patern in my data.
>
> I want to calculate the value of the gradient at each grid point with a
> spectral method. Using simple FFT does not seem to work, especially near the
> borders (probably because the data are not periodic). I have ears about DCT,
> but I am not sure.
>
> I think that I could extend my grid with one row/column at each of the 4
> boundaries, duplicate the real boundary on these new boundary points to get
> a zero derivative boundary for the DCT algorithm. However, I am not sure how
> to get it working. Another possibility would be to extend my grid in some
> way that FFT will not create spurious oscillation near the boundaries, but
> again, I my attempts failed so far.
>
> I would appreciate advice on the best way to do this.

For a very nice introduction to spectral differentiation, see

Weideman, J. A. C., & S. C. Reddy (2000). A MATLAB differentiation
matrix suite. ACM Transactions on Mathematical Software 26 (4),
465-519.  http://dx.doi.org/10.1145/365723.365727

(The examples are in MATLAB, but the discussion is general and the
code simple enough to be read as pseudo-code.)

Basically, I fear you're like to encounter the problem that evenly
spaced grids aren't suited to spectral differentiation.  Ideally the
grid points should be the Gauss points, like the absciss? of a
Gauss-type quadrature rule; these give us series-representations which
are independent of the boundary conditions.  If you can't change where
the grid points are, you're probably better off with lower order
differentiation, like high-order finite differences, using one-sided
rules near the edges.
   The discrete cosine transform that's used in JPEG gets around the
ultimate ill-conditioning of equidistant nodes by only applying it
after the image has been broken into (8, 8) tiles.
   If your grid wasn't of a finite size, you could use sinc functions,
but they have infinite support; again, this indicates the importance
of boundary conditions.


From yuqingpsy at gmail.com  Thu Apr 10 21:32:56 2014
From: yuqingpsy at gmail.com (Qing Yu)
Date: Thu, 10 Apr 2014 21:32:56 -0400
Subject: [SciPy-User] Optimization problem using fmin_slsqp
In-Reply-To: <CANRv7QoiXsNVwDa4eMWn=HJfQLee__MzHiuUNUJJ4LTW=x44-g@mail.gmail.com>
References: <CAFEPp9m3R8vHtQk4y1Qqmrd_Qa8j1OqXDkxKg3P5JV_pRsRU3A@mail.gmail.com>
	<CAFEPp9kbTt0xM6ALFNmkuOP5tGSLDjQiiWR1JS8WHKtYCDL5xg@mail.gmail.com>
	<CANRv7QoNPi245nT3d=4n4qv4+6Wz9xv1s6aki3ZtL+BE6RuYzw@mail.gmail.com>
	<CAFEPp9=bF3g5YzktsiEjrygZRerkF30vfTxmWtTC-zgF9tmDEQ@mail.gmail.com>
	<CANRv7Qp9abVRZi7PJDP3CWK5NHLOrWy3guOL=8tBVEckumAeEA@mail.gmail.com>
	<CAFEPp9muvEJMXo_KxK6-1Ehqs_W_pT=iFJSA1z+3o5NcamFbcw@mail.gmail.com>
	<CANRv7QoiXsNVwDa4eMWn=HJfQLee__MzHiuUNUJJ4LTW=x44-g@mail.gmail.com>
Message-ID: <CAFEPp9nvEFoQPuyJYGBudf9BLqjn1KOYeOcGD0mr3yTfSM2cdQ@mail.gmail.com>

Thanks so much!


Best,
Qing


On Thu, Apr 10, 2014 at 12:31 PM, Yuxiang Wang <yw5aj at virginia.edu> wrote:

> Hi Qing,
>
> Sorry that I have not read the whole thread in my previous postings
> and not noticing that you have already got your full code... I rewrote
> the code and it works now.
>
>
> ####################################################################################
>
> # -*- coding: utf-8 -*-
>
> """
>
> Created on Thu Apr 10 12:23:59 2014
>
> temp.py
>
> @author: Yuxiang Wang
>
> """
>
>
> import numpy as np
>
> import matplotlib.pyplot as plt
>
> from scipy.optimize import minimize
>
>
> # Function definitions
>
> def f(x,a):
>
> return a[0]*np.exp((a[2]**2)*np.cos(x-a[1])-1)+a[3]
>
>
> def residual(a,x,y):
>
> return np.linalg.norm(y-f(x,a))**2
>
>
> # Assign values
>
> x=[0,np.pi/6,np.pi/3,np.pi/2,2*np.pi/3,5*np.pi/6]
>
> y=[0.17,0.36,0.61,0.38,0.17,0.16]
>
> a0=[1,np.pi/3,np.pi/2,0]
>
>
> # Fit!
>
> bounds=((None, None), (0, np.pi), (0, None), (None,None))
>
> res = minimize(residual, a0, args=(x, y), method='L-BFGS-B', bounds=bounds)
>
>
> # Plot
>
> fig, axs = plt.subplots()
>
> axs.plot(x, y, '.k')
>
> axs.plot(x, f(x, res.x), '-r')
>
>
> ####################################################################################
>
> And notice that you can safely change method='L-BFGS-B' to 'SLSQP' and
> it worked perfectly as well.
>
> Best,
> Shawn
>
> On Thu, Apr 10, 2014 at 11:22 AM, Qing Yu <yuqingpsy at gmail.com> wrote:
> > Hi Shawn,
> >
> > Thanks for your detailed explanation! I'm actually not sure how to check
> > whether residual() is spitting out nan; and also the fitting result
> (a~0.1,
> > mu~1, sigma~1.57, b~0.06) I got without bounds still fell in the bounds I
> > defined [(None,None),(0,pi),(0,None),(None,None)].
> >
> > On the other hand I've tried the 'minimize' function. I mentioned it in
> my
> > reply to Matt's email and here it is:
> >> Hi Matt,
> >>
> >> Thanks so much! I've actually tried the 'minimize' function in
> >> scipy.optimize. There are several constrained optimization methods in
> >> 'minimize' and among them 'SLSQP' seems to be equivalent to the method
> >> used
> >> in fmin_slsqp; I got similar bad fitting results using these two
> >> functions.
> >> Meanwhile, using the 'L-BFGS-B' method in 'minimize', I can get similar
> >> good
> >> fitting results as 'leastsq'. But I'm not sure how these two algorithms
> >> differ.
> >
> > So yes I've tried the 'L-BFGS-B' method and it is working well. Also,
> using
> > the 'lmfit' package as Matt suggested, I can get same good fitting
> results
> > as 'L-BFGS-B'. The 'SLSQP' method, however, provides similar bad fitting
> > results as 'fmin_slsqp'. I think that is because they are using the same
> > algorithm (Sequential Least SQuares Programming).
> >
> > Best,
> > Qing
> >   '
> > ""
> >
> >
> > On Wed, Apr 9, 2014 at 9:16 PM, Yuxiang Wang <yw5aj at virginia.edu> wrote:
> >>
> >> Hi Qing,
> >>
> >> I was thinking that your function residual() may give nan and this
> >> would make the fitting uncontrollable.
> >>
> >> I think that this may be the case because in your previous emails you
> >> mentioned the error:
> >>             Iteration limit exceeded    (Exit mode 9)
> >>             Current function value: nan
> >>             Iterations: 101
> >>             Function evaluations: 1606
> >>             Gradient evaluations: 101
> >> The current function value is a "nan".
> >>
> >> It might simply be that in the bounds you defined, the algorithm does
> >> not converge due to the nan; if you get rid of the bounds, it will go
> >> elsewhere for sure but that does not give what you want.
> >>
> >> Could you check that to see whether residual() function is spitting out
> >> nan?
> >>
> >>
> >> PS: I heard that minimize() should always be used and fmin_slsqp etc.
> >> should be deprecated... I personally do prefer minimize because then I
> >> can easily try some other algorithms like L-BFGS-B (this and SLSQP are
> >> my favorite two!).
> >>
> >> -Shawn
> >>
> >> On Wed, Apr 9, 2014 at 11:24 AM, Qing Yu <yuqingpsy at gmail.com> wrote:
> >> > Hi Shawn,
> >> >
> >> > Yes I was actually using a 'try...except...' command in my script.
> It's
> >> > simply because I want an output when the fitting procedure fails.
> Here's
> >> > the
> >> > part of the script:
> >> >
> >> > try:
> >> >        out, fx, its, imode, smode =
> >> >
> >> >
> fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
> >> > full_output=True)
> >> > except:
> >> >        out = [nan, nan, nan, nan]
> >> >
> >> > But I suppose I won't get the nan output unless the fitting fails?
> >> >
> >> > Thanks!
> >> >
> >> >
> >> > Best,
> >> > Qing
> >> >
> >> > On Wed, Apr 9, 2014 at 11:00 AM, Yuxiang Wang <yw5aj at virginia.edu>
> >> > wrote:
> >> >>
> >> >> Hi Qing,
> >> >>
> >> >> In your description I see a line:
> >> >>
> >> >> Current function value: nan
> >> >>
> >> >> Is your function outputting a np.nan for certain inputs? That might
> be
> >> >> bad for the algorithm to work. Could you check that?
> >> >>
> >> >> -Shawn
> >> >>
> >> >> On Mon, Apr 7, 2014 at 4:56 PM, Qing Yu <yuqingpsy at gmail.com> wrote:
> >> >> > Hi Matt,
> >> >> >
> >> >> > Thanks so much for your help! I've now changed the residual
> function
> >> >> > as
> >> >> > below:
> >> >> >
> >> >> > def residual(a,x,y):
> >> >> >         return sum((y-f(x,a))**2)
> >> >> >
> >> >> > the script seems to be working after the change, but I have another
> >> >> > problem
> >> >> > now:
> >> >> >
> >> >> > because I've set the upper and lower bounds of the parameters in
> >> >> > fmin_slsqp
> >> >> > (and that is why I use fmin_slsqp instead of leastsq):
> >> >> > bounds=[(None,None),(0,pi),(0,None),(None,None)], if I keep the
> >> >> > bounds,
> >> >> > I
> >> >> > get the error:
> >> >> >             Iteration limit exceeded    (Exit mode 9)
> >> >> >             Current function value: nan
> >> >> >             Iterations: 101
> >> >> >             Function evaluations: 1606
> >> >> >             Gradient evaluations: 101
> >> >> > I tried changing iter to 10000, the error still existed.
> >> >> > If I delete the bound option, I can get the successful output.
> >> >> > However,
> >> >> > the
> >> >> > fitted values are quite different from what I get using lsqcurvefit
> >> >> > in
> >> >> > matlab. Moreover, if I plot the data, the model fit is indeed worse
> >> >> > using
> >> >> > fmin_slsqp. I'm not sure if the difference is caused by the missing
> >> >> > bounds.
> >> >> > But the outputs from fmin_slsqp did not go out of bounds anyway.
> Does
> >> >> > anyone
> >> >> > have any idea why I could not set the bounds in fmin_slsqp?
> >> >> >
> >> >> > Thanks so much for any help!
> >> >> >
> >> >> > Best,
> >> >> > Qing
> >> >> >
> >> >> >
> >> >> > On Mon, Apr 7, 2014 at 3:48 PM, Qing Yu <yuqingpsy at gmail.com>
> wrote:
> >> >> >>
> >> >> >> Dear SciPy experts,
> >> >> >>
> >> >> >> I'm using the fmin_slsqp function in scipy.optimize to fit my
> data,
> >> >> >> I
> >> >> >> first defined the function f and the residual of fitting, and then
> >> >> >> used
> >> >> >> actual data x and y to fit the curve. However, I encountered an
> >> >> >> error
> >> >> >> that I
> >> >> >> could not explain.
> >> >> >> Here's a short version of my script:
> >> >> >>
> >> >> >> def f(x,a):
> >> >> >> return a[0]*exp((a[2]**2)*cos(x-a[1])-1)+a[3]
> >> >> >> def residual(a,x,y):
> >> >> >> return y-f(x,a)
> >> >> >>
> >> >> >> x=[0,pi/6,pi/3,pi/2,2*pi/3,5*pi/6]
> >> >> >> y=[0.17,0.36,0.61,0.38,0.17,0.16]
> >> >> >> a0=[1,pi/3,pi/2,0]
> >> >> >> out, fx, its, imode, smode =
> >> >> >>
> >> >> >>
> >> >> >>
> fmin_slsqp(residual,a0,args=(x,y),bounds=[(None,None),(0,pi),(0,None),(None,None)],
> >> >> >> full_output=True)
> >> >> >>
> >> >> >> I ended up with the error: "failed in converting 8th argument `g'
> of
> >> >> >> _slsqp.slsqp to C/Fortran array." Does anyone have any idea of why
> >> >> >> this
> >> >> >> happened?
> >> >> >>
> >> >> >> Thanks so much in advance for any help!
> >> >> >>
> >> >> >> Best,
> >> >> >> Qing
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Graduate Student
> >> >> > Qing Yu
> >> >> > Department of Psychological and Brain Sciences
> >> >> > Dartmouth College
> >> >> > Hanover, New Hampshire 03755, US
> >> >> >
> >> >> > _______________________________________________
> >> >> > SciPy-User mailing list
> >> >> > SciPy-User at scipy.org
> >> >> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Yuxiang "Shawn" Wang
> >> >> Gerling Research Lab
> >> >> University of Virginia
> >> >> yw5aj at virginia.edu
> >> >> +1 (434) 284-0836
> >> >> https://sites.google.com/a/virginia.edu/yw5aj/
> >> >> _______________________________________________
> >> >> SciPy-User mailing list
> >> >> SciPy-User at scipy.org
> >> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Graduate Student
> >> > Qing Yu
> >> > Department of Psychological and Brain Sciences
> >> > Dartmouth College
> >> > Hanover, New Hampshire 03755, US
> >> >
> >> > _______________________________________________
> >> > SciPy-User mailing list
> >> > SciPy-User at scipy.org
> >> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >> >
> >>
> >>
> >>
> >> --
> >> Yuxiang "Shawn" Wang
> >> Gerling Research Lab
> >> University of Virginia
> >> yw5aj at virginia.edu
> >> +1 (434) 284-0836
> >> https://sites.google.com/a/virginia.edu/yw5aj/
> >> _______________________________________________
> >> SciPy-User mailing list
> >> SciPy-User at scipy.org
> >> http://mail.scipy.org/mailman/listinfo/scipy-user
> >
> >
> >
> >
> > --
> > Graduate Student
> > Qing Yu
> > Department of Psychological and Brain Sciences
> > Dartmouth College
> > Hanover, New Hampshire 03755, US
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
>
>
>
> --
> Yuxiang "Shawn" Wang
> Gerling Research Lab
> University of Virginia
> yw5aj at virginia.edu
> +1 (434) 284-0836
> https://sites.google.com/a/virginia.edu/yw5aj/
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140410/a0620b5e/attachment.html>

From bouloumag at gmail.com  Thu Apr 10 22:00:21 2014
From: bouloumag at gmail.com (Darcoux Christine)
Date: Thu, 10 Apr 2014 22:00:21 -0400
Subject: [SciPy-User] create sparse matrix from diagonal arrays
Message-ID: <CAD6uFgmnc8gXOPF8hs3na=hiKRgAyY6ajYd_k38vpkDGxniZHA@mail.gmail.com>

I would like to solve a PDE using an implicit finite volume discretization.
Since the problem is 3D, the matrix should be septadiagonal. I have
computed the 7 diagonals (for top, bottom, east, west, north south and
central point) in 7 different arrays.

Now, I would like to create a sparse matrix object from these arrays, but I
feel a little lost with the documentation of scipy.sparse.spdiags.

Help would be greatly appreciated.

Christine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140410/9acaf56f/attachment.html>

From gdmcbain at freeshell.org  Thu Apr 10 22:53:51 2014
From: gdmcbain at freeshell.org (Geordie McBain)
Date: Fri, 11 Apr 2014 12:53:51 +1000
Subject: [SciPy-User] create sparse matrix from diagonal arrays
In-Reply-To: <CAD6uFgmnc8gXOPF8hs3na=hiKRgAyY6ajYd_k38vpkDGxniZHA@mail.gmail.com>
References: <CAD6uFgmnc8gXOPF8hs3na=hiKRgAyY6ajYd_k38vpkDGxniZHA@mail.gmail.com>
Message-ID: <CAB0gBH0XU43yVS0-X8ESFM__w6VpFYM1iFwpa_uRhtpGxNoQRw@mail.gmail.com>

2014-04-11 12:00 GMT+10:00 Darcoux Christine <bouloumag at gmail.com>:
> I would like to solve a PDE using an implicit finite volume discretization.
> Since the problem is 3D, the matrix should be septadiagonal. I have computed
> the 7 diagonals (for top, bottom, east, west, north south and central point)
> in 7 different arrays.
>
> Now, I would like to create a sparse matrix object from these arrays, but I
> feel a little lost with the documentation of scipy.sparse.spdiags.
>
> Help would be greatly appreciated.

Put the diagonals, padded to the length of the main diagonal
(postpadded for the subdiagonals and prepadded for the superdiagonals,
it doesn't matter what with) as the rows of an array, or elements of a
list or other iterable.

For example, to generate the classical tridiagonal one-dimensional Laplacian:

>>> spdiags([[1, 1, np.nan], [-2, -2, -2], [np.nan, 1, 1]], [-1, 0, 1], 3, 3).toarray()
array([[-2.,  1.,  0.],
       [ 1., -2.,  1.],
       [ 0.,  1., -2.]])


From tony at maths.lth.se  Fri Apr 11 02:26:18 2014
From: tony at maths.lth.se (Tony Stillfjord)
Date: Fri, 11 Apr 2014 08:26:18 +0200
Subject: [SciPy-User] create sparse matrix from diagonal arrays
In-Reply-To: <CAB0gBH0XU43yVS0-X8ESFM__w6VpFYM1iFwpa_uRhtpGxNoQRw@mail.gmail.com>
References: <CAD6uFgmnc8gXOPF8hs3na=hiKRgAyY6ajYd_k38vpkDGxniZHA@mail.gmail.com>
	<CAB0gBH0XU43yVS0-X8ESFM__w6VpFYM1iFwpa_uRhtpGxNoQRw@mail.gmail.com>
Message-ID: <CAEWZG1boLaczJr959Uunit2xQtp7jBj_xuRB+GRyd-CzbhegVw@mail.gmail.com>

On Fri, Apr 11, 2014 at 4:53 AM, Geordie McBain <gdmcbain at freeshell.org>wrote:

> 2014-04-11 12:00 GMT+10:00 Darcoux Christine <bouloumag at gmail.com>:
> > I would like to solve a PDE using an implicit finite volume
> discretization.
> > Since the problem is 3D, the matrix should be septadiagonal. I have
> computed
> > the 7 diagonals (for top, bottom, east, west, north south and central
> point)
> > in 7 different arrays.
> >
> > Now, I would like to create a sparse matrix object from these arrays,
> but I
> > feel a little lost with the documentation of scipy.sparse.spdiags.
> >
> > Help would be greatly appreciated.
>
> Put the diagonals, padded to the length of the main diagonal
> (postpadded for the subdiagonals and prepadded for the superdiagonals,
> it doesn't matter what with) as the rows of an array, or elements of a
> list or other iterable.
>
> For example, to generate the classical tridiagonal one-dimensional
> Laplacian:
>
> >>> spdiags([[1, 1, np.nan], [-2, -2, -2], [np.nan, 1, 1]], [-1, 0, 1], 3,
> 3).toarray()
> array([[-2.,  1.,  0.],
>        [ 1., -2.,  1.],
>        [ 0.,  1., -2.]])
>


Or if you don't want to pad your diagonals, use the scipy.sparse.diags
function instead. It was included in 0.11.0. Then you would have something
like

diags([top, bottom, east, west, north, south, central], [i0, i1, i2, i3,
i4, i5, i6])

where the i's are the corresponding diagonal offsets. For example, i6 would
be 0.

Simple example:

>>> e0 = array([1, 2, 3])
>>> e1 = array([4, 5])
>>> e2 = array([6])
>>> diags([e0, e1, e2], [0, 1, -2]).toarray()
array([[ 1.,  4.,  0.],
       [ 0.,  2.,  5.],
       [ 6.,  0.,  3.]])

Tony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140411/e4ce4ae5/attachment.html>

From jacksilver045 at gmail.com  Fri Apr 11 13:52:58 2014
From: jacksilver045 at gmail.com (=?UTF-8?B?2KzYp9mDINin2YTZgdi22Kk=?=)
Date: Fri, 11 Apr 2014 13:52:58 -0400
Subject: [SciPy-User] Question on spectral derivative
In-Reply-To: <CAB0gBH33cSO_rmnAJ5eYhFmPtupBUqgM6Psydikm3pu8ibRhBA@mail.gmail.com>
References: <CAJkLvGUv5c-QAeXDh2Ma4aTTaaWZy9nbfbg0iWv7tVJs96n8Rg@mail.gmail.com>
	<CAB0gBH33cSO_rmnAJ5eYhFmPtupBUqgM6Psydikm3pu8ibRhBA@mail.gmail.com>
Message-ID: <CAJkLvGW4Qa7cOq32LyYjq+SoSndZBW0mAQRCFgJmFugQp+3-0w@mail.gmail.com>

On Thu, Apr 10, 2014 at 8:20 PM, Geordie McBain <gdmcbain at freeshell.org>wrote:

> 2014-04-11 6:53 GMT+10:00 ??? ????? <jacksilver045 at gmail.com>:
> > I have data that form a scalar field on a 2D grid, evenly spaced. The
> grid
> > has a finite size. There is no particular periodicity patern in my data.
> >
> > I want to calculate the value of the gradient at each grid point with a
> > spectral method. Using simple FFT does not seem to work, especially near
> the
> > borders (probably because the data are not periodic). I have ears about
> DCT,
> > but I am not sure.
> >
> > I think that I could extend my grid with one row/column at each of the 4
> > boundaries, duplicate the real boundary on these new boundary points to
> get
> > a zero derivative boundary for the DCT algorithm. However, I am not sure
> how
> > to get it working. Another possibility would be to extend my grid in some
> > way that FFT will not create spurious oscillation near the boundaries,
> but
> > again, I my attempts failed so far.
> >
> > I would appreciate advice on the best way to do this.
>
> For a very nice introduction to spectral differentiation, see
>
> Weideman, J. A. C., & S. C. Reddy (2000). A MATLAB differentiation
> matrix suite. ACM Transactions on Mathematical Software 26 (4),
> 465-519.  http://dx.doi.org/10.1145/365723.365727
>
> (The examples are in MATLAB, but the discussion is general and the
> code simple enough to be read as pseudo-code.)
>
> Basically, I fear you're like to encounter the problem that evenly
> spaced grids aren't suited to spectral differentiation.  Ideally the
> grid points should be the Gauss points, like the absciss? of a
> Gauss-type quadrature rule; these give us series-representations which
> are independent of the boundary conditions.  If you can't change where
> the grid points are, you're probably better off with lower order
> differentiation, like high-order finite differences, using one-sided
> rules near the edges.
>    The discrete cosine transform that's used in JPEG gets around the
> ultimate ill-conditioning of equidistant nodes by only applying it
> after the image has been broken into (8, 8) tiles.
>    If your grid wasn't of a finite size, you could use sinc functions,
> but they have infinite support; again, this indicates the importance
> of boundary conditions.
>

Thank you very much for your answer. I will have a look at the reference
paper. Very appreciated.

I tried first with centered finite difference, using the five-points
stencil described on
http://en.wikipedia.org/wiki/Five-point_stencil#First_derivative
but it does not seems to be accurate enough for my needs.

I think the lack of accuracy is caused by the fact that although the slope
is being computed at x, the value of the function at x is not involved in
the finite difference formula. As far as I know, there is no finite
difference differentiation method that can avoid this problem on a
non-staggered grid. This was the starting point of my investigation for
spectral differentiation methods.

Unless I could find another differentiation method, I will try to see what
are the implications of a change of the position of the grid points. At
first, I do not feel very enthusiast about this because, for the same
number of grid point,  a gaussian grid will have exaggeratedly high
resolution near the boundaries and less resolution at the center of the
grid.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140411/34683a0c/attachment.html>

From anthony.j.mannucci at jpl.nasa.gov  Sun Apr 13 17:16:33 2014
From: anthony.j.mannucci at jpl.nasa.gov (Mannucci, Anthony J (335G))
Date: Sun, 13 Apr 2014 21:16:33 +0000
Subject: [SciPy-User] Can create, but not access,
	netCDF variables with no dimension
Message-ID: <CF704CBE.83A5C%anthony.j.mannucci@jpl.nasa.gov>

I am trying to read a netcdf file that has variables created without dimension (they are scalars). Since the file is obtained externally, and written without python, I cannot modify the file. So, the variable I am trying to access is not accessible using scipy.io.netcdf. (I can see the variable with ncdump).

I cannot fully reproduce the problem here, although I can reproduce something similar. Here is some code for discussion:
---
import scipy.io.netcdf as CDF
f = CDF.netcdf_file('simple.nc', 'w')
f.createDimension('time', 10)
time = f.createVariable('time', 'i', ('time',))
time[:] = np.arange(10)
nodimvar = f.createVariable('nodimvar','f8',()) # Guessing this creates a scalar variable.
nodimvar = 8.0 # nodimvar[:] fails with an error. Scalar not indexable.
f.close()

f = CDF.netcdf_file('simple.nc', 'r')
print f.variables.keys()
print f.variables['time'][:]
print f.variables['nodimvar'] # Does not print variable value, just the object.
print f.variables['nodimvar'][:] # Error, scalar is not indexable.
f.close()
-----
The variable name shows up in f.variables.keys(). However, I do not know how to access the variable. Appending [:] to gives an error "IndexError: 0-d arrays can't be indexed.".

In the file I was provided, not included here because it's very large, the variable with no dimensions does not even show with up in keys().

I note that the package netCDF4 seems to handle this case correctly. Even zero-dimension variables are accessed via [:]. No error is generated and the value of the scalar is returned.

Thanks!

-Tony

--
Tony Mannucci
Supervisor, Ionospheric and Atmospheric Remote Sensing Group
 Mail-Stop 138-308,                     Tel > (818) 354-1699
 Jet Propulsion Laboratory,              Fax > (818) 393-5115
 California Institute of Technology,     Email > Tony.Mannucci at jpl.nasa.gov
 4800 Oak Grove Drive,                   http://scienceandtechnology.jpl.nasa.gov/people/a_mannucci/
 Pasadena, CA 91109

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140413/862ee0bc/attachment.html>

From guziy.sasha at gmail.com  Sun Apr 13 20:12:46 2014
From: guziy.sasha at gmail.com (Oleksandr Huziy)
Date: Sun, 13 Apr 2014 20:12:46 -0400
Subject: [SciPy-User] Can create, but not access,
	netCDF variables with no dimension
In-Reply-To: <CF704CBE.83A5C%anthony.j.mannucci@jpl.nasa.gov>
References: <CF704CBE.83A5C%anthony.j.mannucci@jpl.nasa.gov>
Message-ID: <CAN3p1sX8cm0W2XSvSLc6tpXUEt3EFGMyoPaqrstHDC1PGShTyA@mail.gmail.com>

Can you share ncdump -h on the file in question with the variable in
question indicated?

Cheers


2014-04-13 17:16 GMT-04:00 Mannucci, Anthony J (335G) <
anthony.j.mannucci at jpl.nasa.gov>:

>   I am trying to read a netcdf file that has variables created without
> dimension (they are scalars). Since the file is obtained externally, and
> written without python, I cannot modify the file. So, the variable I am
> trying to access is not accessible using scipy.io.netcdf. (I can see the
> variable with ncdump).
>
>  I cannot fully reproduce the problem here, although I can reproduce
> something similar. Here is some code for discussion:
> ---
>  import scipy.io.netcdf as CDF
> f = CDF.netcdf_file('simple.nc', 'w')
> f.createDimension('time', 10)
> time = f.createVariable('time', 'i', ('time',))
> time[:] = np.arange(10)
> nodimvar = f.createVariable('nodimvar','f8',()) # Guessing this creates a
> scalar variable.
> nodimvar = 8.0 # nodimvar[:] fails with an error. Scalar not indexable.
> f.close()
>
>  f = CDF.netcdf_file('simple.nc', 'r')
> print f.variables.keys()
> print f.variables['time'][:]
> print f.variables['nodimvar'] # Does not print variable value, just the
> object.
> print f.variables['nodimvar'][:] # Error, scalar is not indexable.
> f.close()
>  -----
> The variable name shows up in f.variables.keys(). However, I do not know
> how to access the variable. Appending [:] to gives an error "IndexError:
> 0-d arrays can't be indexed.".
>
>  In the file I was provided, not included here because it's very large,
> the variable with no dimensions does not even show with up in keys().
>
>  I note that the package netCDF4 seems to handle this case correctly.
> Even zero-dimension variables are accessed via [:]. No error is generated
> and the value of the scalar is returned.
>
>  Thanks!
>
>  -Tony
>
>   --
>  Tony Mannucci
> Supervisor, Ionospheric and Atmospheric Remote Sensing Group
>  Mail-Stop 138-308,                     Tel > (818) 354-1699
>  Jet Propulsion Laboratory,              Fax > (818) 393-5115
>  California Institute of Technology,     Email >
> Tony.Mannucci at jpl.nasa.gov
>  4800 Oak Grove Drive,
> http://scienceandtechnology.jpl.nasa.gov/people/a_mannucci/
>  Pasadena, CA 91109
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Sasha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140413/20a9f1c5/attachment.html>

From guziy.sasha at gmail.com  Sun Apr 13 21:49:13 2014
From: guziy.sasha at gmail.com (Oleksandr Huziy)
Date: Sun, 13 Apr 2014 21:49:13 -0400
Subject: [SciPy-User] Can create, but not access,
	netCDF variables with no dimension
In-Reply-To: <CF704CBE.83A5C%anthony.j.mannucci@jpl.nasa.gov>
References: <CF704CBE.83A5C%anthony.j.mannucci@jpl.nasa.gov>
Message-ID: <CAN3p1sU+L7TgCR-3JK3NsZ=b8vwsGxt=+-yK0vyfKinzE+bTyA@mail.gmail.com>

This seems to work fine:

import numpy as np
import scipy.io.netcdf as CDF
f = CDF.netcdf_file('simple.nc', 'w')
f.createDimension('time', 10)
time = f.createVariable('time', 'i', ('time',))
time[:] = np.arange(10)
nodimvar = f.createVariable('nodimvar','f8',()) # Guessing this creates a
scalar variable.
*nodimvar.assignValue(8.0)* # nodimvar[:] fails with an error. Scalar not
indexable.
f.close()

f = CDF.netcdf_file('simple.nc', 'r')
print f.variables.keys()
print f.variables['time'][:]
print f.variables['nodimvar'] # Does not print variable value, just the
object.
the_var = f.variables['nodimvar'] # Error, scalar is not indexable.
*print the_var.getValue()*
f.close()

here is the demo notebook:
http://nbviewer.ipython.org/urls/raw.githubusercontent.com/guziy/PyNotebooks/master/scipy_netcdf.ipynb?create=1

Cheers


2014-04-13 17:16 GMT-04:00 Mannucci, Anthony J (335G) <
anthony.j.mannucci at jpl.nasa.gov>:

>   I am trying to read a netcdf file that has variables created without
> dimension (they are scalars). Since the file is obtained externally, and
> written without python, I cannot modify the file. So, the variable I am
> trying to access is not accessible using scipy.io.netcdf. (I can see the
> variable with ncdump).
>
>  I cannot fully reproduce the problem here, although I can reproduce
> something similar. Here is some code for discussion:
> ---
>  import scipy.io.netcdf as CDF
> f = CDF.netcdf_file('simple.nc', 'w')
> f.createDimension('time', 10)
> time = f.createVariable('time', 'i', ('time',))
> time[:] = np.arange(10)
> nodimvar = f.createVariable('nodimvar','f8',()) # Guessing this creates a
> scalar variable.
> nodimvar = 8.0 # nodimvar[:] fails with an error. Scalar not indexable.
> f.close()
>
>  f = CDF.netcdf_file('simple.nc', 'r')
> print f.variables.keys()
> print f.variables['time'][:]
> print f.variables['nodimvar'] # Does not print variable value, just the
> object.
> print f.variables['nodimvar'][:] # Error, scalar is not indexable.
> f.close()
>  -----
> The variable name shows up in f.variables.keys(). However, I do not know
> how to access the variable. Appending [:] to gives an error "IndexError:
> 0-d arrays can't be indexed.".
>
>  In the file I was provided, not included here because it's very large,
> the variable with no dimensions does not even show with up in keys().
>
>  I note that the package netCDF4 seems to handle this case correctly.
> Even zero-dimension variables are accessed via [:]. No error is generated
> and the value of the scalar is returned.
>
>  Thanks!
>
>  -Tony
>
>   --
>  Tony Mannucci
> Supervisor, Ionospheric and Atmospheric Remote Sensing Group
>  Mail-Stop 138-308,                     Tel > (818) 354-1699
>  Jet Propulsion Laboratory,              Fax > (818) 393-5115
>  California Institute of Technology,     Email >
> Tony.Mannucci at jpl.nasa.gov
>  4800 Oak Grove Drive,
> http://scienceandtechnology.jpl.nasa.gov/people/a_mannucci/
>  Pasadena, CA 91109
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Sasha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140413/0bd3b7d4/attachment.html>

From fdg at voo.it  Mon Apr 14 06:00:54 2014
From: fdg at voo.it (Francesco De Gasperin)
Date: Mon, 14 Apr 2014 12:00:54 +0200
Subject: [SciPy-User] defining new distributions with rv_continuous
Message-ID: <534BB1D6.8060000@voo.it>

Hi,

I have some problems in defining a new distribution using rv_continuous.
As an example I tried to create a gaussian defined only for positive 
values of the random variable.

This is the code:

###########################################
from scipy.stats import distributions
class gauss_pos(distributions.rv_continuous):
    def _pdf(self, x):
         if x <= 0: return 0
         sigma = 1.
         c = 0.
         return 1/(np.sqrt(2*np.pi)*sigma) * np.exp( -(x-c)**2 / 
(2*sigma**2) )

t_obj = gauss_pos(name="gauss_pos")
print (t_obj.rvs(size=10))
#############################################

but when I run it I always end up with:
OverflowError: (34, 'Numerical result out of range')

Then, I removed the "if x <= 0: return 0" line and I tried to define the 
"a" and "b" parameters when creating the t_obj. Defining only the "a=0" 
produces the same error, while using something as:

t_obj = gauss_pos(a=0,b=2)

instead produces a result of this type:
[ 2.          2.          1.5460072   1.09621782  1.62184086  0.42107306
   2.          1.09281742  0.82957403  2.        ]

which I still do not understand... it seems that "b" is returned every 
time the constraint that the random var must be between a and b is 
violated...

any help is appreciated!
cheers,
Francesco


From evgeny.burovskiy at gmail.com  Mon Apr 14 06:13:09 2014
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Mon, 14 Apr 2014 11:13:09 +0100
Subject: [SciPy-User] defining new distributions with rv_continuous
In-Reply-To: <534BB1D6.8060000@voo.it>
References: <534BB1D6.8060000@voo.it>
Message-ID: <CAMRo0itnV7Di=_CNuLwrV4BN_hSem4HpwCPojigZ6pNeD07FgQ@mail.gmail.com>

In this specific case, have a look at truncnorm:

http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.stats.truncnorm.html

>>> import scipy.stats as stats
>>> import numpy as np
>>> stats.truncnorm.pdf(0.1, 0, np.inf)
0.79390509495402362
>>> stats.truncnorm.pdf(-0.1, 0, np.inf)
0.0



On Mon, Apr 14, 2014 at 11:00 AM, Francesco De Gasperin <fdg at voo.it> wrote:
> Hi,
>
> I have some problems in defining a new distribution using rv_continuous.
> As an example I tried to create a gaussian defined only for positive
> values of the random variable.
>
> This is the code:
>
> ###########################################
> from scipy.stats import distributions
> class gauss_pos(distributions.rv_continuous):
>     def _pdf(self, x):
>          if x <= 0: return 0
>          sigma = 1.
>          c = 0.
>          return 1/(np.sqrt(2*np.pi)*sigma) * np.exp( -(x-c)**2 /
> (2*sigma**2) )
>
> t_obj = gauss_pos(name="gauss_pos")
> print (t_obj.rvs(size=10))
> #############################################
>
> but when I run it I always end up with:
> OverflowError: (34, 'Numerical result out of range')
>
> Then, I removed the "if x <= 0: return 0" line and I tried to define the
> "a" and "b" parameters when creating the t_obj. Defining only the "a=0"
> produces the same error, while using something as:
>
> t_obj = gauss_pos(a=0,b=2)
>
> instead produces a result of this type:
> [ 2.          2.          1.5460072   1.09621782  1.62184086  0.42107306
>    2.          1.09281742  0.82957403  2.        ]
>
> which I still do not understand... it seems that "b" is returned every
> time the constraint that the random var must be between a and b is
> violated...
>
> any help is appreciated!
> cheers,
> Francesco
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From josef.pktd at gmail.com  Mon Apr 14 06:18:33 2014
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 14 Apr 2014 06:18:33 -0400
Subject: [SciPy-User] defining new distributions with rv_continuous
In-Reply-To: <CAMRo0itnV7Di=_CNuLwrV4BN_hSem4HpwCPojigZ6pNeD07FgQ@mail.gmail.com>
References: <534BB1D6.8060000@voo.it>
	<CAMRo0itnV7Di=_CNuLwrV4BN_hSem4HpwCPojigZ6pNeD07FgQ@mail.gmail.com>
Message-ID: <CAMMTP+DXc_xWJ=A5TBWXw-pMfC_OCuDrP7_JTivvysvusX-R0Q@mail.gmail.com>

On Mon, Apr 14, 2014 at 6:13 AM, Evgeni Burovski
<evgeny.burovskiy at gmail.com> wrote:
> In this specific case, have a look at truncnorm:
>
> http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.stats.truncnorm.html
>
>>>> import scipy.stats as stats
>>>> import numpy as np
>>>> stats.truncnorm.pdf(0.1, 0, np.inf)
> 0.79390509495402362
>>>> stats.truncnorm.pdf(-0.1, 0, np.inf)
> 0.0
>
>
>
> On Mon, Apr 14, 2014 at 11:00 AM, Francesco De Gasperin <fdg at voo.it> wrote:
>> Hi,
>>
>> I have some problems in defining a new distribution using rv_continuous.
>> As an example I tried to create a gaussian defined only for positive
>> values of the random variable.
>>
>> This is the code:
>>
>> ###########################################
>> from scipy.stats import distributions
>> class gauss_pos(distributions.rv_continuous):
>>     def _pdf(self, x):
>>          if x <= 0: return 0
>>          sigma = 1.
>>          c = 0.
>>          return 1/(np.sqrt(2*np.pi)*sigma) * np.exp( -(x-c)**2 /
>> (2*sigma**2) )
>>
>> t_obj = gauss_pos(name="gauss_pos")
>> print (t_obj.rvs(size=10))
>> #############################################
>>
>> but when I run it I always end up with:
>> OverflowError: (34, 'Numerical result out of range')
>>
>> Then, I removed the "if x <= 0: return 0" line and I tried to define the
>> "a" and "b" parameters when creating the t_obj. Defining only the "a=0"
>> produces the same error, while using something as:
>>
>> t_obj = gauss_pos(a=0,b=2)
>>
>> instead produces a result of this type:
>> [ 2.          2.          1.5460072   1.09621782  1.62184086  0.42107306
>>    2.          1.09281742  0.82957403  2.        ]
>>
>> which I still do not understand... it seems that "b" is returned every
>> time the constraint that the random var must be between a and b is
>> violated...

The specific problem is that your pdf is not normalized correctly to
integrate to 1.
If you truncate the normal distribution, then you need to normalize
for the truncated probability.

The ppf will not be correct, and it is used in to create the rvs in a
generic way.

As Evgeni pointed out truncnorm, the best way is to look at the source
code for some distributions and use them as pattern.

There is also halfnorm.

Josef


>>
>> any help is appreciated!
>> cheers,
>> Francesco
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From fdg at voo.it  Mon Apr 14 06:23:52 2014
From: fdg at voo.it (Francesco De Gasperin)
Date: Mon, 14 Apr 2014 12:23:52 +0200
Subject: [SciPy-User] defining new distributions with,	rv_continuous
Message-ID: <534BB738.9060602@voo.it>

Thanks for the tip. I am actually working with a quite more complex 
distribution which is not a simple truncated normal distribution. My 
example was to simplify the code which generates the problems.

 > In this specific case, have a look at truncnorm:
 >
 > 
http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.stats.truncnorm.html
 >
 > import scipy.stats as stats
 > import numpy as np
 > stats.truncnorm.pdf(0.1, 0, np.inf)
 > 0.79390509495402362
 > stats.truncnorm.pdf(-0.1, 0, np.inf)
 > 0.0


From maria-rosaria.antonelli at curie.fr  Mon Apr 14 06:28:05 2014
From: maria-rosaria.antonelli at curie.fr (Antonelli Maria Rosaria)
Date: Mon, 14 Apr 2014 10:28:05 +0000
Subject: [SciPy-User] Saving into file.txt
Message-ID: <CF71840C.782C%maria-rosaria.antonelli@curie.fr>

Hi, 

I need to save same data arrays into a file.txt.
I would like to save them as different columns.
At the moment I can manage to save them as different lines.


Can you please give me some hints ?
Best regards, 
Rosa



From evgeny.burovskiy at gmail.com  Mon Apr 14 06:29:34 2014
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Mon, 14 Apr 2014 11:29:34 +0100
Subject: [SciPy-User] defining new distributions with rv_continuous
In-Reply-To: <CAMMTP+DXc_xWJ=A5TBWXw-pMfC_OCuDrP7_JTivvysvusX-R0Q@mail.gmail.com>
References: <534BB1D6.8060000@voo.it>
	<CAMRo0itnV7Di=_CNuLwrV4BN_hSem4HpwCPojigZ6pNeD07FgQ@mail.gmail.com>
	<CAMMTP+DXc_xWJ=A5TBWXw-pMfC_OCuDrP7_JTivvysvusX-R0Q@mail.gmail.com>
Message-ID: <CAMRo0iv18mbha=Sg-53SvwCOfPgFk42FiPp9VD1rmNcoezFkig@mail.gmail.com>

A barebone example of subclassing:

>>> class TNC(stats.rv_continuous):
...    def _pdf(self, x):
...      return stats.norm.pdf(x) * 2.      # cf Josef's email
...
>>> tnc = TNC(name='foo', a=0.)
>>>
>>> tnc.rvs(size=3)
array([ 0.19215378,  2.43396508,  0.58672056])
>>>
>>> tnc.expect(lambda x: 1.)
0.9999999999999998

* Notice there's no need for explicit branching in _pdf, since this is
taken care of in generic pdf once you provide the `a` arg to ctor.
* While this works for small examples, you better be aware that the
generic rvs incurs a *lot* of overhead. You might want to define
explicit _ppf or _rvs.

Evgeni



On Mon, Apr 14, 2014 at 11:18 AM,  <josef.pktd at gmail.com> wrote:
> On Mon, Apr 14, 2014 at 6:13 AM, Evgeni Burovski
> <evgeny.burovskiy at gmail.com> wrote:
>> In this specific case, have a look at truncnorm:
>>
>> http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.stats.truncnorm.html
>>
>>>>> import scipy.stats as stats
>>>>> import numpy as np
>>>>> stats.truncnorm.pdf(0.1, 0, np.inf)
>> 0.79390509495402362
>>>>> stats.truncnorm.pdf(-0.1, 0, np.inf)
>> 0.0
>>
>>
>>
>> On Mon, Apr 14, 2014 at 11:00 AM, Francesco De Gasperin <fdg at voo.it> wrote:
>>> Hi,
>>>
>>> I have some problems in defining a new distribution using rv_continuous.
>>> As an example I tried to create a gaussian defined only for positive
>>> values of the random variable.
>>>
>>> This is the code:
>>>
>>> ###########################################
>>> from scipy.stats import distributions
>>> class gauss_pos(distributions.rv_continuous):
>>>     def _pdf(self, x):
>>>          if x <= 0: return 0
>>>          sigma = 1.
>>>          c = 0.
>>>          return 1/(np.sqrt(2*np.pi)*sigma) * np.exp( -(x-c)**2 /
>>> (2*sigma**2) )
>>>
>>> t_obj = gauss_pos(name="gauss_pos")
>>> print (t_obj.rvs(size=10))
>>> #############################################
>>>
>>> but when I run it I always end up with:
>>> OverflowError: (34, 'Numerical result out of range')
>>>
>>> Then, I removed the "if x <= 0: return 0" line and I tried to define the
>>> "a" and "b" parameters when creating the t_obj. Defining only the "a=0"
>>> produces the same error, while using something as:
>>>
>>> t_obj = gauss_pos(a=0,b=2)
>>>
>>> instead produces a result of this type:
>>> [ 2.          2.          1.5460072   1.09621782  1.62184086  0.42107306
>>>    2.          1.09281742  0.82957403  2.        ]
>>>
>>> which I still do not understand... it seems that "b" is returned every
>>> time the constraint that the random var must be between a and b is
>>> violated...
>
> The specific problem is that your pdf is not normalized correctly to
> integrate to 1.
> If you truncate the normal distribution, then you need to normalize
> for the truncated probability.
>
> The ppf will not be correct, and it is used in to create the rvs in a
> generic way.
>
> As Evgeni pointed out truncnorm, the best way is to look at the source
> code for some distributions and use them as pattern.
>
> There is also halfnorm.
>
> Josef
>
>
>>>
>>> any help is appreciated!
>>> cheers,
>>> Francesco
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From lists at hilboll.de  Mon Apr 14 06:42:15 2014
From: lists at hilboll.de (Andreas Hilboll)
Date: Mon, 14 Apr 2014 12:42:15 +0200
Subject: [SciPy-User] Saving into file.txt
In-Reply-To: <CF71840C.782C%maria-rosaria.antonelli@curie.fr>
References: <CF71840C.782C%maria-rosaria.antonelli@curie.fr>
Message-ID: <534BBB87.9020407@hilboll.de>

On 14.04.2014 12:28, Antonelli Maria Rosaria wrote:
> Hi, 
> 
> I need to save same data arrays into a file.txt.
> I would like to save them as different columns.
> At the moment I can manage to save them as different lines.
> 
> 
> Can you please give me some hints ?
> Best regards, 
> Rosa
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 

Hi,

take a look at numpy.savetxt()

-- Andreas.


From guziy.sasha at gmail.com  Mon Apr 14 06:42:45 2014
From: guziy.sasha at gmail.com (Oleksandr Huziy)
Date: Mon, 14 Apr 2014 06:42:45 -0400
Subject: [SciPy-User] Saving into file.txt
In-Reply-To: <CF71840C.782C%maria-rosaria.antonelli@curie.fr>
References: <CF71840C.782C%maria-rosaria.antonelli@curie.fr>
Message-ID: <CAN3p1sWwXxK-zG==vq6USpxfsGhhgnQ19oPnKa=_=bCS3gujnA@mail.gmail.com>

Transposing the array does the trick:

In [1]: import numpy as np

In [2]: x = np.arange(10)

In [3]: y = np.random.randn(10)

In [4]: y
Out[4]:
array([ 0.77039544,  1.27781447,  0.96214621, -0.04514849,  0.12023634,
        0.33786337, -0.88450054, -0.67355859,  2.11181118,  0.96484793])

In [5]: x
Out[5]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

In [9]: np.savetxt("test.txt", [x, y])

In [10]: cat test.txt
0.000000000000000000e+00 1.000000000000000000e+00 2.000000000000000000e+00
3.000000000000000000e+00 4.000000000000000000e+00 5.000000000000000000e+00
6.000000000000000000e+00 7.000000000000000000e+00 8.000000000000000000e+00
9.000000000000000000e+00
7.703954403007949869e-01 1.277814469462716263e+00 9.621462065528501029e-01
-4.514849461497021099e-02 1.202363398274446937e-01 3.378633716328510173e-01
-8.845005407517207985e-01 -6.735585891743005593e-01
2.111811179340500555e+00 9.648479328801105037e-01

In [11]: np.savetxt("test.txt", np.asarray([x, y]).T)

In [12]: cat test.txt
0.000000000000000000e+00 7.703954403007949869e-01
1.000000000000000000e+00 1.277814469462716263e+00
2.000000000000000000e+00 9.621462065528501029e-01
3.000000000000000000e+00 -4.514849461497021099e-02
4.000000000000000000e+00 1.202363398274446937e-01
5.000000000000000000e+00 3.378633716328510173e-01
6.000000000000000000e+00 -8.845005407517207985e-01
7.000000000000000000e+00 -6.735585891743005593e-01
8.000000000000000000e+00 2.111811179340500555e+00
9.000000000000000000e+00 9.648479328801105037e-01

Cheers



2014-04-14 6:28 GMT-04:00 Antonelli Maria Rosaria <
maria-rosaria.antonelli at curie.fr>:

> Hi,
>
> I need to save same data arrays into a file.txt.
> I would like to save them as different columns.
> At the moment I can manage to save them as different lines.
>
>
> Can you please give me some hints ?
> Best regards,
> Rosa
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Sasha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140414/574ce2f8/attachment.html>

From fdg at voo.it  Mon Apr 14 07:15:05 2014
From: fdg at voo.it (Francesco De Gasperin)
Date: Mon, 14 Apr 2014 13:15:05 +0200
Subject: [SciPy-User] defining new distributions with, rv_continuous
Message-ID: <534BC339.8010105@voo.it>

Many thanks for the quick help. I see that the overhead is quite large 
but I don't look for high performances now, I need to generate only 1000 
numbers. For interested people, I attach here a more flexible code which 
should work as it numerically normalizes the pdf given "a" and "b".

#####################################################
from scipy.stats import distributions
from scipy import integrate

class test(distributions.rv_continuous):

     def __init__(self,a=-np.inf,b=np.inf,name=''):
         super().__init__(a=a,b=b,name=name)
         self.integral = integrate.quad(self.f, self.a, self.b)[0]

     def f(self, x):
	# in this case is a gaussian, but it could be a more complex pdf
         sigma = 1.
         c = 0.
         return np.exp( -(x-c)**2 / (2*sigma**2) )

     def _pdf(self, x):
         return self.f(x)/self.integral

t_obj = test(a=0,name="test")
######################################################

thanks,
Francesco


From bouloumag at gmail.com  Mon Apr 14 08:36:52 2014
From: bouloumag at gmail.com (Darcoux Christine)
Date: Mon, 14 Apr 2014 08:36:52 -0400
Subject: [SciPy-User] create sparse matrix from diagonal arrays
In-Reply-To: <CAEWZG1boLaczJr959Uunit2xQtp7jBj_xuRB+GRyd-CzbhegVw@mail.gmail.com>
References: <CAD6uFgmnc8gXOPF8hs3na=hiKRgAyY6ajYd_k38vpkDGxniZHA@mail.gmail.com>
	<CAB0gBH0XU43yVS0-X8ESFM__w6VpFYM1iFwpa_uRhtpGxNoQRw@mail.gmail.com>
	<CAEWZG1boLaczJr959Uunit2xQtp7jBj_xuRB+GRyd-CzbhegVw@mail.gmail.com>
Message-ID: <CAD6uFgkkejGXbK+a+6CUGTb0YPU1MN1sG-O+Kr9Ja8WWah9eVg@mail.gmail.com>

Le vendredi 11 avril 2014, Tony Stillfjord <tony at maths.lth.se> a ?crit :

> On Fri, Apr 11, 2014 at 4:53 AM, Geordie McBain <gdmcbain at freeshell.org<javascript:_e(%7B%7D,'cvml','gdmcbain at freeshell.org');>
> > wrote:
>
>> 2014-04-11 12:00 GMT+10:00 Darcoux Christine <bouloumag at gmail.com<javascript:_e(%7B%7D,'cvml','bouloumag at gmail.com');>
>> >:
>> > I would like to solve a PDE using an implicit finite volume
>> discretization.
>> > Since the problem is 3D, the matrix should be septadiagonal. I have
>> computed
>> > the 7 diagonals (for top, bottom, east, west, north south and central
>> point)
>> > in 7 different arrays.
>> >
>> > Now, I would like to create a sparse matrix object from these arrays,
>> but I
>> > feel a little lost with the documentation of scipy.sparse.spdiags.
>> >
>> > Help would be greatly appreciated.
>>
>> Put the diagonals, padded to the length of the main diagonal
>> (postpadded for the subdiagonals and prepadded for the superdiagonals,
>> it doesn't matter what with) as the rows of an array, or elements of a
>> list or other iterable.
>>
>> For example, to generate the classical tridiagonal one-dimensional
>> Laplacian:
>>
>> >>> spdiags([[1, 1, np.nan], [-2, -2, -2], [np.nan, 1, 1]], [-1, 0, 1],
>> 3, 3).toarray()
>> array([[-2.,  1.,  0.],
>>        [ 1., -2.,  1.],
>>        [ 0.,  1., -2.]])
>>
>
>
> Or if you don't want to pad your diagonals, use the scipy.sparse.diags
> function instead. It was included in 0.11.0. Then you would have something
> like
>
> diags([top, bottom, east, west, north, south, central], [i0, i1, i2, i3,
> i4, i5, i6])
>
> where the i's are the corresponding diagonal offsets. For example, i6
> would be 0.
>
> Simple example:
>
> >>> e0 = array([1, 2, 3])
> >>> e1 = array([4, 5])
> >>> e2 = array([6])
> >>> diags([e0, e1, e2], [0, 1, -2]).toarray()
> array([[ 1.,  4.,  0.],
>        [ 0.,  2.,  5.],
>        [ 6.,  0.,  3.]])
>
> Tony
>

I have found the stencil_grid function in pyamg, which is exactly what I
need.

BTW, it would be nice if scipy could include a similar function.

Christine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140414/d7e78d40/attachment.html>

From anthony.j.mannucci at jpl.nasa.gov  Mon Apr 14 12:04:29 2014
From: anthony.j.mannucci at jpl.nasa.gov (Mannucci, Anthony J (335G))
Date: Mon, 14 Apr 2014 16:04:29 +0000
Subject: [SciPy-User] Can create, but not access,
 netCDF variables  with no dimension
Message-ID: <CF71551E.83B33%anthony.j.mannucci@jpl.nasa.gov>

Thanks for your help using getValue. That should help with the example I provided. However, the variable does not show up in the keys(). Here is an ncdump excerpt.

More information: I'm using scipy 0.10. Perhaps this issue has been resolved in a later version?

-Tony

netcdf apr2011storm_s008D110429_0100_110501_0000 {
dimensions:
        time = UNLIMITED ; // (48 currently)
        lon = 72 ;
        lat = 36 ;
        lev = 29 ;
        ilev = 29 ;
        mlon = 81 ;
        mlat = 97 ;
        mlev = 32 ;
        imlev = 32 ;
        mtimedim = 3 ;
        latlon = 2 ;
        dtidedim = 2 ;
        sdtidedim = 10 ;
        datelen = 24 ;
        filelen = 1024 ;
variables:
        double time(time) ;
                time:long_name = "time" ;
                time:units = "minutes since 2011-3-21 0:0:0" ;
                time:initial_year = 2011 ;
                time:initial_day = 80 ;
                time:initial_mtime = 80, 0, 0 ;
        double lon(lon) ;
                lon:long_name = "geographic longitude (-west, +east)" ;
                lon:units = "degrees_east" ;
        double lat(lat) ;
                lat:long_name = "geographic latitude (-south, +north)" ;
                lat:units = "degrees_north" ;
        double lev(lev) ;
                lev:long_name = "midpoint levels" ;
                lev:short_name = "ln(p0/p)" ;
                lev:units = "" ;
                lev:positive = "up" ;
                lev:standard_name = "atmosphere_ln_pressure_coordinate" ;
                lev:formula_terms = "p0: p0 lev: lev" ;
                lev:formula = "p(k) = p0 * exp(-lev(k))" ;
        double ilev(ilev) ;
                ilev:long_name = "interface levels" ;
                ilev:short_name = "ln(p0/p)" ;
                ilev:units = "" ;
                ilev:positive = "up" ;
                ilev:standard_name = "atmosphere_ln_pressure_coordinate" ;
                ilev:formula_terms = "p0: p0 lev: ilev" ;
                ilev:formula = "p(k) = p0 * exp(-ilev(k))" ;

<<< More stuff. The following don't show up>>>
        double p0 ;
                p0:long_name = "Reference pressure" ;
                p0:units = "millibars" ;
        double p0_model ;
                p0_model:long_name = "Reference pressure (as used by the model)" ;
                p0_model:units = "microbars" ;


--
Tony Mannucci
Supervisor, Ionospheric and Atmospheric Remote Sensing Group
 Mail-Stop 138-308,                     Tel > (818) 354-1699
 Jet Propulsion Laboratory,              Fax > (818) 393-5115
 California Institute of Technology,     Email > Tony.Mannucci at jpl.nasa.gov
 4800 Oak Grove Drive,                   http://scienceandtechnology.jpl.nasa.gov/people/a_mannucci/
 Pasadena, CA 91109

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140414/7897abd8/attachment.html>

From maria-rosaria.antonelli at curie.fr  Mon Apr 14 12:13:51 2014
From: maria-rosaria.antonelli at curie.fr (Antonelli Maria Rosaria)
Date: Mon, 14 Apr 2014 16:13:51 +0000
Subject: [SciPy-User] Saving into file.txt
In-Reply-To: <CAN3p1sWwXxK-zG==vq6USpxfsGhhgnQ19oPnKa=_=bCS3gujnA@mail.gmail.com>
References: <CF71840C.782C%maria-rosaria.antonelli@curie.fr>
	<CAN3p1sWwXxK-zG==vq6USpxfsGhhgnQ19oPnKa=_=bCS3gujnA@mail.gmail.com>
Message-ID: <CF71D5B1.783C%maria-rosaria.antonelli@curie.fr>

Thank you !
This is what I needed.
Have a nice evening,
Rosa

From: Oleksandr Huziy <guziy.sasha at gmail.com<mailto:guziy.sasha at gmail.com>>
Reply-To: SciPy Users List <scipy-user at scipy.org<mailto:scipy-user at scipy.org>>
Date: Monday, April 14, 2014 12:42 PM
To: SciPy Users List <scipy-user at scipy.org<mailto:scipy-user at scipy.org>>
Subject: Re: [SciPy-User] Saving into file.txt

Transposing the array does the trick:

In [1]: import numpy as np

In [2]: x = np.arange(10)

In [3]: y = np.random.randn(10)

In [4]: y
Out[4]:
array([ 0.77039544,  1.27781447,  0.96214621, -0.04514849,  0.12023634,
        0.33786337, -0.88450054, -0.67355859,  2.11181118,  0.96484793])

In [5]: x
Out[5]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

In [9]: np.savetxt("test.txt", [x, y])

In [10]: cat test.txt
0.000000000000000000e+00 1.000000000000000000e+00 2.000000000000000000e+00 3.000000000000000000e+00 4.000000000000000000e+00 5.000000000000000000e+00 6.000000000000000000e+00 7.000000000000000000e+00 8.000000000000000000e+00 9.000000000000000000e+00
7.703954403007949869e-01 1.277814469462716263e+00 9.621462065528501029e-01 -4.514849461497021099e-02 1.202363398274446937e-01 3.378633716328510173e-01 -8.845005407517207985e-01 -6.735585891743005593e-01 2.111811179340500555e+00 9.648479328801105037e-01

In [11]: np.savetxt("test.txt", np.asarray([x, y]).T)

In [12]: cat test.txt
0.000000000000000000e+00 7.703954403007949869e-01
1.000000000000000000e+00 1.277814469462716263e+00
2.000000000000000000e+00 9.621462065528501029e-01
3.000000000000000000e+00 -4.514849461497021099e-02
4.000000000000000000e+00 1.202363398274446937e-01
5.000000000000000000e+00 3.378633716328510173e-01
6.000000000000000000e+00 -8.845005407517207985e-01
7.000000000000000000e+00 -6.735585891743005593e-01
8.000000000000000000e+00 2.111811179340500555e+00
9.000000000000000000e+00 9.648479328801105037e-01

Cheers



2014-04-14 6:28 GMT-04:00 Antonelli Maria Rosaria <maria-rosaria.antonelli at curie.fr<mailto:maria-rosaria.antonelli at curie.fr>>:
Hi,

I need to save same data arrays into a file.txt.
I would like to save them as different columns.
At the moment I can manage to save them as different lines.


Can you please give me some hints ?
Best regards,
Rosa

_______________________________________________
SciPy-User mailing list
SciPy-User at scipy.org<mailto:SciPy-User at scipy.org>
http://mail.scipy.org/mailman/listinfo/scipy-user



--
Sasha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140414/758ea080/attachment.html>

From guziy.sasha at gmail.com  Mon Apr 14 12:24:42 2014
From: guziy.sasha at gmail.com (Oleksandr Huziy)
Date: Mon, 14 Apr 2014 12:24:42 -0400
Subject: [SciPy-User] Can create, but not access,
	netCDF variables with no dimension
In-Reply-To: <CF71551E.83B33%anthony.j.mannucci@jpl.nasa.gov>
References: <CF71551E.83B33%anthony.j.mannucci@jpl.nasa.gov>
Message-ID: <CAN3p1sVxtb_y78nN-UjTdyj1pyWb+0G3V2Lo-Em628Agts+2fw@mail.gmail.com>

Do you have nco installed?
You could select onlyt p0 into a separate file, then I could look into it?

ncks -v p0 <your complete file> only_p0.nc

I think it is also possible to do with cdo.

In any case it is better to upgrade scipy, though I am not sure if netcdf
has changed since 1.10.

Cheers



2014-04-14 12:04 GMT-04:00 Mannucci, Anthony J (335G) <
anthony.j.mannucci at jpl.nasa.gov>:

>   Thanks for your help using getValue. That should help with the example
> I provided. However, the variable does not show up in the keys(). Here is
> an ncdump excerpt.
>
>  More information: I'm using scipy 0.10. Perhaps this issue has been
> resolved in a later version?
>
>  -Tony
>
>  netcdf apr2011storm_s008D110429_0100_110501_0000 {
> dimensions:
>         time = UNLIMITED ; // (48 currently)
>         lon = 72 ;
>         lat = 36 ;
>         lev = 29 ;
>         ilev = 29 ;
>         mlon = 81 ;
>         mlat = 97 ;
>         mlev = 32 ;
>         imlev = 32 ;
>         mtimedim = 3 ;
>         latlon = 2 ;
>         dtidedim = 2 ;
>         sdtidedim = 10 ;
>         datelen = 24 ;
>         filelen = 1024 ;
> variables:
>         double time(time) ;
>                 time:long_name = "time" ;
>                 time:units = "minutes since 2011-3-21 0:0:0" ;
>                 time:initial_year = 2011 ;
>                 time:initial_day = 80 ;
>                 time:initial_mtime = 80, 0, 0 ;
>         double lon(lon) ;
>                 lon:long_name = "geographic longitude (-west, +east)" ;
>                 lon:units = "degrees_east" ;
>         double lat(lat) ;
>                 lat:long_name = "geographic latitude (-south, +north)" ;
>                 lat:units = "degrees_north" ;
>         double lev(lev) ;
>                 lev:long_name = "midpoint levels" ;
>                 lev:short_name = "ln(p0/p)" ;
>                 lev:units = "" ;
>                 lev:positive = "up" ;
>                 lev:standard_name = "atmosphere_ln_pressure_coordinate" ;
>                 lev:formula_terms = "p0: p0 lev: lev" ;
>                 lev:formula = "p(k) = p0 * exp(-lev(k))" ;
>         double ilev(ilev) ;
>                 ilev:long_name = "interface levels" ;
>                 ilev:short_name = "ln(p0/p)" ;
>                 ilev:units = "" ;
>                 ilev:positive = "up" ;
>                 ilev:standard_name = "atmosphere_ln_pressure_coordinate" ;
>                 ilev:formula_terms = "p0: p0 lev: ilev" ;
>                 ilev:formula = "p(k) = p0 * exp(-ilev(k))" ;
>
>  <<< More stuff. The following don't show up>>>
>          double p0 ;
>                 p0:long_name = "Reference pressure" ;
>                 p0:units = "millibars" ;
>         double p0_model ;
>                 p0_model:long_name = "Reference pressure (as used by the
> model)" ;
>                 p0_model:units = "microbars" ;
>
>
>   --
>  Tony Mannucci
> Supervisor, Ionospheric and Atmospheric Remote Sensing Group
>  Mail-Stop 138-308,                     Tel > (818) 354-1699
>  Jet Propulsion Laboratory,              Fax > (818) 393-5115
>  California Institute of Technology,     Email >
> Tony.Mannucci at jpl.nasa.gov
>  4800 Oak Grove Drive,
> http://scienceandtechnology.jpl.nasa.gov/people/a_mannucci/
>  Pasadena, CA 91109
>
>


-- 
Sasha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140414/08db4db6/attachment.html>

From john.krasting at noaa.gov  Tue Apr 15 14:52:29 2014
From: john.krasting at noaa.gov (John Krasting - NOAA Federal)
Date: Tue, 15 Apr 2014 14:52:29 -0400
Subject: [SciPy-User] scipy.signal vs Matlab: filtfilt and reflection
Message-ID: <CAEDGu3dg9t4LOhHJO5_dc5F-44p14BgOav09xkd9HPkz7CNMUQ@mail.gmail.com>

Hi Scipy Users -

Am I correct in reading that filtfilt in scipy.signal (v. 13.0) does not
extrapolate data at the beginning and the end of a time series when using
the filtfilt function?

Based on the Matlab code (
http://chmielowski.eu/POLITECHNIKA/Dydaktyka/AUTOMATYKA/AutoLab/Matlab/TOOLBOX/SIGNAL/FILTFILT.M),
I see that there is a block of code that does the reflection:

% Extrapolate beginning and end of data sequence using a "reflection
% method".  Slopes of original and extrapolated sequences match at
% the end points.
% This reduces end effects.
    y = [2*x(1)-x((nfact+1):-1:2);x;2*x(len)-x((len-1):-1:len-nfact)];

I did not see this in the source for scipy's filtfilt function.   Are there
any plans to add this as a possible option to the function?

Thanks,
John K.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140415/89ea1b30/attachment.html>

From warren.weckesser at gmail.com  Tue Apr 15 16:38:53 2014
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Tue, 15 Apr 2014 16:38:53 -0400
Subject: [SciPy-User] scipy.signal vs Matlab: filtfilt and reflection
In-Reply-To: <CAEDGu3dg9t4LOhHJO5_dc5F-44p14BgOav09xkd9HPkz7CNMUQ@mail.gmail.com>
References: <CAEDGu3dg9t4LOhHJO5_dc5F-44p14BgOav09xkd9HPkz7CNMUQ@mail.gmail.com>
Message-ID: <CAGzF1ucPX7jhd30Wzqam4D3UVKWj064R779A2-UzAQT17etzOA@mail.gmail.com>

On 4/15/14, John Krasting - NOAA Federal <john.krasting at noaa.gov> wrote:
> Hi Scipy Users -
>
> Am I correct in reading that filtfilt in scipy.signal (v. 13.0) does not
> extrapolate data at the beginning and the end of a time series when using
> the filtfilt function?
>
> Based on the Matlab code (
> http://chmielowski.eu/POLITECHNIKA/Dydaktyka/AUTOMATYKA/AutoLab/Matlab/TOOLBOX/SIGNAL/FILTFILT.M),
> I see that there is a block of code that does the reflection:
>
> % Extrapolate beginning and end of data sequence using a "reflection
> % method".  Slopes of original and extrapolated sequences match at
> % the end points.
> % This reduces end effects.
>     y = [2*x(1)-x((nfact+1):-1:2);x;2*x(len)-x((len-1):-1:len-nfact)];
>
> I did not see this in the source for scipy's filtfilt function.   Are there
> any plans to add this as a possible option to the function?
>
> Thanks,
> John K.
>


That matlab code makes an odd extension of the data.  E.g.


octave:35> len = 8
len =  8
octave:36> x = [1:len]';
octave:37> x'
ans =

   1   2   3   4   5   6   7   8

octave:38> nfact = 3;
octave:39> y = [2*x(1)-x((nfact+1):-1:2);x;2*x(len)-x((len-1):-1:len-nfact)];
octave:40> y'
ans =

   -2   -1    0    1    2    3    4    5    6    7    8    9   10   11


That type of extension is the default behavior of filtfilt
(http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.filtfilt.html).
 You can change the type of the extension and the length of the
padding with the padtype and padlen arguments.

Warren


From john.krasting at noaa.gov  Tue Apr 15 18:33:41 2014
From: john.krasting at noaa.gov (John Krasting - NOAA Federal)
Date: Tue, 15 Apr 2014 18:33:41 -0400
Subject: [SciPy-User] scipy.signal vs Matlab: filtfilt and reflection
In-Reply-To: <CAGzF1ucPX7jhd30Wzqam4D3UVKWj064R779A2-UzAQT17etzOA@mail.gmail.com>
References: <CAEDGu3dg9t4LOhHJO5_dc5F-44p14BgOav09xkd9HPkz7CNMUQ@mail.gmail.com>
	<CAGzF1ucPX7jhd30Wzqam4D3UVKWj064R779A2-UzAQT17etzOA@mail.gmail.com>
Message-ID: <CAEDGu3dFfrYC4zzWoZ4cCugqTTR8=W8kW47pmZ5qPeRppww5ig@mail.gmail.com>

Hi Warren -

Thanks for your reply.  Sorry for being mistaken, I now see that filtfilt
does the odd padding by default.

I am still seeing differences between Matlab and scipy at the second
decimal place.  This concerns me.  Has anybody else seen this?

I took a sample 15 numbers (pulled from a random dist.), built a
Butterworth filter, and ran filtfilt on the data using Matlab and scipy.
 The results are in the same ballpark, but I am still seeing differences
between Matlab and scipy at the second decimal place.  This concerns me
since I expected to see differences at the lower order bits.  Has anybody
else seen this or know why this is the case?

I've pasted my 15 random numbers and output from both Matlab and scipy
below.

Thanks,
John K.


PS - I tried taking the Butterworth filter from scip


---- scipy v0.13.3, numpy v.1.6.2 ----

data = numpy.array([0.401808033751942, 0.075966691690842,
0.239916153553658, 0.123318934835166, 0.183907788282417, 0.239952525664903,
0.417267069084370, 0.049654430325742, 0.902716109915281, 0.944787189721646,
0.490864092468080, 0.489252638400019, 0.337719409821377, 0.900053846417662,
0.369246781120215])
[
b,a] = scipy.signal.butter(3,0.2)

>>> b
array([ 0.01809893,  0.0542968 ,  0.0542968 ,  0.01809893])

>>> b[0]
0.018098933007514431
>>> b[1]
0.054296799022543286
>>> b[2]
0.054296799022543286
>>> b[3]
0.018098933007514431

>>> a
array([ 1.        , -1.76004188,  1.18289326, -0.27805992])

>>> a[0]
1.0
>>> a[1]
-1.7600418803431686
>>> a[2]
1.1828932620378303
>>> a[3]
-0.27805991763454624

scipy.signal.filtfilt(b,a,data)

array([ 0.40421981,  0.29292145,  0.20773242,  0.1682317 ,  0.18250909,
        0.24674659,  0.34676359,  0.46045534,  0.56190086,  0.62844665,
        0.64819638,  0.62192427,  0.55831894,  0.46881537,  0.36653733])


---- MATLAB v.7.9.0.529 ----

data = [0.401808033751942, 0.075966691690842, 0.239916153553658,
0.123318934835166, 0.183907788282417, 0.239952525664903, 0.417267069084370,
0.049654430325742, 0.902716109915281, 0.944787189721646, 0.490864092468080,
0.489252638400019, 0.337719409821377, 0.900053846417662, 0.369246781120215]

>> [b,a] = butter(3,0.2)

b =

   0.018098933007514   0.054296799022543   0.054296799022543
0.018098933007514


a =

   1.000000000000000  -1.760041880343169   1.182893262037831
 -0.278059917634546

>> filtfilt(b,a,data)

ans =

  Columns 1 through 5

   0.397839293822166   0.286273839374148   0.202969550095165
0.166087236796221   0.182700682986968

  Columns 6 through 10

   0.248436130460523   0.348904396853114   0.461998356964929
0.561930803937072   0.626352300416756

  Columns 11 through 15

   0.643941843065195   0.616392843959916   0.553574502943643
0.468014071436841   0.373188133985282



On Tue, Apr 15, 2014 at 4:38 PM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

> On 4/15/14, John Krasting - NOAA Federal <john.krasting at noaa.gov> wrote:
> > Hi Scipy Users -
> >
> > Am I correct in reading that filtfilt in scipy.signal (v. 13.0) does not
> > extrapolate data at the beginning and the end of a time series when using
> > the filtfilt function?
> >
> > Based on the Matlab code (
> >
> http://chmielowski.eu/POLITECHNIKA/Dydaktyka/AUTOMATYKA/AutoLab/Matlab/TOOLBOX/SIGNAL/FILTFILT.M
> ),
> > I see that there is a block of code that does the reflection:
> >
> > % Extrapolate beginning and end of data sequence using a "reflection
> > % method".  Slopes of original and extrapolated sequences match at
> > % the end points.
> > % This reduces end effects.
> >     y = [2*x(1)-x((nfact+1):-1:2);x;2*x(len)-x((len-1):-1:len-nfact)];
> >
> > I did not see this in the source for scipy's filtfilt function.   Are
> there
> > any plans to add this as a possible option to the function?
> >
> > Thanks,
> > John K.
> >
>
>
> That matlab code makes an odd extension of the data.  E.g.
>
>
> octave:35> len = 8
> len =  8
> octave:36> x = [1:len]';
> octave:37> x'
> ans =
>
>    1   2   3   4   5   6   7   8
>
> octave:38> nfact = 3;
> octave:39> y =
> [2*x(1)-x((nfact+1):-1:2);x;2*x(len)-x((len-1):-1:len-nfact)];
> octave:40> y'
> ans =
>
>    -2   -1    0    1    2    3    4    5    6    7    8    9   10   11
>
>
> That type of extension is the default behavior of filtfilt
> (
> http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.filtfilt.html
> ).
>  You can change the type of the extension and the length of the
> padding with the padtype and padlen arguments.
>
> Warren
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140415/c0d5fb0e/attachment.html>

From john.krasting at noaa.gov  Tue Apr 15 18:36:08 2014
From: john.krasting at noaa.gov (John Krasting - NOAA Federal)
Date: Tue, 15 Apr 2014 18:36:08 -0400
Subject: [SciPy-User] scipy.signal vs Matlab: filtfilt and reflection
In-Reply-To: <CAEDGu3dFfrYC4zzWoZ4cCugqTTR8=W8kW47pmZ5qPeRppww5ig@mail.gmail.com>
References: <CAEDGu3dg9t4LOhHJO5_dc5F-44p14BgOav09xkd9HPkz7CNMUQ@mail.gmail.com>
	<CAGzF1ucPX7jhd30Wzqam4D3UVKWj064R779A2-UzAQT17etzOA@mail.gmail.com>
	<CAEDGu3dFfrYC4zzWoZ4cCugqTTR8=W8kW47pmZ5qPeRppww5ig@mail.gmail.com>
Message-ID: <CAEDGu3f4eEKB0Af+ErXX+E7amtR1X_uRbXJD3v3sdbUd0us0Fg@mail.gmail.com>

The postscript was cutoff ...

PS - I tried taking the Butterworth filter from scipy and Matlab and
interchanged them to see if the extra decimal places mattered ... it did
not make a difference for me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140415/4e1749a1/attachment.html>

From warren.weckesser at gmail.com  Tue Apr 15 19:25:27 2014
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Tue, 15 Apr 2014 19:25:27 -0400
Subject: [SciPy-User] scipy.signal vs Matlab: filtfilt and reflection
In-Reply-To: <CAEDGu3dFfrYC4zzWoZ4cCugqTTR8=W8kW47pmZ5qPeRppww5ig@mail.gmail.com>
References: <CAEDGu3dg9t4LOhHJO5_dc5F-44p14BgOav09xkd9HPkz7CNMUQ@mail.gmail.com>
	<CAGzF1ucPX7jhd30Wzqam4D3UVKWj064R779A2-UzAQT17etzOA@mail.gmail.com>
	<CAEDGu3dFfrYC4zzWoZ4cCugqTTR8=W8kW47pmZ5qPeRppww5ig@mail.gmail.com>
Message-ID: <CAGzF1ud+vNG5hGPL9bKBy-MpQK2muFXmaM_PxBi-=DNY-V5sBg@mail.gmail.com>

On Tue, Apr 15, 2014 at 6:33 PM, John Krasting - NOAA Federal <
john.krasting at noaa.gov> wrote:

> Hi Warren -
>
> Thanks for your reply.  Sorry for being mistaken, I now see that filtfilt
> does the odd padding by default.
>
> I am still seeing differences between Matlab and scipy at the second
> decimal place.  This concerns me.  Has anybody else seen this?
>
> I took a sample 15 numbers (pulled from a random dist.), built a
> Butterworth filter, and ran filtfilt on the data using Matlab and scipy.
>  The results are in the same ballpark, but I am still seeing differences
> between Matlab and scipy at the second decimal place.  This concerns me
> since I expected to see differences at the lower order bits.  Has anybody
> else seen this or know why this is the case?
>
> I've pasted my 15 random numbers and output from both Matlab and scipy
> below.
>
> Thanks,
> John K.
>
>
> PS - I tried taking the Butterworth filter from scip
>
>
> ?? scipy v0.13.3, numpy v.1.6.2 ??
>
> data = numpy.array([0.401808033751942, 0.075966691690842,
> 0.239916153553658, 0.123318934835166, 0.183907788282417, 0.239952525664903,
> 0.417267069084370, 0.049654430325742, 0.902716109915281, 0.944787189721646,
> 0.490864092468080, 0.489252638400019, 0.337719409821377, 0.900053846417662,
> 0.369246781120215])
> [
> b,a] = scipy.signal.butter(3,0.2)
>
> >>> b
> array([ 0.01809893,  0.0542968 ,  0.0542968 ,  0.01809893])
>
> >>> b[0]
> 0.018098933007514431
> >>> b[1]
> 0.054296799022543286
> >>> b[2]
> 0.054296799022543286
> >>> b[3]
> 0.018098933007514431
>
> >>> a
> array([ 1.        , -1.76004188,  1.18289326, -0.27805992])
>
> >>> a[0]
> 1.0
> >>> a[1]
> -1.7600418803431686
> >>> a[2]
> 1.1828932620378303
> >>> a[3]
> -0.27805991763454624
>
> scipy.signal.filtfilt(b,a,data)
>
> array([ 0.40421981,  0.29292145,  0.20773242,  0.1682317 ,  0.18250909,
>         0.24674659,  0.34676359,  0.46045534,  0.56190086,  0.62844665,
>         0.64819638,  0.62192427,  0.55831894,  0.46881537,  0.36653733])
>
>
> ?? MATLAB v.7.9.0.529 ??
>
> data = [0.401808033751942, 0.075966691690842, 0.239916153553658,
> 0.123318934835166, 0.183907788282417, 0.239952525664903, 0.417267069084370,
> 0.049654430325742, 0.902716109915281, 0.944787189721646, 0.490864092468080,
> 0.489252638400019, 0.337719409821377, 0.900053846417662, 0.369246781120215]
>
> >> [b,a] = butter(3,0.2)
>
> b =
>
>    0.018098933007514   0.054296799022543   0.054296799022543
> 0.018098933007514
>
>
> a =
>
>    1.000000000000000  -1.760041880343169   1.182893262037831
>  -0.278059917634546
>
> >> filtfilt(b,a,data)
>
> ans =
>
>   Columns 1 through 5
>
>    0.397839293822166   0.286273839374148   0.202969550095165
> 0.166087236796221   0.182700682986968
>
>   Columns 6 through 10
>
>    0.248436130460523   0.348904396853114   0.461998356964929
> 0.561930803937072   0.626352300416756
>
>   Columns 11 through 15
>
>    0.643941843065195   0.616392843959916   0.553574502943643
> 0.468014071436841   0.373188133985282
>
>
>

John,

The difference is in the default padding length.  In matlab's filtfilt, it
is 3*(max(len(a), len(b)) - 1), and in scipy's filtfilt, it is
3*max(len(a), len(b)).  If you set padlen to the same value used in
matlab, the results match:

 In [47]: filtfilt(b, a, data, padlen=9)
Out[47]:
array([ 0.39783929,  0.28627384,  0.20296955,  0.16608724,  0.18270068,
        0.24843613,  0.3489044 ,  0.46199836,  0.5619308 ,  0.6263523 ,
        0.64394184,  0.61639284,  0.5535745 ,  0.46801407,  0.37318813])


Warren



> On Tue, Apr 15, 2014 at 4:38 PM, Warren Weckesser <
> warren.weckesser at gmail.com> wrote:
>
>> On 4/15/14, John Krasting - NOAA Federal <john.krasting at noaa.gov> wrote:
>> > Hi Scipy Users -
>> >
>> > Am I correct in reading that filtfilt in scipy.signal (v. 13.0) does not
>> > extrapolate data at the beginning and the end of a time series when
>> using
>> > the filtfilt function?
>> >
>> > Based on the Matlab code (
>> >
>> http://chmielowski.eu/POLITECHNIKA/Dydaktyka/AUTOMATYKA/AutoLab/Matlab/TOOLBOX/SIGNAL/FILTFILT.M
>> ),
>> > I see that there is a block of code that does the reflection:
>> >
>> > % Extrapolate beginning and end of data sequence using a "reflection
>> > % method".  Slopes of original and extrapolated sequences match at
>> > % the end points.
>> > % This reduces end effects.
>> >     y = [2*x(1)-x((nfact+1):-1:2);x;2*x(len)-x((len-1):-1:len-nfact)];
>> >
>> > I did not see this in the source for scipy's filtfilt function.   Are
>> there
>> > any plans to add this as a possible option to the function?
>> >
>> > Thanks,
>> > John K.
>> >
>>
>>
>> That matlab code makes an odd extension of the data.  E.g.
>>
>>
>> octave:35> len = 8
>> len =  8
>> octave:36> x = [1:len]';
>> octave:37> x'
>> ans =
>>
>>    1   2   3   4   5   6   7   8
>>
>> octave:38> nfact = 3;
>> octave:39> y =
>> [2*x(1)-x((nfact+1):-1:2);x;2*x(len)-x((len-1):-1:len-nfact)];
>> octave:40> y'
>> ans =
>>
>>    -2   -1    0    1    2    3    4    5    6    7    8    9   10   11
>>
>>
>> That type of extension is the default behavior of filtfilt
>> (
>> http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.filtfilt.html
>> ).
>>  You can change the type of the extension and the length of the
>> padding with the padtype and padlen arguments.
>>
>> Warren
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140415/8ba2bca0/attachment.html>

From john.krasting at noaa.gov  Tue Apr 15 19:34:11 2014
From: john.krasting at noaa.gov (John Krasting - NOAA Federal)
Date: Tue, 15 Apr 2014 19:34:11 -0400
Subject: [SciPy-User] scipy.signal vs Matlab: filtfilt and reflection
In-Reply-To: <CAGzF1ud+vNG5hGPL9bKBy-MpQK2muFXmaM_PxBi-=DNY-V5sBg@mail.gmail.com>
References: <CAEDGu3dg9t4LOhHJO5_dc5F-44p14BgOav09xkd9HPkz7CNMUQ@mail.gmail.com>
	<CAGzF1ucPX7jhd30Wzqam4D3UVKWj064R779A2-UzAQT17etzOA@mail.gmail.com>
	<CAEDGu3dFfrYC4zzWoZ4cCugqTTR8=W8kW47pmZ5qPeRppww5ig@mail.gmail.com>
	<CAGzF1ud+vNG5hGPL9bKBy-MpQK2muFXmaM_PxBi-=DNY-V5sBg@mail.gmail.com>
Message-ID: <CAEDGu3e0UiuqU26FuMQbtDM1c9VAN9-Co2xm+Li+pvvWjax6mg@mail.gmail.com>

Hi Warren -

Thanks for your reply.  Yes, when I set padlen=9 in my case I am able to
reproduce Matlab's results.   Thanks for helping me see the difference!

-John K.

--------------------------------
Dr. John Krasting
Physical Scientist (NOAA Federal)
NOAA/Geophysical Fluid Dynamics Laboratory
Biogeochemistry, Ecosystems, and Climate Group
Princeton University Forrestal Campus
201 Forrestal Road
Princeton, NJ 08540

P. (609) 452-5359
F. (609) 987-5063


On Tue, Apr 15, 2014 at 7:25 PM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

>
>
> On Tue, Apr 15, 2014 at 6:33 PM, John Krasting - NOAA Federal <
> john.krasting at noaa.gov> wrote:
>
>> Hi Warren -
>>
>> Thanks for your reply.  Sorry for being mistaken, I now see that filtfilt
>> does the odd padding by default.
>>
>> I am still seeing differences between Matlab and scipy at the second
>> decimal place.  This concerns me.  Has anybody else seen this?
>>
>> I took a sample 15 numbers (pulled from a random dist.), built a
>> Butterworth filter, and ran filtfilt on the data using Matlab and scipy.
>>  The results are in the same ballpark, but I am still seeing differences
>> between Matlab and scipy at the second decimal place.  This concerns me
>> since I expected to see differences at the lower order bits.  Has anybody
>> else seen this or know why this is the case?
>>
>> I've pasted my 15 random numbers and output from both Matlab and scipy
>> below.
>>
>> Thanks,
>> John K.
>>
>>
>> PS - I tried taking the Butterworth filter from scip
>>
>>
>> ---- scipy v0.13.3, numpy v.1.6.2 ----
>>
>> data = numpy.array([0.401808033751942, 0.075966691690842,
>> 0.239916153553658, 0.123318934835166, 0.183907788282417, 0.239952525664903,
>> 0.417267069084370, 0.049654430325742, 0.902716109915281, 0.944787189721646,
>> 0.490864092468080, 0.489252638400019, 0.337719409821377, 0.900053846417662,
>> 0.369246781120215])
>> [
>> b,a] = scipy.signal.butter(3,0.2)
>>
>> >>> b
>> array([ 0.01809893,  0.0542968 ,  0.0542968 ,  0.01809893])
>>
>> >>> b[0]
>> 0.018098933007514431
>> >>> b[1]
>> 0.054296799022543286
>> >>> b[2]
>> 0.054296799022543286
>> >>> b[3]
>> 0.018098933007514431
>>
>> >>> a
>> array([ 1.        , -1.76004188,  1.18289326, -0.27805992])
>>
>> >>> a[0]
>> 1.0
>> >>> a[1]
>> -1.7600418803431686
>> >>> a[2]
>> 1.1828932620378303
>> >>> a[3]
>> -0.27805991763454624
>>
>> scipy.signal.filtfilt(b,a,data)
>>
>> array([ 0.40421981,  0.29292145,  0.20773242,  0.1682317 ,  0.18250909,
>>         0.24674659,  0.34676359,  0.46045534,  0.56190086,  0.62844665,
>>         0.64819638,  0.62192427,  0.55831894,  0.46881537,  0.36653733])
>>
>>
>> ---- MATLAB v.7.9.0.529 ----
>>
>> data = [0.401808033751942, 0.075966691690842, 0.239916153553658,
>> 0.123318934835166, 0.183907788282417, 0.239952525664903, 0.417267069084370,
>> 0.049654430325742, 0.902716109915281, 0.944787189721646, 0.490864092468080,
>> 0.489252638400019, 0.337719409821377, 0.900053846417662, 0.369246781120215]
>>
>> >> [b,a] = butter(3,0.2)
>>
>> b =
>>
>>    0.018098933007514   0.054296799022543   0.054296799022543
>> 0.018098933007514
>>
>>
>> a =
>>
>>    1.000000000000000  -1.760041880343169   1.182893262037831
>>  -0.278059917634546
>>
>> >> filtfilt(b,a,data)
>>
>> ans =
>>
>>   Columns 1 through 5
>>
>>    0.397839293822166   0.286273839374148   0.202969550095165
>> 0.166087236796221   0.182700682986968
>>
>>   Columns 6 through 10
>>
>>    0.248436130460523   0.348904396853114   0.461998356964929
>> 0.561930803937072   0.626352300416756
>>
>>   Columns 11 through 15
>>
>>    0.643941843065195   0.616392843959916   0.553574502943643
>> 0.468014071436841   0.373188133985282
>>
>>
>>
>
> John,
>
> The difference is in the default padding length.  In matlab's filtfilt, it
> is 3*(max(len(a), len(b)) - 1), and in scipy's filtfilt, it is
> 3*max(len(a), len(b)).  If you set padlen to the same value used in
> matlab, the results match:
>
>  In [47]: filtfilt(b, a, data, padlen=9)
> Out[47]:
> array([ 0.39783929,  0.28627384,  0.20296955,  0.16608724,  0.18270068,
>         0.24843613,  0.3489044 ,  0.46199836,  0.5619308 ,  0.6263523 ,
>         0.64394184,  0.61639284,  0.5535745 ,  0.46801407,  0.37318813])
>
>
> Warren
>
>
>
>> On Tue, Apr 15, 2014 at 4:38 PM, Warren Weckesser <
>> warren.weckesser at gmail.com> wrote:
>>
>>>  On 4/15/14, John Krasting - NOAA Federal <john.krasting at noaa.gov>
>>> wrote:
>>> > Hi Scipy Users -
>>> >
>>> > Am I correct in reading that filtfilt in scipy.signal (v. 13.0) does
>>> not
>>> > extrapolate data at the beginning and the end of a time series when
>>> using
>>> > the filtfilt function?
>>> >
>>> > Based on the Matlab code (
>>> >
>>> http://chmielowski.eu/POLITECHNIKA/Dydaktyka/AUTOMATYKA/AutoLab/Matlab/TOOLBOX/SIGNAL/FILTFILT.M
>>> ),
>>> > I see that there is a block of code that does the reflection:
>>> >
>>> > % Extrapolate beginning and end of data sequence using a "reflection
>>> > % method".  Slopes of original and extrapolated sequences match at
>>> > % the end points.
>>> > % This reduces end effects.
>>> >     y = [2*x(1)-x((nfact+1):-1:2);x;2*x(len)-x((len-1):-1:len-nfact)];
>>> >
>>> > I did not see this in the source for scipy's filtfilt function.   Are
>>> there
>>> > any plans to add this as a possible option to the function?
>>> >
>>> > Thanks,
>>> > John K.
>>> >
>>>
>>>
>>> That matlab code makes an odd extension of the data.  E.g.
>>>
>>>
>>> octave:35> len = 8
>>> len =  8
>>> octave:36> x = [1:len]';
>>> octave:37> x'
>>> ans =
>>>
>>>    1   2   3   4   5   6   7   8
>>>
>>> octave:38> nfact = 3;
>>> octave:39> y =
>>> [2*x(1)-x((nfact+1):-1:2);x;2*x(len)-x((len-1):-1:len-nfact)];
>>> octave:40> y'
>>> ans =
>>>
>>>    -2   -1    0    1    2    3    4    5    6    7    8    9   10   11
>>>
>>>
>>> That type of extension is the default behavior of filtfilt
>>> (
>>> http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.filtfilt.html
>>> ).
>>>  You can change the type of the extension and the length of the
>>> padding with the padtype and padlen arguments.
>>>
>>> Warren
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140415/d88e381c/attachment.html>

From tweetsforzion at gmail.com  Thu Apr 17 10:10:35 2014
From: tweetsforzion at gmail.com (Steve Miller)
Date: Thu, 17 Apr 2014 08:10:35 -0600
Subject: [SciPy-User] How to map spherical coordinate data to cartesian grid?
Message-ID: <CANtDh-42_eZtiwHhb1cMCaR_=9oK+hLJ9DKKRQJAu9xRaETWKg@mail.gmail.com>

Dear all.  I was send 4 long 1D arrays (flattened) representing 3D data in
spherical coordinates:

rho: density array
r:     spherical coordinate r array
th:   spherical coordinate theta array
phi: spherical coordinate phi array

where the r, th, and phi arrays are the coordinates for the density array
rho. These arrays are not uniformly spaces in r, th and phi.

I would like to map rho to cartesian coordinates of a 3D cube. IE, define:

X, Y, Z = mgrid[etc...]

Is there a handy scipy routine that will map/interpolate these rho values
in spherical coordinates across my new cartesian box? Naively I thought I
could use something like map_coordinates but I am unclear how I would use
this for this calculation. Could someone provide a simple example script
for how I could use scipy to do this? Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140417/160809cb/attachment.html>

From marcocamma at gmail.com  Thu Apr 17 10:59:59 2014
From: marcocamma at gmail.com (marco cammarata)
Date: Thu, 17 Apr 2014 14:59:59 +0000 (UTC)
Subject: [SciPy-User] numpy array assignment
Message-ID: <loom.20140417T164828-27@post.gmane.org>

Hi,

I have used many times syntax like

import numpy as np

def result_of_long_calculation_returning_the_right_shape():
  return np.random.random( (2,3) )


N=10
out = np.empty( (N,2,3) )
for i in range(N):
  out[i] = result_of_long_calculation_returning_the_right_shape()

if I have to loop trough the axis 1 I would do something like

N=10
out = np.empty( (2,N,3) )
for i in range(N):
  out[:,i,:] = result_of_long_calculation_returning_the_right_shape()

The problem is that - in the function I am writing - the number of 
dimensions and the axis to loop trough are not known in advance.

Is there a smart what to do it ?
I would expect that a function like np.assign(array,element=0,axis=0) that  
but I can't find such function.

For the moment I ended up doing something really really inelegant:
  res = result_of_long_calculation_returning_the_right_shape()
  s = "out[" + (":,")*axis + "i" + (",:")*(nDim-axis-1) + "]=res"
  exec(s)

Thanks a lot in advance,
marco






From tritemio at gmail.com  Thu Apr 17 18:12:23 2014
From: tritemio at gmail.com (Antonino Ingargiola)
Date: Thu, 17 Apr 2014 15:12:23 -0700
Subject: [SciPy-User] numpy array assignment
In-Reply-To: <loom.20140417T164828-27@post.gmane.org>
References: <loom.20140417T164828-27@post.gmane.org>
Message-ID: <CANn2QUyrbc03tSkqxHUPCETQnYK4G4GAjBPrphQ_Zj8rf2CYxg@mail.gmail.com>

On Thu, Apr 17, 2014 at 7:59 AM, marco cammarata <marcocamma at gmail.com>wrote:

> For the moment I ended up doing something really really inelegant:
>   res = result_of_long_calculation_returning_the_right_shape()
>   s = "out[" + (":,")*axis + "i" + (",:")*(nDim-axis-1) + "]=res"
>   exec(s)
>

When you do a[:, :, :], that's just syntactic sugar for a[(slice(None),
slice(None), slice(None),)] (note that slice(None) means take all
elements). In other words, the index of a ndarray is a tuple of
objects/slices.

You can build the tuple programmatically. Another example:

    import numpy as np
    a = np.arange(1000).reshape(10, 5, 20)
    index = (slice(None), slice(0, 1), slice(None))
    a[index] = 3
    a

Cheers,
Antonio
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140417/38eb96f4/attachment.html>

From marcocamma at gmail.com  Fri Apr 18 03:51:32 2014
From: marcocamma at gmail.com (marco cammarata)
Date: Fri, 18 Apr 2014 07:51:32 +0000 (UTC)
Subject: [SciPy-User] numpy array assignment
References: <loom.20140417T164828-27@post.gmane.org>
	<CANn2QUyrbc03tSkqxHUPCETQnYK4G4GAjBPrphQ_Zj8rf2CYxg@mail.gmail.com>
Message-ID: <loom.20140418T094834-261@post.gmane.org>

Antonino Ingargiola <tritemio <at> gmail.com> writes:

> 
> 
> On Thu, Apr 17, 2014 at 7:59 AM, marco cammarata <marcocamma <at> 
gmail.com> wrote:?
> 
> 
> 
> 
> For the moment I ended up doing something really really inelegant:
> ? res = result_of_long_calculation_returning_the_right_shape()
> ? s = "out[" + (":,")*axis + "i" + (",:")*(nDim-axis-1) + "]=res"
> ? exec(s)
> 
> 
> When you do a[:, :, :], that's just syntactic sugar for a[(slice(None), 
slice(None), slice(None),)] (note that slice(None) means take all elements). 
In other words, the index of a ndarray is a tuple of objects/slices.
> 
> You can build the tuple programmatically. Another example:
> 
> ? ? import numpy as np
> ? ??a = np.arange(1000).reshape(10, 5, 20)
> ? ??index = (slice(None), slice(0, 1), slice(None))
> 
> ? ??a[index] = 3
> ? ??a
> 
> 
> Cheers,
> Antonio
> 
> ?
> 
> 
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
> 

Antonio,

thanks for your comment.
Indeed your solution is better than mine because avoid the exec but would 
still require the construction of the tuple by checking the axis and the 
number of dimension.

What I was really looking for is a more general (compact) solution.

Thanks anyway,
ciao
marco





From daoust.mj at gmail.com  Fri Apr 18 06:39:16 2014
From: daoust.mj at gmail.com (Mark Daoust)
Date: Fri, 18 Apr 2014 06:39:16 -0400
Subject: [SciPy-User] numpy array assignment
In-Reply-To: <loom.20140418T094834-261@post.gmane.org>
References: <loom.20140417T164828-27@post.gmane.org>
	<CANn2QUyrbc03tSkqxHUPCETQnYK4G4GAjBPrphQ_Zj8rf2CYxg@mail.gmail.com>
	<loom.20140418T094834-261@post.gmane.org>
Message-ID: <CAFg8gYxi71XwmhZevScPsZFyCw-HtJJUhi8_A5G-UNQR3e9LoQ@mail.gmail.com>

Using slices and tuples, as suggested by Antonio, we can redo your original
example like this:

    colon = (slice(None),)
    out[colon*axis+(i,)+colon*(nDim-axis-1,)]=result

Ellipsis, which represents "as many colons as are needed", can simplify it
farther:

    out[colon*axis+(i,...)]=result

but numpy slicing always has an implied trailing ellipsis so, I think, this
is equivalent:

    out[colon*axis+(i,)]=result

The ellipses are more useful if you have a "negative" axis, counting from
the right instead of left. For example you can hit all but the last (-1)
and second to last (-2) axis  with:

    out[...,i]=result
    out[...,i,:]=result

or all but axis (-n) with:

    out[(...,i)+colon*(n)] = result




Mark Daoust


> You can build the tuple programmatically. Another example:
> >
> >     import numpy as np
> >     a = np.arange(1000).reshape(10, 5, 20)
> >     index = (slice(None), slice(0, 1), slice(None))
> >
> >     a[index] = 3
> >     a
> >
> >
> > Cheers,
> > Antonio
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140418/07ad2369/attachment.html>

From sturla.molden at gmail.com  Thu Apr 24 07:49:35 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Thu, 24 Apr 2014 11:49:35 +0000 (UTC)
Subject: [SciPy-User] How to map spherical coordinate data to cartesian
	grid?
References: <CANtDh-42_eZtiwHhb1cMCaR_=9oK+hLJ9DKKRQJAu9xRaETWKg@mail.gmail.com>
Message-ID: <2039582185420032741.050360sturla.molden-gmail.com@news.gmane.org>

Steve Miller <tweetsforzion at gmail.com> wrote:
> Dear all.  I was send 4 long 1D arrays (flattened) representing 3D data in
> spherical coordinates:
> 
> rho: density array
> r:     spherical coordinate r array
> th:   spherical coordinate theta array
> phi: spherical coordinate phi array
> 

Usually you transform to Cartesian coordinates like so:

x = r * sin(theta) * cos(phi)
y = r * cos(theta) * sin(phi)
z = r * cos(theta)


Sturla



From sturla.molden at gmail.com  Thu Apr 24 08:04:21 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Thu, 24 Apr 2014 12:04:21 +0000 (UTC)
Subject: [SciPy-User] How to map spherical coordinate data to cartesian
	grid?
References: <CANtDh-42_eZtiwHhb1cMCaR_=9oK+hLJ9DKKRQJAu9xRaETWKg@mail.gmail.com>
	<2039582185420032741.050360sturla.molden-gmail.com@news.gmane.org>
Message-ID: <220334344420033359.075979sturla.molden-gmail.com@news.gmane.org>

Sturla Molden <sturla.molden at gmail.com> wrote:

> Usually you transform to Cartesian coordinates like so:
> 
> x = r * sin(theta) * cos(phi)
> y = r * cos(theta) * sin(phi)
> z = r * cos(theta)

And once you have the points in Cartesian space, you can use
scipy.spatial.cKDTree to quickly find the nearest-neighbor in your grid. If
you need the voxel, you can see how the point is located with respect to
the nearest neighbor. There are 8 different possibilities to consider.
Alternatively you can use bisection search along x, y, and z axes to map
each point into a voxel. The latter would take some Cython coding to be
fast.

Sturla



From sebastian at sipsolutions.net  Thu Apr 24 11:42:54 2014
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Thu, 24 Apr 2014 17:42:54 +0200
Subject: [SciPy-User] DAQmx python wrapper
Message-ID: <1398354174.2782.8.camel@sebastian-t440>

Hey,

We are thinking about writing some DAQ programs in python, and I was
wondering if anyone can recommend a lib for that. There seem to be a
bunch of wrappers using ctypes, which seem all good enough, but if there
is something more highlevel or very well done, I would be happy to use
that :).

Regards,

Sebastian



From charlesr.harris at gmail.com  Thu Apr 24 22:23:26 2014
From: charlesr.harris at gmail.com (Charles R Harris)
Date: Thu, 24 Apr 2014 20:23:26 -0600
Subject: [SciPy-User] DAQmx python wrapper
In-Reply-To: <1398354174.2782.8.camel@sebastian-t440>
References: <1398354174.2782.8.camel@sebastian-t440>
Message-ID: <CAB6mnxLoja67mA0E0=yDEhA97hZwQGpSuBjEBik2o3jSL4Cnpg@mail.gmail.com>

Hi Sebastian,


On Thu, Apr 24, 2014 at 9:42 AM, Sebastian Berg
<sebastian at sipsolutions.net>wrote:

> Hey,
>
> We are thinking about writing some DAQ programs in python, and I was
> wondering if anyone can recommend a lib for that. There seem to be a
> bunch of wrappers using ctypes, which seem all good enough, but if there
> is something more highlevel or very well done, I would be happy to use
> that :).
>
> Regards,
>
>
Is DAQ data acquisition? If so, what sort of data?

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140424/d28be0a9/attachment.html>

From hasslerjc at comcast.net  Sat Apr 26 09:04:17 2014
From: hasslerjc at comcast.net (John Hassler)
Date: Sat, 26 Apr 2014 09:04:17 -0400
Subject: [SciPy-User] DAQmx python wrapper
In-Reply-To: <CAB6mnxLoja67mA0E0=yDEhA97hZwQGpSuBjEBik2o3jSL4Cnpg@mail.gmail.com>
References: <1398354174.2782.8.camel@sebastian-t440>
	<CAB6mnxLoja67mA0E0=yDEhA97hZwQGpSuBjEBik2o3jSL4Cnpg@mail.gmail.com>
Message-ID: <535BAED1.6000207@comcast.net>

An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140426/c9329632/attachment.html>

From sebastian at sipsolutions.net  Sat Apr 26 12:53:15 2014
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Sat, 26 Apr 2014 18:53:15 +0200
Subject: [SciPy-User] DAQmx python wrapper
In-Reply-To: <535BAED1.6000207@comcast.net>
References: <1398354174.2782.8.camel@sebastian-t440>
	<CAB6mnxLoja67mA0E0=yDEhA97hZwQGpSuBjEBik2o3jSL4Cnpg@mail.gmail.com>
	<535BAED1.6000207@comcast.net>
Message-ID: <1398531195.7311.25.camel@sebastian-t440>

On Sa, 2014-04-26 at 09:04 -0400, John Hassler wrote:
> 
> On 4/24/2014 10:23 PM, Charles R Harris wrote:
> 
> > Hi Sebastian,
> > 
> > 
> > On Thu, Apr 24, 2014 at 9:42 AM, Sebastian Berg
> > <sebastian at sipsolutions.net> wrote:
> >         Hey,
> >         
> >         We are thinking about writing some DAQ programs in python,
> >         and I was
> >         wondering if anyone can recommend a lib for that. There seem
> >         to be a
> >         bunch of wrappers using ctypes, which seem all good enough,
> >         but if there
> >         is something more highlevel or very well done, I would be
> >         happy to use
> >         that :).
> >         
> >         Regards,
> >         
> > 
> > 
> > Is DAQ data acquisition? If so, what sort of data?
> > 
> > 
> > Chuck
> > 
> > 
> Which DAQ are you using?  There is 'comedi' for internal boards, but I
> don't know of anything except the "ctype wrappers" for usb boxes.
> When I started doing this several years ago, I wrote my own ctype
> wrappers to access the dll for the Measurement Computing USB-1208 and
> for the NI-USB6008.  Mine were very simple, and did only what I
> needed.  There's a number of them on the web now, but I haven't looked
> an any of them in detail.
> 

Sorry, thought the DAQ was more clear then it probably is. What we have
is a National instruments USB DAQ unit (M-Series). We are actually using
it more for analog output then input right now.

> Andrew Straw wrote a very nice object-oriented driver for the
> Measurement Computing USB devices in Python.  However, I found
> references to it in Cookbook/Data Acquisition, but I couldn't find the
> actual PyUniversalLibrary.  Maybe it's no longer active.
> 

I will have a look if I find it, that sounds interesting, I also was
hoping that a wrapper could maybe help with multiprocessing magic to
separate the gui as much as possible from the signal generation. Though
maybe that is too high level to really do in any case.
I have started playing with one of the ctypes wrappers and it indeed
works very well. Rather annoying, because I initially wrote labview and
now have to realize that the C-Api is identical and can easily be called
from python, too :)...

- Sebastian

> 
> "Ctype wrappers" have worked very well for me.  We have one experiment
> using two USB-1208 and one NI-USB-6008 simultaneously in a single
> Python program with a Tkinter GUI, and it works fine.
> 
> john
> 
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user




From josef.pktd at gmail.com  Sun Apr 27 22:31:31 2014
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 27 Apr 2014 22:31:31 -0400
Subject: [SciPy-User] question: OLS with sparse design and dense parameter
Message-ID: <CAMMTP+ADtToLDfC6VXSJHK9zXcyWipLW7LmEA=QjG9D7WW8SPw@mail.gmail.com>

(sunday night quest for another sparse recipe)

This is a sparse algorithm question:

scipy.sparse.linalg has a choice of linear system solvers, but I have not
much idea about the relative merits. Is there a recommendation for the
following case?

suppose I want to do estimate a standard linear model by OLS
y = x * b,  find b = argmin(y - x*b)

x is sparse, with blocks, medium large (20000 by 2000) but if it works
quite a bit larger,
but we don't have a small number of dominant singular values, the solution
of the linear system will be dense.

the basic parts of my minimum example

nobsi, n_groups, k_vars = 10, 2000, 3

#nobsi, n_groups, k_vars = 2, 3, 3


xi = np.arange(nobsi)[:,None]**np.arange(3) #vander

x_sp = sparse.kron(sparse.eye(n_groups), xi).tocsr()

(balanced kron structure is just to make a quick example)


x_sp.shape

(20000, 6000)


first try


xpx = x_sp.T.dot(x_sp) # sparse

xpy = x_sp.T.dot(y) # dense


p_dense = np.linalg.solve(xpx.toarray(), xpy)

p_sp1 = sparse.linalg.spsolve(xpx, xpy)

p_sp2 = sparse.linalg.lsmr(x_sp, y)[0]


timing: 15.6659998894 0.00500011444092 0.00600004196167


Is this too small to worry, and anything works, or is there a
recommendation?

definitely don't use dense.


Thanks,


Josef
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140427/dff3c453/attachment.html>

From njs at pobox.com  Mon Apr 28 02:42:28 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 28 Apr 2014 07:42:28 +0100
Subject: [SciPy-User] question: OLS with sparse design and dense
	parameter
In-Reply-To: <CAMMTP+ADtToLDfC6VXSJHK9zXcyWipLW7LmEA=QjG9D7WW8SPw@mail.gmail.com>
References: <CAMMTP+ADtToLDfC6VXSJHK9zXcyWipLW7LmEA=QjG9D7WW8SPw@mail.gmail.com>
Message-ID: <CAPJVwBkJ4LQeU3KADuQiORZ7RHpcwLVtbf3p1_H2cF0TizxruQ@mail.gmail.com>

I guess the classic solution for ols are sparse cholesky or sparse qr.
scikits.sparse has a cholmod wrapper, and could be modified to have a
suitesparseqr wrapper relatively easily.
On 28 Apr 2014 03:31, <josef.pktd at gmail.com> wrote:

> (sunday night quest for another sparse recipe)
>
> This is a sparse algorithm question:
>
> scipy.sparse.linalg has a choice of linear system solvers, but I have not
> much idea about the relative merits. Is there a recommendation for the
> following case?
>
> suppose I want to do estimate a standard linear model by OLS
> y = x * b,  find b = argmin(y - x*b)
>
> x is sparse, with blocks, medium large (20000 by 2000) but if it works
> quite a bit larger,
> but we don't have a small number of dominant singular values, the solution
> of the linear system will be dense.
>
> the basic parts of my minimum example
>
> nobsi, n_groups, k_vars = 10, 2000, 3
>
> #nobsi, n_groups, k_vars = 2, 3, 3
>
>
> xi = np.arange(nobsi)[:,None]**np.arange(3) #vander
>
> x_sp = sparse.kron(sparse.eye(n_groups), xi).tocsr()
>
> (balanced kron structure is just to make a quick example)
>
>
> x_sp.shape
>
> (20000, 6000)
>
>
> first try
>
>
> xpx = x_sp.T.dot(x_sp) # sparse
>
> xpy = x_sp.T.dot(y) # dense
>
>
> p_dense = np.linalg.solve(xpx.toarray(), xpy)
>
> p_sp1 = sparse.linalg.spsolve(xpx, xpy)
>
> p_sp2 = sparse.linalg.lsmr(x_sp, y)[0]
>
>
> timing: 15.6659998894 0.00500011444092 0.00600004196167
>
>
> Is this too small to worry, and anything works, or is there a
> recommendation?
>
> definitely don't use dense.
>
>
> Thanks,
>
>
> Josef
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140428/ec63572b/attachment.html>

From sholzergr at gmail.com  Mon Apr 28 02:45:29 2014
From: sholzergr at gmail.com (Severin Holzer-Graf)
Date: Mon, 28 Apr 2014 08:45:29 +0200
Subject: [SciPy-User] question: OLS with sparse design and dense
	parameter
In-Reply-To: <CAPJVwBkJ4LQeU3KADuQiORZ7RHpcwLVtbf3p1_H2cF0TizxruQ@mail.gmail.com>
References: <CAMMTP+ADtToLDfC6VXSJHK9zXcyWipLW7LmEA=QjG9D7WW8SPw@mail.gmail.com>
	<CAPJVwBkJ4LQeU3KADuQiORZ7RHpcwLVtbf3p1_H2cF0TizxruQ@mail.gmail.com>
Message-ID: <CAFk72czM2AqSAYaLrRUwR1nT_dS8BsXM+jT=zwWhPu42y83YMQ@mail.gmail.com>

I needed SuiteSparseQR for my work and added the backslash function of
it to scikits.sparse.
The modification is available here: https://github.com/duga3/scikits-sparse

2014-04-28 8:42 GMT+02:00 Nathaniel Smith <njs at pobox.com>:
> I guess the classic solution for ols are sparse cholesky or sparse qr.
> scikits.sparse has a cholmod wrapper, and could be modified to have a
> suitesparseqr wrapper relatively easily.
>
> On 28 Apr 2014 03:31, <josef.pktd at gmail.com> wrote:
>>
>> (sunday night quest for another sparse recipe)
>>
>> This is a sparse algorithm question:
>>
>> scipy.sparse.linalg has a choice of linear system solvers, but I have not
>> much idea about the relative merits. Is there a recommendation for the
>> following case?
>>
>> suppose I want to do estimate a standard linear model by OLS
>> y = x * b,  find b = argmin(y - x*b)
>>
>> x is sparse, with blocks, medium large (20000 by 2000) but if it works
>> quite a bit larger,
>> but we don't have a small number of dominant singular values, the solution
>> of the linear system will be dense.
>>
>> the basic parts of my minimum example
>>
>> nobsi, n_groups, k_vars = 10, 2000, 3
>>
>> #nobsi, n_groups, k_vars = 2, 3, 3
>>
>>
>> xi = np.arange(nobsi)[:,None]**np.arange(3) #vander
>>
>> x_sp = sparse.kron(sparse.eye(n_groups), xi).tocsr()
>>
>> (balanced kron structure is just to make a quick example)
>>
>>
>> x_sp.shape
>>
>> (20000, 6000)
>>
>>
>> first try
>>
>>
>> xpx = x_sp.T.dot(x_sp) # sparse
>>
>> xpy = x_sp.T.dot(y) # dense
>>
>>
>> p_dense = np.linalg.solve(xpx.toarray(), xpy)
>>
>> p_sp1 = sparse.linalg.spsolve(xpx, xpy)
>>
>> p_sp2 = sparse.linalg.lsmr(x_sp, y)[0]
>>
>>
>> timing: 15.6659998894 0.00500011444092 0.00600004196167
>>
>>
>> Is this too small to worry, and anything works, or is there a
>> recommendation?
>>
>> definitely don't use dense.
>>
>>
>> Thanks,
>>
>>
>> Josef
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>


From gary.pajer at gmail.com  Mon Apr 28 13:51:10 2014
From: gary.pajer at gmail.com (Gary Pajer)
Date: Mon, 28 Apr 2014 13:51:10 -0400
Subject: [SciPy-User] DAQmx python wrapper
In-Reply-To: <1398531195.7311.25.camel@sebastian-t440>
References: <1398354174.2782.8.camel@sebastian-t440>
	<CAB6mnxLoja67mA0E0=yDEhA97hZwQGpSuBjEBik2o3jSL4Cnpg@mail.gmail.com>
	<535BAED1.6000207@comcast.net>
	<1398531195.7311.25.camel@sebastian-t440>
Message-ID: <CALGGngAkMnn-8LiPLCKs0uN=VB68zK731wRSLioFBvad8ggA8Q@mail.gmail.com>

I've used Andrew's wrapper, and it worked well.

But in the end I wrote my own routines using ctypes (stealing some ideas
from Andrew).   I can't remember exactly why I wrote my own... it was
several years ago.  But I do remember that it was really very easy to
implement calls using ctypes.

~-~-~


On Sat, Apr 26, 2014 at 12:53 PM, Sebastian Berg <sebastian at sipsolutions.net
> wrote:

> On Sa, 2014-04-26 at 09:04 -0400, John Hassler wrote:
> >
> > On 4/24/2014 10:23 PM, Charles R Harris wrote:
> >
> > > Hi Sebastian,
> > >
> > >
> > > On Thu, Apr 24, 2014 at 9:42 AM, Sebastian Berg
> > > <sebastian at sipsolutions.net> wrote:
> > >         Hey,
> > >
> > >         We are thinking about writing some DAQ programs in python,
> > >         and I was
> > >         wondering if anyone can recommend a lib for that. There seem
> > >         to be a
> > >         bunch of wrappers using ctypes, which seem all good enough,
> > >         but if there
> > >         is something more highlevel or very well done, I would be
> > >         happy to use
> > >         that :).
> > >
> > >         Regards,
> > >
> > >
> > >
> > > Is DAQ data acquisition? If so, what sort of data?
> > >
> > >
> > > Chuck
> > >
> > >
> > Which DAQ are you using?  There is 'comedi' for internal boards, but I
> > don't know of anything except the "ctype wrappers" for usb boxes.
> > When I started doing this several years ago, I wrote my own ctype
> > wrappers to access the dll for the Measurement Computing USB-1208 and
> > for the NI-USB6008.  Mine were very simple, and did only what I
> > needed.  There's a number of them on the web now, but I haven't looked
> > an any of them in detail.
> >
>
> Sorry, thought the DAQ was more clear then it probably is. What we have
> is a National instruments USB DAQ unit (M-Series). We are actually using
> it more for analog output then input right now.
>
> > Andrew Straw wrote a very nice object-oriented driver for the
> > Measurement Computing USB devices in Python.  However, I found
> > references to it in Cookbook/Data Acquisition, but I couldn't find the
> > actual PyUniversalLibrary.  Maybe it's no longer active.
> >
>
> I will have a look if I find it, that sounds interesting, I also was
> hoping that a wrapper could maybe help with multiprocessing magic to
> separate the gui as much as possible from the signal generation. Though
> maybe that is too high level to really do in any case.
> I have started playing with one of the ctypes wrappers and it indeed
> works very well. Rather annoying, because I initially wrote labview and
> now have to realize that the C-Api is identical and can easily be called
> from python, too :)...
>
> - Sebastian
>
> >
> > "Ctype wrappers" have worked very well for me.  We have one experiment
> > using two USB-1208 and one NI-USB-6008 simultaneously in a single
> > Python program with a Tkinter GUI, and it works fine.
> >
> > john
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140428/2db18f07/attachment.html>

From gary.pajer at gmail.com  Mon Apr 28 15:39:03 2014
From: gary.pajer at gmail.com (Gary Pajer)
Date: Mon, 28 Apr 2014 15:39:03 -0400
Subject: [SciPy-User] DAQmx python wrapper
In-Reply-To: <CALGGngAkMnn-8LiPLCKs0uN=VB68zK731wRSLioFBvad8ggA8Q@mail.gmail.com>
References: <1398354174.2782.8.camel@sebastian-t440>
	<CAB6mnxLoja67mA0E0=yDEhA97hZwQGpSuBjEBik2o3jSL4Cnpg@mail.gmail.com>
	<535BAED1.6000207@comcast.net>
	<1398531195.7311.25.camel@sebastian-t440>
	<CALGGngAkMnn-8LiPLCKs0uN=VB68zK731wRSLioFBvad8ggA8Q@mail.gmail.com>
Message-ID: <CALGGngBY=R=K3TcVFbZON8=nAjG+U_vftUWe71PpDbCqoyy2LA@mail.gmail.com>

Andrew Straw's PyUniversalLibrary appears to be on
PyPi<https://pypi.python.org/pypi/PyUniversalLibrary/>

~-~-~


On Mon, Apr 28, 2014 at 1:51 PM, Gary Pajer <gary.pajer at gmail.com> wrote:

> I've used Andrew's wrapper, and it worked well.
>
> But in the end I wrote my own routines using ctypes (stealing some ideas
> from Andrew).   I can't remember exactly why I wrote my own... it was
> several years ago.  But I do remember that it was really very easy to
> implement calls using ctypes.
>
> ~-~-~
>
>
> On Sat, Apr 26, 2014 at 12:53 PM, Sebastian Berg <
> sebastian at sipsolutions.net> wrote:
>
>> On Sa, 2014-04-26 at 09:04 -0400, John Hassler wrote:
>> >
>> > On 4/24/2014 10:23 PM, Charles R Harris wrote:
>> >
>> > > Hi Sebastian,
>> > >
>> > >
>> > > On Thu, Apr 24, 2014 at 9:42 AM, Sebastian Berg
>> > > <sebastian at sipsolutions.net> wrote:
>> > >         Hey,
>> > >
>> > >         We are thinking about writing some DAQ programs in python,
>> > >         and I was
>> > >         wondering if anyone can recommend a lib for that. There seem
>> > >         to be a
>> > >         bunch of wrappers using ctypes, which seem all good enough,
>> > >         but if there
>> > >         is something more highlevel or very well done, I would be
>> > >         happy to use
>> > >         that :).
>> > >
>> > >         Regards,
>> > >
>> > >
>> > >
>> > > Is DAQ data acquisition? If so, what sort of data?
>> > >
>> > >
>> > > Chuck
>> > >
>> > >
>> > Which DAQ are you using?  There is 'comedi' for internal boards, but I
>> > don't know of anything except the "ctype wrappers" for usb boxes.
>> > When I started doing this several years ago, I wrote my own ctype
>> > wrappers to access the dll for the Measurement Computing USB-1208 and
>> > for the NI-USB6008.  Mine were very simple, and did only what I
>> > needed.  There's a number of them on the web now, but I haven't looked
>> > an any of them in detail.
>> >
>>
>> Sorry, thought the DAQ was more clear then it probably is. What we have
>> is a National instruments USB DAQ unit (M-Series). We are actually using
>> it more for analog output then input right now.
>>
>> > Andrew Straw wrote a very nice object-oriented driver for the
>> > Measurement Computing USB devices in Python.  However, I found
>> > references to it in Cookbook/Data Acquisition, but I couldn't find the
>> > actual PyUniversalLibrary.  Maybe it's no longer active.
>> >
>>
>> I will have a look if I find it, that sounds interesting, I also was
>> hoping that a wrapper could maybe help with multiprocessing magic to
>> separate the gui as much as possible from the signal generation. Though
>> maybe that is too high level to really do in any case.
>> I have started playing with one of the ctypes wrappers and it indeed
>> works very well. Rather annoying, because I initially wrote labview and
>> now have to realize that the C-Api is identical and can easily be called
>> from python, too :)...
>>
>> - Sebastian
>>
>> >
>> > "Ctype wrappers" have worked very well for me.  We have one experiment
>> > using two USB-1208 and one NI-USB-6008 simultaneously in a single
>> > Python program with a Tkinter GUI, and it works fine.
>> >
>> > john
>> >
>> >
>> > _______________________________________________
>> > SciPy-User mailing list
>> > SciPy-User at scipy.org
>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140428/506012f5/attachment.html>

From josef.pktd at gmail.com  Mon Apr 28 15:47:30 2014
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 28 Apr 2014 15:47:30 -0400
Subject: [SciPy-User] DAQmx python wrapper
In-Reply-To: <CALGGngBY=R=K3TcVFbZON8=nAjG+U_vftUWe71PpDbCqoyy2LA@mail.gmail.com>
References: <1398354174.2782.8.camel@sebastian-t440>
	<CAB6mnxLoja67mA0E0=yDEhA97hZwQGpSuBjEBik2o3jSL4Cnpg@mail.gmail.com>
	<535BAED1.6000207@comcast.net>
	<1398531195.7311.25.camel@sebastian-t440>
	<CALGGngAkMnn-8LiPLCKs0uN=VB68zK731wRSLioFBvad8ggA8Q@mail.gmail.com>
	<CALGGngBY=R=K3TcVFbZON8=nAjG+U_vftUWe71PpDbCqoyy2LA@mail.gmail.com>
Message-ID: <CAMMTP+DQaOx8Mo+4ubQrU4KdaeR5Hue8Qwf0EZY3UTmWYjsDkA@mail.gmail.com>

On Mon, Apr 28, 2014 at 3:39 PM, Gary Pajer <gary.pajer at gmail.com> wrote:

> Andrew Straw's PyUniversalLibrary appears to be on PyPi<https://pypi.python.org/pypi/PyUniversalLibrary/>
>
>
google search points here
https://code.astraw.com/PyUniversalLibrary/
http://code.astraw.com/PyUniversalLibrary-downloads/

Josef

> ~-~-~
>
>
> On Mon, Apr 28, 2014 at 1:51 PM, Gary Pajer <gary.pajer at gmail.com> wrote:
>
>> I've used Andrew's wrapper, and it worked well.
>>
>> But in the end I wrote my own routines using ctypes (stealing some ideas
>> from Andrew).   I can't remember exactly why I wrote my own... it was
>> several years ago.  But I do remember that it was really very easy to
>> implement calls using ctypes.
>>
>> ~-~-~
>>
>>
>> On Sat, Apr 26, 2014 at 12:53 PM, Sebastian Berg <
>> sebastian at sipsolutions.net> wrote:
>>
>>> On Sa, 2014-04-26 at 09:04 -0400, John Hassler wrote:
>>> >
>>> > On 4/24/2014 10:23 PM, Charles R Harris wrote:
>>> >
>>> > > Hi Sebastian,
>>> > >
>>> > >
>>> > > On Thu, Apr 24, 2014 at 9:42 AM, Sebastian Berg
>>> > > <sebastian at sipsolutions.net> wrote:
>>> > >         Hey,
>>> > >
>>> > >         We are thinking about writing some DAQ programs in python,
>>> > >         and I was
>>> > >         wondering if anyone can recommend a lib for that. There seem
>>> > >         to be a
>>> > >         bunch of wrappers using ctypes, which seem all good enough,
>>> > >         but if there
>>> > >         is something more highlevel or very well done, I would be
>>> > >         happy to use
>>> > >         that :).
>>> > >
>>> > >         Regards,
>>> > >
>>> > >
>>> > >
>>> > > Is DAQ data acquisition? If so, what sort of data?
>>> > >
>>> > >
>>> > > Chuck
>>> > >
>>> > >
>>> > Which DAQ are you using?  There is 'comedi' for internal boards, but I
>>> > don't know of anything except the "ctype wrappers" for usb boxes.
>>> > When I started doing this several years ago, I wrote my own ctype
>>> > wrappers to access the dll for the Measurement Computing USB-1208 and
>>> > for the NI-USB6008.  Mine were very simple, and did only what I
>>> > needed.  There's a number of them on the web now, but I haven't looked
>>> > an any of them in detail.
>>> >
>>>
>>> Sorry, thought the DAQ was more clear then it probably is. What we have
>>> is a National instruments USB DAQ unit (M-Series). We are actually using
>>> it more for analog output then input right now.
>>>
>>> > Andrew Straw wrote a very nice object-oriented driver for the
>>> > Measurement Computing USB devices in Python.  However, I found
>>> > references to it in Cookbook/Data Acquisition, but I couldn't find the
>>> > actual PyUniversalLibrary.  Maybe it's no longer active.
>>> >
>>>
>>> I will have a look if I find it, that sounds interesting, I also was
>>> hoping that a wrapper could maybe help with multiprocessing magic to
>>> separate the gui as much as possible from the signal generation. Though
>>> maybe that is too high level to really do in any case.
>>> I have started playing with one of the ctypes wrappers and it indeed
>>> works very well. Rather annoying, because I initially wrote labview and
>>> now have to realize that the C-Api is identical and can easily be called
>>> from python, too :)...
>>>
>>> - Sebastian
>>>
>>> >
>>> > "Ctype wrappers" have worked very well for me.  We have one experiment
>>> > using two USB-1208 and one NI-USB-6008 simultaneously in a single
>>> > Python program with a Tkinter GUI, and it works fine.
>>> >
>>> > john
>>> >
>>> >
>>> > _______________________________________________
>>> > SciPy-User mailing list
>>> > SciPy-User at scipy.org
>>> > http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140428/ab2214c6/attachment.html>

From sgarcia at olfac.univ-lyon1.fr  Tue Apr 29 16:11:50 2014
From: sgarcia at olfac.univ-lyon1.fr (Samuel Garica)
Date: Tue, 29 Apr 2014 22:11:50 +0200
Subject: [SciPy-User] DAQmx python wrapper
In-Reply-To: <CAMMTP+DQaOx8Mo+4ubQrU4KdaeR5Hue8Qwf0EZY3UTmWYjsDkA@mail.gmail.com>
References: <1398354174.2782.8.camel@sebastian-t440>	<CAB6mnxLoja67mA0E0=yDEhA97hZwQGpSuBjEBik2o3jSL4Cnpg@mail.gmail.com>	<535BAED1.6000207@comcast.net>	<1398531195.7311.25.camel@sebastian-t440>	<CALGGngAkMnn-8LiPLCKs0uN=VB68zK731wRSLioFBvad8ggA8Q@mail.gmail.com>	<CALGGngBY=R=K3TcVFbZON8=nAjG+U_vftUWe71PpDbCqoyy2LA@mail.gmail.com>
	<CAMMTP+DQaOx8Mo+4ubQrU4KdaeR5Hue8Qwf0EZY3UTmWYjsDkA@mail.gmail.com>
Message-ID: <53600786.7060104@olfac.univ-lyon1.fr>

Hi,
I have wrote a project for acquisition in ptyhon. It is alpha level. And 
so, not released yet. But we use it everyday in my lab.
It is a simple layer on top of pycomedi (linux) or universal librabry 
via ctypes (on windows) and some other device.

It use zmq for dispatching buffer.
It also offer an osilloscope and timefrequency viewer in Qt.

Have a look if you want
https://github.com/samuelgarcia/pyacq


Best

Samuel


Le 28/04/2014 21:47, josef.pktd at gmail.com a ?crit :
>
>
>
> On Mon, Apr 28, 2014 at 3:39 PM, Gary Pajer <gary.pajer at gmail.com 
> <mailto:gary.pajer at gmail.com>> wrote:
>
>     Andrew Straw's PyUniversalLibrary appears to be on PyPi
>     <https://pypi.python.org/pypi/PyUniversalLibrary/>
>
>
> google search points here
> https://code.astraw.com/PyUniversalLibrary/
> http://code.astraw.com/PyUniversalLibrary-downloads/
>
> Josef
>
>     ~-~-~
>
>
>     On Mon, Apr 28, 2014 at 1:51 PM, Gary Pajer <gary.pajer at gmail.com
>     <mailto:gary.pajer at gmail.com>> wrote:
>
>         I've used Andrew's wrapper, and it worked well.
>
>         But in the end I wrote my own routines using ctypes (stealing
>         some ideas from Andrew).   I can't remember exactly why I
>         wrote my own... it was several years ago.  But I do remember
>         that it was really very easy to implement calls using ctypes.
>
>         ~-~-~
>
>
>         On Sat, Apr 26, 2014 at 12:53 PM, Sebastian Berg
>         <sebastian at sipsolutions.net
>         <mailto:sebastian at sipsolutions.net>> wrote:
>
>             On Sa, 2014-04-26 at 09:04 -0400, John Hassler wrote:
>             >
>             > On 4/24/2014 10:23 PM, Charles R Harris wrote:
>             >
>             > > Hi Sebastian,
>             > >
>             > >
>             > > On Thu, Apr 24, 2014 at 9:42 AM, Sebastian Berg
>             > > <sebastian at sipsolutions.net
>             <mailto:sebastian at sipsolutions.net>> wrote:
>             > >         Hey,
>             > >
>             > >         We are thinking about writing some DAQ
>             programs in python,
>             > >         and I was
>             > >         wondering if anyone can recommend a lib for
>             that. There seem
>             > >         to be a
>             > >         bunch of wrappers using ctypes, which seem all
>             good enough,
>             > >         but if there
>             > >         is something more highlevel or very well done,
>             I would be
>             > >         happy to use
>             > >         that :).
>             > >
>             > >         Regards,
>             > >
>             > >
>             > >
>             > > Is DAQ data acquisition? If so, what sort of data?
>             > >
>             > >
>             > > Chuck
>             > >
>             > >
>             > Which DAQ are you using?  There is 'comedi' for internal
>             boards, but I
>             > don't know of anything except the "ctype wrappers" for
>             usb boxes.
>             > When I started doing this several years ago, I wrote my
>             own ctype
>             > wrappers to access the dll for the Measurement Computing
>             USB-1208 and
>             > for the NI-USB6008.  Mine were very simple, and did only
>             what I
>             > needed.  There's a number of them on the web now, but I
>             haven't looked
>             > an any of them in detail.
>             >
>
>             Sorry, thought the DAQ was more clear then it probably is.
>             What we have
>             is a National instruments USB DAQ unit (M-Series). We are
>             actually using
>             it more for analog output then input right now.
>
>             > Andrew Straw wrote a very nice object-oriented driver
>             for the
>             > Measurement Computing USB devices in Python.  However, I
>             found
>             > references to it in Cookbook/Data Acquisition, but I
>             couldn't find the
>             > actual PyUniversalLibrary.  Maybe it's no longer active.
>             >
>
>             I will have a look if I find it, that sounds interesting,
>             I also was
>             hoping that a wrapper could maybe help with
>             multiprocessing magic to
>             separate the gui as much as possible from the signal
>             generation. Though
>             maybe that is too high level to really do in any case.
>             I have started playing with one of the ctypes wrappers and
>             it indeed
>             works very well. Rather annoying, because I initially
>             wrote labview and
>             now have to realize that the C-Api is identical and can
>             easily be called
>             from python, too :)...
>
>             - Sebastian
>
>             >
>             > "Ctype wrappers" have worked very well for me.  We have
>             one experiment
>             > using two USB-1208 and one NI-USB-6008 simultaneously in
>             a single
>             > Python program with a Tkinter GUI, and it works fine.
>             >
>             > john
>             >
>             >
>             > _______________________________________________
>             > SciPy-User mailing list
>             > SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
>             > http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>             _______________________________________________
>             SciPy-User mailing list
>             SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
>             http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>
>
>     _______________________________________________
>     SciPy-User mailing list
>     SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140429/d526f361/attachment.html>

From gb.gabrielebrambilla at gmail.com  Wed Apr 30 10:38:07 2014
From: gb.gabrielebrambilla at gmail.com (Gabriele Brambilla)
Date: Wed, 30 Apr 2014 10:38:07 -0400
Subject: [SciPy-User] creating a particular numpy array
Message-ID: <CABmgkicam1cbgzPr8sy96Q1=orEOacTd9mNiTy_dg1R6W0db_w@mail.gmail.com>

Hi,

I have a particular array

MYMAP = [np.zeros([npha, nobs, nex], dtype=float)]

how can I create an array of 210 of MYMAP?

thanks

Gabriele
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140430/6143b46c/attachment.html>

From gb.gabrielebrambilla at gmail.com  Wed Apr 30 11:10:47 2014
From: gb.gabrielebrambilla at gmail.com (Gabriele Brambilla)
Date: Wed, 30 Apr 2014 11:10:47 -0400
Subject: [SciPy-User] problems handling a numpy array
Message-ID: <CABmgkid_FdbpMeueC-vRw8N2utLaTBOVHH8-GMwVXnHP=ELVgQ@mail.gmail.com>

Hi,

I have a 4d numpy array.

I want to select an "index" of the fourth dimension to obtain a 3d array.

I try to di it in this way:

MYMAP = MYMAPS[:, :, :,  1]

that is defined in this way:
MYMAPS = [np.zeros([npha, nobs, nex, last-first+1], dtype=float)]

but I receive an error:
TypeError: list indices must be integers, not tuple

what am I doing wrong?

thanks

Gabriele
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140430/57ff8866/attachment.html>

From robert.kern at gmail.com  Wed Apr 30 11:13:04 2014
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 30 Apr 2014 16:13:04 +0100
Subject: [SciPy-User] problems handling a numpy array
In-Reply-To: <CABmgkid_FdbpMeueC-vRw8N2utLaTBOVHH8-GMwVXnHP=ELVgQ@mail.gmail.com>
References: <CABmgkid_FdbpMeueC-vRw8N2utLaTBOVHH8-GMwVXnHP=ELVgQ@mail.gmail.com>
Message-ID: <CAF6FJivRzhjv0bc3Qq78OMKDoB3CF_bTK_1z7ETb3rUKpWNyJg@mail.gmail.com>

On Wed, Apr 30, 2014 at 4:10 PM, Gabriele Brambilla
<gb.gabrielebrambilla at gmail.com> wrote:
> Hi,
>
> I have a 4d numpy array.
>
> I want to select an "index" of the fourth dimension to obtain a 3d array.
>
> I try to di it in this way:
>
> MYMAP = MYMAPS[:, :, :,  1]
>
> that is defined in this way:
> MYMAPS = [np.zeros([npha, nobs, nex, last-first+1], dtype=float)]
>
> but I receive an error:
> TypeError: list indices must be integers, not tuple
>
> what am I doing wrong?

You have made MYMAPS a list with an array inside it. Did you mean to
do this, instead?

  MYMAPS = np.zeros([npha, nobs, nex, last-first+1], dtype=float)

-- 
Robert Kern


From gb.gabrielebrambilla at gmail.com  Wed Apr 30 11:20:40 2014
From: gb.gabrielebrambilla at gmail.com (Gabriele Brambilla)
Date: Wed, 30 Apr 2014 11:20:40 -0400
Subject: [SciPy-User] problems handling a numpy array
In-Reply-To: <CAF6FJivRzhjv0bc3Qq78OMKDoB3CF_bTK_1z7ETb3rUKpWNyJg@mail.gmail.com>
References: <CABmgkid_FdbpMeueC-vRw8N2utLaTBOVHH8-GMwVXnHP=ELVgQ@mail.gmail.com>
	<CAF6FJivRzhjv0bc3Qq78OMKDoB3CF_bTK_1z7ETb3rUKpWNyJg@mail.gmail.com>
Message-ID: <CABmgkidSWPT9_VEM1_DEnhiqM=vMjCFKgwetgFZOFpf6a47_AA@mail.gmail.com>

No you are right,
I have to write it in this way:
MYMAPS = np.zeros([npha, nobs, nex, last-first+1], dtype=float)

thanks

Gabriele


2014-04-30 11:13 GMT-04:00 Robert Kern <robert.kern at gmail.com>:

> On Wed, Apr 30, 2014 at 4:10 PM, Gabriele Brambilla
> <gb.gabrielebrambilla at gmail.com> wrote:
> > Hi,
> >
> > I have a 4d numpy array.
> >
> > I want to select an "index" of the fourth dimension to obtain a 3d array.
> >
> > I try to di it in this way:
> >
> > MYMAP = MYMAPS[:, :, :,  1]
> >
> > that is defined in this way:
> > MYMAPS = [np.zeros([npha, nobs, nex, last-first+1], dtype=float)]
> >
> > but I receive an error:
> > TypeError: list indices must be integers, not tuple
> >
> > what am I doing wrong?
>
> You have made MYMAPS a list with an array inside it. Did you mean to
> do this, instead?
>
>   MYMAPS = np.zeros([npha, nobs, nex, last-first+1], dtype=float)
>
> --
> Robert Kern
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140430/ba9bfe98/attachment.html>

From evgeny.burovskiy at gmail.com  Wed Apr 30 11:28:34 2014
From: evgeny.burovskiy at gmail.com (Evgeni Burovski)
Date: Wed, 30 Apr 2014 16:28:34 +0100
Subject: [SciPy-User] creating a particular numpy array
In-Reply-To: <CABmgkicam1cbgzPr8sy96Q1=orEOacTd9mNiTy_dg1R6W0db_w@mail.gmail.com>
References: <CABmgkicam1cbgzPr8sy96Q1=orEOacTd9mNiTy_dg1R6W0db_w@mail.gmail.com>
Message-ID: <CAMRo0is0G3bVjQO3CUi1jLqiUmQRtPhb+PJJ-=5nLbHGR+vvAQ@mail.gmail.com>

The way you wrote it, you've a list of length one, with the array being its
only element:

MYMAP =[....]
On Apr 30, 2014 3:38 PM, "Gabriele Brambilla" <
gb.gabrielebrambilla at gmail.com> wrote:

> Hi,
>
> I have a particular array
>
> MYMAP = [np.zeros([npha, nobs, nex], dtype=float)]
>
> how can I create an array of 210 of MYMAP?
>
> thanks
>
> Gabriele
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140430/c9c20cc6/attachment.html>

From gb.gabrielebrambilla at gmail.com  Wed Apr 30 11:53:53 2014
From: gb.gabrielebrambilla at gmail.com (Gabriele Brambilla)
Date: Wed, 30 Apr 2014 11:53:53 -0400
Subject: [SciPy-User] creating a particular numpy array
In-Reply-To: <CAMRo0is0G3bVjQO3CUi1jLqiUmQRtPhb+PJJ-=5nLbHGR+vvAQ@mail.gmail.com>
References: <CABmgkicam1cbgzPr8sy96Q1=orEOacTd9mNiTy_dg1R6W0db_w@mail.gmail.com>
	<CAMRo0is0G3bVjQO3CUi1jLqiUmQRtPhb+PJJ-=5nLbHGR+vvAQ@mail.gmail.com>
Message-ID: <CABmgkida2ohNXu0O1=6ByJYDzVbPDJp5SZtMwG2g2jQ473qmng@mail.gmail.com>

Yes,
I have corrected it. sorry

MYMAP = np.zeros([npha, nobs, nex, 210], dtype=float)

now there is no problem.

thanks


2014-04-30 11:28 GMT-04:00 Evgeni Burovski <evgeny.burovskiy at gmail.com>:

> The way you wrote it, you've a list of length one, with the array being
> its only element:
>
> MYMAP =[....]
> On Apr 30, 2014 3:38 PM, "Gabriele Brambilla" <
> gb.gabrielebrambilla at gmail.com> wrote:
>
>> Hi,
>>
>> I have a particular array
>>
>> MYMAP = [np.zeros([npha, nobs, nex], dtype=float)]
>>
>> how can I create an array of 210 of MYMAP?
>>
>> thanks
>>
>> Gabriele
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140430/799b8597/attachment.html>

From guziy.sasha at gmail.com  Wed Apr 30 12:15:21 2014
From: guziy.sasha at gmail.com (Oleksandr Huziy)
Date: Wed, 30 Apr 2014 12:15:21 -0400
Subject: [SciPy-User] creating a particular numpy array
In-Reply-To: <CABmgkida2ohNXu0O1=6ByJYDzVbPDJp5SZtMwG2g2jQ473qmng@mail.gmail.com>
References: <CABmgkicam1cbgzPr8sy96Q1=orEOacTd9mNiTy_dg1R6W0db_w@mail.gmail.com>
	<CAMRo0is0G3bVjQO3CUi1jLqiUmQRtPhb+PJJ-=5nLbHGR+vvAQ@mail.gmail.com>
	<CABmgkida2ohNXu0O1=6ByJYDzVbPDJp5SZtMwG2g2jQ473qmng@mail.gmail.com>
Message-ID: <CAN3p1sV0H9v4b1QkOvKv6eeW8RX8Tg9fNJgmMrXdHj=nTOj6MQ@mail.gmail.com>

If you are OK with having the same obj for each array element:

x = np.asarray(210*[np.zeros([npha, nobs, nex], dtype=float)])

Cheers


2014-04-30 11:53 GMT-04:00 Gabriele Brambilla <
gb.gabrielebrambilla at gmail.com>:

> Yes,
> I have corrected it. sorry
>
> MYMAP = np.zeros([npha, nobs, nex, 210], dtype=float)
>
> now there is no problem.
>
> thanks
>
>
> 2014-04-30 11:28 GMT-04:00 Evgeni Burovski <evgeny.burovskiy at gmail.com>:
>
> The way you wrote it, you've a list of length one, with the array being
>> its only element:
>>
>> MYMAP =[....]
>> On Apr 30, 2014 3:38 PM, "Gabriele Brambilla" <
>> gb.gabrielebrambilla at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a particular array
>>>
>>> MYMAP = [np.zeros([npha, nobs, nex], dtype=float)]
>>>
>>> how can I create an array of 210 of MYMAP?
>>>
>>> thanks
>>>
>>> Gabriele
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Sasha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140430/5640d358/attachment.html>

From warren.weckesser at gmail.com  Wed Apr 30 15:19:23 2014
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Wed, 30 Apr 2014 15:19:23 -0400
Subject: [SciPy-User] Solving complex and array differential equations with
	odeint
Message-ID: <CAGzF1ufwCWYLzz4VTNWMU+m4+sE4iY2FUrE_=+p-M1wbB=70zg@mail.gmail.com>

Occasionally on the mailing list and stackoverflow, the question of how to
solve a system of differential equations in which the dependent variables
are complex or an array is raised.  `scipy.integrate.ode` (with the `zvode`
integrator) and `scipy.integrate.complex_ode` can handle complex equations,
but neither `ode` nor `scipy.integrate.odeint` can handle arrays.  To solve
an array differential equation with either of them, you have to implement
the equations with the array flattened to a 1-d vector.

I've created a wrapper of `odeint`, cleverly called `odeintw`, that takes
care of wrapping complex and array equations so they can be solved with
`odeint`.  The github repository is here:
https://github.com/WarrenWeckesser/odeintw

Eventually, I'll work on getting this into scipy itself, but these days the
distribution of "eventually" has a fat tail, so it might be a while.

Warren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140430/61697e0f/attachment.html>