From josef.pktd at gmail.com  Thu May  1 10:50:01 2014
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 1 May 2014 10:50:01 -0400
Subject: [SciPy-User] updating sparse matrix
Message-ID: <CAMMTP+A13Y2QaoeJw8Anny5OeTSc7zLt+5+DORAVuLw-Lp4D2g@mail.gmail.com>

I need to create a new sparse matrix inside a loop that has the same
sparsity structure as before, or essentially I just need to update the
numbers.

Is it allowed to directly assign new values to the .data attribute?
And is this the fastest way to update a sparse array?

>>> from scipy import sparse, linalg

>>> z0 = linalg.toeplitz([1,2,3])
>>> z1 = sparse.kron(sparse.eye(2), z0).tocsr()
>>> z1.toarray()
array([[ 1.,  2.,  3.,  0.,  0.,  0.],
       [ 2.,  1.,  2.,  0.,  0.,  0.],
       [ 3.,  2.,  1.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  1.,  2.,  3.],
       [ 0.,  0.,  0.,  2.,  1.,  2.],
       [ 0.,  0.,  0.,  3.,  2.,  1.]])
>>> z1.data
array([ 1.,  2.,  3.,  2.,  1.,  2.,  3.,  2.,  1.,  1.,  2.,  3.,  2.,
        1.,  2.,  3.,  2.,  1.])

update keeping same structure

>>> z1.data[:9] = np.arange(1, 10)
>>> z1.toarray()
array([[ 1.,  2.,  3.,  0.,  0.,  0.],
       [ 4.,  5.,  6.,  0.,  0.,  0.],
       [ 7.,  8.,  9.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  1.,  2.,  3.],
       [ 0.,  0.,  0.,  2.,  1.,  2.],
       [ 0.,  0.,  0.,  3.,  2.,  1.]])


>>> z1.data[:] = np.tile(np.arange(1, 10), 2)
>>> z1.toarray()
array([[ 1.,  2.,  3.,  0.,  0.,  0.],
       [ 4.,  5.,  6.,  0.,  0.,  0.],
       [ 7.,  8.,  9.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  1.,  2.,  3.],
       [ 0.,  0.,  0.,  4.,  5.,  6.],
       [ 0.,  0.,  0.,  7.,  8.,  9.]])

Josef


From josef.pktd at gmail.com  Thu May  1 11:00:21 2014
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Thu, 1 May 2014 11:00:21 -0400
Subject: [SciPy-User] question: OLS with sparse design and dense
	parameter
In-Reply-To: <CAFk72czM2AqSAYaLrRUwR1nT_dS8BsXM+jT=zwWhPu42y83YMQ@mail.gmail.com>
References: <CAMMTP+ADtToLDfC6VXSJHK9zXcyWipLW7LmEA=QjG9D7WW8SPw@mail.gmail.com>
	<CAPJVwBkJ4LQeU3KADuQiORZ7RHpcwLVtbf3p1_H2cF0TizxruQ@mail.gmail.com>
	<CAFk72czM2AqSAYaLrRUwR1nT_dS8BsXM+jT=zwWhPu42y83YMQ@mail.gmail.com>
Message-ID: <CAMMTP+Cc3Q1vgPuAzwYLq71N71PkXp-paBFHoqAv4jLqyR7amw@mail.gmail.com>

On Mon, Apr 28, 2014 at 2:45 AM, Severin Holzer-Graf
<sholzergr at gmail.com> wrote:
> I needed SuiteSparseQR for my work and added the backslash function of
> it to scikits.sparse.
> The modification is available here: https://github.com/duga3/scikits-sparse
>
> 2014-04-28 8:42 GMT+02:00 Nathaniel Smith <njs at pobox.com>:
>> I guess the classic solution for ols are sparse cholesky or sparse qr.
>> scikits.sparse has a cholmod wrapper, and could be modified to have a
>> suitesparseqr wrapper relatively easily.
>>
>> On 28 Apr 2014 03:31, <josef.pktd at gmail.com> wrote:
>>>
>>> (sunday night quest for another sparse recipe)
>>>
>>> This is a sparse algorithm question:
>>>
>>> scipy.sparse.linalg has a choice of linear system solvers, but I have not
>>> much idea about the relative merits. Is there a recommendation for the
>>> following case?
>>>
>>> suppose I want to do estimate a standard linear model by OLS
>>> y = x * b,  find b = argmin(y - x*b)
>>>
>>> x is sparse, with blocks, medium large (20000 by 2000) but if it works
>>> quite a bit larger,
>>> but we don't have a small number of dominant singular values, the solution
>>> of the linear system will be dense.
>>>
>>> the basic parts of my minimum example
>>>
>>> nobsi, n_groups, k_vars = 10, 2000, 3
>>>
>>> #nobsi, n_groups, k_vars = 2, 3, 3
>>>
>>>
>>> xi = np.arange(nobsi)[:,None]**np.arange(3) #vander
>>>
>>> x_sp = sparse.kron(sparse.eye(n_groups), xi).tocsr()
>>>
>>> (balanced kron structure is just to make a quick example)
>>>
>>>
>>> x_sp.shape
>>>
>>> (20000, 6000)
>>>
>>>
>>> first try
>>>
>>>
>>> xpx = x_sp.T.dot(x_sp) # sparse
>>>
>>> xpy = x_sp.T.dot(y) # dense
>>>
>>>
>>> p_dense = np.linalg.solve(xpx.toarray(), xpy)
>>>
>>> p_sp1 = sparse.linalg.spsolve(xpx, xpy)
>>>
>>> p_sp2 = sparse.linalg.lsmr(x_sp, y)[0]
>>>
>>>
>>> timing: 15.6659998894 0.00500011444092 0.00600004196167
>>>
>>>
>>> Is this too small to worry, and anything works, or is there a
>>> recommendation?
>>>
>>> definitely don't use dense.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Josef

Thanks for the answers,

I would like to stick to scipy for now.
I don't know if sparse usage in statsmodels will get heavy enough to
require additional optional dependencies.
At least not any time soon.

scipy.sparse has several solvers for linear systems and one of them
should work well enough for my purposes (traditional
statistics/econometrics instead of "Big Data"), for now.

Josef


>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From patrick.westphal at informatik.uni-leipzig.de  Fri May  2 04:56:34 2014
From: patrick.westphal at informatik.uni-leipzig.de (Patrick Westphal)
Date: Fri, 02 May 2014 10:56:34 +0200
Subject: [SciPy-User] Column extraction from line based text
Message-ID: <53635DC2.6050202@informatik.uni-leipzig.de>

Hello Scipy users,

I have to deal with messy line based text files that have certain
fields, e.g.

One Bowling Green, 3rd Floor New York, NY 10004
228 Park Ave S New York, NY 10003-1502
85 West Street at Albany Street  New York, NY 10006
13031 West Jefferson Boulevard, Suite 200 Los Angeles, CA 90094
550 S Flower St, Los Angeles, CA 90071

and I'd like to have these columns extracted

      | One Bowling Green, 3rd Floor        | New York    | NY | 10004
  228| Park Ave S                          | New York    | NY | 10003-1502
   85| West Street at Albany Street        | New York    | NY | 10006
13031 | West Jefferson Boulevard, Suite 200 | Los Angeles | CA | 90094
  550 | S Flower St                         | Los Angeles | CA | 90071

I searched quite a lot and found projects like Asciitable [0], but these
were not built to find homogeneous column structures by combining
multiple line tokens, as e.g. in 'One Bowling Green, 3rd Floor', but
rather try to generate one column per token.

My question is, if you are aware of any projects that better suit my
task, or if you could give me some hints where and what to search.

Thanks in advance,
Patrick

[0] http://cxc.cfa.harvard.edu/contrib/asciitable/


From athanastasiou at gmail.com  Fri May  2 05:24:41 2014
From: athanastasiou at gmail.com (Athanasios Anastasiou)
Date: Fri, 2 May 2014 10:24:41 +0100
Subject: [SciPy-User] Column extraction from line based text
In-Reply-To: <53635DC2.6050202@informatik.uni-leipzig.de>
References: <53635DC2.6050202@informatik.uni-leipzig.de>
Message-ID: <CAH9P_kR-qhTpO1R7=5cGa_5uF3DY85gKVP0PDmeXc9dsQo2MGg@mail.gmail.com>

Hello Patrick

In general, you can use "Regular Expressions" to extract your fields and
re-format them in any way you like.

I don't think that this is a "homogeneous" structure though. The postal
address seems to have been formatted following a specific template of

number (maybe character, maybe numeric), street name (could go up to 3
space separated tokens) coma (sometimes) city, postcode

Furthermore: street name -> street (any number of space separated tokens)
coma(always?) location (Floor, Suite, other? followed by a number (always?))

The city and postcode fields are the easiest to extract. For example, you
could establish a regular expression like ".*? [,]? ("Los Angeles"|"New
York"), [A-Z]{2}[0-9]{5}" which is roughly translated to "Match in a
non-greedy way everything up to an optional coma followed by something that
looks like a city name followed by a coma followed by a postcode".

You can extract such fields with FROMREGEXP (
http://docs.scipy.org/doc/numpy/reference/generated/numpy.fromregex.html)
or pyparsing (http://pyparsing.wikispaces.com/)

As you can see, some things have to be kept or assumed "fixed".

For example, you might have to run a quick check of how many cities you
have in your dataset or at least, how many distinct words you expect a city
to occupy to be able to describe roughly how does a city name (or other
field) looks like. The same applies when you try to fix the way that the
location is being described. Is it always Floor? Could it be for example
"block, Block, Floor, floor, Apartment, Apt, apt, apartment, room, Room,
RM, rm, ....". Could it be 1st Floor? Maybe Floor 1, Floor 2 and so on.

Hope this helps.

All the best.


On Fri, May 2, 2014 at 9:56 AM, Patrick Westphal <
patrick.westphal at informatik.uni-leipzig.de> wrote:

> Hello Scipy users,
>
> I have to deal with messy line based text files that have certain
> fields, e.g.
>
> One Bowling Green, 3rd Floor New York, NY 10004
> 228 Park Ave S New York, NY 10003-1502
> 85 West Street at Albany Street  New York, NY 10006
> 13031 West Jefferson Boulevard, Suite 200 Los Angeles, CA 90094
> 550 S Flower St, Los Angeles, CA 90071
>
> and I'd like to have these columns extracted
>
>       | One Bowling Green, 3rd Floor        | New York    | NY | 10004
>   228| Park Ave S                          | New York    | NY | 10003-1502
>    85| West Street at Albany Street        | New York    | NY | 10006
> 13031 | West Jefferson Boulevard, Suite 200 | Los Angeles | CA | 90094
>   550 | S Flower St                         | Los Angeles | CA | 90071
>
> I searched quite a lot and found projects like Asciitable [0], but these
> were not built to find homogeneous column structures by combining
> multiple line tokens, as e.g. in 'One Bowling Green, 3rd Floor', but
> rather try to generate one column per token.
>
> My question is, if you are aware of any projects that better suit my
> task, or if you could give me some hints where and what to search.
>
> Thanks in advance,
> Patrick
>
> [0] http://cxc.cfa.harvard.edu/contrib/asciitable/
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140502/4e51d0a1/attachment.html>

From patrick.westphal at informatik.uni-leipzig.de  Fri May  2 05:56:11 2014
From: patrick.westphal at informatik.uni-leipzig.de (Patrick Westphal)
Date: Fri, 02 May 2014 11:56:11 +0200
Subject: [SciPy-User] Column extraction from line based text
In-Reply-To: <CAH9P_kR-qhTpO1R7=5cGa_5uF3DY85gKVP0PDmeXc9dsQo2MGg@mail.gmail.com>
References: <53635DC2.6050202@informatik.uni-leipzig.de>
	<CAH9P_kR-qhTpO1R7=5cGa_5uF3DY85gKVP0PDmeXc9dsQo2MGg@mail.gmail.com>
Message-ID: <53636BBB.4020502@informatik.uni-leipzig.de>

Hi Athanasios,

thanks for your answer. Unfortunately, this was just an example and the
content is rather random. So fixed regexes won't help here. I thought of
a solution, that takes the field type, (maybe field lengths) and number
of fields into account to find homogeneities. In the given example, this
would mean that it is detected, that
- the line usually starts with a string that contains a number
- usually followed by some inhomogeneous text (e.g. West Street at
Albany Street  New York)
- usually followed by a two letter word (e.g. NY)
- and the line ends with a string, again containing numbers
which would at least suffice to generate some of the desired columns, e.g.

85 | West Street at Albany Street  New York | NY | 10006

But, I could try to generate regexes from the text lines and use
numpy.fromregex. I'll think about that.

Thanks!

On 02.05.2014 11:24, Athanasios Anastasiou wrote:
> Hello Patrick
>
> In general, you can use "Regular Expressions" to extract your fields
> and re-format them in any way you like.
>
> I don't think that this is a "homogeneous" structure though. The
> postal address seems to have been formatted following a specific
> template of
>
> number (maybe character, maybe numeric), street name (could go up to 3
> space separated tokens) coma (sometimes) city, postcode
>
> Furthermore: street name -> street (any number of space separated
> tokens) coma(always?) location (Floor, Suite, other? followed by a
> number (always?))
>
> The city and postcode fields are the easiest to extract. For example,
> you could establish a regular expression like ".*? [,]? ("Los
> Angeles"|"New York"), [A-Z]{2}[0-9]{5}" which is roughly translated to
> "Match in a non-greedy way everything up to an optional coma followed
> by something that looks like a city name followed by a coma followed
> by a postcode".
>
> You can extract such fields with FROMREGEXP
> (http://docs.scipy.org/doc/numpy/reference/generated/numpy.fromregex.html)
> or pyparsing (http://pyparsing.wikispaces.com/)
>
> As you can see, some things have to be kept or assumed "fixed".
>
> For example, you might have to run a quick check of how many cities
> you have in your dataset or at least, how many distinct words you
> expect a city to occupy to be able to describe roughly how does a city
> name (or other field) looks like. The same applies when you try to fix
> the way that the location is being described. Is it always Floor?
> Could it be for example "block, Block, Floor, floor, Apartment, Apt,
> apt, apartment, room, Room, RM, rm, ....". Could it be 1st Floor?
> Maybe Floor 1, Floor 2 and so on.
>
> Hope this helps.
>
> All the best.
>
>
> On Fri, May 2, 2014 at 9:56 AM, Patrick Westphal
> <patrick.westphal at informatik.uni-leipzig.de
> <mailto:patrick.westphal at informatik.uni-leipzig.de>> wrote:
>
>     Hello Scipy users,
>
>     I have to deal with messy line based text files that have certain
>     fields, e.g.
>
>     One Bowling Green, 3rd Floor New York, NY 10004
>     228 Park Ave S New York, NY 10003-1502
>     85 West Street at Albany Street  New York, NY 10006
>     13031 West Jefferson Boulevard, Suite 200 Los Angeles, CA 90094
>     550 S Flower St, Los Angeles, CA 90071
>
>     and I'd like to have these columns extracted
>
>           | One Bowling Green, 3rd Floor        | New York    | NY | 10004
>       228| Park Ave S                          | New York    | NY |
>     10003-1502
>        85| West Street at Albany Street        | New York    | NY | 10006
>     13031 | West Jefferson Boulevard, Suite 200 | Los Angeles | CA | 90094
>       550 | S Flower St                         | Los Angeles | CA | 90071
>
>     I searched quite a lot and found projects like Asciitable [0], but
>     these
>     were not built to find homogeneous column structures by combining
>     multiple line tokens, as e.g. in 'One Bowling Green, 3rd Floor', but
>     rather try to generate one column per token.
>
>     My question is, if you are aware of any projects that better suit my
>     task, or if you could give me some hints where and what to search.
>
>     Thanks in advance,
>     Patrick
>
>     [0] http://cxc.cfa.harvard.edu/contrib/asciitable/
>     _______________________________________________
>     SciPy-User mailing list
>     SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/scipy-user
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140502/f907de81/attachment.html>

From trive at astro.su.se  Fri May  2 06:52:42 2014
From: trive at astro.su.se (=?ISO-8859-1?Q?Th=F8ger_Emil_Rivera-Thorsen?=)
Date: Fri, 02 May 2014 12:52:42 +0200
Subject: [SciPy-User] Column extraction from line based text
In-Reply-To: <53636BBB.4020502@informatik.uni-leipzig.de>
References: <53635DC2.6050202@informatik.uni-leipzig.de>	<CAH9P_kR-qhTpO1R7=5cGa_5uF3DY85gKVP0PDmeXc9dsQo2MGg@mail.gmail.com>
	<53636BBB.4020502@informatik.uni-leipzig.de>
Message-ID: <536378FA.2000701@astro.su.se>

Maybe you could use the Pandas read_csv() function and give the '|'
character as delimiter; and then, for the offending columns, use the
'map()' method to split up the strings in the column and move them to a
separate column. This of course requires you to know which columns may
have offending content, and adress them individually.
Could be done like:

|import pandas as pd

table = pd.read_csv('filename.csv', 
                    delimiter='|', 
                    names=['number', 'street', 'city', 'state', 'zip'])

# Do something like the following for all columns with offending content
table['address2'] = table.street.map(lambda x: nan if len(x.split(',')) < 2 else x.split(',')[-1].strip())
table.street = table.street.map(lambda x: x.split(',')[0].strip())

# Now order columns to your liking:
table = table[['number', 'street', 'address2', 'city', 'state', 'zip']]

print table.to_string()
|

|   number                        street   address2           city state          zip
0                     One Bowling Green  3rd Floor   New York       NY         10004
1     228                    Park Ave S        NaN   New York       NY    10003-1502
2      85  West Street at Albany Street        NaN   New York       NY         10006
3  13031       West Jefferson Boulevard  Suite 200   Los Angeles    CA         90094
4    550                    S Flower St        NaN   Los Angeles    CA         90071
|

On 2014-05-02 11:56, Patrick Westphal wrote:

> Hi Athanasios,
>
> thanks for your answer. Unfortunately, this was just an example and
> the content is rather random. So fixed regexes won't help here. I
> thought of a solution, that takes the field type, (maybe field
> lengths) and number of fields into account to find homogeneities. In
> the given example, this would mean that it is detected, that
> - the line usually starts with a string that contains a number
> - usually followed by some inhomogeneous text (e.g. West Street at
> Albany Street  New York)
> - usually followed by a two letter word (e.g. NY)
> - and the line ends with a string, again containing numbers
> which would at least suffice to generate some of the desired columns, e.g.
>
> 85 | West Street at Albany Street  New York | NY | 10006
>
> But, I could try to generate regexes from the text lines and use
> numpy.fromregex. I'll think about that.
>
> Thanks!
>
> On 02.05.2014 11:24, Athanasios Anastasiou wrote:
>> Hello Patrick
>>
>> In general, you can use "Regular Expressions" to extract your fields
>> and re-format them in any way you like.
>>
>> I don't think that this is a "homogeneous" structure though. The
>> postal address seems to have been formatted following a specific
>> template of
>>
>> number (maybe character, maybe numeric), street name (could go up to
>> 3 space separated tokens) coma (sometimes) city, postcode
>>
>> Furthermore: street name -> street (any number of space separated
>> tokens) coma(always?) location (Floor, Suite, other? followed by a
>> number (always?))
>>
>> The city and postcode fields are the easiest to extract. For example,
>> you could establish a regular expression like ".*? [,]? ("Los
>> Angeles"|"New York"), [A-Z]{2}[0-9]{5}" which is roughly translated
>> to "Match in a non-greedy way everything up to an optional coma
>> followed by something that looks like a city name followed by a coma
>> followed by a postcode".
>>
>> You can extract such fields with FROMREGEXP
>> (http://docs.scipy.org/doc/numpy/reference/generated/numpy.fromregex.html)
>> or pyparsing (http://pyparsing.wikispaces.com/)
>>
>> As you can see, some things have to be kept or assumed "fixed".
>>
>> For example, you might have to run a quick check of how many cities
>> you have in your dataset or at least, how many distinct words you
>> expect a city to occupy to be able to describe roughly how does a
>> city name (or other field) looks like. The same applies when you try
>> to fix the way that the location is being described. Is it always
>> Floor? Could it be for example "block, Block, Floor, floor,
>> Apartment, Apt, apt, apartment, room, Room, RM, rm, ....". Could it
>> be 1st Floor? Maybe Floor 1, Floor 2 and so on.
>>
>> Hope this helps.
>>
>> All the best.
>>
>>
>> On Fri, May 2, 2014 at 9:56 AM, Patrick Westphal
>> <patrick.westphal at informatik.uni-leipzig.de
>> <mailto:patrick.westphal at informatik.uni-leipzig.de>> wrote:
>>
>>     Hello Scipy users,
>>
>>     I have to deal with messy line based text files that have certain
>>     fields, e.g.
>>
>>     One Bowling Green, 3rd Floor New York, NY 10004
>>     228 Park Ave S New York, NY 10003-1502
>>     85 West Street at Albany Street  New York, NY 10006
>>     13031 West Jefferson Boulevard, Suite 200 Los Angeles, CA 90094
>>     550 S Flower St, Los Angeles, CA 90071
>>
>>     and I'd like to have these columns extracted
>>
>>           | One Bowling Green, 3rd Floor        | New York    | NY |
>>     10004
>>       228| Park Ave S                          | New York    | NY |
>>     10003-1502
>>        85| West Street at Albany Street        | New York    | NY | 10006
>>     13031 | West Jefferson Boulevard, Suite 200 | Los Angeles | CA |
>>     90094
>>       550 | S Flower St                         | Los Angeles | CA |
>>     90071
>>
>>     I searched quite a lot and found projects like Asciitable [0],
>>     but these
>>     were not built to find homogeneous column structures by combining
>>     multiple line tokens, as e.g. in 'One Bowling Green, 3rd Floor', but
>>     rather try to generate one column per token.
>>
>>     My question is, if you are aware of any projects that better suit my
>>     task, or if you could give me some hints where and what to search.
>>
>>     Thanks in advance,
>>     Patrick
>>
>>     [0] http://cxc.cfa.harvard.edu/contrib/asciitable/
>>     _______________________________________________
>>     SciPy-User mailing list
>>     SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
>>     http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-- 
--------------------------
Th?ger Emil Rivera-Thorsen
Ph.D. student
Stockholm University, 
Department of Astronomy
--------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140502/66ae22a6/attachment.html>

From sunk.cs at gmail.com  Sat May  3 07:02:36 2014
From: sunk.cs at gmail.com (K. Sun)
Date: Sat, 3 May 2014 13:02:36 +0200
Subject: [SciPy-User] question about numerical integration
Message-ID: <20140503110236.GA7881@probe.unige.ch>

Dear scipy community,

I am using the following code to compute a 2D integration
\int_0^1 \inf_0^1 f(x,y) dx dy.

def opts0(*args, **kwargs):
    return { 'epsabs':0, 'epsrel':1e-2, 'limit':10 }

print scipy.integrate.nquad( f, \
            [(0, 1), (0, 1)], \
            opts=(opts0,opts0) )

The problem is that f(x,y) is a very expensive function, cost
a lot of time to compute. This makes the integration very slow.
According to my study, f(x,y) gets called >= 100,000 times.

My question is: is there any way to control the number of 
calls to f(x,y), e.g. <= 10000. I only need a rough estimation
and can tolerant +-10% error.

Thanks,
K.


From davidmenhur at gmail.com  Sat May  3 08:31:05 2014
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Sat, 3 May 2014 14:31:05 +0200
Subject: [SciPy-User] question about numerical integration
In-Reply-To: <20140503110236.GA7881@probe.unige.ch>
References: <20140503110236.GA7881@probe.unige.ch>
Message-ID: <CAJhcF=14kxruZ5CV1rh2XDojURDaESAhAXOM9GzNrE1mN9Nj1w@mail.gmail.com>

On 3 May 2014 13:02, K. Sun <sunk.cs at gmail.com> wrote:

> My question is: is there any way to control the number of
> calls to f(x,y), e.g. <= 10000. I only need a rough estimation
> and can tolerant +-10% error.


That is a good case for Monte Carlo integration. The error goes as
1/sqrt(N) independently of the number of dimensions, so you will be there
with a few hundred evaluations. In the classic example to compute pi:

In [3]: x = np.random.random(100)
In [4]: y = np.random.random(100)
In [5]: np.sum(x*x + y*y < 1)
Out[5]: 78
In [6]: 4.0 * np.sum(x*x + y*y < 1)/100.
Out[6]: 3.12

where the function is x^2 + y^2.

/David.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140503/fd8e8ce8/attachment.html>

From garyr at fidalgo.net  Sat May  3 10:17:26 2014
From: garyr at fidalgo.net (garyr)
Date: Sat, 3 May 2014 07:17:26 -0700
Subject: [SciPy-User] Matplotlib books
Message-ID: <4B29845F89234CC2AB10F6F3472AFBF8@owner59bf8d40c>

Amazon lists two books on Matplotlib, are there others? Which would you 
recommend?




From ralf.gommers at gmail.com  Sun May  4 04:15:52 2014
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 4 May 2014 10:15:52 +0200
Subject: [SciPy-User] ANN: Scipy 0.14.0 release
Message-ID: <CABL7CQh+tRroV21EbVNUO7+-ceACT1bhzdO_isZvDDWHejyr_w@mail.gmail.com>

Hi,

On behalf of the Scipy development team I'm pleased to announce the
availability of Scipy 0.14.0. This release contains new features (see
release notes below) and 8 months worth of maintenance work. 80 people
contributed to this release.

This is also the first release for which binary wheels are available on
PyPi for OS X, supporting the python.org Python. Wheels for Windows are
still being worked on, those may follow at a later date.

This release requires Python 2.6, 2.7 or 3.2-3.4 and NumPy 1.5.1 or
greater. Sources and binaries can be found at
https://sourceforge.net/projects/scipy/files/scipy/0.14.0/.

Enjoy,
Ralf


==========================
SciPy 0.14.0 Release Notes
==========================

.. contents::

SciPy 0.14.0 is the culmination of 8 months of hard work. It contains
many new features, numerous bug-fixes, improved test coverage and
better documentation.  There have been a number of deprecations and
API changes in this release, which are documented below.  All users
are encouraged to upgrade to this release, as there are a large number
of bug-fixes and optimizations.  Moreover, our development attention
will now shift to bug-fix releases on the 0.14.x branch, and on adding
new features on the master branch.

This release requires Python 2.6, 2.7 or 3.2-3.4 and NumPy 1.5.1 or greater.


New features
============

``scipy.interpolate`` improvements
----------------------------------

A new wrapper function `scipy.interpolate.interpn` for interpolation on
regular
grids has been added. `interpn` supports linear and nearest-neighbor
interpolation in arbitrary dimensions and spline interpolation in two
dimensions.

Faster implementations of piecewise polynomials in power and Bernstein
polynomial bases have been added as `scipy.interpolate.PPoly` and
`scipy.interpolate.BPoly`. New users should use these in favor of
`scipy.interpolate.PiecewisePolynomial`.

`scipy.interpolate.interp1d` now accepts non-monotonic inputs and sorts
them.
If performance is critical, sorting can be turned off by using the new
``assume_sorted`` keyword.

Functionality for evaluation of bivariate spline derivatives in
``scipy.interpolate`` has been added.

The new class `scipy.interpolate.Akima1DInterpolator` implements the
piecewise
cubic polynomial interpolation scheme devised by H. Akima.

Functionality for fast interpolation on regular, unevenly spaced grids
in arbitrary dimensions has been added as
`scipy.interpolate.RegularGridInterpolator` .


``scipy.linalg`` improvements
-----------------------------

The new function `scipy.linalg.dft` computes the matrix of the
discrete Fourier transform.

A condition number estimation function for matrix exponential,
`scipy.linalg.expm_cond`, has been added.


``scipy.optimize`` improvements
-------------------------------

A set of benchmarks for optimize, which can be run with
``optimize.bench()``,
has been added.

`scipy.optimize.curve_fit` now has more controllable error estimation via
the
``absolute_sigma`` keyword.

Support for passing custom minimization methods to ``optimize.minimize()``
and ``optimize.minimize_scalar()`` has been added, currently useful
especially
for combining ``optimize.basinhopping()`` with custom local optimizer
routines.


``scipy.stats`` improvements
----------------------------

A new class `scipy.stats.multivariate_normal` with functionality for
multivariate normal random variables has been added.

A lot of work on the ``scipy.stats`` distribution framework has been done.
Moment calculations (skew and kurtosis mainly) are fixed and verified, all
examples are now runnable, and many small accuracy and performance
improvements
for individual distributions were merged.

The new function `scipy.stats.anderson_ksamp` computes the k-sample
Anderson-Darling test for the null hypothesis that k samples come from
the same parent population.


``scipy.signal`` improvements
-----------------------------

``scipy.signal.iirfilter`` and related functions to design Butterworth,
Chebyshev, elliptical and Bessel IIR filters now all use pole-zero ("zpk")
format internally instead of using transformations to numerator/denominator
format.  The accuracy of the produced filters, especially high-order ones,
is
improved significantly as a result.

The new function `scipy.signal.vectorstrength` computes the vector strength,
a measure of phase synchrony, of a set of events.


``scipy.special`` improvements
------------------------------

The functions `scipy.special.boxcox` and `scipy.special.boxcox1p`, which
compute the Box-Cox transformation, have been added.


``scipy.sparse`` improvements
-----------------------------

- Significant performance improvement in CSR, CSC, and DOK indexing speed.
- When using Numpy >= 1.9 (to be released in MM 2014), sparse matrices
function
  correctly when given to arguments of ``np.dot``, ``np.multiply`` and other
  ufuncs.  With earlier Numpy and Scipy versions, the results of such
  operations are undefined and usually unexpected.
- Sparse matrices are no longer limited to ``2^31`` nonzero elements.  They
  automatically switch to using 64-bit index data type for matrices
containing
  more elements.  User code written assuming the sparse matrices use int32
as
  the index data type will continue to work, except for such large matrices.
  Code dealing with larger matrices needs to accept either int32 or int64
  indices.


Deprecated features
===================

``anneal``
----------

The global minimization function `scipy.optimize.anneal` is deprecated.
All users should use the `scipy.optimize.basinhopping` function instead.

``scipy.stats``
---------------

``randwcdf`` and ``randwppf`` functions are deprecated. All users should use
distribution-specific ``rvs`` methods instead.

Probability calculation aliases ``zprob``, ``fprob`` and ``ksprob`` are
deprecated. Use instead the ``sf`` methods of the corresponding
distributions
or the ``special`` functions directly.

``scipy.interpolate``
---------------------

``PiecewisePolynomial`` class is deprecated.


Backwards incompatible changes
==============================

scipy.special.lpmn
------------------

``lpmn`` no longer accepts complex-valued arguments. A new function
``clpmn`` with uniform complex analytic behavior has been added, and
it should be used instead.

scipy.sparse.linalg
-------------------

Eigenvectors in the case of generalized eigenvalue problem are normalized to
unit vectors in 2-norm, rather than following the LAPACK normalization
convention.

The deprecated UMFPACK wrapper in ``scipy.sparse.linalg`` has been removed
due
to license and install issues.  If available, ``scikits.umfpack`` is still
used
transparently in the ``spsolve`` and ``factorized`` functions.  Otherwise,
SuperLU is used instead in these functions.

scipy.stats
-----------

The deprecated functions ``glm``, ``oneway`` and ``cmedian`` have been
removed
from ``scipy.stats``.

``stats.scoreatpercentile`` now returns an array instead of a list of
percentiles.

scipy.interpolate
-----------------

The API for computing derivatives of a monotone piecewise interpolation has
changed: if `p` is a ``PchipInterpolator`` object, `p.derivative(der)`
returns a callable object representing the derivative of `p`. For in-place
derivatives use the second argument of the `__call__` method:
`p(0.1, der=2)` evaluates the second derivative of `p` at `x=0.1`.

The method `p.derivatives` has been removed.


Other changes
=============


Authors
=======

* Marc Abramowitz +
* Anders Bech Borchersen +
* Vincent Arel-Bundock +
* Petr Baudis +
* Max Bolingbroke
* Fran?ois Boulogne
* Matthew Brett
* Lars Buitinck
* Evgeni Burovski
* CJ Carey +
* Thomas A Caswell +
* Pawel Chojnacki +
* Phillip Cloud +
* Stefano Costa +
* David Cournapeau
* David Menendez Hurtado +
* Matthieu Dartiailh +
* Christoph Deil +
* J?rg Dietrich +
* endolith
* Francisco de la Pe?a +
* Ben FrantzDale +
* Jim Garrison +
* Andr? Gaul
* Christoph Gohlke
* Ralf Gommers
* Robert David Grant
* Alex Griffing
* Blake Griffith
* Yaroslav Halchenko
* Andreas Hilboll
* Kat Huang
* Gert-Ludwig Ingold
* James T. Webber +
* Dorota Jarecka +
* Todd Jennings +
* Thouis (Ray) Jones
* Juan Luis Cano Rodr?guez
* ktritz +
* Jacques Kvam +
* Eric Larson +
* Justin Lavoie +
* Denis Laxalde
* Jussi Leinonen +
* lemonlaug +
* Tim Leslie
* Alain Leufroy +
* George Lewis +
* Max Linke +
* Brandon Liu +
* Benny Malengier +
* Matthias K?mmerer +
* Cimarron Mittelsteadt +
* Eric Moore
* Andrew Nelson +
* Niklas Hamb?chen +
* Joel Nothman +
* Clemens Novak
* Emanuele Olivetti +
* Stefan Otte +
* peb +
* Josef Perktold
* pjwerneck
* poolio
* J?r?me Roy +
* Carl Sandrock +
* Andrew Sczesnak +
* Shauna +
* Fabrice Silva
* Daniel B. Smith
* Patrick Snape +
* Thomas Spura +
* Jacob Stevenson
* Julian Taylor
* Tomas Tomecek
* Richard Tsai
* Jacob Vanderplas
* Joris Vankerschaver +
* Pauli Virtanen
* Warren Weckesser

A total of 80 people contributed to this release.
People with a "+" by their names contributed a patch for the first time.
This list of names is automatically generated, and may not be fully
complete.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140504/09c5dafd/attachment.html>

From wenlei.xie at gmail.com  Mon May  5 19:35:37 2014
From: wenlei.xie at gmail.com (Wenlei Xie)
Date: Mon, 5 May 2014 19:35:37 -0400
Subject: [SciPy-User] SpMV operation performance
Message-ID: <CABTg2xpYTHe0nstHtAbxRS7BuY5VeU=gpV35vC22hyO9WOEjog@mail.gmail.com>

Hi all,

I got very strange behavior of SpMV operations on SciPy. The operation is A
* b, where A is a 160 * 3494258  sparse matrix with 134K non-zero entries,
b is a 3494258 * 100 dense matrix. I tried to use a generated b and random
b, which got very different performance:

PROFILE: A.dot(b1) finished, times taken 3.31 sec.
PROFILE: A.dot(b2) finished, times taken 0.06 sec.

Here is the microbenchmark code:

A = scipy.sparse.rand(160, 3494258, density = 0.00024, format = 'csr')
print A.nnz

b1 = np.load('xxx.npy')
b2 = numpy.random.random(b1.shape)

print b1.shape
print b1
print b2

tic = time.time()
c = A.dot(b1)
toc = time.time()
print >> sys.stdout,  "PROFILE: A.dot(b1) finished, times taken %.2lf sec."
% (toc - tic)

tic = time.time()
c = A.dot(b2)
toc = time.time()
print >> sys.stdout,  "PROFILE: A.dot(b2) finished, times taken %.2lf sec."
% (toc - tic)

I am wondering if there are any reason that A * b1 is so slow? (It's just
about 13M operations)

Here is part of b1:
[[ -3.92341357e-05   1.01987226e-04   4.21401120e-04 ...,   4.32603023e-04
    2.27787136e-04  -1.49943074e-04]
 [ -3.46766744e-05   9.18044044e-05   3.15756595e-04 ...,  -8.95272393e-05
   -1.67456582e-05   3.47852846e-06]
 [ -3.17974655e-05   8.45500240e-05   2.66199806e-04 ...,   1.30058938e-04
    4.92882776e-05   4.25645134e-05]
 ...,
 [ -1.05413991e-05  -5.47764556e-06  -4.46550403e-07 ...,  -1.17936250e-05
    3.81992402e-05   2.52261317e-05]
 [ -1.00702731e-05  -5.34535785e-06  -1.23229510e-06 ...,   1.58569376e-07
    3.11790336e-05  -3.28347371e-05]
 [ -2.03765705e-03  -1.08504060e-03  -2.57745529e-04 ...,  -7.23532559e-04
    1.09431507e-03  -8.37872641e-04]]


Part of b2:
[[ 0.1774481   0.18137227  0.31403885 ...,  0.39728013  0.55467338
   0.36584007]
 [ 0.67578084  0.38609356  0.78373177 ...,  0.48574942  0.83005304
   0.6256593 ]
 [ 0.19390028  0.36458137  0.99752705 ...,  0.73648617  0.3931888
   0.75647987]
 ...,
 [ 0.243743    0.3885193   0.32879056 ...,  0.18071315  0.19830516
   0.39236967]
 [ 0.49944973  0.81659732  0.10394949 ...,  0.72219989  0.35894712
   0.61801771]
 [ 0.14000652  0.46018156  0.44108854 ...,  0.23251014  0.45843828
   0.02769163]]


Thank you so much!

Best,
Wenlei
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140505/3b5780ae/attachment.html>

From aronne.merrelli at gmail.com  Tue May  6 10:12:21 2014
From: aronne.merrelli at gmail.com (Aronne Merrelli)
Date: Tue, 6 May 2014 09:12:21 -0500
Subject: [SciPy-User] SpMV operation performance
In-Reply-To: <CABTg2xpYTHe0nstHtAbxRS7BuY5VeU=gpV35vC22hyO9WOEjog@mail.gmail.com>
References: <CABTg2xpYTHe0nstHtAbxRS7BuY5VeU=gpV35vC22hyO9WOEjog@mail.gmail.com>
Message-ID: <CAHNdQ4JuhM4_PN3VEe9KtZa4F84k+4DZyPv-zZA+0hVj0Tq2oQ@mail.gmail.com>

On Mon, May 5, 2014 at 6:35 PM, Wenlei Xie <wenlei.xie at gmail.com> wrote:

> Hi all,
>
> I got very strange behavior of SpMV operations on SciPy. The operation is
> A * b, where A is a 160 * 3494258  sparse matrix with 134K non-zero
> entries, b is a 3494258 * 100 dense matrix. I tried to use a generated b
> and random b, which got very different performance:
>
> PROFILE: A.dot(b1) finished, times taken 3.31 sec.
> PROFILE: A.dot(b2) finished, times taken 0.06 sec.
>
> Here is the microbenchmark code:
>
> A = scipy.sparse.rand(160, 3494258, density = 0.00024, format = 'csr')
> print A.nnz
>
> b1 = np.load('xxx.npy')
> b2 = numpy.random.random(b1.shape)
>
> print b1.shape
> print b1
> print b2
>
> <snip>
>

What is the dtype of b1? When you create b2 with np.random.random(), you
make a new array with the same shape, of dtype np.float64, regardless of
the dtype of b1. I can get a similar slowdown if b1 is float32. My guess is
that if the dtype does not match, the optimized BLAS code is not being used
for the dot product, or maybe another temporary array has be be created.

For example:

A = scipy.sparse.rand(160, 3494258, density = 0.00024, format = 'csr')
b1 = np.zeros( (3494258, 100), dtype=np.float32)
b2 = numpy.random.random(b1.shape)

t0=time.time(); Ab1 = A.dot(b1); print time.time()-t0
1.4849190712

t0=time.time(); Ab2 = A.dot(b2); print time.time()-t0
0.0752549171448


Hope that helps,
Aronne
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140506/c43b01bd/attachment.html>

From pav at iki.fi  Tue May  6 13:06:00 2014
From: pav at iki.fi (Pauli Virtanen)
Date: Tue, 06 May 2014 20:06:00 +0300
Subject: [SciPy-User] SpMV operation performance
In-Reply-To: <CABTg2xpYTHe0nstHtAbxRS7BuY5VeU=gpV35vC22hyO9WOEjog@mail.gmail.com>
References: <CABTg2xpYTHe0nstHtAbxRS7BuY5VeU=gpV35vC22hyO9WOEjog@mail.gmail.com>
Message-ID: <lkb4po$gpr$1@ger.gmane.org>

06.05.2014 02:35, Wenlei Xie kirjoitti:
> I got very strange behavior of SpMV operations on SciPy. The operation is A
> * b, where A is a 160 * 3494258  sparse matrix with 134K non-zero entries,
> b is a 3494258 * 100 dense matrix. I tried to use a generated b and random
> b, which got very different performance:
> 
> PROFILE: A.dot(b1) finished, times taken 3.31 sec.
> PROFILE: A.dot(b2) finished, times taken 0.06 sec.
[clip]

Some possibilities that may slow down floating point math:

- denormal numbers (|a| < ~1e-300 and |a| > ~1e300)
- NaNs

The first can in principle be controlled by compiler flags used when
compiling Scipy.

-- 
Pauli Virtanen



From wenlei.xie at gmail.com  Tue May  6 23:01:30 2014
From: wenlei.xie at gmail.com (Wenlei Xie)
Date: Tue, 6 May 2014 23:01:30 -0400
Subject: [SciPy-User] SpMV operation performance
In-Reply-To: <lkb4po$gpr$1@ger.gmane.org>
References: <CABTg2xpYTHe0nstHtAbxRS7BuY5VeU=gpV35vC22hyO9WOEjog@mail.gmail.com>
	<lkb4po$gpr$1@ger.gmane.org>
Message-ID: <CABTg2xq=NuQT7cgWUsDT033e6AD+jWPJgMSRhDO7fV+Rbq+opg@mail.gmail.com>

Thank you guys! However it looks like none of these holds:

In [50]: np.isnan(V).any()
Out[50]: False

In [51]: (abs(V) < 1e-100).any()
Out[51]: False

In [52]: (abs(V) > 1e+100).any()
Out[52]: False

In [53]: V.dtype
Out[53]: dtype('float64')


I think it's probably the matrix contains some strange values that slowed
things down. But I don't have a clear clue (it is generated by scipy.svd).

Can any one give me a reference to the implantation to the .dot() for
sparse matrices?

Another interesting observation is that if I .dot() each column of the
matrix V, the running time is much shorter than .dot() all of them:

In [97]: time T = A.dot(V[:, 0])
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s

In [98]: time T = A.dot(V[:, 1])
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s

In [99]: time T = A.dot(V[:, 0:2])
CPU times: user 0.06 s, sys: 0.01 s, total: 0.07 s
Wall time: 0.07 s

In [100]: time T = A.dot(V)
CPU times: user 3.38 s, sys: 0.52 s, total: 3.90 s
Wall time: 3.91 s

(And, most of the time seems to be spent on user-mode?)


Thank you!

Best,
Wenlei



On Tue, May 6, 2014 at 1:06 PM, Pauli Virtanen <pav at iki.fi> wrote:

> 06.05.2014 02:35, Wenlei Xie kirjoitti:
> > I got very strange behavior of SpMV operations on SciPy. The operation
> is A
> > * b, where A is a 160 * 3494258  sparse matrix with 134K non-zero
> entries,
> > b is a 3494258 * 100 dense matrix. I tried to use a generated b and
> random
> > b, which got very different performance:
> >
> > PROFILE: A.dot(b1) finished, times taken 3.31 sec.
> > PROFILE: A.dot(b2) finished, times taken 0.06 sec.
> [clip]
>
> Some possibilities that may slow down floating point math:
>
> - denormal numbers (|a| < ~1e-300 and |a| > ~1e300)
> - NaNs
>
> The first can in principle be controlled by compiler flags used when
> compiling Scipy.
>
> --
> Pauli Virtanen
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Wenlei Xie (???)

Department of Computer Science
5132 Upson Hall, Cornell University
Ithaca, NY 14853, USA
Phone: (607) 255-5577
Email: wenlei.xie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140506/40bd0f2c/attachment.html>

From wenlei.xie at gmail.com  Tue May  6 23:36:11 2014
From: wenlei.xie at gmail.com (Wenlei Xie)
Date: Tue, 6 May 2014 23:36:11 -0400
Subject: [SciPy-User] SpMV operation performance
In-Reply-To: <CABTg2xq=NuQT7cgWUsDT033e6AD+jWPJgMSRhDO7fV+Rbq+opg@mail.gmail.com>
References: <CABTg2xpYTHe0nstHtAbxRS7BuY5VeU=gpV35vC22hyO9WOEjog@mail.gmail.com>
	<lkb4po$gpr$1@ger.gmane.org>
	<CABTg2xq=NuQT7cgWUsDT033e6AD+jWPJgMSRhDO7fV+Rbq+opg@mail.gmail.com>
Message-ID: <CABTg2xqzCMwoLXJYUnDwcbRtQ5b4kOXdRg_H8am9OjuQ_-_+SA@mail.gmail.com>

Hi all,

It looks like the problem is the organization of the matrix read from the
disk. After flatten and reshape it, problem got solved.

In [135]: VV = V.flatten().reshape(3494258,100)

In [136]: time T = A.dot(VV)
CPU times: user 0.08 s, sys: 0.00 s, total: 0.08 s
Wall time: 0.08 s

I suspect there has something related to how the data is
internally represented.

Thank you!

Best Regards,
Wenlei




On Tue, May 6, 2014 at 11:01 PM, Wenlei Xie <wenlei.xie at gmail.com> wrote:

> Thank you guys! However it looks like none of these holds:
>
> In [50]: np.isnan(V).any()
> Out[50]: False
>
> In [51]: (abs(V) < 1e-100).any()
> Out[51]: False
>
> In [52]: (abs(V) > 1e+100).any()
> Out[52]: False
>
> In [53]: V.dtype
> Out[53]: dtype('float64')
>
>
> I think it's probably the matrix contains some strange values that slowed
> things down. But I don't have a clear clue (it is generated by scipy.svd).
>
> Can any one give me a reference to the implantation to the .dot() for
> sparse matrices?
>
> Another interesting observation is that if I .dot() each column of the
> matrix V, the running time is much shorter than .dot() all of them:
>
> In [97]: time T = A.dot(V[:, 0])
> CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
> Wall time: 0.00 s
>
> In [98]: time T = A.dot(V[:, 1])
> CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
> Wall time: 0.00 s
>
> In [99]: time T = A.dot(V[:, 0:2])
> CPU times: user 0.06 s, sys: 0.01 s, total: 0.07 s
> Wall time: 0.07 s
>
> In [100]: time T = A.dot(V)
> CPU times: user 3.38 s, sys: 0.52 s, total: 3.90 s
> Wall time: 3.91 s
>
> (And, most of the time seems to be spent on user-mode?)
>
>
> Thank you!
>
> Best,
> Wenlei
>
>
>
> On Tue, May 6, 2014 at 1:06 PM, Pauli Virtanen <pav at iki.fi> wrote:
>
>> 06.05.2014 02:35, Wenlei Xie kirjoitti:
>> > I got very strange behavior of SpMV operations on SciPy. The operation
>> is A
>> > * b, where A is a 160 * 3494258  sparse matrix with 134K non-zero
>> entries,
>> > b is a 3494258 * 100 dense matrix. I tried to use a generated b and
>> random
>> > b, which got very different performance:
>> >
>> > PROFILE: A.dot(b1) finished, times taken 3.31 sec.
>> > PROFILE: A.dot(b2) finished, times taken 0.06 sec.
>> [clip]
>>
>> Some possibilities that may slow down floating point math:
>>
>> - denormal numbers (|a| < ~1e-300 and |a| > ~1e300)
>> - NaNs
>>
>> The first can in principle be controlled by compiler flags used when
>> compiling Scipy.
>>
>> --
>> Pauli Virtanen
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>
>
>
> --
> Wenlei Xie (???)
>
> Department of Computer Science
> 5132 Upson Hall, Cornell University
> Ithaca, NY 14853, USA
> Phone: (607) 255-5577
> Email: wenlei.xie at gmail.com
>



-- 
Wenlei Xie (???)

Department of Computer Science
5132 Upson Hall, Cornell University
Ithaca, NY 14853, USA
Phone: (607) 255-5577
Email: wenlei.xie at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140506/e40f3c14/attachment.html>

From sebastian at sipsolutions.net  Wed May  7 02:56:51 2014
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Wed, 07 May 2014 08:56:51 +0200
Subject: [SciPy-User] SpMV operation performance
In-Reply-To: <CABTg2xqzCMwoLXJYUnDwcbRtQ5b4kOXdRg_H8am9OjuQ_-_+SA@mail.gmail.com>
References: <CABTg2xpYTHe0nstHtAbxRS7BuY5VeU=gpV35vC22hyO9WOEjog@mail.gmail.com>
	<lkb4po$gpr$1@ger.gmane.org>
	<CABTg2xq=NuQT7cgWUsDT033e6AD+jWPJgMSRhDO7fV+Rbq+opg@mail.gmail.com>
	<CABTg2xqzCMwoLXJYUnDwcbRtQ5b4kOXdRg_H8am9OjuQ_-_+SA@mail.gmail.com>
Message-ID: <1399445811.8776.2.camel@sebastian-t440>

On Di, 2014-05-06 at 23:36 -0400, Wenlei Xie wrote:
> Hi all,
> 
> 
> It looks like the problem is the organization of the matrix read from
> the disk. After flatten and reshape it, problem got solved.
> 

In that case, using numpy 1.8 or later is likely to fix it as well.

- Sebastian

> 
> In [135]: VV = V.flatten().reshape(3494258,100)
> 
> 
> In [136]: time T = A.dot(VV)
> CPU times: user 0.08 s, sys: 0.00 s, total: 0.08 s
> Wall time: 0.08 s
> 
> 
> I suspect there has something related to how the data is
> internally represented.
> 
> 
> Thank you!
> 
> 
> Best Regards,
> Wenlei
> 
> 
> 
> 
> 
> 
> On Tue, May 6, 2014 at 11:01 PM, Wenlei Xie <wenlei.xie at gmail.com>
> wrote:
>         Thank you guys! However it looks like none of these holds:
>         
>         
>         In [50]: np.isnan(V).any()
>         Out[50]: False
>         
>         
>         In [51]: (abs(V) < 1e-100).any()
>         Out[51]: False
>         
>         
>         In [52]: (abs(V) > 1e+100).any()
>         Out[52]: False
>         
>         
>         In [53]: V.dtype
>         Out[53]: dtype('float64')
>         
>         
>         
>         
>         I think it's probably the matrix contains some strange values
>         that slowed things down. But I don't have a clear clue (it is
>         generated by scipy.svd).
>         
>         
>         Can any one give me a reference to the implantation to
>         the .dot() for sparse matrices?
>         
>         
>         Another interesting observation is that if I .dot() each
>         column of the matrix V, the running time is much shorter
>         than .dot() all of them:
>         
>         
>         In [97]: time T = A.dot(V[:, 0])
>         CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
>         Wall time: 0.00 s
>         
>         
>         In [98]: time T = A.dot(V[:, 1])
>         CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
>         Wall time: 0.00 s
>         
>         
>         In [99]: time T = A.dot(V[:, 0:2])
>         CPU times: user 0.06 s, sys: 0.01 s, total: 0.07 s
>         Wall time: 0.07 s
>         
>         
>         In [100]: time T = A.dot(V)
>         CPU times: user 3.38 s, sys: 0.52 s, total: 3.90 s
>         Wall time: 3.91 s
>         
>         
>         (And, most of the time seems to be spent on user-mode?)
>         
>         
>         
>         
>         Thank you!
>         
>         
>         Best,
>         Wenlei
>         
>         
>         
>         
>         On Tue, May 6, 2014 at 1:06 PM, Pauli Virtanen <pav at iki.fi>
>         wrote:
>                 06.05.2014 02:35, Wenlei Xie kirjoitti:
>                 > I got very strange behavior of SpMV operations on
>                 SciPy. The operation is A
>                 > * b, where A is a 160 * 3494258  sparse matrix with
>                 134K non-zero entries,
>                 > b is a 3494258 * 100 dense matrix. I tried to use a
>                 generated b and random
>                 > b, which got very different performance:
>                 >
>                 > PROFILE: A.dot(b1) finished, times taken 3.31 sec.
>                 > PROFILE: A.dot(b2) finished, times taken 0.06 sec.
>                 
>                 [clip]
>                 
>                 Some possibilities that may slow down floating point
>                 math:
>                 
>                 - denormal numbers (|a| < ~1e-300 and |a| > ~1e300)
>                 - NaNs
>                 
>                 The first can in principle be controlled by compiler
>                 flags used when
>                 compiling Scipy.
>                 
>                 --
>                 Pauli Virtanen
>                 
>                 _______________________________________________
>                 SciPy-User mailing list
>                 SciPy-User at scipy.org
>                 http://mail.scipy.org/mailman/listinfo/scipy-user
>         
>         
>         
>         
>         -- 
>         Wenlei Xie (???)
>         
>         Department of Computer Science
>         5132 Upson Hall, Cornell University
>         Ithaca, NY 14853, USA
>         Phone: (607) 255-5577
>         Email: wenlei.xie at gmail.com
> 
> 
> 
> 
> -- 
> Wenlei Xie (???)
> 
> Department of Computer Science
> 5132 Upson Hall, Cornell University
> Ithaca, NY 14853, USA
> Phone: (607) 255-5577
> Email: wenlei.xie at gmail.com
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140507/caa0bee9/attachment.sig>

From kevin.bache at gmail.com  Thu May  8 15:56:53 2014
From: kevin.bache at gmail.com (Kevin Bache)
Date: Thu, 8 May 2014 12:56:53 -0700
Subject: [SciPy-User] Saving Function Values in Optimize
Message-ID: <CAK9LM1DHJR9ucTzQoMBk7ASBtMG24KmdUPUeyRbmgNjFsjapDQ@mail.gmail.com>

Hi Everyone,
Quick question: is there a preferred way to save function values in
optimize.minimize?  The callback function format only passes 'xk', the
current parameters for the optimization problem.  For some problem types,
it's computationally trivial to convert that to f(xk), but in many, that
process is expensive.  I'd like to save all function evaluations as I
progress through the optimization process, yielding an object with the
information:

(xk_1, f(xk_1)), (xk_2, f(xk_2)), ...  (xk_n, f(xk_n))

Does anyone have advice on how to go about this without hacking the
optimize code?

Thanks in advance!

Best Regards,
Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140508/47482e11/attachment.html>

From sebastian at sipsolutions.net  Thu May  8 19:04:32 2014
From: sebastian at sipsolutions.net (Sebastian Berg)
Date: Fri, 09 May 2014 01:04:32 +0200
Subject: [SciPy-User] Saving Function Values in Optimize
In-Reply-To: <CAK9LM1DHJR9ucTzQoMBk7ASBtMG24KmdUPUeyRbmgNjFsjapDQ@mail.gmail.com>
References: <CAK9LM1DHJR9ucTzQoMBk7ASBtMG24KmdUPUeyRbmgNjFsjapDQ@mail.gmail.com>
Message-ID: <1399590272.29507.2.camel@sebastian-t440>

On Do, 2014-05-08 at 12:56 -0700, Kevin Bache wrote:
> Hi Everyone,
> 
> Quick question: is there a preferred way to save function values in
> optimize.minimize?  The callback function format only passes 'xk', the
> current parameters for the optimization problem.  For some problem
> types, it's computationally trivial to convert that to f(xk), but in
> many, that process is expensive.  I'd like to save all function
> evaluations as I progress through the optimization process, yielding
> an object with the information:
> 
> 
> (xk_1, f(xk_1)), (xk_2, f(xk_2)), ...  (xk_n, f(xk_n))
> 
> 
> Does anyone have advice on how to go about this without hacking the
> optimize code?
> 
The sniplet below isn't perfect, but I think you should be able to adept
it to your needs. Decorators can do this kind of magic pretty nicely
(though of course nothing stops you from just implementing it into the
function itself).

- Sebastian


def store_cost(func):
    """Decorator for cost functions, as long as the cost function is only
    called with the same arguments this works good. Defines func.x and
    func.cost.

    Example:
    ========

    from scipy.optimize import fmin

    @store_cost
    def func(x):
        return (x - 10)**2
    fmin(func, [0])

    print func.x
    print func.cost
    """
    x_list = []
    cost_list = []
    def new_func(x, *args):
        x_list.append(x.copy())
        e = func(x, *args)
        cost_list.append(e)
        return e
    new_func.x = x_list
    new_func.cost = cost_list
    return new_func


> 
> Thanks in advance!
> 
> 
> Best Regards,
> Kevin
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140509/66aed5ef/attachment.sig>

From kevin.bache at gmail.com  Thu May  8 21:57:43 2014
From: kevin.bache at gmail.com (Kevin Bache)
Date: Thu, 8 May 2014 18:57:43 -0700
Subject: [SciPy-User] Saving Function Values in Optimize
In-Reply-To: <1399590272.29507.2.camel@sebastian-t440>
References: <CAK9LM1DHJR9ucTzQoMBk7ASBtMG24KmdUPUeyRbmgNjFsjapDQ@mail.gmail.com>
	<1399590272.29507.2.camel@sebastian-t440>
Message-ID: <CAK9LM1C117XOj_Jn_nCo2+J6seeB1sHj+sWZvNS2C0RU3dgfhQ@mail.gmail.com>

Thanks, Sebastian.  Good idea.
Kevin


On Thu, May 8, 2014 at 4:04 PM, Sebastian Berg
<sebastian at sipsolutions.net>wrote:

> On Do, 2014-05-08 at 12:56 -0700, Kevin Bache wrote:
> > Hi Everyone,
> >
> > Quick question: is there a preferred way to save function values in
> > optimize.minimize?  The callback function format only passes 'xk', the
> > current parameters for the optimization problem.  For some problem
> > types, it's computationally trivial to convert that to f(xk), but in
> > many, that process is expensive.  I'd like to save all function
> > evaluations as I progress through the optimization process, yielding
> > an object with the information:
> >
> >
> > (xk_1, f(xk_1)), (xk_2, f(xk_2)), ...  (xk_n, f(xk_n))
> >
> >
> > Does anyone have advice on how to go about this without hacking the
> > optimize code?
> >
> The sniplet below isn't perfect, but I think you should be able to adept
> it to your needs. Decorators can do this kind of magic pretty nicely
> (though of course nothing stops you from just implementing it into the
> function itself).
>
> - Sebastian
>
>
> def store_cost(func):
>     """Decorator for cost functions, as long as the cost function is only
>     called with the same arguments this works good. Defines func.x and
>     func.cost.
>
>     Example:
>     ========
>
>     from scipy.optimize import fmin
>
>     @store_cost
>     def func(x):
>         return (x - 10)**2
>     fmin(func, [0])
>
>     print func.x
>     print func.cost
>     """
>     x_list = []
>     cost_list = []
>     def new_func(x, *args):
>         x_list.append(x.copy())
>         e = func(x, *args)
>         cost_list.append(e)
>         return e
>     new_func.x = x_list
>     new_func.cost = cost_list
>     return new_func
>
>
> >
> > Thanks in advance!
> >
> >
> > Best Regards,
> > Kevin
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140508/2b637f25/attachment.html>

From sturla.molden at gmail.com  Fri May  9 04:28:56 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Fri, 9 May 2014 08:28:56 +0000 (UTC)
Subject: [SciPy-User] Saving Function Values in Optimize
References: <CAK9LM1DHJR9ucTzQoMBk7ASBtMG24KmdUPUeyRbmgNjFsjapDQ@mail.gmail.com>
	<1399590272.29507.2.camel@sebastian-t440>
	<CAK9LM1C117XOj_Jn_nCo2+J6seeB1sHj+sWZvNS2C0RU3dgfhQ@mail.gmail.com>
Message-ID: <258420934421313172.715059sturla.molden-gmail.com@news.gmane.org>

If function evaluations is used to approximate the derivatives, you might
want to skip those evaluations in the tally, as they don't show the
convergence path. Then we end up with something like this (not tested, but
should work unless I made a typo), where the result object gets an 'fevals'
attribute that stores the function evaluation path, excluding approximation
of derivatives:


def minimize(f, x0, jac=None, hessp=None, epsilon=1.E-12, **kwargs):

    fevals = list()
    approx_fprime = sp.optimize.approx_fprime

    def fun(xk, *args):
        v = f(xk,*args)
        fevals.append(v)
        return v

    def approx_jac(xk, *args):
        return approx_fprime(xk, f, epsilon, *args)

    if jac is None:
        jac = approx_jac

    def approx_hessp(x0, p, *args):
        f2 = jac(*((x0 + epsilon*p,) + args))
        f1 = jac(*((x0,) + args))
        return (f2 - f1) / epsilon

    if hessp = None:
        hessp = approx_hessp

    res = scipy.optimize.minimize(fun, x0, args=args, 
        jac=jac, hessp=hessp, **kwargs)
    
    res.fevals = fevals 
    return res


:-)


Regards,
Sturla

Kevin Bache <kevin.bache at gmail.com> wrote:
> Thanks, Sebastian.  Good idea.
> Kevin
> 
> On Thu, May 8, 2014 at 4:04 PM, Sebastian Berg
> <sebastian at sipsolutions.net>wrote:
> 
>> On Do, 2014-05-08 at 12:56 -0700, Kevin Bache wrote:
>>> Hi Everyone,
>>> 
>>> Quick question: is there a preferred way to save function values in
>>> optimize.minimize?  The callback function format only passes 'xk', the
>>> current parameters for the optimization problem.  For some problem
>>> types, it's computationally trivial to convert that to f(xk), but in
>>> many, that process is expensive.  I'd like to save all function
>>> evaluations as I progress through the optimization process, yielding
>>> an object with the information:
>>> 
>>> 
>>> (xk_1, f(xk_1)), (xk_2, f(xk_2)), ...  (xk_n, f(xk_n))
>>> 
>>> 
>>> Does anyone have advice on how to go about this without hacking the
>>> optimize code?
>>> 
>> The sniplet below isn't perfect, but I think you should be able to adept
>> it to your needs. Decorators can do this kind of magic pretty nicely
>> (though of course nothing stops you from just implementing it into the
>> function itself).
>> 
>> - Sebastian
>> 
>> 
>> def store_cost(func):
>>     """Decorator for cost functions, as long as the cost function is only
>>     called with the same arguments this works good. Defines func.x and
>>     func.cost.
>> 
>>     Example:
>>     ========
>> 
>>     from scipy.optimize import fmin
>> 
>>     @store_cost
>>     def func(x):
>>         return (x - 10)**2
>>     fmin(func, [0])
>> 
>>     print func.x
>>     print func.cost
>>     """
>>     x_list = []
>>     cost_list = []
>>     def new_func(x, *args):
>>         x_list.append(x.copy())
>>         e = func(x, *args)
>>         cost_list.append(e)
>>         return e
>>     new_func.x = x_list
>>     new_func.cost = cost_list
>>     return new_func
>> 
>> 
>>> 
>>> Thanks in advance!
>>> 
>>> 
>>> Best Regards,
>>> Kevin
>>> 
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> <a
>>> href="http://mail.scipy.org/mailman/listinfo/scipy-user">http://mail.scipy.org/mailman/listinfo/scipy-user</a>
>> 
>> 
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> <a
>> href="http://mail.scipy.org/mailman/listinfo/scipy-user">http://mail.scipy.org/mailman/listinfo/scipy-user</a>
>> 
>> 
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> <a
> href="http://mail.scipy.org/mailman/listinfo/scipy-user">http://mail.scipy.org/mailman/listinfo/scipy-user</a>



From gb.gabrielebrambilla at gmail.com  Fri May  9 10:33:42 2014
From: gb.gabrielebrambilla at gmail.com (Gabriele Brambilla)
Date: Fri, 9 May 2014 10:33:42 -0400
Subject: [SciPy-User] forcing curve fit
Message-ID: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>

Hi,

is it possible to force curve_fit to search the values for the best fit
parameters in a certain interval of values?

Because I have a fit routine that returns to me a lot of good curves but
sometimes returns to me a curve with an amplitude that is 100, 1000 times
smaller!

thanks

Gabriele
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140509/0e3678ee/attachment.html>

From msarahan at gmail.com  Fri May  9 12:56:39 2014
From: msarahan at gmail.com (Michael Sarahan)
Date: Fri, 9 May 2014 09:56:39 -0700
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
Message-ID: <CAB9hrOpZoY=3Y1MtYRC9Nu9xLtoB0EupgDoBLaeiHV+phzyyUg@mail.gmail.com>

The word that you are looking for is "constraint."  Here's a Stack Overflow
post for you:
http://stackoverflow.com/questions/16541171/how-do-i-put-a-constraint-on-scipy-curve-fit

HTH,
Mike


On Fri, May 9, 2014 at 7:33 AM, Gabriele Brambilla <
gb.gabrielebrambilla at gmail.com> wrote:

> Hi,
>
> is it possible to force curve_fit to search the values for the best fit
> parameters in a certain interval of values?
>
> Because I have a fit routine that returns to me a lot of good curves but
> sometimes returns to me a curve with an amplitude that is 100, 1000 times
> smaller!
>
> thanks
>
> Gabriele
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140509/5faab6d0/attachment.html>

From sturla.molden at gmail.com  Fri May  9 15:00:30 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Fri, 9 May 2014 19:00:30 +0000 (UTC)
Subject: [SciPy-User] forcing curve fit
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
Message-ID: <1830143610421354715.570103sturla.molden-gmail.com@news.gmane.org>

It sounds like you need scipy.optimize.minimize with method SLSQP.

Sturla

Gabriele Brambilla <gb.gabrielebrambilla at gmail.com> wrote:
> Hi,
> 
> is it possible to force curve_fit to search the values for the best fit
> parameters in a certain interval of values?
> 
> Because I have a fit routine that returns to me a lot of good curves but
> sometimes returns to me a curve with an amplitude that is 100, 1000 times
> smaller!
> 
> thanks
> 
> Gabriele
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> <a
> href="http://mail.scipy.org/mailman/listinfo/scipy-user">http://mail.scipy.org/mailman/listinfo/scipy-user</a>



From newville at cars.uchicago.edu  Fri May  9 15:21:27 2014
From: newville at cars.uchicago.edu (Matt Newville)
Date: Fri, 9 May 2014 14:21:27 -0500
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
Message-ID: <CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>

Gabriele,


On Fri, May 9, 2014 at 9:33 AM, Gabriele Brambilla <
gb.gabrielebrambilla at gmail.com> wrote:

> Hi,
>
> is it possible to force curve_fit to search the values for the best fit
> parameters in a certain interval of values?
>
> Because I have a fit routine that returns to me a lot of good curves but
> sometimes returns to me a curve with an amplitude that is 100, 1000 times
> smaller!
>
> thanks
>
> Gabriele
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>

You might find the lmfit library (http://lmfit.github.io/lmfit-py) useful.
  This allows one to write least squares problems in terms of Parameter
objects which can have bounds set on their values, be fixed, or have values
written as mathematical expressions of other Parameter values.  The lmfit
library also allows you to switch between minimization algorithms easily,
and also explore confidence intervals in detail.

--Matt




-- 
--Matt Newville <newville at cars.uchicago.edu> 630-252-0431
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140509/40feb935/attachment.html>

From trive at astro.su.se  Fri May  9 17:02:13 2014
From: trive at astro.su.se (=?UTF-8?B?VGjDuGdlciBFbWlsIFJpdmVyYS1UaG9yc2Vu?=)
Date: Fri, 09 May 2014 23:02:13 +0200
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
	<CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
Message-ID: <536D4255.80603@astro.su.se>

Matt, are you the aothor of lmfit?
I am using it for a project I am working on at the moment and am very 
happy with it.

On Fri 09 May 2014 09:21:27 PM CEST, Matt Newville wrote:
> Gabriele,
>
>
> On Fri, May 9, 2014 at 9:33 AM, Gabriele Brambilla
> <gb.gabrielebrambilla at gmail.com
> <mailto:gb.gabrielebrambilla at gmail.com>> wrote:
>
>     Hi,
>
>     is it possible to force curve_fit to search the values for the
>     best fit parameters in a certain interval of values?
>
>     Because I have a fit routine that returns to me a lot of good
>     curves but sometimes returns to me a curve with an amplitude that
>     is 100, 1000 times smaller!
>
>     thanks
>
>     Gabriele
>
>     _______________________________________________
>     SciPy-User mailing list
>     SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>
> You might find the lmfit library (http://lmfit.github.io/lmfit-py)
> useful.   This allows one to write least squares problems in terms of
> Parameter objects which can have bounds set on their values, be fixed,
> or have values written as mathematical expressions of other Parameter
> values.  The lmfit library also allows you to switch between
> minimization algorithms easily, and also explore confidence intervals
> in detail.
>
> --Matt
>
>
>
>
> --
> --Matt Newville <newville at cars.uchicago.edu
> <http://cars.uchicago.edu>> 630-252-0431 <tel:630-252-0431>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


From newville at cars.uchicago.edu  Fri May  9 20:37:50 2014
From: newville at cars.uchicago.edu (Matt Newville)
Date: Fri, 9 May 2014 19:37:50 -0500
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <536D4255.80603@astro.su.se>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
	<CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
	<536D4255.80603@astro.su.se>
Message-ID: <CA+7ESbqdw6=kZ5JawYpAHRPzd3yn3Xt9_NYP7t-6GwpDTDYJPA@mail.gmail.com>

Hi Thoger,

On Fri, May 9, 2014 at 4:02 PM, Th?ger Emil Rivera-Thorsen <
trive at astro.su.se> wrote:

> Matt, are you the aothor of lmfit?
> I am using it for a project I am working on at the moment and am very
> happy with it.
>
>
I am, or I think I'm the main author.   There have been many contributions,
especially from Till Stensitzki (who may have written more code than I),
but several others as well, and it is certainly built on many previous
works (ie, scipy.optimize).  I think such a high level wrapping of
scipy.optimize addresses the common needs of fitting experimental data
(such as adding bounds and "frozen" parameters)  is needed, and believe
that may people using curve_fit() would find it useful.

But I also feel somewhat detached from this. It's one of several
side-projects somewhat related to my main research and responsibilities.
One might view it as essentially a "scikit", and if so I might be able to
kid myself that this was giving something of value back to the scipy
ecosystem.  I know that's not true, but I will support lmfit to the best of
my abilities for the foreseeable future.  Then again, I feel no ownership
of it, and if someone is interested in contributing or taking over, I'd be
happy to help and encourage as much as I can.

Anyway, I'm very glad to hear you're happy with it.

--Matt Newville
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140509/4970bd64/attachment.html>

From yw5aj at virginia.edu  Sat May 10 12:36:25 2014
From: yw5aj at virginia.edu (Yuxiang Wang)
Date: Sat, 10 May 2014 12:36:25 -0400
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CA+7ESbqdw6=kZ5JawYpAHRPzd3yn3Xt9_NYP7t-6GwpDTDYJPA@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
	<CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
	<536D4255.80603@astro.su.se>
	<CA+7ESbqdw6=kZ5JawYpAHRPzd3yn3Xt9_NYP7t-6GwpDTDYJPA@mail.gmail.com>
Message-ID: <CANRv7QpSstFoKm3X=-jK5rOvo27c0OYo0fwrZLs6nyN2x2dhdQ@mail.gmail.com>

Hi Matt,

Thank you for the great contribution! I have a quick question - does
lmfit scale the parameters by its initial values? For example, a
parameter maybe on the order of 1e-13, and during the fit we would
like to scale that number to 1 so it is easier to converge to an
optimum.

-Shawn

On Fri, May 9, 2014 at 8:37 PM, Matt Newville
<newville at cars.uchicago.edu> wrote:
> Hi Thoger,
>
>
> On Fri, May 9, 2014 at 4:02 PM, Th?ger Emil Rivera-Thorsen
> <trive at astro.su.se> wrote:
>>
>> Matt, are you the aothor of lmfit?
>> I am using it for a project I am working on at the moment and am very
>> happy with it.
>>
>
> I am, or I think I'm the main author.   There have been many contributions,
> especially from Till Stensitzki (who may have written more code than I), but
> several others as well, and it is certainly built on many previous works
> (ie, scipy.optimize).  I think such a high level wrapping of scipy.optimize
> addresses the common needs of fitting experimental data (such as adding
> bounds and "frozen" parameters)  is needed, and believe that may people
> using curve_fit() would find it useful.
>
> But I also feel somewhat detached from this. It's one of several
> side-projects somewhat related to my main research and responsibilities.
> One might view it as essentially a "scikit", and if so I might be able to
> kid myself that this was giving something of value back to the scipy
> ecosystem.  I know that's not true, but I will support lmfit to the best of
> my abilities for the foreseeable future.  Then again, I feel no ownership of
> it, and if someone is interested in contributing or taking over, I'd be
> happy to help and encourage as much as I can.
>
> Anyway, I'm very glad to hear you're happy with it.
>
> --Matt Newville
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>



-- 
Yuxiang "Shawn" Wang
Gerling Research Lab
University of Virginia
yw5aj at virginia.edu
+1 (434) 284-0836
https://sites.google.com/a/virginia.edu/yw5aj/


From newville at cars.uchicago.edu  Sat May 10 13:17:07 2014
From: newville at cars.uchicago.edu (Matt Newville)
Date: Sat, 10 May 2014 12:17:07 -0500
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CANRv7QpSstFoKm3X=-jK5rOvo27c0OYo0fwrZLs6nyN2x2dhdQ@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
	<CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
	<536D4255.80603@astro.su.se>
	<CA+7ESbqdw6=kZ5JawYpAHRPzd3yn3Xt9_NYP7t-6GwpDTDYJPA@mail.gmail.com>
	<CANRv7QpSstFoKm3X=-jK5rOvo27c0OYo0fwrZLs6nyN2x2dhdQ@mail.gmail.com>
Message-ID: <CA+7ESbpW-_HwwtjdJwen59CfyLT20c294vUt6R35+f+edxH3ww@mail.gmail.com>

On May 10, 2014 11:37 AM, "Yuxiang Wang" <yw5aj at virginia.edu> wrote:
>
> Hi Matt,
>
> Thank you for the great contribution! I have a quick question - does
> lmfit scale the parameters by its initial values? For example, a
> parameter maybe on the order of 1e-13, and during the fit we would
> like to scale that number to 1 so it is easier to converge to an
> optimum.
>
> -Shawn

My understanding and experience  is that leastsq() does this internally
well enough to usually not have to worry about it.   But if you know the
scales differ by very many orders of magnitude,  it probably wouldn't hurt
to rescale values.

>
> On Fri, May 9, 2014 at 8:37 PM, Matt Newville
> <newville at cars.uchicago.edu> wrote:
> > Hi Thoger,
> >
> >
> > On Fri, May 9, 2014 at 4:02 PM, Th?ger Emil Rivera-Thorsen
> > <trive at astro.su.se> wrote:
> >>
> >> Matt, are you the aothor of lmfit?
> >> I am using it for a project I am working on at the moment and am very
> >> happy with it.
> >>
> >
> > I am, or I think I'm the main author.   There have been many
contributions,
> > especially from Till Stensitzki (who may have written more code than
I), but
> > several others as well, and it is certainly built on many previous works
> > (ie, scipy.optimize).  I think such a high level wrapping of
scipy.optimize
> > addresses the common needs of fitting experimental data (such as adding
> > bounds and "frozen" parameters)  is needed, and believe that may people
> > using curve_fit() would find it useful.
> >
> > But I also feel somewhat detached from this. It's one of several
> > side-projects somewhat related to my main research and responsibilities.
> > One might view it as essentially a "scikit", and if so I might be able
to
> > kid myself that this was giving something of value back to the scipy
> > ecosystem.  I know that's not true, but I will support lmfit to the
best of
> > my abilities for the foreseeable future.  Then again, I feel no
ownership of
> > it, and if someone is interested in contributing or taking over, I'd be
> > happy to help and encourage as much as I can.
> >
> > Anyway, I'm very glad to hear you're happy with it.
> >
> > --Matt Newville
> >
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > http://mail.scipy.org/mailman/listinfo/scipy-user
> >
>
>
>
> --
> Yuxiang "Shawn" Wang
> Gerling Research Lab
> University of Virginia
> yw5aj at virginia.edu
> +1 (434) 284-0836
> https://sites.google.com/a/virginia.edu/yw5aj/
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140510/5e2dfc34/attachment.html>

From ralf.gommers at gmail.com  Sun May 11 04:03:34 2014
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 11 May 2014 10:03:34 +0200
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CA+7ESbqdw6=kZ5JawYpAHRPzd3yn3Xt9_NYP7t-6GwpDTDYJPA@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
	<CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
	<536D4255.80603@astro.su.se>
	<CA+7ESbqdw6=kZ5JawYpAHRPzd3yn3Xt9_NYP7t-6GwpDTDYJPA@mail.gmail.com>
Message-ID: <CABL7CQjLZWMBhf4++AApWE=9raYw+8ZoN+utR51CZ-v6y_kP3A@mail.gmail.com>

On Sat, May 10, 2014 at 2:37 AM, Matt Newville
<newville at cars.uchicago.edu>wrote:

> Hi Thoger,
>
>
> On Fri, May 9, 2014 at 4:02 PM, Th?ger Emil Rivera-Thorsen <
> trive at astro.su.se> wrote:
>
>> Matt, are you the aothor of lmfit?
>> I am using it for a project I am working on at the moment and am very
>> happy with it.
>>
>>
> I am, or I think I'm the main author.   There have been many
> contributions, especially from Till Stensitzki (who may have written more
> code than I), but several others as well, and it is certainly built on
> many previous works (ie, scipy.optimize).  I think such a high level
> wrapping of scipy.optimize addresses the common needs of fitting
> experimental data (such as adding bounds and "frozen" parameters)  is
> needed, and believe that may people using curve_fit() would find it useful.
>
>
> But I also feel somewhat detached from this. It's one of several
> side-projects somewhat related to my main research and responsibilities.
> One might view it as essentially a "scikit", and if so I might be able to
> kid myself that this was giving something of value back to the scipy
> ecosystem.  I know that's not true,
>

Not sure if you meant that or not, but I'm going to have to disagree
anyway. Packages like lmfit, which provide regularly requested
functionality and are documented and maintained well, are *very* valuable.
The scientific Python ecosystem is so strong mainly because there are
packages like that for almost everything one needs. A few big packages like
numpy/scipy/pandas/scikit-learn could never contain all that functionality
(for lots of reasons, release timing, maintainer load, etc.), we need a lot
of small specialized ones.

So thank you for lmfit!

Cheers,
Ralf


but I will support lmfit to the best of my abilities for the foreseeable
> future.  Then again, I feel no ownership of it, and if someone is
> interested in contributing or taking over, I'd be happy to help and
> encourage as much as I can.
>
> Anyway, I'm very glad to hear you're happy with it.
>
> --Matt Newville
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140511/35fecd58/attachment.html>

From josef.pktd at gmail.com  Sun May 11 09:43:21 2014
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Sun, 11 May 2014 09:43:21 -0400
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CABL7CQjLZWMBhf4++AApWE=9raYw+8ZoN+utR51CZ-v6y_kP3A@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
	<CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
	<536D4255.80603@astro.su.se>
	<CA+7ESbqdw6=kZ5JawYpAHRPzd3yn3Xt9_NYP7t-6GwpDTDYJPA@mail.gmail.com>
	<CABL7CQjLZWMBhf4++AApWE=9raYw+8ZoN+utR51CZ-v6y_kP3A@mail.gmail.com>
Message-ID: <CAMMTP+C=HiTRV763qnCt3HsxQfRHAaf2kJA5wg6-ZnkZR5_6kA@mail.gmail.com>

On Sun, May 11, 2014 at 4:03 AM, Ralf Gommers <ralf.gommers at gmail.com>wrote:

>
>
>
> On Sat, May 10, 2014 at 2:37 AM, Matt Newville <newville at cars.uchicago.edu
> > wrote:
>
>> Hi Thoger,
>>
>>
>> On Fri, May 9, 2014 at 4:02 PM, Th?ger Emil Rivera-Thorsen <
>> trive at astro.su.se> wrote:
>>
>>> Matt, are you the aothor of lmfit?
>>> I am using it for a project I am working on at the moment and am very
>>> happy with it.
>>>
>>>
>> I am, or I think I'm the main author.   There have been many
>> contributions, especially from Till Stensitzki (who may have written more
>> code than I), but several others as well, and it is certainly built on
>> many previous works (ie, scipy.optimize).  I think such a high level
>> wrapping of scipy.optimize addresses the common needs of fitting
>> experimental data (such as adding bounds and "frozen" parameters)  is
>> needed, and believe that may people using curve_fit() would find it useful.
>>
>>
>> But I also feel somewhat detached from this. It's one of several
>> side-projects somewhat related to my main research and responsibilities.
>> One might view it as essentially a "scikit", and if so I might be able to
>> kid myself that this was giving something of value back to the scipy
>> ecosystem.  I know that's not true,
>>
>
> Not sure if you meant that or not, but I'm going to have to disagree
> anyway. Packages like lmfit, which provide regularly requested
> functionality and are documented and maintained well, are *very* valuable.
> The scientific Python ecosystem is so strong mainly because there are
> packages like that for almost everything one needs. A few big packages like
> numpy/scipy/pandas/scikit-learn could never contain all that functionality
> (for lots of reasons, release timing, maintainer load, etc.), we need a lot
> of small specialized ones.
>
> So thank you for lmfit!
>

I fully agree with Ralf.
And I'm glad to point to lmfit when someone asks why statsmodels doesn't
have non-linear least squares.

And even when statsmodels gets nonlinear least squares, it won't have the
same parameter oriented interface.

Josef



>
> Cheers,
> Ralf
>
>
> but I will support lmfit to the best of my abilities for the foreseeable
>> future.  Then again, I feel no ownership of it, and if someone is
>> interested in contributing or taking over, I'd be happy to help and
>> encourage as much as I can.
>>
>> Anyway, I'm very glad to hear you're happy with it.
>>
>> --Matt Newville
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140511/e269620f/attachment.html>

From newville at cars.uchicago.edu  Sun May 11 13:33:18 2014
From: newville at cars.uchicago.edu (Matt Newville)
Date: Sun, 11 May 2014 12:33:18 -0500
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CAMMTP+C=HiTRV763qnCt3HsxQfRHAaf2kJA5wg6-ZnkZR5_6kA@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
	<CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
	<536D4255.80603@astro.su.se>
	<CA+7ESbqdw6=kZ5JawYpAHRPzd3yn3Xt9_NYP7t-6GwpDTDYJPA@mail.gmail.com>
	<CABL7CQjLZWMBhf4++AApWE=9raYw+8ZoN+utR51CZ-v6y_kP3A@mail.gmail.com>
	<CAMMTP+C=HiTRV763qnCt3HsxQfRHAaf2kJA5wg6-ZnkZR5_6kA@mail.gmail.com>
Message-ID: <CA+7ESbqGTHerqpdsAMA30_HmvG=1J3Hwr7MkiwgofS4=ZFGR2g@mail.gmail.com>

Hi Yuxiang, Ralf, Josef,

Thanks for the kinds words and sorry for the rather botched expression.
I was trying to express my view of the value of the work that I added to
lmfit compared to the work that it builds on (MINPACK, scipy.optimize, the
whole numpy/scipy stack).   I don't really identify lmfit as a main part of
my work, but I do think it's a valuable approach to fitting data. I'm happy
to be able to maintain this relatively small contribution for the
foreseeable future.  If someone was interested, I would be happy to pass it
on.
?
--Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140511/6cc47e73/attachment.html>

From ralf.gommers at gmail.com  Sun May 11 16:45:02 2014
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Sun, 11 May 2014 22:45:02 +0200
Subject: [SciPy-User] ANN: Weave 0.15.0 release
Message-ID: <CABL7CQj4FKT3pKgQPPQkW0J-HPNLOqcpXd4Ur2+uXC=5koBYiQ@mail.gmail.com>

Hi,

I'm pleased to announce the release of Weave 0.15.0. Weave provides
tools for including C/C++ code within Python code. Inlining C/C++ code
within Python generally results in speedups of 1.5x to 30x over algorithms
written in pure Python.

Weave is the stand-alone version of the deprecated Scipy submodule
scipy.weave.
It is Python 2.x only, and is provided for users that need new versions of
Scipy (from which the weave submodule will be removed in the future) but
have existing code that still depends on scipy.weave.
For new code, users are recommended to use Cython.

Weave 0.15.0 is the first release of Weave as a standalone package.  It is
numbered 0.15.0, because it was split from Scipy after the 0.14.0 release of
that package.  No new functionality is included in this release compared to
Scipy 0.14.0, only changes needed to make Weave a standalone package.

This release requires Python 2.6 or 2.7. The source code can be found on
https://github.com/scipy/weave and the release itself on PyPi:
https://pypi.python.org/pypi/weave

Note that the Scipy developers are not planning to make any further
improvements
to Weave. They may however merge pull requests and create maintenance
releases for urgent issues. If someone is interested in maintaining Weave,
that would be very welcome. Questions and discussions relating to Weave
should
be directed to the scipy-dev mailing list (see
http://scipy.org/scipylib/mailing-lists.html).

Cheers,
Ralf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140511/79eb55aa/attachment.html>

From gb.gabrielebrambilla at gmail.com  Mon May 12 08:56:28 2014
From: gb.gabrielebrambilla at gmail.com (Gabriele Brambilla)
Date: Mon, 12 May 2014 08:56:28 -0400
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
	<CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
Message-ID: <CABmgkifze0Okhdb7VFQgGL=Qjr2zKmojQCYHgtk_P+3X475Xhg@mail.gmail.com>

Hi,
I tried to install it but I receive this error:

C:\Users\Gabriele\Documents\universit?\TESI\code>easy_install -U lmfit
Searching for lmfit
Reading http://pypi.python.org/simple/lmfit/
Best match: lmfit 0.7.4
Downloading
https://pypi.python.org/packages/source/l/lmfit/lmfit-0.7.4.tar.gz#m
d5=121cc508575b6c9e84f50d89fe6c40d0
Processing lmfit-0.7.4.tar.gz
Writing
c:\users\gabriele\appdata\local\temp\easy_install-vzljat\lmfit-0.7.4\set
up.cfg
Running lmfit-0.7.4\setup.py -q bdist_egg --dist-dir
c:\users\gabriele\appdata\l
ocal\temp\easy_install-vzljat\lmfit-0.7.4\egg-dist-tmp-5uakdk
error: SandboxViolation:
os.open('C:\\Users\\Gabriele\\.matplotlib\\tmp3euipt',
34242, 384) {}

The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.

This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand.  Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.

I am using Anaconda on Windows 7, 64bit.

thanks

Gabriele


2014-05-09 15:21 GMT-04:00 Matt Newville <newville at cars.uchicago.edu>:

> Gabriele,
>
>
> On Fri, May 9, 2014 at 9:33 AM, Gabriele Brambilla <
> gb.gabrielebrambilla at gmail.com> wrote:
>
>> Hi,
>>
>> is it possible to force curve_fit to search the values for the best fit
>> parameters in a certain interval of values?
>>
>> Because I have a fit routine that returns to me a lot of good curves but
>> sometimes returns to me a curve with an amplitude that is 100, 1000 times
>> smaller!
>>
>> thanks
>>
>> Gabriele
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> You might find the lmfit library (http://lmfit.github.io/lmfit-py)
> useful.   This allows one to write least squares problems in terms of
> Parameter objects which can have bounds set on their values, be fixed, or
> have values written as mathematical expressions of other Parameter values.
> The lmfit library also allows you to switch between minimization algorithms
> easily, and also explore confidence intervals in detail.
>
> --Matt
>
>
>
>
> --
> --Matt Newville <newville at cars.uchicago.edu> 630-252-0431
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140512/8824d15c/attachment.html>

From josef.pktd at gmail.com  Mon May 12 09:18:02 2014
From: josef.pktd at gmail.com (josef.pktd at gmail.com)
Date: Mon, 12 May 2014 09:18:02 -0400
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CABmgkifze0Okhdb7VFQgGL=Qjr2zKmojQCYHgtk_P+3X475Xhg@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
	<CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
	<CABmgkifze0Okhdb7VFQgGL=Qjr2zKmojQCYHgtk_P+3X475Xhg@mail.gmail.com>
Message-ID: <CAMMTP+AntjdXBXzqReB2jCaf3+RZwS8C1Ww7mTHH1t3dK-rPdw@mail.gmail.com>

On Mon, May 12, 2014 at 8:56 AM, Gabriele Brambilla <
gb.gabrielebrambilla at gmail.com> wrote:

> Hi,
> I tried to install it but I receive this error:
>
> C:\Users\Gabriele\Documents\universit?\TESI\code>easy_install -U lmfit
> Searching for lmfit
> Reading http://pypi.python.org/simple/lmfit/
> Best match: lmfit 0.7.4
> Downloading
> https://pypi.python.org/packages/source/l/lmfit/lmfit-0.7.4.tar.gz#m
> d5=121cc508575b6c9e84f50d89fe6c40d0
> Processing lmfit-0.7.4.tar.gz
> Writing
> c:\users\gabriele\appdata\local\temp\easy_install-vzljat\lmfit-0.7.4\set
> up.cfg
> Running lmfit-0.7.4\setup.py -q bdist_egg --dist-dir
> c:\users\gabriele\appdata\l
> ocal\temp\easy_install-vzljat\lmfit-0.7.4\egg-dist-tmp-5uakdk
> error: SandboxViolation:
> os.open('C:\\Users\\Gabriele\\.matplotlib\\tmp3euipt',
> 34242, 384) {}
>
> The package setup script has attempted to modify files on your system
> that are not within the EasyInstall build area, and has been aborted.
>
> This package cannot be safely installed by EasyInstall, and may not
> support alternate installation locations even if you run its setup
> script by hand.  Please inform the package's author and the EasyInstall
> maintainers to find out if a fix or workaround is available.
>
> I am using Anaconda on Windows 7, 64bit.
>

statsmodels has this in the beginning in the setup.py

# temporarily redirect config directory to prevent matplotlib importing
# testing that for writeable directory which results in sandbox error in
# certain easy_install versions
os.environ["MPLCONFIGDIR"] = "."


I don't remember what was triggering an import of matplotlib.

https://github.com/lmfit/lmfit-py/issues


Josef


>
> thanks
>
> Gabriele
>
>
> 2014-05-09 15:21 GMT-04:00 Matt Newville <newville at cars.uchicago.edu>:
>
> Gabriele,
>>
>>
>> On Fri, May 9, 2014 at 9:33 AM, Gabriele Brambilla <
>> gb.gabrielebrambilla at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> is it possible to force curve_fit to search the values for the best fit
>>> parameters in a certain interval of values?
>>>
>>> Because I have a fit routine that returns to me a lot of good curves but
>>> sometimes returns to me a curve with an amplitude that is 100, 1000 times
>>> smaller!
>>>
>>> thanks
>>>
>>> Gabriele
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>> You might find the lmfit library (http://lmfit.github.io/lmfit-py)
>> useful.   This allows one to write least squares problems in terms of
>> Parameter objects which can have bounds set on their values, be fixed, or
>> have values written as mathematical expressions of other Parameter values.
>> The lmfit library also allows you to switch between minimization algorithms
>> easily, and also explore confidence intervals in detail.
>>
>> --Matt
>>
>>
>>
>>
>> --
>> --Matt Newville <newville at cars.uchicago.edu> 630-252-0431
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140512/8c9f6222/attachment.html>

From trive at astro.su.se  Mon May 12 09:33:50 2014
From: trive at astro.su.se (=?ISO-8859-1?Q?Th=F8ger_Emil_Rivera-Thorsen?=)
Date: Mon, 12 May 2014 15:33:50 +0200
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CABL7CQjLZWMBhf4++AApWE=9raYw+8ZoN+utR51CZ-v6y_kP3A@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>	<CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>	<536D4255.80603@astro.su.se>	<CA+7ESbqdw6=kZ5JawYpAHRPzd3yn3Xt9_NYP7t-6GwpDTDYJPA@mail.gmail.com>
	<CABL7CQjLZWMBhf4++AApWE=9raYw+8ZoN+utR51CZ-v6y_kP3A@mail.gmail.com>
Message-ID: <5370CDBE.5000804@astro.su.se>

What Ralph said.
lmfit has definitely halped me get some results quickly due to some nice
features and, not least, its great transparency and excellent
documentatioj, freeing my energy for other matters.
I do indeed think that is very valuable for the community and I have
often recommended it.

On 2014-05-11 10:03, Ralf Gommers wrote:
>
>
>
> On Sat, May 10, 2014 at 2:37 AM, Matt Newville
> <newville at cars.uchicago.edu <mailto:newville at cars.uchicago.edu>> wrote:
>
>     Hi Thoger,
>
>
>     On Fri, May 9, 2014 at 4:02 PM, Th?ger Emil Rivera-Thorsen
>     <trive at astro.su.se <mailto:trive at astro.su.se>> wrote:
>
>         Matt, are you the aothor of lmfit?
>         I am using it for a project I am working on at the moment and
>         am very
>         happy with it.
>
>      
>     I am, or I think I'm the main author.   There have been many
>     contributions, especially from Till Stensitzki (who may have
>     written more code than I), but several others as well, and it is
>     certainly built on many previous works (ie, scipy.optimize).  I
>     think such a high level wrapping of scipy.optimize addresses the
>     common needs of fitting experimental data (such as adding bounds
>     and "frozen" parameters)  is needed, and believe that may people
>     using curve_fit() would find it useful.  
>
>     But I also feel somewhat detached from this. It's one of several
>     side-projects somewhat related to my main research and
>     responsibilities.   One might view it as essentially a "scikit",
>     and if so I might be able to kid myself that this was giving
>     something of value back to the scipy ecosystem.  I know that's not
>     true,
>
>
> Not sure if you meant that or not, but I'm going to have to disagree
> anyway. Packages like lmfit, which provide regularly requested
> functionality and are documented and maintained well, are *very*
> valuable. The scientific Python ecosystem is so strong mainly because
> there are packages like that for almost everything one needs. A few
> big packages like numpy/scipy/pandas/scikit-learn could never contain
> all that functionality (for lots of reasons, release timing,
> maintainer load, etc.), we need a lot of small specialized ones.
>
> So thank you for lmfit!
>
> Cheers,
> Ralf
>
>
>     but I will support lmfit to the best of my abilities for the
>     foreseeable future.  Then again, I feel no ownership of it, and if
>     someone is interested in contributing or taking over, I'd be happy
>     to help and encourage as much as I can.
>      
>     Anyway, I'm very glad to hear you're happy with it.
>
>     --Matt Newville
>
>
>     _______________________________________________
>     SciPy-User mailing list
>     SciPy-User at scipy.org <mailto:SciPy-User at scipy.org>
>     http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user


-- 
--------------------------
Th?ger Emil Rivera-Thorsen
Ph.D. student
Stockholm University, 
Department of Astronomy
--------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140512/efa78e3f/attachment.html>

From gb.gabrielebrambilla at gmail.com  Mon May 12 10:09:00 2014
From: gb.gabrielebrambilla at gmail.com (Gabriele Brambilla)
Date: Mon, 12 May 2014 10:09:00 -0400
Subject: [SciPy-User] forcing curve fit
In-Reply-To: <CAMMTP+AntjdXBXzqReB2jCaf3+RZwS8C1Ww7mTHH1t3dK-rPdw@mail.gmail.com>
References: <CABmgkidzqm2Vk_2czSj6=8t7p=rEjMG+zYH6qehMeOtXt5q0Mg@mail.gmail.com>
	<CA+7ESbrObGLp36GL6CcC-PVPvVKUdM==1Zb8DEAY7Mep6_x0SA@mail.gmail.com>
	<CABmgkifze0Okhdb7VFQgGL=Qjr2zKmojQCYHgtk_P+3X475Xhg@mail.gmail.com>
	<CAMMTP+AntjdXBXzqReB2jCaf3+RZwS8C1Ww7mTHH1t3dK-rPdw@mail.gmail.com>
Message-ID: <CABmgkifUPvJ-TdqcFE3rbMKp0NDVeun9WZBB7YiutF2zXYHYwQ@mail.gmail.com>

But so what I have to do to install it?

thanks

Gabriele


2014-05-12 9:18 GMT-04:00 <josef.pktd at gmail.com>:

>
>
>
> On Mon, May 12, 2014 at 8:56 AM, Gabriele Brambilla <
> gb.gabrielebrambilla at gmail.com> wrote:
>
>> Hi,
>> I tried to install it but I receive this error:
>>
>> C:\Users\Gabriele\Documents\universit?\TESI\code>easy_install -U lmfit
>> Searching for lmfit
>> Reading http://pypi.python.org/simple/lmfit/
>> Best match: lmfit 0.7.4
>> Downloading
>> https://pypi.python.org/packages/source/l/lmfit/lmfit-0.7.4.tar.gz#m
>> d5=121cc508575b6c9e84f50d89fe6c40d0
>> Processing lmfit-0.7.4.tar.gz
>> Writing
>> c:\users\gabriele\appdata\local\temp\easy_install-vzljat\lmfit-0.7.4\set
>> up.cfg
>> Running lmfit-0.7.4\setup.py -q bdist_egg --dist-dir
>> c:\users\gabriele\appdata\l
>> ocal\temp\easy_install-vzljat\lmfit-0.7.4\egg-dist-tmp-5uakdk
>> error: SandboxViolation:
>> os.open('C:\\Users\\Gabriele\\.matplotlib\\tmp3euipt',
>> 34242, 384) {}
>>
>> The package setup script has attempted to modify files on your system
>> that are not within the EasyInstall build area, and has been aborted.
>>
>> This package cannot be safely installed by EasyInstall, and may not
>> support alternate installation locations even if you run its setup
>> script by hand.  Please inform the package's author and the EasyInstall
>> maintainers to find out if a fix or workaround is available.
>>
>> I am using Anaconda on Windows 7, 64bit.
>>
>
> statsmodels has this in the beginning in the setup.py
>
> # temporarily redirect config directory to prevent matplotlib importing
> # testing that for writeable directory which results in sandbox error in
> # certain easy_install versions
> os.environ["MPLCONFIGDIR"] = "."
>
>
> I don't remember what was triggering an import of matplotlib.
>
> https://github.com/lmfit/lmfit-py/issues
>
>
> Josef
>
>
>>
>> thanks
>>
>> Gabriele
>>
>>
>> 2014-05-09 15:21 GMT-04:00 Matt Newville <newville at cars.uchicago.edu>:
>>
>> Gabriele,
>>>
>>>
>>> On Fri, May 9, 2014 at 9:33 AM, Gabriele Brambilla <
>>> gb.gabrielebrambilla at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> is it possible to force curve_fit to search the values for the best fit
>>>> parameters in a certain interval of values?
>>>>
>>>> Because I have a fit routine that returns to me a lot of good curves
>>>> but sometimes returns to me a curve with an amplitude that is 100, 1000
>>>> times smaller!
>>>>
>>>> thanks
>>>>
>>>> Gabriele
>>>>
>>>> _______________________________________________
>>>> SciPy-User mailing list
>>>> SciPy-User at scipy.org
>>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>>
>>>>
>>>
>>> You might find the lmfit library (http://lmfit.github.io/lmfit-py)
>>> useful.   This allows one to write least squares problems in terms of
>>> Parameter objects which can have bounds set on their values, be fixed, or
>>> have values written as mathematical expressions of other Parameter values.
>>> The lmfit library also allows you to switch between minimization algorithms
>>> easily, and also explore confidence intervals in detail.
>>>
>>> --Matt
>>>
>>>
>>>
>>>
>>> --
>>> --Matt Newville <newville at cars.uchicago.edu> 630-252-0431
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140512/33714285/attachment.html>

From maria-rosaria.antonelli at curie.fr  Tue May 13 13:01:24 2014
From: maria-rosaria.antonelli at curie.fr (Antonelli Maria Rosaria)
Date: Tue, 13 May 2014 17:01:24 +0000
Subject: [SciPy-User] problem in converting myfile.ipynb to Pdf
In-Reply-To: <CF981C43.7E8F%maria-rosaria.antonelli@curie.fr>
References: <CF981C43.7E8F%maria-rosaria.antonelli@curie.fr>
Message-ID: <CF981C79.7E91%maria-rosaria.antonelli@curie.fr>

Hi,

I have a problem converting a notebook into a pdf.
In the shell I use the command : nbconvert --to latex my file.ipynb --post PDF, I did not check the latex files but I obtain a corrupted pdf file that I cannot open.
Any ideas ?

Many thanks,
Cheers,
Rosa
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140513/22e93523/attachment.html>

From guziy.sasha at gmail.com  Tue May 13 15:05:50 2014
From: guziy.sasha at gmail.com (Oleksandr Huziy)
Date: Tue, 13 May 2014 15:05:50 -0400
Subject: [SciPy-User] problem in converting myfile.ipynb to Pdf
In-Reply-To: <CF981C79.7E91%maria-rosaria.antonelli@curie.fr>
References: <CF981C43.7E8F%maria-rosaria.antonelli@curie.fr>
	<CF981C79.7E91%maria-rosaria.antonelli@curie.fr>
Message-ID: <CAN3p1sVC-d30m4UGAUk5h253pyXAFExu6h8AtqmYMKgptmLugg@mail.gmail.com>

Could you share your notebook?


Cheers


2014-05-13 13:01 GMT-04:00 Antonelli Maria Rosaria <
maria-rosaria.antonelli at curie.fr>:

>  Hi,
>
>  I have a problem converting a notebook into a pdf.
> In the shell I use the command : nbconvert --to latex my file.ipynb --post
> PDF, I did not check the latex files but I obtain a corrupted pdf file that
> I cannot open.
> Any ideas ?
>
>  Many thanks,
> Cheers,
> Rosa
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Sasha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140513/3c74e39d/attachment.html>

From martin.noblia at openmailbox.org  Tue May 13 16:41:59 2014
From: martin.noblia at openmailbox.org (=?ISO-8859-1?Q?Martin_Nobl=EDa?=)
Date: Tue, 13 May 2014 17:41:59 -0300
Subject: [SciPy-User] problem in converting myfile.ipynb to Pdf
In-Reply-To: <CF981C79.7E91%maria-rosaria.antonelli@curie.fr>
References: <CF981C43.7E8F%maria-rosaria.antonelli@curie.fr>
	<CF981C79.7E91%maria-rosaria.antonelli@curie.fr>
Message-ID: <53728397.1010009@openmailbox.org>

If you OS is a some flavor of GNU/Linux:

first check the dependencies (pandoc, texlive(i recomended texlive-full))
later
for one notebook that i wrote pandoc converts wrong some sentences 
aggregate some \

Regards
El 13/05/14 14:01, Antonelli Maria Rosaria escribi?:
> Hi,
>
> I have a problem converting a notebook into a pdf.
> In the shell I use the command : nbconvert --to latex my 
> file.ipynb --post PDF, I did not check the latex files but I obtain a 
> corrupted pdf file that I cannot open.
> Any ideas ?
>
> Many thanks,
> Cheers,
> Rosa
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user

-- 
*Martin Nobl?a*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140513/cd00bfda/attachment.html>

From srean.list at gmail.com  Wed May 14 05:15:17 2014
From: srean.list at gmail.com (srean)
Date: Wed, 14 May 2014 04:15:17 -0500
Subject: [SciPy-User] ANN: Weave 0.15.0 release
In-Reply-To: <CABL7CQj4FKT3pKgQPPQkW0J-HPNLOqcpXd4Ur2+uXC=5koBYiQ@mail.gmail.com>
References: <CABL7CQj4FKT3pKgQPPQkW0J-HPNLOqcpXd4Ur2+uXC=5koBYiQ@mail.gmail.com>
Message-ID: <CAJewx8-rmQaKO3gpHSMiviLkC0xgfkS_QDQ_AtKxoWMU-ozCXw@mail.gmail.com>

Very happy to see this, I thought weave had been relegated to abandon-ware.
Although Weave did not get a lot of love I have found it very useful.

Although Cython is the recommended way these days, I have found that the
way to speed array operations in Cython is to write low level indexing code
in the tight inner loop. For me the reason to use a higher level language
is to avoid doing exactly that. Weave fit that hole perfectly. Wished it
integrated new Blitz++ release.

So, thanks.



On Sun, May 11, 2014 at 3:45 PM, Ralf Gommers <ralf.gommers at gmail.com>wrote:

> Hi,
>
> I'm pleased to announce the release of Weave 0.15.0. Weave provides
> tools for including C/C++ code within Python code. Inlining C/C++ code
> within Python generally results in speedups of 1.5x to 30x over algorithms
> written in pure Python.
>
> Weave is the stand-alone version of the deprecated Scipy submodule
> scipy.weave.
> It is Python 2.x only, and is provided for users that need new versions of
> Scipy (from which the weave submodule will be removed in the future) but
> have existing code that still depends on scipy.weave.
> For new code, users are recommended to use Cython.
>
> Weave 0.15.0 is the first release of Weave as a standalone package.  It is
> numbered 0.15.0, because it was split from Scipy after the 0.14.0 release
> of
> that package.  No new functionality is included in this release compared
> to
> Scipy 0.14.0, only changes needed to make Weave a standalone package.
>
> This release requires Python 2.6 or 2.7. The source code can be found on
> https://github.com/scipy/weave and the release itself on PyPi:
> https://pypi.python.org/pypi/weave
>
> Note that the Scipy developers are not planning to make any further
> improvements
> to Weave. They may however merge pull requests and create maintenance
> releases for urgent issues. If someone is interested in maintaining Weave,
> that would be very welcome. Questions and discussions relating to Weave
> should
> be directed to the scipy-dev mailing list (see
> http://scipy.org/scipylib/mailing-lists.html).
>
> Cheers,
> Ralf
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140514/b63b0906/attachment.html>

From robert.kern at gmail.com  Wed May 14 05:31:01 2014
From: robert.kern at gmail.com (Robert Kern)
Date: Wed, 14 May 2014 10:31:01 +0100
Subject: [SciPy-User] ANN: Weave 0.15.0 release
In-Reply-To: <CAJewx8-rmQaKO3gpHSMiviLkC0xgfkS_QDQ_AtKxoWMU-ozCXw@mail.gmail.com>
References: <CABL7CQj4FKT3pKgQPPQkW0J-HPNLOqcpXd4Ur2+uXC=5koBYiQ@mail.gmail.com>
	<CAJewx8-rmQaKO3gpHSMiviLkC0xgfkS_QDQ_AtKxoWMU-ozCXw@mail.gmail.com>
Message-ID: <CAF6FJiu9R4mYCx6A3_9qfqQKZfh3d8Y1a7G1a9WN59rp+HYhrQ@mail.gmail.com>

On Wed, May 14, 2014 at 10:15 AM, srean <srean.list at gmail.com> wrote:
> Very happy to see this, I thought weave had been relegated to abandon-ware.
> Although Weave did not get a lot of love I have found it very useful.

Actually, it's even more abandoned than before. It's been carved off
into its own project such that its abandonment doesn't hold up the
rest of scipy. See Ralf's last paragraph:

> On Sun, May 11, 2014 at 3:45 PM, Ralf Gommers <ralf.gommers at gmail.com>
> wrote:

>> Note that the Scipy developers are not planning to make any further
>> improvements
>> to Weave. They may however merge pull requests and create maintenance
>> releases for urgent issues. If someone is interested in maintaining Weave,
>> that would be very welcome. Questions and discussions relating to Weave
>> should
>> be directed to the scipy-dev mailing list (see
>> http://scipy.org/scipylib/mailing-lists.html).

If you wish to take up maintenance of weave to save it from its
abandonment, please let us know, and we will give you admin rights to
the repo. Thanks!

-- 
Robert Kern


From ndbecker2 at gmail.com  Fri May 16 08:55:30 2014
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 16 May 2014 08:55:30 -0400
Subject: [SciPy-User] DFT over narrow frequency range
Message-ID: <ll51s2$t1t$1@ger.gmane.org>

I have an application requiring a 1d DFT to be taken on a regular spaced 
frequency grid which covers a narrow range of frequencies.  I'm wondering about 
efficient ways to do this.

If the number of frequency bins is small, probably the most efficient approach 
is to directly compute the DFT.

Another possibility would be to filter and decimate the signal to the range of 
interest where an FFT could be applied.

I wonder if there are some other possibilities?  I was discovered NFFT, which 
doesn't appear to directly apply, but shows me there are many tricks I haven't 
learned.



From gregor.thalhammer at gmail.com  Fri May 16 09:24:54 2014
From: gregor.thalhammer at gmail.com (Gregor Thalhammer)
Date: Fri, 16 May 2014 15:24:54 +0200
Subject: [SciPy-User] DFT over narrow frequency range
In-Reply-To: <ll51s2$t1t$1@ger.gmane.org>
References: <ll51s2$t1t$1@ger.gmane.org>
Message-ID: <EC51B90D-01A1-47A7-A764-FC7C6C2B4706@gmail.com>


Am 16.05.2014 um 14:55 schrieb Neal Becker <ndbecker2 at gmail.com>:

> I have an application requiring a 1d DFT to be taken on a regular spaced 
> frequency grid which covers a narrow range of frequencies.  I'm wondering about 
> efficient ways to do this.
> 
> If the number of frequency bins is small, probably the most efficient approach 
> is to directly compute the DFT.
> 
> Another possibility would be to filter and decimate the signal to the range of 
> interest where an FFT could be applied.
> 
> I wonder if there are some other possibilities?  I was discovered NFFT, which 
> doesn't appear to directly apply, but shows me there are many tricks I haven't 
> learned.

You can also use the chirp-z or Bluestein-FFT algorithm to get a narrow part of the spectrum on a finer grid than with a conventional FFT (interpolating the spectrum). Gives more beaitiful results, but it is as costly as an FFT, probably not what you are looking for. 

Gregor

From ndbecker2 at gmail.com  Fri May 16 14:28:34 2014
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 16 May 2014 14:28:34 -0400
Subject: [SciPy-User] DFT over narrow frequency range
References: <ll51s2$t1t$1@ger.gmane.org>
	<EC51B90D-01A1-47A7-A764-FC7C6C2B4706@gmail.com>
Message-ID: <ll5lci$5jm$1@ger.gmane.org>

Gregor Thalhammer wrote:

> 
> Am 16.05.2014 um 14:55 schrieb Neal Becker <ndbecker2 at gmail.com>:
> 
>> I have an application requiring a 1d DFT to be taken on a regular spaced
>> frequency grid which covers a narrow range of frequencies.  I'm wondering
>> about efficient ways to do this.
>> 
>> If the number of frequency bins is small, probably the most efficient
>> approach is to directly compute the DFT.
>> 
>> Another possibility would be to filter and decimate the signal to the range
>> of interest where an FFT could be applied.
>> 
>> I wonder if there are some other possibilities?  I was discovered NFFT, which
>> doesn't appear to directly apply, but shows me there are many tricks I
>> haven't learned.
> 
> You can also use the chirp-z or Bluestein-FFT algorithm to get a narrow part
> of the spectrum on a finer grid than with a conventional FFT (interpolating
> the spectrum). Gives more beaitiful results, but it is as costly as an FFT,
> probably not what you are looking for.
> 
> Gregor

I found some nice zoomfft code in an old post:
http://scipy-user.10969.n7.nabble.com/Chirp-Z-transform-td9556.html



From pyviennacl at tsmithe.net  Sun May 18 07:56:22 2014
From: pyviennacl at tsmithe.net (Toby St Clere Smithe)
Date: Sun, 18 May 2014 12:56:22 +0100
Subject: [SciPy-User] ANN: PyViennaCL 1.0.3 -- very easy GPGPU linear algebra
Message-ID: <87wqdjz4ex.fsf@tsmithe.net>

Hello everybody,

I am pleased to announce the 1.0.3 release of PyViennaCL! This release
fixes a number of important bugs, and improves performance on nVidia
Kepler GPUs. The ChangeLog is below, and the associated ViennaCL version
is 1.5.2.


About PyViennaCL
================

*PyViennaCL* aims to make fast, powerful GPGPU and heterogeneous
scientific computing really transparently easy, especially for users
already using NumPy for representing matrices.

PyViennaCL does this by harnessing the `ViennaCL
<http://viennacl.sourceforge.net/>`_ linear algebra and numerical computation
library for GPGPU and heterogeneous systems, thereby making available to Python
programmers ViennaCL?s fast *OpenCL* and *CUDA* algorithms. PyViennaCL does
this in a way that is idiomatic and compatible with the Python community?s most
popular scientific packages, *NumPy* and *SciPy*.

PyViennaCL exposes the following functionality:

* sparse (compressed, co-ordinate, ELL, and hybrid) and dense
  (row-major and column-major) matrices, vectors and scalars on your
  compute device using OpenCL;
* standard arithmetic operations and mathematical functions;
* fast matrix products for sparse and dense matrices, and inner and
  outer products for vectors;
* direct solvers for dense triangular systems;
* iterative solvers for sparse and dense systems, using the BiCGStab,
  CG, and GMRES algorithms;
* iterative algorithms for eigenvalue estimation problems.

PyViennaCL has also been designed for straightforward use in the context
of NumPy and SciPy: PyViennaCL objects can be constructed using NumPy
arrays, and arithmetic operations and comparisons in PyViennaCL are
type-agnostic.

See the following link for documentation and example code:
http://viennacl.sourceforge.net/pyviennacl/doc/


Get PyViennaCL
==============

PyViennaCL is easily installed from PyPI.

If you are on Windows, there are binaries for Python versions 2.7, 3.2,
3.3, and 3.4.

If you are on Mac OS X and want to provide binaries, then please get in
touch! Otherwise, the installation process will build PyViennaCL from
source, which can take a while.

If you are on Debian or Ubuntu, binaries are available in Debian testing
and unstable, and Ubuntu utopic. Just run::

    apt-get install python-pyviennacl python3-pyviennacl

To install PyViennaCL from PyPI, make sure you've got a recent version
of the *pip* package manager, and run::

    pip install pyviennacl


Bugs and support
================

If you find a problem in PyViennaCL, then please report it at
https://github.com/viennacl/pyviennacl-dev/issues


ChangeLog
=========

2014-05-15  Toby St Clere Smithe  <pyviennacl at tsmithe.net>

	* Release 1.0.3.

	* Update external/viennacl-dev to version 1.5.2.
	  [91b7589a8fccc92927306e0ae3e061d85ac1ae93]

	  This contains two important fixes: one for a build failure on
	  Windows (PyViennaCL issue #17) relating to the re-enabling of the
	  Lanczos algorithm in 1.0.2, and one for an issue relating to
	  missing support for matrix transposition in the ViennaCL scheduler
	  (PyViennaCL issue #19, ViennaCL issue #73).

	  This release is also benefitial for performance on nVidia Kepler
	  GPUs, increasing the performance of matrix-matrix multiplications
	  to 600 GFLOPs in single precision on a GeForce GTX 680.

	* Fix bug when using integers in matrix and vector index key
	  [dbb1911fd788e66475f5717c1692be49d083a506]

	* Fix slicing of dense matrices (issue #18).
	  [9c745710ebc2a1066c7074b6c5de61b227017cc6]

	* Enable test for matrix transposition
	  [9e951103b883a3848aa2115df3edce73d347c09b]

	* Add non-square matrix-vector product test
	  [21dd29cd10ebe02a96ee23c20ee55401bc6c874f]

2014-05-06  Toby St Clere Smithe  <pyviennacl at tsmithe.net>

	* Release 1.0.2.

	* Re-enable Lanczos algorithm for eigenvalues (issue #11).
	[cbfb41fca3fb1f3db42fd7b3ccb8332b701d1e20]

	* Enable eigenvalue computations for compressed and coordinate
  	matrices.
	[8ecee3b200a92ae99b72653a823c1f60e62f75dd]

	* Fix matrix-vector product for non-square matrices (issue #13).
	[bf3aa2bf91339df72b6f7561afaf8b12aad57cda]

	* Link against rt on Linux (issue #12).
	[d5784b62b353ebbfd78fe1335fd96971b5089f53]




Best regards,

-- 
Toby St Clere Smithe
http://tsmithe.net


From ndbecker2 at gmail.com  Mon May 19 08:06:42 2014
From: ndbecker2 at gmail.com (Neal Becker)
Date: Mon, 19 May 2014 08:06:42 -0400
Subject: [SciPy-User] ANN: PyViennaCL 1.0.3 -- very easy GPGPU linear
	algebra
References: <87wqdjz4ex.fsf@tsmithe.net>
	<31928456-F70E-45AF-A603-4DD764007143@iap.fr>
Message-ID: <llcs4j$mfh$2@ger.gmane.org>

Marquette Jean-Baptiste wrote:

> Hi Toby,
> 
>> If you are on Mac OS X and want to provide binaries, then please get in
>> touch! Otherwise, the installation process will build PyViennaCL from
>> source, which can take a while.
> 
> I could contribute, though I just opened an issue about a compilation error on
> Mavericks 10.9.3. Curiously, the source install seems OK on Lion 10.7.5
> 
> Cheers,
> JB

Has anyone built on fedora?

What are the build requirements?



From pyviennacl at tsmithe.net  Mon May 19 08:32:40 2014
From: pyviennacl at tsmithe.net (Toby St Clere Smithe)
Date: Mon, 19 May 2014 13:32:40 +0100
Subject: [SciPy-User] ANN: PyViennaCL 1.0.3 -- very easy GPGPU linear
	algebra
References: <87wqdjz4ex.fsf@tsmithe.net>
	<31928456-F70E-45AF-A603-4DD764007143@iap.fr>
	<llcs4j$mfh$2@ger.gmane.org>
Message-ID: <87r43qx82f.fsf@tsmithe.net>

Hi Neal,

Neal Becker <ndbecker2 at gmail.com> writes:
> Has anyone built on fedora?
>
> What are the build requirements?

You'll need NumPy (>= 1.7) and an OpenCL library installed, but then you
should just be able to run `pip install pyviennacl`. Or you can grab the
tree from github[1] and do it the manual way with setup.py.

[1] https://github.com/viennacl/pyviennacl-dev

Cheers,

-- 
Toby St Clere Smithe
http://tsmithe.net



From pyviennacl at tsmithe.net  Mon May 19 19:21:36 2014
From: pyviennacl at tsmithe.net (Toby St Clere Smithe)
Date: Tue, 20 May 2014 00:21:36 +0100
Subject: [SciPy-User] ANN: PyViennaCL 1.0.3 -- very easy GPGPU linear
	algebra
In-Reply-To: <87wqdjz4ex.fsf@tsmithe.net> (Toby St Clere Smithe's message of
	"Sun, 18 May 2014 12:56:22 +0100")
References: <87wqdjz4ex.fsf@tsmithe.net>
Message-ID: <87mwedwe0v.fsf@tsmithe.net>

Just to say that, thanks to Matthew Brett, binary wheels for Mac OS X
are now available, for Python versions 2.7, 3.3, and 3.4. This means
that, if you're on that platform, you won't have to build from source!
As usual, just run `pip install pyviennacl`, and please report any
issues you encounter to https://github.com/viennacl/pyviennacl-dev/issues !

Thanks,

Toby


Toby St Clere Smithe <pyviennacl at tsmithe.net> writes:
> Hello everybody,
>
> I am pleased to announce the 1.0.3 release of PyViennaCL! This release
> fixes a number of important bugs, and improves performance on nVidia
> Kepler GPUs. The ChangeLog is below, and the associated ViennaCL version
> is 1.5.2.
>
>
> About PyViennaCL
> ================
>
> *PyViennaCL* aims to make fast, powerful GPGPU and heterogeneous
> scientific computing really transparently easy, especially for users
> already using NumPy for representing matrices.
>
> PyViennaCL does this by harnessing the `ViennaCL
> <http://viennacl.sourceforge.net/>`_ linear algebra and numerical computation
> library for GPGPU and heterogeneous systems, thereby making available to Python
> programmers ViennaCL?s fast *OpenCL* and *CUDA* algorithms. PyViennaCL does
> this in a way that is idiomatic and compatible with the Python community?s most
> popular scientific packages, *NumPy* and *SciPy*.
>
> PyViennaCL exposes the following functionality:
>
> * sparse (compressed, co-ordinate, ELL, and hybrid) and dense
>   (row-major and column-major) matrices, vectors and scalars on your
>   compute device using OpenCL;
> * standard arithmetic operations and mathematical functions;
> * fast matrix products for sparse and dense matrices, and inner and
>   outer products for vectors;
> * direct solvers for dense triangular systems;
> * iterative solvers for sparse and dense systems, using the BiCGStab,
>   CG, and GMRES algorithms;
> * iterative algorithms for eigenvalue estimation problems.
>
> PyViennaCL has also been designed for straightforward use in the context
> of NumPy and SciPy: PyViennaCL objects can be constructed using NumPy
> arrays, and arithmetic operations and comparisons in PyViennaCL are
> type-agnostic.
>
> See the following link for documentation and example code:
> http://viennacl.sourceforge.net/pyviennacl/doc/
>
>
> Get PyViennaCL
> ==============
>
> PyViennaCL is easily installed from PyPI.
>
> If you are on Windows, there are binaries for Python versions 2.7, 3.2,
> 3.3, and 3.4.
>
> If you are on Mac OS X and want to provide binaries, then please get in
> touch! Otherwise, the installation process will build PyViennaCL from
> source, which can take a while.
>
> If you are on Debian or Ubuntu, binaries are available in Debian testing
> and unstable, and Ubuntu utopic. Just run::
>
>     apt-get install python-pyviennacl python3-pyviennacl
>
> To install PyViennaCL from PyPI, make sure you've got a recent version
> of the *pip* package manager, and run::
>
>     pip install pyviennacl
>
>
> Bugs and support
> ================
>
> If you find a problem in PyViennaCL, then please report it at
> https://github.com/viennacl/pyviennacl-dev/issues
>
>
> ChangeLog
> =========
>
> 2014-05-15  Toby St Clere Smithe  <pyviennacl at tsmithe.net>
>
> 	* Release 1.0.3.
>
> 	* Update external/viennacl-dev to version 1.5.2.
> 	  [91b7589a8fccc92927306e0ae3e061d85ac1ae93]
>
> 	  This contains two important fixes: one for a build failure on
> 	  Windows (PyViennaCL issue #17) relating to the re-enabling of the
> 	  Lanczos algorithm in 1.0.2, and one for an issue relating to
> 	  missing support for matrix transposition in the ViennaCL scheduler
> 	  (PyViennaCL issue #19, ViennaCL issue #73).
>
> 	  This release is also benefitial for performance on nVidia Kepler
> 	  GPUs, increasing the performance of matrix-matrix multiplications
> 	  to 600 GFLOPs in single precision on a GeForce GTX 680.
>
> 	* Fix bug when using integers in matrix and vector index key
> 	  [dbb1911fd788e66475f5717c1692be49d083a506]
>
> 	* Fix slicing of dense matrices (issue #18).
> 	  [9c745710ebc2a1066c7074b6c5de61b227017cc6]
>
> 	* Enable test for matrix transposition
> 	  [9e951103b883a3848aa2115df3edce73d347c09b]
>
> 	* Add non-square matrix-vector product test
> 	  [21dd29cd10ebe02a96ee23c20ee55401bc6c874f]
>
> 2014-05-06  Toby St Clere Smithe  <pyviennacl at tsmithe.net>
>
> 	* Release 1.0.2.
>
> 	* Re-enable Lanczos algorithm for eigenvalues (issue #11).
> 	[cbfb41fca3fb1f3db42fd7b3ccb8332b701d1e20]
>
> 	* Enable eigenvalue computations for compressed and coordinate
>   	matrices.
> 	[8ecee3b200a92ae99b72653a823c1f60e62f75dd]
>
> 	* Fix matrix-vector product for non-square matrices (issue #13).
> 	[bf3aa2bf91339df72b6f7561afaf8b12aad57cda]
>
> 	* Link against rt on Linux (issue #12).
> 	[d5784b62b353ebbfd78fe1335fd96971b5089f53]
>
>
>
>
> Best regards,

-- 
Toby St Clere Smithe
http://tsmithe.net


From franck.dernoncourt at gmail.com  Mon May 19 23:22:39 2014
From: franck.dernoncourt at gmail.com (Franck Dernoncourt)
Date: Mon, 19 May 2014 23:22:39 -0400
Subject: [SciPy-User] Where can I see the list of built-in wavelet functions
 that I can pass to scipy.signal.cwt?
Message-ID: <CA+e8+rgdvXr2veHnHqDCmOsxuOg7xLdyDwEM-Qt2-QJS7jnFhg@mail.gmail.com>

Hi,

scipy.signal.cwt's documentation says (
http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.signal.cwt.html
):

scipy.signal.cwt(data, wavelet, widths):

wavelet : function Wavelet function, which should take 2 arguments. The
> first argument is the number of points that the returned vector will have
> (len(wavelet(width,length)) == length). The second is a width parameter,
> defining the size of the wavelet (e.g. standard deviation of a gaussian).
> See ricker, which satisfies these requirements.wavelet : function Wavelet
> function, *which should take 2 arguments*.


Beyond scipy.signal.ricket, what are the other built-in wavelet functions
that I can pass to scipy.signal.cwt?

I see in scipy / scipy / signal / wavelets.py (
https://github.com/scipy/scipy/blob/v0.13.0/scipy/signal/wavelets.py)

 __all__ = ['daub', 'qmf', 'cascade', 'morlet', 'ricker', 'cwt']


but looking at the arguments of each of those wavelet functions, only
ricket seems to work with scipy.signal.cwt(data, wavelet, widths) (as only
ricker takes precisely 2 arguments).

Thanks,

----
Franck Dernoncourt
http://francky.me
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140519/3e800e0b/attachment.html>

From davidmenhur at gmail.com  Tue May 20 04:33:12 2014
From: davidmenhur at gmail.com (=?UTF-8?B?RGHPgGlk?=)
Date: Tue, 20 May 2014 10:33:12 +0200
Subject: [SciPy-User] Where can I see the list of built-in wavelet
 functions that I can pass to scipy.signal.cwt?
In-Reply-To: <CA+e8+rgdvXr2veHnHqDCmOsxuOg7xLdyDwEM-Qt2-QJS7jnFhg@mail.gmail.com>
References: <CA+e8+rgdvXr2veHnHqDCmOsxuOg7xLdyDwEM-Qt2-QJS7jnFhg@mail.gmail.com>
Message-ID: <CAJhcF=1fLA17SQh_kcRnXS+MHXHq1OESQXWZ5pxU5RXERjVkWw@mail.gmail.com>

I found the module for CWT quite confusing, so I rolled my own:

https://github.com/Dapid/fast-pycwt

It is built for speed (I got my running time from 4 h down to 20 min). It
is not thoroughly tested, and it is limited to single and double; but for
me it is in a "good enough" state.


/David.


On 20 May 2014 05:22, Franck Dernoncourt <franck.dernoncourt at gmail.com>wrote:

> Hi,
>
> scipy.signal.cwt's documentation says (
> http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.signal.cwt.html
> ):
>
> scipy.signal.cwt(data, wavelet, widths):
>
> wavelet : function Wavelet function, which should take 2 arguments. The
>> first argument is the number of points that the returned vector will have
>> (len(wavelet(width,length)) == length). The second is a width parameter,
>> defining the size of the wavelet (e.g. standard deviation of a gaussian).
>> See ricker, which satisfies these requirements.wavelet : function Wavelet
>> function, *which should take 2 arguments*.
>
>
> Beyond scipy.signal.ricket, what are the other built-in wavelet functions
> that I can pass to scipy.signal.cwt?
>
> I see in scipy / scipy / signal / wavelets.py (
> https://github.com/scipy/scipy/blob/v0.13.0/scipy/signal/wavelets.py)
>
>  __all__ = ['daub', 'qmf', 'cascade', 'morlet', 'ricker', 'cwt']
>
>
> but looking at the arguments of each of those wavelet functions, only
> ricket seems to work with scipy.signal.cwt(data, wavelet, widths) (as only
> ricker takes precisely 2 arguments).
>
> Thanks,
>
> ----
> Franck Dernoncourt
> http://francky.me
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140520/7da5dbdd/attachment.html>

From aaron.oleary at gmail.com  Tue May 20 07:33:58 2014
From: aaron.oleary at gmail.com (Aaron O'Leary)
Date: Tue, 20 May 2014 12:33:58 +0100
Subject: [SciPy-User] Where can I see the list of built-in wavelet
 functions that I can pass to scipy.signal.cwt?
In-Reply-To: <CAJhcF=1fLA17SQh_kcRnXS+MHXHq1OESQXWZ5pxU5RXERjVkWw@mail.gmail.com>
References: <CA+e8+rgdvXr2veHnHqDCmOsxuOg7xLdyDwEM-Qt2-QJS7jnFhg@mail.gmail.com>
	<CAJhcF=1fLA17SQh_kcRnXS+MHXHq1OESQXWZ5pxU5RXERjVkWw@mail.gmail.com>
Message-ID: <20140520113358.GA20871@tk422>

> On 20 May 2014 05:22, Franck Dernoncourt <franck.dernoncourt at gmail.com>wrote:
> > Beyond scipy.signal.ricket, what are the other built-in wavelet functions
> > that I can pass to scipy.signal.cwt?

On Tue 20 May, Da?id wrote:
> I found the module for CWT quite confusing, so I rolled my own:
> 
> https://github.com/Dapid/fast-pycwt

you might also find my version useful:

https://github.com/aaren/wavelets

I also found scipy wavelets confusing. My version includes a faster cwt
that can take wavelets expressed in either frequency or time.

I found it more intuitive to have wavelet functions that take
time/frequency and width as arguments rather than the present method
(I prefer thinking in real space rather than sample space).

Presently, the morlet wavelet that comes with scipy,
`scipy.signal.wavelets.morlet`, cannot be used as input to cwt. This is
unfortunate I think.

Additionally, the present cwt doesn't allow complex output. This doesn't
make a difference for ricker but wavelet functions are complex in
general.

My modified 'cwt' method is here:

https://github.com/aaren/wavelets/blob/master/wavelets.py#L15

It can accept wavelet functions defined in time or frequency space,
uses fftconvolve, and allows complex output.

My background on this is based on a reading of Torrence and Compo:

Torrence and Compo, 'A Practical Guide to Wavelet Analysis' (BAMS, 1998)

http://paos.colorado.edu/research/wavelets/

hope that helps a bit,

aaron


From klaus.schaefers at gmail.com  Tue May 20 11:28:41 2014
From: klaus.schaefers at gmail.com (Klausen Schaefersinho)
Date: Tue, 20 May 2014 17:28:41 +0200
Subject: [SciPy-User] Gamma Distribution
Message-ID: <CADv9Q=eNooWA2vyCAwPTmK-PverCNiJveerb9pj57mzpBFHn9A@mail.gmail.com>

Hi,

I have a short questions with regards to the parametrization of the gamma
distribution in the scipy.stats module. According to wikipedia, there are
three common ways to parameterize the gamma distrubution:


   1. With a shape parameter <http://en.wikipedia.org/wiki/Shape_parameter>
   *k* and a scale parameter <http://en.wikipedia.org/wiki/Scale_parameter>?.
   2. With a shape parameter *?* = *k* and an inverse scale parameter
   ? = 1/?, called a rate parameter<http://en.wikipedia.org/wiki/Rate_parameter>
   .
   3. With a shape parameter *k* and a mean parameter ? = *k*/?.


Which one is used in Scipy and how do those parameters map to the a ,
location and shape?

Best Regards,

Klaus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140520/ad6c8657/attachment.html>

From warren.weckesser at gmail.com  Tue May 20 11:48:55 2014
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Tue, 20 May 2014 11:48:55 -0400
Subject: [SciPy-User] Gamma Distribution
In-Reply-To: <CADv9Q=eNooWA2vyCAwPTmK-PverCNiJveerb9pj57mzpBFHn9A@mail.gmail.com>
References: <CADv9Q=eNooWA2vyCAwPTmK-PverCNiJveerb9pj57mzpBFHn9A@mail.gmail.com>
Message-ID: <CAGzF1ufdcptvuHP0ytHsbb_0LGiZt7aDzCKLpuCX7tNV_kyYyg@mail.gmail.com>

On Tue, May 20, 2014 at 11:28 AM, Klausen Schaefersinho <
klaus.schaefers at gmail.com> wrote:

> Hi,
>
> I have a short questions with regards to the parametrization of the gamma
> distribution in the scipy.stats module. According to wikipedia, there are
> three common ways to parameterize the gamma distrubution:
>
>
>    1. With a shape parameter<http://en.wikipedia.org/wiki/Shape_parameter>
>    *k* and a scale parameter<http://en.wikipedia.org/wiki/Scale_parameter>?.
>    2. With a shape parameter *?* = *k* and an inverse scale parameter
>    ? = 1/?, called a rate parameter<http://en.wikipedia.org/wiki/Rate_parameter>
>    .
>    3. With a shape parameter *k* and a mean parameter ? = *k*/?.
>
>
> Which one is used in Scipy and how do those parameters map to the a ,
> location and shape?
>
>
scipy.stats.gamma uses the first parametrization.  wikipedia's `k`
corresponds to the shape parameter `a`, and `theta` corresponds to the
scale parameter.  Like all of the distributions in scipy.stats, gamma also
has the location parameter `loc`, but it is usually not used.

This script reproduces the first plot on the wikipedia page.


import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import gamma


x = np.linspace(0, 20, 201)

params = [(1, 2, 'r'), (2, 2, 'g'), (3, 2, 'b'), (5, 1, 'c'), (9, 0.5, 'y')]
for k, scale, clr in params:
    lbl = r"k = %3.1f, $\theta$ = %3.1f" % (k, scale)
    plt.plot(x, gamma.pdf(x, k, scale=scale), clr, label=lbl)
plt.title("Probability density function")
plt.legend()

plt.show()


Warren


Best Regards,
>
> Klaus
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140520/2b039cac/attachment.html>

From chocold12 at gmail.com  Wed May 21 14:28:19 2014
From: chocold12 at gmail.com (lily li)
Date: Wed, 21 May 2014 12:28:19 -0600
Subject: [SciPy-User] about opening netcdf file
Message-ID: <CAN5afy-7-_BCVuaiWcrDnFRN1U12P6we8rnWrH0sUc6d1AGB0A@mail.gmail.com>

Hi scipy users,

I want to ask that how to open a NetCDF file? I tried to import netCDF4,
but it says no such module. So which module or package to install
beforehand? Thanks very much.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140521/e5b9a5d3/attachment.html>

From guziy.sasha at gmail.com  Wed May 21 14:31:14 2014
From: guziy.sasha at gmail.com (Oleksandr Huziy)
Date: Wed, 21 May 2014 14:31:14 -0400
Subject: [SciPy-User] about opening netcdf file
In-Reply-To: <CAN5afy-7-_BCVuaiWcrDnFRN1U12P6we8rnWrH0sUc6d1AGB0A@mail.gmail.com>
References: <CAN5afy-7-_BCVuaiWcrDnFRN1U12P6we8rnWrH0sUc6d1AGB0A@mail.gmail.com>
Message-ID: <CAN3p1sUfy3jsVz+H5GYfeEAtGkWLd--HY9neJS9oxzoPDFEbKg@mail.gmail.com>

You need to install netCDF4:


pip install netCDF4


You will have to install netcdf and hdf separately. On ubuntu those are
installed using apt-get. And if you do not have pip yet you will also need
to install it.



2014-05-21 14:28 GMT-04:00 lily li <chocold12 at gmail.com>:

> Hi scipy users,
>
> I want to ask that how to open a NetCDF file? I tried to import netCDF4,
> but it says no such module. So which module or package to install
> beforehand? Thanks very much.
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Sasha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140521/9d5b420d/attachment.html>

From chocold12 at gmail.com  Wed May 21 14:32:43 2014
From: chocold12 at gmail.com (lily li)
Date: Wed, 21 May 2014 12:32:43 -0600
Subject: [SciPy-User] about opening netcdf file
In-Reply-To: <CAN3p1sUfy3jsVz+H5GYfeEAtGkWLd--HY9neJS9oxzoPDFEbKg@mail.gmail.com>
References: <CAN5afy-7-_BCVuaiWcrDnFRN1U12P6we8rnWrH0sUc6d1AGB0A@mail.gmail.com>
	<CAN3p1sUfy3jsVz+H5GYfeEAtGkWLd--HY9neJS9oxzoPDFEbKg@mail.gmail.com>
Message-ID: <CAN5afy9hVFGD1cyf8Z1DPBOqxh0t0Wp5menWRHSVLE5WpBjE_Q@mail.gmail.com>

How to do in python in windows? I'm not working on ubuntu. Thanks again.


On Wed, May 21, 2014 at 12:31 PM, Oleksandr Huziy <guziy.sasha at gmail.com>wrote:

> You need to install netCDF4:
>
>
> pip install netCDF4
>
>
> You will have to install netcdf and hdf separately. On ubuntu those are
> installed using apt-get. And if you do not have pip yet you will also need
> to install it.
>
>
>
> 2014-05-21 14:28 GMT-04:00 lily li <chocold12 at gmail.com>:
>
>> Hi scipy users,
>>
>> I want to ask that how to open a NetCDF file? I tried to import netCDF4,
>> but it says no such module. So which module or package to install
>> beforehand? Thanks very much.
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
>
> --
> Sasha
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140521/6602ab82/attachment.html>

From guziy.sasha at gmail.com  Wed May 21 14:39:35 2014
From: guziy.sasha at gmail.com (Oleksandr Huziy)
Date: Wed, 21 May 2014 14:39:35 -0400
Subject: [SciPy-User] about opening netcdf file
In-Reply-To: <CAN5afy9hVFGD1cyf8Z1DPBOqxh0t0Wp5menWRHSVLE5WpBjE_Q@mail.gmail.com>
References: <CAN5afy-7-_BCVuaiWcrDnFRN1U12P6we8rnWrH0sUc6d1AGB0A@mail.gmail.com>
	<CAN3p1sUfy3jsVz+H5GYfeEAtGkWLd--HY9neJS9oxzoPDFEbKg@mail.gmail.com>
	<CAN5afy9hVFGD1cyf8Z1DPBOqxh0t0Wp5menWRHSVLE5WpBjE_Q@mail.gmail.com>
Message-ID: <CAN3p1sUUEQWEatR7P6CBcAzdRoX6Jjh6C12nCx+3bTo7An7b0g@mail.gmail.com>

Never done it for windows but suppose it is easier:


select the version you need download and double click
http://www.lfd.uci.edu/~gohlke/pythonlibs/#netcdf4


Cheers


2014-05-21 14:32 GMT-04:00 lily li <chocold12 at gmail.com>:

> How to do in python in windows? I'm not working on ubuntu. Thanks again.
>
>
> On Wed, May 21, 2014 at 12:31 PM, Oleksandr Huziy <guziy.sasha at gmail.com>wrote:
>
>> You need to install netCDF4:
>>
>>
>> pip install netCDF4
>>
>>
>> You will have to install netcdf and hdf separately. On ubuntu those are
>> installed using apt-get. And if you do not have pip yet you will also need
>> to install it.
>>
>>
>>
>> 2014-05-21 14:28 GMT-04:00 lily li <chocold12 at gmail.com>:
>>
>>> Hi scipy users,
>>>
>>> I want to ask that how to open a NetCDF file? I tried to import netCDF4,
>>> but it says no such module. So which module or package to install
>>> beforehand? Thanks very much.
>>>
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>>
>>>
>>
>>
>> --
>> Sasha
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>


-- 
Sasha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140521/5b8a27c4/attachment.html>

From parrenin at ujf-grenoble.fr  Thu May 22 10:53:50 2014
From: parrenin at ujf-grenoble.fr (=?UTF-8?Q?Fr=C3=A9d=C3=A9ric_Parrenin?=)
Date: Thu, 22 May 2014 16:53:50 +0200
Subject: [SciPy-User] interpolation (using averaging) on a staircase function
Message-ID: <CABrqr5pdjo5S3zyVy=cFDQ8DaT+YnfpEs81cuL_h+-znn6kZNg@mail.gmail.com>

Dear all,

I have a serie (x,y) which I would like to interpolate at new abscissas xp,
but not using a simple interpolation. I would like to the yp to be equal to
the average over each interval (xp[i],xp[i+1]) of the staircase function
defined by the (x,y) serie.

How to do that using scipy?

Best regards,

Fr?d?ric Parrenin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140522/091437ab/attachment.html>

From parrenin at ujf-grenoble.fr  Thu May 22 11:20:31 2014
From: parrenin at ujf-grenoble.fr (=?UTF-8?Q?Fr=C3=A9d=C3=A9ric_Parrenin?=)
Date: Thu, 22 May 2014 17:20:31 +0200
Subject: [SciPy-User] leastsq and multiprocessing
In-Reply-To: <l91dvi$5qf$1@ger.gmane.org>
References: <CABrqr5oxiPw2QmpQEgRVWes03uav4V4_iGuiresP253s8=UksQ@mail.gmail.com>
	<CAMMTP+Cjghr7=0OhKf5joqRMGvv2TLFVG+kfQOg2Bxip=ULY5g@mail.gmail.com>
	<CA+7ESboWrQw_VvxpauTAEJq5TY1-c81RoLZwnLSJEdZrmf0xBA@mail.gmail.com>
	<l91dvi$5qf$1@ger.gmane.org>
Message-ID: <CABrqr5pZYYYceLnbiccmaQi2NMhaAu9yiwJOVjgPV5_9umHpDA@mail.gmail.com>

Dear all,

Coming back to an old thread...

I tried Jeremy's method since it is the easiest to implement.
Below is the Dfun function I provided to leastsq.
In my experiment, I used a pool of 6 since I have 8 cores in my PC.

However, the computer becomes extremely slow, almost unusable, during the
experiment.
Do you know why this happens?

Best regards,

Fr?d?ric

________________

def Dres(var):
    """Calculate derivatives for each parameter using pool."""
    zeropred = residuals(var)
    derivparams = []
    results=[]
    delta = m.sqrt(np.finfo(float).eps) #Stolen from the leastsq code
    for i in range(len(var)):
        copy = np.array(var)
        copy[i] += delta
        derivparams.append(copy)
#        results.append(residuals(derivparams))
    if __name__ == "__main__":
        pool = multiprocessing.Pool(nb_nodes)
    results = pool.map(residuals, derivparams)
    derivs = [ (r - zeropred)/delta for r in results ]
    return derivs



2013-12-20 13:43 GMT+01:00 Jeremy Sanders <jeremy at jeremysanders.net>:

> Matt Newville wrote:
>
> > Currently, scipy's leastsq() simply calls the Fortran lmdif() (for
> > finite-diff Jacobian).   I think replacing fdjac2() with a
> > multiprocessing version would require reimplementing both lmdif() and
> > fdjac2(), probably using cython.  If calls to MINPACKs lmpar() and
> > qrfac() could be left untouched, this translation does not look too
> > insane -- the two routines lmdif() and fdjac2() themselves are not
> > that complicated.   It would be a fair amount of work, and I cannot
> > volunteer to do this myself any time soon.  But, I do think it
> > actually would improve the speed of leastsq() for many use cases.
>
> Computing the Jacobian using using multiprocessing definitely helps the
> speed. I wrote the unrated answer (xioxox) there which shows how to do it
> in
> Python.
>
> Jeremy
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140522/f94c94b6/attachment.html>

From newville at cars.uchicago.edu  Thu May 22 22:23:50 2014
From: newville at cars.uchicago.edu (Matt Newville)
Date: Thu, 22 May 2014 21:23:50 -0500
Subject: [SciPy-User] leastsq and multiprocessing
In-Reply-To: <CABrqr5pZYYYceLnbiccmaQi2NMhaAu9yiwJOVjgPV5_9umHpDA@mail.gmail.com>
References: <CABrqr5oxiPw2QmpQEgRVWes03uav4V4_iGuiresP253s8=UksQ@mail.gmail.com>
	<CAMMTP+Cjghr7=0OhKf5joqRMGvv2TLFVG+kfQOg2Bxip=ULY5g@mail.gmail.com>
	<CA+7ESboWrQw_VvxpauTAEJq5TY1-c81RoLZwnLSJEdZrmf0xBA@mail.gmail.com>
	<l91dvi$5qf$1@ger.gmane.org>
	<CABrqr5pZYYYceLnbiccmaQi2NMhaAu9yiwJOVjgPV5_9umHpDA@mail.gmail.com>
Message-ID: <CA+7ESbobejjW=ML+k8mN3XnkiuQZap1DGY7v9bsjxxPGLyMAeA@mail.gmail.com>

Hi Frederic,

On Thu, May 22, 2014 at 10:20 AM, Fr?d?ric Parrenin <
parrenin at ujf-grenoble.fr> wrote:

> Dear all,
>
> Coming back to an old thread...
>
> I tried Jeremy's method since it is the easiest to implement.
> Below is the Dfun function I provided to leastsq.
> In my experiment, I used a pool of 6 since I have 8 cores in my PC.
>
> However, the computer becomes extremely slow, almost unusable, during the
> experiment.
> Do you know why this happens?
>
> Best regards,
>
> Fr?d?ric
>

Yes, my observation, based on the code at

https://github.com/newville/scipy/commit/3d0ac1da3bcd1d34a1bec8226ea0284f04fcb5dc

was that there was about a 10x performance hit.    So, similar to your
observations.

This approach assumes that the cost of setting up multiple processes is
small compared to the execution time of the objective function itself.  It
also assumes that having a Jacobian function in Python (as compared to
Fortran) is a small performance hit.  Again, this is more likely to be true
for a time-consuming objective function, and almost certainly not true for
any small test case.

I could be persuaded that this approach is still a reasonable idea, but (at
least if implemented in pure Python) all the evidence is that it is much
slower.  Using Cython may help, but I have not tried this.

Any multiprocessing approach that includes calling the objective function
from different processes is going to be limited by the "picklability"
issue. To me, this is a fairly significant limitation.  I've been lead to
believe that the Mystic framework may have worked around this problem, but
I don't know the details.

Others have suggested that doing the QR factorization with multiprocessing
would be the better approach.  This seems worth trying, but, In my
experience, the bulk of the time is actually spent in the objective
function.

--Matt Newville
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140522/7901e950/attachment.html>

From parrenin at ujf-grenoble.fr  Fri May 23 02:52:32 2014
From: parrenin at ujf-grenoble.fr (=?UTF-8?Q?Fr=C3=A9d=C3=A9ric_Parrenin?=)
Date: Fri, 23 May 2014 08:52:32 +0200
Subject: [SciPy-User] leastsq and multiprocessing
In-Reply-To: <CA+7ESbobejjW=ML+k8mN3XnkiuQZap1DGY7v9bsjxxPGLyMAeA@mail.gmail.com>
References: <CABrqr5oxiPw2QmpQEgRVWes03uav4V4_iGuiresP253s8=UksQ@mail.gmail.com>
	<CAMMTP+Cjghr7=0OhKf5joqRMGvv2TLFVG+kfQOg2Bxip=ULY5g@mail.gmail.com>
	<CA+7ESboWrQw_VvxpauTAEJq5TY1-c81RoLZwnLSJEdZrmf0xBA@mail.gmail.com>
	<l91dvi$5qf$1@ger.gmane.org>
	<CABrqr5pZYYYceLnbiccmaQi2NMhaAu9yiwJOVjgPV5_9umHpDA@mail.gmail.com>
	<CA+7ESbobejjW=ML+k8mN3XnkiuQZap1DGY7v9bsjxxPGLyMAeA@mail.gmail.com>
Message-ID: <CABrqr5qA7Vfix1uejfs1y2bAUd=Hzv7USmFtLLNLKqDhE4Gm5Q@mail.gmail.com>

Answering to my own question:
Actually, the same code runs on debian 7 instead of ubuntu 13.10 does not
slow down my computer. So this may be an ubuntu-specific problem.

For the gain, my program runs in 545 s on one core and in 123 s using 10
cores.
So it seems there is a factor of 2 performance hit in this case, this is
not two bad.

Best regards,

Fr?d?ric Parrenin




2014-05-23 4:23 GMT+02:00 Matt Newville <newville at cars.uchicago.edu>:

> Hi Frederic,
>
> On Thu, May 22, 2014 at 10:20 AM, Fr?d?ric Parrenin <
> parrenin at ujf-grenoble.fr> wrote:
>
>> Dear all,
>>
>> Coming back to an old thread...
>>
>> I tried Jeremy's method since it is the easiest to implement.
>> Below is the Dfun function I provided to leastsq.
>> In my experiment, I used a pool of 6 since I have 8 cores in my PC.
>>
>> However, the computer becomes extremely slow, almost unusable, during the
>> experiment.
>> Do you know why this happens?
>>
>> Best regards,
>>
>> Fr?d?ric
>>
>
> Yes, my observation, based on the code at
>
> https://github.com/newville/scipy/commit/3d0ac1da3bcd1d34a1bec8226ea0284f04fcb5dc
>
> was that there was about a 10x performance hit.    So, similar to your
> observations.
>
> This approach assumes that the cost of setting up multiple processes is
> small compared to the execution time of the objective function itself.  It
> also assumes that having a Jacobian function in Python (as compared to
> Fortran) is a small performance hit.  Again, this is more likely to be true
> for a time-consuming objective function, and almost certainly not true for
> any small test case.
>
> I could be persuaded that this approach is still a reasonable idea, but
> (at least if implemented in pure Python) all the evidence is that it is
> much slower.  Using Cython may help, but I have not tried this.
>
> Any multiprocessing approach that includes calling the objective function
> from different processes is going to be limited by the "picklability"
> issue. To me, this is a fairly significant limitation.  I've been lead to
> believe that the Mystic framework may have worked around this problem, but
> I don't know the details.
>
> Others have suggested that doing the QR factorization with multiprocessing
> would be the better approach.  This seems worth trying, but, In my
> experience, the bulk of the time is actually spent in the objective
> function.
>
> --Matt Newville
>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140523/d6c90d5e/attachment.html>

From cimrman3 at ntc.zcu.cz  Fri May 23 07:24:56 2014
From: cimrman3 at ntc.zcu.cz (Robert Cimrman)
Date: Fri, 23 May 2014 13:24:56 +0200
Subject: [SciPy-User] ANN: SfePy 2014.2
Message-ID: <537F3008.6090804@ntc.zcu.cz>

I am pleased to announce release 2014.2 of SfePy.

Description
-----------
SfePy (simple finite elements in Python) is a software for solving
systems of coupled partial differential equations by the finite element
method. The code is based on NumPy and SciPy packages. It is distributed
under the new BSD license.

This release brings a preliminary support for isogeometric analysis - a
recently developed computational approach that allows using the NURBS-based
domain description from CAD design tools also for approximation purposes
similar to the finite element method.

Home page: http://sfepy.org
Mailing list: http://groups.google.com/group/sfepy-devel
Git (source) repository, issue tracker, wiki: http://github.com/sfepy

Highlights of this release
--------------------------
- preliminary support for isogeometric analysis
- improved post-processing and visualization script for time-dependent problems
   with adaptive time steps
- three new terms

For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1
(rather long and technical).

Best regards,
Robert Cimrman and Contributors (*)

(*) Contributors to this release (alphabetical order):

Vladim?r Luke?


From parrenin at ujf-grenoble.fr  Mon May 26 10:20:20 2014
From: parrenin at ujf-grenoble.fr (=?UTF-8?Q?Fr=C3=A9d=C3=A9ric_Parrenin?=)
Date: Mon, 26 May 2014 16:20:20 +0200
Subject: [SciPy-User] leastsq and multiprocessing
In-Reply-To: <CABrqr5qA7Vfix1uejfs1y2bAUd=Hzv7USmFtLLNLKqDhE4Gm5Q@mail.gmail.com>
References: <CABrqr5oxiPw2QmpQEgRVWes03uav4V4_iGuiresP253s8=UksQ@mail.gmail.com>
	<CAMMTP+Cjghr7=0OhKf5joqRMGvv2TLFVG+kfQOg2Bxip=ULY5g@mail.gmail.com>
	<CA+7ESboWrQw_VvxpauTAEJq5TY1-c81RoLZwnLSJEdZrmf0xBA@mail.gmail.com>
	<l91dvi$5qf$1@ger.gmane.org>
	<CABrqr5pZYYYceLnbiccmaQi2NMhaAu9yiwJOVjgPV5_9umHpDA@mail.gmail.com>
	<CA+7ESbobejjW=ML+k8mN3XnkiuQZap1DGY7v9bsjxxPGLyMAeA@mail.gmail.com>
	<CABrqr5qA7Vfix1uejfs1y2bAUd=Hzv7USmFtLLNLKqDhE4Gm5Q@mail.gmail.com>
Message-ID: <CABrqr5rWAP79FxFx8bsWkiUjAx2kgn7SvX8HAT1gR=gpKK96yg@mail.gmail.com>

Actually, the parallel leastsq code is very unstable on both debian 7 and
ubuntu 13.10.
Sometimes it works, sometimes it freezes my computer.

I would be glad if anybody could explain to me the origin of this problem.

Best regards,

Fr?d?ric Parrenin




2014-05-23 8:52 GMT+02:00 Fr?d?ric Parrenin <parrenin at ujf-grenoble.fr>:

> Answering to my own question:
> Actually, the same code runs on debian 7 instead of ubuntu 13.10 does not
> slow down my computer. So this may be an ubuntu-specific problem.
>
> For the gain, my program runs in 545 s on one core and in 123 s using 10
> cores.
> So it seems there is a factor of 2 performance hit in this case, this is
> not two bad.
>
> Best regards,
>
> Fr?d?ric Parrenin
>
>
>
>
> 2014-05-23 4:23 GMT+02:00 Matt Newville <newville at cars.uchicago.edu>:
>
>> Hi Frederic,
>>
>> On Thu, May 22, 2014 at 10:20 AM, Fr?d?ric Parrenin <
>> parrenin at ujf-grenoble.fr> wrote:
>>
>>> Dear all,
>>>
>>> Coming back to an old thread...
>>>
>>> I tried Jeremy's method since it is the easiest to implement.
>>> Below is the Dfun function I provided to leastsq.
>>> In my experiment, I used a pool of 6 since I have 8 cores in my PC.
>>>
>>> However, the computer becomes extremely slow, almost unusable, during
>>> the experiment.
>>> Do you know why this happens?
>>>
>>> Best regards,
>>>
>>> Fr?d?ric
>>>
>>
>> Yes, my observation, based on the code at
>>
>> https://github.com/newville/scipy/commit/3d0ac1da3bcd1d34a1bec8226ea0284f04fcb5dc
>>
>> was that there was about a 10x performance hit.    So, similar to your
>> observations.
>>
>> This approach assumes that the cost of setting up multiple processes is
>> small compared to the execution time of the objective function itself.  It
>> also assumes that having a Jacobian function in Python (as compared to
>> Fortran) is a small performance hit.  Again, this is more likely to be true
>> for a time-consuming objective function, and almost certainly not true for
>> any small test case.
>>
>> I could be persuaded that this approach is still a reasonable idea, but
>> (at least if implemented in pure Python) all the evidence is that it is
>> much slower.  Using Cython may help, but I have not tried this.
>>
>> Any multiprocessing approach that includes calling the objective function
>> from different processes is going to be limited by the "picklability"
>> issue. To me, this is a fairly significant limitation.  I've been lead to
>> believe that the Mystic framework may have worked around this problem, but
>> I don't know the details.
>>
>> Others have suggested that doing the QR factorization with
>> multiprocessing would be the better approach.  This seems worth trying,
>> but, In my experience, the bulk of the time is actually spent in the
>> objective function.
>>
>> --Matt Newville
>>
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140526/1ab0a280/attachment.html>

From mdekauwe at gmail.com  Wed May 28 10:45:55 2014
From: mdekauwe at gmail.com (mdekauwe)
Date: Wed, 28 May 2014 07:45:55 -0700 (PDT)
Subject: [SciPy-User] about opening netcdf file
In-Reply-To: <CAN5afy-7-_BCVuaiWcrDnFRN1U12P6we8rnWrH0sUc6d1AGB0A@mail.gmail.com>
References: <CAN5afy-7-_BCVuaiWcrDnFRN1U12P6we8rnWrH0sUc6d1AGB0A@mail.gmail.com>
Message-ID: <1401288355677-19455.post@n7.nabble.com>

Personally I use the https://code.google.com/p/netcdf4-python/

I think at some point when I tested it was considerably quicker, but no idea
if this is really true. Anyway...

import netCDF4 as nc

f = nc.Dataset(fname, 'r')             

# Extract data from NetCDF file
lats = f.variables['latitude'][:]  # extract/copy the data
lons = f.variables['longitude'][:]
time = nc.num2date(f.variables['time'][:], f.variables['time'].units)

temp = f.variables['Tair']

HTH,

Martin



--
View this message in context: http://scipy-user.10969.n7.nabble.com/SciPy-User-about-opening-netcdf-file-tp19445p19455.html
Sent from the Scipy-User mailing list archive at Nabble.com.


From jsch at demuc.de  Wed May 28 20:51:05 2014
From: jsch at demuc.de (=?iso-8859-1?Q?Johannes_Sch=0F=F6nberger?=)
Date: Wed, 28 May 2014 20:51:05 -0400
Subject: [SciPy-User] ANN: scikit-image 0.10.0
References: <FA0CAD4E-D769-4822-8014-AFE3C9F5C47D@demuc.de>
Message-ID: <AC616B42-B299-4D3E-9EB8-03AD06C6AA85@demuc.de>

Announcement: scikit-image 0.10.0
=================================

We're happy to announce the release of scikit-image v0.10.0!

scikit-image is an image processing toolbox for SciPy that includes algorithms
for segmentation, geometric transformations, color space manipulation,
analysis, filtering, morphology, feature detection, and more.

For more information, examples, and documentation, please visit our website:

 http://scikit-image.org


New Features
------------

In addition to many bug fixes, (speed) improvements and new examples, the 118 pull
requests (1112 commits) merged for this release include the following new
features (PR number in brackets):

- Viewer returns overlays (#810)
- Luv colorspace conversion (#798)
- Phase unwrapping (#644)
- Add Wiener deconvolution (#800)
- Sum filter to rank filters (#844)
- Updates to MCP algorithm. (#854)
- Add ISODATA thresholding (#859)
- Add `imread_collection` automatically if plugin provides `imread` (#862)
- Enforce SLIC superpixels connectivity (#857)
- Estimator class parameters now public (#840)
- BRIEF, ORB and CENSURE features (#834)
- SLIC-zero capabilities for SLIC segmentation. (#864)
- Shaded Polygon for LineProfile Plugin Output (#875)
- Add string inputs for range parameters of `rescale_intensity` (#897)
- Blob Detection (#903)
- `peak_local_max` support for N-d images (#906)
- Add loading from URL in novice module (#916)
- Correct mesh orientation in marching cubes algorithm (#882)
- Added alpha channel support to novice (#946)
- Equality for regionprops (#956)
- IPython notebook export for examples (#1000)


API changes
-----------

The following backward-incompatible API changes were made between 0.9 and 0.10:

- Removed deprecated functions in `skimage.filter.rank.*`
- Removed deprecated parameter `epsilon` of `skimage.viewer.LineProfile`
- Removed backwards-compatability of `skimage.measure.regionprops`
- Removed {`ratio`, `sigma`} deprecation warnings of `skimage.segmentation.slic`
and also remove explicit `sigma` parameter from doc-string example
- Changed default mode of random_walker segmentation to 'cg_mg' > 'cg' > 'bf',
depending on which optional dependencies are available.
- Removed deprecated `out` parameter of `skimage.morphology.binary_*`
- Removed deprecated parameter `depth` in `skimage.segmentation.random_walker`
- Removed deprecated logger function in `skimage/__init__.py`
- Removed deprecated function `filter.median_filter`
- Removed deprecated `skimage.color.is_gray` and `skimage.color.is_rgb`
functions
- Removed deprecated `skimage.segmentation.visualize_boundaries`
- Removed deprecated `skimage.morphology.greyscale_*`
- Removed deprecated `skimage.exposure.equalize`



Contributors to this release
----------------------------

This release was made possible by the collaborative efforts of many
contributors, both new and old.  They are listed in alphabetical order by
surname:

- Raphael Ackermann
- Ankit Agrawal
- Maximilian Albert
- Pietro Berkes
- Vighnesh Birodkar
- Fran?ois Boulogne
- Olivier Debeir
- Christoph Deil
- Jaidev Deshpande
- Jostein B? Fl?ystad
- Neeraj Gangwar
- Michael Hansen
- Almar Klein
- Jeremy Metz
- Juan Nunez-Iglesias
- Fran?ois Orieux
- Guillem Palou
- Rishabh Raj
- Thomas Robitaille
- Michal Romaniuk
- Johannes L. Sch?nberger
- Julian Taylor
- Gregor Thalhammer
- Matthew Trentacoste
- Siva Prasad Varma
- Guillem Palou Visa
- Stefan van der Walt
- Josh Warner
- Tony S Yu
- blink1073
- cgohlke
- jaimefrio
- radioxoma
- sharky93





From gary.ruben at gmail.com  Thu May 29 02:19:30 2014
From: gary.ruben at gmail.com (gary ruben)
Date: Thu, 29 May 2014 16:19:30 +1000
Subject: [SciPy-User] about opening netcdf file
In-Reply-To: <1401288355677-19455.post@n7.nabble.com>
References: <CAN5afy-7-_BCVuaiWcrDnFRN1U12P6we8rnWrH0sUc6d1AGB0A@mail.gmail.com>
	<1401288355677-19455.post@n7.nabble.com>
Message-ID: <CAExsOdZswO9t07SZ2NaVVMc_Xse=rcCU+=Mb6S0YnzefLVfpoQ@mail.gmail.com>

In my experience pupynere is faster than netcdf4. pupynere is also the
basis of the netcdf reader that is available in scipy, e.g.:

from scipy.io import netcdf_file

f = netcdf_file(path, 'r')
array_data = f.variables['array_data']

f.close()

I haven't checked recently, but from memory I think it doesn't support
contexts, so opening the file using "with open as f" doesn't work; you have
to open and close it explicitly as shown.

Gary



On 29 May 2014 00:45, mdekauwe <mdekauwe at gmail.com> wrote:

> Personally I use the https://code.google.com/p/netcdf4-python/
>
> I think at some point when I tested it was considerably quicker, but no
> idea
> if this is really true. Anyway...
>
> import netCDF4 as nc
>
> f = nc.Dataset(fname, 'r')
>
> # Extract data from NetCDF file
> lats = f.variables['latitude'][:]  # extract/copy the data
> lons = f.variables['longitude'][:]
> time = nc.num2date(f.variables['time'][:], f.variables['time'].units)
>
> temp = f.variables['Tair']
>
> HTH,
>
> Martin
>
>
>
> --
> View this message in context:
> http://scipy-user.10969.n7.nabble.com/SciPy-User-about-opening-netcdf-file-tp19445p19455.html
> Sent from the Scipy-User mailing list archive at Nabble.com.
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140529/7515c1f5/attachment.html>

From sturla.molden at gmail.com  Thu May 29 12:17:30 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Thu, 29 May 2014 16:17:30 +0000 (UTC)
Subject: [SciPy-User] leastsq and multiprocessing
References: <CABrqr5pZYYYceLnbiccmaQi2NMhaAu9yiwJOVjgPV5_9umHpDA@mail.gmail.com>
	<CA+7ESbobejjW=ML+k8mN3XnkiuQZap1DGY7v9bsjxxPGLyMAeA@mail.gmail.com>
	<CABrqr5qA7Vfix1uejfs1y2bAUd=Hzv7USmFtLLNLKqDhE4Gm5Q@mail.gmail.com>
	<CABrqr5rWAP79FxFx8bsWkiUjAx2kgn7SvX8HAT1gR=gpKK96yg@mail.gmail.com>
Message-ID: <729088418423070966.298979sturla.molden-gmail.com@news.gmane.org>

I think this is fundamentally the wrong approach to a parallel leastsq. We
should replace the MINPACK supplied QR-solver with one based on LAPACK.
Then MKL, Accelerate or OpenBLAS will take care of the parallel processing.
This is often the dominating part of the computation. So just parallel
processing in the Python callbacks will not be very scalable. If you really
care about a parallel leastsq, this is where you should put your effort.
The computational compexity here is O(N**3), compared to O(N) for the
callbacks. The bigger the problem, the more the QR part will dominate. 

As for the callback functions that produces residuals and Jacobian, the
easiest solution would be a prange in Cython or Numba, or use Python
threads and release the GIL. I would not use multiprocessing without shared
memory here, because otherwise the IPC overhead will be too big. The
functions that compute the residuals and Jacobian are called repeatedly.
The major IPC overhead is multiprocessing's internal use of pickle to
serialize the ndarrays, not the communication over the pipes. I would
instead just copy data to and from shared memory. You can find a shared
memory system that works with multiprocessing on 
https://github.com/sturlamolden/sharedmem-numpy
Note that it does not remove the pickle overhead, so you should reuse the
shared memory arrays in the Python callbacks. This way the IPC overhead
will be reduced to a memcpy.

Sturla


Fr?d?ric Parrenin <parrenin at ujf-grenoble.fr> wrote:
> Actually, the parallel leastsq code is very unstable on both debian 7 and
> ubuntu 13.10.
> Sometimes it works, sometimes it freezes my computer.
> 
> I would be glad if anybody could explain to me the origin of this problem.
> 
> Best regards,
> 
> Fr?d?ric Parrenin
> 
> 2014-05-23 8:52 GMT+02:00 Fr?d?ric Parrenin <parrenin at ujf-grenoble.fr>:
> 
>> Answering to my own question:
>> Actually, the same code runs on debian 7 instead of ubuntu 13.10 does not
>> slow down my computer. So this may be an ubuntu-specific problem.
>> 
>> For the gain, my program runs in 545 s on one core and in 123 s using 10
>> cores.
>> So it seems there is a factor of 2 performance hit in this case, this is
>> not two bad.
>> 
>> Best regards,
>> 
>> Fr?d?ric Parrenin
>> 
>> 
>> 
>> 
>> 2014-05-23 4:23 GMT+02:00 Matt Newville <newville at cars.uchicago.edu>:
>> 
>>> Hi Frederic,
>>> 
>>> On Thu, May 22, 2014 at 10:20 AM, Fr?d?ric Parrenin <
>>> parrenin at ujf-grenoble.fr> wrote:
>>> 
>>>> Dear all,
>>>> 
>>>> Coming back to an old thread...
>>>> 
>>>> I tried Jeremy's method since it is the easiest to implement.
>>>> Below is the Dfun function I provided to leastsq.
>>>> In my experiment, I used a pool of 6 since I have 8 cores in my PC.
>>>> 
>>>> However, the computer becomes extremely slow, almost unusable, during
>>>> the experiment.
>>>> Do you know why this happens?
>>>> 
>>>> Best regards,
>>>> 
>>>> Fr?d?ric
>>>> 
>>> 
>>> Yes, my observation, based on the code at
>>> 
>>> <a
>>> href="https://github.com/newville/scipy/commit/3d0ac1da3bcd1d34a1bec8226ea0284f04fcb5dc">https://github.com/newville/scipy/commit/3d0ac1da3bcd1d34a1bec8226ea0284f04fcb5dc</a>
>>> 
>>> was that there was about a 10x performance hit.    So, similar to your
>>> observations.
>>> 
>>> This approach assumes that the cost of setting up multiple processes is
>>> small compared to the execution time of the objective function itself.  It
>>> also assumes that having a Jacobian function in Python (as compared to
>>> Fortran) is a small performance hit.  Again, this is more likely to be true
>>> for a time-consuming objective function, and almost certainly not true for
>>> any small test case.
>>> 
>>> I could be persuaded that this approach is still a reasonable idea, but
>>> (at least if implemented in pure Python) all the evidence is that it is
>>> much slower.  Using Cython may help, but I have not tried this.
>>> 
>>> Any multiprocessing approach that includes calling the objective function
>>> from different processes is going to be limited by the "picklability"
>>> issue. To me, this is a fairly significant limitation.  I've been lead to
>>> believe that the Mystic framework may have worked around this problem, but
>>> I don't know the details.
>>> 
>>> Others have suggested that doing the QR factorization with
>>> multiprocessing would be the better approach.  This seems worth trying,
>>> but, In my experience, the bulk of the time is actually spent in the
>>> objective function.
>>> 
>>> --Matt Newville
>>> 
>>> 
>>> _______________________________________________
>>> SciPy-User mailing list
>>> SciPy-User at scipy.org
>>> <a
>>> href="http://mail.scipy.org/mailman/listinfo/scipy-user">http://mail.scipy.org/mailman/listinfo/scipy-user</a>
>>> 
>>> 
>> 
> 
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> <a
> href="http://mail.scipy.org/mailman/listinfo/scipy-user">http://mail.scipy.org/mailman/listinfo/scipy-user</a>



From sturla.molden at gmail.com  Thu May 29 13:55:41 2014
From: sturla.molden at gmail.com (Sturla Molden)
Date: Thu, 29 May 2014 17:55:41 +0000 (UTC)
Subject: [SciPy-User] leastsq and multiprocessing
References: <CABrqr5pZYYYceLnbiccmaQi2NMhaAu9yiwJOVjgPV5_9umHpDA@mail.gmail.com>
	<CA+7ESbobejjW=ML+k8mN3XnkiuQZap1DGY7v9bsjxxPGLyMAeA@mail.gmail.com>
	<CABrqr5qA7Vfix1uejfs1y2bAUd=Hzv7USmFtLLNLKqDhE4Gm5Q@mail.gmail.com>
	<CABrqr5rWAP79FxFx8bsWkiUjAx2kgn7SvX8HAT1gR=gpKK96yg@mail.gmail.com>
	<729088418423070966.298979sturla.molden-gmail.com@news.gmane.org>
Message-ID: <384741976423078282.054743sturla.molden-gmail.com@news.gmane.org>

Sturla Molden <sturla.molden at gmail.com> wrote:

> The computational compexity here is O(N**3), compared to O(N) for the
> callbacks. The bigger the problem, the more the QR part will dominate. 

In other words, as the size of the problems increases, the speed
improvement from using multiprocessing will go towards 0. You will have no
benefit from using multiprocessing with leastsq when you really need it. So
put the parallel computing in leastsq in the right place.

In the long run, it would be better to replace the Fortran MINPACK leastsq
with one that uses LAPACK *GELS driver for each linearized least squares
fit. (BTW: Using *GELSS (scipy.linalg.lstsq) kind of defies the purpose of
using a regularization in the linearized least squares fits, as the SVD
solves this problem automatically.)

Sturla



From jonathantu at gmail.com  Thu May 29 16:20:46 2014
From: jonathantu at gmail.com (Jonathan Tu)
Date: Thu, 29 May 2014 13:20:46 -0700
Subject: [SciPy-User] GMRES iteration number
Message-ID: <2EAF2F65-0385-4899-8897-E62628EAD50D@gmail.com>

Hi,

Is there any way to access the number of iterations it takes to complete a GMRES computation?  I've checked the documentation at http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.sparse.linalg.gmres.html and it doesn't appear so.  I am doing some testing with passing in initial guesses x0, and I am interested to know whether or not this significantly reduces the required number of iterations.



Thanks,
Jonathan Tu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140529/3e49e53d/attachment.html>

From newville at cars.uchicago.edu  Thu May 29 17:49:57 2014
From: newville at cars.uchicago.edu (Matt Newville)
Date: Thu, 29 May 2014 16:49:57 -0500
Subject: [SciPy-User] leastsq and multiprocessing
In-Reply-To: <729088418423070966.298979sturla.molden-gmail.com@news.gmane.org>
References: <CABrqr5pZYYYceLnbiccmaQi2NMhaAu9yiwJOVjgPV5_9umHpDA@mail.gmail.com>
	<CA+7ESbobejjW=ML+k8mN3XnkiuQZap1DGY7v9bsjxxPGLyMAeA@mail.gmail.com>
	<CABrqr5qA7Vfix1uejfs1y2bAUd=Hzv7USmFtLLNLKqDhE4Gm5Q@mail.gmail.com>
	<CABrqr5rWAP79FxFx8bsWkiUjAx2kgn7SvX8HAT1gR=gpKK96yg@mail.gmail.com>
	<729088418423070966.298979sturla.molden-gmail.com@news.gmane.org>
Message-ID: <CA+7ESbr=9yJLbty7bmi+FXCf7gSLTX45+TKM0oatSJzzZe0qjA@mail.gmail.com>

Hi Sturla,

On Thu, May 29, 2014 at 11:17 AM, Sturla Molden <sturla.molden at gmail.com>
wrote:
>
> I think this is fundamentally the wrong approach to a parallel leastsq. We
> should replace the MINPACK supplied QR-solver with one based on LAPACK.
> Then MKL, Accelerate or OpenBLAS will take care of the parallel
processing.
> This is often the dominating part of the computation. So just parallel
> processing in the Python callbacks will not be very scalable. If you
really
> care about a parallel leastsq, this is where you should put your effort.
> The computational compexity here is O(N**3), compared to O(N) for the
> callbacks. The bigger the problem, the more the QR part will dominate.
>

I agree that, at some order of magnitude, QR will dominate.  I'm not sure
that this is "often the dominating part".   Do you have any data on this?
Certainly the crossover will depend on the complexity of the objective
function, no?

>
> As for the callback functions that produces residuals and Jacobian, the
> easiest solution would be a prange in Cython or Numba, or use Python
> threads and release the GIL. I would not use multiprocessing without
shared
> memory here, because otherwise the IPC overhead will be too big. The
> functions that compute the residuals and Jacobian are called repeatedly.
> The major IPC overhead is multiprocessing's internal use of pickle to
> serialize the ndarrays, not the communication over the pipes. I would
> instead just copy data to and from shared memory. You can find a shared
> memory system that works with multiprocessing on
> https://github.com/sturlamolden/sharedmem-numpy
> Note that it does not remove the pickle overhead, so you should reuse the
> shared memory arrays in the Python callbacks. This way the IPC overhead
> will be reduced to a memcpy.
>
>
> Sturla
>

That seems reasonable for static ndarray data.  But I think the bigger
overhead may be function calls.

For me, the bigger issue is requiring pickle-able objects, especially
instance methods. That is, I'd like to create a Data or Model object and
pass that into a fitting function, using methods a part of the residual
calculation.  Those instances would need to be pickleable to work with
multiprocessing. I don't know how to that in a general way.

--Matt
 On May 29, 2014 11:17 AM, "Sturla Molden" <sturla.molden at gmail.com> wrote:

> I think this is fundamentally the wrong approach to a parallel leastsq. We
> should replace the MINPACK supplied QR-solver with one based on LAPACK.
> Then MKL, Accelerate or OpenBLAS will take care of the parallel processing.
> This is often the dominating part of the computation. So just parallel
> processing in the Python callbacks will not be very scalable. If you really
> care about a parallel leastsq, this is where you should put your effort.
> The computational compexity here is O(N**3), compared to O(N) for the
> callbacks. The bigger the problem, the more the QR part will dominate.
>
> As for the callback functions that produces residuals and Jacobian, the
> easiest solution would be a prange in Cython or Numba, or use Python
> threads and release the GIL. I would not use multiprocessing without shared
> memory here, because otherwise the IPC overhead will be too big. The
> functions that compute the residuals and Jacobian are called repeatedly.
> The major IPC overhead is multiprocessing's internal use of pickle to
> serialize the ndarrays, not the communication over the pipes. I would
> instead just copy data to and from shared memory. You can find a shared
> memory system that works with multiprocessing on
> https://github.com/sturlamolden/sharedmem-numpy
> Note that it does not remove the pickle overhead, so you should reuse the
> shared memory arrays in the Python callbacks. This way the IPC overhead
> will be reduced to a memcpy.
>
> Sturla
>
>
> Fr?d?ric Parrenin <parrenin at ujf-grenoble.fr> wrote:
> > Actually, the parallel leastsq code is very unstable on both debian 7 and
> > ubuntu 13.10.
> > Sometimes it works, sometimes it freezes my computer.
> >
> > I would be glad if anybody could explain to me the origin of this
> problem.
> >
> > Best regards,
> >
> > Fr?d?ric Parrenin
> >
> > 2014-05-23 8:52 GMT+02:00 Fr?d?ric Parrenin <parrenin at ujf-grenoble.fr>:
> >
> >> Answering to my own question:
> >> Actually, the same code runs on debian 7 instead of ubuntu 13.10 does
> not
> >> slow down my computer. So this may be an ubuntu-specific problem.
> >>
> >> For the gain, my program runs in 545 s on one core and in 123 s using 10
> >> cores.
> >> So it seems there is a factor of 2 performance hit in this case, this is
> >> not two bad.
> >>
> >> Best regards,
> >>
> >> Fr?d?ric Parrenin
> >>
> >>
> >>
> >>
> >> 2014-05-23 4:23 GMT+02:00 Matt Newville <newville at cars.uchicago.edu>:
> >>
> >>> Hi Frederic,
> >>>
> >>> On Thu, May 22, 2014 at 10:20 AM, Fr?d?ric Parrenin <
> >>> parrenin at ujf-grenoble.fr> wrote:
> >>>
> >>>> Dear all,
> >>>>
> >>>> Coming back to an old thread...
> >>>>
> >>>> I tried Jeremy's method since it is the easiest to implement.
> >>>> Below is the Dfun function I provided to leastsq.
> >>>> In my experiment, I used a pool of 6 since I have 8 cores in my PC.
> >>>>
> >>>> However, the computer becomes extremely slow, almost unusable, during
> >>>> the experiment.
> >>>> Do you know why this happens?
> >>>>
> >>>> Best regards,
> >>>>
> >>>> Fr?d?ric
> >>>>
> >>>
> >>> Yes, my observation, based on the code at
> >>>
> >>> <a
> >>> href="
> https://github.com/newville/scipy/commit/3d0ac1da3bcd1d34a1bec8226ea0284f04fcb5dc
> ">
> https://github.com/newville/scipy/commit/3d0ac1da3bcd1d34a1bec8226ea0284f04fcb5dc
> </a>
> >>>
> >>> was that there was about a 10x performance hit.    So, similar to your
> >>> observations.
> >>>
> >>> This approach assumes that the cost of setting up multiple processes is
> >>> small compared to the execution time of the objective function itself.
>  It
> >>> also assumes that having a Jacobian function in Python (as compared to
> >>> Fortran) is a small performance hit.  Again, this is more likely to be
> true
> >>> for a time-consuming objective function, and almost certainly not true
> for
> >>> any small test case.
> >>>
> >>> I could be persuaded that this approach is still a reasonable idea, but
> >>> (at least if implemented in pure Python) all the evidence is that it is
> >>> much slower.  Using Cython may help, but I have not tried this.
> >>>
> >>> Any multiprocessing approach that includes calling the objective
> function
> >>> from different processes is going to be limited by the "picklability"
> >>> issue. To me, this is a fairly significant limitation.  I've been lead
> to
> >>> believe that the Mystic framework may have worked around this problem,
> but
> >>> I don't know the details.
> >>>
> >>> Others have suggested that doing the QR factorization with
> >>> multiprocessing would be the better approach.  This seems worth trying,
> >>> but, In my experience, the bulk of the time is actually spent in the
> >>> objective function.
> >>>
> >>> --Matt Newville
> >>>
> >>>
> >>> _______________________________________________
> >>> SciPy-User mailing list
> >>> SciPy-User at scipy.org
> >>> <a
> >>> href="http://mail.scipy.org/mailman/listinfo/scipy-user">
> http://mail.scipy.org/mailman/listinfo/scipy-user</a>
> >>>
> >>>
> >>
> >
> > _______________________________________________
> > SciPy-User mailing list
> > SciPy-User at scipy.org
> > <a
> > href="http://mail.scipy.org/mailman/listinfo/scipy-user">
> http://mail.scipy.org/mailman/listinfo/scipy-user</a>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140529/149fd00d/attachment.html>

From jeremy at jeremysanders.net  Fri May 30 03:37:43 2014
From: jeremy at jeremysanders.net (Jeremy Sanders)
Date: Fri, 30 May 2014 09:37:43 +0200
Subject: [SciPy-User] leastsq and multiprocessing
References: <CABrqr5pZYYYceLnbiccmaQi2NMhaAu9yiwJOVjgPV5_9umHpDA@mail.gmail.com>
	<CA+7ESbobejjW=ML+k8mN3XnkiuQZap1DGY7v9bsjxxPGLyMAeA@mail.gmail.com>
	<CABrqr5qA7Vfix1uejfs1y2bAUd=Hzv7USmFtLLNLKqDhE4Gm5Q@mail.gmail.com>
	<CABrqr5rWAP79FxFx8bsWkiUjAx2kgn7SvX8HAT1gR=gpKK96yg@mail.gmail.com>
	<729088418423070966.298979sturla.molden-gmail.com@news.gmane.org>
Message-ID: <lm9cfp$10d$1@ger.gmane.org>

Sturla Molden wrote:

> I think this is fundamentally the wrong approach to a parallel leastsq. We
> should replace the MINPACK supplied QR-solver with one based on LAPACK.
> Then MKL, Accelerate or OpenBLAS will take care of the parallel
> processing. This is often the dominating part of the computation. So just
> parallel processing in the Python callbacks will not be very scalable. If
> you really care about a parallel leastsq, this is where you should put
> your effort. The computational compexity here is O(N**3), compared to O(N)
> for the callbacks. The bigger the problem, the more the QR part will
> dominate.

This is certainly not true for many of the problems I've been recently 
solving. Calculating the objective function is very slow and my models only 
have a few tens of parameters. The IPC is minimal here, so I get get pretty 
good utilisation of several cores.

I think support for both parallel processing optimisations is necessary.

Jeremy




From arun.gokule at gmail.com  Fri May 30 04:37:15 2014
From: arun.gokule at gmail.com (Arun Gokule)
Date: Fri, 30 May 2014 01:37:15 -0700
Subject: [SciPy-User] GMRES iteration number
In-Reply-To: <2EAF2F65-0385-4899-8897-E62628EAD50D@gmail.com>
References: <2EAF2F65-0385-4899-8897-E62628EAD50D@gmail.com>
Message-ID: <CAEAJEXcfHirTLqCpUS6Co1ViU3F90WcFexTpZDg1YaDbAFdvgQ@mail.gmail.com>

AFAICT no.


On Thu, May 29, 2014 at 1:20 PM, Jonathan Tu <jonathantu at gmail.com> wrote:

> Hi,
>
> Is there any way to access the number of iterations it takes to complete a
> GMRES computation?  I've checked the documentation at
> http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.sparse.linalg.gmres.html and
> it doesn't appear so.  I am doing some testing with passing in initial
> guesses x0, and I am interested to know whether or not this significantly
> reduces the required number of iterations.
>
>
>
> Thanks,
> Jonathan Tu
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140530/f75b05d3/attachment.html>

From ralf.gommers at gmail.com  Fri May 30 04:47:52 2014
From: ralf.gommers at gmail.com (Ralf Gommers)
Date: Fri, 30 May 2014 10:47:52 +0200
Subject: [SciPy-User] GMRES iteration number
In-Reply-To: <CAEAJEXcfHirTLqCpUS6Co1ViU3F90WcFexTpZDg1YaDbAFdvgQ@mail.gmail.com>
References: <2EAF2F65-0385-4899-8897-E62628EAD50D@gmail.com>
	<CAEAJEXcfHirTLqCpUS6Co1ViU3F90WcFexTpZDg1YaDbAFdvgQ@mail.gmail.com>
Message-ID: <CABL7CQjtgnxtLTmAg6dqMNnh8GXcv29-pLEn_VRpsA26OtseLA@mail.gmail.com>

On Fri, May 30, 2014 at 10:37 AM, Arun Gokule <arun.gokule at gmail.com> wrote:

> AFAICT no.
>
>
> On Thu, May 29, 2014 at 1:20 PM, Jonathan Tu <jonathantu at gmail.com> wrote:
>
>> Hi,
>>
>> Is there any way to access the number of iterations it takes to complete
>> a GMRES computation?  I've checked the documentation at
>> http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.sparse.linalg.gmres.html and
>> it doesn't appear so.  I am doing some testing with passing in initial
>> guesses x0, and I am interested to know whether or not this significantly
>> reduces the required number of iterations.
>>
>
There's no return value that tells you tells (and we can't add one in nice
a backwards-compatible way), but you can use a callback function to do
this. Just provide a callback that increments some counter each time it is
called.

Ralf



>>
>>
>> Thanks,
>> Jonathan Tu
>>
>> _______________________________________________
>> SciPy-User mailing list
>> SciPy-User at scipy.org
>> http://mail.scipy.org/mailman/listinfo/scipy-user
>>
>>
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140530/f50d446f/attachment.html>

From vanforeest at gmail.com  Fri May 30 05:35:55 2014
From: vanforeest at gmail.com (nicky van foreest)
Date: Fri, 30 May 2014 11:35:55 +0200
Subject: [SciPy-User] find zero
Message-ID: <CAK_GA2FMFKPfuGH=AJn+QFmXhn9=Mdu+zFCmRPTRWQR0qYGziw@mail.gmail.com>

Hi,

I am trying to find a zero in an array V that is positive at the left some
index i, say, and negative at the right of i. V is not necessarily
descending, so that numpy.searchsorted does not always find the right
answer. My solution to find i is like this

i = np.searchsorted(V<0, 0.5)

This, however, seems a bit overkill, as the complexity of this algorithm is
at least as large as the length of V, (due to V<0) .

Does anybody know of a smarter, a more direct, solution? Of course, I can
build a function, use scipy.optimize.brentq, but that appears to be even
more overkill.

thanks

Nicky
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140530/892f1395/attachment.html>

From alan.isaac at gmail.com  Fri May 30 08:15:33 2014
From: alan.isaac at gmail.com (Alan G Isaac)
Date: Fri, 30 May 2014 08:15:33 -0400
Subject: [SciPy-User] find zero
In-Reply-To: <CAK_GA2FMFKPfuGH=AJn+QFmXhn9=Mdu+zFCmRPTRWQR0qYGziw@mail.gmail.com>
References: <CAK_GA2FMFKPfuGH=AJn+QFmXhn9=Mdu+zFCmRPTRWQR0qYGziw@mail.gmail.com>
Message-ID: <53887665.5030109@gmail.com>

On 5/30/2014 5:35 AM, nicky van foreest wrote:
> I am trying to find a zero in an array V that is positive at the left some index i, say, and negative at the right of i.


The insertion point is
1+np.argmax(np.diff(np.sign(a)) != 0)
but a simple bisection algorithm would be much faster for large arrays.
Without testing::

def bisect(a, i1, i2):
   if a[i1]*a[i2] > 0:
     raise ValueError("Sign changing interval required.")
   while abs(i1-i2) > 1:
     midpt = (i1+i2) // 2
     if a[midpt]*a[i1] > 0:
       i1 = midpt
     else:
       i2 = midpt
   return max(i1,i2)

fwiw,
Alan Isaac



From jeffreback at gmail.com  Fri May 30 13:44:47 2014
From: jeffreback at gmail.com (Jeff Reback)
Date: Fri, 30 May 2014 13:44:47 -0400
Subject: [SciPy-User] ANN: Pandas 0.14.0 released
Message-ID: <CAHMnJKgKcJQx-0Bm+8eqLkxOuw4xQakXi1u8C_ttO_CMDxfK9g@mail.gmail.com>

Hello,

We are proud to announce v0.14.0 of pandas, a major release from 0.13.1.

This release includes a small number of API changes, several new features,
enhancements, and performance improvements along with a large number of bug
fixes.

This was 4 months of work with 1014 commits by 121 authors encompassing 757
issues.

We recommend that all users upgrade to this version.

*Highlights:*

   -   Officially support Python 3.4
   -   SQL interfaces updated to use sqlalchemy
   -   Display interface changes
   -   MultiIndexing Using Slicers
   -   Ability to join a singly-indexed DataFrame with a multi-indexed
   DataFrame
   -   More consistency in groupby results and more flexible groupby
   specifications
   -   Holiday calendars are now supported in CustomBusinessDay
   -   Several improvements in plotting functions, including: hexbin, area
   and pie plots
   -   Performance doc section on I/O operations

See a full description of Whatsnew for v0.14.0 here:
http://pandas.pydata.org/pandas-docs/stable/whatsnew.html


*What is it:*

*pandas* is a Python package providing fast, flexible, and expressive data
structures designed to make working with ?relational? or ?labeled? data both
easy and intuitive. It aims to be the fundamental high-level building block
for
doing practical, real world data analysis in Python. Additionally, it has
the
broader goal of becoming the most powerful and flexible open source data
analysis / manipulation tool available in any language.


Documentation:
http://pandas.pydata.org/pandas-docs/stable/

Source tarballs, windows binaries are available on PyPI:
https://pypi.python.org/pypi/pandas

windows binaries are courtesy of  Christoph Gohlke and are built on Numpy
1.8
macosx wheels will be available soon, courtesy of Matthew Brett

Please report any issues here:
https://github.com/pydata/pandas/issues


Thanks

The Pandas Development Team


Contributors to the 0.14.0 release

   - Acanthostega
   - Adam Marcus
   - agijsberts
   - akittredge
   - Alex Gaudio
   - Alex Rothberg
   - AllenDowney
   - Andrew Rosenfeld
   - Andy Hayden
   - ankostis
   - anomrake
   - Antoine Mazi?res
   - anton-d
   - bashtage
   - Benedikt Sauer
   - benjamin
   - Brad Buran
   - bwignall
   - cgohlke
   - chebee7i
   - Christopher Whelan
   - Clark Fitzgerald
   - clham
   - Dale Jung
   - Dan Allan
   - Dan Birken
   - danielballan
   - Daniel Waeber
   - David Jung
   - David Stephens
   - Douglas McNeil
   - DSM
   - Garrett Drapala
   - Gouthaman Balaraman
   - Guillaume Poulin
   - hshimizu77
   - hugo
   - immerrr
   - ischwabacher
   - Jacob Howard
   - Jacob Schaer
   - jaimefrio
   - Jason Sexauer
   - Jeff Reback
   - Jeffrey Starr
   - Jeff Tratner
   - John David Reaver
   - John McNamara
   - John W. O'Brien
   - Jonathan Chambers
   - Joris Van den Bossche
   - jreback
   - jsexauer
   - Julia Evans
   - J?lio
   - Katie Atkinson
   - kdiether
   - Kelsey Jordahl
   - Kevin Sheppard
   - K.-Michael Aye
   - Matthias Kuhn
   - Matt Wittmann
   - Max Grender-Jones
   - Michael E. Gruen
   - michaelws
   - mikebailey
   - Mike Kelly
   - Nipun Batra
   - Noah Spies
   - ojdo
   - onesandzeroes
   - Patrick O'Keeffe
   - phaebz
   - Phillip Cloud
   - Pietro Battiston
   - PKEuS
   - Randy Carnevale
   - ribonoous
   - Robert Gibboni
   - rockg
   - sinhrks
   - Skipper Seabold
   - SplashDance
   - Stephan Hoyer
   - Tim Cera
   - Tobias Brandt
   - Todd Jennings
   - TomAugspurger
   - Tom Augspurger
   - unutbu
   - westurner
   - Yaroslav Halchenko
   - y-p
   - zach powers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140530/02f088ef/attachment.html>

From matthew.brett at gmail.com  Fri May 30 18:31:29 2014
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 30 May 2014 15:31:29 -0700
Subject: [SciPy-User] [Numpy-discussion] ANN: Pandas 0.14.0 released
In-Reply-To: <lmavvn$saj$2@ger.gmane.org>
References: <CAHMnJKgKcJQx-0Bm+8eqLkxOuw4xQakXi1u8C_ttO_CMDxfK9g@mail.gmail.com>
	<lmavvn$saj$2@ger.gmane.org>
Message-ID: <CAH6Pt5rsuuyzLEyokbXO_v7JxFzzUrbsScVrT8pjSxcbzJ7fxA@mail.gmail.com>

Hi,

On Fri, May 30, 2014 at 3:16 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> pip install --user --up pandas
> Downloading/unpacking pandas from
> https://pypi.python.org/packages/source/p/pandas/pandas-0.14.0.tar.gz#md5=b775987c0ceebcc8d5ace4a1241c967a
> ...
>
> Downloading/unpacking numpy>=1.6.1 from
> https://pypi.python.org/packages/source/n/numpy/numpy-1.8.1.tar.gz#md5=be95babe263bfa3428363d6db5b64678
> (from pandas)
>   Downloading numpy-1.8.1.tar.gz (3.8MB): 3.8MB downloaded
>   Running setup.py egg_info for package numpy
>     Running from numpy source directory.
>
>     warning: no files found matching 'tools/py3tool.py'
>     warning: no files found matching '*' under directory 'doc/f2py'
>     warning: no previously-included files matching '*.pyc' found anywhere in
> distribution
>     warning: no previously-included files matching '*.pyo' found anywhere in
> distribution
>     warning: no previously-included files matching '*.pyd' found anywhere in
> distribution
> Downloading/unpacking six from
> https://pypi.python.org/packages/source/s/six/six-1.6.1.tar.gz#md5=07d606ac08595d795bf926cc9985674f
> (from python-dateutil->pandas)
>   Downloading six-1.6.1.tar.gz
>   Running setup.py egg_info for package six
>
>     no previously-included directories found matching 'documentation/_build'
> Installing collected packages: pandas, pytz, numpy, six
> ....
>
> What?  I already have numpy-1.8.0 installed (also have six, pytz).

Yes, this is a very unfortunate feature of pip --upgrade - it does a
recursive upgrade of all dependent packages:

http://pip.readthedocs.org/en/latest/reference/pip_install.html#cmdoption-U
https://github.com/pypa/pip/issues/304

Maybe you could just do:

pip install --ignore-install pandas

instead?

Cheers,

Matthew


From njs at pobox.com  Fri May 30 18:39:59 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 30 May 2014 23:39:59 +0100
Subject: [SciPy-User] [pydata] Re: [Numpy-discussion] ANN: Pandas 0.14.0
	released
In-Reply-To: <CAH6Pt5rsuuyzLEyokbXO_v7JxFzzUrbsScVrT8pjSxcbzJ7fxA@mail.gmail.com>
References: <CAHMnJKgKcJQx-0Bm+8eqLkxOuw4xQakXi1u8C_ttO_CMDxfK9g@mail.gmail.com>
	<lmavvn$saj$2@ger.gmane.org>
	<CAH6Pt5rsuuyzLEyokbXO_v7JxFzzUrbsScVrT8pjSxcbzJ7fxA@mail.gmail.com>
Message-ID: <CAPJVwBmEnCg-jmcCDD9NmzUx932TQCj+yx5k4ZV4oo8bdGRRMg@mail.gmail.com>

I sometimes do

  pip install pandas==0.14.0

This requires you know the version number, but is still much easier than
the arcane mutterings that are otherwise needed if you want to be fully
correct (pull in new dependencies, etc.).

-n
On 30 May 2014 23:31, "Matthew Brett" <matthew.brett at gmail.com> wrote:

> Hi,
>
> On Fri, May 30, 2014 at 3:16 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
> > pip install --user --up pandas
> > Downloading/unpacking pandas from
> >
> https://pypi.python.org/packages/source/p/pandas/pandas-0.14.0.tar.gz#md5=b775987c0ceebcc8d5ace4a1241c967a
> > ...
> >
> > Downloading/unpacking numpy>=1.6.1 from
> >
> https://pypi.python.org/packages/source/n/numpy/numpy-1.8.1.tar.gz#md5=be95babe263bfa3428363d6db5b64678
> > (from pandas)
> >   Downloading numpy-1.8.1.tar.gz (3.8MB): 3.8MB downloaded
> >   Running setup.py egg_info for package numpy
> >     Running from numpy source directory.
> >
> >     warning: no files found matching 'tools/py3tool.py'
> >     warning: no files found matching '*' under directory 'doc/f2py'
> >     warning: no previously-included files matching '*.pyc' found
> anywhere in
> > distribution
> >     warning: no previously-included files matching '*.pyo' found
> anywhere in
> > distribution
> >     warning: no previously-included files matching '*.pyd' found
> anywhere in
> > distribution
> > Downloading/unpacking six from
> >
> https://pypi.python.org/packages/source/s/six/six-1.6.1.tar.gz#md5=07d606ac08595d795bf926cc9985674f
> > (from python-dateutil->pandas)
> >   Downloading six-1.6.1.tar.gz
> >   Running setup.py egg_info for package six
> >
> >     no previously-included directories found matching
> 'documentation/_build'
> > Installing collected packages: pandas, pytz, numpy, six
> > ....
> >
> > What?  I already have numpy-1.8.0 installed (also have six, pytz).
>
> Yes, this is a very unfortunate feature of pip --upgrade - it does a
> recursive upgrade of all dependent packages:
>
> http://pip.readthedocs.org/en/latest/reference/pip_install.html#cmdoption-U
> https://github.com/pypa/pip/issues/304
>
> Maybe you could just do:
>
> pip install --ignore-install pandas
>
> instead?
>
> Cheers,
>
> Matthew
>
> --
> You received this message because you are subscribed to the Google Groups
> "PyData" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pydata+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140530/8d6ed504/attachment.html>

From matthew.brett at gmail.com  Fri May 30 18:54:37 2014
From: matthew.brett at gmail.com (Matthew Brett)
Date: Fri, 30 May 2014 15:54:37 -0700
Subject: [SciPy-User] [Numpy-discussion] ANN: Pandas 0.14.0 released
In-Reply-To: <CAH6Pt5rsuuyzLEyokbXO_v7JxFzzUrbsScVrT8pjSxcbzJ7fxA@mail.gmail.com>
References: <CAHMnJKgKcJQx-0Bm+8eqLkxOuw4xQakXi1u8C_ttO_CMDxfK9g@mail.gmail.com>
	<lmavvn$saj$2@ger.gmane.org>
	<CAH6Pt5rsuuyzLEyokbXO_v7JxFzzUrbsScVrT8pjSxcbzJ7fxA@mail.gmail.com>
Message-ID: <CAH6Pt5o5AmHmzxfNZt7mF1iKY=cWS_GJxP7UGjUqUEDicbmjCw@mail.gmail.com>

Hi,

On Fri, May 30, 2014 at 3:31 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Fri, May 30, 2014 at 3:16 PM, Neal Becker <ndbecker2 at gmail.com> wrote:
>> pip install --user --up pandas
>> Downloading/unpacking pandas from
>> https://pypi.python.org/packages/source/p/pandas/pandas-0.14.0.tar.gz#md5=b775987c0ceebcc8d5ace4a1241c967a
>> ...
>>
>> Downloading/unpacking numpy>=1.6.1 from
>> https://pypi.python.org/packages/source/n/numpy/numpy-1.8.1.tar.gz#md5=be95babe263bfa3428363d6db5b64678
>> (from pandas)
>>   Downloading numpy-1.8.1.tar.gz (3.8MB): 3.8MB downloaded
>>   Running setup.py egg_info for package numpy
>>     Running from numpy source directory.
>>
>>     warning: no files found matching 'tools/py3tool.py'
>>     warning: no files found matching '*' under directory 'doc/f2py'
>>     warning: no previously-included files matching '*.pyc' found anywhere in
>> distribution
>>     warning: no previously-included files matching '*.pyo' found anywhere in
>> distribution
>>     warning: no previously-included files matching '*.pyd' found anywhere in
>> distribution
>> Downloading/unpacking six from
>> https://pypi.python.org/packages/source/s/six/six-1.6.1.tar.gz#md5=07d606ac08595d795bf926cc9985674f
>> (from python-dateutil->pandas)
>>   Downloading six-1.6.1.tar.gz
>>   Running setup.py egg_info for package six
>>
>>     no previously-included directories found matching 'documentation/_build'
>> Installing collected packages: pandas, pytz, numpy, six
>> ....
>>
>> What?  I already have numpy-1.8.0 installed (also have six, pytz).
>
> Yes, this is a very unfortunate feature of pip --upgrade - it does a
> recursive upgrade of all dependent packages:
>
> http://pip.readthedocs.org/en/latest/reference/pip_install.html#cmdoption-U
> https://github.com/pypa/pip/issues/304
>
> Maybe you could just do:
>
> pip install --ignore-install pandas
>
> instead?

Seconding Nathaniel's suggestion instead:

pip install --ignore-installed pandas

(note fixed typo s/ignore-install/ignore-installed/) also tries to
upgrade numpy.

Cheers,

Matthew


From njs at pobox.com  Fri May 30 19:10:12 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 31 May 2014 00:10:12 +0100
Subject: [SciPy-User] [pydata] Re: [Numpy-discussion] ANN: Pandas 0.14.0
	released
In-Reply-To: <CAH6Pt5o5AmHmzxfNZt7mF1iKY=cWS_GJxP7UGjUqUEDicbmjCw@mail.gmail.com>
References: <CAHMnJKgKcJQx-0Bm+8eqLkxOuw4xQakXi1u8C_ttO_CMDxfK9g@mail.gmail.com>
	<lmavvn$saj$2@ger.gmane.org>
	<CAH6Pt5rsuuyzLEyokbXO_v7JxFzzUrbsScVrT8pjSxcbzJ7fxA@mail.gmail.com>
	<CAH6Pt5o5AmHmzxfNZt7mF1iKY=cWS_GJxP7UGjUqUEDicbmjCw@mail.gmail.com>
Message-ID: <CAPJVwBku1B-RqU-m=4Aeh0v74KC2ysD3wrE9766LiV+f5YvFOw@mail.gmail.com>

If you really want to use complicated command line switches I think the
correct ones are:

pip install -U --no-deps pandas
pip install pandas

(Yes, you have to run both commands in order to handle all cases correctly.)

-n
On 30 May 2014 23:54, "Matthew Brett" <matthew.brett at gmail.com> wrote:

> Hi,
>
> On Fri, May 30, 2014 at 3:31 PM, Matthew Brett <matthew.brett at gmail.com>
> wrote:
> > Hi,
> >
> > On Fri, May 30, 2014 at 3:16 PM, Neal Becker <ndbecker2 at gmail.com>
> wrote:
> >> pip install --user --up pandas
> >> Downloading/unpacking pandas from
> >>
> https://pypi.python.org/packages/source/p/pandas/pandas-0.14.0.tar.gz#md5=b775987c0ceebcc8d5ace4a1241c967a
> >> ...
> >>
> >> Downloading/unpacking numpy>=1.6.1 from
> >>
> https://pypi.python.org/packages/source/n/numpy/numpy-1.8.1.tar.gz#md5=be95babe263bfa3428363d6db5b64678
> >> (from pandas)
> >>   Downloading numpy-1.8.1.tar.gz (3.8MB): 3.8MB downloaded
> >>   Running setup.py egg_info for package numpy
> >>     Running from numpy source directory.
> >>
> >>     warning: no files found matching 'tools/py3tool.py'
> >>     warning: no files found matching '*' under directory 'doc/f2py'
> >>     warning: no previously-included files matching '*.pyc' found
> anywhere in
> >> distribution
> >>     warning: no previously-included files matching '*.pyo' found
> anywhere in
> >> distribution
> >>     warning: no previously-included files matching '*.pyd' found
> anywhere in
> >> distribution
> >> Downloading/unpacking six from
> >>
> https://pypi.python.org/packages/source/s/six/six-1.6.1.tar.gz#md5=07d606ac08595d795bf926cc9985674f
> >> (from python-dateutil->pandas)
> >>   Downloading six-1.6.1.tar.gz
> >>   Running setup.py egg_info for package six
> >>
> >>     no previously-included directories found matching
> 'documentation/_build'
> >> Installing collected packages: pandas, pytz, numpy, six
> >> ....
> >>
> >> What?  I already have numpy-1.8.0 installed (also have six, pytz).
> >
> > Yes, this is a very unfortunate feature of pip --upgrade - it does a
> > recursive upgrade of all dependent packages:
> >
> >
> http://pip.readthedocs.org/en/latest/reference/pip_install.html#cmdoption-U
> > https://github.com/pypa/pip/issues/304
> >
> > Maybe you could just do:
> >
> > pip install --ignore-install pandas
> >
> > instead?
>
> Seconding Nathaniel's suggestion instead:
>
> pip install --ignore-installed pandas
>
> (note fixed typo s/ignore-install/ignore-installed/) also tries to
> upgrade numpy.
>
> Cheers,
>
> Matthew
>
> --
> You received this message because you are subscribed to the Google Groups
> "PyData" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pydata+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140531/8b9e9bf5/attachment.html>

From vanforeest at gmail.com  Sat May 31 16:45:52 2014
From: vanforeest at gmail.com (nicky van foreest)
Date: Sat, 31 May 2014 22:45:52 +0200
Subject: [SciPy-User] find zero
In-Reply-To: <53887665.5030109@gmail.com>
References: <CAK_GA2FMFKPfuGH=AJn+QFmXhn9=Mdu+zFCmRPTRWQR0qYGziw@mail.gmail.com>
	<53887665.5030109@gmail.com>
Message-ID: <CAK_GA2GRsKRH3FtQAB9cSxZ0qHQ7BjJv+eLWt-JEJp5gszj-ow@mail.gmail.com>

Hi Alan,

Thanks for your suggestion. I guess your solution

1+np.argmax(np.diff(np.sign(a)) != 0)

is slower than my code

np.searchsorted(a<0, 0.5)

as it requires a threefold loop over a. However, I did not test it. I was
also hoping to find a numpy/scipy function that would enable me to avoid to
build a bisection in python, such as your solution.

Nicky




On 30 May 2014 14:15, Alan G Isaac <alan.isaac at gmail.com> wrote:

> On 5/30/2014 5:35 AM, nicky van foreest wrote:
> > I am trying to find a zero in an array V that is positive at the left
> some index i, say, and negative at the right of i.
>
>
> The insertion point is
> 1+np.argmax(np.diff(np.sign(a)) != 0)
> but a simple bisection algorithm would be much faster for large arrays.
> Without testing::
>
> def bisect(a, i1, i2):
>    if a[i1]*a[i2] > 0:
>      raise ValueError("Sign changing interval required.")
>    while abs(i1-i2) > 1:
>      midpt = (i1+i2) // 2
>      if a[midpt]*a[i1] > 0:
>        i1 = midpt
>      else:
>        i2 = midpt
>    return max(i1,i2)
>
> fwiw,
> Alan Isaac
>
> _______________________________________________
> SciPy-User mailing list
> SciPy-User at scipy.org
> http://mail.scipy.org/mailman/listinfo/scipy-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.scipy.org/pipermail/scipy-user/attachments/20140531/ef86eeea/attachment.html>