<div dir="ltr"><div><div>Dear Smith,<br>that's exactly what I want. Thank!<br>Dear Josef,<br>I'm not thinking in publishing nothing with code. If you have some interesting I can show some codes. But it's probably very basic. Mainly I'm constructing some basics functions for model selection. R it's very good with this (bestglm, leaps...) and I see few things in python.<br>
Finaly,<br>Have scipy discussion list yet? I'm not received nothing to months.<br><br></div>abraços,<br></div>Koblitz<br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">2014-05-15 14:00 GMT-03:00 <span dir="ltr"><<a href="mailto:numpy-discussion-request@scipy.org" target="_blank">numpy-discussion-request@scipy.org</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Send NumPy-Discussion mailing list submissions to<br>
<a href="mailto:numpy-discussion@scipy.org">numpy-discussion@scipy.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:numpy-discussion-request@scipy.org">numpy-discussion-request@scipy.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:numpy-discussion-owner@scipy.org">numpy-discussion-owner@scipy.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of NumPy-Discussion digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. smoothing function (rodrigo koblitz)<br>
2. Fancy Indexing of Structured Arrays is Slow (Dave Hirschfeld)<br>
3. [JOB] Scientific software engineer at the Met Office (Phil Elson)<br>
4. Re: smoothing function (<a href="mailto:josef.pktd@gmail.com">josef.pktd@gmail.com</a>)<br>
5. Re: smoothing function (Nathaniel Smith)<br>
6. Re: smoothing function (<a href="mailto:josef.pktd@gmail.com">josef.pktd@gmail.com</a>)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Thu, 15 May 2014 09:04:03 -0300<br>
From: rodrigo koblitz <<a href="mailto:rodrigokoblitz@gmail.com">rodrigokoblitz@gmail.com</a>><br>
Subject: [Numpy-discussion] smoothing function<br>
To: <a href="mailto:numpy-discussion@scipy.org">numpy-discussion@scipy.org</a><br>
Message-ID:<br>
<<a href="mailto:CAAZkdU_5yw9qigWVofVrPZLptgs75q14Y7vaWoGpQW_nqtrpdA@mail.gmail.com">CAAZkdU_5yw9qigWVofVrPZLptgs75q14Y7vaWoGpQW_nqtrpdA@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Buenos,<br>
I'm reading Zuur book (ecology models with R) and try make it entire in<br>
python.<br>
Have this function in R:<br>
M4 <- gam(So ? s(De) + factor(ID), subset = I1)<br>
<br>
the 's' term indicated with So is modelled as a smoothing function of De<br>
<br>
I'm looking for something close to this in python.<br>
<br>
Someone can help me?<br>
<br>
abra?os,<br>
Koblitz<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://mail.scipy.org/pipermail/numpy-discussion/attachments/20140515/04d32736/attachment-0001.html" target="_blank">http://mail.scipy.org/pipermail/numpy-discussion/attachments/20140515/04d32736/attachment-0001.html</a><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Thu, 15 May 2014 12:31:50 +0000 (UTC)<br>
From: Dave Hirschfeld <<a href="mailto:dave.hirschfeld@gmail.com">dave.hirschfeld@gmail.com</a>><br>
Subject: [Numpy-discussion] Fancy Indexing of Structured Arrays is<br>
Slow<br>
To: <a href="mailto:numpy-discussion@scipy.org">numpy-discussion@scipy.org</a><br>
Message-ID: <<a href="mailto:loom.20140515T135603-598@post.gmane.org">loom.20140515T135603-598@post.gmane.org</a>><br>
Content-Type: text/plain; charset=us-ascii<br>
<br>
As can be seen from the code below (or in the notebook linked beneath) fancy<br>
indexing of a structured array is twice as slow as indexing both fields<br>
independently - making it 4x slower?<br>
<br>
I found that fancy indexing was a bottleneck in my application so I was<br>
hoping to reduce the overhead by combining the arrays into a structured<br>
array and only doing one indexing operation. Unfortunately that doubled the<br>
time that it took!<br>
<br>
Is there any reason for this? If not, I'm happy to open an enhancement issue<br>
on GitHub - just let me know.<br>
<br>
Thanks,<br>
Dave<br>
<br>
<br>
In [32]: nrows, ncols = 365, 10000<br>
<br>
In [33]: items = np.rec.fromarrays(randn(2,nrows, ncols), names=<br>
['widgets','gadgets'])<br>
<br>
In [34]: row_idx = randint(0, nrows, ncols)<br>
...: col_idx = np.arange(ncols)<br>
<br>
In [35]: %timeit filtered_items = items[row_idx, col_idx]<br>
100 loops, best of 3: 3.45 ms per loop<br>
<br>
In [36]: %%timeit<br>
...: widgets = items['widgets'][row_idx, col_idx]<br>
...: gadgets = items['gadgets'][row_idx, col_idx]<br>
...:<br>
1000 loops, best of 3: 1.57 ms per loop<br>
<br>
<br>
<a href="http://nbviewer.ipython.org/urls/gist.githubusercontent.com/dhirschfeld/98b9
970fb68adf23dfea/raw/10c0f968ea1489f0a24da80d3af30de7106848ac/Slow%20Structu" target="_blank">http://nbviewer.ipython.org/urls/gist.githubusercontent.com/dhirschfeld/98b9<br>
970fb68adf23dfea/raw/10c0f968ea1489f0a24da80d3af30de7106848ac/Slow%20Structu</a><br>
red%20Array%20Indexing.ipynb<br>
<br>
<a href="https://gist.github.com/dhirschfeld/98b9970fb68adf23dfea" target="_blank">https://gist.github.com/dhirschfeld/98b9970fb68adf23dfea</a><br>
<br>
<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Thu, 15 May 2014 16:13:10 +0100<br>
From: Phil Elson <<a href="mailto:pelson.pub@gmail.com">pelson.pub@gmail.com</a>><br>
Subject: [Numpy-discussion] [JOB] Scientific software engineer at the<br>
Met Office<br>
To: Discussion of Numerical Python <<a href="mailto:numpy-discussion@scipy.org">numpy-discussion@scipy.org</a>>,<br>
matplotlib development list <<a href="mailto:matplotlib-devel@lists.sourceforge.net">matplotlib-devel@lists.sourceforge.net</a>><br>
Message-ID:<br>
<<a href="mailto:CA%2BL60sAj1zoedxALDhuHp6aTo%2BKvcJxzVRJV7nq76Xy_OirurQ@mail.gmail.com">CA+L60sAj1zoedxALDhuHp6aTo+KvcJxzVRJV7nq76Xy_OirurQ@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
I just wanted to let you know that there is currently a vacancy for a<br>
full-time developer at the Met Office, the UK's National Weather Service,<br>
within our Analysis, Visualisation and Data (AVD) team.<br>
<br>
I'm posting on this list as the Met Office's AVD team are heavily involved<br>
in the development of Python packages to support the work that our<br>
scientists undertake on a daily basis. The vast majority of the AVD team's<br>
time is spent working on our own open source Python packages Iris, cartopy<br>
and biggus as well as working on packages such as numpy, scipy, matplotlib<br>
and IPython; so we don't see this as just a great opportunity to work<br>
within a world class scientific organisation, but a role which will also<br>
deliver real benefits to the wider scientific Python community.<br>
<br>
Please see <a href="http://goo.gl/3ScFaZ" target="_blank">http://goo.gl/3ScFaZ</a> for full details and how to apply, or<br>
contact <a href="mailto:HREnquiries@metoffice.gov.uk">HREnquiries@metoffice.gov.uk</a> if you have any questions.<br>
<br>
Many Thanks,<br>
<br>
Phil<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://mail.scipy.org/pipermail/numpy-discussion/attachments/20140515/9ed32579/attachment-0001.html" target="_blank">http://mail.scipy.org/pipermail/numpy-discussion/attachments/20140515/9ed32579/attachment-0001.html</a><br>
<br>
------------------------------<br>
<br>
Message: 4<br>
Date: Thu, 15 May 2014 11:54:30 -0400<br>
From: <a href="mailto:josef.pktd@gmail.com">josef.pktd@gmail.com</a><br>
Subject: Re: [Numpy-discussion] smoothing function<br>
To: Discussion of Numerical Python <<a href="mailto:numpy-discussion@scipy.org">numpy-discussion@scipy.org</a>><br>
Message-ID:<br>
<<a href="mailto:CAMMTP%2BAkRLNgqiXO0PtfW_KRdGThdP8%2B%2BWcy3Bc23YZMV-h%2BPA@mail.gmail.com">CAMMTP+AkRLNgqiXO0PtfW_KRdGThdP8++Wcy3Bc23YZMV-h+PA@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
On Thu, May 15, 2014 at 8:04 AM, rodrigo koblitz<br>
<<a href="mailto:rodrigokoblitz@gmail.com">rodrigokoblitz@gmail.com</a>>wrote:<br>
<br>
> Buenos,<br>
> I'm reading Zuur book (ecology models with R) and try make it entire in<br>
> python.<br>
> Have this function in R:<br>
> M4 <- gam(So ? s(De) + factor(ID), subset = I1)<br>
><br>
> the 's' term indicated with So is modelled as a smoothing function of De<br>
><br>
> I'm looking for something close to this in python.<br>
><br>
<br>
These kind of general questions are better asked on the scipy-user mailing<br>
list which covers more general topics than numpy-discussion.<br>
<br>
As far as I know, GAMs are not available in python, at least I never came<br>
across any.<br>
<br>
statsmodels has an ancient GAM in the sandbox that has never been connected<br>
to any smoother, since, lowess, spline and kernel regression support was<br>
missing. Nobody is working on that right now.<br>
If you have only a single nonparametric variable, then statsmodels also has<br>
partial linear model based on kernel regression, that is not cleaned up or<br>
verified, but Padarn is currently working on this.<br>
<br>
I think in this case using a penalized linear model with spline basis<br>
functions would be more efficient, but there is also nothing clean<br>
available, AFAIK.<br>
<br>
It's not too difficult to write the basic models, but it takes time to<br>
figure out the last 10% and to verify the results and write unit tests.<br>
<br>
<br>
If you make your code publicly available, then I would be very interested<br>
in a link. I'm trying to collect examples from books that have a python<br>
solution.<br>
<br>
Josef<br>
<br>
<br>
><br>
> Someone can help me?<br>
><br>
> abra?os,<br>
> Koblitz<br>
><br>
> _______________________________________________<br>
> NumPy-Discussion mailing list<br>
> <a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
> <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
><br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://mail.scipy.org/pipermail/numpy-discussion/attachments/20140515/e15c73fe/attachment-0001.html" target="_blank">http://mail.scipy.org/pipermail/numpy-discussion/attachments/20140515/e15c73fe/attachment-0001.html</a><br>
<br>
------------------------------<br>
<br>
Message: 5<br>
Date: Thu, 15 May 2014 17:17:43 +0100<br>
From: Nathaniel Smith <<a href="mailto:njs@pobox.com">njs@pobox.com</a>><br>
Subject: Re: [Numpy-discussion] smoothing function<br>
To: Discussion of Numerical Python <<a href="mailto:numpy-discussion@scipy.org">numpy-discussion@scipy.org</a>><br>
Message-ID:<br>
<CAPJVwBns59n=Ddd3O-M7ESc0=<a href="mailto:deA%2BMTdp1CfsPkeTr3qfPiVzg@mail.gmail.com">deA+MTdp1CfsPkeTr3qfPiVzg@mail.gmail.com</a>><br>
Content-Type: text/plain; charset=UTF-8<br>
<br>
On Thu, May 15, 2014 at 1:04 PM, rodrigo koblitz<br>
<<a href="mailto:rodrigokoblitz@gmail.com">rodrigokoblitz@gmail.com</a>> wrote:<br>
> Buenos,<br>
> I'm reading Zuur book (ecology models with R) and try make it entire in<br>
> python.<br>
> Have this function in R:<br>
> M4 <- gam(So ? s(De) + factor(ID), subset = I1)<br>
><br>
> the 's' term indicated with So is modelled as a smoothing function of De<br>
><br>
> I'm looking for something close to this in python.<br>
<br>
The closest thing that doesn't require writing your own code is<br>
probably to use patsy's [1] support for (simple unpenalized) spline<br>
basis transformations [2]. I think using statsmodels this works like:<br>
<br>
import statsmodels.formula.api as smf<br>
# adjust '5' to taste -- bigger = wigglier, less bias, more overfitting<br>
results = smf.ols("So ~ bs(De, 5) + C(ID)", data=my_df).fit()<br>
print results.summary()<br>
<br>
To graph the resulting curve you'll want to use the results to somehow<br>
do "prediction" -- I'm not sure what the API for that looks like in<br>
statsmodels. If you need help figuring it out then the asking on the<br>
statsmodels list or stackoverflow is probably the quickest way to get<br>
help.<br>
<br>
-n<br>
<br>
[1] <a href="http://patsy.readthedocs.org/en/latest/" target="_blank">http://patsy.readthedocs.org/en/latest/</a><br>
[2] <a href="http://patsy.readthedocs.org/en/latest/builtins-reference.html#patsy.builtins.bs" target="_blank">http://patsy.readthedocs.org/en/latest/builtins-reference.html#patsy.builtins.bs</a><br>
<br>
--<br>
Nathaniel J. Smith<br>
Postdoctoral researcher - Informatics - University of Edinburgh<br>
<a href="http://vorpus.org" target="_blank">http://vorpus.org</a><br>
<br>
<br>
------------------------------<br>
<br>
Message: 6<br>
Date: Thu, 15 May 2014 12:47:25 -0400<br>
From: <a href="mailto:josef.pktd@gmail.com">josef.pktd@gmail.com</a><br>
Subject: Re: [Numpy-discussion] smoothing function<br>
To: Discussion of Numerical Python <<a href="mailto:numpy-discussion@scipy.org">numpy-discussion@scipy.org</a>><br>
Message-ID:<br>
<<a href="mailto:CAMMTP%2BBe-OZfidm-Gw%2BEzJm4fcb9zyQZX_aF%2BmWfSMaH9GZPhQ@mail.gmail.com">CAMMTP+Be-OZfidm-Gw+EzJm4fcb9zyQZX_aF+mWfSMaH9GZPhQ@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
On Thu, May 15, 2014 at 12:17 PM, Nathaniel Smith <<a href="mailto:njs@pobox.com">njs@pobox.com</a>> wrote:<br>
<br>
> On Thu, May 15, 2014 at 1:04 PM, rodrigo koblitz<br>
> <<a href="mailto:rodrigokoblitz@gmail.com">rodrigokoblitz@gmail.com</a>> wrote:<br>
> > Buenos,<br>
> > I'm reading Zuur book (ecology models with R) and try make it entire in<br>
> > python.<br>
> > Have this function in R:<br>
> > M4 <- gam(So ? s(De) + factor(ID), subset = I1)<br>
> ><br>
> > the 's' term indicated with So is modelled as a smoothing function of De<br>
> ><br>
> > I'm looking for something close to this in python.<br>
><br>
> The closest thing that doesn't require writing your own code is<br>
> probably to use patsy's [1] support for (simple unpenalized) spline<br>
> basis transformations [2]. I think using statsmodels this works like:<br>
><br>
> import statsmodels.formula.api as smf<br>
> # adjust '5' to taste -- bigger = wigglier, less bias, more overfitting<br>
> results = smf.ols("So ~ bs(De, 5) + C(ID)", data=my_df).fit()<br>
> print results.summary()<br>
><br>
<br>
Nice<br>
<br>
<br>
><br>
> To graph the resulting curve you'll want to use the results to somehow<br>
> do "prediction" -- I'm not sure what the API for that looks like in<br>
> statsmodels. If you need help figuring it out then the asking on the<br>
> statsmodels list or stackoverflow is probably the quickest way to get<br>
> help.<br>
><br>
<br>
seems to work (in a very simple made up example)<br>
<br>
results.predict({'De':np.arange(1,5), 'ID':['a']*4}, transform=True)<br>
#array([ 0.75 , 1.08333333, 0.75 , 0.41666667])<br>
<br>
Josef<br>
<br>
<br>
> -n<br>
><br>
> [1] <a href="http://patsy.readthedocs.org/en/latest/" target="_blank">http://patsy.readthedocs.org/en/latest/</a><br>
> [2]<br>
> <a href="http://patsy.readthedocs.org/en/latest/builtins-reference.html#patsy.builtins.bs" target="_blank">http://patsy.readthedocs.org/en/latest/builtins-reference.html#patsy.builtins.bs</a><br>
><br>
> --<br>
> Nathaniel J. Smith<br>
> Postdoctoral researcher - Informatics - University of Edinburgh<br>
> <a href="http://vorpus.org" target="_blank">http://vorpus.org</a><br>
> _______________________________________________<br>
> NumPy-Discussion mailing list<br>
> <a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
> <a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://mail.scipy.org/pipermail/numpy-discussion/attachments/20140515/c98cbd0a/attachment-0001.html" target="_blank">http://mail.scipy.org/pipermail/numpy-discussion/attachments/20140515/c98cbd0a/attachment-0001.html</a><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
NumPy-Discussion mailing list<br>
<a href="mailto:NumPy-Discussion@scipy.org">NumPy-Discussion@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/numpy-discussion" target="_blank">http://mail.scipy.org/mailman/listinfo/numpy-discussion</a><br>
<br>
<br>
End of NumPy-Discussion Digest, Vol 92, Issue 19<br>
************************************************<br>
</blockquote></div><br></div>