[pypy-dev] Questions on the pypy+numpy project

[holger krekel]

On Wed, Oct 19, 2011 at 10:38 -0400, Gary Robinson wrote:
> > By the way, did you ever considered the possibility of running pypy and cpython side-by-side?
> > You do your pure-python computation on pypy, then you pipe them (e.g. by using execnet) to a cpython process which does the processing using scipy. Depending on how big the data is, the overhead of passing the data around should not be too high
> .
> Absolutely -- I've thought about that general approach though this is the first time I recall hearing about execnet. Of course I'm concerned that the overhead would be too much in some cases, such as huge numbers of calls to scipy.stats.stats.chisqprob. Such overhead seems like it might cancel all the benefit of PyPy, depending on the script. But maybe it's not as much overhead as I fear. For example I see that execnet does not do pickling. Hm.

You can add pickling on top of execnet by using dumps/loads and sending
the bytes which is fast.  I recommend to use pickling with great care
and not everywhere though.

holger

> -- 
> 
> Gary Robinson
> CTO
> Emergent Discovery, LLC
> personal email: garyrob at me.com
> work email: grobinson at emergentdiscovery.com
> Company: http://www.emergentdiscovery.com
> Blog:    http://www.garyrobinson.net
> 
> 
> 
> 
> On Oct 19, 2011, at 10:27 AM, Antonio Cuni wrote:
> 
> > Hello Gary,
> > 
> > On 19/10/11 15:38, Gary Robinson wrote:
> >>>> You would like pypy+numpy+scipy so that you could write fast
> >>>> python-only algorithms and still use the existing libraries.  I
> >>>> suppose this is a perfectly reasonable usecase, and indeed
> >>>> the current plan does not focus on this.
> >>> 
> >> 
> >> Yes. That is exactly what I want.
> > [cut]
> > 
> > thank you for the input: indeed, I agree that for your usecase the current plan is not the best. OTOH, there is probably someone else for which the current plan is better than others, we cannot make everyone happy at the same time, although we might do it eventually :-).
> > 
> > By the way, did you ever considered the possibility of running pypy and cpython side-by-side?
> > You do your pure-python computation on pypy, then you pipe them (e.g. by using execnet) to a cpython process which does the processing using scipy. Depending on how big the data is, the overhead of passing the data around should not be too high
> > 
> > It's not ideal, but it might be worth of being tried.
> > 
> > ciao,
> > Anto
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
>