<div dir="ltr">Hi Brian,<div><br></div><div>Yes, ok I wasn't clear either. Meta thing - IPython and NB have spoilt me - I want IPy as a cmd line for everything - and be able to launch all cmdline programs from IPy and IPyNB. So that's the meta goal. Every new cmdline I encounter I try to see if !ls works if not it is not a good enough cmdline any more and I try to see if there is a cell magic for that cmdline :-) !</div>
<div><br></div><div>In the context of Spark/Shark and family, they are early efforts and I want to be able to play with the many moving parts in there fast and furiously, without being limited by the earliness of their interface. So if I can plug these into IPy then all the better.</div>
<div><br></div><div>I am not sure if there's any value in integrating the two parallel computing models as they seem to serve different audiences. The IPy parallel computing model seems closer to what the scientific community needs and at first sight the Spark/Shark model seems to serve the more business oriented data demographic.</div>
<div><br></div><div>Where there is an intersection is IMHO, in the ML area - we will learn about that tomorrow in the conference. In any case in the spirit of "IPy over everything" I'd like to hope I can do some integration. <br>
</div><div><br></div><div>Also Fernando is here too and we chatted at lunch but pretty much about everything else except the AMP stuff. I think it makes more sense to wait till the end of day tomorrow to report on the content.</div>
<div><br></div><div>Nitin</div><div><br></div><div>P.S. In an IETF meetings decades ago Vint Cerf wore a t-shirt that said "IP over everything", so "IPy over everything" is my homage to that t-shirt.<br>
</div><div><br></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><br>------------------------------------------------------------------<br>Nitin Borwankar <br><a href="mailto:nborwankar@gmail.com">nborwankar@gmail.com</a></div>
<br><br><div class="gmail_quote">On Thu, Aug 29, 2013 at 2:58 PM, Brian Granger <span dir="ltr"><<a href="mailto:ellisonbg@gmail.com" target="_blank">ellisonbg@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Sorry I wasn't clear in my question. I am very aware of how amazing<br>
Spark and Shark are. I do think you are right that they are looking<br>
very promising right now. What I don't see is what IPython can offer<br>
in working with them. Given their architecture, I don't see how for<br>
example you could run spark jobs from the IPython Notebook<br>
interactively. Is that the type of thing you are thinking about? Or<br>
are you more thinking about direct integration of spark and<br>
IPython.parallel. I am more wondering what the benefit of<br>
IPython+Spark integration would be. I know that Fernando and Min have<br>
talked with some of the AMP lab people and I would love to see what<br>
can be done. I would probably be best to sit down and talk further<br>
with the spark/shark devs at some point. But if you can learn more<br>
about their architecture and investigate the possibilities and report<br>
back, that would be fantastic.<br>
<div class="HOEnZb"><div class="h5"><br>
On Thu, Aug 29, 2013 at 2:41 PM, Nitin Borwankar <<a href="mailto:nborwankar@gmail.com">nborwankar@gmail.com</a>> wrote:<br>
> Hi Brian,<br>
><br>
> The advantage IMHO is that pyspark and the larger UCB AMP effort are a huge<br>
> open source effort for distributed parallel computing that improves upon the<br>
> Hadoop model. Spark the underlying layer + Shark the Hive compatible query<br>
> language adds performance gains of 10x - 100x. The effort has 20+ companies<br>
> contributing code including Yahoo and 70+ contributors. AMP has a 10M$ grant<br>
> from NSF. So<br>
> a) it's not going away soon<br>
> b) it may be hard to compete with it without that level of resources<br>
> c) they do have a Python shell (have not used it yet) and they appear<br>
> committed to have Python as a first class language in their effort.<br>
> d) lets see if we can find ways to integrate with it.<br>
><br>
> I think integration at the level of the interactive interface might make<br>
> sense.<br>
><br>
> Just my 2c but I think this effort may leapfrog pure Hadoop over the next<br>
> 2-3 years.<br>
><br>
><br>
> Nitin.<br>
><br>
><br>
><br>
><br>
> ------------------------------------------------------------------<br>
> Nitin Borwankar<br>
> <a href="mailto:nborwankar@gmail.com">nborwankar@gmail.com</a><br>
><br>
><br>
> On Thu, Aug 29, 2013 at 1:35 PM, Brian Granger <<a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a>> wrote:<br>
>><br>
>> >From a quick glance, it looks like both pyspark and IPython use<br>
>> similar parallel computing models in terms of the process model. You<br>
>> might think that would help them to integrate, but in this case I<br>
>> think it will get in the way of integration. Without learning more<br>
>> about the low-level details of their architecture it is really<br>
>> difficult to know if it is possible or not. But I think the bigger<br>
>> question is what would the motivation for integration be? Both<br>
>> IPython and spark provide self-contained parallel computing<br>
>> capabilties - what usage cases are there for using both at the same<br>
>> time? I think the biggest potential show stopper is that pyspark is<br>
>> not designed in any way to be interactive as far as I can tell.<br>
>> Pyspark jobs basically run in batch mode, which is going to make it<br>
>> really tough to fit into IPython's interactive model. Worth looking<br>
>> more into though..<br>
>><br>
>> Cheers,<br>
>><br>
>> Brian<br>
>><br>
>> On Thu, Aug 29, 2013 at 11:28 AM, Nitin Borwankar <<a href="mailto:nborwankar@gmail.com">nborwankar@gmail.com</a>><br>
>> wrote:<br>
>> > I'm at AmpCamp3 at UCB and see that there would be huge benefits to<br>
>> > integrating pyspark with IPython and IPyNB.<br>
>> ><br>
>> > Questions:<br>
>> ><br>
>> > a) has this been attempted/done? if so pointers pl.<br>
>> ><br>
>> > b) does this overlap the IPyNB parallel computing effort in<br>
>> > conflicting/competing ways?<br>
>> ><br>
>> > c) if this has not been done yet - does anyone have a sense of how much<br>
>> > effort this might be? (I've done a small hack integrating postgres psql<br>
>> > into<br>
>> > ipynb so I'm not terrified by that level of deep digging, but are there<br>
>> > any<br>
>> > show stopper gotchas?)<br>
>> ><br>
>> > Thanks much,<br>
>> ><br>
>> > Nitin<br>
>> > ------------------------------------------------------------------<br>
>> > Nitin Borwankar<br>
>> > <a href="mailto:nborwankar@gmail.com">nborwankar@gmail.com</a><br>
>> ><br>
>> > _______________________________________________<br>
>> > IPython-dev mailing list<br>
>> > <a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
>> > <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
>> ><br>
>><br>
>><br>
>><br>
>> --<br>
>> Brian E. Granger<br>
>> Cal Poly State University, San Luis Obispo<br>
>> <a href="mailto:bgranger@calpoly.edu">bgranger@calpoly.edu</a> and <a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a><br>
>> _______________________________________________<br>
>> IPython-dev mailing list<br>
>> <a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
>> <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> IPython-dev mailing list<br>
> <a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
> <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
><br>
<br>
<br>
<br>
--<br>
Brian E. Granger<br>
Cal Poly State University, San Luis Obispo<br>
<a href="mailto:bgranger@calpoly.edu">bgranger@calpoly.edu</a> and <a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a><br>
_______________________________________________<br>
IPython-dev mailing list<br>
<a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
<a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
</div></div></blockquote></div><br></div>