<br><br><div class="gmail_quote">On Fri, Oct 29, 2010 at 23:25, Brian Granger <span dir="ltr"><<a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Min,<br>
<div class="im"><br>
On Fri, Oct 29, 2010 at 10:55 PM, MinRK <<a href="mailto:benjaminrk@gmail.com">benjaminrk@gmail.com</a>> wrote:<br>
> This is more agressive than I expected, I'll have to get the new parallel<br>
> stuff in gear.<br>
<br>
</div>If you stopped writing great code, we wouldn't be tempted to do crazy<br>
things like this ;-)<br>
<div class="im"><br>
> The main roadblock for me is merging work into the Kernels. I plan to spend<br>
> tomorrow working on getting the new parallel code ready for review, and<br>
> identifying what needs to happen with code in master in order for this to go<br>
> into 0.11. The only work that needs merge rather than drop-in is in Kernels<br>
> and Session. I expect that just using the new Session will just be fine<br>
> after a rewview, but getting the existing Kernels to provide what is<br>
> necessary for the parallel code will be some work, and I'll try to identify<br>
> exactly what that will look like.<br>
<br>
</div>Are you thinking of having only one top-level kernel script that<br>
handles both the parallel computing stuff and the interactive IPython?<br>
I think the idea of that is fantastic, but I am not sure we need to<br>
have all of that working to merge your stuff. I am not opposed to<br>
attempting this before/during the merge, but I don't view it as<br>
absolutely needed. Also, it may make sense to review your code<br>
standalone first and then discuss merging the kernel and session stuff<br>
with what we already have.<br></blockquote><div><br></div><div>I was thinking that we already have a remote execution object, and the only difference between the two is the connection patterns. New features/bugfixes will likely want to be shared by both. My StreamKernel was derived from the original pykernel, but I kept working on it while you were developing on it, so they diverged. I think they can be merged, as long as we do a few things, mostly to do with abstracting the connections:</div>
<div><br></div><div> * allow Kernels to connect, not just bind</div><div> * use action-based, not socket-type names</div><div> * allow execution requests to come from a *list* of connections, not just one</div>
<div> * use sessions/ioloop instead of direct send/recv_json</div><div><br></div><div>I also think using a KernelManager would be good, because it gets nice process management (restart the kernel, etc.), and I can't really do that without a Kernel, but I could subclass.</div>
<div><br></div><div>Related question:</div><div><br></div><div>why is ipkernel not a subclass of pykernel? There's lots of identical code there.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im"><br>
> The main things I know already:<br>
> * Names should change (GH-178). It's really a coincidence that we had just<br>
> one action per socket type, and the parallel code has several sockets of the<br>
> same type, and some actions that can be on different socket types, depending<br>
> on the scheduler.<br>
<br>
</div>Yep.<br>
<div class="im"><br>
> * Use IOLoop/ZMQStream - this isn't necessarily critical, and I can probably<br>
> do it with a subclass if we don't want it in the main kernels.<br>
<br>
</div>At this point I think that zmqstream has stablized enough that we<br>
*should* be using it in the kernel and kernel manager code anyways. I<br>
am completely fine with this.<br>
<div class="im"><br>
> * apply_request. This should be all new code, and shouldn't collide with<br>
> anything.<br>
<br>
</div>Ok.<br>
<br>
One other point that Fernando and I talked about is actually shipping<br>
the rest of tornado with pyzmq. I have been thinking more about the<br>
architecture of the html notebook that James has been working on and<br>
it is an absolutely perfect fit for implementing the server using our<br>
zmq enabled Tornado event loop with tornado's regular http handling.<br>
It would also give us ssl support, authentication and lots of other<br>
web server goodies like websockets. If we did this, I think it would<br>
be possible to have a decent prototype of James' html notebook in<br>
0.11. What do you think about this Min? We are already shipping a<br>
good portion of tornado already with pyzmq and the rest is just a<br>
dozen or so .py files (there is one .c file that we don't need for<br>
python 2.6 and up).<br>
Eventually I would like to contribute our ioloop.py and zmqstream to<br>
tornado itself, but I don't think we have to worry about that yet.<br></blockquote><div><br></div><div>I'm not very familiar with Tornado other than our use in pyzmq. If we can use it for authentication</div><div>
without significant performance penalty, then that's a pretty big deal, and well worth it.</div><div><br></div><div>It sounds like it would definitely provide a good toolkit for web backends, so using it is probably a good idea.</div>
<div><br></div><div>I'm not sure that it should be *shipped* with pyzmq, though. I think it would be fine to ship with IPython</div><div>if we use it there, but I don't see a need to include it inside pyzmq. If we depend on it, then depend on it in PyPI,</div>
<div>but if it's only for some extended functionality, I don't see any problem with asking people to install it, since it is</div><div>easy_installable (and apt-installable on Ubuntu). PyZMQ is a pretty low-level library - I don't think shipping someone else's</div>
<div>project inside it is a good idea unless there are significant benefits.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Also, moving tornado into pyzmq would allow us to so secure https<br>
connections for the parallel computing client - controller connection.<br></blockquote><div><br></div><div>Secure connections would be *great* if the performance is good enough.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
Cheers,<br>
<font color="#888888"><br>
Brian<br>
</font><div><div></div><div class="h5"><br>
> Let me know what I can do to help things along.<br>
> -MinRK<br>
><br>
> On Fri, Oct 29, 2010 at 20:28, Fernando Perez <<a href="http://fperez.net" target="_blank">fperez.net</a>@<a href="http://gmail.com" target="_blank">gmail.com</a>> wrote:<br>
>><br>
>> On Fri, Oct 29, 2010 at 11:23 AM, Brian Granger <<a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a>><br>
>> wrote:<br>
>> > Remove all of the twisted stuff from 0.11 and put the new zmq stuff in<br>
>> > place as a prototype.<br>
>> ><br>
>> > Here is my logic:<br>
>> ><br>
>> > * The Twisted parallel stuff is *already* broken in 0.11 and if anyone<br>
>> > has stable code running on it, they should be using 0.10.<br>
>> > * If someone is happy to run non-production ready code, there is no<br>
>> > reason they should be using the Twisted stuff, they should use the<br>
>> > pyzmq stuff.<br>
>> > * Twisted is a *massive* burden on our code base:<br>
>> > - For package managers, it brings in Twisted, Foolscap and<br>
>> > zope.interface.<br>
>> > - It makes our test suite unstable and fragile because we have to<br>
>> > run tests in subprocesses and use trial sometimes and nose other<br>
>> > times.<br>
>> > - It is a huge # of LOC.<br>
>> > - It means that most of our codebase is Python 3 ready.<br>
>> ><br>
>> > There are lots of cons to this proposal:<br>
>> ><br>
>> > * That is really quick to drop support for the Twisted stuff.<br>
>> > * We may piss some people off.<br>
>> > * It possibly means maintaining the 0.10 series longer than we imagined.<br>
>> > * We don't have a security story for the pyzmq parallel stuff yet.<br>
>><br>
>> I have to say that I simply didn't have Brian's boldness to propose<br>
>> this, but I think it's the right thing to do, ultimately. It *is*<br>
>> painful in the short term, but it's also the honest approach. I keep<br>
>> forgetting but Brian reminded me that even the Twisted-based code in<br>
>> 0.11 has serious regressions re. the 0.10.x series, since in the big<br>
>> refactoring for 0.11 not quite everything made it through.<br>
>><br>
>> The 0.10 maintenance doesn't worry me a whole lot: as long as we limit<br>
>> it to small changes, by now merging them as self-contained pull<br>
>> requests is really easy (as I just did recently with the ones Paul and<br>
>> Tom sent). And rolling out a new release when the total delta is<br>
>> small is actually not that much work.<br>
>><br>
>> So I'm totally +1 on this radical, but I think ultimately beneficial,<br>
>> approach. It's important to keep in mind that doing this will lift a<br>
>> big load off our shoulders, and we're a small enough team that this<br>
>> benefit is significant. It will let us concentrate on moving the new<br>
>> machinery forward quickly without having to worry about the large<br>
>> Twisted code. It will also help Thomas with his py3 efforts, as it's<br>
>> one less thing he has to keep getting out of his way.<br>
>><br>
>> Concrete plan:<br>
>><br>
>> - Wait a week or two for feedback.<br>
>> - If we decide to move ahead, make a shared branch on the main repo<br>
>> where we can do this work and review it, with all having the chance to<br>
>> contribute while it happens.<br>
>> - Move all twisted-using code (IPython/kernel and some code in<br>
>> IPython/testing) into IPython/deathrow. This will let anyone who<br>
>> reall wants it find it easily, without having to dig through version<br>
>> control history. Note that deathrow does *not* make it into official<br>
>> release tarballs.<br>
>><br>
>> Cheers,<br>
>><br>
>> f<br>
>> _______________________________________________<br>
>> IPython-dev mailing list<br>
>> <a href="mailto:IPython-dev@scipy.org">IPython-dev@scipy.org</a><br>
>> <a href="http://mail.scipy.org/mailman/listinfo/ipython-dev" target="_blank">http://mail.scipy.org/mailman/listinfo/ipython-dev</a><br>
><br>
><br>
<br>
<br>
<br>
</div></div><div><div></div><div class="h5">--<br>
Brian E. Granger, Ph.D.<br>
Assistant Professor of Physics<br>
Cal Poly State University, San Luis Obispo<br>
<a href="mailto:bgranger@calpoly.edu">bgranger@calpoly.edu</a><br>
<a href="mailto:ellisonbg@gmail.com">ellisonbg@gmail.com</a><br>
</div></div></blockquote></div><br>