) image -- bytes of an image file (anything loadable by PIL, so no need to have different PNG and JPEG type strings) mathtext -- just the TeX-lite text (the frontend can render it itself) When given an object for display, the DisplayHook will give it to each of the DisplayFormatters in turn. If the formatter can handle the object, it will return some JSONable object[1]. The DisplayHook will append a 3-tuple (formatter.id, formatter.type, data) to a list. The DisplayHook will give this to whatever is forming the response message. Most likely, there won't be too many of these formatters for the same type active at any time and there should always be the (id='default', type='text') formatter. A simple frontend can just look for that. A more complicated GUI frontend may prefer a type='html' response and only fall back to a type='text' format. It may have an ordered list of formatter IDs that it will try to display before falling back in order. It might allow the user to flip through the different representations for each cell. For example, if I have a type='mathtext' formatter showing sympy expressions, I might wish to go back to a simple repr so I know what to type to reproduce the expression. I'm certain this is overengineered, but I think we have use cases for all of the features in it. I think most of the complexity is optional. The basic in-process terminal frontend doesn't even bother with most of this and just uses the default formatter to get the text and prints it. [1] Why a general JSONable object instead of just bytes? It would be nice to be able to define a formatter that could give some structured information about the object. For example, we could define an ArrayMetadataFormatter that gives a dict with shape, dtype, etc. A GUI frontend could display this information nicely formatted along with one of the other representations. > If the latter, I'm not sure I like the approach of passing a dict > through and letting each formatter modify it. Sate that mutates > as-it-goes tends to produce harder to understand code, at least in my > experience. Instead, we can call all the formatters in sequence and > get from each a pair of key, value. We can then insert the keys into > a dict as they come on our side (so if the storage structure ever > changes from a dict to anything else, likely the formatters can stay > unmodified). Does that sound reasonable to you? That's actually how I would have implemented it [my original ipwx code notwithstanding ;-)]. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Thu Oct 28 21:10:21 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 28 Oct 2010 18:10:21 -0700 Subject: [IPython-dev] Extensible pretty-printing In-Reply-To: <201010281654.07105.mark.voorhies@ucsf.edu> References: <201010281654.07105.mark.voorhies@ucsf.edu> Message-ID: On Thu, Oct 28, 2010 at 4:54 PM, Mark Voorhies wrote: > On Thursday, October 28, 2010 04:11:18 pm Fernando Perez wrote: >> On Thu, Oct 28, 2010 at 4:00 PM, Robert Kern wrote: >> > I recommend exactly what I did in ipwx. The DisplayTrap is configured with a >> > list of DisplayFormatters. Each DisplayFormatter gets a chance to decorate the >> > return messaged with an additional entry, keyed by the type of the >> > DisplayFormatter (probably something like 'string', 'html', 'image', etc. but >> > also perhaps 'repr', 'pretty', 'mathtext'; needs some more thought). > > Would it make sense to use DisplayFormatter class names as keys? ?That would > avoid name collisions. ?Clients wanting more abstract/semantic formatting names > could use an auxiliary hierarchy of general->specific->formatter strings (e.g., > provided by the pretty module) to find the best match to a target formatting > in the message (e.g., like resolving fonts in CSS). I think Robert picked up this theme as well, so I'll reply in his message to this idea... >> If the latter, I'm not sure I like the approach of passing a dict >> through and letting each formatter modify it. > > Given that the outputs should be independent (i.e., shouldn't be modifying each > other), it seems like the main advantage of chaining the formatters would be to > avoid duplicating work (e.g., the html formatter could work off of the string result). > This could also be done by linking the formatters directly (e.g., passing a result-caching > string formatter to the html formatter's constructor) as long as we know the order that > they will be called in. What I don't like about this is that it introduces a fair amount of coupling between formatters: order dependencies and mutation of state. In my mind, these guys are just an example of the observer pattern, and I think in that context the most robust implementations are those that have minimal/zero coupling between observers. Each observer gets notified of the relevant event (output ready for display) and both order of execution and failures of some shouldn't impact the others. I'm not sure I see the real-world benefit of the tighter coupling and I do see a cost... Cheers, f From fperez.net at gmail.com Thu Oct 28 21:17:55 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 28 Oct 2010 18:17:55 -0700 Subject: [IPython-dev] Extensible pretty-printing In-Reply-To: References:

Message-ID: On Thu, Oct 28, 2010 at 5:13 PM, Robert Kern wrote: >> OK, so how do you want to proceed: do you want to reopen your pull >> request (possibly rebasing it if necessary) as it was, or do you want >> to go ahead and implement the above approach right away? > > I'd rather implement this approach right away. We just need to decide what the > keys should be and what they should mean. I originally used the ID of the > DisplayFormatter. This would allow both a "normal" representation and an > enhanced one both of the same type (plain text, HTML, PNG image) to coexist. > Then the frontend could pick which one to display and let the user flip back and > forth as desired even for old Out[] entries without reexecuting code. This may > be a case of YAGNI. Actually I don't think it's YAGNI, and I have a specific use case in mind, with a practical example. Lyx shows displayed equations, but if you copy one, it's nice enough to actually feed the clipboard with the raw Latex for the equation. This is very convenient, and I often use it to edit complex formulas in lyx that I then paste into reST docs. We could similarly have pretty display of e.g. sympy output, but where one could copy the raw latex fort the output cell. The ui could expose this via a context menu that offers 'copy image, copy latex, copy string' for example. So this does strike me like genuinely useful and valuable functionality. > However, that means that the frontend needs to know about the IDs of the > DisplayFormatters. It needs to know that 'my-tweaked-html' formatter is HTML. I > might propose this as the fully-general solution: > > Each DisplayFormatter has a unique ID and a non-unique type. The type string > determines how a frontend would actually interpret the data for display. If a > frontend can display a particular type, it can display it for any > DisplayFormatter of that type. There will be a few predefined type strings with > meanings, but implementors can define new ones as long as they pick new names. > > ? text -- monospaced plain text (unicode) > ? html -- snippet of HTML (anything one can slap inside of a

) > ? image -- bytes of an image file (anything loadable by PIL, so no need to have > different PNG and JPEG type strings) > ? mathtext -- just the TeX-lite text (the frontend can render it itself) > > When given an object for display, the DisplayHook will give it to each of the > DisplayFormatters in turn. If the formatter can handle the object, it will > return some JSONable object[1]. The DisplayHook will append a 3-tuple > > ? (formatter.id, formatter.type, data) > > to a list. The DisplayHook will give this to whatever is forming the response > message. > > Most likely, there won't be too many of these formatters for the same type > active at any time and there should always be the (id='default', type='text') > formatter. A simple frontend can just look for that. A more complicated GUI > frontend may prefer a type='html' response and only fall back to a type='text' > format. It may have an ordered list of formatter IDs that it will try to display > before falling back in order. It might allow the user to flip through the > different representations for each cell. For example, if I have a > type='mathtext' formatter showing sympy expressions, I might wish to go back to > a simple repr so I know what to type to reproduce the expression. > > I'm certain this is overengineered, but I think we have use cases for all of the > features in it. I think most of the complexity is optional. The basic in-process > terminal frontend doesn't even bother with most of this and just uses the > default formatter to get the text and prints it. > > [1] Why a general JSONable object instead of just bytes? It would be nice to be > able to define a formatter that could give some structured information about the > object. For example, we could define an ArrayMetadataFormatter that gives a dict > with shape, dtype, etc. A GUI frontend could display this information nicely > formatted along with one of the other representations. Most of this I agree with. Just one question: why not use real mime types for the type info? I keep thinking that for our payloads and perhaps also for this, we might as well encode type metadata as mimetypes: they're reasonably standardized, python has a mime library, and browsers are wired to do something sensible with mime-tagged data already. Am I missing something? >> If the latter, I'm not sure I like the approach of passing a dict >> through and letting each formatter modify it. ?Sate that mutates >> as-it-goes tends to produce harder to understand code, at least in my >> experience. ?Instead, we can call all the formatters in sequence and >> get from each a pair of key, value. ?We can then insert the keys into >> a dict as they come on our side (so if the storage structure ever >> changes from a dict to anything else, likely the formatters can stay >> unmodified). ?Does that sound reasonable to you? > > That's actually how I would have implemented it [my original ipwx code > notwithstanding ;-)]. OK. It seems we're converging design wise to the point where code can continue the conversation :) Thanks! Cheers, f From ellisonbg at gmail.com Thu Oct 28 21:31:15 2010 From: ellisonbg at gmail.com (Brian Granger) Date: Thu, 28 Oct 2010 18:31:15 -0700 Subject: [IPython-dev] Extensible pretty-printing In-Reply-To: References:

Message-ID: I am in the middle of lab (they are taking a quiz), so I don't have time to dig into the full thread ATM, but I do have a few comments: * The main thing that I am concerned about is how we answer the question i) "how do I (a developer of foo) make my class Foo, print nice HTML/SVG. IOW, what is does the public API for all of this look like? * In the current IPython, displayhook is only triggered 1x per block. Thus, you can't use displayhook to get the str/html/svg/png representation of an object inside a block block or loop. This is a serious limitation, that Fernando and I feel is a good thing in the end. But, this means that we will also need top-level functions that users can put in their code to trigger all of this logic independent of displayhook. Like this: for t in times: a = compute_thing(t) print_html(a) # This should use the APIs that we are designing and the payload system to deliver the html to the frontend. We should also have functions like print_png, print_svg, print_latex that we inject into builtins. What this means is that we need to design an implementation that is independent from displayhook and that is cleanly integrated with the payload system. Cheers, Brian On Thu, Oct 28, 2010 at 6:17 PM, Fernando Perez wrote: > On Thu, Oct 28, 2010 at 5:13 PM, Robert Kern wrote: > >>> OK, so how do you want to proceed: do you want to reopen your pull >>> request (possibly rebasing it if necessary) as it was, or do you want >>> to go ahead and implement the above approach right away? >> >> I'd rather implement this approach right away. We just need to decide what the >> keys should be and what they should mean. I originally used the ID of the >> DisplayFormatter. This would allow both a "normal" representation and an >> enhanced one both of the same type (plain text, HTML, PNG image) to coexist. >> Then the frontend could pick which one to display and let the user flip back and >> forth as desired even for old Out[] entries without reexecuting code. This may >> be a case of YAGNI. > > Actually I don't think it's YAGNI, and I have a specific use case in > mind, with a practical example. ?Lyx shows displayed equations, but if > you copy one, it's nice enough to actually feed the clipboard with the > raw Latex for the equation. ?This is very convenient, and I often use > it to edit complex formulas in lyx that I then paste into reST docs. > > We could similarly have pretty display of e.g. sympy output, but where > one could copy the raw latex fort the output cell. ?The ui could > expose this via a context menu that offers 'copy image, copy latex, > copy string' for example. > > So this does strike me like genuinely useful and valuable functionality. > >> However, that means that the frontend needs to know about the IDs of the >> DisplayFormatters. It needs to know that 'my-tweaked-html' formatter is HTML. I >> might propose this as the fully-general solution: >> >> Each DisplayFormatter has a unique ID and a non-unique type. The type string >> determines how a frontend would actually interpret the data for display. If a >> frontend can display a particular type, it can display it for any >> DisplayFormatter of that type. There will be a few predefined type strings with >> meanings, but implementors can define new ones as long as they pick new names. >> >> ? text -- monospaced plain text (unicode) >> ? html -- snippet of HTML (anything one can slap inside of a

) >> ? image -- bytes of an image file (anything loadable by PIL, so no need to have >> different PNG and JPEG type strings) >> ? mathtext -- just the TeX-lite text (the frontend can render it itself) >> >> When given an object for display, the DisplayHook will give it to each of the >> DisplayFormatters in turn. If the formatter can handle the object, it will >> return some JSONable object[1]. The DisplayHook will append a 3-tuple >> >> ? (formatter.id, formatter.type, data) >> >> to a list. The DisplayHook will give this to whatever is forming the response >> message. >> >> Most likely, there won't be too many of these formatters for the same type >> active at any time and there should always be the (id='default', type='text') >> formatter. A simple frontend can just look for that. A more complicated GUI >> frontend may prefer a type='html' response and only fall back to a type='text' >> format. It may have an ordered list of formatter IDs that it will try to display >> before falling back in order. It might allow the user to flip through the >> different representations for each cell. For example, if I have a >> type='mathtext' formatter showing sympy expressions, I might wish to go back to >> a simple repr so I know what to type to reproduce the expression. >> >> I'm certain this is overengineered, but I think we have use cases for all of the >> features in it. I think most of the complexity is optional. The basic in-process >> terminal frontend doesn't even bother with most of this and just uses the >> default formatter to get the text and prints it. >> >> [1] Why a general JSONable object instead of just bytes? It would be nice to be >> able to define a formatter that could give some structured information about the >> object. For example, we could define an ArrayMetadataFormatter that gives a dict >> with shape, dtype, etc. A GUI frontend could display this information nicely >> formatted along with one of the other representations. > > Most of this I agree with. ?Just one question: why not use real mime > types for the type info? ?I keep thinking that for our payloads and > perhaps also for this, we might as well encode type metadata as > mimetypes: they're reasonably standardized, python has a mime library, > and browsers are wired to do something sensible with mime-tagged data > already. ?Am I missing something? > >>> If the latter, I'm not sure I like the approach of passing a dict >>> through and letting each formatter modify it. ?Sate that mutates >>> as-it-goes tends to produce harder to understand code, at least in my >>> experience. ?Instead, we can call all the formatters in sequence and >>> get from each a pair of key, value. ?We can then insert the keys into >>> a dict as they come on our side (so if the storage structure ever >>> changes from a dict to anything else, likely the formatters can stay >>> unmodified). ?Does that sound reasonable to you? >> >> That's actually how I would have implemented it [my original ipwx code >> notwithstanding ;-)]. > > OK. ?It seems we're converging design wise to the point where code can > continue the conversation :) > > Thanks! > > Cheers, > > f > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgranger at calpoly.edu ellisonbg at gmail.com From robert.kern at gmail.com Thu Oct 28 22:38:55 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 28 Oct 2010 21:38:55 -0500 Subject: [IPython-dev] Extensible pretty-printing In-Reply-To: References:

Message-ID: On 2010-10-28 20:17 , Fernando Perez wrote: > On Thu, Oct 28, 2010 at 5:13 PM, Robert Kern wrote: >> [1] Why a general JSONable object instead of just bytes? It would be nice to be >> able to define a formatter that could give some structured information about the >> object. For example, we could define an ArrayMetadataFormatter that gives a dict >> with shape, dtype, etc. A GUI frontend could display this information nicely >> formatted along with one of the other representations. > > Most of this I agree with. Just one question: why not use real mime > types for the type info? I keep thinking that for our payloads and > perhaps also for this, we might as well encode type metadata as > mimetypes: they're reasonably standardized, python has a mime library, > and browsers are wired to do something sensible with mime-tagged data > already. Am I missing something? I guess you could use MIME types for the ones that make sense to do so. However, it would be nice to have an 'image' type that didn't really care about whether it was PNG or JPEG. There's no reason a frontend should have to look for every image/ MIME type that PIL can handle. We'd never actually *use* the MIME library anywhere. We'd treat the MIME types just like our own identifiers anyways. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Thu Oct 28 22:45:02 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 28 Oct 2010 19:45:02 -0700 Subject: [IPython-dev] Extensible pretty-printing In-Reply-To: References:

Message-ID: On Thu, Oct 28, 2010 at 6:31 PM, Brian Granger wrote: > I am in the middle of lab (they are taking a quiz), so I don't have > time to dig into the full thread ATM, but I do have a few comments: > > * The main thing that I am concerned about is how we answer the > question i) "how do I (a developer of foo) make my class Foo, print > nice HTML/SVG. ?IOW, what is does the public API for all of this look > like? I would propose having a baseline protocol with the usual python approach of __methods__. We can define a few basic ones that are auto-recognized: __repr_html__, __repr_latex__, __repr_image__ (bitmap images), __repr_svg__, __repr_pdf__? The system could by default call any of these that are found, and stuff the output in the repr output structure. For the 'default' field of the repr dict, we'd use either __pretty__ if defined and pretty-printing is on, or __repr__ if pretty-printing is off (I think that's how pretty.py works). This lets classes define their own rich representations for common formats. And then, the complementary approach from pretty allows anyone to define a formatter that pretty-prints any object that hadn't been customized. Though the cost of computing all of these every time does become a concern... There needs to be an easy way to toggle them on/off for the user, so that you're not waiting for a bunch of > * In the current IPython, displayhook is only triggered 1x per block. > Thus, you can't use displayhook to get the str/html/svg/png > representation of an object inside a block block or loop. ?This is a > serious limitation, that Fernando and I feel is a good thing in the > end. ?But, this means that we will also need top-level functions that > users can put in their code to trigger all of this logic independent > of displayhook. > > Like this: > > for t in times: > ? ?a = compute_thing(t) > ? ?print_html(a) ?# This should use the APIs that we are designing > and the payload system to deliver the html to the frontend. > > We should also have functions like print_png, print_svg, print_latex > that we inject into builtins. > > What this means is that we need to design an implementation that is > independent from displayhook and that is cleanly integrated with the > payload system. That seems reasonable, and it can reuse the same machinery from displayhook. Basically the displayhook does it when called and passed an object, these functions do it by explicit user request... Cheers, f From fperez.net at gmail.com Thu Oct 28 22:46:25 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 28 Oct 2010 19:46:25 -0700 Subject: [IPython-dev] Extensible pretty-printing In-Reply-To: References:

Message-ID: On Thu, Oct 28, 2010 at 7:38 PM, Robert Kern wrote: > I guess you could use MIME types for the ones that make sense to do so. However, > it would be nice to have an 'image' type that didn't really care about whether > it was PNG or JPEG. There's no reason a frontend should have to look for every > image/ MIME type that PIL can handle. We'd never actually *use* the MIME > library anywhere. We'd treat the MIME types just like our own identifiers anyways. Agreed. I just think that if we tag things with real MIME types, the web frontend has less logic to worry about. Other frontends are free to treat all image/ identically and just pass them to PIL, for example. Cheers, f From robert.kern at gmail.com Thu Oct 28 23:04:31 2010 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 28 Oct 2010 22:04:31 -0500 Subject: [IPython-dev] Extensible pretty-printing In-Reply-To: References:

Message-ID: On 2010-10-28 20:31 , Brian Granger wrote: > I am in the middle of lab (they are taking a quiz), so I don't have > time to dig into the full thread ATM, but I do have a few comments: > > * The main thing that I am concerned about is how we answer the > question i) "how do I (a developer of foo) make my class Foo, print > nice HTML/SVG. IOW, what is does the public API for all of this look > like? Depends on how you write the HTMLFormatter and SVGFormatter. We can probably write a general base class that handles looking for _html_() methods and to allow registration of functions for types that cannot be modified. I've broken out the deferred type-dispatching I use in our pretty.py before; I'll try to find it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From takowl at gmail.com Fri Oct 29 06:30:29 2010 From: takowl at gmail.com (Thomas Kluyver) Date: Fri, 29 Oct 2010 11:30:29 +0100 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) Message-ID: On 29 October 2010 02:10, wrote: > We now have a massive amount of new code in the pipeline, and it's > really high time we start getting this out in the hands of users > beyond those willing to run from a git HEAD. 0.11 will be a 'tech > preview' release, especially because the situation with regards to the > parallel code is a bit messy right now. But we shouldn't wait for too > much longer. > I don't know what sort of QA process it would need to go through, but could we look at doing a parallel release of IPython on Python 3? As it stands, it would have to just be the plain terminal version, but I think that's still valuable. I'm fairly happy that that side is working, and it passes all the automated tests. The Qt frontend is tantalisingly close to working, largely thanks to MinRK's work with pyzmq, and I hope I can clear the remaining roadblocks soon, but I don't know if it will be ready in a month. And if the new HTML frontend can work with it too, that's just icing on the cake. Thanks, Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg at gmail.com Fri Oct 29 13:44:22 2010 From: ellisonbg at gmail.com (Brian Granger) Date: Fri, 29 Oct 2010 10:44:22 -0700 Subject: [IPython-dev] Code review of html frontend Message-ID: James, Here is a longer code review of the html frontend. Overall, this is really nice. Great job! ==================== HTML Notebook Review ==================== General ======= * Add IPython standard copyright to each file and put an author section with yourself in the module level docstrings. * Use a space after the "#" in comments. * We should refactor the main KernelManager to consist of two classes, one that handles the starting of the kernel process and a second that handles the channels. In this case, there is not reason to have all the channel stuff, as I think we can simple use comet or websockets to pass the raw ZMQ JSON messages between the client and the kernel. The only reason we might not want to do this is to allow us to validate the messages in the web server, but really we should be doing that in the kernel anyways. This would make the webserver stuff even thinner. * Let's document clearly the URL structure we are using. It will be much easier to do this if we move to tornado. * Let's make sure we develop the html/js stuff in a way that will be independent of the webserver side of things, that way, in the future, we can easily swap out either component if we want. * Please add brief docstrings to the important methods. * We will probably have multiple html frontends, so we should probably put your stuff in something like html/basic or html/basicnb. We did this for the qt frontend (qt/console). * For now I would mostly focus on the Javascript and UI side of things, as I think we probably want to move to Tornado even for simple servers like this. UI == * The results of ? and ?? only go to one client. ipythonhttp.py ============== * We should discuss the possibility of using Tornado as our web server. It will be a much better match for working with zmq and many things will be much easier. We are already shipping a good part of Tornado with pyzmq and could possibly ship all of it. Using the stdlib for now is fine though. Tornado also has websocket support that would work extremely well with pyzmq. * Remove command line options that you don't support yet. * Move defer to the top level and rename something like "start_browser". Move import threading to top level as well. kernelmanager.py ================ * In do_GET and do_POST, document what each elif clause is handling. * I see that you are using time.time for the client id. Would it be better to use a real uuid? If you want to also have the time you could still pass that to manager.register as well. * When a kernel dies, the client should note that and there should be an option to restart the kernel. * We should probaby have top level URLs for the different ZMQ sockets like /XREQ, /SUB, etc. so that the GET and POST traffic has different URLS. CometManager.py ================ * Change filename to cometmanager.py. -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgranger at calpoly.edu ellisonbg at gmail.com From ellisonbg at gmail.com Fri Oct 29 13:45:20 2010 From: ellisonbg at gmail.com (Brian Granger) Date: Fri, 29 Oct 2010 10:45:20 -0700 Subject: [IPython-dev] Multiline bug screenshot Message-ID: James, Here is a screenshot of what the multiline input bug looks like in Chrome 6. Cheers, Brian -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgranger at calpoly.edu ellisonbg at gmail.com -------------- next part -------------- A non-text attachment was scrubbed... Name: multiline_bug.jpg Type: image/jpeg Size: 7308 bytes Desc: not available URL: From ellisonbg at gmail.com Fri Oct 29 14:02:16 2010 From: ellisonbg at gmail.com (Brian Granger) Date: Fri, 29 Oct 2010 11:02:16 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References: Message-ID: Fernando, Thanks for summarizing this. I think this sounds like a good plan. Cheers, Brian On Thu, Oct 28, 2010 at 5:01 PM, Fernando Perez wrote: > Hi all, > > I know we've said several times that we should release 0.11 'soon', so > I forgive anyone for laughing at this email. ?But don't ignore it, > this time we mean it :) > > We now have a massive amount of ?new code in the pipeline, and it's > really high time we start getting this out in the hands of users > beyond those willing to run from a git HEAD. ?0.11 will be a 'tech > preview' release, especially because the situation with regards to the > parallel code is a bit messy right now. ?But we shouldn't wait for too > much longer. > > Brian and I tried to compile a list of the main things that need work > before we can make a release, and this is ?our best estimate right > now: > > - Unicode: this is totally broken and totally unacceptable. ?But I'm > pretty sure with a few clean hours I can get it done. ?It's not super > hard, just detail-oriented work that I need a quiet block of time to > tackle. > > - Updating top-level entry points to use the new config system, > especially the Qt console. ?Brian said he could tackle this one. > > - Final checks on the state of the GUI/event loop support. ?Things are > looking fairly good from usage, but we have concerns that there may > still be problems lurking just beneath the surface. > > - Continue/finish the displayhook discussion: we're well on our way on > this, we just need to finish it up. ?We mark it here because it's an > important part of the api and a good test case for how we want to > expose this kind of functionality. > > - Move all payloads to pub channel. ?This is also a big api item that > affects all clients, so we might as well get it right from the start. > I can try to work on this. > > - James' web frontend: I'd really like to get that code in for early > battle-testing, even though it's clear it's early functionality > subject still to much evolution. > > That's all I have in my list. ?Anything else you can all think of? > > As for non-blockers, we have: > > - the parallel code is not in a good situation right now: we have a > few regressions re. the Twisted 0.10.1 code (e.g. the SGE code isn't > ported yet), the Twisted winhpc scheduler is only in 0.11, and while > the new zmq tools are looking great, they are NOT production-ready > quite yet. ?In summary, we'll have to warn in bright, blinking pink > letters 1995-style, everyone who uses the parallel code in production > systems to stick with the 0.10 series for a little longer. ?Annoying, > yes, but unfortunately such is life. > > - our docs have unfortunately gone fairly stale in a few places. ?We > have no docs for the new Qt console and a lot of information is partly > or completely stale. ?This is an area where volunteers could make a > huge difference: any help here has a big impact in letting the project > better serve users, and doc pull requests are likely to be reviewed > very quickly. ?Additionally, you don't need to know too much about the > code's intimate details to help with documenting the user-facing > functionality. > > Anything else? > > Plan: I'd love to get 0.11 out in the first week of December. ?John > Hunter, Stefan van der Walt and I (all three contributors) will be at > Scipy India in Hyderabad Dec 11-18, and there will be sprint time > there. ?Ideally, we'd have a stable release out for potential sprint > participants who want to hack on IPython to work from. ?It would also > be a good way to wrap up a great year of development and de-stagnation > of IPython, leaving us with a nice fresh ball of warm code to play > with over the winter holidays. > > Cheers, > > f > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgranger at calpoly.edu ellisonbg at gmail.com From ellisonbg at gmail.com Fri Oct 29 14:15:45 2010 From: ellisonbg at gmail.com (Brian Granger) Date: Fri, 29 Oct 2010 11:15:45 -0700 Subject: [IPython-dev] Error in test_unicode for InputSplitter Message-ID: I am seeing the following on Python 2.6, Mac OS X 10.5: ====================================================================== ERROR: test_unicode (IPython.core.tests.test_inputsplitter.InputSplitterTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/bgranger/Documents/Computation/IPython/code/ipython/IPython/core/tests/test_inputsplitter.py", line 353, in test_unicode self.isp.push("u'\xc3\xa9'") File "/Users/bgranger/Documents/Computation/IPython/code/ipython/IPython/core/inputsplitter.py", line 374, in push self._store(lines) File "/Users/bgranger/Documents/Computation/IPython/code/ipython/IPython/core/inputsplitter.py", line 607, in _store setattr(self, store, self._set_source(buffer)) File "/Users/bgranger/Documents/Computation/IPython/code/ipython/IPython/core/inputsplitter.py", line 610, in _set_source return ''.join(buffer).encode(self.encoding) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128) ---------------------------------------------------------------------- Ran 270 tests in 1.974s Is this a regression or a known issue? Cheers, Brian -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgranger at calpoly.edu ellisonbg at gmail.com From ellisonbg at gmail.com Fri Oct 29 14:23:22 2010 From: ellisonbg at gmail.com (Brian Granger) Date: Fri, 29 Oct 2010 11:23:22 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References: Message-ID: > As for non-blockers, we have: > > - the parallel code is not in a good situation right now: we have a > few regressions re. the Twisted 0.10.1 code (e.g. the SGE code isn't > ported yet), the Twisted winhpc scheduler is only in 0.11, and while > the new zmq tools are looking great, they are NOT production-ready > quite yet. ?In summary, we'll have to warn in bright, blinking pink > letters 1995-style, everyone who uses the parallel code in production > systems to stick with the 0.10 series for a little longer. ?Annoying, > yes, but unfortunately such is life. I want to at least proposed the following solution to this: Remove all of the twisted stuff from 0.11 and put the new zmq stuff in place as a prototype. Here is my logic: * The Twisted parallel stuff is *already* broken in 0.11 and if anyone has stable code running on it, they should be using 0.10. * If someone is happy to run non-production ready code, there is no reason they should be using the Twisted stuff, they should use the pyzmq stuff. * Twisted is a *massive* burden on our code base: - For package managers, it brings in Twisted, Foolscap and zope.interface. - It makes our test suite unstable and fragile because we have to run tests in subprocesses and use trial sometimes and nose other times. - It is a huge # of LOC. - It means that most of our codebase is Python 3 ready. There are lots of cons to this proposal: * That is really quick to drop support for the Twisted stuff. * We may piss some people off. * It possibly means maintaining the 0.10 series longer than we imagined. * We don't have a security story for the pyzmq parallel stuff yet. I am not convinced this is the right thing to do, but the benefits are significant. Cheers, Brian -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgranger at calpoly.edu ellisonbg at gmail.com From fperez.net at gmail.com Fri Oct 29 19:34:11 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 29 Oct 2010 16:34:11 -0700 Subject: [IPython-dev] Error in test_unicode for InputSplitter In-Reply-To: References: Message-ID: Hey, On Fri, Oct 29, 2010 at 11:15 AM, Brian Granger wrote: > > ? ?return ''.join(buffer).encode(self.encoding) > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position > 2: ordinal not in range(128) No, I don't see it here on trunk: Ran 9 test groups in 45.840s Status: OK Is this on your unmodified copy of trunk? If so that's bad news, because it means we have a unicode problem that's platform-dependent :( But let's not worry too much about it yet, since we know we have a full complement of problems with unicode anyways. Once I've had a go at that code, we'll make sure this is gone. Thanks for pointing it out though. I've made a note about it in our main unicode ticket just to make sure we don't forget: http://github.com/ipython/ipython/issues/#issue/25/comment/503524 Cheers, f From benjaminrk at gmail.com Fri Oct 29 19:48:56 2010 From: benjaminrk at gmail.com (MinRK) Date: Fri, 29 Oct 2010 16:48:56 -0700 Subject: [IPython-dev] Error in test_unicode for InputSplitter In-Reply-To: References: Message-ID: Check the default encoding. It could be that your Python's default encoding is ascii, or some other such thing, causing a problem. sys.getdefaultencoding() Somethings this is utf8, sometimes it's ascii. -MinRK On Fri, Oct 29, 2010 at 16:34, Fernando Perez wrote: > Hey, > > On Fri, Oct 29, 2010 at 11:15 AM, Brian Granger > wrote: > > > > return ''.join(buffer).encode(self.encoding) > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position > > 2: ordinal not in range(128) > > No, I don't see it here on trunk: > > Ran 9 test groups in 45.840s > > Status: > OK > > Is this on your unmodified copy of trunk? If so that's bad news, > because it means we have a unicode problem that's platform-dependent > :( > > But let's not worry too much about it yet, since we know we have a > full complement of problems with unicode anyways. Once I've had a go > at that code, we'll make sure this is gone. > > Thanks for pointing it out though. I've made a note about it in our > main unicode ticket just to make sure we don't forget: > > http://github.com/ipython/ipython/issues/#issue/25/comment/503524 > > Cheers, > > f > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Fri Oct 29 20:26:58 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 29 Oct 2010 19:26:58 -0500 Subject: [IPython-dev] Error in test_unicode for InputSplitter In-Reply-To: References:

Message-ID: On 2010-10-29 18:48 , MinRK wrote: > Check the default encoding. It could be that your Python's default encoding is > ascii, or some other such thing, causing a problem. > > sys.getdefaultencoding() > > Somethings this is utf8, sometimes it's ascii. It should never, ever be anything but ascii. If you have it set to something else, you have a broken Python. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Oct 29 20:35:52 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 29 Oct 2010 19:35:52 -0500 Subject: [IPython-dev] Error in test_unicode for InputSplitter In-Reply-To: References: Message-ID: On 2010-10-29 18:34 , Fernando Perez wrote: > Hey, > > On Fri, Oct 29, 2010 at 11:15 AM, Brian Granger wrote: >> >> return ''.join(buffer).encode(self.encoding) >> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position >> 2: ordinal not in range(128) > > No, I don't see it here on trunk: > > Ran 9 test groups in 45.840s > > Status: > OK > > Is this on your unmodified copy of trunk? If so that's bad news, > because it means we have a unicode problem that's platform-dependent > :( I can verify this test failure on OS X with an unmodified trunk. The code is just wrong (at least on Python 2) since it calls .encode() on a byte string, not a unicode string. You've never decoded it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From benjaminrk at gmail.com Fri Oct 29 20:39:03 2010 From: benjaminrk at gmail.com (Min RK) Date: Fri, 29 Oct 2010 17:39:03 -0700 Subject: [IPython-dev] Error in test_unicode for InputSplitter In-Reply-To: References:

Message-ID: <1334DB12-86AA-4139-9EC2-C15E34863C6C@gmail.com> My mistake. I do remember there being an earlier problem related to someone's default encoding. What would be broken if Python's default encoding were utf8? Why is this a configurable option, if everything but ascii is broken? -MinRK On Oct 29, 2010, at 17:26, Robert Kern wrote: > On 2010-10-29 18:48 , MinRK wrote: >> Check the default encoding. It could be that your Python's default encoding is >> ascii, or some other such thing, causing a problem. >> >> sys.getdefaultencoding() >> >> Somethings this is utf8, sometimes it's ascii. > > It should never, ever be anything but ascii. If you have it set to something > else, you have a broken Python. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > that is made terrible by our own mad attempt to interpret it as though it had > an underlying truth." > -- Umberto Eco > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev From robert.kern at gmail.com Fri Oct 29 20:42:31 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 29 Oct 2010 19:42:31 -0500 Subject: [IPython-dev] Error in test_unicode for InputSplitter In-Reply-To: <1334DB12-86AA-4139-9EC2-C15E34863C6C@gmail.com> References:

<1334DB12-86AA-4139-9EC2-C15E34863C6C@gmail.com> Message-ID: On 2010-10-29 19:39 , Min RK wrote: > My mistake. I do remember there being an earlier problem related to someone's default encoding. > > What would be broken if Python's default encoding were utf8? Things like this would work differently on different people's machines. Internally, the __hash__/__eq__ relationship between unicode and str objects would fail to hold. > Why is this a configurable option, if everything but ascii is broken? It's not a configurable option. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Oct 29 20:39:55 2010 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 29 Oct 2010 19:39:55 -0500 Subject: [IPython-dev] Error in test_unicode for InputSplitter In-Reply-To: References:

Message-ID: On 2010-10-29 19:26 , Robert Kern wrote: > On 2010-10-29 18:48 , MinRK wrote: >> Check the default encoding. It could be that your Python's default encoding is >> ascii, or some other such thing, causing a problem. >> >> sys.getdefaultencoding() >> >> Somethings this is utf8, sometimes it's ascii. > > It should never, ever be anything but ascii. If you have it set to something > else, you have a broken Python. Or rather, on Python 2, it must always be ascii. On Python 3, it must always be utf-8. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From fperez.net at gmail.com Fri Oct 29 21:01:21 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 29 Oct 2010 18:01:21 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References: Message-ID: Hi Thomas, On Fri, Oct 29, 2010 at 3:30 AM, Thomas Kluyver wrote: > I don't know what sort of QA process it would need to go through, but could > we look at doing a parallel release of IPython on Python 3? As it stands, it > would have to just be the plain terminal version, but I think that's still > valuable. I'm fairly happy that that side is working, and it passes all the > automated tests. > > The Qt frontend is tantalisingly close to working, largely thanks to MinRK's > work with pyzmq, and I hope I can clear the remaining roadblocks soon, but I > don't know if it will be ready in a month. And if the new HTML frontend can > work with it too, that's just icing on the cake. > This would be fantastic. Given we have exactly *zero* official IPython on python3, pretty much anything that works reasonably well at the terminal would be great to have. As long as the test suite and basic interactive use are there, it will already be a great starting point for anyone wanting to use python3. With numpy already on python3, we really want this moving. To get things going, I've made a new Python3 team for the IPython organization: https://github.com/organizations/ipython/teams For now I only added you and me to it because I don't want to forcefully volunteer anyone else :) But anyone who wants to help out with this, just write back on list and I'll add you right away. Thomas, I think the easiest way forward will be to move your py3 repo to be owned by this team, as that will make it easier to get collaboration from others, pull requests reviewed, etc. Doing that on a personally-held repo is awkward. You need to ask github manually to do it: http://support.github.com/discussions/organization-issues/123-request-to-move-a-personal-repo-to-an-organization Once they've moved it, we'll announce it on the -user list as well as the numpy one, and hopefully others will begin using this as well, testing it, etc. Many thanks for taking the initiative on this!!! Regards, f From ellisonbg at gmail.com Fri Oct 29 21:35:53 2010 From: ellisonbg at gmail.com (Brian Granger) Date: Fri, 29 Oct 2010 18:35:53 -0700 Subject: [IPython-dev] Error in test_unicode for InputSplitter In-Reply-To: References:

Message-ID: Python 2.6.5 |CUSTOM| (r265:79063, May 28 2010, 15:13:03) Type "copyright", "credits" or "license" for more information. IPython 0.11.alpha1.git -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. In [1]: import sys In [2]: sys.getdefaultencoding() Out[2]: 'ascii' This is EPD on the Mac... Brian On Fri, Oct 29, 2010 at 5:39 PM, Robert Kern wrote: > On 2010-10-29 19:26 , Robert Kern wrote: >> On 2010-10-29 18:48 , MinRK wrote: >>> Check the default encoding. ?It could be that your Python's default encoding is >>> ascii, or some other such thing, causing a problem. >>> >>> sys.getdefaultencoding() >>> >>> Somethings this is utf8, sometimes it's ascii. >> >> It should never, ever be anything but ascii. If you have it set to something >> else, you have a broken Python. > > Or rather, on Python 2, it must always be ascii. On Python 3, it must always be > utf-8. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > ?that is made terrible by our own mad attempt to interpret it as though it had > ?an underlying truth." > ? -- Umberto Eco > > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgranger at calpoly.edu ellisonbg at gmail.com From fperez.net at gmail.com Fri Oct 29 23:28:19 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 29 Oct 2010 20:28:19 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References:

Message-ID: On Fri, Oct 29, 2010 at 11:23 AM, Brian Granger wrote: > Remove all of the twisted stuff from 0.11 and put the new zmq stuff in > place as a prototype. > > Here is my logic: > > * The Twisted parallel stuff is *already* broken in 0.11 and if anyone > has stable code running on it, they should be using 0.10. > * If someone is happy to run non-production ready code, there is no > reason they should be using the Twisted stuff, they should use the > pyzmq stuff. > * Twisted is a *massive* burden on our code base: > ?- For package managers, it brings in Twisted, Foolscap and zope.interface. > ?- It makes our test suite unstable and fragile because we have to > run tests in subprocesses and use trial sometimes and nose other > times. > ?- It is a huge # of LOC. > ?- It means that most of our codebase is Python 3 ready. > > There are lots of cons to this proposal: > > * That is really quick to drop support for the Twisted stuff. > * We may piss some people off. > * It possibly means maintaining the 0.10 series longer than we imagined. > * We don't have a security story for the pyzmq parallel stuff yet. I have to say that I simply didn't have Brian's boldness to propose this, but I think it's the right thing to do, ultimately. It *is* painful in the short term, but it's also the honest approach. I keep forgetting but Brian reminded me that even the Twisted-based code in 0.11 has serious regressions re. the 0.10.x series, since in the big refactoring for 0.11 not quite everything made it through. The 0.10 maintenance doesn't worry me a whole lot: as long as we limit it to small changes, by now merging them as self-contained pull requests is really easy (as I just did recently with the ones Paul and Tom sent). And rolling out a new release when the total delta is small is actually not that much work. So I'm totally +1 on this radical, but I think ultimately beneficial, approach. It's important to keep in mind that doing this will lift a big load off our shoulders, and we're a small enough team that this benefit is significant. It will let us concentrate on moving the new machinery forward quickly without having to worry about the large Twisted code. It will also help Thomas with his py3 efforts, as it's one less thing he has to keep getting out of his way. Concrete plan: - Wait a week or two for feedback. - If we decide to move ahead, make a shared branch on the main repo where we can do this work and review it, with all having the chance to contribute while it happens. - Move all twisted-using code (IPython/kernel and some code in IPython/testing) into IPython/deathrow. This will let anyone who reall wants it find it easily, without having to dig through version control history. Note that deathrow does *not* make it into official release tarballs. Cheers, f From benjaminrk at gmail.com Sat Oct 30 01:55:32 2010 From: benjaminrk at gmail.com (MinRK) Date: Fri, 29 Oct 2010 22:55:32 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References:

Message-ID: This is more agressive than I expected, I'll have to get the new parallel stuff in gear. The main roadblock for me is merging work into the Kernels. I plan to spend tomorrow working on getting the new parallel code ready for review, and identifying what needs to happen with code in master in order for this to go into 0.11. The only work that needs merge rather than drop-in is in Kernels and Session. I expect that just using the new Session will just be fine after a rewview, but getting the existing Kernels to provide what is necessary for the parallel code will be some work, and I'll try to identify exactly what that will look like. The main things I know already: * Names should change (GH-178). It's really a coincidence that we had just one action per socket type, and the parallel code has several sockets of the same type, and some actions that can be on different socket types, depending on the scheduler. * Use IOLoop/ZMQStream - this isn't necessarily critical, and I can probably do it with a subclass if we don't want it in the main kernels. * apply_request. This should be all new code, and shouldn't collide with anything. Let me know what I can do to help things along. -MinRK On Fri, Oct 29, 2010 at 20:28, Fernando Perez wrote: > On Fri, Oct 29, 2010 at 11:23 AM, Brian Granger > wrote: > > Remove all of the twisted stuff from 0.11 and put the new zmq stuff in > > place as a prototype. > > > > Here is my logic: > > > > * The Twisted parallel stuff is *already* broken in 0.11 and if anyone > > has stable code running on it, they should be using 0.10. > > * If someone is happy to run non-production ready code, there is no > > reason they should be using the Twisted stuff, they should use the > > pyzmq stuff. > > * Twisted is a *massive* burden on our code base: > > - For package managers, it brings in Twisted, Foolscap and > zope.interface. > > - It makes our test suite unstable and fragile because we have to > > run tests in subprocesses and use trial sometimes and nose other > > times. > > - It is a huge # of LOC. > > - It means that most of our codebase is Python 3 ready. > > > > There are lots of cons to this proposal: > > > > * That is really quick to drop support for the Twisted stuff. > > * We may piss some people off. > > * It possibly means maintaining the 0.10 series longer than we imagined. > > * We don't have a security story for the pyzmq parallel stuff yet. > > I have to say that I simply didn't have Brian's boldness to propose > this, but I think it's the right thing to do, ultimately. It *is* > painful in the short term, but it's also the honest approach. I keep > forgetting but Brian reminded me that even the Twisted-based code in > 0.11 has serious regressions re. the 0.10.x series, since in the big > refactoring for 0.11 not quite everything made it through. > > The 0.10 maintenance doesn't worry me a whole lot: as long as we limit > it to small changes, by now merging them as self-contained pull > requests is really easy (as I just did recently with the ones Paul and > Tom sent). And rolling out a new release when the total delta is > small is actually not that much work. > > So I'm totally +1 on this radical, but I think ultimately beneficial, > approach. It's important to keep in mind that doing this will lift a > big load off our shoulders, and we're a small enough team that this > benefit is significant. It will let us concentrate on moving the new > machinery forward quickly without having to worry about the large > Twisted code. It will also help Thomas with his py3 efforts, as it's > one less thing he has to keep getting out of his way. > > Concrete plan: > > - Wait a week or two for feedback. > - If we decide to move ahead, make a shared branch on the main repo > where we can do this work and review it, with all having the chance to > contribute while it happens. > - Move all twisted-using code (IPython/kernel and some code in > IPython/testing) into IPython/deathrow. This will let anyone who > reall wants it find it easily, without having to dig through version > control history. Note that deathrow does *not* make it into official > release tarballs. > > Cheers, > > f > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ellisonbg at gmail.com Sat Oct 30 02:25:18 2010 From: ellisonbg at gmail.com (Brian Granger) Date: Fri, 29 Oct 2010 23:25:18 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References:

Message-ID: Min, On Fri, Oct 29, 2010 at 10:55 PM, MinRK wrote: > This is more agressive than I expected, I'll have to get the new parallel > stuff in gear. If you stopped writing great code, we wouldn't be tempted to do crazy things like this ;-) > The main roadblock for me is merging work into the Kernels. ?I plan to spend > tomorrow working on getting the new parallel code ready for review, and > identifying what needs to happen with code in master in order for this to go > into 0.11. ?The only work that needs merge rather than drop-in is in Kernels > and Session. ?I expect that just using the new Session will just be fine > after a rewview, but getting the existing Kernels to provide what is > necessary for the parallel code will be some work, and I'll try to identify > exactly what that will look like. Are you thinking of having only one top-level kernel script that handles both the parallel computing stuff and the interactive IPython? I think the idea of that is fantastic, but I am not sure we need to have all of that working to merge your stuff. I am not opposed to attempting this before/during the merge, but I don't view it as absolutely needed. Also, it may make sense to review your code standalone first and then discuss merging the kernel and session stuff with what we already have. > The main things I know already: > * Names should change (GH-178). It's really a coincidence that we had just > one action per socket type, and the parallel code has several sockets of the > same type, and some actions that can be on different socket types, depending > on the scheduler. Yep. > * Use IOLoop/ZMQStream - this isn't necessarily critical, and I can probably > do it with a subclass if we don't want it in the main kernels. At this point I think that zmqstream has stablized enough that we *should* be using it in the kernel and kernel manager code anyways. I am completely fine with this. > * apply_request. This should be all new code, and shouldn't collide with > anything. Ok. One other point that Fernando and I talked about is actually shipping the rest of tornado with pyzmq. I have been thinking more about the architecture of the html notebook that James has been working on and it is an absolutely perfect fit for implementing the server using our zmq enabled Tornado event loop with tornado's regular http handling. It would also give us ssl support, authentication and lots of other web server goodies like websockets. If we did this, I think it would be possible to have a decent prototype of James' html notebook in 0.11. What do you think about this Min? We are already shipping a good portion of tornado already with pyzmq and the rest is just a dozen or so .py files (there is one .c file that we don't need for python 2.6 and up). Eventually I would like to contribute our ioloop.py and zmqstream to tornado itself, but I don't think we have to worry about that yet. Also, moving tornado into pyzmq would allow us to so secure https connections for the parallel computing client - controller connection. Cheers, Brian > Let me know what I can do to help things along. > -MinRK > > On Fri, Oct 29, 2010 at 20:28, Fernando Perez wrote: >> >> On Fri, Oct 29, 2010 at 11:23 AM, Brian Granger >> wrote: >> > Remove all of the twisted stuff from 0.11 and put the new zmq stuff in >> > place as a prototype. >> > >> > Here is my logic: >> > >> > * The Twisted parallel stuff is *already* broken in 0.11 and if anyone >> > has stable code running on it, they should be using 0.10. >> > * If someone is happy to run non-production ready code, there is no >> > reason they should be using the Twisted stuff, they should use the >> > pyzmq stuff. >> > * Twisted is a *massive* burden on our code base: >> > ?- For package managers, it brings in Twisted, Foolscap and >> > zope.interface. >> > ?- It makes our test suite unstable and fragile because we have to >> > run tests in subprocesses and use trial sometimes and nose other >> > times. >> > ?- It is a huge # of LOC. >> > ?- It means that most of our codebase is Python 3 ready. >> > >> > There are lots of cons to this proposal: >> > >> > * That is really quick to drop support for the Twisted stuff. >> > * We may piss some people off. >> > * It possibly means maintaining the 0.10 series longer than we imagined. >> > * We don't have a security story for the pyzmq parallel stuff yet. >> >> I have to say that I simply didn't have Brian's boldness to propose >> this, but I think it's the right thing to do, ultimately. ?It *is* >> painful in the short term, but it's also the honest approach. ?I keep >> forgetting but Brian reminded me that even the Twisted-based code in >> 0.11 has serious regressions re. the 0.10.x series, since in the big >> refactoring for 0.11 not quite everything made it through. >> >> The 0.10 maintenance doesn't worry me a whole lot: as long as we limit >> it to small changes, by now merging them as self-contained pull >> requests is really easy (as I just did recently with the ones Paul and >> Tom sent). ?And rolling out a new release when the total delta is >> small is actually not that much work. >> >> So I'm totally +1 on this radical, but I think ultimately beneficial, >> approach. ?It's important to keep in mind that doing this will lift a >> big load off our shoulders, and we're a small enough team that this >> benefit is significant. ?It will let us concentrate on moving the new >> machinery forward quickly without having to worry about the large >> Twisted code. ?It will also help Thomas with his py3 efforts, as it's >> one less thing he has to keep getting out of his way. >> >> Concrete plan: >> >> - Wait a week or two for feedback. >> - If we decide to move ahead, make a shared branch on the main repo >> where we can do this work and review it, with all having the chance to >> contribute while it happens. >> - Move all twisted-using code (IPython/kernel and some code in >> IPython/testing) into IPython/deathrow. ?This will let anyone who >> reall wants it find it easily, without having to dig through version >> control history. ?Note that deathrow does *not* make it into official >> release tarballs. >> >> Cheers, >> >> f >> _______________________________________________ >> IPython-dev mailing list >> IPython-dev at scipy.org >> http://mail.scipy.org/mailman/listinfo/ipython-dev > > -- Brian E. Granger, Ph.D. Assistant Professor of Physics Cal Poly State University, San Luis Obispo bgranger at calpoly.edu ellisonbg at gmail.com From benjaminrk at gmail.com Sat Oct 30 03:10:44 2010 From: benjaminrk at gmail.com (MinRK) Date: Sat, 30 Oct 2010 00:10:44 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References:

Message-ID: On Fri, Oct 29, 2010 at 23:25, Brian Granger wrote: > Min, > > On Fri, Oct 29, 2010 at 10:55 PM, MinRK wrote: > > This is more agressive than I expected, I'll have to get the new parallel > > stuff in gear. > > If you stopped writing great code, we wouldn't be tempted to do crazy > things like this ;-) > > > The main roadblock for me is merging work into the Kernels. I plan to > spend > > tomorrow working on getting the new parallel code ready for review, and > > identifying what needs to happen with code in master in order for this to > go > > into 0.11. The only work that needs merge rather than drop-in is in > Kernels > > and Session. I expect that just using the new Session will just be fine > > after a rewview, but getting the existing Kernels to provide what is > > necessary for the parallel code will be some work, and I'll try to > identify > > exactly what that will look like. > > Are you thinking of having only one top-level kernel script that > handles both the parallel computing stuff and the interactive IPython? > I think the idea of that is fantastic, but I am not sure we need to > have all of that working to merge your stuff. I am not opposed to > attempting this before/during the merge, but I don't view it as > absolutely needed. Also, it may make sense to review your code > standalone first and then discuss merging the kernel and session stuff > with what we already have. > I was thinking that we already have a remote execution object, and the only difference between the two is the connection patterns. New features/bugfixes will likely want to be shared by both. My StreamKernel was derived from the original pykernel, but I kept working on it while you were developing on it, so they diverged. I think they can be merged, as long as we do a few things, mostly to do with abstracting the connections: * allow Kernels to connect, not just bind * use action-based, not socket-type names * allow execution requests to come from a *list* of connections, not just one * use sessions/ioloop instead of direct send/recv_json I also think using a KernelManager would be good, because it gets nice process management (restart the kernel, etc.), and I can't really do that without a Kernel, but I could subclass. Related question: why is ipkernel not a subclass of pykernel? There's lots of identical code there. > > > The main things I know already: > > * Names should change (GH-178). It's really a coincidence that we had > just > > one action per socket type, and the parallel code has several sockets of > the > > same type, and some actions that can be on different socket types, > depending > > on the scheduler. > > Yep. > > > * Use IOLoop/ZMQStream - this isn't necessarily critical, and I can > probably > > do it with a subclass if we don't want it in the main kernels. > > At this point I think that zmqstream has stablized enough that we > *should* be using it in the kernel and kernel manager code anyways. I > am completely fine with this. > > > * apply_request. This should be all new code, and shouldn't collide with > > anything. > > Ok. > > One other point that Fernando and I talked about is actually shipping > the rest of tornado with pyzmq. I have been thinking more about the > architecture of the html notebook that James has been working on and > it is an absolutely perfect fit for implementing the server using our > zmq enabled Tornado event loop with tornado's regular http handling. > It would also give us ssl support, authentication and lots of other > web server goodies like websockets. If we did this, I think it would > be possible to have a decent prototype of James' html notebook in > 0.11. What do you think about this Min? We are already shipping a > good portion of tornado already with pyzmq and the rest is just a > dozen or so .py files (there is one .c file that we don't need for > python 2.6 and up). > Eventually I would like to contribute our ioloop.py and zmqstream to > tornado itself, but I don't think we have to worry about that yet. > I'm not very familiar with Tornado other than our use in pyzmq. If we can use it for authentication without significant performance penalty, then that's a pretty big deal, and well worth it. It sounds like it would definitely provide a good toolkit for web backends, so using it is probably a good idea. I'm not sure that it should be *shipped* with pyzmq, though. I think it would be fine to ship with IPython if we use it there, but I don't see a need to include it inside pyzmq. If we depend on it, then depend on it in PyPI, but if it's only for some extended functionality, I don't see any problem with asking people to install it, since it is easy_installable (and apt-installable on Ubuntu). PyZMQ is a pretty low-level library - I don't think shipping someone else's project inside it is a good idea unless there are significant benefits. > > Also, moving tornado into pyzmq would allow us to so secure https > connections for the parallel computing client - controller connection. > Secure connections would be *great* if the performance is good enough. > > Cheers, > > Brian > > > Let me know what I can do to help things along. > > -MinRK > > > > On Fri, Oct 29, 2010 at 20:28, Fernando Perez > wrote: > >> > >> On Fri, Oct 29, 2010 at 11:23 AM, Brian Granger > >> wrote: > >> > Remove all of the twisted stuff from 0.11 and put the new zmq stuff in > >> > place as a prototype. > >> > > >> > Here is my logic: > >> > > >> > * The Twisted parallel stuff is *already* broken in 0.11 and if anyone > >> > has stable code running on it, they should be using 0.10. > >> > * If someone is happy to run non-production ready code, there is no > >> > reason they should be using the Twisted stuff, they should use the > >> > pyzmq stuff. > >> > * Twisted is a *massive* burden on our code base: > >> > - For package managers, it brings in Twisted, Foolscap and > >> > zope.interface. > >> > - It makes our test suite unstable and fragile because we have to > >> > run tests in subprocesses and use trial sometimes and nose other > >> > times. > >> > - It is a huge # of LOC. > >> > - It means that most of our codebase is Python 3 ready. > >> > > >> > There are lots of cons to this proposal: > >> > > >> > * That is really quick to drop support for the Twisted stuff. > >> > * We may piss some people off. > >> > * It possibly means maintaining the 0.10 series longer than we > imagined. > >> > * We don't have a security story for the pyzmq parallel stuff yet. > >> > >> I have to say that I simply didn't have Brian's boldness to propose > >> this, but I think it's the right thing to do, ultimately. It *is* > >> painful in the short term, but it's also the honest approach. I keep > >> forgetting but Brian reminded me that even the Twisted-based code in > >> 0.11 has serious regressions re. the 0.10.x series, since in the big > >> refactoring for 0.11 not quite everything made it through. > >> > >> The 0.10 maintenance doesn't worry me a whole lot: as long as we limit > >> it to small changes, by now merging them as self-contained pull > >> requests is really easy (as I just did recently with the ones Paul and > >> Tom sent). And rolling out a new release when the total delta is > >> small is actually not that much work. > >> > >> So I'm totally +1 on this radical, but I think ultimately beneficial, > >> approach. It's important to keep in mind that doing this will lift a > >> big load off our shoulders, and we're a small enough team that this > >> benefit is significant. It will let us concentrate on moving the new > >> machinery forward quickly without having to worry about the large > >> Twisted code. It will also help Thomas with his py3 efforts, as it's > >> one less thing he has to keep getting out of his way. > >> > >> Concrete plan: > >> > >> - Wait a week or two for feedback. > >> - If we decide to move ahead, make a shared branch on the main repo > >> where we can do this work and review it, with all having the chance to > >> contribute while it happens. > >> - Move all twisted-using code (IPython/kernel and some code in > >> IPython/testing) into IPython/deathrow. This will let anyone who > >> reall wants it find it easily, without having to dig through version > >> control history. Note that deathrow does *not* make it into official > >> release tarballs. > >> > >> Cheers, > >> > >> f > >> _______________________________________________ > >> IPython-dev mailing list > >> IPython-dev at scipy.org > >> http://mail.scipy.org/mailman/listinfo/ipython-dev > > > > > > > > -- > Brian E. Granger, Ph.D. > Assistant Professor of Physics > Cal Poly State University, San Luis Obispo > bgranger at calpoly.edu > ellisonbg at gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jorgen.stenarson at bostream.nu Sat Oct 30 03:12:13 2010 From: jorgen.stenarson at bostream.nu (=?ISO-8859-1?Q?J=F6rgen_Stenarson?=) Date: Sat, 30 Oct 2010 09:12:13 +0200 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References:

Message-ID: <4CCBC54D.9030807@bostream.nu> Hi, Brian Granger skrev 2010-10-30 08:25: > Eventually I would like to contribute our ioloop.py and zmqstream to > tornado itself, but I don't think we have to worry about that yet. > > Also, moving tornado into pyzmq would allow us to so secure https > connections for the parallel computing client - controller connection. I did a quick websearch for tornado and found tornadoweb.org is that what you are talking about? Looking at their webpage I can't see anything about windows. What would going that route mean for supporting the windows platform? /J?rgen From benjaminrk at gmail.com Sat Oct 30 03:17:12 2010 From: benjaminrk at gmail.com (MinRK) Date: Sat, 30 Oct 2010 00:17:12 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: <4CCBC54D.9030807@bostream.nu> References:

<4CCBC54D.9030807@bostream.nu> Message-ID: On Sat, Oct 30, 2010 at 00:12, J?rgen Stenarson < jorgen.stenarson at bostream.nu> wrote: > Hi, > > Brian Granger skrev 2010-10-30 08:25: > > Eventually I would like to contribute our ioloop.py and zmqstream to > > tornado itself, but I don't think we have to worry about that yet. > > > > Also, moving tornado into pyzmq would allow us to so secure https > > connections for the parallel computing client - controller connection. > > I did a quick websearch for tornado and found tornadoweb.org is that > what you are talking about? Looking at their webpage I can't see > anything about windows. What would going that route mean for supporting > the windows platform? > Yes, that's the right tornado. Tornado is (almost) pure Python, and at least the parts we use work just fine on Windows. > > /J?rgen > _______________________________________________ > IPython-dev mailing list > IPython-dev at scipy.org > http://mail.scipy.org/mailman/listinfo/ipython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gael.varoquaux at normalesup.org Sat Oct 30 03:36:03 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sat, 30 Oct 2010 09:36:03 +0200 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References:

Message-ID: <20101030073603.GA1308@phare.normalesup.org> Hi there, On Fri, Oct 29, 2010 at 08:28:19PM -0700, Fernando Perez wrote: > I have to say that I simply didn't have Brian's boldness to propose > this, but I think it's the right thing to do, ultimately. It *is* > painful in the short term, but it's also the honest approach. I keep > forgetting but Brian reminded me that even the Twisted-based code in > 0.11 has serious regressions re. the 0.10.x series, since in the big > refactoring for 0.11 not quite everything made it through. I haven't been contributing to the discussion, because I don't have any time to contribute. Here is my gut feeling and an end user: Progress made in 0.11 looks awesome. However, I am not sure what the net gain is for an user. It seems to me that the core architecture is there, but the end-user aspects are not finished. The twisted code is gone, but I hear that the pyzmq code to replace it is not stable yet. So, here is the question: why release now, and not in 6 months? If you release now, distributions will almost automatically package 0.11, so it will land on user's boxes. This is a naive question, I may very well have missed important aspect of the discussion. Ga?l From shr066 at gmail.com Sat Oct 30 10:33:55 2010 From: shr066 at gmail.com (Steve Rogers) Date: Sat, 30 Oct 2010 08:33:55 -0600 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: <20101030073603.GA1308@phare.normalesup.org> References:

<20101030073603.GA1308@phare.normalesup.org> Message-ID: +1 on transitioning from Twisted to pyzmq for the 0.11 release.. I'm investigating pyzmq for a project that requires some security and I may be able to help in that area. -- Steve Rogers http://www.linkedin.com/in/shrogers ?Do what you can, with what you have, where you are.? -- Theodore Roosevelt -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjaminrk at gmail.com Sat Oct 30 16:27:22 2010 From: benjaminrk at gmail.com (MinRK) Date: Sat, 30 Oct 2010 13:27:22 -0700 Subject: [IPython-dev] Parallel IPython with ZMQ slides Message-ID: As requested, the slides from my presentation of the new parallel code to Py4Science last week: http://ptsg.berkeley.edu/~minrk/ipzmq/ipzmq.p4s.pdf And code for some of the demos (+ NetworkX DAG dependencies): http://ptsg.berkeley.edu/~minrk/ipzmq/demo.zip -MinRK -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjaminrk at gmail.com Sat Oct 30 17:42:53 2010 From: benjaminrk at gmail.com (MinRK) Date: Sat, 30 Oct 2010 14:42:53 -0700 Subject: [IPython-dev] ipqt doc started Message-ID: Fernando, I realized that I totally forgot about the skeleton doc for the qt console last week. Sorry! I'm putting it together in my ipqt-docs branch, but I will do it directly in master, if you want. You can see it in progress: http://ptsg.berkeley.edu/~minrk/ipdocs/interactive/qtconsole.html -MinRK -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sat Oct 30 18:24:28 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 30 Oct 2010 15:24:28 -0700 Subject: [IPython-dev] ipqt doc started In-Reply-To: References: Message-ID: Hey, On Sat, Oct 30, 2010 at 2:42 PM, MinRK wrote: > Fernando, > I realized that I totally forgot about the skeleton doc for the qt console > last week. Sorry! > I'm putting it together in my ipqt-docs branch, but I will do it directly in > master, if you want. > You can see it in progress: > http://ptsg.berkeley.edu/~minrk/ipdocs/interactive/qtconsole.html > -MinRK This is great, thanks! Just push straight to trunk: for pure doc work, especially of this kind (where anything is an improvement, since we have zero), there's no need to add a layer of friction; we trust you :) Cheers, f From fperez.net at gmail.com Sat Oct 30 18:32:22 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 30 Oct 2010 15:32:22 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References:

Message-ID: On Fri, Oct 29, 2010 at 10:55 PM, MinRK wrote: > This is more agressive than I expected, I'll have to get the new parallel > stuff in gear. Sorry if it sounded like we were putting you on the spot, that was most definitely *not* our intention. As Brian said, if you hadn't done such a crazy good job, we wouldn't even have dreamed of going in this direction :) But I think that Brian's point, with which I agree, is basically that 0.11 should go with the pyzmq code *even if it's very raw*, because the 0.11 Twisted code is broken enough that it would be a dis-service to our users to lure them into that code. 0.11 will be released as a "tech preview" release, prominently and clearly labeled as not for production systems, with a note to distributors that they probably should *not* package it by default as a replacement for 0.10 yet (that may take another two or three releases before it's a sensible thing to do). But the Twisted code in 0.10 is just not in really usable shape, and this is a way of being upfront and honest about it. So don't worry too much on your side, we're *not* putting all the burden on your shoulders. Cheers, f From fperez.net at gmail.com Sat Oct 30 18:32:47 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 30 Oct 2010 15:32:47 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References:

<20101030073603.GA1308@phare.normalesup.org> Message-ID: On Sat, Oct 30, 2010 at 7:33 AM, Steve Rogers wrote: > > I'm investigating pyzmq for a project that requires some security and I may > be able to help in that area That would be fantastic, by all means pitch in. Regards, f From fperez.net at gmail.com Sat Oct 30 18:47:29 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 30 Oct 2010 15:47:29 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: <20101030073603.GA1308@phare.normalesup.org> References:

<20101030073603.GA1308@phare.normalesup.org> Message-ID: On Sat, Oct 30, 2010 at 12:36 AM, Gael Varoquaux wrote: > I haven't been contributing to the discussion, because I don't have any > time to contribute. Here is my gut feeling and an end user: > > Progress made in 0.11 looks awesome. However, I am not sure what the net > gain is for an user. It seems to me that the core architecture is there, > but the end-user aspects are not finished. The twisted code is gone, but > I hear that the pyzmq code to replace it is not stable yet. So, here is > the question: why release now, and not in 6 months? If you release now, > distributions will almost automatically package 0.11, so it will land on > user's boxes. > > This is a naive question, I may very well have missed important aspect of > the discussion. Yes, you missed the whole point, I'm afraid. 0.11 has major new *features* for end-users that are already very usable. The Qt console isn't perfect, but it's very, very, very nice as an everyday working environment, and it has a massive amount of improvements for many use cases over the terminal-only one. James' html client also offers remote collaboration and zero-install capabilities that we don't have anywhere else, and even as the architecture there is refined, that's already a big win for end users. The new configuration system is also much cleaner and nicer, and the layout of the actual source code is far more rational and navigable than before, a big win for potential new contributors. Most importantly, now there's a proper *architecture* for how the entire ipython machinery works across all modalities (one-process, multi-process interactive, parallel) that is well thought out *and* documented. A data point that indicates that we probably got more things right than wrong was that I spent probably less than one hour explaining to James the messaging design, and without *ever* having done any ipython coding, in a couple of days he came back with a fully working html notebook. Doing something like that with the 0.10-era code would have been unthinkable (as someone who wrestled that code into the ground for embedding as a client, you know full well the nightmare it was, and how much nasty work it took). We've recently had Evan Patterson, Mark Voorhies and James Gao land into the IPython codebase and immediately make real contributions: this is an indicator that for the first time ever, we have a codebase that can actually accept new participants without dragging them into the nightmare-inducing maze of object-oriented spaghetti we had before. Having seen this, the best way to gain contributors is to have people use the new features, find something they like and then find something they don't like but are willing to fix/improve. Furthermore, many many apis have changed in backwards-incompatible ways. It's important that people who had developed other projects using ipython as a library have a chance to adapt their projects, give us feedback so we can fine-tune things to make their job easier, make suggestions of what could be done better from our side, etc. So we want those projects to have a clear starting point where their developers can test things out. If we wait until everything has settled 6 months from now, it may be harder to adjust to their feedback. In summary, both from the perspective of end users and that of developers building atop ipython, we have very good reasons to not delay a release much further (4-8 weeks from now is OK, 6 months is most definitely not). We'll label it clearly so that nobody gets 0.11 when they don't expect it, but we do want these new features out for people to both enjoy and to hammer on while they're still warm enough that we can mold them. Cheers, f From takowl at gmail.com Sat Oct 30 19:18:41 2010 From: takowl at gmail.com (Thomas Kluyver) Date: Sun, 31 Oct 2010 00:18:41 +0100 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References: Message-ID: On 30 October 2010 02:01, Fernando Perez wrote: > Thomas, I think the easiest way forward will be to move your py3 repo > to be owned by this team, as that will make it easier to get > collaboration from others, pull requests reviewed, etc. Doing that on > a personally-held repo is awkward. You need to ask github manually > to do it: > I've asked, but apparently they can't do it, because it's a fork of the repo the IPython organisation already owns: http://support.github.com/discussions/organization-issues/315-request-to-transfer-my-personal-repository-to-an-organisation I guess we could either push my branch to the organisation's existing repo (without merging it into trunk), or set up a new ipython-py3k repository, and push my branch as master there. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.tollerud at gmail.com Sat Oct 30 20:55:33 2010 From: erik.tollerud at gmail.com (Erik Tollerud) Date: Sat, 30 Oct 2010 17:55:33 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References:

<20101030073603.GA1308@phare.normalesup.org> Message-ID: +1 on removing Twisted (mostly from a user perspective) - I've had a number of different strange dependency problems with Twisted while trying to get IPython working on a cluster, so ditching it entirely would make such things easier for experimenting with .11 With that in mind, I'm in the early stages of developing a small application intended for use on the aforementioned cluster. I'd like to write it for use with .11 instead of .10 if possible (thinking of it partly as a test case to help with debugging .11). Because of the network setup and the possible need for remote collaborator input, the ideal situation would be to use the html frontend to get at the cluster. But that's a non-starter if there's no authentication - is it at all realistic to try for the html frontend with authentication in .11 (presumably w/ Tornado, based on the above discussion)? -- Erik Tollerud From fperez.net at gmail.com Sat Oct 30 21:45:55 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 30 Oct 2010 18:45:55 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References: Message-ID: On Sat, Oct 30, 2010 at 4:18 PM, Thomas Kluyver wrote: > I've asked, but apparently they can't do it, because it's a fork of the repo > the IPython organisation already owns: > http://support.github.com/discussions/organization-issues/315-request-to-transfer-my-personal-repository-to-an-organisation Of course, I should have thought of that from the start, sorry for sending you on a wild goose chase. > I guess we could either push my branch to the organisation's existing repo > (without merging it into trunk), or set up a new ipython-py3k repository, > and push my branch as master there. Your wish is my command :) I pushed from your master as it was just now. http://github.com/ipython/ipython-py3k I followed this route because I can imagine work on this repo being a little erratic, and possibly doing large rebases/cleanups before a final merge in the ipython trunk. So having it isolated in a separate repo seems like a cleaner alternative. Cheers, f From fperez.net at gmail.com Sat Oct 30 21:49:32 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Sat, 30 Oct 2010 18:49:32 -0700 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References:

<20101030073603.GA1308@phare.normalesup.org>

Message-ID: On Sat, Oct 30, 2010 at 5:55 PM, Erik Tollerud wrote: > +1 on removing Twisted (mostly from a user perspective) - I've had a > number of different strange dependency problems with Twisted while > trying to get IPython working on a cluster, so ditching it entirely > would make such things easier for experimenting with .11 Thanks for the feedback. > With that in mind, I'm in the early stages of developing a small > application intended for use on the aforementioned cluster. ?I'd like > to write it for use with .11 instead of .10 if possible (thinking of > it partly as a test case to help with debugging .11). ?Because of the > network setup and the possible need for remote collaborator input, the > ideal situation would be to use the html frontend to get at the > cluster. ?But that's a non-starter if there's no authentication - is > it at all realistic to try for the html frontend with authentication > in .11 (presumably w/ Tornado, based on the above discussion)? That's our hope, but obviously I can't promise anything :) But yes, that's one big reason to think of Tornado, so that we can at least have basic auth/ssl support for the web entry point. The kernels by default only listen on localhost, so it's not that bad: you have a problem with being open to other local users, but not to the whole internet. But since the webnb effectively forwards the obscure/multiport/local zmq connections over nice, clean, everybody-understands-it http, it all of a sudden turns an improbable problem into a likely catastrophe. Let's see how the next few weeks go, and all help on this front will be very welcome. Cheers, f From gael.varoquaux at normalesup.org Sun Oct 31 05:00:24 2010 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Sun, 31 Oct 2010 10:00:24 +0100 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) In-Reply-To: References:

<20101030073603.GA1308@phare.normalesup.org> Message-ID: <20101031090024.GA937@phare.normalesup.org> On Sat, Oct 30, 2010 at 03:47:29PM -0700, Fernando Perez wrote: > > Progress made in 0.11 looks awesome. However, I am not sure what the net > > gain is for an user. It seems to me that the core architecture is there, > > but the end-user aspects are not finished. > > [...] > Yes, you missed the whole point, I'm afraid. Fair enough, I should have thought about the Qt-based stuff :), especially since I have been following it quite closely. > 0.11 has major new *features* for end-users that are already very > usable. The Qt console isn't perfect, but it's very, very, very nice > as an everyday working environment, and it has a massive amount of > improvements for many use cases over the terminal-only one. James' > html client also offers remote collaboration and zero-install > capabilities that we don't have anywhere else, and even as the > architecture there is refined, that's already a big win for end users. > [...] My question just boils down to: is it worth giving this win to end users while removing other things they already have? Should a '0.X' release ever temporarily remove features that will come back later? As an outsider from a project, I expect the feature set always to be increasing, or to be simply features to be dropped. > Most importantly, now there's a proper *architecture* for how the > entire ipython machinery works across all modalities (one-process, > multi-process interactive, parallel) that is well thought out *and* > documented. A data point that indicates that we probably got more > things right than wrong was that I spent probably less than one hour > explaining to James the messaging design, and without *ever* having > done any ipython coding, in a couple of days he came back with a fully > working html notebook. I agree with the progress. I can see the gain. It's great, but it seems to be more of a developper point of view that an end user point of view. > We've recently had Evan Patterson, Mark Voorhies and James Gao land > into the IPython codebase and immediately make real contributions: > this is an indicator that for the first time ever, we have a codebase > that can actually accept new participants without dragging them into > the nightmare-inducing maze of object-oriented spaghetti we had > before. [...] Absolutely, I have been witnessing this and I can tangibly feel this improvement. There has clearly been a shift in the project's dynamics that is awesome. Anyhow, it won't impact me much, because at work we are stuck with really old stuff (.8, I believe), and I myself already use the latest code from git updated every once in a while. I am just wondering if, from the point of view of the end user, it is not worth waiting another 6 months to have things stabilize a bit more. Release an 'alpha', or 'technology preview' might be a better message to give. Now I am just going to shut up, and watch the fantastic progress that is being made. Gael From takowl at gmail.com Sun Oct 31 14:19:57 2010 From: takowl at gmail.com (Thomas Kluyver) Date: Sun, 31 Oct 2010 18:19:57 +0000 Subject: [IPython-dev] Starting to plan for 0.11 (this time for real) Message-ID: On 31 October 2010 17:00, wrote: > I am just wondering if, from the point > of view of the end user, it is not worth waiting another 6 months to have > things stabilize a bit more. Release an 'alpha', or 'technology preview' > might be a better message to give. > I'll just chime in on this: if we make an official 'release', so we put out 0.11 and continue development towards 0.12, it is possible that distributions will package it as an upgrade, no matter how clearly we say "not ready for production." This happened quite prominently with KDE 4.0: once it reached 'final release', it got out to users, then suffered a backlash because it wasn't really ready. Obviously IPython isn't quite as high profile as KDE, so it may be easier to avoid that sort of debacle. But there is something to be said for calling things alphas/betas if that's what they are. I'm not saying that I think we should or shouldn't make a release. But I can see the argument against it. Thomas -------------- next part -------------- An HTML attachment was scrubbed... URL: From fperez.net at gmail.com Sun Oct 31 15:08:46 2010 From: fperez.net at gmail.com (Fernando Perez) Date: Sun, 31 Oct 2010 12:08:46 -0700 Subject: [IPython-dev] IPython on python3, for the adventurous... Message-ID: Hi all, thanks to the great work done by Thomas Kluyver, for those of you starting to use Numpy (or other projects) on Python 3 and who wanted to have IPython as your working environment, things are getting off the ground: (py3k)amirbar[py3k]> python3 ipython.py Python 3.1.2 (release31-maint, Sep 17 2010, 20:34:23) Type "copyright", "credits" or "license" for more information. IPython 0.11.alpha1.git -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. In [1]: import io In [2]: import sys In [3]: sys.ver sys.version sys.version_info In [3]: sys.version Out[3]: '3.1.2 (release31-maint, Sep 17 2010, 20:34:23) \n[GCC 4.4.5]' etc... This is still very much a work in progress, and we haven't made any official release. But there's a github repo up and running with a proper bug tracker, so by all means let us know what works and what doesnt' (and even better, join in with help!): http://github.com/ipython/ipython-py3k Our hope is to eventually merge this work into ipython proper at some point in the future, but since the history of this branch is likely to be messy (with many merges from trunk, auto-generated code and possible rebases down the road) we're keeping it in a separate repo for now. Cheers, f From Fernando.Perez at berkeley.edu Sun Oct 31 15:21:00 2010 From: Fernando.Perez at berkeley.edu (Fernando Perez) Date: Sun, 31 Oct 2010 12:21:00 -0700 Subject: [IPython-dev] Work on new Viz.Engine Fos In-Reply-To: <1288257563.1734.20.camel@dragonfly> References: <1288257563.1734.20.camel@dragonfly> Message-ID: Hi Guys, these questions are best asked on the ipython-dev list, so that others can both benefit from the discussion and contribute to it (often others will be more knowledgeable than myself on specific topics). I'm cc'ing the list here with the original question for full context. There are two (well, three if you count the old 0.10.x threads-based approach, but I'm ignoring that as we move forward and so should you, if you value your sanity) models to consider in the 0.11.x development tree: - single-process, in-terminal ipython ("classic" ipython, if you will): the gui handling is done by tying the gui event loop to calls into PyOSInputHook. Relevant files: http://github.com/ipython/ipython/blob/master/IPython/lib/inputhook.py # General support http://github.com/ipython/ipython/blob/master/IPython/lib/inputhookgtk.py # gtk specifics http://github.com/ipython/ipython/blob/master/IPython/lib/inputhookwx.py # wx specifics The details are toolkits-pecific, and you'll need to find what the right steps are for pyglet, but the basic idea is always the same: tie the toolkit's own event loop to sync with python's PyOSInputHoook, which fires every time the terminal is ready to read user input. - multi-process, zeromq-based ipython: in this case there's no PyOSInputHook to fire, because there's no terminal waiting to read. All inputs to the kernel arrive over the network from a different process. In this case, a different approach is needed, based on the same general idea: the toolkit's event loop is wired to operate on an idle timer of some kind, the specifics being again toolkit-specific. The details are in this file (see top-level docstring): http://github.com/ipython/ipython/blob/master/IPython/lib/guisupport.py and for GTK this file has the particular class: http://github.com/ipython/ipython/blob/master/IPython/zmq/gui/gtkembed.py In summary, you'll want to figure out how to sync the pyglet event loop in both scenarios: at the terminal with PyOS_InputHook and at the multiprocess ipython with some kind of idle timer. Let us know how it goes, and we'll be happy to include the generic support in ipython proper so that we support pyglet in addition to the major GUI toolkits. Cheers, f On Thu, Oct 28, 2010 at 02:19, Stephan Gerhard wrote: > Hi Fernando, > > Hope you are doing fine! > > Eleftherios and me (you might remember me from HBM, the ConnectomeViewer > guy ;) are currently working together on Fos, the new 3d scientific > visualization engine for neuroimaging based on pyglet. > > We came across an issue that might be related to new developments of > IPython and we wanted to make sure that we are going in the right > direction. It is also about event loops, where you are more expert than > both of us. > > We would like to be able to interact with the Fos viz window (derived > pyglet.window.Window) from IPython without Fos blocking IPython. We > looked into Sympy which seems to have solved this issue and we are > trying to understand what they did. > > How could we benefit from the 2 processes model that IPython will > implement? I read in an email that the event handling of IPython is > going to change. How would this impact the solution that Sympy came up > with for this problem? > > At some point, we might want to have a fos viz server running, and > communicate with it over sockets to send geometry and commands. But I am > not sure how this would fit in this picture. Probably zeromq might be > helpful here. Do you know any good tutorial? > > Is there anything you can think of we should take care of? > > Well, many questions, but I hope this is fine for you :) > > Cheers, > Stephan & Eleftherios From benjaminrk at gmail.com Fri Oct 22 16:04:24 2010 From: benjaminrk at gmail.com (MinRK) Date: Fri, 22 Oct 2010 20:04:24 -0000 Subject: [IPython-dev] ZMQ Parallel IPython Performance preview In-Reply-To: References:

Message-ID: Re-run for throughput with data: submit 16 tasks of a given size, plot against size. new-style: def echo(a): return a old-style: task = StringTask("b=a", push=dict(a=a), pull=['b']) The input chosen was random numpy arrays (64b float, so len(A)/8 ~= size in B). Notable points: * ZMQ submission remains flat, independent of size, due to non-copying sends * size doesn't come into account until ~100kB, and clearly dominates both after 1MB the turning point for Twisted is a little earlier than for ZMQ * at 4MB, Twisted is submitting < 2 tasks per sec, while ZMQ is submitting ~90 * roundtrip, ZMQ is fairly consistently ~40x faster. memory usage: * Peak memory for the engines is 20% higher with ZMQ, because more than one task can now be waiting in the queue on the engine at a time. * Peak memory for the Controller including schedulers is 25% less than Twisted with pure ZMQ, and 20% less with the Python scheduler. Note that all results still reside in memory, since I haven't implemented the db backend yet. * Peak memory for the Python scheduler is approximately the same as the engines * Peak memory for the zmq scheduler is about half that. -MinRK On Fri, Oct 22, 2010 at 09:52, MinRK wrote: > I'll get on the new tests, I already have a bandwidth one written, so I'm > running it now. As for Twisted's throughput performance, it's at least > partly our fault. Since the receiving is in Python, every time we try to > send there are incoming results getting in the way. If we wrote it such > that sending prevented the receipt of results, I'm sure the Twisted code > would be faster for large numbers of messages. With ZMQ, though, we don't > have to be receiving in Python to get the results to the client process, so > they arrive in ZMQ and await simple memcpy/deserialization. > > -MinRK > > > On Fri, Oct 22, 2010 at 09:27, Brian Granger wrote: > >> Min, >> >> Also, can you get memory consumption numbers for the controller and >> queues. I want to see how much worse Twisted is in that respect. >> >> Cheers, >> >> Brian >> >> On Thu, Oct 21, 2010 at 11:53 PM, MinRK wrote: >> >>> I have my first performance numbers for throughput with the new parallel >>> code riding on ZeroMQ, and results are fairly promising. Roundtrip time for >>> ~512 tiny tasks submitted as fast as they can is ~100x faster than with >>> Twisted. >>> >>> As a throughput test, I submitted a flood of many very small tasks that >>> should take ~no time: >>> new-style: >>> def wait(t=0): >>> import time >>> time.sleep(t) >>> submit: >>> client.apply(wait, args=(t,)) >>> >>> Twisted: >>> task = StringTask("import time; time.sleep(%f)"%t) >>> submit: >>> client.run(task) >>> >>> Flooding the queue with these tasks with t=0, and then waiting for the >>> results, I tracked two times: >>> Sent: the time from the first submit until the last submit returns >>> Roundtrip: the time from the first submit to getting the last result >>> >>> Plotting these times vs number of messages, we see some decent numbers: >>> * The pure ZMQ scheduler is fastest, 10-100 times faster than Twisted >>> roundtrip >>> * The Python scheduler is ~3x slower roundtrip than pure ZMQ, but no >>> penalty to the submission rate >>> * Twisted performance falls off very quickly as the number of tasks grows >>> * ZMQ performance is quite flat >>> >>> Legend: >>> zmq: the pure ZMQ Device is used for routing tasks >>> lru/weighted: the simplest/most complicated routing schemes respectively >>> in the Python ZMQ Scheduler (which supports dependencies) >>> twisted: the old IPython.kernel >>> >>> [image: roundtrip.png] >>> [image: sent.png] >>> Test system: >>> Core-i7 930, 4x2 cores (ht), 4-engine cluster all over tcp/loopback, >>> Ubuntu 10.04, Python 2.6.5 >>> >>> -MinRK >>> http://github.com/minrk >>> >> >> >> >> -- >> Brian E. Granger, Ph.D. >> Assistant Professor of Physics >> Cal Poly State University, San Luis Obispo >> bgranger at calpoly.edu >> ellisonbg at gmail.com >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: roundtrip.png Type: image/png Size: 30731 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sent.png Type: image/png Size: 31114 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: echo.png Type: image/png Size: 49607 bytes Desc: not available URL: From benjaminrk at gmail.com Fri Oct 22 16:11:29 2010 From: benjaminrk at gmail.com (MinRK) Date: Fri, 22 Oct 2010 20:11:29 -0000 Subject: [IPython-dev] ZMQ Parallel IPython Performance preview In-Reply-To: References: