Mailman 3 February 2007 - Python-Dev

Re: [Python-Dev] Trial balloon: microthreads library in stdlib
by glyph＠divmod.com Feb. 15, 2007

Feb. 15, 2007

On 04:49 pm, richard.m.tew(a)gmail.com wrote: >On 2/13/07, glyph(a)divmod.com <glyph(a)divmod.com> wrote: >>Tristan is correct: this should be a patch against Twisted, or perhaps as a >>separate library that could implement a reactor. > >I think there is some confusion here. Quite. >I am not writing a competing event driven mechanism. What I was doing >was feeling out whether there was any interest in better support for >asynchronous calls. Perhaps you could describe what you were proposing a bit better, because both to Tristan and myself, it sounded (and frankly, still sounds) like you were describing that would directly replace Twisted's mainloop core API. "asynchronous calls" is a very vague term with at least three completely separate definitions. >Yes I have looked at Twisted. It was the first place I looked, several >months ago, and what drew me to it was the IOCP reactor. However >as I also explained in my reply to Tristan it is not suitable for my >needs. As far as I can tell, you still haven't even clearly expressed what your needs are, let alone whether or not Twisted is suitable. In the reply you're citing, you said that "this" sounded like something "low level" that "twisted would be written on top of" - but the "this" you were talking about, based on your previous messages, sounded like monkeypatching the socket and asyncore modules to provide asynchronous file I/O based on the platform-specific IOCP API for Windows. Twisted can already provide that functionality in a _much_ cleaner fashion already, although you might need to edit some code or monkeypatch some objects specifically to get asynchronous file I/O, at least Twisted is *designed* for such applications. To quote you from the original message that Tristan replied to: >> I can't go into low level details because I do not have the necessary >> experience or knowledge to know which approach would be best. ... >> I of course have no concept of how this might be done on other platforms. As someone who _does_ have the necessary experience and knowledge, I can tell you that Twisted *IS* the "one stop shop" for events that you're describing. I do know how one would do such a thing on other platforms and it is not a simple task - so much so that async file I/O is still not available in Twisted today. >It is a large dependency and it is a secondary framework. Has it occurred to you that it is a "large" dependency not because we like making bloated and redundant code, but because it is doing something that is actually complex and involved? >And I did not feel the need to verify the implication that it wasn't >ready because this matched my own recollections. It meaning... Twisted? Twisted's IOCP support? Ready for... what? IOCPReactor definitely not up to the standard of much of the rest of the code in Twisted, but it's clearly ready for _something_ since BitTorrent uses it. >But I hope you realise that asserting that things should be within >Twisted without giving any reason, I have suggested that work proceed in Twisted rather than in Python because adding async file I/O to Twisted would be much easier than adding an entirely new event-loop core to Python, and then adding async file I/O to *that*. I thought that I provided several reasons before as well, but let me state them as clearly as I can here. Twisted is a large and mature framework with several years of development and an active user community. The pluggable event loop it exposes is solving a hard problem, and its implementation encodes a lot of knowledge about how such a thing might be done. It's also tested on a lot of different platforms. Writing all this from scratch - even a small subset of it - is a lot of work, and is unlikely to result in something robust enough for use by all Python programs without quite a lot of effort. It would be a lot easier to get the Twisted team involved in standardizing event-driven communications in Python. Every other effort I'm aware of is either much smaller, entirely proprietary, or both. Again, I would love to be corrected here, and shown some other large event-driven framework in Python that I was heretofore unaware of. Standardization is much easier to achieve when you have multiple interested parties with different interests to accommodate. As Yitzhak Rabin used to say, "you don't engage in API standardization with your friends, you engage in API standardization with your enemies" - or... something like that. >especially when the first person >saying it just stated that the matching work in Twisted wasn't even >ready, feels like Twisted is trying to push itself forward by bullying >people to work within it whenever something can be asserted as >laying within whatever domain it is which Twisted covers. You say that you weren't proposing an alternate implementation of an event loop core, so I may be reacting to something you didn't say at all. However, again, at least Tristan thought the same thing, so I'm not the only one. I think that *that* could be described as bullying. I -- and many others -- have a considerable investment in Twisted, and attempts to "standardize" on another, new main-loop within the standard library is an attempt to subvert that investment and pre-empt honest competition between different event-driven frameworks, by de-facto forcing all Python users (and Twisted itself) to learn and deal with what the standard library is doing. Ignoring the Twisted API while doing this adds insult to injury. There are some clear and very deep misunderstandings in this discussion about what the Twisted reactor is, how it works, and what it does, so I do really feel like it is being ignored. Not all of this is a direct reaction to things you've said, but to the continuing pattern of posts to this list which encourage development of new event-driven mechanisms or changes to asyncore to add features, while still ignoring Twisted and all the lessons learned from it. If we're talking about inclusion in the standard library, maybe Twisted's event loop *isn't* ready. Certainly all the _rest_ of Twisted isn't. I have admitted that already in this thread. That doesn't mean another from-scratch toy event loop, even one that Twisted could be implemented "on top of" _is_ ready. Stackless - I'd say that's certainly ready, although I understand the other concerns voiced in this thread. It's got a proven track-record and it, also, solves a difficult problem. Re-implementing microthreads for python from scratch, while ignoring stackless, specifically for inclusion in the standard library would also, presumably, be a bad idea. >Even if it doesn't suit their needs. If someone doesn't "have the necessary experience or knowledge to know which approach would be best", then how, exactly, do they know whether things suit their needs or not? You specifically claimed ignorance, and then said you were being "bullied" when someone tried to give you the information that you are lacking. >I don't think Tristan intended >that, but when followed up with your reply and JP's interesting >response to Greg, it feels like it. I thought JP's response to Greg was useful, since much of this discussion has been hopelessly vague. For example, your earlier claim that "it wasn't ready". I've tried to restrict my comments here to the possibility of a new event-driven core in the Python standard library. There are a lot of other features also being discussed here - a microthread scheduler API, for example - which I am _not_ advocating the use of Twisted for. Twisted could definitely make use of such a scheduler API if it existed in the stdlib.

3 2

Interning string subtype instances
by Hrvoje Nikšić Feb. 15, 2007

Feb. 15, 2007

I propose modifying PyString_InternInPlace to better cope with string subtype instances. Current implementation of PyString_InternInPlace does nothing and returns if passed an instance of a subtype of PyString_Type. This is a problem for applications that need to support string subtypes, but also must intern the strings for faster equivalence testing. Such an application, upon receiving a string subtype, will silently fail to work. There is good reason for PyString_InternInPlace not accepting string subtypes: since a subtype can have modified behavior, interning it can cause problems for other users of the interned string. I agree with the reasoning, but propose a different solution: when interning an instance of a string subtype, PyString_InternInPlace could simply intern a copy. This should be a fully backward compatible change because: 1) code that passes PyString instances (99.99% cases) will work as before, and 2) code that passes something else silently failed to intern the string anyway. Speed should be exactly the same as before, with the added benefit that interning PyString subtype instances now does something, but without the problems that interning the actual instance can produce. The patch could look like this. If there is interest in this, I can produce a complete patch. @@ -5,10 +5,6 @@ PyObject *t; if (s == NULL || !PyString_Check(s)) Py_FatalError("PyString_InternInPlace: strings only please!"); - /* If it's a string subclass, we don't really know what putting - it in the interned dict might do. */ - if (!PyString_CheckExact(s)) - return; if (PyString_CHECK_INTERNED(s)) return; if (interned == NULL) { @@ -25,6 +21,18 @@ *p = t; return; } + /* Make sure we don't intern a string subclass, since we don't + really know what putting it in the interned dict might do. */ + if (!PyString_CheckExact(s)) { + PyObject *copy; + copy = PyString_FromStringAndSize(PyString_AS_STRING(*p), + PyString_GET_SIZE(*p)); + if (!copy) + return; + Py_DECREF(*p); + *p = copy; + s = (PyStringObject *) copy; + } if (PyDict_SetItem(interned, (PyObject *)s, (PyObject *)s) < 0) { PyErr_Clear();

6 18

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
by glyph＠divmod.com Feb. 15, 2007

Feb. 15, 2007

On 12:31 am, greg.ewing(a)canterbury.ac.nz wrote: >glyph(a)divmod.com wrote: >> On 08:52 pm, theller(a)ctypes.org wrote: >> >> >When I last looked at twisted (that is several years ago), there were >> >several reactors - win32reactor, wxreactor, maybe even more. >> >> Only the very top-most level decides which reactor the application will use. > >This is a worry, because it implies that there has to >*be* a top level that knows what kind of reactor the >whole application will use, and all parts of the >application need to know that they will be getting >their reactor from that top level. The default reactor is the most portable one, 'select', and if no other reactor is installed, that's the one that will be used. >That may not be the case. For example, you want to >incorporate some piece of event-driven code written >by someone else into your gtk application. But it >wasn't written with gtk in mind, so it doesn't know >to use a gtkreactor, or how to get a reactor that it >can use from your application. "from twisted.internet import reactor" is the way you get at the reactor, regardless of which one is currently installed. There can only be one reactor active at any given time, because at the very bottom of the event-handling machinery _some_ platform multiplexing API must be called, and that is mostly what the reactor represents. The GTK reactor does not have its own API. It simply allows you to use GTK APIs as well, by back-ending to the glib mainloop. That is, in fact, the whole point of having a "reactor" API in the first place. >This is not my idea of what another poster called a >"one-stop shop" -- a common API that different pieces >of code can call independently without having to know >about each other. >To my mind, there shouldn't be a "reactor" object >exposed to the application at all. There should just >be functions for setting up callbacks. That's what the Twisted "reactor" object *is*, exactly. Functions (well, methods) for setting up callbacks. >The choice of >implementation should be made somewhere deep inside >the library, based on what platform is being used. The "deep inside the library" decision is actually a policy decision made by a server's administrator, or dependent upon the GUI library being used if you need to interact with a GUI event loop. Perhaps the default selection could be better, but installing a reactor is literally one line of code, or a single command-line option to the "twistd" daemon. See: http://twistedmatrix.com/projects/core/documentation/howto/choosing-reactor… It is completely transparent to the application, _unless_ the application wants to make use of platform-specific features. See the following for more information: http://www.cs.wustl.edu/~schmidt/PDF/reactor-siemens.pdf although technically Twisted's "reactor" is more like the slightly higher level POSA "proactor" pattern; asyncore is more like a true "reactor" in the sense discussed in that paper. Twisted exposes various APIs for "setting up callbacks" exactly as you describe: http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.… http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.… >So at this point I'm skeptical that the Twisted >API for these things should be adopted as-is Given that your supporting arguments were almost exactly the opposite of the truth in every case, I think this conclusion should be re-examined :). If you're interested in how a normal Twisted application looks, see this tutorial: http://twistedmatrix.com/projects/core/documentation/howto/servers.html

1 0

Re: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax
by glyph＠divmod.com Feb. 15, 2007

Feb. 15, 2007

On 01:04 pm, ben(a)redfrontdoor.org wrote: >glyph(a)divmod.com: >> I really, really wish that every feature proposal for Python had to meet >> some burden of proof [...]. I suspect this would kill 90% of "hey >> wouldn't this syntax be neat" proposals on day zero [...] > >This is what I understood the initial posting to python-ideas to be >about. If the feedback from there had been "that's a stupid idea", then >that would have been the end of it. I think it's a good thing that >there's the python-ideas mechanism for speculative suggestions. Discussion, with any audience, no matter how discerning, isn't really going to be a useful filter unless the standards of that audience are clear. What I was trying to say there is that the proposal of new ideas should not begin with "Hey, I think this might be 'good'" - that's too ill defined. It should be, "I noticed (myself/my users/my students/other open source projects) writing lots of code that looks like *this*, and I think it would be a real savings if it could look like *that*, instead". Some back-of-the-envelope math might be nice, in terms of savings of lines of code, or an explanation of the new functionality that might be enabled by a feature. More than the way proposals should work, I'm suggesting that the standards of the community in _evaluating_ to the proposals should be clearer. The cost-benefit analysis should be done in terms of programmer time, or ease of learning, or integration possibilities, or something. It doesn't *necessarily* have to be in terms of "lines of code saved", but especially for syntax proposals, that is probably a necessary start. It's fine to have some aesthetic considerations, but there's a lot more to Python than aesthetics, and right now it seems like the majority of posts coming into threads like this one are "I don't like the proposed ':==|', it's ugly. How about '?!?+' instead?" The short version of this is that any feature should describe what task it makes easier, for whom, and by how much. This description should be done up front, so that the discussion can center around whether than analysis is correct and whether it is worth the ever-present tradeoff against upgrade headaches.

2 1

Why is nb_inplace_power ternary?
by "Martin v. Löwis" Feb. 15, 2007

Feb. 15, 2007

#1653736 reports that slot_nb_inplace_power has the wrong type: it should be a ternary function, but only is a binary. The proposed change is to make it ternary, and to invoke __ipow__ with three arguments. In researching this, I came to wonder why nb_inplace_power is ternary in the first place. It is the implementation of foo **= bar (and that's its only use), so it ought to be binary. Historically, the reason probably is that it was copied from nb_power, which is ternary because it also implements pow()). So here is my proposed change: 1. For 2.5.1, rewrite slot_nb_inplace_power to raise an exception if the third argument is not None, and then invoke __ipow__ with only one argument. 2. For 2.6, change the API to make nb_inplace_power binary. Optionally, in 2.5, the exception could be added into other places as well, e.g. PyNumber_InPlacePower and instance_ipow (rather than invoking __ipow__ with a third argument if its not None). Comments? Regards, Martin

5 12

Re: [Python-Dev] Trial balloon: microthreads library in stdlib
by glyph＠divmod.com Feb. 15, 2007

Feb. 15, 2007

On 12 Feb, 05:11 pm, richard.m.tew(a)gmail.com wrote: >On 2/12/07, Tristan Seligmann <mithrandi-python-dev(a)mithrandi.za.net> wrote: >> * Richard Tew <richard.m.tew(a)gmail.com> [2007-02-12 13:46:43 +0000]: >> > Perhaps there is a better way. And I of course have no concept of >> > how this might be done on other platforms. >> >> Building on an existing framework that does this seems better than >> reinventing the wheel, for something of this magnitude. > >This to me seems to be the low level support you would build >something like Twisted on top of. Pushing Twisted so that others >can get it seems a little over the top. It sounds like you don't really understand what Twisted is, what it does, or the difficulty involved in building that "low level" so that it's usable by something like Twisted. Tristan is correct: this should be a patch against Twisted, or perhaps as a separate library that could implement a reactor. I have no problem with other, competing event-driven mechanisms being developed; in fact, I think it's great that the style of programming is getting some attention. But this is not a robust and straightforward wrapping of some lower level. This is an entirely new, experimental project, and the place to start developing it is _NOT_ in the Python core. Another post in this thread outlined that the first thing you should do is develop something and get people in the community to use it. Please do that, start its own mailing list, and stop discussing it here. On a personal note, I believe that "those who do not understand twisted are doomed to repeat it". I predict that the ultimate outcome of this project will be that all concerned will realize that, actually, Twisted already does almost everything that it proposes, and the microthreading features being discussed here are a trivial hack somewhere in its mainloop machinery, not an entirely new subsystem that it should be implemented in terms of.

6 6

Re: [Python-Dev] [Python-3000] pre-PEP: Default Argument Expressions
by Greg Ewing Feb. 14, 2007

Feb. 14, 2007

Another thing I don't like about this default argument proposal is that using it in any non-trivial way would tend to put implementation details of the function up in the header, which should be reserved for info describing the calling signature. The following example from the PEP is an extreme instance of this: > def bar(b=my_sum((["b"] * (2 * 3))[:4])): > print b -- Greg

1 0

Any value in setting up pybots for py3k?
by Grig Gheorghiu Feb. 14, 2007

Feb. 14, 2007

Sidnei da Silva asked on the pybots mailing list if we should be testing packages with py3k. I think it's probably too early for that, but on the other hand I'm sure many package creators/maintainers would be curious to see how their packages fare with py3k. So is there any value or interest in setting up a svn notification hook for py3k commits that would go to the pybots buildbot master? Grig -- http://agiletesting.blogspot.com

3 2

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
by glyph＠divmod.com Feb. 14, 2007

Feb. 14, 2007

On 02:22 pm, aahz(a)pythoncraft.com wrote: >On Wed, Feb 14, 2007, glyph(a)divmod.com wrote: >> >> As I said, I don't have time to write the PEPs myself, but I might fix >> some specific bugs if there were a clear set of issues preventing this >> from moving forward. Better integration with the standard library >> would definitely be a big win for both Twisted and Python. > >Here's where I'm coming from: > >My first experience with Twisted was excellent: I needed a web server in >fifteen minutes to do my PyCon presentation, and it Just Worked. > >My second experience with Twisted? Well, I didn't really have one. Thanks for the feedback, Aahz... I'm a little confused, though. Is this just a personal anecdote, or were you trying to make a larger point about Twisted's suitability for the stdlib?

1 0

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
by glyph＠divmod.com Feb. 14, 2007

Feb. 14, 2007

On 03:32 pm, joachim.koenig-baltes(a)emesgarten.de wrote: >glyph(a)divmod.com wrote: >>When you boil it down, Twisted's event loop ... >But that is exactly the problem I have with Twisted. For HTTP ... ^^^^ >From that word on out, you have completely lost the plot. I am talking about Twisted's _event loop_. I specifically mentioned that the higher levels (and specifically "web") are separate, and can be discussed separately. If there were interest in porting the simpler XXXHTTPServer APIs in the stdlib to be event-driven and work with a Twisted-style event loop, it would be a relatively simple hack.

1 0