Mailman 3 March 2007 - Python-Dev

Breaking calls to object.__init__/__new__
by Guido van Rossum 22 Mar '07

22 Mar '07

See python.org/sf/1683368. I'd like to invite opinions on whether it's worth breaking an unknown amount of user code in 2.6 for the sake of stricter argument checking for object.__init__ and object.__new__. I think it probably isn't; but the strict version could be added to 3.0 and a warning issued in 2.6 in -Wpy3k mode. Alternatively, we could introduce the stricter code in 2.6, fix the stdlib modules that it breaks, and hope for the best. Opinions? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

7 17

Adding timeout to socket.py and httplib.py - Updated
by Facundo Batista 22 Mar '07

22 Mar '07

I updated the patch #1676823, reflecting discussion here: - The function name changed, now it's _create_connection(). Its signature also changed: now, timeout is mandatorily named. - HTTPConnection has the posibility to call timeout with a number, and also with None. In both cases, it updates sock.timeout (changing effectively the default timeout). Docs and tests cases are also updated (note: now there's no doc in libsocket.tex, as this function is now private). Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

2 2

Patchs and bugs resume
by Facundo Batista 22 Mar '07

22 Mar '07

People: At the beginning of March, there was a thread in this list about patchs and bugs that teorically weren't checked out. >From that discussion, I asked myself: "How can I know the temporal location of a patch/bug?". Are there a lot of old patchs/bugs? Those that are old, don't have any update or there're a big discussion with each one? Are they abandoned? To help me with this analisys, I made a tool that taking information from SourceForge it creates a resume table, for the patchs... http://www.taniquetil.com.ar/facundo/py_patchs.html ...and the bugs: http://www.taniquetil.com.ar/facundo/py_bugs.html My idea is to update them periodically (something like each day, at the end of the html you have the update date and time). Enjoy it. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

4 7

Re: [Python-Dev] Breaking calls to object.__init__/__new__
by Phillip J. Eby 22 Mar '07

22 Mar '07

At 09:41 PM 3/21/2007 -0700, Guido van Rossum wrote: >On 3/21/07, Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > > Every time I've considered using super, I've eventually > > decided against it for reasons like this. I've always > > found a better way that doesn't introduce so may > > strange interactions between the classes involved. > >I end up feeling the same way, for purely pragmatic reasons. However, >cooperative multiple inheritance is a respected religion, and when >practiced systematically can work well. > >Also make a big distinction between super calls of __init__ (which are >a Pythonic wart and don't exist in other languages practicing multiple >inheritance AFAIK) and super calls of regular methods (which are >virtually problem-free as long as you agree on a base class that >defines a method and derive from that class when you want to extend >it). FYI, the only time I ever use super() for __init__ calls is for metaclasses. However, there the signature is fixed, so this problem doesn't really arise. However, in Py3K the metaclass __init__ will accept keyword arguments. What's the right thing to do there? Should we pass them all through? I'm not sure if there's any inherent symmetry between this and object.__init__ -- but it seems to be an interesting question in its own right. The whole point of being co-operative in a metaclass is to allow other metaclasses to be safely mixed in -- and they may be metaclasses from a completely different library or framework. So, it's not as simple as saying "use a specialized base class" -- the class author may not even *know* that metaclasses are involved, let alone multiple ones.

2 2

Re: [Python-Dev] Breaking calls to object.__init__/__new__
by Terry Jones 22 Mar '07

22 Mar '07

Following up to myself, with some examples. I probably haven't done this as cleanly as is possible, but below are a bunch of classes and subclasses that cleanly deal with passing around arguments, even when those args conflict in name, etc., as outlined in my previous mail. Here's the general class __init__ pattern: class myclass(A, B, C, ...): def __init__(self, **kwargs): initargs = kwargs.setdefault('__initargs__', {}) # 1 Aargs = initargs.setdefault(A, {}) # 2 Aargs['limit'] = 10 # 3 Bargs = initargs.setdefault(B, {}) # 4 Bargs['limit'] = 'wednesday' # 5 super(myclass, self).__init__(**kwargs) # 6 myargs = initargs.get(myclass, {}) # 7 limit = myargs.get('limit', 7) # 8 if 'huh?' in myargs: raise Exception # 9 In words, the steps are: 1. Pull the __initargs__ key out of **kwargs, or create it. This gives you the top-level dictionary containing your own arguments, if any, and the args for your superclass(es), if any. 2. Get the arguments for superclass A, and 3. Add an argument for A.__init__ 4 & 5. Do the same for superclass B. We don't alter args for Superclass C. 6. Call super, passing **kwargs along (this contains the original __initargs__ that was sent to us, or the one we made in step 1 if there was no prior __initargs__. 7. Our arguments, if any, are in initargs too. Get them. 8. Pull out one of our args, or set a default. 9. Check we didn't get any unexpected args, etc. Some comments: - This isn't too ugly, I don't think, but it does require mentioning your class and superclasses by name. There are many advantages (see last mail) if you're willing to do this. - You can combine positional args and explicit keyword args in a class if you choose, and you can then disambiguate or complain if you like (class sub4 in the example code below does this). You can also take those explicit args and stuff them into the __initargs__ for your superclasses, as/if needed. Explicit args might be a slight pain to deal with, but you need them as you don't want to destroy the simple class instantiation mechanism of python that passes args normally to __init__. - It does require that classes cooperate to a small extent. E.g., you don't probably don't want the __init__ class in one of your super classes to go fiddling with initargs[myclass] before myclass even gets to see its own args (though this particular nit could be solved by having myclass copy / remove its args before calling super). - A class could also optionally delete it args once done. In that case it's cleaner to have line 7 use initargs.setdefault(myclass, {}) so a del initargs[myclass] always just works. - If you don't want to mess with the args to your superclasses, don't. In that case lines 2-5 go away in the above (and 1 and 6 could be swapped). - You can add *args to all your __init__ methods. This allows your class to sit in the middle of a class chain where a lower method wants to pass a positional argument on to a higher class (that wasn't written according to the above pattern). So the args just gets passed along. It's not as nice, but it doesn't break anything (by silently not passing on any positional args that may be present). Classes written using the above pattern can just ignore all positional args. - If I were encouraging doing something like the above in python proper, I think I'd allow line 1 to read initargs = kwargs.setdefault(None, {}) which adopts the convention that the __init__ keywords are passed in the None slot of kwargs. Currently you can't do this, as python complains that all keyword args must be strings. While this goes against explicit is better that implicit, using __initargs__ as I have done is polluting the keyword space and would probably one day cause someone a problem. There's some example code below. Terry class err(Exception): pass class sum(object): def __init__(self, **kwargs): print "-> in sum" initargs = kwargs.setdefault('__initargs__', {}) super(sum, self).__init__(**kwargs) myargs = initargs.get(sum, {}) limit = myargs.get('limit', 5) print "my limit is %d" % limit class daylimitmaker(object): def __init__(self, **kwargs): print "-> in daylimitmaker" initargs = kwargs.setdefault('__initargs__', {}) super(daylimitmaker, self).__init__(**kwargs) myargs = initargs.get(daylimitmaker, {}) limit = myargs.get('limit', 'sunday (last day of the week)') print "my limit is '%s'" % limit class lastday(object): def __init__(self, **kwargs): print "-> in lastday" initargs = kwargs.setdefault('__initargs__', {}) super(lastday, self).__init__(**kwargs) myargs = initargs.get(lastday, {}) limit = myargs.get('limit', 'yesterday') print "my limit is '%s'" % limit class onelimitarg(object): def __init__(self, **kwargs): print "-> in onelimitarg" initargs = kwargs.setdefault('__initargs__', {}) super(onelimitarg, self).__init__(**kwargs) myargs = initargs.get(onelimitarg, {}) if list(myargs.keys()) != ('limit'): raise err, "hey, someone passed me unknown arguments: %s" % myargs limit = myargs.get('limit', 1) print "my limit is %d" % limit class sub0(sum): '''Subclass sum, and pass it an explicit limit.''' def __init__(self, **kwargs): print "-> in sub0" initargs = kwargs.setdefault('__initargs__', {}) sumargs = initargs.setdefault(sum, {}) sumargs['limit'] = 99 super(sub0, self).__init__(**kwargs) class sub1(sum, lastday): '''Subclass sum and lastday, passing both an explicit limit.''' def __init__(self, **kwargs): print "-> in sub1" initargs = kwargs.setdefault('__initargs__', {}) sumargs = initargs.setdefault(sum, {}) sumargs['limit'] = 4 lastdayargs = initargs.setdefault(lastday, {}) lastdayargs['limit'] = 'friday' super(sub1, self).__init__(**kwargs) myargs = initargs.get(sub1, {}) limit = myargs.get('limit', 23) # etc... class sub2(lastday, sum): '''Subclass lastday and sum (opposite order from sub1), passing both an explicit limit.''' def __init__(self, **kwargs): print "-> in sub2" initargs = kwargs.setdefault('__initargs__', {}) sumargs = initargs.setdefault(sum, {}) sumargs['limit'] = 10 lastdayargs = initargs.setdefault(lastday, {}) lastdayargs['limit'] = 'wednesday' super(sub2, self).__init__(**kwargs) myargs = initargs.get(sub2, {}) # etc... class sub3(lastday, sum): '''Subclass lastday and sum, letting both use their default limit.''' def __init__(self, **kwargs): print "-> in sub3" initargs = kwargs.setdefault('__initargs__', {}) super(sub3, self).__init__(**kwargs) myargs = initargs.get(sub3, {}) # etc... class sub4(lastday, sum): '''Subclass lastday and sum, let our __init__ take a limit arg and check for ambiguity.''' def __init__(self, limit=10, **kwargs): '''This is deliberately ugly, but it can be done.''' print "-> in sub4" initargs = kwargs.setdefault('__initargs__', {}) sumargs = initargs.setdefault(sum, {}) sumargs['limit'] = limit super(sub4, self).__init__(**kwargs) myargs = initargs.get(sub4, {}) if 'limit' in myargs: raise err, "hey, someone passed me an __initargs__ limit argument, but i like to get an explicit limit arg.\n" class sub5(sub3, sub4): def __init__(self, **kwargs): print "-> in sub5" initargs = kwargs.setdefault('__initargs__', {}) super(sub5, self).__init__(**kwargs) myargs = initargs.get(sub5, {}) # etc... class sub6(sum, daylimitmaker): def __init__(self, **kwargs): print "-> in sub6" initargs = kwargs.setdefault('__initargs__', {}) super(sub6, self).__init__(**kwargs) myargs = initargs.get(sub6, {}) # etc... class sub7(onelimitarg): def __init__(self, **kwargs): print "-> in sub7" initargs = kwargs.setdefault('__initargs__', {}) onelimitargs = initargs.setdefault(onelimitarg, {}) onelimitargs['limit'] = 10 onelimitargs['dummy'] = 'dummy' # Pass an unknown arg. super(sub7, self).__init__(**kwargs) if __name__ == '__main__': for klass in (sub0, sub1, sub2, sub3, sub4, sub5, sub6, sub7): print "Instantiating %s: mro = %s" % (klass.__name__, [k.__name__ for k in klass.__mro__]) try: klass() except err, e: print e print print "Instantiating sub4(limit=22)" sub4(limit=22) # Call with explicit arg, the way sub4 likes it. print try: print "Instantiating sub4 with __initargs__ limit." # (The way sub4 doesn't like it.) sub4(__initargs__={sub4 : {'limit' : None}}) except err, e: print e

1 0

Breaking calls to object.__init__/__new__
by Blake Ross 22 Mar '07

22 Mar '07

At 09:41 PM 3/21/2007 -0700, Guido van Rossum wrote: > Also make a big distinction between super calls of __init__ (which are > a Pythonic wart and don't exist in other languages practicing multiple > inheritance AFAIK) Since I filed the bug, I should clarify that the primary reason I'm using super-init is to avoid multiple construction of a shared base (even if the base is just object, since I'd prefer not to rely on object's initializer being a no-op). C++ ensures that virtual bases are only constructed once, so there's no concern as to multiple construction. Is there a Python pattern better than super-init that provides the same guarantee? (Apologies if this appears in the wrong place; I just joined the list and I'm not seeing a way to participate in an existing thread.)

4 3

Re: [Python-Dev] Breaking calls to object.__init__/__new__
by Terry Jones 22 Mar '07

22 Mar '07

>>>>> "G" == Guido van Rossum <guido(a)python.org> writes: G> There are different philosophies about the correct style for G> cooperative super calls. G> The submitter of the bug report likes to remove "consumed" arguments G> and pass the others on, having something at the root that complains G> about any unused arguments. It has the problem that you mention: if G> multiple classes might be interested in the *same* argument they G> won't see it. The other style is to pass *all* arguments down, and G> let everyone cherry-pick them. The last call then just throws them G> away. This has the problem that misspelled arguments are silently G> ignored rather than being diagnosed at the point where you can do G> something about it. G> I don't know what the "best practice" is (like Greg Ewing, I don't G> use either style myself) but I've got a feeling that it must be G> easier to solve the former problem than the latter (also I don't G> know that the former actually occurs in practice). I'm following up to the above, as I'm not sure where the best place to jump into this discussion is. I think the problem is more general that just consuming and/or passing on. In the typical style of passing args to super classes that I've seen, __init__ methods in the call chain also don't really even know *whether* they should use an argument. There is also the problem of argument change over time. E.g., suppose I'm the writer of a class X and, unbeknownst to me, someone writes class Y(X,Z), where Z is some other class. Suppose Z's __init__ expects an argument called z. If I one day add an argument called z, that has totally different semantics, things get really hairy. I'm the writer of X, I'm unaware of classes Y and Z, and if someone passes me an z argument then naturally I want to (try to) use it. There are other examples like this, and argument name clashes can be more specific (like 'limit' or 'debug'). In general, we're faced with a situation that is also both time-sensitive and semantics-sensitive. Class writers shouldn't need to know all future argument changes to other classes potentially upstream or downstream from them in the call chain. Given an unstructured mess of arguments, you don't know what to consume, what to ignore, what to pass on, etc., and you don't know the future. Although this sounds much more difficult than simple diamond inheritance, I think there's an easy way to solve it. When writing a class, you know 1) what arguments your __init__ method wants, and 2) what arguments your superclass(es) __init__ methods want. You know the second thing from reading the API/docs of your superclasses - that's why you're calling them. You want to be able, for example, to route a 'limit' argument to one of your super classes and another different 'limit' argument to another of your super classes. You want this to work even if one or more of your superclasses (now or in the future) in turn decides to call some other super class that also has a 'limit' argument with entirely different (or the same) semantics. Etc. An easy solution is for __init__ to receive and pass a dictionary whose keys are class names and whose values are dictionaries of arguments intended for that class. That way, an __init__ method in a class knows exactly which arguments are intended for it. It can detect extra args, missing args, misspelt args, etc. It can also prepare args for its known immediate superclasses (those from which it subclasses) - and it does know these classes as it is explicitly subclassing from them. It can leave arguments that are intended for the __init__ methods in other classes alone. It can create different arguments of the same name for its different superclasses. It can change its signature (adding and dropping args) without affecting the args passed to its superclass. Classes earlier in the call chain can do the same without affecting the args received by this class. It is also immune to differences in superclass name ordering by subclasses (i.e., a subclass of X and Y should be able to be class Z(X,Y) or class(Y,Z) without screwing up the args received by either X.__init__ or Y.__init__. An __init__ could also safely del its own arguments once it has extracted/used them. In some odd cases it might like to pass them along, perhaps even adding something (like a None key to its sub-dictionary) to indicate that it has already been called (yes, this sounds like weird - I'm just mentioning that it would be possible). There's also no need for any special top-level subclasses of object that simply ignore all their arguments (though you _could_ add one if you wanted to be strict and insist that all __init__ methods del their args). This approach adheres nicely to the explicit is better than implicit maxim. I think the implicit situation is unsolvable. I guess this could all be dressed up nicely using a metaclass that pulled out any available args, made them available to the __init__ as regular keyword args, and made available the special dictionary that holds arguments for later classes (any given __init__ may want to add/alter this). I don't know enough about metaclasses to know if this could be done completely cleanly though. I'd probably prefer it all to be explicit and manual. Sorry for such a long post without much code. I can write some example classes if the above is unclear or people think this worth looking at further. Terry

1 0

Calling base class methods from C
by Raymond Hettinger 22 Mar '07

22 Mar '07

The xxsubtype.c module gives an example of calling a parent method if it is in a slot: static int spamlist_init(spamlistobject *self, PyObject *args, PyObject *kwds) { if (PyList_Type.tp_init((PyObject *)self, args, kwds) < 0) return -1; self->state = 0; return 0; } How you call non-slotted named methods in parent classes? class List(list): def append(self, x): print x List.append(self, x) # What is the C equivalent of this call? Raymond

3 2

Re: [Python-Dev] I vote to reject: Adding timeout to socket.py and httplib.py.
by Alan Kennedy 22 Mar '07

22 Mar '07

[Facundo Batista] > Remember that this function primary use is for > higher level libraries Yes, I see that clearly now. But remember that by adding a new function to the socket module to support httplib et al, you are also adding a function to the socket module that will be used directly by end users. I vote to reject this patch. The underlying problem it is trying to address is that httplib.HTTPConnection objects do not support timeouts. But solving that problem by moving the functionality of the HTTPConnection.connect() method inside the socket module as a standalone function is the *wrong* solution. The proposed new function does not belong in the socket module. In contrast to all of the other socket creation and management functionality in the socket module, the new function does not handle non-blocking IO. Also, the new functions' use-case is too restricted, to client connections with a positive timeout: this functionality is trivially available using existing functionality, e.g. mysock = socket(AF_INET, SOCK_STREAM) mysocket.settimeout(1.0) mysocket.connect( (host, port) ) # etc In contrast to the new function, the existing functionality in the socket module is much more general, and much better designed to handle the wide range of situations in which sockets are used. Socket APIs are hard to do right because sockets are complex and hard to do right. The problem the patch seeks to fix is is in httplib; the problem should be fixed in httplib. I recommend modifying the patch to remove *all* proposed changes to the socket module. Instead, the patch should restrict itself to fixing the httplib module. Sorry, Alan.

12 24

Rationale for NamedTemporaryFile?
by Greg Ewing 21 Mar '07

21 Mar '07

I've just discovered the hard way that NamedTemporaryFile automatically deletes the file when you close it. That doesn't seem very useful to me, since surely the reason you're using NamedTemporaryFile instead of TemporaryFile is that you want to do something else with it afterwards? What's the rationale for this behaviour? -- Greg

9 16