From tjreedy at udel.edu Sun Jan 1 09:24:59 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 01 Jan 2012 03:24:59 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On 12/31/2011 5:57 PM, Nathan Schneider wrote: > The necessary resolution of key conflicts is what makes "+" feel less > natural to me for dicts than it does for sequences, where the order of > the operands is transparently reflected in the output. For dicts, in > contrast, we are talking about a "lossy" union/combination where what > is lost depends on the order of the operands?this is less transparent > in the output. However it is spelled, though, I would be thrilled if > such an operation were built in. :) You mean like dict.update? -- Terry Jan Reedy From jeanpierreda at gmail.com Sun Jan 1 09:40:01 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 1 Jan 2012 03:40:01 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On Sun, Jan 1, 2012 at 3:24 AM, Terry Reedy wrote: > You mean like dict.update? Nah nah, that's "+= "; "+" is dict(d1, **d2). -- Devin On Sun, Jan 1, 2012 at 3:24 AM, Terry Reedy wrote: > On 12/31/2011 5:57 PM, Nathan Schneider wrote: > >> The necessary resolution of key conflicts is what makes "+" feel less >> natural to me for dicts than it does for sequences, where the order of >> the operands is transparently reflected in the output. For dicts, in >> contrast, we are talking about a "lossy" union/combination where what >> is lost depends on the order of the operands?this is less transparent >> in the output. However it is spelled, though, I would be thrilled if >> such an operation were built in. :) > > > You mean like dict.update? > > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From julien at tayon.net Mon Jan 2 11:51:01 2012 From: julien at tayon.net (julien tayon) Date: Mon, 2 Jan 2012 11:51:01 +0100 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: 2011/12/31 Nathan Schneider : > On Sat, Dec 31, 2011 at 4:55 PM, Terry Reedy wrote: >> On 12/31/2011 11:07 AM, julien tayon wrote: >> >>> The question is a dict the same as a vector or as a set ? >> >> >> A dict is a mapping, which is to say, a functional set of key,value pairs. >> 'Functional' means that the keys are unique. This is a key property (pun >> intended). A vector can be regarded as a functional set of count,value >> pairs. Some languages use dicts for vectors. Even in Python, people use >> dicts for sparse arrays of various dimensions, with unspecified pairs >> assumed to have a 0 value. Yes, a potentialy indefinite sparse value vector on a base at least greater. For which the dimension is the path of keys to a value. And I lamely admitted these were orthogonal in my mind to one another. (therefore dicthing coupling problematics in the first time since I will think of matrix later) That is the reason, I intend to back down on any argumentation on vectoriel meaning of dict without proper code to back up my claims :) . > > It occurs to me that any argument for a vector interpretation of > dicts, with element-wise operations, should apply equally well to > lists and tuples, which are perhaps even more natural as > representations for vectors. But of course, the + operator for > sequences is interpreted as concatenation, not element-wise addition; > it would be strange if the reverse were true for dicts. I do admit I came to the fact you are right. And, I think that if two keys were equals but not the value it should raise a collision exception. I could provide a consistent "set addition" for dict as an amend for making so much noise :) I would also provide sub. I see a way to provide a dynamic dict to sets translator that would be quite weiredl. > > The necessary resolution of key conflicts is what makes "+" feel less > natural to me for dicts than it does for sequences, where the order of > the operands is transparently reflected in the output. For dicts, in > contrast, we are talking about a "lossy" union/combination where what > is lost depends on the order of the operands?this is less transparent > in the output. However it is spelled, though, I would be thrilled if > such an operation were built in. :) > I can write my module for this. :) (I need to document myself on magic methods) Well, if dict are vectors I must define : - matrix (projection of vectors in an another base) - distance (cosine similarity) I will therefore be able to define the notion of proximity once done that will tell if something is almost another thing a word counter of a text should be close to the word counters of the keyword it triggers. PS I did not quite catch the functionnal thingy. Cheers -- jul From jamesghutchison at gmail.com Mon Jan 2 19:09:13 2012 From: jamesghutchison at gmail.com (James Hutchison) Date: Mon, 2 Jan 2012 11:09:13 -0700 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: Here are some "utility" functions I had to write in my current project: # add entries from d2 to d1 # supports only dict and numeric types for now # recursive, applies function to all subdicts def addDictEntries(d1, d2) # take the max of entries from d2 to d1 # supports only dict and numeric types for now # recursive, applies function to all subdicts def maxDictEntries(d1, d2): # take the min of entries from d2 to d1 # supports only dict and numeric types for now # recursive, applies function to all subdicts def minDictEntries(d1, d2) # merges entries from d2 to d1 like update does # if an entry already exists and the values differ then throws an exception # recursive, applies function to all subdicts def mergeDictEntriesSafe(d1, d2) # traverses through the dict and converts all defaultdicts to dicts # this is done so that missing elements will throw an exception rather than creating a default entry # also, defaultdicts aren't picklable so this is required in order to serialize a dict def unDefaultDict(d) # same as unDefaultDict but isn't recursive. Only applies to top level keys def unDefaultDictShallow(d) The usage for all this varies but I found that combining / adding dicts together becomes very prevalent when you split datasets up for multihreading/multiprocessing purposes and need to recombine it later. so I propose that a "combine" function is added to the dict class and all dict subclasses. The function would have the following arguments: def combine(self, d2, conflictFunc=None): If combineFunc is a function, then that function is called and the result is used. val = conflictFunc(k1,k2); If combineFunc is a string, then self's key value calls conflictFunc and the new value is the return value def combine(self, d2, combineFunc=None): if combineFunc == None: for k,i in d2.items(): if(not k in self): self[k] = copy.deepcopy(i); else: if(hasattr(i,'keys')): self[k].combine(i); else: if self[k] != i: raise someUsefulException(); return; if (type(combineFunc) == str): for k,i in d2.items(): if(not k in self): self[k] = copy.deepcopy(i); else: if(hasattr(i,'keys')): self[k].combine(i,combineFunc); else: self[k] = getattr(self[k],combineFunc)(i); else: for k,i in d2.items(): if(not k in self): self[k] = copy.deepcopy(i); else: if(hasattr(i,'keys')): self[k].combine(i,combineFunc); else: self[k] = combineFunc(self[k],i); examples: d.combine(d2); # update with d2 but throw an exception if any keys have different values d.combine(d2, myCustomFunc); # if conflict, then result is = myCustomFunc(v1,v2) which would be defined elsewhere d.combine(d2, '__add__'); # if conflict, then result is = v1 + v2 >>print(d1) {'a': 'b', 'c': {'g': 5}} >>print(d2) {'a': 'ecause', 'c': {'g': 10}} >>d1.combine(d2,'__add__'); >>print(d1) {'a': 'because', 'c': {'g': 15}} I think doing so would be more powerful and less ambiguous than trying to give dict a '+' A second function I suggest adding is asDict(). This would return a dictionary with all sub dictionaries-types converted to the dict type. I.E. all defaultdicts and counters become just plain dicts. This would be the same as my unDefaultDict function. It could take an optional argument to do the conversion only to the shallow keys def asDict(self, shallow=False) From goodman.m.w at gmail.com Mon Jan 2 23:49:16 2012 From: goodman.m.w at gmail.com (goodman.m.w at gmail.com) Date: Mon, 2 Jan 2012 14:49:16 -0800 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: It's funny, I joined this list several days ago to ask about this very problem. Perhaps I can add something (there's code below if you want to skip the discussion): I do research in computational linguistics, and adding/incrementing dictionary values is a very common task (e.g. counting unique words in a corpus). A defaultdict and a for loop have served me well toward this end (and Counter may be nice, but I haven't tried it), but I've always wanted something more functional. I found in Haskell a nice method Map.fromListWith that takes a list of (key, value) pairs and a function describing how to deal with collisions. It's useful and flexible. Summing up what has been said, as I understand it, it's not obvious what a good behaviour for + is with dicts, and we can't assure it follows algebraic properties like commutativity. As Guido hinted at in the first messages, addition over disjoint dicts (and we can include the identity dict) should be the same no matter the operation. That is: {k1:v1} + {k2:v2} = {k1:v1, k2:v2} {k1:v1} + {} = {k1:v1} The only interesting cases are when we have the same key: {k1:v1} + {k1:v2} = {k1:op(v1,v2)} where op is some binary function. While dict doesn't have __add__ defined, it does resolve conflicts by just using the latter value (i.e. op=lambda x, y: y), as in the following three statements: d = dict([(k1,v1), (k1,v2)]) # result: {k1:v2} d[k1] = v3 # result: {k1:v3} d.update([(k1,v4) # result: {k1:v4} I think the latter two, at least, should not be counted as addition, but rather stay as assignment. Someone brought up the idea of a __collision__ method that is called when __setitem__ is called on an existing key. I like this idea, and there's probably some use for it, but for implementing addition it might not be practical (how, then, would you re-assign a key's value without forcing "op" to apply on the new and existing value?). I propose something like the following proof-of-concept: class AccumulationDict(dict): def __init__(self, accumulator, *args, **kwargs): if not callable(accumulator): raise TypeError('Accumulator must be callable.') self.accumulator = accumulator self.accumulate(*args, **kwargs) def __additem__(self, key, value): if key in self: self[key] = self.accumulator(self[key], value) else: self[key] = value def __add__(self, other): result = AccumulationDict(self.accumulator, self) # check if other's accumulator is same? result.accumulate(other) return result def accumulate(self, *args, **kwargs): for arg in args: if isinstance(arg, list): for (key, value) in arg: self.__additem__(key, value) elif isinstance(arg, dict): for (key, value) in arg.items(): self.__additem__(key, value) else: raise TypeError('Argument must be of type list or dict.') for key in kwargs: self.__additem__(key, kwargs[key]) Which has the following properties: * takes a binary function for accumulating existing and new values * otherwise can be instantiated like a dict * accumulate() function is like update(), but uses the accumulator to deal with conflicts * KeyErrors still occur when getting a non-existent key * addition is supported and returns a new object with the merged dicts * __setitem__ and update() are unchanged so they can be used to assign values >>> d = AccumulationDict(operator.add, [('a',1),('b',1),('a',1)], b=1) >>> d {'a': 2, 'b': 2} >>> d['a'] = 0 >>> d {'a': 0, 'b': 2} >>> d + {'a':3,'b':1} {'a': 3, 'b': 3} >>> d {'a': 0, 'b': 2} >>> d['c'] Traceback (most recent call last): File "", line 1, in KeyError: 'c' >>> d.update({'c':1}) >>> d {'a': 0, 'c': 1, 'b': 2} >>> d.accumulate({'c':1}) >>> d {'a': 0, 'c': 2, 'b': 2} I hope this contributes to the discussion, and I'd be happy to hear your thoughts. Thanks, -- -Michael Wayne Goodman From eliben at gmail.com Tue Jan 3 04:08:27 2012 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 3 Jan 2012 05:08:27 +0200 Subject: [Python-ideas] working on a PEP for the __preview__ package Message-ID: Hello, I plan to take the ideas discussed in the thread on the __experimental__/__preview__ package (1), and further elaborated by Nick Coghlan in (2), and turn it into a PEP. Further, I'm willing to invest the time resources to make the package release-worthy before Alpha 1 of 3.3, with at least one module in it. This work will be done under the mentorship of Nick Coghlan. Any ideas, suggestions or objections are welcome. Eli ---- (1) http://mail.python.org/pipermail/python-ideas/2011-August/011317.html (2) http://readthedocs.org/docs/ncoghlan_devs-python-notes/en/latest/pep_ideas/preview_namespace.html From ncoghlan at gmail.com Tue Jan 3 05:32:12 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Jan 2012 14:32:12 +1000 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: Message-ID: On Tue, Jan 3, 2012 at 1:08 PM, Eli Bendersky wrote: > Hello, > > I plan to take the ideas discussed in the thread on the > __experimental__/__preview__ package (1), and further elaborated by > Nick Coghlan in (2), and turn it into a PEP. Further, I'm willing to > invest the time resources to make the package release-worthy before > Alpha 1 of 3.3, with at least one module in it. This work will be done > under the mentorship of Nick Coghlan. > > Any ideas, suggestions or objections are welcome. I think one thing that needs to be made crystal clear (both in the PEP and in the documentation for the namespace) is that anyone attempting to combine the preview namespace with persistence mechanisms that rely on module names are setting themselves up for migration problems in the future. It isn't just the case that the __preview__ namespace APIs *might* change: they're actually *guaranteed* to change, when they move to their final location (probably, but not necessarily, in the next release). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From anacrolix at gmail.com Tue Jan 3 15:57:26 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 4 Jan 2012 01:57:26 +1100 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: Message-ID: Just a suggestion: Golang uses an "exp" namespace, this would translate to a exp package for grouping like modules in a similar way that xml, http and concurrent are currently used. http://golang.org/pkg/exp/ The name is unoffensive, and somewhat more palatable than __preview__, which really isn't a protocol, or a bag for special interpreter switches like __future__. Under this scheme, the currently proposed modules would be: exp.regex exp.daemon exp.ipaddr Rather than __preview__.regex __preview__.daemon __preview__.ipaddr Either way I really look forward to something for PEP 3153 being trialed. http://www.python.org/dev/peps/pep-3153/ Matt On Tue, Jan 3, 2012 at 3:32 PM, Nick Coghlan wrote: > On Tue, Jan 3, 2012 at 1:08 PM, Eli Bendersky wrote: >> Hello, >> >> I plan to take the ideas discussed in the thread on the >> __experimental__/__preview__ package (1), and further elaborated by >> Nick Coghlan in (2), and turn it into a PEP. Further, I'm willing to >> invest the time resources to make the package release-worthy before >> Alpha 1 of 3.3, with at least one module in it. This work will be done >> under the mentorship of Nick Coghlan. >> >> Any ideas, suggestions or objections are welcome. > > I think one thing that needs to be made crystal clear (both in the PEP > and in the documentation for the namespace) is that anyone attempting > to combine the preview namespace with persistence mechanisms that rely > on module names are setting themselves up for migration problems in > the future. > > It isn't just the case that the __preview__ namespace APIs *might* > change: they're actually *guaranteed* to change, when they move to > their final location (probably, but not necessarily, in the next > release). > > Cheers, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- ?_? From digitalxero at gmail.com Tue Jan 3 16:06:09 2012 From: digitalxero at gmail.com (Dj Gilcrease) Date: Tue, 3 Jan 2012 10:06:09 -0500 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: Message-ID: On Tue, Jan 3, 2012 at 9:57 AM, Matt Joiner wrote: > The name is unoffensive, and somewhat more palatable than __preview__, > which really isn't a protocol, or a bag for special interpreter > switches like __future__. It can be a bag for special interpreter switches or packages that are up for api review. Also not using a dunder name space indicates stability so it isnt an option From techtonik at gmail.com Tue Jan 3 16:47:47 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 3 Jan 2012 18:47:47 +0300 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: Message-ID: On Tue, Jan 3, 2012 at 6:08 AM, Eli Bendersky wrote: > Hello, > > I plan to take the ideas discussed in the thread on the > __experimental__/__preview__ package (1), and further elaborated by > Nick Coghlan in (2), and turn it into a PEP. Further, I'm willing to > invest the time resources to make the package release-worthy before > Alpha 1 of 3.3, with at least one module in it. This work will be done > under the mentorship of Nick Coghlan. > > Any ideas, suggestions or objections are welcome. > > Eli > > ---- > (1) http://mail.python.org/pipermail/python-ideas/2011-August/011317.html > (2) > http://readthedocs.org/docs/ncoghlan_devs-python-notes/en/latest/pep_ideas/preview_namespace.html Before attempting any library refactoring or addition, I would first think about placing a feedback mechanism in place to collect public response about satisfaction with current stdlib contents, and collecting proposals about parts that should be included into stdlib (or another bundle shipped with Python). -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Jan 3 17:27:35 2012 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 03 Jan 2012 16:27:35 +0000 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: Message-ID: On 1/3/12 2:57 PM, Matt Joiner wrote: > Just a suggestion: > > Golang uses an "exp" namespace, this would translate to a exp package > for grouping like modules in a similar way that xml, http and > concurrent are currently used. > http://golang.org/pkg/exp/ > > The name is unoffensive, and somewhat more palatable than __preview__, > which really isn't a protocol, or a bag for special interpreter > switches like __future__. > > Under this scheme, the currently proposed modules would be: > > exp.regex > exp.daemon > exp.ipaddr > > Rather than > > __preview__.regex > __preview__.daemon > __preview__.ipaddr The nice thing about the dunders is that no one will make a module with a conflicting name. That's really the common factor behind all uses of dunders: reserving a name for Python's use. Whether something is a protocol or bag of switches doesn't really enter into it. Like "new" and "random", "exp" or anything else that means something like "experimental" is going to trigger some problems when people use that name for *their* experimental modules. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nathan.alexander.rice at gmail.com Tue Jan 3 17:59:45 2012 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Tue, 3 Jan 2012 11:59:45 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: This is slightly tangential, but I've always wondered... Why aren't set operations implemented on dicts? It is fairly natural to view a dictionary as a set of (key, value) pairs. Things like subset/superset checking (with concomitant operator support) make sense. I have written stuff like set(dict1.items()) < set(dict2.items()) many times. I don't even know what rich comparison operators on dictionaries do now, it isn't intuitive at all. Nathan From robert.kern at gmail.com Tue Jan 3 18:10:51 2012 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 03 Jan 2012 17:10:51 +0000 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On 1/3/12 4:59 PM, Nathan Rice wrote: > This is slightly tangential, but I've always wondered... Why aren't > set operations implemented on dicts? It is fairly natural to view a > dictionary as a set of (key, value) pairs. Things like > subset/superset checking (with concomitant operator support) make > sense. I have written stuff like set(dict1.items())< > set(dict2.items()) many times. The values are unrestricted Python objects. They do not have to be hashable or sortable. The set operations you describe would have to be require one or both (or else do something algorithmically horrendous). Further, you cannot treat dicts as sets of (key, value) pairs because dicts have unique keys, not unique (key, value) pairs. > I don't even know what rich comparison operators on dictionaries do > now, it isn't intuitive at all. The rich comparison operators are only defined for == and !=. In Python 2.x, there is a legacy implementation of __cmp__ that does something more complicated. I recommend ignoring it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From solipsis at pitrou.net Tue Jan 3 18:29:42 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 3 Jan 2012 18:29:42 +0100 Subject: [Python-ideas] working on a PEP for the __preview__ package References: Message-ID: <20120103182942.1552481e@pitrou.net> On Tue, 03 Jan 2012 16:27:35 +0000 Robert Kern wrote: > On 1/3/12 2:57 PM, Matt Joiner wrote: > > Just a suggestion: > > > > Golang uses an "exp" namespace, this would translate to a exp package > > for grouping like modules in a similar way that xml, http and > > concurrent are currently used. > > http://golang.org/pkg/exp/ > > > > The name is unoffensive, and somewhat more palatable than __preview__, > > which really isn't a protocol, or a bag for special interpreter > > switches like __future__. > > > > Under this scheme, the currently proposed modules would be: > > > > exp.regex > > exp.daemon > > exp.ipaddr > > > > Rather than > > > > __preview__.regex > > __preview__.daemon > > __preview__.ipaddr > > The nice thing about the dunders is that no one will make a module with a > conflicting name. That's really the common factor behind all uses of dunders: > reserving a name for Python's use. Whether something is a protocol or bag of > switches doesn't really enter into it. Plus, writing "from __preview__ import regex" looks cooler than "from exp import regex". Regards Antoine. From nathan.alexander.rice at gmail.com Tue Jan 3 18:39:48 2012 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Tue, 3 Jan 2012 12:39:48 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On Tue, Jan 3, 2012 at 12:10 PM, Robert Kern wrote: > On 1/3/12 4:59 PM, Nathan Rice wrote: >> >> This is slightly tangential, but I've always wondered... Why aren't >> set operations implemented on dicts? ?It is fairly natural to view a >> dictionary as a set of (key, value) pairs. ?Things like >> subset/superset checking (with concomitant operator support) make >> sense. ?I have written stuff like set(dict1.items())< >> set(dict2.items()) many times. > > > The values are unrestricted Python objects. They do not have to be hashable > or sortable. The set operations you describe would have to be require one or > both (or else do something algorithmically horrendous). I haven't had any problems with using set(somedict.items()). I will admit that I primarily do this in simple contexts. This brings me to another curiosity... Why do mutable items not implement hash using id()? > Further, you cannot treat dicts as sets of (key, value) pairs because dicts > have unique keys, not unique (key, value) pairs. I'm confused. Because keys are unique, (key, value) pairs are unique as well. I realize the values are not unique but the tuple is guaranteed to be. >> I don't even know what rich comparison operators on dictionaries do >> now, it isn't intuitive at all. > > > The rich comparison operators are only defined for == and !=. In Python 2.x, > there is a legacy implementation of __cmp__ that does something more > complicated. I recommend ignoring it. Recommendation definitely noted :D Nathan From robert.kern at gmail.com Tue Jan 3 19:45:56 2012 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 03 Jan 2012 18:45:56 +0000 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On 1/3/12 5:39 PM, Nathan Rice wrote: > On Tue, Jan 3, 2012 at 12:10 PM, Robert Kern wrote: >> On 1/3/12 4:59 PM, Nathan Rice wrote: >>> >>> This is slightly tangential, but I've always wondered... Why aren't >>> set operations implemented on dicts? It is fairly natural to view a >>> dictionary as a set of (key, value) pairs. Things like >>> subset/superset checking (with concomitant operator support) make >>> sense. I have written stuff like set(dict1.items())< >>> set(dict2.items()) many times. >> >> >> The values are unrestricted Python objects. They do not have to be hashable >> or sortable. The set operations you describe would have to be require one or >> both (or else do something algorithmically horrendous). > > I haven't had any problems with using set(somedict.items()). I will > admit that I primarily do this in simple contexts. > > This brings me to another curiosity... Why do mutable items not > implement hash using id()? Usually because they do not implement __eq__ using id(). The invariant that needs to be maintained is that if two objects compare equal, then they need to hash equal. Many mutable objects compare by value, and thus two non-identical objects can compare equal. >> Further, you cannot treat dicts as sets of (key, value) pairs because dicts >> have unique keys, not unique (key, value) pairs. > > I'm confused. Because keys are unique, (key, value) pairs are unique > as well. I realize the values are not unique but the tuple is > guaranteed to be. Yes, the dict.items() would be a valid, unique set, but set operations on those sets may not give valid dicts because the same key could point to different values in the two dict operands. For example: [~] |7> a = {'one': 1} [~] |8> b = {'one': '1'} [~] |9> set(a.items()) | set(b.items()) set([('one', '1'), ('one', 1)]) [~] |10> dict(set(a.items()) | set(b.items())) {'one': 1} You've lost a value there. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From joshua.landau.ws at gmail.com Tue Jan 3 19:55:45 2012 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 3 Jan 2012 18:55:45 +0000 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On 3 January 2012 18:45, Robert Kern wrote: > [~] > |7> a = {'one': 1} > > [~] > |8> b = {'one': '1'} > > [~] > |9> set(a.items()) | set(b.items()) > set([('one', '1'), ('one', 1)]) > > [~] > |10> dict(set(a.items()) | set(b.items())) > {'one': 1} > I don't know what you've done here: >>> a = {'one': 1} >>> b = {'one': 1} >>> set(a.items()) | set(b.items()) {('one', 1)} >>> dict(set(a.items()) | set(b.items())) {'one': 1} >>> ( However, your point remains: >>> a = {'number': 1} >>> b = {'number': 2} >>> set(a.items()) | set(b.items()) {('number', 1), ('number', 2)} >>> dict(set(a.items()) | set(b.items())) {'number': 2} >>> ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Jan 3 19:58:46 2012 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 03 Jan 2012 18:58:46 +0000 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On 1/3/12 6:55 PM, Joshua Landau wrote: > On 3 January 2012 18:45, Robert Kern > > wrote: > > [~] > |7> a = {'one': 1} > > [~] > |8> b = {'one': '1'} > > [~] > |9> set(a.items()) | set(b.items()) > set([('one', '1'), ('one', 1)]) > > [~] > |10> dict(set(a.items()) | set(b.items())) > {'one': 1} > > > I don't know what you've done here: > > >>> a = {'one': 1} > >>> b = {'one': 1} > >>> set(a.items()) | set(b.items()) > {('one', 1)} > >>> dict(set(a.items()) | set(b.items())) > {'one': 1} > >>> Sorry, I was a little too clever for my own good: a = {'one': 1} b = {'one': '1'} The first has the integer 1; the second has the string '1'. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nathan.alexander.rice at gmail.com Tue Jan 3 21:37:42 2012 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Tue, 3 Jan 2012 15:37:42 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On Tue, Jan 3, 2012 at 1:45 PM, Robert Kern wrote: > On 1/3/12 5:39 PM, Nathan Rice wrote: >> I haven't had any problems with using set(somedict.items()). ?I will >> admit that I primarily do this in simple contexts. >> >> This brings me to another curiosity... Why do mutable items not >> implement hash using id()? > > > Usually because they do not implement __eq__ using id(). The invariant that > needs to be maintained is that if two objects compare equal, then they need > to hash equal. Many mutable objects compare by value, and thus two > non-identical objects can compare equal. Ok. I missed that invariant in the data model (and a couple of other places, apparently). Thanks. >>> Further, you cannot treat dicts as sets of (key, value) pairs because >>> dicts >>> have unique keys, not unique (key, value) pairs. >> >> >> I'm confused. ?Because keys are unique, (key, value) pairs are unique >> as well. ?I realize the values are not unique but the tuple is >> guaranteed to be. > > > Yes, the dict.items() would be a valid, unique set, but set operations on > those sets may not give valid dicts because the same key could point to > different values in the two dict operands. For example: > > [~] > |7> a = {'one': 1} > > [~] > |8> b = {'one': '1'} > > [~] > |9> set(a.items()) | set(b.items()) > set([('one', '1'), ('one', 1)]) > > [~] > |10> dict(set(a.items()) | set(b.items())) > {'one': 1} > > You've lost a value there. I have been following the earlier discussion of + in the context of dictionaries, and I agree that many operators do not cleanly map over. It seems to me that the comparison operators are well behaved in this regard though. As an additional aside, perhaps the return type of elements in the items call could be made into a nice "Item" namedtuple. I always unpack my items into (k, v) or something similar when possible, but some people aren't as considerate. Having item.key and item.value available (and used in examples) encourages nice code; even if someone chose a horrible name for their items() unpacking, it is easier to grok "badname.key ... badname.value" than "badname[0] ... badname[1]". Nathan From jeanpierreda at gmail.com Tue Jan 3 21:50:51 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 3 Jan 2012 15:50:51 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: > The values are unrestricted Python objects. They do not have to be hashable > or sortable. The set operations you describe would have to be require one or > both (or else do something algorithmically horrendous). He only describes <, which can be implemented in linear time as: def __lt__(self, d2): if not isinstance(d2, dict): return NotImplemented return all(key in d2 and d2[key] == value for key, value in self.items()) Which set operations are you thinking of? -- Devin On Tue, Jan 3, 2012 at 12:10 PM, Robert Kern wrote: > On 1/3/12 4:59 PM, Nathan Rice wrote: >> >> This is slightly tangential, but I've always wondered... Why aren't >> set operations implemented on dicts? ?It is fairly natural to view a >> dictionary as a set of (key, value) pairs. ?Things like >> subset/superset checking (with concomitant operator support) make >> sense. ?I have written stuff like set(dict1.items())< >> set(dict2.items()) many times. > > > The values are unrestricted Python objects. They do not have to be hashable > or sortable. The set operations you describe would have to be require one or > both (or else do something algorithmically horrendous). > > Further, you cannot treat dicts as sets of (key, value) pairs because dicts > have unique keys, not unique (key, value) pairs. > > >> I don't even know what rich comparison operators on dictionaries do >> now, it isn't intuitive at all. > > > The rich comparison operators are only defined for == and !=. In Python 2.x, > there is a legacy implementation of __cmp__ that does something more > complicated. I recommend ignoring it. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > ?that is made terrible by our own mad attempt to interpret it as though it > had > ?an underlying truth." > ?-- Umberto Eco > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From robert.kern at gmail.com Tue Jan 3 21:57:25 2012 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 03 Jan 2012 20:57:25 +0000 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On 1/3/12 8:50 PM, Devin Jeanpierre wrote: >> The values are unrestricted Python objects. They do not have to be hashable >> or sortable. The set operations you describe would have to be require one or >> both (or else do something algorithmically horrendous). > > He only describes<, which can be implemented in linear time as: > > def __lt__(self, d2): > if not isinstance(d2, dict): > return NotImplemented > > return all(key in d2 and d2[key] == value for key, value in > self.items()) Actually, this implements __le__. For __lt__, you need to exclude the case where they are exactly equal. You're right though that it wouldn't be algorithmically horrendous to do this. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tjreedy at udel.edu Tue Jan 3 22:02:04 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 03 Jan 2012 16:02:04 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On 1/3/2012 11:59 AM, Nathan Rice wrote: > This is slightly tangential, but I've always wondered... Why aren't > set operations implemented on dicts? It is fairly natural to view a > dictionary as a set of (key, value) pairs. Things like > subset/superset checking (with concomitant operator support) make > sense. I have written stuff like set(dict1.items())< > set(dict2.items()) many times. As a said earlier in this thread, being able to view dicts as sets is such a good idea that it was already added a few years ago. >>> {1:1}.items() < {1:1, 2:2}.items() True > I don't even know what rich comparison operators on dictionaries do > now, it isn't intuitive at all. TypeError: unorderable types: dict() < dict() -- Terry Jan Reedy From tjreedy at udel.edu Tue Jan 3 22:10:30 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 03 Jan 2012 16:10:30 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On 1/3/2012 1:45 PM, Robert Kern wrote: > |7> a = {'one': 1} > |8> b = {'one': '1'} > |9> set(a.items()) | set(b.items()) > set([('one', '1'), ('one', 1)]) In Python 3, the subject of this list, converting set views to sets is redundant. >>> a.items() | b.items() {('one', '1'), ('one', 1)} >>> set(a.items()) | set(b.items()) {('one', '1'), ('one', 1)} -- Terry Jan Reedy From barry at python.org Tue Jan 3 22:18:43 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 3 Jan 2012 16:18:43 -0500 Subject: [Python-ideas] Idea: Google Groups web interface for Ideas References: <11616830.383.1325005310654.JavaMail.geo-discussion-forums@yqlp13> Message-ID: <20120103161843.25044187@resist.wooz.org> On Dec 27, 2011, at 09:01 AM, anatoly techtonik wrote: >As you may know, the python-ideas list is opened only to subscribers. This >is inconvenient, because: >1. it requires three step subscription process >2. it is impossible to post a reply to existing thread/idea > >There is a web-interface in Google Groups at >https://groups.google.com/forum/#!forum/python-ideas that can solve >problems above and provide some more nifty features such as embedded >search. But there comes another problem that messages posted through the >group doesn't end in list, because list requires subscription. I've >already tried to find a solution, but run out of time, so I summarized the >proposal at http://wiki.python.org/moin/MailmanWithGoogleGroups I disagree with some of the statements on that wiki page. Most if not all of our mailing lists are mirrored on Gmane and very likely many other public archive sites. You can read and post to mailing lists through Gmane, after an initial verification dance, just like I'm doing here to this message. The problem with search and "thread notifications" (if I understand the latter correctly) is a problem with Pipermail, not specifically Mailman even though the former is bundled with the latter. I've been begging people for at least a decade to help out with modernizing Pipermail, but the truth is that the state of the art in open source archivers has been abysmal for years. OpenID/OpenAuth (maybe BrowserID, etc.) - come join us in the Mailman 3 project, and help us get things polished so we can release it. I'd certainly welcome a Google Groups mirror of any of our mailing lists, just as I welcome mirrors on Gmane, The Mail Archive, etc. I do think our core technology ought to be open source, but that doesn't mean we shouldn't maximize the reach of these valuable assets of ours. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Tue Jan 3 22:22:58 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 3 Jan 2012 16:22:58 -0500 Subject: [Python-ideas] Idea: Google Groups web interface for Ideas References: <11616830.383.1325005310654.JavaMail.geo-discussion-forums@yqlp13> <14270238.69.1325067063613.JavaMail.geo-discussion-forums@yqip20> Message-ID: <20120103162258.0978cd1e@resist.wooz.org> On Dec 28, 2011, at 02:11 AM, anatoly techtonik wrote: >Does Mailman support confirmation for messages with high SPAM ratio? Not out of the box, no. First, spam detection is not Mailman's mission. This is better done with other tools in integration with an MTA. There could be some integration points with Mailman here, but it's not clear they are worth it, especially when other more interesting work could be done (e.g. DKIM, OpenPGP signatures, etc.). Of course, I welcome anyone interested to join us on mailman-developers at python.org if you want to discuss stuff like this further. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From nathan.alexander.rice at gmail.com Tue Jan 3 22:25:44 2012 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Tue, 3 Jan 2012 16:25:44 -0500 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: On Tue, Jan 3, 2012 at 4:02 PM, Terry Reedy wrote: > On 1/3/2012 11:59 AM, Nathan Rice wrote: >> >> This is slightly tangential, but I've always wondered... Why aren't >> set operations implemented on dicts? ?It is fairly natural to view a >> dictionary as a set of (key, value) pairs. ?Things like >> subset/superset checking (with concomitant operator support) make >> sense. ?I have written stuff like set(dict1.items())< >> set(dict2.items()) many times. > > > As a said earlier in this thread, being able to view dicts as sets is such a > good idea that it was already added a few years ago. > >>>> {1:1}.items() < {1:1, 2:2}.items() > True Yes, my apologies, I knew about views but I missed the part where they have set operations. I was under the impression when I wrote that they were just plain iterators. I try to keep current on Py3 nuances, but since my work ties me to Py2, I will miss subtle but important points like that from time to time. I still don't think there is anything wrong with having the comparison operations present on the dictionary directly, it seems intuitive *to me*. Of course I understand that I may be an outlier in this (and it certainly wouldn't be the first time :P). Nathan From ncoghlan at gmail.com Wed Jan 4 02:29:37 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Jan 2012 11:29:37 +1000 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: <20120103182942.1552481e@pitrou.net> References: <20120103182942.1552481e@pitrou.net> Message-ID: On Wed, Jan 4, 2012 at 3:29 AM, Antoine Pitrou wrote: > On Tue, 03 Jan 2012 16:27:35 +0000 > Robert Kern wrote: >> The nice thing about the dunders is that no one will make a module with a >> conflicting name. That's really the common factor behind all uses of dunders: >> reserving a name for Python's use. Whether something is a protocol or bag of >> switches doesn't really enter into it. > > Plus, writing "from __preview__ import regex" looks cooler than > "from exp import regex". Dunder names also alert people that something special is going on (in this case, it helps alert them to the fact that the modules in this namespace *will* be moved in the future). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Jan 4 02:34:22 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Jan 2012 11:34:22 +1000 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: Message-ID: On Wed, Jan 4, 2012 at 1:47 AM, anatoly techtonik wrote: > Before attempting any library refactoring or addition, I would first think > about placing a feedback mechanism in place to collect public response about > satisfaction with current stdlib contents, and collecting proposals about > parts that should be included into stdlib (or another bundle shipped with > Python). Sorry anatoly, you're not going to win that one. We're well aware you don't like the use of mailing lists, the issue tracker, IRC, blogs, Reddit, Hacker News, Stack Overflow, etc for feedback and want something else that is formal and structured, but isn't the issue tracker (since you object to that for reasons I don't really understand). If you want to set up your own feedback mechanism, encourage people to use it, and act as a conduit between your new mechanism and the channels the current core devs are already paying attention to, feel free. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Wed Jan 4 02:37:08 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 4 Jan 2012 02:37:08 +0100 Subject: [Python-ideas] working on a PEP for the __preview__ package References: <20120103182942.1552481e@pitrou.net> Message-ID: <20120104023708.75e25be8@pitrou.net> (and by the way I think __preview__ is a *very* interesting idea!) cheers Antoine. On Wed, 4 Jan 2012 11:29:37 +1000 Nick Coghlan wrote: > On Wed, Jan 4, 2012 at 3:29 AM, Antoine Pitrou wrote: > > On Tue, 03 Jan 2012 16:27:35 +0000 > > Robert Kern wrote: > >> The nice thing about the dunders is that no one will make a module with a > >> conflicting name. That's really the common factor behind all uses of dunders: > >> reserving a name for Python's use. Whether something is a protocol or bag of > >> switches doesn't really enter into it. > > > > Plus, writing "from __preview__ import regex" looks cooler than > > "from exp import regex". > > Dunder names also alert people that something special is going on (in > this case, it helps alert them to the fact that the modules in this > namespace *will* be moved in the future). > > Cheers, > Nick. From stephen at xemacs.org Wed Jan 4 03:48:03 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 04 Jan 2012 11:48:03 +0900 Subject: [Python-ideas] Idea: Google Groups web interface for Ideas In-Reply-To: <20120103162258.0978cd1e@resist.wooz.org> References: <11616830.383.1325005310654.JavaMail.geo-discussion-forums@yqlp13> <14270238.69.1325067063613.JavaMail.geo-discussion-forums@yqip20> <20120103162258.0978cd1e@resist.wooz.org> Message-ID: <87mxa441zw.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > On Dec 28, 2011, at 02:11 AM, anatoly techtonik wrote: > > >Does Mailman support confirmation for messages with high SPAM ratio? > > Not out of the box, no. But it would be trivial to implement efficiently, I think, even in MM2, as long as there is a front-end (eg, spamassassin) that does the spam-checking efficiently. > First, spam detection is not Mailman's mission. A big +1 to that, but I've had problems on the XEmacs lists with marginally spammy stuff. (Eg, one frequent contributor had a French-castle-guard+some-even-more-awful-Aussie-slang version of spook.el hooked up to Gnus, and he regularly ended up snagged by the "male potency enhancers" filter. But actual spam outnumbered his posts about 99 to 1. :^) So you might want to let some spammy stuff through *to* Mailman, yet still have Mailman hold the spammy-ER stuff among that. I don't know if Python lists ever have that kind of problem, though. > There could be some integration points with Mailman here, but it's > not clear they are worth it, For mailman-developers, I think it's preferable to teach people to fish (here, write and install simple Handlers), and provide a contrib page on the wiki for posting them. From techtonik at gmail.com Wed Jan 4 11:45:52 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 4 Jan 2012 13:45:52 +0300 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: <20120103182942.1552481e@pitrou.net> Message-ID: On Wed, Jan 4, 2012 at 4:29 AM, Nick Coghlan wrote: > On Wed, Jan 4, 2012 at 3:29 AM, Antoine Pitrou > wrote: > > On Tue, 03 Jan 2012 16:27:35 +0000 > > Robert Kern wrote: > >> The nice thing about the dunders is that no one will make a module with > a > >> conflicting name. That's really the common factor behind all uses of > dunders: > >> reserving a name for Python's use. Whether something is a protocol or > bag of > >> switches doesn't really enter into it. > > > > Plus, writing "from __preview__ import regex" looks cooler than > > "from exp import regex". > > Dunder names also alert people that something special is going on (in > this case, it helps alert them to the fact that the modules in this > namespace *will* be moved in the future). I am afraid that dunders will become a standard thing in many Python packages, because of everybody's natural desire to use latest features that work right. That means that new people to Python will have to deal with Python magic right from the start, which will make language more complicated for them. A correct way IMO would be to introduce a versioned exp27, exp271 packages: 1. It makes it explicitly clear when imported package version was considered experimental 2. It allows to switch between multiple experimental package versions 3. It will provide future compatibility if a package released from `exp` breaks API 5. as a sum of 2+3 above - it provides a development process and extends time to stabilize API with public testing and feedback before freezing the package to die in stdlib -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Wed Jan 4 12:23:55 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 4 Jan 2012 14:23:55 +0300 Subject: [Python-ideas] Public feedback tools (Was: working on a PEP for the __preview__ package) Message-ID: On Wed, Jan 4, 2012 at 4:34 AM, Nick Coghlan wrote: > On Wed, Jan 4, 2012 at 1:47 AM, anatoly techtonik > wrote: > > Before attempting any library refactoring or addition, I would first > think > > about placing a feedback mechanism in place to collect public response > about > > satisfaction with current stdlib contents, and collecting proposals about > > parts that should be included into stdlib (or another bundle shipped with > > Python). > > Sorry anatoly, you're not going to win that one. We're well aware you > don't like the use of mailing lists, the issue tracker, IRC, blogs, > Reddit, Hacker News, Stack Overflow, etc for feedback and want > something else that is formal and structured, but isn't the issue > tracker (since you object to that for reasons I don't really > understand). I want something that will accumulate and summarize the outcome of all these informal activities in a structured way. With a tight time limits it is impossible to read and follow all discussions, so there should be an easy way to say +1 or -1 to filter useful stuff. There is nothing wrong with Reddit, Stack Overflow and etc., except that it is very hard to figure out trends in Python development. Without visible trends (or Roadmap FWIW) - you can't see what is going to be changed and how can you help (in accordance with your own skills and motivations). Even if you don't want to (or can't) help, you at least want to place your +1, -1 and see what other people think is right or wrong about Python. At least you will learn something new that will help to change things you've become aware of in the future. If you want to set up your own feedback mechanism, > encourage people to use it, and act as a conduit between your new > mechanism and the channels the current core devs are already paying > attention to, feel free. If only I could finish anything alone.. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Wed Jan 4 12:29:33 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 04 Jan 2012 20:29:33 +0900 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: <20120103182942.1552481e@pitrou.net> Message-ID: <878vln4sf6.fsf@uwakimon.sk.tsukuba.ac.jp> anatoly techtonik writes: > I am afraid that dunders will become a standard thing in many Python > packages, because of everybody's natural desire to use latest features that > work right. __preview__ is for the latest features that have not been confirmed to work right (yet). Where's the problem? Newbies certainly should not be using such in production applications, although it's an excellent way to learn everything you always wanted to know about Python internals but were afraid to ask (and more!) From techtonik at gmail.com Wed Jan 4 12:36:27 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 4 Jan 2012 14:36:27 +0300 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: <878vln4sf6.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20120103182942.1552481e@pitrou.net> <878vln4sf6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Wed, Jan 4, 2012 at 2:29 PM, Stephen J. Turnbull wrote: > anatoly techtonik writes: > > > I am afraid that dunders will become a standard thing in many Python > > packages, because of everybody's natural desire to use latest features > that > > work right. > > __preview__ is for the latest features that have not been confirmed to > work right (yet). Where's the problem? Newbies certainly should not > be using such in production applications, although it's an excellent > way to learn everything you always wanted to know about Python > internals but were afraid to ask (and more!) > What's the point in developing code you won't use? __preview__ features don't have a Roadmap, so you may wait forever to release you package into production. With indefinite release cycle __preview__ will become useless for the most people, who will quickly lose enthusiasm to use __preview__ and continuously update their code for API breaks. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Wed Jan 4 12:49:11 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 4 Jan 2012 06:49:11 -0500 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: <20120103182942.1552481e@pitrou.net> Message-ID: > A correct way IMO would be to introduce a versioned exp27, exp271 packages: +2. Rather than silently introducing subtle semantics changes, wouldn't it be better for older imports to fail with an ImportError? Since this is just for toying around, this shouldn't break any play, but it _should_ discourage (very strongly) people using experimental stuff in their published code. Perhaps a dunder name like __preview32__ ? __preview__._32 ? > 1. It makes it explicitly clear when imported package version was considered > experimental I agree with this too. It's a good benefit. > 2. It allows to switch between multiple experimental package versions > 3. It will provide future compatibility if a package released from `exp` > breaks API Less sold on these. This isn't the purpose of __preview__ as I understand it, and including every revision of every __preview__ package could increase Python's size very dramatically over time. -- Devin On Wed, Jan 4, 2012 at 5:45 AM, anatoly techtonik wrote: > On Wed, Jan 4, 2012 at 4:29 AM, Nick Coghlan wrote: >> >> On Wed, Jan 4, 2012 at 3:29 AM, Antoine Pitrou >> wrote: >> > On Tue, 03 Jan 2012 16:27:35 +0000 >> > Robert Kern wrote: >> >> The nice thing about the dunders is that no one will make a module with >> >> a >> >> conflicting name. That's really the common factor behind all uses of >> >> dunders: >> >> reserving a name for Python's use. Whether something is a protocol or >> >> bag of >> >> switches doesn't really enter into it. >> > >> > Plus, writing "from __preview__ import regex" looks cooler than >> > "from exp import regex". >> >> Dunder names also alert people that something special is going on (in >> this case, it helps alert them to the fact that the modules in this >> namespace *will* be moved in the future). > > > I am afraid that dunders will become a standard thing in many Python > packages,?because of everybody's natural desire to use?latest features that > work right. That means that new people to Python will have to deal with > Python magic right from the start, which will make language more complicated > for them. > > A correct way IMO would be to introduce a versioned exp27, exp271 packages: > 1. It makes it explicitly clear when imported package version was considered > experimental > 2. It allows to switch between multiple experimental package versions > 3. It will provide future compatibility if a package released from `exp` > breaks API > 5. as a sum of 2+3 above - it provides a development process and extends > time to stabilize API with public testing and feedback before freezing the > package to die in stdlib > > -- > anatoly t. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From julien at tayon.net Wed Jan 4 12:54:15 2012 From: julien at tayon.net (julien tayon) Date: Wed, 4 Jan 2012 12:54:15 +0100 Subject: [Python-ideas] [Python-Dev] hello, new dict addition for new eve ? In-Reply-To: References: Message-ID: 2012/1/3 Nathan Rice : >> >> As a said earlier in this thread, being able to view dicts as sets is such a >> good idea that it was already added a few years ago. >> >>>>> {1:1}.items() < {1:1, 2:2}.items() >> True > Making me think of funny stuffs : >>> dict( a = .1 * .1 ) > dict( a = .01 ) True ( value compared, floats are behaving as they always do) >>> dict( a = lambda x : x ) == dict( a = lambda x : x ) False ( value compared, and lambda are id compared I guess) And since I always mixed up truth and lies : >>> False , True = True, False == True and "to be" or not "to be" if "logic" is "insane" else 42 -- Jul From jkbbwr at gmail.com Wed Jan 4 13:03:59 2012 From: jkbbwr at gmail.com (Jakob Bowyer) Date: Wed, 4 Jan 2012 12:03:59 +0000 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: <20120103182942.1552481e@pitrou.net> Message-ID: Surely preview should only contain whats being previewed for the next release then when its released with the next version they carry deprecation warnings and are removed in the following version e.g. v1 __preview__.magic v2 deprecate(__preview__.magic) v3 There is no longer __preview__.magic in __preview__ On Wed, Jan 4, 2012 at 11:49 AM, Devin Jeanpierre wrote: > > A correct way IMO would be to introduce a versioned exp27, exp271 > packages: > > +2. Rather than silently introducing subtle semantics changes, > wouldn't it be better for older imports to fail with an ImportError? > Since this is just for toying around, this shouldn't break any play, > but it _should_ discourage (very strongly) people using experimental > stuff in their published code. > > Perhaps a dunder name like __preview32__ ? __preview__._32 ? > > > 1. It makes it explicitly clear when imported package version was > considered > > experimental > > I agree with this too. It's a good benefit. > > > 2. It allows to switch between multiple experimental package versions > > 3. It will provide future compatibility if a package released from `exp` > > breaks API > > Less sold on these. This isn't the purpose of __preview__ as I > understand it, and including every revision of every __preview__ > package could increase Python's size very dramatically over time. > > -- Devin > > On Wed, Jan 4, 2012 at 5:45 AM, anatoly techtonik > wrote: > > On Wed, Jan 4, 2012 at 4:29 AM, Nick Coghlan wrote: > >> > >> On Wed, Jan 4, 2012 at 3:29 AM, Antoine Pitrou > >> wrote: > >> > On Tue, 03 Jan 2012 16:27:35 +0000 > >> > Robert Kern wrote: > >> >> The nice thing about the dunders is that no one will make a module > with > >> >> a > >> >> conflicting name. That's really the common factor behind all uses of > >> >> dunders: > >> >> reserving a name for Python's use. Whether something is a protocol or > >> >> bag of > >> >> switches doesn't really enter into it. > >> > > >> > Plus, writing "from __preview__ import regex" looks cooler than > >> > "from exp import regex". > >> > >> Dunder names also alert people that something special is going on (in > >> this case, it helps alert them to the fact that the modules in this > >> namespace *will* be moved in the future). > > > > > > I am afraid that dunders will become a standard thing in many Python > > packages, because of everybody's natural desire to use latest features > that > > work right. That means that new people to Python will have to deal with > > Python magic right from the start, which will make language more > complicated > > for them. > > > > A correct way IMO would be to introduce a versioned exp27, exp271 > packages: > > 1. It makes it explicitly clear when imported package version was > considered > > experimental > > 2. It allows to switch between multiple experimental package versions > > 3. It will provide future compatibility if a package released from `exp` > > breaks API > > 5. as a sum of 2+3 above - it provides a development process and extends > > time to stabilize API with public testing and feedback before freezing > the > > package to die in stdlib > > > > -- > > anatoly t. > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > http://mail.python.org/mailman/listinfo/python-ideas > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jan 4 13:09:46 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Jan 2012 22:09:46 +1000 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: <20120103182942.1552481e@pitrou.net> Message-ID: On Wed, Jan 4, 2012 at 9:49 PM, Devin Jeanpierre wrote: >> A correct way IMO would be to introduce a versioned exp27, exp271 packages: > > +2. Rather than silently introducing subtle semantics changes, > wouldn't it be better for older imports to fail with an ImportError? > Since this is just for toying around, this shouldn't break any play, > but it _should_ discourage (very strongly) people using experimental > stuff in their published code. __preview__ isn't just for toying around: so long as you don't need to support multiple Python versions, it will be just as stable and well supported as the rest of the standard library. That means people deploying to controlled environments (such as the folks in corporate and governmental environments that are intended to be the primary beneficiaries) can use it quite happily (although keeping in mind the potential for changes when finally upgrading in the future). That's the entire point of the exercise. It's unlikely that things will be in the __preview__ namespace for more than one release - I expect that most packages that are of an adequate standard to get added in the first place will make it through their preview release period without significant changes. The process just gives us one final chance to get broader feedback before we flip the switch and lock in the API indefinitely. For folks that have an easier time of dependency management, it's likely a better bet for them to depend on a PyPI release of a package rather than using the __preview__ version, but such folks don't depend as much on the standard library in the first place. People also shouldn't read too much into the relaxation of the backwards compatibility guarantees - we aren't going to go nuts and just change things on a whim. We'll just be a little less conservative when we identify problems in the preview APIs that need to be fixed before they're added to the main part of the standard library. Deciding whether or not to rely on __preview__ modules, or to depend on packages that in turn use __preview__ modules will still be a decision developers need to make for themselves. That's no different from any other form of due diligence that professional developers need to conduct on their dependencies today. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Jan 4 13:20:17 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Jan 2012 22:20:17 +1000 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: <20120103182942.1552481e@pitrou.net> Message-ID: On Wed, Jan 4, 2012 at 10:03 PM, Jakob Bowyer wrote: > Surely preview should only contain whats being previewed for the next > release then when its released with the next version they carry deprecation > warnings and are removed in the following version e.g. > > v1 > __preview__.magic > v2 > deprecate(__preview__.magic) > v3 > There is no longer __preview__.magic in __preview__ The expected progression is: 3.X __preview__.example 3.X+1 There is no longer a __preview__.example (What's New will say whether to find it in the standard library or on PyPI) I think there's a useful set of guidelines to come out of this discussion, though: 1. All __preview__ candidates must be available as supported modules on PyPI 2. Entry into __preview__ marks the start of the transition to the standard library. Issues may be reported both via project specific channels and core Python channels. 3. If insurmountable problems are encountered during the preview release (for example, buildbot instability on par with that historically exhibited by bsddb), the project will be removed from __preview__ and revert to standalone status 4. If no such problems are encountered, the module fully enters the standard library in the subsequent release, with the PyPI release remaining available for use in earlier Python versions So multi-version code (such as other packages published on PyPI) should generally depend on the PyPI version, but __preview__ becomes an option if you can't use the PyPI version for some reason. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jeanpierreda at gmail.com Wed Jan 4 13:30:51 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 4 Jan 2012 07:30:51 -0500 Subject: [Python-ideas] working on a PEP for the __preview__ package In-Reply-To: References: <20120103182942.1552481e@pitrou.net> Message-ID: > It's unlikely that things will be in the __preview__ namespace for > more than one release - I expect that most packages that are of an > adequate standard to get added in the first place will make it through > their preview release period without significant changes. The process > just gives us one final chance to get broader feedback before we flip > the switch and lock in the API indefinitely. Ah, this would remove much of the point of explicit version numbering wouldn't it? Your other paragraphs make other good points against it too. It was a good response. I am no longer +2 on explicit versioning. I still find it nice for "experimental" code to be a bit clear about which experiment you're referring to, but it isn't very important if it doesn't last long and doesn't go through that many breaking changes. In fact, the way the use case is shaping up, it isn't important at all. I suspect I misremembered or only selectively remembered the original __preview__ proposal. I suppose this is a good reason to get a PEP out, huh. -- Devin On Wed, Jan 4, 2012 at 7:09 AM, Nick Coghlan wrote: > On Wed, Jan 4, 2012 at 9:49 PM, Devin Jeanpierre wrote: >>> A correct way IMO would be to introduce a versioned exp27, exp271 packages: >> >> +2. Rather than silently introducing subtle semantics changes, >> wouldn't it be better for older imports to fail with an ImportError? >> Since this is just for toying around, this shouldn't break any play, >> but it _should_ discourage (very strongly) people using experimental >> stuff in their published code. > > __preview__ isn't just for toying around: so long as you don't need to > support multiple Python versions, it will be just as stable and well > supported as the rest of the standard library. That means people > deploying to controlled environments (such as the folks in corporate > and governmental environments that are intended to be the primary > beneficiaries) can use it quite happily (although keeping in mind the > potential for changes when finally upgrading in the future). That's > the entire point of the exercise. > > It's unlikely that things will be in the __preview__ namespace for > more than one release - I expect that most packages that are of an > adequate standard to get added in the first place will make it through > their preview release period without significant changes. The process > just gives us one final chance to get broader feedback before we flip > the switch and lock in the API indefinitely. > > For folks that have an easier time of dependency management, it's > likely a better bet for them to depend on a PyPI release of a package > rather than using the __preview__ version, but such folks don't depend > as much on the standard library in the first place. > > People also shouldn't read too much into the relaxation of the > backwards compatibility guarantees - we aren't going to go nuts and > just change things on a whim. We'll just be a little less conservative > when we identify problems in the preview APIs that need to be fixed > before they're added to the main part of the standard library. > > Deciding whether or not to rely on __preview__ modules, or to depend > on packages that in turn use __preview__ modules will still be a > decision developers need to make for themselves. That's no different > from any other form of due diligence that professional developers need > to conduct on their dependencies today. > > Cheers, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jimjjewett at gmail.com Wed Jan 4 15:41:19 2012 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 4 Jan 2012 09:41:19 -0500 Subject: [Python-ideas] Proposed PEP on concurrent programming support Message-ID: (I've added back python-ideas, because I think that is still the appropriate forum.) >.... A new > suite type - the ``transaction`` will be added to the language. The > suite will have the semantics discussed above: modifying an object in > the suite will trigger creation of a thread-local shallow copy to be > used in the Transaction. Further modifications of the original will > cause all existing copies to be discarded and the transaction to be > restarted. ... How will you know that an object has been modified? The only ways I can think of are (1) Timestamp every object -- or at least every mutable object -- and hope that everybody agrees on which modifications should count. (2) Make two copies of every object you're using in the suite; at the end, compare one of them to both the original and the one you were operating on. With this solution, you can decide for youself what counts as a modification, but it still isn't straightforward; I would consider changing a value to be changing a dict, even though nothing in the item (header) itself changed. -jJ From barry at python.org Wed Jan 4 16:18:12 2012 From: barry at python.org (Barry Warsaw) Date: Wed, 4 Jan 2012 10:18:12 -0500 Subject: [Python-ideas] Idea: Google Groups web interface for Ideas In-Reply-To: <87mxa441zw.fsf@uwakimon.sk.tsukuba.ac.jp> References: <11616830.383.1325005310654.JavaMail.geo-discussion-forums@yqlp13> <14270238.69.1325067063613.JavaMail.geo-discussion-forums@yqip20> <20120103162258.0978cd1e@resist.wooz.org> <87mxa441zw.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20120104101812.64d32e8e@limelight.wooz.org> On Jan 04, 2012, at 11:48 AM, Stephen J. Turnbull wrote: >Barry Warsaw writes: > > On Dec 28, 2011, at 02:11 AM, anatoly techtonik wrote: > > > > >Does Mailman support confirmation for messages with high SPAM ratio? > > > > Not out of the box, no. > >But it would be trivial to implement efficiently, I think, even in >MM2, as long as there is a front-end (eg, spamassassin) that does the >spam-checking efficiently. Right. If the front-end added some header containing a pattern that could be parsed for spamminess, it would be nearly trivial to write a handler that could make that determination and hold/discard/reject such a message. Skip (IIRC) and I once had a moderately-well working spambayes plugin for Mailman. >I don't know if Python lists ever have that kind of problem, though. I don't think we have much of a spam problem on the lists these days. It used to be python-list got most of its spam through the Usenet gateway. Probably in another year or so all of the terms "Usenet", "spam", and "email" will produce a reaction in our young intarweb users much like the terms "vinyl records" and "land-line" do today. There's no problem that doesn't disappear if you ignore it long enough . > > There could be some integration points with Mailman here, but it's > > not clear they are worth it, > >For mailman-developers, I think it's preferable to teach people to >fish (here, write and install simple Handlers), and provide a contrib >page on the wiki for posting them. Like all open source, contributions are welcome. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From tjreedy at udel.edu Wed Jan 4 19:11:21 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 04 Jan 2012 13:11:21 -0500 Subject: [Python-ideas] Idea: Google Groups web interface for Ideas In-Reply-To: <20120104101812.64d32e8e@limelight.wooz.org> References: <11616830.383.1325005310654.JavaMail.geo-discussion-forums@yqlp13> <14270238.69.1325067063613.JavaMail.geo-discussion-forums@yqip20> <20120103162258.0978cd1e@resist.wooz.org> <87mxa441zw.fsf@uwakimon.sk.tsukuba.ac.jp> <20120104101812.64d32e8e@limelight.wooz.org> Message-ID: On 1/4/2012 10:18 AM, Barry Warsaw wrote: >> I don't know if Python lists ever have that kind of problem, though. > > I don't think we have much of a spam problem on the lists these days. It used > to be python-list got most of its spam through the Usenet gateway. It still gets a few spam messages a day. -- Terry Jan Reedy From t.landschoff at gmx.net Wed Jan 4 23:32:04 2012 From: t.landschoff at gmx.net (Torsten Landschoff) Date: Wed, 04 Jan 2012 23:32:04 +0100 Subject: [Python-ideas] Sampling Profiler for Python Message-ID: <4F04D364.8030404@gmx.net> Hello world, since I moved from Java development to Python development, I really miss the sampling profiler of jvisualvm (http://visualvm.java.net/). Perhaps it is only me but especially for long running GUI applications, I would really like to know why it sometimes slows down to a crawl. cProfile is not really useful for this as it makes the software unresponsive alone. Also if using multiple threads (like doing some I/O in background, when using http://www.lag.net/paramiko/) it will give meaningless results. I looked for an existing profiler and found statprof, mentioned in this blog: http://wingolog.org/archives/2005/10/28/profiling However, this still fails profiling multiple threads. I don't think one can do any better without touching the interpreter core. So that's what I tried... My approach implements the profiler from the following parts: 1. Sample Trigger: A timer that triggers taking samples of all threads at fixed intervals. 2. Sampling function injection: Ensures that a sampling function is called on each thread at the first opportunity after the sample trigger fired. 3. Sampling function: A function that assigns the current time slice to the executing code on the thread it is called on. Only the second part has to be implemented in the Python interpreter core, the sample trigger and sampling functions can be provided by external modules. My implementation of (2) can be found at https://bitbucket.org/bluehorn/sampling_prof I also prepared a first implementation of (1) and (3) at https://bitbucket.org/bluehorn/profiler_util. Using that library, I can profile pybench with a 3 % slowdown (taking 100 profiling samples per second): $ export PYTHON_PATH=$HOME/python-work/profiler_util $ python3 -m sampling_profiler -- Tools/pybench/pybench.py -c stock.pybench -f prof.pybench [...] Totals: 3119ms 3072ms +1.5% 3198ms 3108ms +2.9% This also shows that the string benchmark takes the most time: Thread MainThread (3842 samples, 38.434401 seconds) ------------------------------------------------------------------------ cpu_time (cum) ticks (cum) samples (cum) filename:lineno function 0.548 0.548 55 55 55 55 Tools/pybench/Strings.py:455 test 0.448 0.448 45 45 45 45 Tools/pybench/Strings.py:568 calibrate 0.428 0.428 43 43 43 43 Tools/pybench/Constructs.py:484 test 0.380 0.380 38 38 38 38 Tools/pybench/Calls.py:445 f 0.324 0.324 34 34 34 34 Tools/pybench/Calls.py:122 f The output is explained as follows: cpu_time: cpu time in seconds accounted to that line. (cum): This is the cpu time including the cumulative time of the functions called on that line. Gathering that data means that the sampling function has to walk the stack for each sample, which will cause a dramatic slowdown for deeply recursive calls. There should be a switch to disable this. ticks: Number of sampling profiler ticks accounted to that line. For my example (where sample ticks are equidistant in wall time) this is a measure of the time spent in that line, including any blocking calls (especially I/O). (cum): ... including called functions samples: Number of times the sampling profiler function was called for that line. This should equal ticks unless execution of that line took longer than a profiler tick. The remaining output should be obvious. I have to admit that this is far from finished, but I hope to get some feedback and perhaps even help by posting it here. What do you think? Greetings, Torsten -------------- next part -------------- An HTML attachment was scrubbed... URL: From sumerc at gmail.com Thu Jan 5 07:45:29 2012 From: sumerc at gmail.com (=?ISO-8859-1?Q?S=FCmer_Cip?=) Date: Thu, 5 Jan 2012 08:45:29 +0200 Subject: [Python-ideas] Sampling Profiler for Python In-Reply-To: <4F04D364.8030404@gmx.net> References: <4F04D364.8030404@gmx.net> Message-ID: Have you looked at yappi? yappi.start() accepts a sampling parameter. Regards, On Thu, Jan 5, 2012 at 12:32 AM, Torsten Landschoff wrote: > ** > Hello world, > > since I moved from Java development to Python development, I really miss > the sampling profiler of jvisualvm (http://visualvm.java.net/). Perhaps > it is only me but especially for long running GUI applications, I would > really like to know why it sometimes slows down to a crawl. > > cProfile is not really useful for this as it makes the software > unresponsive alone. Also if using multiple threads (like doing some I/O in > background, when using http://www.lag.net/paramiko/) it will give > meaningless results. > > I looked for an existing profiler and found statprof, mentioned in this > blog: http://wingolog.org/archives/2005/10/28/profiling > > However, this still fails profiling multiple threads. I don't think one > can do any better without touching the interpreter core. So that's what I > tried... > > My approach implements the profiler from the following parts: > > 1. Sample Trigger: A timer that triggers taking samples of all threads > at fixed intervals. > 2. Sampling function injection: Ensures that a sampling function is > called on each thread at the first opportunity after the sample trigger > fired. > 3. Sampling function: A function that assigns the current time slice > to the executing code on the thread it is called on. > > Only the second part has to be implemented in the Python interpreter core, > the sample trigger and sampling functions can be provided by external > modules. My implementation of (2) can be found at > https://bitbucket.org/bluehorn/sampling_prof > > I also prepared a first implementation of (1) and (3) at > https://bitbucket.org/bluehorn/profiler_util. Using that library, I can > profile pybench with a 3 % slowdown (taking 100 profiling samples per > second): > > $ export PYTHON_PATH=$HOME/python-work/profiler_util > $ python3 -m sampling_profiler -- Tools/pybench/pybench.py -c > stock.pybench -f prof.pybench > [...] > Totals: 3119ms 3072ms +1.5% 3198ms 3108ms > +2.9% > > This also shows that the string benchmark takes the most time: > > Thread MainThread (3842 samples, 38.434401 seconds) > ------------------------------------------------------------------------ > cpu_time (cum) ticks (cum) samples (cum) > filename:lineno function > 0.548 0.548 55 55 55 55 > Tools/pybench/Strings.py:455 test > 0.448 0.448 45 45 45 45 > Tools/pybench/Strings.py:568 calibrate > 0.428 0.428 43 43 43 43 > Tools/pybench/Constructs.py:484 test > 0.380 0.380 38 38 38 38 > Tools/pybench/Calls.py:445 f > 0.324 0.324 34 34 34 34 > Tools/pybench/Calls.py:122 f > > The output is explained as follows: > > cpu_time: cpu time in seconds accounted to that line. > (cum): This is the cpu time including the cumulative time of the functions > called on that line. Gathering that data means that the sampling function > has to walk the stack for each sample, which will cause a dramatic slowdown > for deeply recursive calls. There should be a switch to disable this. > ticks: Number of sampling profiler ticks accounted to that line. For my > example (where sample ticks are equidistant in wall time) this is a measure > of the time spent in that line, including any blocking calls (especially > I/O). > (cum): ... including called functions > samples: Number of times the sampling profiler function was called for > that line. This should equal ticks unless execution of that line took > longer than a profiler tick. > > The remaining output should be obvious. > > I have to admit that this is far from finished, but I hope to get some > feedback and perhaps even help by posting it here. > > What do you think? > > Greetings, Torsten > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -- -- Sumer Cip -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.landschoff at gmx.net Sun Jan 8 02:01:04 2012 From: t.landschoff at gmx.net (Torsten Landschoff) Date: Sun, 08 Jan 2012 02:01:04 +0100 Subject: [Python-ideas] Sampling Profiler for Python In-Reply-To: References: <4F04D364.8030404@gmx.net> Message-ID: <4F08EAD0.20802@gmx.net> Hi S?mer, On 01/05/2012 07:45 AM, S?mer Cip wrote: > Have you looked at yappi? yappi.start() accepts a sampling parameter. no, I haven't because I googled for sampling profiler or statistical profiler. I compared running pybench with yappi vs. my sampling profiler (with 100 samples/sec). First, the run time increases in both cases vs. the original performance of python. The figures are different because I backported to Python2 again for the comparison (is yappi available for Python3?) yappi: Totals: 4692ms 3317ms +41.5% 4831ms 3448ms +40.1% sampling_profiler: Totals: 3491ms 3269ms +6.8% 3638ms 3435ms +5.9% These are the hot spots as reported by yappi: name #n tsub ttot tavg Tools/pybench/Calls.py.test:439 10 1.709853 1.709853 0.170985 Tools/pybench/Strings.py.test:467 10 1.383400 2.133651 0.213365 Tools/pybench/With.py.test:16 10 1.179130 1.614330 0.161433 Tools/pybench/Calls.py.test:176 10 1.088372 1.885379 0.188538 Tools/pybench/Unicode.py.test:388 10 1.062785 1.729250 0.172925 Tools/pybench/Calls.py.test:118 10 1.008680 1.284282 0.128428 Tools/pybench/With.py.test:64 10 0.999503 1.431072 0.143107 Tools/pybench/Lists.py.test:9 10 0.995866 1.394482 0.139448 Tools/pybench/Calls.py.test:9 10 0.979050 1.505686 0.150569 Tools/pybench/Dict.py.test:351 10 0.959278 1.290247 0.129025 Compared with the sampling_profiler: cpu_time (cum) ticks (cum) samples (cum) filename:lineno function 0.552 4.224 55 419 55 419 Tools/pybench/Calls.py:446 f 0.512 0.512 52 52 52 52 Tools/pybench/Unicode.py:452 calibrate 0.436 0.436 44 44 44 44 Tools/pybench/Unicode.py:542 calibrate 0.436 0.436 44 44 44 44 Tools/pybench/Strings.py:562 calibrate 0.372 0.372 37 37 37 37 Tools/pybench/Constructs.py:485 test 0.348 0.348 36 36 36 36 Tools/pybench/Calls.py:122 f 0.272 0.272 27 27 27 27 Tools/pybench/Lists.py:339 test 0.248 0.248 25 25 25 25 Tools/pybench/Calls.py:445 f 0.228 0.228 22 22 22 22 Tools/pybench/Strings.py:247 calibrate 0.212 0.212 21 21 21 21 Tools/pybench/With.py:14 __exit__ Let's look at leading hot-spots: 1) yappi reports Calls.py line 439 as the hot spot. This refers to this code: def test(self): # <-- line 439 global f def f(x): if x > 1: return f(x-1) # <-- line 446 return 1 for i in xrange(self.rounds): f(10) f(10) ... Here yappi identified the outer function, the sampling profiler points directly to the line where most of the time is spent. 2) Most other hot spots reported by sampling_profiler lie in the calibrate functions. I wondered by that might be and noticed that pybench calls these 20 times per default. As the sampling_profiler reports the hot spots based on lines vs. the functions in yappi, this means we can not really compare these results. So let's compare the behaviour of the different profilers on this code: from time import time def fib(n): if n > 1: return fib(n-1) + fib(n-2) else: return n def fib2(n): stack = [n] while stack: n = stack.pop() if n > 1: stack.append(n-1) stack.append(n-2) s = time() fib(34) print "fib: %s seconds" % (time() - s) s = time() fib2(34) print "fib2: %s seconds" % (time() - s) Here are two variants of the fibonnaci function. The first works using recursion and returns the correct result. The second is just for simulating the same behaviour without recursion by using a local stack. The result is not computed since it is not required here - I just want to compare how this behaves wrt. our profilers. When I run this without any profiler, I get fib: 2.87187790871 seconds fib2: 5.26651716232 seconds With yappi, I get this: fib: 6.42756795883 seconds fib2: 14.6631779671 seconds name #n tsub ttot tavg demo.py.:1 1 6.427689 21.090864 21.090864 demo.py.fib2:9 1 10.353305 14.663173 14.663173 ..atprof/Lib/posixpath.py.dirname:11 1 0.000004 0.000005 0.000005 ..prof/Lib/threading.py.setprofile:8 1 0.000001 0.000001 0.000001 ..n2.6/site-packages/yappi.py.stop:4 1 0.000000 0.000000 0.000000 demo.py.fib:3 184549.. 0.000000 0.000000 0.000000 The slow down for fib is about what I expected, but I have no idea why fib2 is slowed down that much. AFAIK, the usual profiling hooks of Python are called on entry and exit of each function, not on each line?! The results are incorrect for the fib function for reasons unknown to me - yappi thinks that no time is spent in that function, while the program output makes it obvious that more than 6 seconds are spent there. I tried using the timing_sample parameter to yappi.start. When set to 2, all of tsub, ttot and tavg are 0 in the output for this example. I can speed up the run to 4.97 seconds for fib and 9.33 s for fib2 using a high timing_sample value of 200 or more. That is still quite some overhead for keeping it active at all times. With the sampling profiler, the output is fib: 3.10812282562 seconds fib2: 5.91388201714 seconds Thread MainThread (902 samples, 9.036564 seconds) ------------------------------------------------------------------------ cpu_time (cum) ticks (cum) samples (cum) filename:lineno function 0.000 9.005 0 902 0 902 ython-work/statprof/Lib/runpy.py:122 _run_module_as_main 0.000 9.005 0 902 0 902 tatprof/Lib/sampling_profiler.py:203 main 0.000 9.005 0 902 0 902 tatprof/Lib/sampling_profiler.py:209 0.000 9.005 0 902 0 902 ython-work/statprof/Lib/runpy.py:34 _run_code 0.000 5.912 0 592 0 592 demo.py:21 2.068 3.092 211 310 211 310 demo.py:5 fib 0.000 3.092 0 310 0 310 demo.py:18 2.196 2.196 217 217 217 217 demo.py:12 fib2 1.576 1.576 157 157 157 157 demo.py:15 fib2 0.980 0.980 101 101 101 101 demo.py:14 fib2 0.900 0.900 87 87 87 87 demo.py:4 fib 0.644 0.644 67 67 67 67 demo.py:11 fib2 0.516 0.516 50 50 50 50 demo.py:13 fib2 0.124 0.124 12 12 12 12 demo.py:7 fib Just a small slow down here. And as you can see, this even tells you on what lines the most cpu time is spent. While I look at this, I am missing per-function totals, but that should be easy to add. As you can see, the profiler correctly detects that line 21 and line 18 of the demo code take most of the time (which are the module-level calls to fib2 and fib). For completeness, let's look what cProfile finds out: torsten at pulsar:~/python-work/statprof$ ./python run_cprofile.py demo.py fib: 5.83289813995 seconds fib2: 11.2586638927 seconds 55364794 function calls (36909866 primitive calls) in 17.092 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 17.092 17.092 :1() 1 0.000 0.000 17.092 17.092 demo.py:1() 18454929/1 5.833 0.000 5.833 5.833 demo.py:3(fib) 1 7.831 7.831 11.259 11.259 demo.py:9(fib2) 18454928 1.421 0.000 1.421 0.000 {method 'append' of 'list' objects} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 18454929 2.007 0.000 2.007 0.000 {method 'pop' of 'list' objects} 4 0.000 0.000 0.000 0.000 {time.time} I am impressed how cProfile detects the recursive calls of the fib function (I guess that's the info in the ncalls column). Anyway, the slow down here is much more noticeable for the same information. So, why would I prefer the sampling profiler? a) Less slow down (could still use some optimization) b) Points directly to the code lines where the hot spots are (based on equidistant time samples). c) Does not distort run time by being active. Greetings, Torsten From mwm at mired.org Sun Jan 8 02:07:13 2012 From: mwm at mired.org (Mike Meyer) Date: Sat, 7 Jan 2012 17:07:13 -0800 Subject: [Python-ideas] Boolean value of file object? Message-ID: <20120107170713.3f3f1b2d@bhuda.mired.org> Just an off-the-wall thought. Is there any reason a file object's boolean value shouldn't be false once it's been closed? This would allow replacing "if f and f.closed:" (to make sure you have a file and it's open) with just "if f:". Is there some use case where you want to verify that an object is a file object instead of None (or another false value), and don't care if it's closed? My own thought is that this is a case where status quo wins, but thought someone else might think the idea has more merit. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From solipsis at pitrou.net Sun Jan 8 02:20:32 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 8 Jan 2012 02:20:32 +0100 Subject: [Python-ideas] Boolean value of file object? References: <20120107170713.3f3f1b2d@bhuda.mired.org> Message-ID: <20120108022032.2ca233f5@pitrou.net> On Sat, 7 Jan 2012 17:07:13 -0800 Mike Meyer wrote: > Just an off-the-wall thought. > > Is there any reason a file object's boolean value shouldn't be false > once it's been closed? This would allow replacing "if f and f.closed:" (to > make sure you have a file and it's open) with just "if f:". Is there > some use case where you want to verify that an object is a file > object instead of None (or another false value), and don't care if > it's closed? That sounds just too smart. It is not obvious that a file should become "false" once it is closed. Writing "f.closed" OTOH is explicit and obvious. Regards Antoine. From ncoghlan at gmail.com Sun Jan 8 02:18:30 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Jan 2012 11:18:30 +1000 Subject: [Python-ideas] Boolean value of file object? In-Reply-To: <20120107170713.3f3f1b2d@bhuda.mired.org> References: <20120107170713.3f3f1b2d@bhuda.mired.org> Message-ID: On Sun, Jan 8, 2012 at 11:07 AM, Mike Meyer wrote: > My own thought is that this is a case where status quo wins, but > thought someone else might think the idea has more merit. Unfortunately, there are way too many "file-like objects" in the wild for it to ever be feasible to rely on such a shorthand. Cute idea, though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From steve at pearwood.info Sun Jan 8 03:06:51 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 08 Jan 2012 13:06:51 +1100 Subject: [Python-ideas] Boolean value of file object? In-Reply-To: <20120107170713.3f3f1b2d@bhuda.mired.org> References: <20120107170713.3f3f1b2d@bhuda.mired.org> Message-ID: <4F08FA3B.40204@pearwood.info> Mike Meyer wrote: > Just an off-the-wall thought. > > Is there any reason a file object's boolean value shouldn't be false > once it's been closed? Is there any reason it should? There's nothing false-like about a closed file object. Truthy and falsy objects should map to "something" vs "nothing" or "non-empty" vs "empty", not "some arbitrary flag that might be useful sometimes". -- Steven From raymond.hettinger at gmail.com Sun Jan 8 03:08:30 2012 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 7 Jan 2012 18:08:30 -0800 Subject: [Python-ideas] Boolean value of file object? In-Reply-To: <20120107170713.3f3f1b2d@bhuda.mired.org> References: <20120107170713.3f3f1b2d@bhuda.mired.org> Message-ID: <9904C12C-7DD7-4C54-B1C3-14BBDCF87B6F@gmail.com> On Jan 7, 2012, at 5:07 PM, Mike Meyer wrote: > Is there any reason a file object's boolean value shouldn't be false > once it's been closed? Because that would make it harder to learn what things in Python can be False and what that implies. Currently, None is always false, numbers are false when they are zero, and containers are false when they are empty. Files objects don't fit into that model. Some would question whether a file could be considered a container. Even if a file was considered a container, there is still an important distinction between files that are closed versus files that are empty (i.e. they have a length of zero). Lastly, the file API is adopted by many objects, so we would need to change them all. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sun Jan 8 17:17:30 2012 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 8 Jan 2012 16:17:30 +0000 (UTC) Subject: [Python-ideas] adding a casefold() method to str Message-ID: Hi, Casefolding (Unicode Standard 3.13) is a more aggressive version of lowercasing. It's purpose to assist in the implementation of caseless mapping. For example, under lowercase "?" -> "?" but under casefolding "?" -> "ss". I propose we add a casefold() method. So, case-insensitive matching should really be "one.casefold() == two.casefold()" rather than "one.lower() == two.lower()". Regards, Benjamin From steve at pearwood.info Sun Jan 8 17:58:17 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 Jan 2012 03:58:17 +1100 Subject: [Python-ideas] adding a casefold() method to str In-Reply-To: References: Message-ID: <4F09CB29.1000106@pearwood.info> Benjamin Peterson wrote: > Hi, > Casefolding (Unicode Standard 3.13) is a more aggressive version of lowercasing. > It's purpose to assist in the implementation of caseless mapping. For example, > under lowercase "?" -> "?" but under casefolding "?" -> "ss". I propose we add a > casefold() method. So, case-insensitive matching should really be > "one.casefold() == two.casefold()" > rather than "one.lower() == two.lower()". +1 in principle, but in practice case folding is more complicated than a single method might imply. The most obvious complication is treatment of dotted and dotless I. See, for example: http://unicode.org/Public/UNIDATA/CaseFolding.txt http://www.w3.org/International/wiki/Case_folding http://en.wikipedia.org/wiki/Letter_case#Unicode_case_folding_and_script_identification So while having proper Unicode case-folding is desirable, I don't know how simple it is to implement. Would it be appropriate for casefold() to take an optional argument as to which mappings to use? E.g. something like: str.casefold() # defaults to simple folding str.casefold(string.SIMPLE & string.TURKIC) str.casefold(string.FULL) or should str.casefold() only apply simple folding, with the others combinations relegated to a function in a module somewhere? I count 4 possible functions: simple casefolding, without Turkic I full casefolding, without Turkic I simple casefolding, with Turkic I full casefolding, with Turkic I -- Steven From benjamin at python.org Sun Jan 8 18:47:24 2012 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 8 Jan 2012 17:47:24 +0000 (UTC) Subject: [Python-ideas] adding a casefold() method to str References: <4F09CB29.1000106@pearwood.info> Message-ID: Steven D'Aprano writes: > or should str.casefold() only apply simple folding, with the others > combinations relegated to a function in a module somewhere? Yes, I think so. str does not have any other features dependent on locale. Section 3.3 defines "Default casefolding" which is what the casefold() method should use. From simon.sapin at kozea.fr Sun Jan 8 22:58:05 2012 From: simon.sapin at kozea.fr (Simon Sapin) Date: Sun, 08 Jan 2012 22:58:05 +0100 Subject: [Python-ideas] A key parameter for heapq.merge Message-ID: <4F0A116D.3030202@kozea.fr> Hi, According to its own documentation, the merge() function in the heapq module is similar to sorted(). However, it has none of the key and reverse parameters that sorted() has. I think they could be just as useful as in sorted(). http://docs.python.org/dev/library/heapq.html#heapq.merge http://docs.python.org/dev/library/functions.html#sorted First of all what do you think of the idea? I?m working on a patch for a key parameter. I think it can be pretty straightforward, but I?ll measure if the "no key" case becomes slower (calls to lambda x: x) At worst we can always duplicate the loop. However, I am not sure how to implement reverse. Not all values have an "opposite value" that reverses order, _nsmallest and _nlargest are quite different, and merge uses neither. Anyway, if I get this to work, it will be my first contribution to CPython. I?m trying to follow the Developer?s guide, but is there something else I should be aware of? http://docs.python.org/devguide/ Regards, -- Simon Sapin From steve at pearwood.info Mon Jan 9 00:46:48 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 09 Jan 2012 10:46:48 +1100 Subject: [Python-ideas] A key parameter for heapq.merge In-Reply-To: <4F0A116D.3030202@kozea.fr> References: <4F0A116D.3030202@kozea.fr> Message-ID: <4F0A2AE8.6090608@pearwood.info> Simon Sapin wrote: > I?m working on a patch for a key parameter. I think it can be pretty > straightforward, but I?ll measure if the "no key" case becomes slower > (calls to lambda x: x) At worst we can always duplicate the loop. In my experience, it is *much* faster to test against None every time through the loop than to call a do-nothing function. Instead of this: if key is None: key = lambda x: x for value in heap: do_something_with(key(value)) this is much faster: for value in heap: if key is None: do_something_with(value) else: do_something_with(key(value)) and this is faster still: if key is None: for value in heap: do_something_with(value) else: for value in heap: do_something_with(key(value)) YMMV; I encourage you to benchmark. -- Steven From simon.sapin at kozea.fr Mon Jan 9 08:17:24 2012 From: simon.sapin at kozea.fr (Simon Sapin) Date: Mon, 09 Jan 2012 08:17:24 +0100 Subject: [Python-ideas] A key parameter for heapq.merge In-Reply-To: <4F0A2AE8.6090608@pearwood.info> References: <4F0A116D.3030202@kozea.fr> <4F0A2AE8.6090608@pearwood.info> Message-ID: <4F0A9484.7050903@kozea.fr> Le 09/01/2012 00:46, Steven D'Aprano a ?crit : > and this is faster still: > > if key is None: > for value in heap: > do_something_with(value) > else: > for value in heap: > do_something_with(key(value)) Yes, this is what I meant by duplicating the loop, though do_something_with() is a bit longer: http://hg.python.org/cpython/file/56e9d025078d/Lib/heapq.py#l336 > YMMV; I encourage you to benchmark. I have a working patch with tests. I?ll do a benchmarks next. After that is filing an issue? Thanks, -- Simon Sapin From raymond.hettinger at gmail.com Mon Jan 9 08:36:41 2012 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 9 Jan 2012 07:36:41 +0000 Subject: [Python-ideas] A key parameter for heapq.merge In-Reply-To: <4F0A116D.3030202@kozea.fr> References: <4F0A116D.3030202@kozea.fr> Message-ID: <73792DF0-F128-437E-8AC8-A9F34D042FF4@gmail.com> On Jan 8, 2012, at 9:58 PM, Simon Sapin wrote: > According to its own documentation, the merge() function in the heapq module is similar to sorted(). However, it has none of the key and reverse parameters that sorted() has. I think they could be just as useful as in sorted(). A key-argument could be useful but a reverse parameter doesn't make as much sense. If you want to make a patch, add it to the tracker and assign it to me for review. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Mon Jan 9 09:35:17 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 9 Jan 2012 11:35:17 +0300 Subject: [Python-ideas] Boolean value of file object? In-Reply-To: <9904C12C-7DD7-4C54-B1C3-14BBDCF87B6F@gmail.com> References: <20120107170713.3f3f1b2d@bhuda.mired.org> <9904C12C-7DD7-4C54-B1C3-14BBDCF87B6F@gmail.com> Message-ID: On Sun, Jan 8, 2012 at 5:08 AM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > > there is still an important distinction between files that are closed > versus files that are empty (i.e. they have a length of zero). > and `file`s that are None. When you check f.closed explicitly - the None objects throw an exception. -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Mon Jan 9 11:30:55 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 9 Jan 2012 13:30:55 +0300 Subject: [Python-ideas] Pythonic buffering in Py3 print() Message-ID: In Python 2 "print 'something', statement calls were unbuffered and immediately appeared on screen. In Python 3 "print('something', end='')" calls are buffered. This breaks progress bars and other stuff. 1. What is more "Pythonic" and why? 2. Should Python 3 be fixed ASAP? -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon.sapin at kozea.fr Mon Jan 9 11:46:46 2012 From: simon.sapin at kozea.fr (Simon Sapin) Date: Mon, 09 Jan 2012 11:46:46 +0100 Subject: [Python-ideas] A key parameter for heapq.merge In-Reply-To: <73792DF0-F128-437E-8AC8-A9F34D042FF4@gmail.com> References: <4F0A116D.3030202@kozea.fr> <73792DF0-F128-437E-8AC8-A9F34D042FF4@gmail.com> Message-ID: <4F0AC596.60906@kozea.fr> Le 09/01/2012 08:36, Raymond Hettinger a ?crit : > > A key-argument could be useful but a reverse parameter doesn't make as > much sense. > > If you want to make a patch, add it to the tracker and assign it to me > for review. > I just opened http://bugs.python.org/issue13742 , but I can?t assign it. (New account on the tracker.) Thanks, -- Simon Sapin From eric at trueblade.com Mon Jan 9 11:51:34 2012 From: eric at trueblade.com (Eric V. Smith) Date: Mon, 09 Jan 2012 05:51:34 -0500 Subject: [Python-ideas] A key parameter for heapq.merge In-Reply-To: <4F0AC596.60906@kozea.fr> References: <4F0A116D.3030202@kozea.fr> <73792DF0-F128-437E-8AC8-A9F34D042FF4@gmail.com> <4F0AC596.60906@kozea.fr> Message-ID: <4F0AC6B6.2040403@trueblade.com> On 1/9/2012 5:46 AM, Simon Sapin wrote: > Le 09/01/2012 08:36, Raymond Hettinger a ?crit : >> >> A key-argument could be useful but a reverse parameter doesn't make as >> much sense. >> >> If you want to make a patch, add it to the tracker and assign it to me >> for review. >> > > I just opened http://bugs.python.org/issue13742 , but I can?t assign it. > (New account on the tracker.) I assigned it to Raymond. Eric. From stefan_ml at behnel.de Mon Jan 9 11:51:49 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Jan 2012 11:51:49 +0100 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: anatoly techtonik, 09.01.2012 11:30: > In Python 2 "print 'something', statement calls were unbuffered and > immediately appeared on screen. > In Python 3 "print('something', end='')" calls are buffered. This breaks > progress bars and other stuff. > > 1. What is more "Pythonic" and why? I find the Py3 behaviour more pythonic: flushing should be explicit (which is better than implicit). The mere fact that the user did *not* provide a terminator shows that there are likely other things to follow. A progress bar is just one specific case where flushing is actually desired. In other cases, it may not be desired and would rather lead to performance issues. Making it implicit if a flush happens or not prevents users from controlling it. > 2. Should Python 3 be fixed ASAP? IMHO, no. Stefan From jeanpierreda at gmail.com Mon Jan 9 11:53:09 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 9 Jan 2012 05:53:09 -0500 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: > In Python 2 "print 'something', statement calls were unbuffered and > immediately appeared on screen. No, they weren't. Python doesn't do any extra buffering or flushing. Usually your terminal emulator line-buffers stdout. If you don't print a newline (in Python 2 OR 3) then it doesn't show up until you either flush (sys.stdout.flush()) or send a newline. -- Devin On Mon, Jan 9, 2012 at 5:30 AM, anatoly techtonik wrote: > In Python 2 "print 'something', statement calls were unbuffered and > immediately appeared on screen. > In Python 3 "print('something', end='')" calls are buffered. This breaks > progress bars and other stuff. > > 1. What is more "Pythonic" and why? > 2. Should Python 3 be fixed ASAP? > -- > anatoly t. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From techtonik at gmail.com Mon Jan 9 13:16:07 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 9 Jan 2012 15:16:07 +0300 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: On Mon, Jan 9, 2012 at 1:51 PM, Stefan Behnel wrote: > anatoly techtonik, 09.01.2012 11:30: > > In Python 2 "print 'something', statement calls were unbuffered and > > immediately appeared on screen. > > In Python 3 "print('something', end='')" calls are buffered. This breaks > > progress bars and other stuff. > > > > 1. What is more "Pythonic" and why? > > I find the Py3 behaviour more pythonic: flushing should be explicit (which > is better than implicit). The mere fact that the user did *not* provide a > terminator shows that there are likely other things to follow. Do you find pythonic that `print` statement should output string to the screen (according to docs), but it doesn't do this? Do you find ability to stack chars into screen buffer before the output (instead of doing stacking to a string) more important than hiding the low level output implementation details from the users? A progress > bar is just one specific case where flushing is actually desired. In other > cases, it may not be desired and would rather lead to performance issues. > Making it implicit if a flush happens or not prevents users from > controlling it. > Can you give some examples of these 'other cases'? Maybe 95% of users are using print for progress bars, and enabling flushing by default will make it more pythonic by making code more beautiful (which is better than ugly). -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Mon Jan 9 13:21:31 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 9 Jan 2012 15:21:31 +0300 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: On Mon, Jan 9, 2012 at 1:53 PM, Devin Jeanpierre wrote: > > In Python 2 "print 'something', statement calls were unbuffered and > > immediately appeared on screen. > > No, they weren't. > --- cut py2print.py --- from time import sleep while 1: sleep(1) print ".", --- cut --- This produces one dot every second with Python 2.7.2 on Windows. Python doesn't do any extra buffering or flushing. Usually your > terminal emulator line-buffers stdout. If you don't print a newline > (in Python 2 OR 3) then it doesn't show up until you either flush > (sys.stdout.flush()) or send a newline. I assume you use Linux. Do you agree that for cross-platform language some behavior should be made consistent? -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Mon Jan 9 13:42:34 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Jan 2012 13:42:34 +0100 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: anatoly techtonik, 09.01.2012 13:16: > On Mon, Jan 9, 2012 at 1:51 PM, Stefan Behnel wrote: >> anatoly techtonik, 09.01.2012 11:30: >>> In Python 2 "print 'something', statement calls were unbuffered and >>> immediately appeared on screen. >>> In Python 3 "print('something', end='')" calls are buffered. This breaks >>> progress bars and other stuff. >>> >>> 1. What is more "Pythonic" and why? >> >> I find the Py3 behaviour more pythonic: flushing should be explicit (which >> is better than implicit). The mere fact that the user did *not* provide a >> terminator shows that there are likely other things to follow. > > Do you find pythonic that `print` statement should output string to the > screen (according to docs), but it doesn't do this? Well, it does it if you tell it to. Lacking an explicit request, the rest is platform specific, as usual with multi-level output buffering. Actually, print() is often the first place where new users get in touch with I/O "specialties" such as flush() and buffering/queuing, and it's helpful to learn about them. Not only in the context of terminal I/O but also for disk or network I/O. > Do you find ability to stack chars into screen buffer before the output > (instead of doing stacking to a string) more important than hiding the low > level output implementation details from the users? All I'm saying is that there is no "right" way to do it, so choosing to flush implicitly will a) break user code and b) keep some (or many?) users from getting the behaviour they want. And that without good reason and without making it "better" or "more correct" through such a change. >> A progress >> bar is just one specific case where flushing is actually desired. In other >> cases, it may not be desired and would rather lead to performance issues. >> Making it implicit if a flush happens or not prevents users from >> controlling it. > > Can you give some examples of these 'other cases'? Maybe 95% of users are > using print for progress bars ... or maybe not ... > and enabling flushing by default will make > it more pythonic by making code more beautiful (which is better than ugly). print() is just one way of pushing output "somewhere", and often not the best one. It's helpful for debugging and mostly acceptable in short scripts, but it's otherwise useless for most applications. That being said, I certainly don't claim to have an idea of what "most" users use print() for and what behaviour "most" users expect... Stefan From p.f.moore at gmail.com Mon Jan 9 13:50:23 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 9 Jan 2012 12:50:23 +0000 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: On 9 January 2012 12:21, anatoly techtonik wrote: >> Python doesn't do any extra buffering or flushing. Usually your >> terminal emulator line-buffers stdout. If you don't print a newline >> (in Python 2 OR 3) then it doesn't show up until you either flush >> (sys.stdout.flush()) or send a newline. > > I assume you use Linux. Do you agree that for cross-platform language some > behavior should be made consistent? No. (At least not for 2.x). The Python 2.x position has always been that the IO layer is a relatively thin wrapper over the platform behaviour, and so this sort of platform dependent behaviour should not come as a surprise to anyone. For Python 3.x, the IO layer was reworked to be more platform-neutral, and I would expect consistent behaviour across platforms there. And the behaviour I would expect (given that stdout is a buffered text stream) is that output from print would be buffered until an implicit or explicit flush. And that is exactly what happens. So I'd say that you're 100% right in what you say as regards Python 3, and that is exactly what is happening. The difference with Python 2 is precisely one of those backward-incompatible breaks that required the 2->3 major version change. Characterising a carefully thought out change implemented in a way that minimises backward compatibility issues as a "bug", is at best unfair and dismissive of the work that was put into this. Paul. From sergio at gruposinternet.com.br Mon Jan 9 14:05:57 2012 From: sergio at gruposinternet.com.br (=?ISO-8859-1?Q?S=E9rgio?= Surkamp) Date: Mon, 9 Jan 2012 11:05:57 -0200 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: <20120109110557.75e23bc3@icedearth.gruposinternet.com.br> Em Mon, 9 Jan 2012 13:30:55 +0300 anatoly techtonik escreveu: > In Python 2 "print 'something', statement calls were unbuffered and > immediately appeared on screen. Only if Python is called with -u command line option, from the man page: -u Force stdin, stdout and stderr to be totally unbuffered. Regards, -- .:''''':. .:' ` S?rgio Surkamp | Administrador de Redes :: ........ sergio at gruposinternet.com.br `:. .:' `:, ,.:' *Grupos Internet S.A.* `: :' R. Lauro Linhares, 2123 Torre B - Sala 201 : : Trindade - Florian?polis - SC :.' :: +55 48 3234-4109 : ' http://www.gruposinternet.com.br From jeanpierreda at gmail.com Mon Jan 9 14:25:23 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 9 Jan 2012 08:25:23 -0500 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: > This produces one dot every second with Python 2.7.2 on Windows. Weird. Must be a windows thing. > I assume you use Linux. Do you agree that for cross-platform language some > behavior should be made consistent? Yes. I guess that's why they changed the behavior of print on Windows in Python 3. -- Devin On Mon, Jan 9, 2012 at 7:21 AM, anatoly techtonik wrote: > On Mon, Jan 9, 2012 at 1:53 PM, Devin Jeanpierre > wrote: >> >> > In Python 2 "print 'something', statement calls were unbuffered and >> > immediately appeared on screen. >> >> No, they weren't. > > > ?--- cut py2print.py --- > from time import sleep > > while 1: > ? sleep(1) > ? print ".", > --- cut --- > > This produces one dot every second with Python 2.7.2 on Windows. > >> Python doesn't do any extra buffering or flushing. Usually your >> terminal emulator line-buffers stdout. If you don't print a newline >> (in Python 2 OR 3) then it doesn't show up until you either flush >> (sys.stdout.flush()) or send a newline. > > > I assume you use Linux. Do you agree that for cross-platform language some > behavior should be made consistent? From victor.stinner at haypocalc.com Mon Jan 9 15:14:37 2012 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 9 Jan 2012 15:14:37 +0100 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: Read also the discussion on the Python bug tracker: http://bugs.python.org/issue11633 Victor From guido at python.org Mon Jan 9 18:52:21 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Jan 2012 09:52:21 -0800 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: This has nothing to do with print or print(). There are no flush() calls implied by print/print() calls, nor should there be(*). print/print() simply calls sys.stdout.write() one or more times, the final call usually being sys.stdout.write('\n') which will trigger a flush() if the file is line-buffered -- but if you use a trailing comma (Python 2) or end=... (Python 3) the last thing written is not '\n' and no flush() will be triggered. Whether writing \n flushes or not depends on the platform, the Python version and implementation, and the environment (e.g. most platforms automatically turn on line buffering for files known to be associated with a terminal emulator or other interactive device, assuming there's an easy way to tell). __________ (*) There are plenty of apps that construct their output using many print/print() calls, and for which flushing after each call would be quite unnecessary and potentially cause lots of redundant I/O operations, ultimately slowing down the app. Apps that need flushing should call flush(). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Jan 9 20:59:40 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 09 Jan 2012 14:59:40 -0500 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: On 1/9/2012 12:52 PM, Guido van Rossum wrote: > This has nothing to do with print or print(). There are no flush() calls > implied by print/print() calls, nor should there be(*). print/print() > simply calls sys.stdout.write() one or more times, the final call > usually being sys.stdout.write('\n') which will trigger a flush() if the > file is line-buffered -- but if you use a trailing comma (Python 2) or > end=... (Python 3) the last thing written is not '\n' and no flush() > will be triggered. > > Whether writing \n flushes or not depends on the platform, the Python > version and implementation, and the environment (e.g. most platforms > automatically turn on line buffering for files known to be associated > with a terminal emulator or other interactive device, assuming there's > an easy way to tell). > > __________ > (*) There are plenty of apps that construct their output using many > print/print() calls, and for which flushing after each call would be > quite unnecessary and potentially cause lots of redundant I/O > operations, ultimately slowing down the app. Thank you for the clarification. I noted on http://bugs.python.org/issue11633 that there should be no code change > Apps that need flushing should call flush(). and that saying something like this in the print doc entry is the only possible change. -- Terry Jan Reedy From songofacandy at gmail.com Tue Jan 10 06:52:39 2012 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 10 Jan 2012 14:52:39 +0900 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: What about adding flush=False keyword argument to print()? On Tue, Jan 10, 2012 at 4:59 AM, Terry Reedy wrote: > On 1/9/2012 12:52 PM, Guido van Rossum wrote: >> >> This has nothing to do with print or print(). There are no flush() calls >> implied by print/print() calls, nor should there be(*). print/print() >> simply calls sys.stdout.write() one or more times, the final call >> usually being sys.stdout.write('\n') which will trigger a flush() if the >> file is line-buffered -- but if you use a trailing comma (Python 2) or >> end=... (Python 3) the last thing written is not '\n' and no flush() >> will be triggered. >> >> Whether writing \n flushes or not depends on the platform, the Python >> version and implementation, and the environment (e.g. most platforms >> automatically turn on line buffering for files known to be associated >> with a terminal emulator or other interactive device, assuming there's >> an easy way to tell). >> >> __________ >> (*) There are plenty of apps that construct their output using many >> print/print() calls, and for which flushing after each call would be >> quite unnecessary and potentially cause lots of redundant I/O >> operations, ultimately slowing down the app. > > > Thank you for the clarification. I noted on > http://bugs.python.org/issue11633 > that there should be no code change > > >> Apps that need flushing should call flush(). > > > and that saying something like this in the print doc entry is the only > possible change. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- INADA Naoki? From cmjohnson.mailinglist at gmail.com Tue Jan 10 09:49:32 2012 From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson) Date: Mon, 9 Jan 2012 22:49:32 -1000 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: On Jan 9, 2012, at 7:52 PM, INADA Naoki wrote: > What about adding flush=False keyword argument to print()? I would expect flush=False to mean "please don't flush this right now, even if I do use a newline." flush=None? From songofacandy at gmail.com Tue Jan 10 10:05:00 2012 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 10 Jan 2012 18:05:00 +0900 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: On Tue, Jan 10, 2012 at 5:49 PM, Carl M. Johnson wrote: > > On Jan 9, 2012, at 7:52 PM, INADA Naoki wrote: > >> What about adding flush=False keyword argument to print()? > > I would expect flush=False to mean "please don't flush this right now, even if I do use a newline." print() doesn't flushes even if there is newline. So the keyword argument means "do or don't call flush() method of the file." > > flush=None? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- INADA Naoki? From anacrolix at gmail.com Tue Jan 10 10:15:27 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Tue, 10 Jan 2012 20:15:27 +1100 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: I don't think it would be straight forward or reasonable to _prevent_ flushing from occurring. Forcing a flush, is only useful when the file object to which you're printing isn't already handy, and this only occurs for sys.stdout. It's not worth adding a "flush-in-case" flag just for this one stream, especially if it's unable to prevent flushing. On Tue, Jan 10, 2012 at 8:05 PM, INADA Naoki wrote: > On Tue, Jan 10, 2012 at 5:49 PM, Carl M. Johnson > wrote: >> >> On Jan 9, 2012, at 7:52 PM, INADA Naoki wrote: >> >>> What about adding flush=False keyword argument to print()? >> >> I would expect flush=False to mean "please don't flush this right now, even if I do use a newline." > > print() doesn't flushes even if there is newline. So the keyword argument means > "do or don't call flush() method of the file." > >> >> flush=None? >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > > > > -- > INADA Naoki? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From barry at python.org Tue Jan 10 10:34:27 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 10 Jan 2012 10:34:27 +0100 Subject: [Python-ideas] Pythonic buffering in Py3 print() References: Message-ID: <20120110103427.285de748@rivendell> On Jan 09, 2012, at 03:16 PM, anatoly techtonik wrote: >Can you give some examples of these 'other cases'? Maybe 95% of users are >using print for progress bars, and enabling flushing by default will make >it more pythonic by making code more beautiful (which is better than ugly). I guess I'm in that other 5% . -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From masklinn at masklinn.net Tue Jan 10 11:15:58 2012 From: masklinn at masklinn.net (Masklinn) Date: Tue, 10 Jan 2012 11:15:58 +0100 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: On 10 janv. 2012, at 10:05, INADA Naoki wrote: > On Tue, Jan 10, 2012 at 5:49 PM, Carl M. Johnson > wrote: >> >> On Jan 9, 2012, at 7:52 PM, INADA Naoki wrote: >> >>> What about adding flush=False keyword argument to print()? >> >> I would expect flush=False to mean "please don't flush this right now, even if I do use a newline." > > print() doesn't flushes even if there is newline. Ni, but the underlying stream may flush itself on newlines. > So the keyword argument means > "do or don't call flush() method of the file." That's not clear from its name since the flushing behavior can depend on the underlying stream type. force_flush would be closer to the actual meaning of the param. From phd at phdru.name Tue Jan 10 12:22:16 2012 From: phd at phdru.name (Oleg Broytman) Date: Tue, 10 Jan 2012 15:22:16 +0400 Subject: [Python-ideas] Posting style Message-ID: <20120110112216.GA21299@iskra.aviel.ru> Hello! I very much hope I do not embarrass the audience but people can we return back to email etiquette? Inline quoting instead of top-posting, proper trimming instead of overquoting, an empty line to separate quoted text from the reply. Sorry for the noise but can I ask at least? Yes, I am an old fart joined the Net at 1991, but I find netiquette really useful for effective communication and deviations from it inconvenient and annoying. A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ncoghlan at gmail.com Tue Jan 10 12:53:05 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Jan 2012 21:53:05 +1000 Subject: [Python-ideas] Posting style In-Reply-To: <20120110112216.GA21299@iskra.aviel.ru> References: <20120110112216.GA21299@iskra.aviel.ru> Message-ID: On Tue, Jan 10, 2012 at 9:22 PM, Oleg Broytman wrote: > Hello! > > ? I very much hope I do not embarrass the audience but people can we > return back to email etiquette? Inline quoting instead of top-posting, > proper trimming instead of overquoting, an empty line to separate quoted > text from the reply. Unfortunately, you're fighting against the default behaviour of many current email clients. As far as I can tell, they're written on the assumption of conversational email chains amongst small numbers of people rather than potentially sprawling permanently archived mailing list threads with multiple branches and a large number of participants. Even the Gmail web client starts with the cursor above the quoted text (although it places the signature block after it). The big pain though is answering email on a (not-so-)smartphone - I know the Android Gmail client places both cursor and signature above the quoted message, and fixing the formatting is annoying enough that the only two realistic choices are either top posting or not replying at all (and I'll often choose to wait until I'm back at a machine with a full keyboard for exactly that reason). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Tue Jan 10 13:01:14 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 10 Jan 2012 13:01:14 +0100 Subject: [Python-ideas] Posting style References: <20120110112216.GA21299@iskra.aviel.ru> Message-ID: <20120110130114.7cb8d59e@rivendell> On Jan 10, 2012, at 09:53 PM, Nick Coghlan wrote: >Unfortunately, you're fighting against the default behaviour of many >current email clients. We've been fighting damaged email clients for probably as long as there has been email. Yes it's a futile fight despite that mail client authors have known these things for a bazillion years. Oleg's probably not as old or farty as me, but I appreciate him at least reminding people of these conventions. Being aware of them, perhaps we'll all take just a few extra seconds to make sure our posts are more readable for everyone. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From phd at phdru.name Tue Jan 10 13:13:28 2012 From: phd at phdru.name (Oleg Broytman) Date: Tue, 10 Jan 2012 16:13:28 +0400 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> Message-ID: <20120110121328.GB21299@iskra.aviel.ru> On Tue, Jan 10, 2012 at 09:53:05PM +1000, Nick Coghlan wrote: > On Tue, Jan 10, 2012 at 9:22 PM, Oleg Broytman wrote: > > I very much hope I do not embarrass the audience but people can we > > return back to email etiquette? Inline quoting instead of top-posting, > > proper trimming instead of overquoting, an empty line to separate quoted > > text from the reply. > > Unfortunately, you're fighting against the default behaviour of many > current email clients. I do this with the assumption that subscribers to python mailing lists are developers and powerful users who understand what's good and what's bad and can configure their software accordingly. Defaults are for mere mortals. ;-) > The big pain though is answering email on a (not-so-)smartphone I know and I can bear this particular situation. People usually don't participate in long and complex discussions using small screen clients, and short top-posted notes are bearable. Especially if the rest is trimmed. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From jeanpierreda at gmail.com Tue Jan 10 13:15:07 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 10 Jan 2012 07:15:07 -0500 Subject: [Python-ideas] Posting style In-Reply-To: <20120110112216.GA21299@iskra.aviel.ru> References: <20120110112216.GA21299@iskra.aviel.ru> Message-ID: > Sorry for the noise but can I ask at least? Yes, I am an old fart > joined the Net at 1991, but I find netiquette really useful for effective > communication and deviations from it inconvenient and annoying. Is replying like this acceptable? -- Devin On Tue, Jan 10, 2012 at 6:22 AM, Oleg Broytman wrote: > Hello! > > ? I very much hope I do not embarrass the audience but people can we > return back to email etiquette? Inline quoting instead of top-posting, > proper trimming instead of overquoting, an empty line to separate quoted > text from the reply. > > Sorry for the noise but can I ask at least? Yes, I am an old fart > joined the Net at 1991, but I find netiquette really useful for effective > communication and deviations from it inconvenient and annoying. > > A: Because it messes up the order in which people normally read text. > Q: Why is top-posting such a bad thing? > A: Top-posting. > Q: What is the most annoying thing in e-mail? > > Oleg. > -- > ? ? Oleg Broytman ? ? ? ? ? ?http://phdru.name/ ? ? ? ? ? ?phd at phdru.name > ? ? ? ? ? Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From phd at phdru.name Tue Jan 10 13:43:54 2012 From: phd at phdru.name (Oleg Broytman) Date: Tue, 10 Jan 2012 16:43:54 +0400 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> Message-ID: <20120110124354.GC21299@iskra.aviel.ru> On Tue, Jan 10, 2012 at 07:15:07AM -0500, Devin Jeanpierre wrote: > > Sorry for the noise but can I ask at least? Yes, I am an old fart > > joined the Net at 1991, but I find netiquette really useful for effective > > communication and deviations from it inconvenient and annoying. > > Is replying like this acceptable? I am not going to mandate anything (first, I don't have power). Anything is acceptable. But decide for yourself what is convenient *for you*. Imagine you are reading through a big and complex discussion with many threads. Or think about looking for information in the mailing list(s) archive. How much excessive information can you bear? What posing style would be the best in that cases? It is easy to see the dilemma. It's convenient for a poster to do a quick reply - hence top-posting without trimming. But that's inconvenient for readers and especially for archive browsers. So if you are going to be a long-term participator wouldn't it be in the public interests if you spend a few additional seconds properly formatting your replies? It's in your interests too because sooner or later you will return to your own discussions in the archive. That's my humble opinion. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From jeanpierreda at gmail.com Tue Jan 10 14:13:42 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 10 Jan 2012 08:13:42 -0500 Subject: [Python-ideas] Posting style In-Reply-To: <20120110124354.GC21299@iskra.aviel.ru> References: <20120110112216.GA21299@iskra.aviel.ru> <20120110124354.GC21299@iskra.aviel.ru> Message-ID: On Tue, Jan 10, 2012 at 7:43 AM, Oleg Broytman wrote: > On Tue, Jan 10, 2012 at 07:15:07AM -0500, Devin Jeanpierre wrote: >> > ? Sorry for the noise but can I ask at least? Yes, I am an old fart >> > joined the Net at 1991, but I find netiquette really useful for effective >> > communication and deviations from it inconvenient and annoying. >> >> Is replying like this acceptable? > > ? I am not going to mandate anything (first, I don't have power). > Anything is acceptable. But decide for yourself what is convenient *for > you*. Imagine you are reading through a big and complex discussion with > many threads. Or think about looking for information in the mailing > list(s) archive. How much excessive information can you bear? What > posing style would be the best in that cases? I don't really understand the objection. My mail client hides unbroken quotes at the bottom of an email, so that I don't see anything except what you quoted (if I read my own email), plus my signature. I take it yours doesn't do this? -- Devin From techtonik at gmail.com Tue Jan 10 14:16:09 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 10 Jan 2012 16:16:09 +0300 Subject: [Python-ideas] An etherpad for every idea Message-ID: The last thread about "Posting style" makes me think that effective communication with email is impossible at all in modern world, where time pressure doesn't allow to read responses, not speaking about properly formatting them or following guidelines. The problem with email threads that they can be hijacked, polluted with irrelevant info (context garbage), personal opinions etc. If the aim of the thread is to "sell" some idea, it is almost impossible to track when people change their minds. And the biggest problem is that *email threads are stateless*. Meaning that to get the current state of the problem everybody needs to waste a lot of time to reread the thread and filter it. Sometimes it just takes too much time. So I propose to have a summary for each thread in Etherpad - http://en.wikipedia.org/wiki/Etherpad - that will provide current current summary of email thread, describing original problem, its evolution, arguments and counter-arguments. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From arnodel at gmail.com Tue Jan 10 14:20:04 2012 From: arnodel at gmail.com (Arnaud Delobelle) Date: Tue, 10 Jan 2012 13:20:04 +0000 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> <20120110124354.GC21299@iskra.aviel.ru> Message-ID: On 10 January 2012 13:13, Devin Jeanpierre wrote: > On Tue, Jan 10, 2012 at 7:43 AM, Oleg Broytman wrote: >> ? I am not going to mandate anything (first, I don't have power). >> Anything is acceptable. But decide for yourself what is convenient *for >> you*. Imagine you are reading through a big and complex discussion with >> many threads. Or think about looking for information in the mailing >> list(s) archive. How much excessive information can you bear? What >> posing style would be the best in that cases? > > I don't really understand the objection. My mail client hides unbroken > quotes at the bottom of an email, so that I don't see anything except > what you quoted (if I read my own email), plus my signature. I take it > yours doesn't do this? People don't usually look at the archives with their email client. -- Arnaud From techtonik at gmail.com Tue Jan 10 14:27:24 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 10 Jan 2012 16:27:24 +0300 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: I see that Python 3 behavior is consistent on Windows too meaning it doesn't output anything on the screen when end='' It seems that basic function print() that was created to just "print a string" got over-engineered breaking unix principle of "doing one thing". It reminds me of subprocess.Popen which was also designed to just "run a process". I would keep it as simple as possible for printing stuff to the screen without any implicit buffering. You called print() - you get the output. KISS. If people need more control with output buffering - provide a pointer to sys.stdout with all gory details about streams etc. To get back on topic, let me give a definition. To me "pythonic" means "intuitive" in the first place. And "intuitive" means "up to user expectations". What does user expect from "print()" function? Writing to a file? I don't think so. Output to printer? Maybe. Printing to the screen? Definitely. "intuitive" also means that users won't have to consult user documentation for every basic thing it needs, especially for such basic thing such as getting some stuff printed to the screen output - the stuff that is essential in the coding process. It is already less convenient in Py3k, because you need to type extra Shift-9 and Shift-0 in the process, and it risks to become completely weird: print('calling external process... ', end='') sys.output.flush() ... print('done') So, py3k print() is not pythonic. To make it pythonic, the output should be immediately available after you call it. If you need to buffer output - do this explicitly, because print() doesn't guarantee that it will be buffered anyway. If you care about performance - use sys.stdout directly and tune it explicitly. On Mon, Jan 9, 2012 at 4:25 PM, Devin Jeanpierre wrote: > > This produces one dot every second with Python 2.7.2 on Windows. > > Weird. Must be a windows thing. > > > I assume you use Linux. Do you agree that for cross-platform language > some > > behavior should be made consistent? > > Yes. I guess that's why they changed the behavior of print on Windows > in Python 3. > > -- Devin > > On Mon, Jan 9, 2012 at 7:21 AM, anatoly techtonik > wrote: > > On Mon, Jan 9, 2012 at 1:53 PM, Devin Jeanpierre > > > wrote: > >> > >> > In Python 2 "print 'something', statement calls were unbuffered and > >> > immediately appeared on screen. > >> > >> No, they weren't. > > > > > > --- cut py2print.py --- > > from time import sleep > > > > while 1: > > sleep(1) > > print ".", > > --- cut --- > > > > This produces one dot every second with Python 2.7.2 on Windows. > > > >> Python doesn't do any extra buffering or flushing. Usually your > >> terminal emulator line-buffers stdout. If you don't print a newline > >> (in Python 2 OR 3) then it doesn't show up until you either flush > >> (sys.stdout.flush()) or send a newline. > > > > > > I assume you use Linux. Do you agree that for cross-platform language > some > > behavior should be made consistent? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Tue Jan 10 14:30:47 2012 From: phd at phdru.name (Oleg Broytman) Date: Tue, 10 Jan 2012 17:30:47 +0400 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> <20120110124354.GC21299@iskra.aviel.ru> Message-ID: <20120110133047.GD21299@iskra.aviel.ru> On Tue, Jan 10, 2012 at 08:13:42AM -0500, Devin Jeanpierre wrote: > I don't really understand the objection. My mail client hides unbroken > quotes at the bottom of an email, so that I don't see anything except > what you quoted (if I read my own email), plus my signature. I take it > yours doesn't do this? When I do googling through web-archives I use web browser to read what I found. Google, in my experience, produces the most relevant results. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From solipsis at pitrou.net Tue Jan 10 14:31:56 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Jan 2012 14:31:56 +0100 Subject: [Python-ideas] An etherpad for every idea References: Message-ID: <20120110143156.1188db76@pitrou.net> On Tue, 10 Jan 2012 16:16:09 +0300 anatoly techtonik wrote: > > So I propose to have a summary for each thread in Etherpad - > http://en.wikipedia.org/wiki/Etherpad - that will provide current current > summary of email thread, describing original problem, its evolution, > arguments and counter-arguments. Are you willing to maintain it? Feel free to create and update etherpad summaries for a few of our current discussion threads, so that we see if that's useful. Regards Antoine. From anacrolix at gmail.com Tue Jan 10 18:08:49 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 11 Jan 2012 04:08:49 +1100 Subject: [Python-ideas] An etherpad for every idea In-Reply-To: <20120110143156.1188db76@pitrou.net> References: <20120110143156.1188db76@pitrou.net> Message-ID: Are these already in an etherpad clone or...? On Wed, Jan 11, 2012 at 12:31 AM, Antoine Pitrou wrote: > On Tue, 10 Jan 2012 16:16:09 +0300 > anatoly techtonik > wrote: >> >> So I propose to have a summary for each thread in Etherpad - >> http://en.wikipedia.org/wiki/Etherpad - that will provide current current >> summary of email thread, describing original problem, its evolution, >> arguments and counter-arguments. > > Are you willing to maintain it? Feel free to create and update etherpad > summaries for a few of our current discussion threads, so that we see > if that's useful. > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From anacrolix at gmail.com Tue Jan 10 18:12:37 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 11 Jan 2012 04:12:37 +1100 Subject: [Python-ideas] Posting style In-Reply-To: <20120110133047.GD21299@iskra.aviel.ru> References: <20120110112216.GA21299@iskra.aviel.ru> <20120110124354.GC21299@iskra.aviel.ru> <20120110133047.GD21299@iskra.aviel.ru> Message-ID: lol On Wed, Jan 11, 2012 at 12:30 AM, Oleg Broytman wrote: > On Tue, Jan 10, 2012 at 08:13:42AM -0500, Devin Jeanpierre wrote: >> I don't really understand the objection. My mail client hides unbroken >> quotes at the bottom of an email, so that I don't see anything except >> what you quoted (if I read my own email), plus my signature. I take it >> yours doesn't do this? > > ? When I do googling through web-archives I use web browser to read > what I found. Google, in my experience, produces the most relevant > results. > > Oleg. > -- > ? ? Oleg Broytman ? ? ? ? ? ?http://phdru.name/ ? ? ? ? ? ?phd at phdru.name > ? ? ? ? ? Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From ethan at stoneleaf.us Tue Jan 10 18:04:28 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 10 Jan 2012 09:04:28 -0800 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> Message-ID: <4F0C6F9C.50409@stoneleaf.us> Devin Jeanpierre wrote: >> Sorry for the noise but can I ask at least? Yes, I am an old fart >> joined the Net at 1991, but I find netiquette really useful for effective >> communication and deviations from it inconvenient and annoying. > > Is replying like this acceptable? It would be better if you deleted the post following your reply. It's already been read, no need to consume the bandwidth on the wire nor on the screen. :) ~Ethan~ From ubershmekel at gmail.com Tue Jan 10 19:36:04 2012 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Tue, 10 Jan 2012 20:36:04 +0200 Subject: [Python-ideas] Posting style In-Reply-To: <4F0C6F9C.50409@stoneleaf.us> References: <20120110112216.GA21299@iskra.aviel.ru> <4F0C6F9C.50409@stoneleaf.us> Message-ID: On Tue, Jan 10, 2012 at 7:04 PM, Ethan Furman wrote: > Devin Jeanpierre wrote: > >> Is replying like this acceptable? >> >> > It would be better if you deleted the post following your reply. It's > already been read, no need to consume the bandwidth on the wire nor on the > screen. :) > > Gmail collapses that so I would have never even known he kept the post following following his reply. -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Tue Jan 10 20:05:18 2012 From: phd at phdru.name (Oleg Broytman) Date: Tue, 10 Jan 2012 23:05:18 +0400 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> <4F0C6F9C.50409@stoneleaf.us> Message-ID: <20120110190518.GA30986@iskra.aviel.ru> On Tue, Jan 10, 2012 at 08:36:04PM +0200, Yuval Greenfield wrote: > On Tue, Jan 10, 2012 at 7:04 PM, Ethan Furman wrote: > > Devin Jeanpierre wrote: > >> Is replying like this acceptable? > >> > > It would be better if you deleted the post following your reply. It's > > already been read, no need to consume the bandwidth on the wire nor on the > > screen. :) > > > Gmail collapses that so I would have never even known he kept the post > following following his reply. And of course everyone and his brother use GMail? Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From mwm at mired.org Tue Jan 10 20:25:21 2012 From: mwm at mired.org (Mike Meyer) Date: Tue, 10 Jan 2012 11:25:21 -0800 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> Message-ID: <20120110112521.12d7531c@mikmeyer-vm-fedora> On Tue, 10 Jan 2012 21:53:05 +1000 Nick Coghlan wrote: > The big pain though is answering email on a (not-so-)smartphone - I > know the Android Gmail client places both cursor and signature above > the quoted message, and fixing the formatting is annoying enough that > the only two realistic choices are either top posting or not replying > at all (and I'll often choose to wait until I'm back at a machine with > a full keyboard for exactly that reason). The android client K-9 is better behaved. I find it more usable in general than the Gmail client (or Google's mail client, for matter that). To me, the real issue is proper editing more than where the quote winds up. If you're just going to fire off a quick quip, editing the quote makes it a lot more work. If you're actually taking time to think about what you're saying, then editing the quote is a minor bit of work, and may well help with the composition of the message. Basically, top posting says "I didn't think about this, just shot off a reply." Bottom posting beneath an unedited quote says "I'm an old fart who didn't think about this, just shot off a reply." Taking the time to properly edit the quoted text says "I thought about what I'm saying, and took the time to remove the irrelevant." Final comment: I saw a number of "My mail reader hides the quotes" replies. If "get a better mailer" is an acceptable response, then consider it used if your mailer makes top posting the easy option. Message-ID: <87mx9v8g1x.fsf@benfinney.id.au> Nick Coghlan writes: > On Tue, Jan 10, 2012 at 9:22 PM, Oleg Broytman wrote: > > I very much hope I do not embarrass the audience but people can we > > return back to email etiquette? Inline quoting instead of > > top-posting, proper trimming instead of overquoting, an empty line > > to separate quoted text from the reply. > > Unfortunately, you're fighting against the default behaviour of many > current email clients. [?] Even the Gmail web client starts with the > cursor above the quoted text (although it places the signature block > after it). But that's the right behaviour: for deleting unnecessary quoted text and writing inline responses, the natural way to work is from top to bottom; so starting the cursor at the top is correct. (only half-joking :-) > The big pain though is answering email on a (not-so-)smartphone [?] > (and I'll often choose to wait until I'm back at a machine with a full > keyboard for exactly that reason). Yes. If one knows enough to feel the need to apologise for one's email client, then one also knows enough to wait until a better email client is available. No apology necessary. -- \ ?Know what I hate most? Rhetorical questions.? ?Henry N. Camp | `\ | _o__) | Ben Finney From alexander.belopolsky at gmail.com Tue Jan 10 21:56:41 2012 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 10 Jan 2012 15:56:41 -0500 Subject: [Python-ideas] Posting style In-Reply-To: <87mx9v8g1x.fsf@benfinney.id.au> References: <20120110112216.GA21299@iskra.aviel.ru> <87mx9v8g1x.fsf@benfinney.id.au> Message-ID: On Tue, Jan 10, 2012 at 3:21 PM, Ben Finney wrote: >> The big pain though is answering email on a (not-so-)smartphone [?] >> (and I'll often choose to wait until I'm back at a machine with a full >> keyboard for exactly that reason). > > Yes. If one knows enough to feel the need to apologise for one's email > client, then one also knows enough to wait until a better email client > is available. No apology necessary. +1 In good old times your usenet client (nn) would say: This program posts news to thousands of machines throughout the entire civilized world. Your message will cost the net hundreds if not thousands of dollars to send everywhere. Please be sure you know what you are doing. Are you absolutely sure that you want to do this? [y / n] The economic realities of the Internet changed since that program was in wide use, but it is still true that any time you save by getting your reply out sooner is not worth it if it will cost untold hours to the global community for the rest of the life of the list archive. A hint to Gmail users: if you highlight a portion of the message before hitting reply, only that portion will be quoted. From guido at python.org Tue Jan 10 22:36:28 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 10 Jan 2012 13:36:28 -0800 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: On Tue, Jan 10, 2012 at 5:27 AM, anatoly techtonik wrote: > So, py3k print() is not pythonic. To make it pythonic, the output should > be immediately available after you call it. If you need to buffer output - > do this explicitly, because print() doesn't guarantee that it will be > buffered anyway. If you care about performance - use sys.stdout directly > and tune it explicitly. That's utter bullshit. However, I would be fine with adding a new keyword argument to print() in 3.3 to force a flush, as long as it defaults to off. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jxo6948 at rit.edu Wed Jan 11 00:50:55 2012 From: jxo6948 at rit.edu (John O'Connor) Date: Tue, 10 Jan 2012 18:50:55 -0500 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> <87mx9v8g1x.fsf@benfinney.id.au> Message-ID: On Tue, Jan 10, 2012 at 3:56 PM, Alexander Belopolsky wrote: > A hint to Gmail users: if you highlight a portion of the message > before hitting reply, only that portion will be quoted. Great tip! Note that you will also have to enable it from the Gmail Labs. From p.f.moore at gmail.com Wed Jan 11 01:28:44 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 11 Jan 2012 00:28:44 +0000 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> <87mx9v8g1x.fsf@benfinney.id.au> Message-ID: On 10 January 2012 23:50, John O'Connor wrote: > Note that you will also have to enable it from the Gmail Labs. Yes, that's the trick :-) Thnks. Paul From cs at zip.com.au Wed Jan 11 01:40:32 2012 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 11 Jan 2012 11:40:32 +1100 Subject: [Python-ideas] Posting style In-Reply-To: References: Message-ID: <20120111004032.GA30538@cskk.homeip.net> On 10Jan2012 21:53, Nick Coghlan wrote: | On Tue, Jan 10, 2012 at 9:22 PM, Oleg Broytman wrote: | > ? I very much hope I do not embarrass the audience but people can we | > return back to email etiquette? Inline quoting instead of top-posting, | > proper trimming instead of overquoting, an empty line to separate quoted | > text from the reply. | | Unfortunately, you're fighting against the default behaviour of many | current email clients. [...] Even the Gmail web client starts with the | cursor above the quoted text [...] At Google one convention was that this is a cue to start at the top of the quoted text and trim rubbish aggressively, and inline one's replies. Of course that is a workflow mindset chosen to overcome this bad UI choice:-) Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ Would you remember a one-line .sig? - Paul Thompson, thompson at apple.com From cs at zip.com.au Wed Jan 11 01:44:43 2012 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 11 Jan 2012 11:44:43 +1100 Subject: [Python-ideas] Posting style In-Reply-To: <20120110133047.GD21299@iskra.aviel.ru> References: <20120110133047.GD21299@iskra.aviel.ru> Message-ID: <20120111004443.GA31682@cskk.homeip.net> On 10Jan2012 17:30, Oleg Broytman wrote: | On Tue, Jan 10, 2012 at 08:13:42AM -0500, Devin Jeanpierre wrote: | > I don't really understand the objection. My mail client hides unbroken | > quotes at the bottom of an email, so that I don't see anything except | > what you quoted (if I read my own email), plus my signature. I take it | > yours doesn't do this? | | When I do googling through web-archives I use web browser to read | what I found. Google, in my experience, produces the most relevant | results. It is worth mentioning (since I routinely see it missed it these discussions) that leaving huge chunks of untrimmed quotes makes it much harder to find the _relevant_ archived message because of all the irrelevant keyword hits. Quote trimming helps put relevant stuff in the email flow and keeps irrelevant stuff out. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ I swear to god, officer, I'm fixing this bridge. Just go divert traffic. From greg.ewing at canterbury.ac.nz Tue Jan 10 23:31:08 2012 From: greg.ewing at canterbury.ac.nz (Greg) Date: Wed, 11 Jan 2012 11:31:08 +1300 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> Message-ID: <4F0CBC2C.8070901@canterbury.ac.nz> Devin Jeanpierre wrote: >> Sorry for the noise but can I ask at least? Yes, I am an old fart >>joined the Net at 1991, but I find netiquette really useful for effective >>communication and deviations from it inconvenient and annoying. > > > Is replying like this acceptable? > > -- Devin > > [rest of message quoted here] Not entirely, because I have to scroll through the rest of the message anyway if I want to be sure there aren't any further comments in it, despite the embedded signature. -- Greg From wuwei23 at gmail.com Wed Jan 11 02:43:54 2012 From: wuwei23 at gmail.com (alex23) Date: Tue, 10 Jan 2012 17:43:54 -0800 (PST) Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: <95040435-d494-4507-a5c0-446256eff84e@n6g2000vbz.googlegroups.com> On Jan 10, 11:27?pm, anatoly techtonik wrote: > To get back on topic, let me give a definition. To me "pythonic" means > "intuitive" in the first place. And "intuitive" means "up to user > expectations". No, what "pythonic" means here is "up to anatoly's expectations". If I could kill one idea dead about Python it's this persistent belief that you shouldn't need to read any documentation in order to be able to use it. > "intuitive" also means that users won't have to consult user documentation > for every basic thing it needs Well, clearly that's not true, as your intuition about print's behaviour was wrong. Other people's "intuitive" understanding of mutable default arguments gets them into trouble; you can either "fix" this behaviour so it's inconsistent with the rest of Python's object model, or you can recommend they read the docs so their understanding changes and they know what it means. The latter is the only sane method, IMO. Similarly, every time your intuition is wrong doesn't make it a regression or bug. From ncoghlan at gmail.com Wed Jan 11 02:59:02 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 11 Jan 2012 11:59:02 +1000 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: On Tue, Jan 10, 2012 at 8:15 PM, Masklinn wrote: >> So the keyword argument means >> "do or don't call flush() method of the file." > That's not clear from its name since the flushing behavior can depend on the underlying stream type. force_flush would be closer to the actual meaning of the param. +0 for being able to write print("whatever", force_flush=True) instead of having to do: import sys # somewhere in the file print("whatever") sys.stdout.flush() Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From python at mrabarnett.plus.com Wed Jan 11 03:36:23 2012 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 11 Jan 2012 02:36:23 +0000 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: <4F0CF5A7.2080504@mrabarnett.plus.com> On 11/01/2012 01:59, Nick Coghlan wrote: > On Tue, Jan 10, 2012 at 8:15 PM, Masklinn > wrote: >>> So the keyword argument means "do or don't call flush() method of >>> the file." >> That's not clear from its name since the flushing behavior can >> depend on the underlying stream type. force_flush would be closer >> to the actual meaning of the param. > > +0 for being able to write print("whatever", force_flush=True) > instead of having to do: > > import sys # somewhere in the file > print("whatever") > sys.stdout.flush() > Given that 'print' is a function in Python 3, it's easy to redefine it to flush. I've done it on occasion. From guido at python.org Wed Jan 11 04:14:11 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 10 Jan 2012 19:14:11 -0800 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: <4F0CF5A7.2080504@mrabarnett.plus.com> References: <4F0CF5A7.2080504@mrabarnett.plus.com> Message-ID: On Tue, Jan 10, 2012 at 6:36 PM, MRAB wrote: > On 11/01/2012 01:59, Nick Coghlan wrote: > >> On Tue, Jan 10, 2012 at 8:15 PM, Masklinn >> wrote: >> >>> So the keyword argument means "do or don't call flush() method of >>>> the file." >>>> >>> That's not clear from its name since the flushing behavior can >>> depend on the underlying stream type. force_flush would be closer >>> to the actual meaning of the param. >>> >> >> +0 for being able to write print("whatever", force_flush=True) >> instead of having to do: >> >> import sys # somewhere in the file >> > > print("whatever") > >> sys.stdout.flush() >> >> Given that 'print' is a function in Python 3, it's easy to redefine it > to flush. I've done it on occasion. > Heh, that's exactly why we made it a function. So is adding e.g. a force_flush flag -- though personally I would be fine calling it "flush" despite the possibility that the underlying stream might still flush when it is not explicitly requests. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Wed Jan 11 04:47:21 2012 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 11 Jan 2012 03:47:21 +0000 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: <4F0CF5A7.2080504@mrabarnett.plus.com> Message-ID: <4F0D0649.6020400@mrabarnett.plus.com> On 11/01/2012 03:14, Guido van Rossum wrote: > On Tue, Jan 10, 2012 at 6:36 PM, MRAB > wrote: > > On 11/01/2012 01:59, Nick Coghlan wrote: > > On Tue, Jan 10, 2012 at 8:15 PM, Masklinn __> > wrote: > > So the keyword argument means "do or don't call flush() > method of > the file." > > That's not clear from its name since the flushing behavior can > depend on the underlying stream type. force_flush would be > closer > to the actual meaning of the param. > > > +0 for being able to write print("whatever", force_flush=True) > instead of having to do: > > import sys # somewhere in the file > > > print("whatever") > > sys.stdout.flush() > > Given that 'print' is a function in Python 3, it's easy to redefine it > to flush. I've done it on occasion. > > > Heh, that's exactly why we made it a function. So is adding e.g. a > force_flush flag -- though personally I would be fine calling it "flush" > despite the possibility that the underlying stream might still flush > when it is not explicitly requests. > Even a buffered stream flushes sometime. :-) From stephen at xemacs.org Wed Jan 11 06:27:30 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 11 Jan 2012 14:27:30 +0900 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> <20120110124354.GC21299@iskra.aviel.ru> Message-ID: <87hb02ajwd.fsf@uwakimon.sk.tsukuba.ac.jp> Devin Jeanpierre writes: > I don't really understand the objection. My mail client hides unbroken > quotes at the bottom of an email, so that I don't see anything except > what you quoted (if I read my own email), plus my signature. I take it > yours doesn't do this? Mine does not, and I like it that way, because almost all the mail I receive is either very short threads where it doesn't matter, or related to fora where the vast majority of participants follow the standard netiquette conventions of interlinear response and fairly aggressive trimming. It is often the case that the trailing text is useful for establishing context (because if it wasn't the poster would have trimmed it). I believe that the top-posting style works well in business because threads are either short, or almost everybody is almost always on the same page anyway, and occasionally refers to something buried deeper in the thread. Also, in most cases any conflicts will be settled more or less arbitrarily by the boss's intuition, so extended analysis with precise rebuttals is rare. It sucks in technical discussion, because people have interesting things to say (aka, conflicting analyses that go beyond Snappy Answers to Stupid Questions one-liners). From cs at zip.com.au Wed Jan 11 07:15:31 2012 From: cs at zip.com.au (Cameron Simpson) Date: Wed, 11 Jan 2012 17:15:31 +1100 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: <20120111061531.GA2804@cskk.homeip.net> On 09Jan2012 22:49, Carl M. Johnson wrote: | On Jan 9, 2012, at 7:52 PM, INADA Naoki wrote: | > What about adding flush=False keyword argument to print()? | | I would expect flush=False to mean "please don't flush this right now, | even if I do use a newline." | | flush=None? +1 from me, if None means "honour the file's default flushing behaviour". -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ Since I've mentioned the subject of geneology, I'll repeat a story I heard about a poor fellow over on airstrip one. Seems he spent the most recent thirty years of his life tracking down his family history. Spent hundreds of pounds, traveled, devoted his life to it. Then, last month, a cousin told him he was adopted. Ahhh, sweet irony. - Tim_Mefford From g.brandl at gmx.net Wed Jan 11 09:28:25 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 11 Jan 2012 09:28:25 +0100 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: Am 10.01.2012 22:36, schrieb Guido van Rossum: > On Tue, Jan 10, 2012 at 5:27 AM, anatoly techtonik > wrote: > > So, py3k print() is not pythonic. To make it pythonic, the output should be > immediately available after you call it. If you need to buffer output - do > this explicitly, because print() doesn't guarantee that it will be buffered > anyway. If you care about performance - use sys.stdout directly and tune it > explicitly. > > > That's utter bullshit. > > However, I would be fine with adding a new keyword argument to print() in 3.3 to > force a flush, as long as it defaults to off. I've created an issue to track this: http://bugs.python.org/issue13761 Georg From victor.stinner at haypocalc.com Wed Jan 11 09:46:24 2012 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 11 Jan 2012 09:46:24 +0100 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: There is already an option for flush: -u command line option. Can't we change print() to flush if end is set and -u option is used, instead of introducing a new option? I understand that a command line option is less practical than a new keyword, especially on Windows. From techtonik at gmail.com Wed Jan 11 14:55:04 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 11 Jan 2012 16:55:04 +0300 Subject: [Python-ideas] An etherpad for every idea In-Reply-To: <20120110143156.1188db76@pitrou.net> References: <20120110143156.1188db76@pitrou.net> Message-ID: On Tue, Jan 10, 2012 at 4:31 PM, Antoine Pitrou wrote: > On Tue, 10 Jan 2012 16:16:09 +0300 > anatoly techtonik > wrote: > > > > So I propose to have a summary for each thread in Etherpad - > > http://en.wikipedia.org/wiki/Etherpad - that will provide current > current > > summary of email thread, describing original problem, its evolution, > > arguments and counter-arguments. > > Are you willing to maintain it? Feel free to create and update etherpad > summaries for a few of our current discussion threads, so that we see > if that's useful. > The idea is about collaborative service, not a "community job offer" for single person, responsible to maintain and update summaries for email threads on some wiki page. The architecture should be inspiring for people to provide summaries when they start to find threads confusing, need to organize ideas around the threads or keep the discussion focused. -------------- next part -------------- An HTML attachment was scrubbed... URL: From masklinn at masklinn.net Wed Jan 11 15:02:58 2012 From: masklinn at masklinn.net (Masklinn) Date: Wed, 11 Jan 2012 15:02:58 +0100 Subject: [Python-ideas] An etherpad for every idea In-Reply-To: References: <20120110143156.1188db76@pitrou.net> Message-ID: <894BC1E9-5D57-4702-A758-45E29A1F2AF5@masklinn.net> On 2012-01-11, at 14:55 , anatoly techtonik wrote: > On Tue, Jan 10, 2012 at 4:31 PM, Antoine Pitrou wrote: > >> On Tue, 10 Jan 2012 16:16:09 +0300 >> anatoly techtonik >> wrote: >>> >>> So I propose to have a summary for each thread in Etherpad - >>> http://en.wikipedia.org/wiki/Etherpad - that will provide current >> current >>> summary of email thread, describing original problem, its evolution, >>> arguments and counter-arguments. >> >> Are you willing to maintain it? Feel free to create and update etherpad >> summaries for a few of our current discussion threads, so that we see >> if that's useful. >> > > The idea is about collaborative service, not a "community job offer" for > single person, responsible to maintain and update summaries for email > threads on some wiki page. That does not preclude maintaining: 1. Somebody needs to handle the hardware and software stack, check that the load is acceptable, test software updates and deploy them, handle ACLs if any, ? 2. Information is not going to self-organize over the whole platform, people will need to keep track of the various documents being created and provide an entry point for the service (linking to various sub-trees) 3. Open-edition systems (wikis, etherpad instances, ?) are subject to vandalism, spam and edit wars, there needs to be people and tools able to manage both issues, cleanup the crap in the first two cases and potentially temporarily lock documents in the latter. You're the one who brought it up and suggested it would be a valuable tool, the onus is on you to demonstrate it and it makes sense that you'd be the one maintaining it until then. Hence Antoine's question/suggestion. > The architecture should be inspiring for people > to provide summaries when they start to find threads confusing, need to > organize ideas around the threads or keep the discussion focused. Good luck with that. From manuel at enigmage.de Wed Jan 11 16:08:20 2012 From: manuel at enigmage.de (=?ISO-8859-1?Q?Manuel_B=E4renz?=) Date: Wed, 11 Jan 2012 16:08:20 +0100 Subject: [Python-ideas] "while ... try" - block or "for ... try" - block Message-ID: <4F0DA5E4.40308@enigmage.de> I propose two new control flows for Python: "while ... try": while expr try: suite1 except SomeException: suite2 else: suite3 This executes suite1 as long as handled exceptions are thrown and expr is True. * If an unhandled exception is thrown, it passes the exception on to the surrounding or the stack. * If no exception occurs, life goes on as normal, suite3 is executed and execution goes on afterwards. The control flow is thus equivalent to: while expr: try: suite1 except SomeException: suite2 else: suite3 break But it's neater, very natural (in my opinion) and saves an indentation level. One further enhancement: If expr is encountered to be False, some special exception "NoMoreTriesException" could be raised. It can be catched in the same "while ... try" block. Usecase: while network_is_up() try: connect_to_server() except ConnectError: time.sleep(timeout) except NoMoreTriesException: print("Couldn't establish connection") else: download_stuff() finally: make_sure_resource_is_freed() Another usecase: while receive_packet() try: check_packet() except ChecksumError: print("You sent the wrong thing. Try again.") except NoMoreTriesException: print("I couldn't get a single useful packet from you :(") else: process_packet() finally: close_connection() A similar thing could be made with "for ... try": for password in passwords_i_remember try: connect(password) except WrongPassError: pass # No pun intended except NoMoreTriesException: print("Not a single one worked.") else: check_mailbox() The advantages are the same as for "while ... try". Cheers, Manuel From anacrolix at gmail.com Wed Jan 11 16:11:18 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 12 Jan 2012 02:11:18 +1100 Subject: [Python-ideas] "while ... try" - block or "for ... try" - block In-Reply-To: <4F0DA5E4.40308@enigmage.de> References: <4F0DA5E4.40308@enigmage.de> Message-ID: I don't like it, you're merging 2 statements at the expense of readability. It's like egyptian exception handling. -1 2012/1/12 Manuel B?renz : > I propose two new control flows for Python: > > "while ... try": > > while expr try: > ? ?suite1 > except SomeException: > ? ?suite2 > else: > ? ?suite3 > > This executes suite1 as long as handled exceptions are thrown and expr > is True. > * If an unhandled exception is thrown, it passes the exception on to the > surrounding or the stack. > * If no exception occurs, life goes on as normal, suite3 is executed and > execution goes on afterwards. > > The control flow is thus equivalent to: > > while expr: > ? ?try: > ? ? ? ?suite1 > ? ?except SomeException: > ? ? ? ?suite2 > ? ?else: > ? ? ? ?suite3 > ? ? ? ?break > > But it's neater, very natural (in my opinion) and saves an indentation > level. > > One further enhancement: If expr is encountered to be False, some > special exception "NoMoreTriesException" could be raised. It can be > catched in the same "while ... try" block. > > Usecase: > > while network_is_up() try: > ? ?connect_to_server() > except ConnectError: > ? ?time.sleep(timeout) > except NoMoreTriesException: > ? ?print("Couldn't establish connection") > else: > ? ?download_stuff() > finally: > ? ?make_sure_resource_is_freed() > > Another usecase: > > while receive_packet() try: > ? ?check_packet() > except ChecksumError: > ? ?print("You sent the wrong thing. Try again.") > except NoMoreTriesException: > ? ?print("I couldn't get a single useful packet from you :(") > else: > ? ?process_packet() > finally: > ? ?close_connection() > > > A similar thing could be made with "for ... try": > > for password in passwords_i_remember try: > ? ?connect(password) > except WrongPassError: > ? ?pass # No pun intended > except NoMoreTriesException: > ? ?print("Not a single one worked.") > else: > ? ?check_mailbox() > > The advantages are the same as for "while ... try". > > Cheers, Manuel > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From manuel at enigmage.de Wed Jan 11 16:17:31 2012 From: manuel at enigmage.de (=?UTF-8?B?TWFudWVsIELDpHJlbno=?=) Date: Wed, 11 Jan 2012 16:17:31 +0100 Subject: [Python-ideas] "while ... try" - block or "for ... try" - block In-Reply-To: References: <4F0DA5E4.40308@enigmage.de> Message-ID: <4F0DA80B.3060901@enigmage.de> On 11.01.2012 16:11, Matt Joiner wrote: > I don't like it, you're merging 2 statements at the expense of readability. You might find it unreadable because you're not used to it. What is so unreadable about it? I find it more readable because it's flat. > It's like egyptian exception handling. -1 What is that supposed to mean? From techtonik at gmail.com Wed Jan 11 16:45:33 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 11 Jan 2012 18:45:33 +0300 Subject: [Python-ideas] An etherpad for every idea In-Reply-To: References: <20120110143156.1188db76@pitrou.net> Message-ID: Created a page for the current thread: http://piratepad.net/python-ideas-an-etherpad-for-every-idea No clone of Etherpad sources. The best way would be to start with Etherpad Lite http://etherpad.org/2012/01/02/a-short-etherpad-lite-talk/ if we are talking about implementation. -- anatoly t. On Tue, Jan 10, 2012 at 8:08 PM, Matt Joiner wrote: > Are these already in an etherpad clone or...? > > On Wed, Jan 11, 2012 at 12:31 AM, Antoine Pitrou > wrote: > > On Tue, 10 Jan 2012 16:16:09 +0300 > > anatoly techtonik > > wrote: > >> > >> So I propose to have a summary for each thread in Etherpad - > >> http://en.wikipedia.org/wiki/Etherpad - that will provide current > current > >> summary of email thread, describing original problem, its evolution, > >> arguments and counter-arguments. > > > > Are you willing to maintain it? Feel free to create and update etherpad > > summaries for a few of our current discussion threads, so that we see > > if that's useful. > > > > Regards > > > > Antoine. > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > http://mail.python.org/mailman/listinfo/python-ideas > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Wed Jan 11 17:03:34 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Jan 2012 17:03:34 +0100 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: Victor Stinner, 11.01.2012 09:46: > There is already an option for flush: -u command line option. Can't we > change print() to flush if end is set and -u option is used, instead > of introducing a new option? > > I understand that a command line option is less practical than a new > keyword, especially on Windows. For one, it doesn't allow the program code to explicitly enforce the flush at a specific point if it depends on a command line switch. Note that the caller of print() may not even know what "print" really is at a given point in the execution, and what (if any) output stream it will write to. Doesn't have to be sys.stdout. So a new keyword argument makes sense to me (and I agree that calling it "flush" should be enough). Stefan From techtonik at gmail.com Wed Jan 11 17:30:21 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 11 Jan 2012 19:30:21 +0300 Subject: [Python-ideas] An etherpad for every idea In-Reply-To: <894BC1E9-5D57-4702-A758-45E29A1F2AF5@masklinn.net> References: <20120110143156.1188db76@pitrou.net> <894BC1E9-5D57-4702-A758-45E29A1F2AF5@masklinn.net> Message-ID: On Wed, Jan 11, 2012 at 5:02 PM, Masklinn wrote: > On 2012-01-11, at 14:55 , anatoly techtonik wrote: > > On Tue, Jan 10, 2012 at 4:31 PM, Antoine Pitrou > wrote: > > > >> On Tue, 10 Jan 2012 16:16:09 +0300 > >> anatoly techtonik > >> wrote: > >>> > >>> So I propose to have a summary for each thread in Etherpad - > >>> http://en.wikipedia.org/wiki/Etherpad - that will provide current > >> current > >>> summary of email thread, describing original problem, its evolution, > >>> arguments and counter-arguments. > >> > >> Are you willing to maintain it? Feel free to create and update etherpad > >> summaries for a few of our current discussion threads, so that we see > >> if that's useful. > >> > > > > The idea is about collaborative service, not a "community job offer" for > > single person, responsible to maintain and update summaries for email > > threads on some wiki page. > > That does not preclude maintaining: 1. Somebody needs to handle the hardware and software stack, check that the > load is acceptable, test software updates and deploy them, handle ACLs if > any, ? > Does that mean that the idea is good enough to start talking about problems with infrastructure and maintenance? =) > 2. Information is not going to self-organize over the whole platform, > people will need to keep track of the various documents being created and > provide an entry point for the service (linking to various sub-trees) > That's right. The idea is to provide an automatic service tied to each email thread, meaning that email threads must be identified first. Then - you're right - there must be a web interface of the entry point that lists threads with pointers to the corresponding Etherpad page. This can further list auxiliary flags, for example if thread was updated after the last change in Etherpad. Etherpad interface then can be tweaked further to implement "approval" system that let's every user place a "tick" next to the reference to a message that the user believes is integrated into summary correctly (or flag messages that are completely missing). > 3. Open-edition systems (wikis, etherpad instances, ?) are subject to > vandalism, spam and edit wars, there needs to be people and tools able to > manage both issues, cleanup the crap in the first two cases and potentially > temporarily lock documents in the latter. > In this case it is possible to limit access to python.org accounts. Is it possible? Is there a central login server for python.org? > You're the one who brought it up and suggested it would be a valuable > tool, the onus is on you to demonstrate it and it makes sense that you'd be > the one maintaining it until then. > I only proposed the idea. If you think this tool won't be valuable then I am open to discuss why. If it is only because I failed to make it a valuable tool then I can make it even more obvious by saying - yes, I will not be able to create a valuable tool out of it alone. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Wed Jan 11 17:58:40 2012 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 11 Jan 2012 11:58:40 -0500 Subject: [Python-ideas] "while ... try" - block or "for ... try" - block In-Reply-To: <4F0DA5E4.40308@enigmage.de> References: <4F0DA5E4.40308@enigmage.de> Message-ID: For me, control flow matching indentation is probably the single most important feature of python. What I would support is a more convenient way to inject an exception handler into an expression. For example, (a more complicated version of) your first use case would be improved by a way to say (except ConnectError) ==>?(time.sleep(timeout); continue) and your password checker example would be improved by a way to say (except WrongPassError) ==> (pass) More details below. 2012/1/11 Manuel B?renz : > while expr try: > ? ?suite1 > except SomeException: > ? ?suite2 > else: > ? ?suite3 Without your explanation, I would still have assumed that the exception terminated the while. > while network_is_up() try: > ? ?connect_to_server() > except ConnectError: > ? ?time.sleep(timeout) > except NoMoreTriesException: > ? ?print("Couldn't establish connection") > else: > ? ?download_stuff() > finally: > ? ?make_sure_resource_is_freed() This would be a lot easier for me to follow with the extra indents. As best I can tell, under normal circumstances, that will just keep making new connections, and never get to the download until the network itself goes down. Or does the else bind to the try, rather than the while -- in which case it is still an infinite loop, but at least succeeds. I'm also not sure which resource needs to be freed, so I'll just call it special_resource. For what I think you wanted, I can read it a lot easier as: with special_resource: for keep_trying in range(3): # not infinity if not network_is_up(): # I would prefer this elsewhere ... time.sleep(timeout) continue try connect_to_server(): download_stuff() break except ConnectError: time.sleep(timeout) else: print("Couldn't establish connection") which I would in turn prefer to reduce to: retry=WaitPolicy('sleep_and_retry', 3) with ensure_network(onfail=retry) as network: with connect_to_server(transport=network, onfail=retry): download_stuff() > while receive_packet() try: > ? ?check_packet() > except ChecksumError: > ? ?print("You sent the wrong thing. Try again.") > except NoMoreTriesException: > ? ?print("I couldn't get a single useful packet from you :(") > else: > ? ?process_packet() > finally: > ? ?close_connection() This time, it looks like only the final packet gets processed. If else binds to the try, it makes more sense, but still complains about not getting a single useful packet even after getting and processing a dozen. I would prefer: with connection: while receive_packet(): if OK == check_packet(): process_packet() else: print("You sent the wrong thing. Try again.") or at least (following your API more closely): with connection: got_any = False while receive_packet(): try check_packet(): process_packet() got_any = True except ChecksumError: print("You sent the wrong thing. Try again.") if not got_any: print("I couldn't get a single useful packet from you :(") > A similar thing could be made with "for ... try": > > for password in passwords_i_remember try: > ? ?connect(password) > except WrongPassError: > ? ?pass # No pun intended > except NoMoreTriesException: > ? ?print("Not a single one worked.") > else: > ? ?check_mailbox() I honestly don't see that as an improvement over for password in passwords_i_remember: try?connect(password): ? ? check_mailbox() break except WrongPassError: ? ? pass # No pun intended else: ? ?print("Not a single one worked.") From arnodel at gmail.com Wed Jan 11 18:06:58 2012 From: arnodel at gmail.com (Arnaud Delobelle) Date: Wed, 11 Jan 2012 17:06:58 +0000 Subject: [Python-ideas] "while ... try" - block or "for ... try" - block In-Reply-To: <4F0DA5E4.40308@enigmage.de> References: <4F0DA5E4.40308@enigmage.de> Message-ID: 2012/1/11 Manuel B?renz : > I propose two new control flows for Python: > > "while ... try": > > while expr try: > ? ?suite1 > except SomeException: > ? ?suite2 > else: > ? ?suite3 > > This executes suite1 as long as handled exceptions are thrown and expr > is True. > * If an unhandled exception is thrown, it passes the exception on to the > surrounding or the stack. > * If no exception occurs, life goes on as normal, suite3 is executed and > execution goes on afterwards. > > The control flow is thus equivalent to: > > while expr: > ? ?try: > ? ? ? ?suite1 > ? ?except SomeException: > ? ? ? ?suite2 > ? ?else: > ? ? ? ?suite3 > ? ? ? ?break Aside from anything else, there is the problem that while: suite else: suite is already valid Python Your proposed syntax changes the semantics of "else" after "while" (and the same applies to "for ... else". -- Arnaud From jimjjewett at gmail.com Wed Jan 11 18:21:05 2012 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 11 Jan 2012 12:21:05 -0500 Subject: [Python-ideas] An etherpad for every idea In-Reply-To: References: <20120110143156.1188db76@pitrou.net> <894BC1E9-5D57-4702-A758-45E29A1F2AF5@masklinn.net> Message-ID: Quick Summary: There used to be python-dev summaries. They were great, but too much work. >> >> On Tue, 10 Jan 2012 16:16:09 +0300 >> >> anatoly techtonik wrote: >> >>> So I propose to have a summary for each thread in Etherpad - >> >>> http://en.wikipedia.org/wiki/Etherpad - that will provide current >> >>> current summary of email thread, describing original problem, >> >>> its evolution, arguments and counter-arguments. >> > On Tue, Jan 10, 2012 at 4:31 PM, Antoine Pitrou >> > wrote: >> >> Are you willing to maintain it? Feel free to create and update >> >> etherpad summaries for a few of our current discussion threads, >> >> so that we see if that's useful. On Wed, Jan 11, 2012 at 11:30 AM, anatoly techtonik wrote: > I only proposed the idea. If you think this tool won't be?valuable?then > I am open to discuss why. (1) Going to a separate website -- let alone one not at python.org -- is a pain. (2) Creating and maintaining the summaries is a pain. For individual discussions, someone will occasionally post a summary, or even write up a PEP. For the discussions where that doesn't happen, perhaps no one is both sufficiently overwhelmed to need a summary and sufficiently motivated to provide one. Python-ideas was spun off from python-dev. For the python-dev mailing list as a whole, various people have provided biweekly summaries in the past. The summaries were quite helpful. Alas, they were almost never complete, and they all soon slipped into "late" and then got discontinued, because they were too much work to create. Why won't the same thing happen again? Is there something about the Etherpad platform that makes the job much easier? -jJ From ethan at stoneleaf.us Wed Jan 11 17:47:49 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 11 Jan 2012 08:47:49 -0800 Subject: [Python-ideas] "while ... try" - block or "for ... try" - block In-Reply-To: References: <4F0DA5E4.40308@enigmage.de> Message-ID: <4F0DBD35.2000309@stoneleaf.us> Matt Joiner wrote: [snip] Please trim your replies. ~Ethan~ From ctb at msu.edu Wed Jan 11 18:43:53 2012 From: ctb at msu.edu (C. Titus Brown) Date: Wed, 11 Jan 2012 09:43:53 -0800 Subject: [Python-ideas] Netiquette issues (was Re: "while ... try" - block or "for ... try" - block) In-Reply-To: <4F0DBD35.2000309@stoneleaf.us> References: <4F0DA5E4.40308@enigmage.de> <4F0DBD35.2000309@stoneleaf.us> Message-ID: <20120111174353.GC13614@idyll.org> Folks, I'm a moderator. Volunteer, true, but a moderator nonetheless. And I've had enough. So: Netiquette discussions are always fascinating, but please, let's stop talking about how other people should post. Most of my e-mail from this list over the last few days has been from people on this list nattering about how gmail foo and mailer blah, and it is OFF TOPIC and NO LONGER WELCOME. (Frankly, you should have figured this out for yourselves.) If you feel absolutely compelled to tell people that they are lousy netizens and/or correct them on their posting style, please do so PRIVATELY and not on this list. Please e-mail me PRIVATELY if you disagree with this. Brett Cannon is the other list admin, and you should feel free to tell him how much you dislike me & my overbearing moderator ways. Further e-mails on netiquette will result in those people no longer being able to post without moderation. thanks, --titus p.s. This is not aimed especially at Ethan -- it was just the latest e-mail that pushed me over the edge :) -- C. Titus Brown, ctb at msu.edu From ubershmekel at gmail.com Wed Jan 11 21:00:38 2012 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Wed, 11 Jan 2012 22:00:38 +0200 Subject: [Python-ideas] An etherpad for every idea In-Reply-To: References: <20120110143156.1188db76@pitrou.net> <894BC1E9-5D57-4702-A758-45E29A1F2AF5@masklinn.net> Message-ID: E.g. for this discussion: http://openetherpad.org/JlL56IJFYo The farthest this idea can go in my opinion is: 1. Someone makes a bot that auto posts the etherpads to each idea in python-ideas. or 2. Whoever cares enough about an idea will make an etherpad for it if he/she likes to. I don't mind experimenting though. Yuval Greenfield -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Thu Jan 12 00:02:51 2012 From: cs at zip.com.au (Cameron Simpson) Date: Thu, 12 Jan 2012 10:02:51 +1100 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: <20120111230251.GA23059@cskk.homeip.net> [ Taken back on list - I can't see anyway personal here, and the objection is a good one. - Cameron ] On 11Jan2012 08:41, Paul Moore wrote: | On 11 January 2012 06:15, Cameron Simpson wrote: | > | flush=None? | > | > +1 from me, if None means "honour the file's default flushing | > behaviour". | | The trouble with this is that we then have: | - no parameter (default None) => default behaviour | - flush=True => Always flush | | and that begs the question of what an explicit flush=False means, and | people will assume it means "never flush" even thought you can't force | that as the underlying stream might auto-flush (e.g., at a newline). | | +1 for flush=True/False, with a default of False (meaning do the | stream's default behaviour). Hmm. Good point about False suggesting "never flush", but I think it should just be considered invalid. What about flush=3, or flush=tuple? Better if we only specify behaviour for True (flush now) and None/default/missing (flush default) and make it slightly clear that other values do not have defined behaviour. That leaves scope later for defining a sane (or at least clear) meaning for False if a _useful_ one is ever agreed upon. Call it "reserved". Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ Copyright and Patents: To prevent the Progress of Science and useful Arts, by securing for unlimited Times to Authors and Inventors and Trolls the exclusive Right to all Writings and Discoveries. - http://science.slashdot.org/story/11/04/26/211258/Copyright-Law-Is-Killing-Science From guido at python.org Thu Jan 12 00:33:33 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 11 Jan 2012 15:33:33 -0800 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: <20120111230251.GA23059@cskk.homeip.net> References: <20120111230251.GA23059@cskk.homeip.net> Message-ID: See what I added to the tracker. On Wed, Jan 11, 2012 at 3:02 PM, Cameron Simpson wrote: > [ Taken back on list - I can't see anyway personal here, and the > objection is a good one. - Cameron > ] > > On 11Jan2012 08:41, Paul Moore wrote: > | On 11 January 2012 06:15, Cameron Simpson wrote: > | > | flush=None? > | > > | > +1 from me, if None means "honour the file's default flushing > | > behaviour". > | > | The trouble with this is that we then have: > | - no parameter (default None) => default behaviour > | - flush=True => Always flush > | > | and that begs the question of what an explicit flush=False means, and > | people will assume it means "never flush" even thought you can't force > | that as the underlying stream might auto-flush (e.g., at a newline). > | > | +1 for flush=True/False, with a default of False (meaning do the > | stream's default behaviour). > > Hmm. Good point about False suggesting "never flush", but I think it should > just be considered invalid. What about flush=3, or flush=tuple? > > Better if we only specify behaviour for True (flush now) and > None/default/missing (flush default) and make it slightly clear that > other values do not have defined behaviour. That leaves scope later > for defining a sane (or at least clear) meaning for False if a _useful_ > one is ever agreed upon. Call it "reserved". > > Cheers, > -- > Cameron Simpson DoD#743 > http://www.cskk.ezoshosting.com/cs/ > > Copyright and Patents: To prevent the Progress of Science and useful Arts, > by > securing for unlimited Times to Authors and Inventors and Trolls the > exclusive Right to all Writings and Discoveries. > - > http://science.slashdot.org/story/11/04/26/211258/Copyright-Law-Is-Killing-Science > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Jan 12 01:00:20 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 12 Jan 2012 11:00:20 +1100 Subject: [Python-ideas] "while ... try" - block or "for ... try" - block In-Reply-To: <4F0DA5E4.40308@enigmage.de> References: <4F0DA5E4.40308@enigmage.de> Message-ID: <4F0E2294.40806@pearwood.info> Manuel B?renz wrote: > I propose two new control flows for Python: > > "while ... try": > > while expr try: > suite1 > except SomeException: > suite2 > else: > suite3 > > This executes suite1 as long as handled exceptions are thrown and expr > is True. This conflates two naturally distinct operations, looping and exception handling, into a single construct that tries to do both. "Loop" is a natural operation. "Catch exceptions" is a natural operation. "Loop and catch exceptions" is not, it is two operations and so should be written as two operations. It isn't obvious that handled exceptions remain in the loop. Because there are two equally good behaviours -- handled exceptions remain in the loop, or handled exceptions exit the loop -- people will get confused by the construct and repeatedly be surprised it doesn't do what they expect. Your proposed syntax doesn't allow the user to write code which couldn't be written before, it adds no new power or expressiveness to Python. The sole advantage is that it saves one line and one indent level, which is a trivial advantage: there are no shortages of either newlines or indents, and if anyone is writing such deeply nested code that they are regularly worried about indents, their code almost certainly is in desperate need of refactoring. Disadvantages include that it increases complexity of the language: the compiler becomes more complex, there is another feature for people to learn. Every time somebody writes a loop with a try, they will have to decide whether to write it the old way or the new way. The try clause can be easily missed while skimming code, making the construct inelegant. The syntax also clashes with existing while...else. What about nested try blocks? If we introduce this, will people try writing this? while expr try try: ... except InnerException: ... ... except OuterException: ... This seems like a reasonable thing to do. I know that's exactly what I'd try. It also raises one special case above all other cases. An arbitrary loop containing a try block looks like this: while expr: preamble # 0 or more lines before the try block try block postscript # 0 or more lines after the try block Your proposed syntax only covers the case where both the preamble and the postscript are 0 lines. What is so special about that case that it needs extra syntax? It's not that common. Existing syntax is consistent, natural and obvious. Your proposal looks like you've picked two block constructs, a while loop and a try block, and nailed them together. -1 on this suggestion. [...] > One further enhancement: If expr is encountered to be False, some > special exception "NoMoreTriesException" could be raised. It can be > catched in the same "while ... try" block. That use-case is already handled in existing Python by the while...else form. There is no need to add an additional built-in exception for this. -- Steven From ubershmekel at gmail.com Thu Jan 12 01:13:09 2012 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Thu, 12 Jan 2012 02:13:09 +0200 Subject: [Python-ideas] "while ... try" - block or "for ... try" - block In-Reply-To: <4F0E2294.40806@pearwood.info> References: <4F0DA5E4.40308@enigmage.de> <4F0E2294.40806@pearwood.info> Message-ID: I also only see trouble with this construct. I initially assumed you meant: while expr try: # ... except IOError: print 'error' would actually equal: try: while expr: # ... except IOError: print 'error' This makes sense because the "except" is after the while loop suite so it should mean that being in the except suite removes the possibility of reentering the while suite. Anyhow, because of the ambiguity, and because we'll immediately need for/try if/try and try comprehensions (some of which have contradicting "else" semantics) I'm -2 on this. Yuval Greenfield -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Jan 12 01:15:03 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 11 Jan 2012 19:15:03 -0500 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: <20120111230251.GA23059@cskk.homeip.net> References: <20120111230251.GA23059@cskk.homeip.net> Message-ID: On 1/11/2012 6:02 PM, Cameron Simpson wrote: > Better if we only specify behaviour for True (flush now) and > None/default/missing (flush default) and make it slightly clear that > other values do not have defined behaviour. That was the intent of my suggested doc rewording -- file controls flushing unless flush=True is given. What I wrote on the issue was a first try and I suspect that there are other and perhaps better concise wordings. Post one on the issue if you have one. (But writing in 'doc style' is not always easy ;-). -- Terry Jan Reedy From cs at zip.com.au Thu Jan 12 02:40:17 2012 From: cs at zip.com.au (Cameron Simpson) Date: Thu, 12 Jan 2012 12:40:17 +1100 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: Message-ID: <20120112014017.GA14688@cskk.homeip.net> On 11Jan2012 19:15, Terry Reedy wrote: | On 1/11/2012 6:02 PM, Cameron Simpson wrote: | >Better if we only specify behaviour for True (flush now) and | >None/default/missing (flush default) and make it slightly clear that | >other values do not have defined behaviour. | | That was the intent of my suggested doc rewording -- file controls | flushing unless flush=True is given. What I wrote on the issue was a | first try and I suspect that there are other and perhaps better | concise wordings. Post one on the issue if you have one. (But | writing in 'doc style' is not always easy ;-). Do I comment here, or in #11633 (which is closed) or in #13761? We seem to have multiple streams of discussion. BTW, regarding #11633, I don't like your sentence: Use ``sys.stdout.flush()`` to ensure immediate appearance on a screen Shouldn't that be "*file*.flush()" ? I would _not_ want to give a shallow reader the idea that flushing stdout magicly sorts their problems with a specific file. And I've seen that level of logical gap in various questions on python-list (and elsewhere, of course). I'd also change "ensure immediate appearance on a screen" to "ensure immediate output, for example to ensure appearance on a screen". Again, it changes the wording from "to perform this specific special case call flush" to "calling flush does blah, for example to aid this specific case". Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ When in doubt, gas it. It may not solve the problem, but it ends the suspense. - Steve Moonitz (92 Ducati 900ss) From wuwei23 at gmail.com Thu Jan 12 02:44:59 2012 From: wuwei23 at gmail.com (alex23) Date: Wed, 11 Jan 2012 17:44:59 -0800 (PST) Subject: [Python-ideas] An etherpad for every idea In-Reply-To: References: <20120110143156.1188db76@pitrou.net> <894BC1E9-5D57-4702-A758-45E29A1F2AF5@masklinn.net> Message-ID: <70c88004-a082-4f59-805d-1713c7c92728@gj9g2000pbc.googlegroups.com> On Jan 12, 6:00?am, Yuval Greenfield wrote: > 1. Someone makes a bot that auto posts the etherpads to each idea in > python-ideas. > or > 2. Whoever cares enough about an idea will make an etherpad for it if > he/she likes to. And _both_ options split attention between the list and the etherpad. Which is primary? What happens if someone chooses to participate only on one and not the other? What happens in 6 months when someone else decides that whatever technology du jour they find is cool should be supported as well? From wuwei23 at gmail.com Thu Jan 12 03:19:38 2012 From: wuwei23 at gmail.com (alex23) Date: Wed, 11 Jan 2012 18:19:38 -0800 (PST) Subject: [Python-ideas] "while ... try" - block or "for ... try" - block In-Reply-To: <4F0DA5E4.40308@enigmage.de> References: <4F0DA5E4.40308@enigmage.de> Message-ID: <240e47e1-e2cf-4bd0-a4f8-f72d94ca8cba@g4g2000pbi.googlegroups.com> On Jan 12, 1:08?am, Manuel B?renz wrote: > I propose two new control flows for Python: Guido recently said: "There seems to be a trend in proposing more random variants of the existing compound statements. Please don't do this. The existing complement of compound statements is quite sufficient and the new proposals do nothing but complicate the parser, the documentation, and the learning process for occasional users." https://groups.google.com/group/python-ideas/msg/5b884fd1caef2f74 From tjreedy at udel.edu Thu Jan 12 04:03:49 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 11 Jan 2012 22:03:49 -0500 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: <20120112014017.GA14688@cskk.homeip.net> References: <20120112014017.GA14688@cskk.homeip.net> Message-ID: On 1/11/2012 8:40 PM, Cameron Simpson wrote: > On 11Jan2012 19:15, Terry Reedy wrote: > | On 1/11/2012 6:02 PM, Cameron Simpson wrote: > |>Better if we only specify behaviour for True (flush now) and > |>None/default/missing (flush default) and make it slightly clear that > |>other values do not have defined behaviour. > | > | That was the intent of my suggested doc rewording -- file controls > | flushing unless flush=True is given. What I wrote on the issue was a > | first try and I suspect that there are other and perhaps better > | concise wordings. Post one on the issue if you have one. (But > | writing in 'doc style' is not always easy ;-). > > Do I comment here, or in #11633 (which is closed) After much discussion that included the possibility of a behavior change, that issue became about changing the print doc, especially for 3.2. > or in #13761? After Guido's approval of leaving the default behavior as is but adding a parameter to change it, that was opened for the behavior change for 3.3. That change will require further doc change for 3.3. > BTW, regarding #11633, I don't like your sentence: > > Use ``sys.stdout.flush()`` to ensure immediate appearance on a screen > > Shouldn't that be "*file*.flush()" ? That was my original idea but I changed it after internal debate. But I was at most 60-40 for what I did. > I would _not_ want to give a > shallow reader the idea that flushing stdout magicly sorts their > problems with a specific file. And I've seen that level of logical gap > in various questions on python-list (and elsewhere, of course). > > I'd also change "ensure immediate appearance on a screen" to "ensure > immediate output, for example to ensure appearance on a screen". > Again, it changes the wording from "to perform this specific special > case call flush" to "calling flush does blah, for example to aid this > specific case". If one of the doc people agree with you enough to refine the effectively temporary change for 3.2, fine, but I do not think it worth the effort. I suggest you focus on the doc further change for 3.3 after the behavior change. That will potentially be in the docs for several future versions. For that, post a post-change suggestion to #13761. -- Terry Jan Reedy From eliben at gmail.com Thu Jan 12 08:50:55 2012 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 12 Jan 2012 09:50:55 +0200 Subject: [Python-ideas] "while ... try" - block or "for ... try" - block In-Reply-To: <4F0DA5E4.40308@enigmage.de> References: <4F0DA5E4.40308@enigmage.de> Message-ID: 2012/1/11 Manuel B?renz > I propose two new control flows for Python: > > "while ... try": > > while expr try: > suite1 > except SomeException: > suite2 > else: > suite3 > Manuel, Thanks for the improvement idea, but I must admit I'm -1 on this. Having first looked at the proposed syntax without your explanation, I found it hard to be sure what it does, and this goes against the spirit of Python. The compound statement it purports to replace is just one indent deeper and much clearer. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From techtonik at gmail.com Thu Jan 12 11:04:58 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 12 Jan 2012 13:04:58 +0300 Subject: [Python-ideas] Posting style In-Reply-To: References: <20120110112216.GA21299@iskra.aviel.ru> <20120110124354.GC21299@iskra.aviel.ru> Message-ID: On Tue, Jan 10, 2012 at 4:20 PM, Arnaud Delobelle wrote: > On 10 January 2012 13:13, Devin Jeanpierre wrote: > > On Tue, Jan 10, 2012 at 7:43 AM, Oleg Broytman wrote: > >> I am not going to mandate anything (first, I don't have power). > >> Anything is acceptable. But decide for yourself what is convenient *for > >> you*. Imagine you are reading through a big and complex discussion with > >> many threads. Or think about looking for information in the mailing > >> list(s) archive. How much excessive information can you bear? What > >> posing style would be the best in that cases? > > > > I don't really understand the objection. My mail client hides unbroken > > quotes at the bottom of an email, so that I don't see anything except > > what you quoted (if I read my own email), plus my signature. I take it > > yours doesn't do this? > > People don't usually look at the archives with their email client. I agree that Pipermail (Mailman's default interface) suxx for browsing threads and archives, but Google Groups are able to handle that pretty well https://groups.google.com/forum/#!forum/python-ideas -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From manuel at enigmage.de Thu Jan 12 11:51:24 2012 From: manuel at enigmage.de (=?ISO-8859-1?Q?Manuel_B=E4renz?=) Date: Thu, 12 Jan 2012 11:51:24 +0100 Subject: [Python-ideas] "while ... try" - block or "for ... try" - block In-Reply-To: References: <4F0DA5E4.40308@enigmage.de> Message-ID: <4F0EBB2C.9020609@enigmage.de> On 11.01.2012 17:58, Jim Jewett wrote: > 2012/1/11 Manuel B?renz : > >> while expr try: >> suite1 >> except SomeException: >> suite2 >> else: >> suite3 > Without your explanation, I would still have assumed that the > exception terminated the while. > #Steve's comment: > people will get confused by the construct > and repeatedly be surprised it doesn't do what they expect. Ok, you convinced me that this is not intuitive for most people. Normally, one expects the try block to work and not to fail. So continuing the loop on failure and not on success is counterintuitive. > retry=WaitPolicy('sleep_and_retry', 3) > > with ensure_network(onfail=retry) as network: > with connect_to_server(transport=network, onfail=retry): > download_stuff() That's a nice and Pythonic suggestion. Thanks. On 12.01.2012 01:00, Steven D'Aprano wrote: > This conflates two naturally distinct operations, looping and exception > handling, into a single construct that tries to do both. "Loop" is a natural > operation. "Catch exceptions" is a natural operation. "Loop and catch > exceptions" is not, it is two operations and so should be written as two > operations. In these words, I wanted to point out that "Loop until it works" is a natural operation and should have a Python control flow. > The syntax also clashes with existing while...else. Not syntactically: >>> while True try: File "", line 1 while True try: ^ SyntaxError: invalid syntax The interpreter could easily distinguish the two, since try is a reserved keyword.. > The try clause can be easily missed > while skimming code, Very good point. > Your proposal looks like you've picked two block constructs, a while > loop and a try block, and nailed them together. Sorry, I'll have to defend against that a bit. This idea really came from a (simple) usecase. I did a performance test of the functools.lru_cache decorator against a naive self-written memoize class and found me writing: try: result = self.cache[n] except KeyError: result = self.cache[n] = self.f(n) return result I would have wanted to write something like: while True try: return self.cache[n] except KeyError: self.cache[n] = self.f(n) That's how I came up with the idea. But you presented enough reasonable arguments against it such that I won't consider it anymore. Thanks for your comments. From julien at tayon.net Thu Jan 12 14:05:44 2012 From: julien at tayon.net (julien tayon) Date: Thu, 12 Jan 2012 14:05:44 +0100 Subject: [Python-ideas] An etherpad for every idea In-Reply-To: <70c88004-a082-4f59-805d-1713c7c92728@gj9g2000pbc.googlegroups.com> References: <20120110143156.1188db76@pitrou.net> <894BC1E9-5D57-4702-A758-45E29A1F2AF5@masklinn.net> <70c88004-a082-4f59-805d-1713c7c92728@gj9g2000pbc.googlegroups.com> Message-ID: Hello 2012/1/12 alex23 : > On Jan 12, 6:00?am, Yuval Greenfield wrote: >> 1. Someone makes a bot that auto posts the etherpads to each idea in >> python-ideas. >> or >> 2. Whoever cares enough about an idea will make an etherpad for it if >> he/she likes to. > > And _both_ options split attention between the list and the etherpad. > Which is primary? What happens if someone chooses to participate only > on one and not the other? > > What happens in 6 months when someone else decides that whatever > technology du jour they find is cool should be supported as well? Well, I may be dumb, but ideas seems like non structured draft of PEP. (like IETF RFC drafts), and by looking at PEP 1, I read that it describes a very workflow on how to do things. Maybe python-ideas should only accept ideas in the form a link to a PEP draft and people would have to maintain there draft as long as it is not accepted. http://www.python.org/dev/peps/pep-0001/ So why duplicates already existing procedure and infrastructure ? Maybe amending PEP 1 could be a good way to achieve the aformentioned goal. By not adding a PEP draft that is not accepted in a documentation database, we achieve one of the very first goal of a documentation database which is lowering the Noise Signal Ratio. My .02 cents, -- Julien From fuzzyman at gmail.com Thu Jan 12 16:02:52 2012 From: fuzzyman at gmail.com (Michael Foord) Date: Thu, 12 Jan 2012 15:02:52 +0000 Subject: [Python-ideas] os.listdir default to '.' Message-ID: Hello all, I frequently use Python from the interactive interpreter as a shell. It is a frequent, but mild, irritant that I have to type os.listdir('.') instead of just os.listdir(). How about making '.' a default argument for os.listdir ? All the best, Michael -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Thu Jan 12 16:14:46 2012 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 12 Jan 2012 15:14:46 +0000 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: References: Message-ID: <4F0EF8E6.6030002@btinternet.com> In Python 2.5 you can at least say os.listdir('') which had the same effect. In Python 2.7 you can't. Strictly speaking I think the default value should be os.path.curdir, not specifically '.' +1 from me +1 on allowing the empty string argument again (may avoid breaking some old code) Rob Cliffe On 12/01/2012 15:02, Michael Foord wrote: > Hello all, > > I frequently use Python from the interactive interpreter as a shell. > It is a frequent, but mild, irritant that I have to type > os.listdir('.') instead of just os.listdir(). > > How about making '.' a default argument for os.listdir ? > > All the best, > > Michael > > -- > http://www.voidspace.org.uk/ > > May you do good and not evil > May you find forgiveness for yourself and forgive others > > May you share freely, never taking more than you give. > -- the sqlite blessinghttp://www.sqlite.org/different.html > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From masklinn at masklinn.net Thu Jan 12 16:22:42 2012 From: masklinn at masklinn.net (Masklinn) Date: Thu, 12 Jan 2012 16:22:42 +0100 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: References: Message-ID: <82AB40F9-9C34-460F-A7CD-9E30BB3BF0C4@masklinn.net> On 2012-01-12, at 16:02 , Michael Foord wrote: > Hello all, > > I frequently use Python from the interactive interpreter as a shell. It is > a frequent, but mild, irritant that I have to type os.listdir('.') instead > of just os.listdir(). > > How about making '.' a default argument for os.listdir ? I was going to say I don't think '.' is a good idea for `os.listdir`, but then? > ls Doc LICENSE Makefile Misc PC Python build configure pybuilddir.txt python.exe Grammar Lib Makefile.pre Modules PCbuild README config.log configure.in pyconfig.h python.exe-gdb.py Include Mac Makefile.pre.in Objects Parser Tools config.status install-sh pyconfig.h.in setup.py > python3.2 Python 3.2.2 (default, Nov 7 2011, 20:32:23) [GCC 4.2.1 (Apple Inc. build 5664)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.listdir() ['.bzrignore', '.gitignore', '.hg', '.hgcheck', '.hgeol', '.hgignore', '.hgtags', 'build', 'config.log', 'config.status', 'configure', 'configure.in', 'Doc', 'Grammar', 'Include', 'install-sh', 'Lib', 'LICENSE', 'Mac', 'Makefile', 'Makefile.pre', 'Makefile.pre.in', 'Misc', 'Modules', 'Objects', 'Parser', 'PC', 'PCbuild', 'pybuilddir.txt', 'pyconfig.h', 'pyconfig.h.in', 'Python', 'python.exe', 'python.exe-gdb.py', 'README', 'setup.py', 'Tools'] looks like '.' is already the default argument for os.listdir in Python 3.2 (but not in 3.1, no idea about 3.3) From massimo.dipierro at gmail.com Thu Jan 12 16:28:13 2012 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Thu, 12 Jan 2012 09:28:13 -0600 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: <4F0EF8E6.6030002@btinternet.com> References: <4F0EF8E6.6030002@btinternet.com> Message-ID: <6B78951D-9135-4C87-B702-EFF05C5A62DA@gmail.com> +1 On Jan 12, 2012, at 9:14 AM, Rob Cliffe wrote: > In Python 2.5 you can at least say os.listdir('') which had the same effect. > In Python 2.7 you can't. > > Strictly speaking I think the default value should be os.path.curdir, not specifically '.' > > +1 from me > +1 on allowing the empty string argument again (may avoid breaking some old code) > > Rob Cliffe > > On 12/01/2012 15:02, Michael Foord wrote: >> >> Hello all, >> >> I frequently use Python from the interactive interpreter as a shell. It is a frequent, but mild, irritant that I have to type os.listdir('.') instead of just os.listdir(). >> >> How about making '.' a default argument for os.listdir ? >> >> All the best, >> >> Michael >> >> -- >> http://www.voidspace.org.uk/ >> >> May you do good and not evil >> May you find forgiveness for yourself and forgive others >> >> May you share freely, never taking more than you give. >> -- the sqlite blessing http://www.sqlite.org/different.html >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Thu Jan 12 16:39:27 2012 From: phd at phdru.name (Oleg Broytman) Date: Thu, 12 Jan 2012 19:39:27 +0400 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: References: Message-ID: <20120112153927.GA30594@iskra.aviel.ru> On Thu, Jan 12, 2012 at 03:02:52PM +0000, Michael Foord wrote: > I frequently use Python from the interactive interpreter as a shell. It is > a frequent, but mild, irritant that I have to type os.listdir('.') instead > of just os.listdir(). Not exactly what you've asked but let me show you my config for interactive sessions: http://phdru.name/Software/dotfiles/init.py.html You can type >>> cd('/etc') >>> ls and got back the list. (-: Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From benjamin at python.org Thu Jan 12 16:56:07 2012 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 12 Jan 2012 15:56:07 +0000 (UTC) Subject: [Python-ideas] os.listdir default to '.' References: Message-ID: Michael Foord writes: > > > Hello all,I frequently use Python from the interactive interpreter as a shell. It is a frequent, but mild, irritant that I have to type os.listdir('.') instead of just os.listdir().How about making '.' a default argument for os.listdir ?All the best,Michael-- $ ./python Python 3.3.0a0 (default:2db3ca05fbb7+, Jan 11 2012, 20:46:04) [GCC 4.5.3] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.listdir() ['autom4te.cache', 'pybuilddir.txt', 'PC', 'Tools', 'pyconfig.h', 'python', '.hgtags', 'Makefile.pre.in', 'install-sh', 'configure.in', '.hg', 'python-gdb.py', 'README', 'Parser', 'setup.py', 'config.log', 'Makefile', 'Doc', 'Lib', 'Objects', 'build', 'LICENSE', 'pyconfig.h.in', 'Python', 'Makefile.pre', '.hgeol', 'Include', 'libpython3.3m.a', '.hgignore', '.bzrignore', 'PCbuild', 'Modules', 'config.status', 'Misc', 'configure', '.gitignore', 'Grammar', 'Mac'] From brett at python.org Thu Jan 12 17:35:00 2012 From: brett at python.org (Brett Cannon) Date: Thu, 12 Jan 2012 11:35:00 -0500 Subject: [Python-ideas] Netiquette issues (was Re: "while ... try" - block or "for ... try" - block) In-Reply-To: <20120111174353.GC13614@idyll.org> References: <4F0DA5E4.40308@enigmage.de> <4F0DBD35.2000309@stoneleaf.us> <20120111174353.GC13614@idyll.org> Message-ID: On Wed, Jan 11, 2012 at 12:43, C. Titus Brown wrote: > Folks, > > I'm a moderator. Volunteer, true, but a moderator nonetheless. And > I've had enough. > > So: > > Netiquette discussions are always fascinating, but please, let's stop > talking > about how other people should post. Most of my e-mail from this list over > the > last few days has been from people on this list nattering about how gmail > foo > and mailer blah, and it is OFF TOPIC and NO LONGER WELCOME. (Frankly, you > should have figured this out for yourselves.) > > If you feel absolutely compelled to tell people that they are lousy > netizens > and/or correct them on their posting style, please do so PRIVATELY and not > on this list. > > Please e-mail me PRIVATELY if you disagree with this. Brett Cannon is the > other list admin, and you should feel free to tell him how much you dislike > me & my overbearing moderator ways. > And just FYI to people, I wouldn't bother complaining to me because I support Titus on this one; stuff has gone off the rails here, but I have been flat-out ignoring the list as a whole because of it instead of getting worked up like Titus. He's just a more "delicate flower" than me when comes to this and apparently cares more. =) -Brett > > Further e-mails on netiquette will result in those people no longer being > able to post without moderation. > > thanks, > --titus > > p.s. This is not aimed especially at Ethan -- it was just the latest e-mail > that pushed me over the edge :) > -- > C. Titus Brown, ctb at msu.edu > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leapyear.org Thu Jan 12 17:57:56 2012 From: bruce at leapyear.org (Bruce Leban) Date: Thu, 12 Jan 2012 08:57:56 -0800 Subject: [Python-ideas] An etherpad for every idea In-Reply-To: References: <20120110143156.1188db76@pitrou.net> <894BC1E9-5D57-4702-A758-45E29A1F2AF5@masklinn.net> <70c88004-a082-4f59-805d-1713c7c92728@gj9g2000pbc.googlegroups.com> Message-ID: On Thu, Jan 12, 2012 at 5:05 AM, julien tayon wrote: > Maybe python-ideas should only accept ideas in the form a link to a > PEP draft and people would have to maintain there draft as long as it > is not accepted. > > http://www.python.org/dev/peps/pep-0001/ > Seems like you didn't read PEP 1 carefully enough: "Vetting an idea publicly before going as far as writing a PEP is meant to save the potential author time. Many ideas have been brought forward for changing Python that have been rejected for various reasons...." Asking people to write a PEP before discussing ideas would stifle discussion and innovation. > By not adding a PEP draft that is not accepted in a documentation > database, we achieve one of the very first goal of a documentation > database which is lowering the Noise Signal Ratio. > python-ideas is not a documentation database. It's a discussion list. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From grosser.meister.morti at gmx.net Thu Jan 12 19:46:12 2012 From: grosser.meister.morti at gmx.net (=?ISO-8859-1?Q?Mathias_Panzenb=F6ck?=) Date: Thu, 12 Jan 2012 19:46:12 +0100 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: References: Message-ID: <4F0F2A74.2090800@gmx.net> IIRC for Python 2 under Windows os.listdir(".") does something different than os.listdir(u"."). Because I think there are two filesystem APIs under Windows, one using char and the other wchar_t. (I don't use Windows so I could be mistaken.) But anyway, under any OS Python 2 returns something else whether you pass a unicode or str object to listdir (a list of unicode or string objects). Under Python 3 there is no unicode because str is what under Python 2 was unicode. So under Python 2 os.listdir() would be not good. What should it return? strs or unicodes? On 01/12/2012 04:02 PM, Michael Foord wrote: > Hello all, > > I frequently use Python from the interactive interpreter as a shell. It is a frequent, but mild, > irritant that I have to type os.listdir('.') instead of just os.listdir(). > > How about making '.' a default argument for os.listdir ? > > All the best, > > Michael > > -- > > http://www.voidspace.org.uk/ > > May you do good and not evil > May you find forgiveness for yourself and forgive others > > May you share freely, never taking more than you give. > -- the sqlite blessinghttp://www.sqlite.org/different.html > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From masklinn at masklinn.net Thu Jan 12 21:00:32 2012 From: masklinn at masklinn.net (Masklinn) Date: Thu, 12 Jan 2012 21:00:32 +0100 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: <4F0F2A74.2090800@gmx.net> References: <4F0F2A74.2090800@gmx.net> Message-ID: <7CE9E4B7-F04F-41BB-9C77-E70E021FA173@masklinn.net> On 2012-01-12, at 19:46 , Mathias Panzenb?ck wrote: > > But anyway, under any OS Python 2 returns something else whether you pass a unicode or str object to listdir (a list of unicode or string objects). Under Python 3 there is no unicode because str is what under Python 2 was unicode. > > So under Python 2 os.listdir() would be not good. What should it return? strs or unicodes? I understand Python 2 is in maintenance-only status anyway, so there's no way python-ideas will apply to it. And since Python 3 seems to already have os.listdir default to '.'... From tjreedy at udel.edu Thu Jan 12 21:34:12 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 12 Jan 2012 15:34:12 -0500 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: References: Message-ID: On 1/12/2012 10:02 AM, Michael Foord wrote: > How about making '.' a default argument for os.listdir ? From 3.2 doc: "os.listdir(path='.')" In general, to find out what is already in future versions, the development docs (now 3.3a0, but later 3.4a0, etc) are at http://docs.python.org/dev/ Enhancement issues on the tracker show existing proposals. -- Terry Jan Reedy From nathan.alexander.rice at gmail.com Thu Jan 12 21:45:48 2012 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Thu, 12 Jan 2012 15:45:48 -0500 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) Message-ID: Greetings, I have been writing a lot of code lately that involves creating symbolic expressions of one form or another, which are then fully evaluated at a later time. Examples of this include Elementwise, where I create expressions that act on every member of an iterable (there is a much improved new version coming soon, by the way), and a design by contract/validation lib I'm working on (which shall remain nameless :D) that uses symbolic expressions in the *args of the metaclass __new__ method to generate a constraint class which validates input using __instancecheck__. I do most of this with lambdas, a little hacking with closures and FunctionType(), and chainable objects. I am very impressed that python is this flexible, but there are some issues with the approach that I would like to rectify, namely: 1. Because of the early binding behavior of most things in Python, if I want to include isinstance(X, someclass) in a symbolic expression, I have to wrap it in a lambda (or use .apply(), in the case of Elementwise). This is not a huge deal for me, but it forces me to create wrappers for lots of functions (e.g. isinstance_(X, someclass)) and/or have users wrap every such function they want to use in a symbolic expression. Having to do this also bloats the code a lot; the github version of Elementwise is over 3,000 LoC at this point (including prodigious documentation, but still...). 2. Python expects that certain functions, such as int(), str(), etc, will have a specific return type. While in general I agree with this, it makes Elementwise somewhat inconsistent (and it will do the same to anything else that wants to work with symbolic expressions). I'm interested in fixing both issues. I believe both issues I've had could be solved by having a robust "symbolic object". These objects would basically usable like ordinary objects, however upon any attribute access or other form of interaction, the object would basically short circuit the calling function, and return a symbolic object directly to the outer scope. The symbolic object would behave like a generator function frozen at the point of attribute access, and upon send()-ing (or whatever method), it would behave exactly as if the values sent had been the ones passed in originally (ideally without consuming the generator). I have thought about ways to approximate this behavior python currently, and while I could hack something together using inspect to pull relevant info from the stack, then break out using a special exception (potentially passing a generator with state as an exception arg), this approach strikes me as VERY brittle, implementation dependent, ugly and difficult to work with. Additionally, you would need to catch the special exception somewhere in the stack, so this trick wouldn't work on the first thing in an expression to be evaluated. As an aside, I'd like to solicit some feedback on the validation syntax I've been working on. Currently, I have code that support things like: X = SymbolicObject() const = Constraints(X * 2 + 1 >= 5, X % 2 != 0) const2 = Constraints(X[-1] == "h") const3 = Constraints(X[-1].upper() == "H") >>> print isinstance(3, const) True >>> print isinstance(2, const) False >>> print isinstance(1, const) False >>> print isinstance("bleh", const2) True >> print isinstance("bleh", const3) True Callables are supported as well, so if you wanted to do something like: Constraints(isinstance(X.attr, someclass), somefunc(X[-2].attr, args)) You could approximate that with: Constraints(lambda x: isinstance(x.attr, someclass), lambda x: somefunc(x[-2].attr, args)) As I mentioned in the first paragraph, Constraints is a metaclass, so your validations are checked using __instancecheck__. I'm also considering having __init__ generate mock objects (for certain straight-forward cases, anyhow). Thanks for your time, Nathan From cs at zip.com.au Thu Jan 12 22:32:23 2012 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 13 Jan 2012 08:32:23 +1100 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: <4F0EF8E6.6030002@btinternet.com> References: <4F0EF8E6.6030002@btinternet.com> Message-ID: <20120112213223.GA7309@cskk.homeip.net> On 12Jan2012 15:14, Rob Cliffe wrote: | In Python 2.5 you can at least say os.listdir('') which had the same effect. | In Python 2.7 you can't. | | Strictly speaking I think the default value should be | os.path.curdir, not specifically '.' | +1 from me I think it should be '.'. It is possible to arrange directory permissions so that you can't open curdir because it traverses a directory to which one lacks permissions, but you can still open '.'. Correspondingly, there can be times you can opendir('.') but not compute curdir (for access reasons); an unpleasant situation but possible (maybe not on platforms where the OS hands you curdir like Linux; historically UNIX platforms compute it by consulting the mount table and walking some directories. | +1 on allowing the empty string argument again (may avoid breaking | some old code) I used to be a fan of the '' UNIX string referring to the current directory (on the tenuous but cool logic that '' lets the kernel hand you the current directory directly, while '.' implies _opening_ the current directory and finding its '.' entry, and opening that:-) but not any more - it makes more trouble than its worth (and I think it went away in POSIX). '.' has better lexical properties than '' - it makes dirname and basename and similar operations easier to code and think about because it is not an empty string. opendir() should be as thin a glue around the OS's opendir as possible. On that basis I'm good with a default argument value but bad on explicitly supporting '' as meaning the current directory when the OS may not; it would at least mean wiring special knowledge into os.opendir for that string. I guess I'm: +1 for '.' being opendir's default. -0.5 for '' being opendir's default. -1 for os.curdir() being opendir's default. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ I'M NOT afraid of insects talking over the world, and you know why? It would take about a billion ant just to aim a gun at me, let alone fire it. And know what I'm doing while they're aiming ti at me? I just sort of slip off to the side, and then suddenly run up and kick the gun out of their hands. - Jack Handey From ben+python at benfinney.id.au Thu Jan 12 23:26:10 2012 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 13 Jan 2012 09:26:10 +1100 Subject: [Python-ideas] os.listdir default to '.' References: <4F0EF8E6.6030002@btinternet.com> <20120112213223.GA7309@cskk.homeip.net> Message-ID: <87ipkg7e2l.fsf@benfinney.id.au> Cameron Simpson writes: > I think it should be '.' [instead of ?os.path.curdir?]. > > It is possible to arrange directory permissions so that you can't open > curdir because it traverses a directory to which one lacks permissions, > but you can still open '.'. What distinction are you drawing? >>> os.path.curdir '.' Do you get a different result? -- \ ?Working out the social politics of who you can trust and why | `\ is, quite literally, what a very large part of our brain has | _o__) evolved to do.? ?Douglas Adams | Ben Finney From sven at marnach.net Thu Jan 12 23:11:36 2012 From: sven at marnach.net (Sven Marnach) Date: Thu, 12 Jan 2012 22:11:36 +0000 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: <20120112213223.GA7309@cskk.homeip.net> References: <4F0EF8E6.6030002@btinternet.com> <20120112213223.GA7309@cskk.homeip.net> Message-ID: <20120112221136.GA32550@pantoffel-wg.de> Cameron Simpson schrieb am Fr, 13. Jan 2012, um 08:32:23 +1100: > On 12Jan2012 15:14, Rob Cliffe wrote: > | Strictly speaking I think the default value should be > | os.path.curdir, not specifically '.' [...] > -1 for os.curdir() being opendir's default. There's a misunderstanding here. You are confusing 'os.getcwd()' and 'os.curdir'. The default value for the parameter of 'os.listdir()' should probably be 'os.curdir', but since it is defined in the different platform-specific OS modules like posixmodule.c, it probably is 'os.curdir'. Moreover, I'm not aware of a platform where os.curdir != '.' -- Sven From tjreedy at udel.edu Fri Jan 13 00:03:17 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 12 Jan 2012 18:03:17 -0500 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: References: Message-ID: On 1/12/2012 3:45 PM, Nathan Rice wrote: > Greetings, > > I have been writing a lot of code lately that involves creating > symbolic expressions of one form or another, which are then fully > evaluated at a later time. Examples of this include Elementwise, Python is designed for concrete, rather than symbolic computation. But the latter has been done. > where I create expressions that act on every member of an iterable > (there is a much improved new version coming soon, by the way), and a > design by contract/validation lib I'm working on (which shall remain > nameless :D) that uses symbolic expressions in the *args of the > metaclass __new__ method to generate a constraint class which > validates input using __instancecheck__. I do most of this with > lambdas, a little hacking with closures and FunctionType(), and > chainable objects. I am very impressed that python is this flexible, > but there are some issues with the approach that I would like to > rectify, namely: > > 1. Because of the early binding behavior of most things in Python, I think you may be confusing name-resolution time with execution time. They are distinct, though somewhat tied together in Python. For example: if I hand you a book and say "Read this", I intend that you immediately resolve 'this' to the book I just handed you and immediately 'read' it. If I instead say, "Tomorrow morning, read this", I still intend that you immediately resolve 'this' to a particular object, but now intend that you package the object with the action for later execution. I believe you want this this mixture of resolution now with action later. If so, your problem is that executing 'read(this)' or 'this.read()' does both things now, while "def f(): read(this)" or "lambda: read(this)" puts off both things until later. Python has several ways to binds objects now with actions later. Let be a reference to 'the book Terry handed me, which I stored wherever'. 0: rewrite the text -- a bit awkward in Python. action = compile("read({this})".format(this=), 'xxx', 'eval') 1. default args -- has to be done when the function is defined. def action(this = ): read(this) 2. closures (nested functions) -- also requires a planned-ahead definition. make_read(x): return lambda: read(x) action = make_read() 3. bound methods -- only works for classes with methods. Class Book: def read(self): pass action = Book().read 4. partial binding of function params -- generalizes bound methods; works for any function and argument. from functools import partial action = partial(read, ) > if I want to include isinstance(X, someclass) in a symbolic expression, Consider using partial, which can totally bind all needed args *now* for *later* action. >>> from functools import partial >>> t = partial(isinstance, 1, int) >>> t() True >>> f = partial(isinstance, 1, float) >>> f() False > I have to wrap it in a lambda Are you sure that partial will not work for you? Since partial is written in Python, you can grab and adjust the code to your needs. It amounts to adding default args after the fact by using a generic closure. > (or use .apply(), in the case of > Elementwise). This is not a huge deal for me, but it forces me to > create wrappers for lots of functions (e.g. isinstance_(X, someclass)) > and/or have users wrap every such function they want to use in a > symbolic expression. Having to do this also bloats the code a lot; > the github version of Elementwise is over 3,000 LoC at this point > (including prodigious documentation, but still...). > > 2. Python expects that certain functions, such as int(), str(), etc, > will have a specific return type. While in general I agree with this, People expect that class constructors produce an instance of the class. It is surprising when one does otherwise ;-). Builtin classes like int, bool, and str are classes just like ones you write. > it makes Elementwise somewhat inconsistent (and it will do the same to > anything else that wants to work with symbolic expressions). > > I'm interested in fixing both issues. I believe both issues I've had > could be solved by having a robust "symbolic object". These objects > would basically usable like ordinary objects, however upon any > attribute access or other form of interaction, the object would > basically short circuit the calling function, and return a symbolic > object directly to the outer scope. The symbolic object would behave > like a generator function frozen at the point of attribute access, and > upon send()-ing (or whatever method), it would behave exactly as if > the values sent had been the ones passed in originally (ideally > without consuming the generator). > > I have thought about ways to approximate this behavior python > currently, and while I could hack something together using inspect to > pull relevant info from the stack, then break out using a special > exception (potentially passing a generator with state as an exception > arg), this approach strikes me as VERY brittle, implementation > dependent, ugly and difficult to work with. Additionally, you would > need to catch the special exception somewhere in the stack, so this > trick wouldn't work on the first thing in an expression to be > evaluated. > > As an aside, I'd like to solicit some feedback on the validation > syntax I've been working on. Currently, I have code that support > things like: > > X = SymbolicObject() > > const = Constraints(X * 2 + 1>= 5, X % 2 != 0) > const2 = Constraints(X[-1] == "h") > const3 = Constraints(X[-1].upper() == "H") Using a special class is a standard way to delay concrete execution. >>>> print isinstance(3, const) > True A Contraints instance defines a set. 'const' is the set 'odd_ge_3' It would look better if you used standard syntax and do the inclusion check in a __contains__ method. >>> 3 in odd_ge_3 True >>>> print isinstance(2, const) > False 2 in odd_ge_3 1 in odd_ge_3 >>>> print isinstance(1, const) > False > >>>> print isinstance("bleh", const2) > True > >>> print isinstance("bleh", const3) > True "bleh" in ends_in_h "bleh" in ends_in_h_or_H > Callables are supported as well, so if you wanted to do something like: > > Constraints(isinstance(X.attr, someclass), somefunc(X[-2].attr, args)) > > You could approximate that with: > > Constraints(lambda x: isinstance(x.attr, someclass), lambda x: > somefunc(x[-2].attr, args)) > > As I mentioned in the first paragraph, Constraints is a metaclass, This is your first mention, actually. > so your validations are checked using __instancecheck__. But it is a fake check in that 3 is not really an instance of the class, which has no instances. It is hard for me to believe that you cannot put the same constraint data in an instance of Constraint as a class and the inclusion logic in __contains__. If you customize __instancecheck__ for each instance of Constraints, then write def__contains__(self, ob): return self._check(ob) where _check does the same as your current __instancecheck__ Even if you *have* to make Constraints a metaclass, for other reasons, I believe you could still give it the same __contains__ method. A metaclass *is* a class, and if its class instances represent collections, inclusion in the colleciton should be tested in the standard way. > I'm also > considering having __init__ generate mock objects (for certain > straight-forward cases, anyhow). -- Terry Jan Reedy From guido at python.org Fri Jan 13 00:11:24 2012 From: guido at python.org (Guido van Rossum) Date: Thu, 12 Jan 2012 15:11:24 -0800 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: <20120112221136.GA32550@pantoffel-wg.de> References: <4F0EF8E6.6030002@btinternet.com> <20120112213223.GA7309@cskk.homeip.net> <20120112221136.GA32550@pantoffel-wg.de> Message-ID: On Thu, Jan 12, 2012 at 2:11 PM, Sven Marnach wrote: > Cameron Simpson schrieb am Fr, 13. Jan 2012, um 08:32:23 +1100: > > On 12Jan2012 15:14, Rob Cliffe wrote: > > | Strictly speaking I think the default value should be > > | os.path.curdir, not specifically '.' > [...] > > -1 for os.curdir() being opendir's default. > > There's a misunderstanding here. You are confusing 'os.getcwd()' and > 'os.curdir'. > > The default value for the parameter of 'os.listdir()' should probably > be 'os.curdir', but since it is defined in the different > platform-specific OS modules like posixmodule.c, it probably is > 'os.curdir'. Moreover, I'm not aware of a platform where > os.curdir != '.' > The default for os.listdir() should be the current directory by whatever name it is known. os.curdir is ':' in macpath.py (which applies to MacOS 1-9, not OS X). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Fri Jan 13 00:24:59 2012 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 13 Jan 2012 10:24:59 +1100 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: <87ipkg7e2l.fsf@benfinney.id.au> References: <87ipkg7e2l.fsf@benfinney.id.au> Message-ID: <20120112232459.GA32459@cskk.homeip.net> On 13Jan2012 09:26, Ben Finney wrote: | Cameron Simpson writes: | > I think it should be '.' [instead of ?os.path.curdir?]. | > | > It is possible to arrange directory permissions so that you can't open | > curdir because it traverses a directory to which one lacks permissions, | > but you can still open '.'. | | What distinction are you drawing? | | >>> os.path.curdir | '.' A totally bogus distinction. I had os.getcwd() in my head instead. Please ignore my post:-( Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ This signature was originally recorded on analog equipment. While we have attempted to preserve the meaning of the original, the high resolution of this media may reveal limitations of the source. - paul-michael agapow From cs at zip.com.au Fri Jan 13 00:26:04 2012 From: cs at zip.com.au (Cameron Simpson) Date: Fri, 13 Jan 2012 10:26:04 +1100 Subject: [Python-ideas] os.listdir default to '.' In-Reply-To: <20120112221136.GA32550@pantoffel-wg.de> References: <20120112221136.GA32550@pantoffel-wg.de> Message-ID: <20120112232604.GA914@cskk.homeip.net> On 12Jan2012 22:11, Sven Marnach wrote: | Cameron Simpson schrieb am Fr, 13. Jan 2012, um 08:32:23 +1100: | > On 12Jan2012 15:14, Rob Cliffe wrote: | > | Strictly speaking I think the default value should be | > | os.path.curdir, not specifically '.' | [...] | > -1 for os.curdir() being opendir's default. | | There's a misunderstanding here. You are confusing 'os.getcwd()' and | 'os.curdir'. Yes. Please ignore the rest of that post, based as it is on a brain fart on my part. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ It must be public fact, because I'm not the only one who knows about it. - Stefan A. Werner From pyideas at rebertia.com Fri Jan 13 07:22:26 2012 From: pyideas at rebertia.com (Chris Rebert) Date: Thu, 12 Jan 2012 22:22:26 -0800 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: References: Message-ID: On Thu, Jan 12, 2012 at 3:03 PM, Terry Reedy wrote: > On 1/12/2012 3:45 PM, Nathan Rice wrote: >>>>> print isinstance(3, const) >> >> True > > > A Contraints instance defines a set. 'const' is the set 'odd_ge_3' > It would look better if you used standard syntax and do the inclusion check > in a __contains__ method. > >>>> 3 in odd_ge_3 > > True But what are types but abstract sets of values? Phrasing it as a typecheck is perfectly sensible from a type-theoretic standpoint. Also, the problem of representing `isinstance(X.attr, someclass)` [for non-Constraint someclass, e.g. str] in a Constraint would still remain, since there's no "__lcontains__" (thus precluding `X.attr in str`). >> so your validations are checked using __instancecheck__. > > But it is a fake check in that 3 is not really an instance of the class, > which has no instances. The same can be true for abstract base classes, which have been sufficiently accepted to warrant adding __instancecheck__() in the first place and also to be added to the std lib (witness the `abc` and `collections` modules). It may seem unfamiliar, but then again it was only made possible starting with v2.6, which is pretty recent. Cheers, Chris -- http://rebertia.com From nathan.alexander.rice at gmail.com Fri Jan 13 07:24:35 2012 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Fri, 13 Jan 2012 01:24:35 -0500 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: References: Message-ID: >> I have been writing a lot of code lately that involves creating >> symbolic expressions of one form or another, which are then fully >> evaluated at a later time. ?Examples of this include Elementwise, > > > Python is designed for concrete, rather than symbolic computation. But the > latter has been done. Being able to create abstract expressions that are later realized is super useful and neat. You can do this decently right now, but life would be better if you didn't have to jump through so many hoops. Having symbolic variables override *anything* would also make lambdas obsolete and greatly increase the potential for lazy evaluation. > 0: rewrite the text -- a bit awkward in Python. > > action = compile("read({this})".format(this=), 'xxx', 'eval') Yeah, that is something I'd expect to see in Perl code :) > 1. default args -- has to be done when the function is defined. > > def action(this = ): read(this) > > 2. closures (nested functions) -- also requires a planned-ahead definition. > > make_read(x): > ?return lambda: read(x) > action = make_read() I use this extensively in Elementwise. > 3. bound methods -- only works for classes with methods. > > Class Book: > ? ?def read(self): pass > action = Book().read > > 4. partial binding of function params -- generalizes bound methods; works > for any function and argument. > > from functools import partial > action = partial(read, ) > >> if I want to include isinstance(X, someclass) in a symbolic expression, > > > Consider using partial, which can totally bind all needed args *now* for > *later* action. The issue isn't so much that I *can't* do things as they are more trouble than they should be, and it makes the end user interface for the stuff I write less elegant. For example, if I want to work with a symbolic object, but include a function that is not well behaved, or if I was creating a constraint on the result of a function applied to a symbolic object, I have to know ahead of time everything I want to do, so I can wrap the whole expression in a lambda. Once I wrap it, the nice generative chaining property disappears and I'm stuck with a callable. >>>> from functools import partial >>>> t = partial(isinstance, 1, int) >>>> t() > True >>>> f = partial(isinstance, 1, float) >>>> f() > False > > >> I have to wrap it in a lambda > > > Are you sure that partial will not work for you? Since partial is written in > Python, you can grab and adjust the code to your needs. It amounts to adding > default args after the fact by using a generic closure. In some cases it would, in some cases it wouldn't. Since I basically never do */** expansion on wrappers, lambdas tend to be my go-to more often. >> (or use .apply(), in the case of >> >> Elementwise). ?This is not a huge deal for me, but it forces me to >> create wrappers for lots of functions (e.g. isinstance_(X, someclass)) >> and/or have users wrap every such function they want to use in a >> symbolic expression. ? Having to do this also bloats the code a lot; >> the github version of Elementwise is over 3,000 LoC at this point >> (including prodigious documentation, but still...). >> >> 2. ?Python expects that certain functions, such as int(), str(), etc, >> will have a specific return type. ?While in general I agree with this, > > > People expect that class constructors produce an instance of the class. It > is surprising when one does otherwise ;-). Builtin classes like int, bool, > and str are classes just like ones you write. type/str/int/etc as types is definitely semi-coherent, since the language really treats them more like functions. They are treated that way all over the docs as well. >From the data model page: "object.__str__(self) Called by the str() built-in function and by the..." "object.__nonzero__(self) Called to implement truth value testing and the built-in operation bool()" "object.__complex__(self) object.__int__(self) object.__long__(self) object.__float__(self) Called to implement the built-in functions complex(), int(), long(), and float(). Should return a value of the appropriate type." So clearly this is an area that needs some polish :) >> X = SymbolicObject() >> >> const = Constraints(X * 2 + 1>= 5, X % 2 != 0) >> const2 = Constraints(X[-1] == "h") >> const3 = Constraints(X[-1].upper() == "H") > > > Using a special class is a standard way to delay concrete execution. Standard, and currently a pain in the butt, starting from the fact that operators don't hook into __getattribute__ and getting progressively worse from there. >>>>> print isinstance(3, const) >> >> True > > > A Contraints instance defines a set. 'const' is the set 'odd_ge_3' > It would look better if you used standard syntax and do the inclusion check > in a __contains__ method. Used standard syntax? Can you elaborate please? Also, a set is one of many things a Constraints instance could logically be represented with, as well as a discontinuous interval, a class in the colloquial sense, etc. The nice thing about __instancecheck__ is that every possible constraint reduces to a type check. Of course, you could rephrase all type checking in terms of set membership easily, but I don't think it is quite as intuitive to most folks. Your preference has been noted though. >> As I mentioned in the first paragraph, Constraints is a metaclass, > > > This is your first mention, actually. Feeling feisty? I'm actually curious to see what sort of rational you come up with to back up that statement; should be interesting :) > >> so your validations are checked using __instancecheck__. > > But it is a fake check in that 3 is not really an instance of the class, > which has no instances. It is hard for me to believe that you cannot put the > same constraint data in an instance of Constraint as a class and the > inclusion logic in __contains__. If you customize __instancecheck__ for each > instance of Constraints, then write It is a fake check, like any abstract base class instancecheck is a fake check. > ?def__contains__(self, ob): return self._check(ob) > > where _check does the same as your current __instancecheck__ > > Even if you *have* to make Constraints a metaclass, for other reasons, I > believe you could still give it the same __contains__ method. A metaclass > *is* a class, and if its class instances represent collections, inclusion in > the colleciton should be tested in the standard way. It could be any sort of callable. __instancecheck__ is the only reason it is a metaclass. Otherwise, I probably wouldn't bother with classes at all; returning a check inner function with constraints in the closure would be easy. Cheers, Nathan From jeanpierreda at gmail.com Fri Jan 13 14:45:38 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Fri, 13 Jan 2012 08:45:38 -0500 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: References: Message-ID: On Thu, Jan 12, 2012 at 3:45 PM, Nathan Rice wrote: > I'm interested in fixing both issues. I believe both issues I've had > could be solved by having a robust "symbolic object". These objects > would basically usable like ordinary objects, however upon any > attribute access or other form of interaction, the object would > basically short circuit the calling function, and return a symbolic > object directly to the outer scope. The symbolic object would behave > like a generator function frozen at the point of attribute access, and > upon send()-ing (or whatever method), it would behave exactly as if > the values sent had been the ones passed in originally (ideally > without consuming the generator). I find the way you've formalized this a bit "weird". It looks like you're suggesting adding laziness to Python. If that's what you want, maybe you should try PyPy and the thunk object space: http://doc.pypy.org/en/latest/objspace-proxies.html#the-thunk-object-space -- Devin From nathan.alexander.rice at gmail.com Fri Jan 13 15:21:07 2012 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Fri, 13 Jan 2012 09:21:07 -0500 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: References: Message-ID: On Fri, Jan 13, 2012 at 8:45 AM, Devin Jeanpierre wrote: > On Thu, Jan 12, 2012 at 3:45 PM, Nathan Rice > wrote: >> I'm interested in fixing both issues. I believe both issues I've had >> could be solved by having a robust "symbolic object". ?These objects >> would basically usable like ordinary objects, however upon any >> attribute access or other form of interaction, the object would >> basically short circuit the calling function, and return a symbolic >> object directly to the outer scope. ?The symbolic object would behave >> like a generator function frozen at the point of attribute access, and >> upon send()-ing (or whatever method), it would behave exactly as if >> the values sent had been the ones passed in originally (ideally >> without consuming the generator). > > I find the way you've formalized this a bit "weird". It looks like > you're suggesting adding laziness to Python. > > If that's what you want, maybe you should try PyPy and the thunk object space: While thunk is neat, it doesn't accomplish precisely what I'm describing in this instance. When a function starts to run under thunk, the computations take place as soon as the function gets to the object inside its scope. What I'm after is the ability to basically create functions using only expressions, in a generative manner. Lambda does accomplish this, but in an extremely clunky manner... for example: X = lambda x: x + 1 Y = lambda y: y * 2 Z = lambda z: z % 3 (or XYZ = lambda x: (((x + 1) * 2) % 3) If I want to perform a second step after this, I need create another lambda, because they don't chain/aren't generative. The thing that Elementwise does that is very appealing is that most operations are generative, so if you have an ElementwiseProxy object x, x2 = (((x + 1) * 2) % 3) basically does the same thing as the above, but you can then do x3 = x2 ** 3, and so on until you are ready to get your results. Once you have your results, you can use the same ElementwiseProxy again with different inputs. It is just a function generation technique which has some elegant properties. Having a native object type designed to facilitate this would let people do a lot of interesting things, and make things like Elementwise be a lot more "consistent". I bet you could do some really neat things by subclassing such an object as well. This is certainly not a common programming paradigm, but I think most people could reap a lot of benefits from it while being oblivious to its presence. Nathan From eike.welk.lists1 at gmx.de Fri Jan 13 15:53:35 2012 From: eike.welk.lists1 at gmx.de (Eike Welk) Date: Fri, 13 Jan 2012 15:53:35 +0100 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: References: Message-ID: <12518550.K8fIVbB0Kl@lixie> On Friday 13.01.2012 08:45:38 Devin Jeanpierre wrote: > I find the way you've formalized this a bit "weird". It looks like > you're suggesting adding laziness to Python. IMHO this is quoting like in Lisp: He stores a bit of code in an un-evaluated form. The bit of code is then interpreted somewhere else, in special contexts, that he can choose with his hacked version of "isinstance". Eike. From phd at phdru.name Fri Jan 13 16:10:01 2012 From: phd at phdru.name (Oleg Broytman) Date: Fri, 13 Jan 2012 19:10:01 +0400 Subject: [Python-ideas] Symbolic expressions In-Reply-To: <12518550.K8fIVbB0Kl@lixie> References: <12518550.K8fIVbB0Kl@lixie> Message-ID: <20120113151001.GC26517@iskra.aviel.ru> On Fri, Jan 13, 2012 at 03:53:35PM +0100, Eike Welk wrote: > On Friday 13.01.2012 08:45:38 Devin Jeanpierre wrote: > > > I find the way you've formalized this a bit "weird". It looks like > > you're suggesting adding laziness to Python. > > IMHO this is quoting like in Lisp: He stores a bit of code in an un-evaluated > form. The bit of code is then interpreted somewhere else, in special contexts, > that he can choose with his hacked version of "isinstance". Symbolic algebra is a widely-deployed technique in Python world. It is popular, e.g., in object-relational mappers. IMHO Python is powerful enough in that area, there is no need to change the language. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From jeanpierreda at gmail.com Fri Jan 13 17:24:31 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Fri, 13 Jan 2012 11:24:31 -0500 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: <12518550.K8fIVbB0Kl@lixie> References: <12518550.K8fIVbB0Kl@lixie> Message-ID: On Fri, Jan 13, 2012 at 9:53 AM, Eike Welk wrote: > IMHO this is quoting like in Lisp: He stores a bit of code in an un-evaluated > form. The bit of code is then interpreted somewhere else, in special contexts, > that he can choose with his hacked version of "isinstance". > I wanted to think he wanted quote, because quote is awesome, but that doesn't gel with his desired behavior inside functions. def foo(A): return (A + 2) * 5 (define (foo A) (* (+ A 2) 5))) If in Python one calls foo(SymbolicExpr + 3), then it returns a SymbolicExpr that can be lazily evaluated to compute the expression. This is fairly general and could work even with mutating statements and all sorts of nasty things. If, in Scheme, one calls (foo '(+ X 3)), it is an error. There's no automatic laziness associated with quoted expressions -- they're just data. If one called (foo '3) then it would outright eagerly evaluate the function, returning 25, because '3 == 3. OTOH, you can define a foo that operates on quoted expressions and would return '(* (+ (+ X 3) 2) 5), as follows: (define (foo A) `(* (+ ,A 2) 5)) That might be sufficient for his use-case, can't tell. It isn't specifically what he asked for, though. (See also: XY problem). -- Devin From eike.welk.lists1 at gmx.de Sat Jan 14 04:25:50 2012 From: eike.welk.lists1 at gmx.de (Eike Welk) Date: Sat, 14 Jan 2012 04:25:50 +0100 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: References: Message-ID: <1462421.BIX0U07Ti7@lixie> I think something like your SymbolicObject class should be in Python's standard library. Quite a few projects implement similar classes. If there is a basic infrastructure for symbolic computation in the standard library, symbolic algorithms can be reused much more simple across projects. The pattern you employ with SymbolicObject, is a nice way to mix code that is executed immediately, with code that is executed later. Symbolic algebra languages, like Maple and Mathematica, function this way too. I propose however a slightly different interface: There should be a special object "quoted" to create instances of "SymbolicObject". The objects should be created in the "__getattribute__" method. This way the objects know their name, which is quite useful. A symbolic object would be created like this: X = quoted.X You could also write: const = Constraints(quoted.X * 2 + 1 >= 5, quoted.X % 2 != 0) The (tree of) symbolic objects should have a method that translate it to an abstract syntax tree (ast.AST from the standard library). Subsequent algorithms should operate on this AST. This way symbolic algorithms can also operate on all possible Python code (SymbolicObject can only create expressions). my_ast = (quoted.X * 2 + 1).to_ast() For convenience I think that all symbolic algorithms should also accept SymbolicObject, and convert them to ast.AST internally. The standard library should contain at least these symbolic algorithms: evalsym(tree, environment, return_type="any") Evaluates an expression and returns the result. Works similar to: eval(compile(tree, "", "eval"), environment) This call would return 7: evalsym(quoted.X * 2 + 1, {"X":3}) However it should do partial evaluation if some symbols are unknown, and return an ast.AST in this case (similar to Maple and Mathematica). This expression should be True: evalsym(quoted.X + quoted.Y, {"Y":3}) == quoted.X + 3 The optional argument "return_type" specifies the type of the result. This simplifies algorithms using "eval". Argument "return_type" can be: "any", "ast", or "object". substitute(tree, substitutions) Replaces a symbol with a value (a sub-tree of ast.Ast). This expression should be True: substitute(quoted.X + quoted.Y, {"Y": quoted.X * 2}) \ == (quoted.X + quoted.X * 2).to_ast() Maybe this algorithm should also be able to match general fragments of the tree and replace them. Eike. From liu at cs.stonybrook.edu Sat Jan 14 06:25:19 2012 From: liu at cs.stonybrook.edu (Annie Liu) Date: Sat, 14 Jan 2012 05:25:19 +0000 Subject: [Python-ideas] quantifications, and tuple patterns Message-ID: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> Hi! I'd like to propose adding quantifications to python. It'd be very helpful to have tuple patterns as well. I found extensive use of quantifications while learning and teaching distributed algorithms in the last few years, and I saw more and more places where they can help make programming easier and make programs clearer and less error-prone. They didn't show up this much until I started to pay attention, since my students (even the best ones) and myself too just coded them away too quickly. For example, even the simplest "some x in s has pred", when a witness for x is not needed, could end up like "{x for x in s if pred}", "any(pred for x in s)", "len(...)!=0", "len...>0", len...>=1", ... not to mention the many more kinds of loops and tests written by the majority of students not used to writing queries/comprehensions. Even more kinds of loops and bigger hassles are involved when a witness for x is needed. For another example, a simple "each x in s has x > y" tends to end up like "min(s)>y", which is incorrect, since s could be empty, so one has to either add this condition or invent some boundary value for min, which is error-prone since it is sensitive to the possible values of elements of s and the uses of these values. For an example distributed algorithm, one may look at Robbert von Renesse's "Paxos made moderately complex", at http://www.cs.cornell.edu/courses/CS7412/2011sp/paxos.pdf (he has newer versions but the pseudo code didn't change). It has 8 uses of quantifications. His algorithm can be coded almost exactly like that in our language DistAlgo (python with small extensions for distributed programming) if we have quantifications and tuple patterns. For example (using "!" to indicate a bound variable in tuple patterns), while some (!slot_num,p1) in decisions: if some (!slot_num,p2) in proposals has p2 != p1: is currently written as follows (it happens to be correct here to use "for" for "if" but generally requires more code to be correct): while {p1 for (s0,p1) in decisions if s0==slot_num}: p1 = {p1 for (s0,p1) in decisions if s0==slot_num}.pop() for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 != p1}: Both a witness for p1 and a witness for p2 are needed here. (BTW a best student working on this, after given advanced warnings and hints, wrote a 4-line nested "for"s and "if"s that were actually incorrect) So we started to think about adding quantifications and tuple patterns to DistAlgo too (other extensions are all distributed programming features). In general, "some x in s has pred" should return true/false AND bind x to a witness if true is returned. One might object to the side effect of the binding, but it is most natural here for ease of programming and clarity of programs, and it helps program efficiency too. Witness in existential quantification is also standard in logic. Of course, there is the obvious problem of new keywords: "some", "each", "has" (perhaps one may use ":" for "has") or some others. The "!" syntax would be ok for tuple patterns but only if there are no "equal cards", i.e., multiple occurrences of a same variable (though such occurrences don't seem to be useful in current python semantics for "for". A special symbol for wildcard would be more useful). So we are looking for good syntax. Thanks, Annie Liu From cmjohnson.mailinglist at gmail.com Sat Jan 14 07:27:35 2012 From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson) Date: Fri, 13 Jan 2012 20:27:35 -1000 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: <1462421.BIX0U07Ti7@lixie> References: <1462421.BIX0U07Ti7@lixie> Message-ID: <0567BC83-2437-4881-8F8D-F6E47DB76B72@gmail.com> You guys know this can more or less be done in Python today, right? >>> class Symbolic: ... def __init__(self, func=lambda x: x): ... self.func = func ... ... def __call__(self, arg): ... return self.func(arg) ... ... def __add__(self, other): ... func = lambda x: self.func(x) + other ... return Symbolic(func) ... ... __radd__ = __add__ ... ... def __mul__(self, other): ... func = lambda x: self.func(x) * other ... return Symbolic(func) ... ... __rmul__ = __mul__ ... >>> >>> X = Symbolic() >>> Y = (2 * X) + 10 >>> Y(20) 50 I leave further extensions as an exercise for the reader, but it's not especially tricky to add various boolean tests and whatnot. I don't claim any originality here. I'm fairly sure I read about this idea in blog post somewhere years ago. From p.f.moore at gmail.com Sat Jan 14 13:32:05 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 14 Jan 2012 12:32:05 +0000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> Message-ID: Sorry, should have replied on list. On 14 January 2012 05:25, Annie Liu wrote: > For example, even the simplest "some x in s has pred", when a witness > for x is not needed, could end up like "{x for x in s if pred}", > "any(pred for x in s)", "len(...)!=0", "len...>0", len...>=1", ... not > to mention the many more kinds of loops and tests written by the > majority of students not used to writing queries/comprehensions. ?Even > more kinds of loops and bigger hassles are involved when a witness for > x is needed. I'm confused as to why any(pred for x in s) is unacceptable here? > For another example, a simple "each x in s has x > y" tends to end up > like "min(s)>y", which is incorrect, since s could be empty, so one > has to either add this condition or invent some boundary value for > min, which is error-prone since it is sensitive to the possible values > of elements of s and the uses of these values. all(x > y for x in s) seems to me both idiomatic and correct. I'm not sure quite what you're proposing here - it seems like you're asking for new syntax to do the same job as any and all, which already do exactly what's needed. New syntax has to provide something much, much better if it's likely to be accepted. Please explain more clearly. Paul. From liu at cs.stonybrook.edu Sat Jan 14 14:57:14 2012 From: liu at cs.stonybrook.edu (Annie Liu) Date: Sat, 14 Jan 2012 13:57:14 +0000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> Message-ID: <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> Hi! For those two simplest examples, yes, "any" and "all" can be used. However, there are two general problems. 1. The order of coding using "any/all" is the opposite order of thinking in one's head. In our experience, those kinds of simple examples are coded much more often using "len" than "any/all". The ordering problem gets worse when quantifications are needed in more complex conditions or are nested. 2. The much worse problem is when a witness is needed from existential quantification. In my last/third example from a distributed algorithm, this is the desired code (recall that ! indicates a bound variable): while some (!slot_num,p1) in decisions: if some (!slot_num,p2) in proposals has p2 != p1: propose(p2) perform(p1) "any" is not sufficient to code this easily and clearly. Such witnesses are needed in all worklist algorithms. Thanks, Annie From p.f.moore at gmail.com Sat Jan 14 17:19:29 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 14 Jan 2012 16:19:29 +0000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> Message-ID: On 14 January 2012 13:57, Annie Liu wrote: > For those two simplest examples, yes, "any" and "all" can be used. > However, there are two general problems. > > 1. ?The order of coding using "any/all" is the opposite order of > thinking in one's head. ?In our experience, those kinds of simple > examples are coded much more often using "len" than "any/all". ?The > ordering problem gets worse when quantifications are needed in more > complex conditions or are nested. To be honest, that's a matter of opinion. I find any/all with a generator expression more readable than your syntax, because I am used to generator expressions as they are common in Python. > 2. ?The much worse problem is when a witness is needed from existential > quantification. ?In my last/third example from a distributed algorithm, > this is the desired code (recall that ! indicates a bound variable): > > ?while some (!slot_num,p1) in decisions: > ? ? if some (!slot_num,p2) in proposals has p2 != p1: > ? ? ? ?propose(p2) > ? ? perform(p1) Wow, I find that unreadable!. The ! doesn't read as any sort of assignment to me. Don't forget that in Python, assignments are statements and don't get embedded in expressions. There is no way you're going to get new syntax that creates an expression which embeds a hidden assignment accepted into Python. I'd offer to re write this in idiomatic Python, but looking at it closer, I've no idea how to. I don't know what slot_num is meant to be, at first I thought p1 and p2 were predicates, but then I notice you are comparing them later. Can I suggest you write your example out in Python that works today, and then show how it looks with your proposed syntax alongside? If you can't find the "best" way of writing it in existing Python, just write it however works, no need to try to make it compact, or elegant. There'll be plenty of people here who will show you how to write idiomatic Python versions of what you post :-) Paul. From guido at python.org Sat Jan 14 20:24:11 2012 From: guido at python.org (Guido van Rossum) Date: Sat, 14 Jan 2012 11:24:11 -0800 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> Message-ID: On Sat, Jan 14, 2012 at 8:19 AM, Paul Moore wrote: > On 14 January 2012 13:57, Annie Liu wrote: > Hi Annie! (*) > > For those two simplest examples, yes, "any" and "all" can be used. > > However, there are two general problems. > > > > 1. The order of coding using "any/all" is the opposite order of > > thinking in one's head. In our experience, those kinds of simple > > examples are coded much more often using "len" than "any/all". The > > ordering problem gets worse when quantifications are needed in more > > complex conditions or are nested. > > To be honest, that's a matter of opinion. I find any/all with a > generator expression more readable than your syntax, because I am used > to generator expressions as they are common in Python. > But Paul, aren't you missing the fact that for the algorithms that Annie and her students want to write, the "witness" concept is essential? I.e. they can't just use any(P(x) for x in xs) because if it returns True, they want to know the x that made P(x) be true. Her ! notation is a (perhaps unpythonic) attempt at exporting this witness from the quantification. > 2. The much worse problem is when a witness is needed from existential > > quantification. In my last/third example from a distributed algorithm, > > this is the desired code (recall that ! indicates a bound variable): > > > > while some (!slot_num,p1) in decisions: > > if some (!slot_num,p2) in proposals has p2 != p1: > > propose(p2) > > perform(p1) > > Wow, I find that unreadable!. The ! doesn't read as any sort of > assignment to me. Don't forget that in Python, assignments are > statements and don't get embedded in expressions. I think she's well aware of that, and proposing to change it. :-) There is no way > you're going to get new syntax that creates an expression which embeds > a hidden assignment accepted into Python. > That's your opinion. But this is python-ideas. :-) I'd offer to re write this in idiomatic Python, but looking at it > closer, I've no idea how to. I don't know what slot_num is meant to > be, at first I thought p1 and p2 were predicates, but then I notice > you are comparing them later. > > Can I suggest you write your example out in Python that works today, > and then show how it looks with your proposed syntax alongside? If you > can't find the "best" way of writing it in existing Python, just write > it however works, no need to try to make it compact, or elegant. > There'll be plenty of people here who will show you how to write > idiomatic Python versions of what you post :-) > Actually she gave one in her first post. Here it is again: while {p1 for (s0,p1) in decisions if s0==slot_num}: p1 = {p1 for (s0,p1) in decisions if s0==slot_num}.pop() for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 != p1}: Note that the set {p1 for (s0,p1) in decisions if s0==slot_num} is computed twice, once to decide whether to stop, and then again to compute the witness (p1). Obviously this is inefficient, and that's what she's after. To make this same code more efficient in Python, you'd have to do the following, which is natural for us programmers (since we're so used to working around limitations and inefficiencies in the systems we work with) but unnatural for mathematicians, who count to infinity at the drop of a hat: while True: temp = {p1 for (s0,p1) in decisions if s0==slot_num} if not temp: break p1 = temp.pop() for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 != p1}: The 5 tedious lines from "while" through "pop()" would be collapsed into a single line if you could write while some s0, p1 in decisions if s0 == slot_num: for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 != p1}: TBH I'm not sure what the !slot_num notation is for -- it appears that while some (!slot_num, p1) in decisions: is equivalent in Annie's proposal to while some s0, p1 in decisions if s0 == slot_num: but I'm not sure and it doesn't feel necessary to me. Also note that SOME and EACH quantifiers were present in ABC (Python's predecessor: http://homepages.cwi.nl/~steven/abc/qr.html#TESTS); I dropped them for simplicity, not because I didn't like them. If we wanted to we could have them back (except for the problems of introducing new keywords). __________ (*) See http://neopythonic.blogspot.com/2011/06/depth-and-breadth-of-python.html -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan.alexander.rice at gmail.com Sat Jan 14 21:13:36 2012 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Sat, 14 Jan 2012 15:13:36 -0500 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: <0567BC83-2437-4881-8F8D-F6E47DB76B72@gmail.com> References: <1462421.BIX0U07Ti7@lixie> <0567BC83-2437-4881-8F8D-F6E47DB76B72@gmail.com> Message-ID: On Sat, Jan 14, 2012 at 1:27 AM, Carl M. Johnson wrote: > You guys know this can more or less be done in Python today, right? I did mention in the first post that I am the author of a library that does this right? :D The issue is of consistency. some functions can break out of this behavior. Some "type" functions also want to behave a certain way regardless of input (which makes sense in most cases but breaks chaining symbolic expressions). Additionally, the amount of code required to do that sort of object proxying is really obnoxious. Things like operator methods should be hooked into__getattribute__ on symbolic objects. There are also a LOT of neat possibilities that come out of symbolic objects, for instance, function templates, fairly easy currying and generation of closed form solutions by performing calculations on functions rather than data. Nathan From greg.ewing at canterbury.ac.nz Sat Jan 14 21:18:37 2012 From: greg.ewing at canterbury.ac.nz (Greg) Date: Sun, 15 Jan 2012 09:18:37 +1300 Subject: [Python-ideas] Symbolic expressions In-Reply-To: <20120113151001.GC26517@iskra.aviel.ru> References: <12518550.K8fIVbB0Kl@lixie> <20120113151001.GC26517@iskra.aviel.ru> Message-ID: <4F11E31D.50403@canterbury.ac.nz> Oleg Broytman wrote: > Symbolic algebra is a widely-deployed technique in Python world. It > is popular, e.g., in object-relational mappers. IMHO Python is powerful > enough in that area, there is no need to change the language. The existing techniques for this get a bit awkward at times, though, for example when you want boolean operations, since 'and' etc. can't be overridden. My Overloadable Boolean Operators PEP was one attempt to address this problem, but it hasn't met with a very enthusiastic reception. Recently I've been working on another approach using that I've been calling "deferred expressions" (although I'll probably change that name because "deferred" seems to have taken on a different meaning already). The idea is that you write something like this (sorry about the ugly syntax, I haven't thought of anything better yet): e = <(a + b * c)> This gives you something that behaves in one respect like a parameterless lambda -- you can evaluate it, and any free variables are interpreted in their original environment. However, its internal structure is effectively an AST that you can pick apart and process however you want, including selectively evaluating sub-expressions and building up new expressions. Currently they're not generative the way the OP wants, i.e. you can't do 'e1 + e2' and get a new deferred expression, but that could be arranged, and would provide a nice way of building new expressions. The only problem would be attribute access, since they currently have attributes modelled after those of the corresponding AST nodes, whereas under a generative regime, 'e.a' ought to return a new expression node representing the access of attribute 'a'. I have about 90% of this implemented. I'll make a release as soon as I have something fit for public consumption. -- Greg From p.f.moore at gmail.com Sat Jan 14 22:38:29 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 14 Jan 2012 21:38:29 +0000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> Message-ID: On 14 January 2012 19:24, Guido van Rossum wrote: > But Paul, aren't you missing the fact that for the algorithms that Annie and > her students want to write, the "witness" concept is essential? I.e. they > can't just use any(P(x) for x in xs) because if it returns True, they want > to know the x that made P(x) be true. Her ! notation is a (perhaps > unpythonic) attempt at exporting this witness from the quantification. Fair point. I was thinking that common cases worked with any(), and more complex cases that needed a witness would be sufficiently rare that the extra verbosity would (a) not be a huge burden, and (b) help to make the intent clearer. >> > ?while some (!slot_num,p1) in decisions: >> > ? ? if some (!slot_num,p2) in proposals has p2 != p1: >> > ? ? ? ?propose(p2) >> > ? ? perform(p1) [...] >> Can I suggest you write your example out in Python that works today, >> and then show how it looks with your proposed syntax alongside? If you >> can't find the "best" way of writing it in existing Python, just write >> it however works, no need to try to make it compact, or elegant. >> There'll be plenty of people here who will show you how to write >> idiomatic Python versions of what you post :-) > > Actually she gave one in her first post. Here it is again: I'm sorry about that! I got confused part-way through the original post and skimmed from there, and then didn't go back and reread it in the context of the follow-up, so I missed the example. My mistake. > ? ? ?? while {p1 for (s0,p1) in decisions if s0==slot_num}: > ? ? ? ? ? p1 = {p1 for (s0,p1) in decisions if s0==slot_num}.pop() > ? ? ? ? ? for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 != > p1}: > > Note that the set {p1 for (s0,p1) in decisions if s0==slot_num} is computed > twice, once to decide whether to stop, and then again to compute the witness > (p1). Obviously this is inefficient, and that's what she's after. Agreed, that's inefficient, and it also violates DRY - I can easily imagine those two lines getting out of sync after a while... > To make > this same code more efficient in Python, you'd have to do the following, > which is natural for us programmers (since we're so used to working around > limitations and inefficiencies in the systems we work with) but unnatural > for mathematicians, who count to infinity at the drop of a hat: Hmm, I have an aversion to languages (or constructs) based around theoretical principles. Blame a couple of encounters with Haskell a few years ago. I lost. :-) But it tends to be the over-compressed syntax rather than the ideas that I'm particularly allergic to. > while True: > ? temp = {p1 for (s0,p1) in decisions if s0==slot_num} > ? if not temp: > ??? break > ? p1 = temp.pop() > > ? for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 != p1}: > ? ? Point taken. On the other hand, if (x := val) was an expression form of assignment, like C's assignment, you could write while temp := {p1 for (s0,p1) in decisions if s0==slot_num}: p1 = temp.pop() for s2 ... which is to my mind as succinct, and clearer to a non-mathematician, as your proposal below (and Annie's proposal). It also builds off a much more commonly requested feature :-) Actually, while any(p1 := p for (s,p) in decisions if s == slot_num): for s2 ... works, and is just as short as your proposal or Annie's. > The 5 tedious lines from "while" through "pop()" would be collapsed into a > single line if you could write > > ? while some s0, p1 in decisions if s0 == slot_num: > > ? for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 != p1}: > ? ? > > TBH I'm not sure what the !slot_num notation is for -- it appears that > > ? while some (!slot_num, p1) in decisions: > > is equivalent in Annie's proposal to > > ? while some s0, p1 in decisions if s0 == slot_num: > > but I'm not sure and it doesn't feel necessary to me. Yes, I think that's right. It looks like the idea comes from the concept of unification in logic languages (and the ! notation means "don't unify this value, but rather treat it as a fixed value that must match"). Thanks for your analysis, by the way, I think understand the proposal a lot better now. So, going back to what Annie was referring to, there seem to be three key concepts: Quantifications, which are covered in Python by any() and all() Capturing a "witness", which can be done using assignment-as-expression Tuple matching, which you have shown can be handled using tuple unpacking plus the generator expression if clause, but could probably gain from a more compact notation. BTW, as I'm sure everyone knows, you can simulate assignment-as-expression with def asgn(o, val): o.ans = val return val >>> any(asgn(c,x) for x in (1,2,3) if x%2 == 0) True >>> c.ans 2 > Also note that SOME and EACH quantifiers were present in ABC (Python's > predecessor: http://homepages.cwi.nl/~steven/abc/qr.html#TESTS); I dropped > them for simplicity, not because I didn't like them. If we wanted to we > could have them back (except for the problems of introducing new keywords). One day, I really must read up on ABC. Paul. From guido at python.org Sun Jan 15 00:07:49 2012 From: guido at python.org (Guido van Rossum) Date: Sat, 14 Jan 2012 15:07:49 -0800 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> Message-ID: On Sat, Jan 14, 2012 at 1:38 PM, Paul Moore wrote: > On 14 January 2012 19:24, Guido van Rossum wrote: > > But Paul, aren't you missing the fact that for the algorithms that Annie > and > > her students want to write, the "witness" concept is essential? I.e. they > > can't just use any(P(x) for x in xs) because if it returns True, they > want > > to know the x that made P(x) be true. Her ! notation is a (perhaps > > unpythonic) attempt at exporting this witness from the quantification. > > Fair point. I was thinking that common cases worked with any(), and > more complex cases that needed a witness would be sufficiently rare > that the extra verbosity would (a) not be a huge burden, and (b) help > to make the intent clearer. > > >> > while some (!slot_num,p1) in decisions: > >> > if some (!slot_num,p2) in proposals has p2 != p1: > >> > propose(p2) > >> > perform(p1) > [...] > >> Can I suggest you write your example out in Python that works today, > >> and then show how it looks with your proposed syntax alongside? If you > >> can't find the "best" way of writing it in existing Python, just write > >> it however works, no need to try to make it compact, or elegant. > >> There'll be plenty of people here who will show you how to write > >> idiomatic Python versions of what you post :-) > > > > Actually she gave one in her first post. Here it is again: > > I'm sorry about that! I got confused part-way through the original > post and skimmed from there, and then didn't go back and reread it in > the context of the follow-up, so I missed the example. My mistake. > > > while {p1 for (s0,p1) in decisions if s0==slot_num}: > > p1 = {p1 for (s0,p1) in decisions if s0==slot_num}.pop() > > for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 > != p1}: > > > > Note that the set {p1 for (s0,p1) in decisions if s0==slot_num} is > computed > > twice, once to decide whether to stop, and then again to compute the > witness > > (p1). Obviously this is inefficient, and that's what she's after. > > Agreed, that's inefficient, and it also violates DRY - I can easily > imagine those two lines getting out of sync after a while... > Actually it's worse. neither expression needs to compute the full set -- they only need to iterate until the first match. > > To make > > this same code more efficient in Python, you'd have to do the following, > > which is natural for us programmers (since we're so used to working > around > > limitations and inefficiencies in the systems we work with) but unnatural > > for mathematicians, who count to infinity at the drop of a hat: > > Hmm, I have an aversion to languages (or constructs) based around > theoretical principles. Blame a couple of encounters with Haskell a > few years ago. I lost. :-) In my case the jury is still out, but I'm not worried about Haskell ever overtaking Python. :-) FWIW, comprehensions also come from these theoretical principles, so it's not all bad... > But it tends to be the over-compressed > syntax rather than the ideas that I'm particularly allergic to. > > > while True: > > temp = {p1 for (s0,p1) in decisions if s0==slot_num} > > if not temp: > > break > > p1 = temp.pop() > > > > for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 != p1}: > > > > Point taken. On the other hand, if (x := val) was an expression form > of assignment, like C's assignment, you could write > > while temp := {p1 for (s0,p1) in decisions if s0==slot_num}: > p1 = temp.pop() > > for s2 ... > > which is to my mind as succinct, and clearer to a non-mathematician, > as your proposal below (and Annie's proposal). It also builds off a > much more commonly requested feature :-) > > Actually, > > while any(p1 := p for (s,p) in decisions if s == slot_num): > for s2 ... > > works, and is just as short as your proposal or Annie's. > Yeah, and it also avoids computing the elements of the set beyond the first. > > The 5 tedious lines from "while" through "pop()" would be collapsed into > a > > single line if you could write > > > > while some s0, p1 in decisions if s0 == slot_num: > > > > for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 != p1}: > > > > > > TBH I'm not sure what the !slot_num notation is for -- it appears that > > > > while some (!slot_num, p1) in decisions: > > > > is equivalent in Annie's proposal to > > > > while some s0, p1 in decisions if s0 == slot_num: > > > > but I'm not sure and it doesn't feel necessary to me. > > Yes, I think that's right. It looks like the idea comes from the > concept of unification in logic languages (and the ! notation means > "don't unify this value, but rather treat it as a fixed value that > must match"). Thanks for your analysis, by the way, I think understand > the proposal a lot better now. > And thanks for the link with logic languages, that's an area I know even less about... > So, going back to what Annie was referring to, there seem to be three > key concepts: > > Quantifications, which are covered in Python by any() and all() > Capturing a "witness", which can be done using assignment-as-expression > Tuple matching, which you have shown can be handled using tuple > unpacking plus the generator expression if clause, but could probably > gain from a more compact notation. > I'm not sure we need a new construct for tuple matching. Witness capturing seems the most important missing feature here. > BTW, as I'm sure everyone knows, you can simulate assignment-as-expression > with > > def asgn(o, val): > o.ans = val > return val > > >>> any(asgn(c,x) for x in (1,2,3) if x%2 == 0) > True > >>> c.ans > 2 > Eew. :-( > > Also note that SOME and EACH quantifiers were present in ABC (Python's > > predecessor: http://homepages.cwi.nl/~steven/abc/qr.html#TESTS); I > dropped > > them for simplicity, not because I didn't like them. If we wanted to we > > could have them back (except for the problems of introducing new > keywords). > > One day, I really must read up on ABC. > Just follow the above link and scroll to the top. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jan 15 00:27:48 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 14 Jan 2012 23:27:48 +0000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> Message-ID: On 14 January 2012 23:07, Guido van Rossum wrote: >> BTW, as I'm sure everyone knows, you can simulate assignment-as-expression >> with >> >> def asgn(o, val): >> ?o.ans = val >> ?return val >> >> >>> any(asgn(c,x) for x in (1,2,3) if x%2 == 0) >> True >> >>> c.ans >> 2 > > > Eew. :-( Agreed. Assignment-as-expression is nicer. But I have to say I've never really missed it in Python until this discussion started. Question. Do you object to assignment-as-expression per se, or merely as the default way of looking at assignment? Or to put it another way, would something like my x := val (or maybe x <- val or something else) be acceptable? Paul. From cmjohnson.mailinglist at gmail.com Sun Jan 15 01:03:10 2012 From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson) Date: Sat, 14 Jan 2012 14:03:10 -1000 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: References: <1462421.BIX0U07Ti7@lixie> <0567BC83-2437-4881-8F8D-F6E47DB76B72@gmail.com> Message-ID: On Jan 14, 2012, at 10:13 AM, Nathan Rice wrote: > I did mention in the first post that I am the author of a library that > does this right? :D Sorry, my mistake for only skimming the thread. I just kept wondering why none of the later emails mentioned that it was possible? From steve at pearwood.info Sun Jan 15 03:01:24 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 15 Jan 2012 13:01:24 +1100 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> Message-ID: <4F123374.5030402@pearwood.info> Guido van Rossum wrote: > On Sat, Jan 14, 2012 at 1:38 PM, Paul Moore wrote: [...] >> So, going back to what Annie was referring to, there seem to be three >> key concepts: >> >> Quantifications, which are covered in Python by any() and all() >> Capturing a "witness", which can be done using assignment-as-expression >> Tuple matching, which you have shown can be handled using tuple >> unpacking plus the generator expression if clause, but could probably >> gain from a more compact notation. >> > > I'm not sure we need a new construct for tuple matching. Witness capturing > seems the most important missing feature here. If I recall correctly, there have been occasional discussions about changing any() and all() to return the item found rather than a flag. Given the need for backwards compatibility, I don't think we can or should do this, but a hypothetical quant module, or a few new built-ins, could possibly help here. I'm not sure that quantifications are quite important enough to justify new syntax. For lack of better names: def all2(iterable, pred=bool): """Like all(pred(obj) for obj in iterable), returning True if it is true, otherwise obj, the first witness that it false. """ for obj in iterable: if not pred(obj): return obj return True def any2(iterable, pred=bool): """Like any(pred(x) for x in iterable), returning False if it is false, otherwise obj, the first witness that it is true. """ for obj in iterable: if pred(obj): return obj # I look forward to the bike-shedding about returning None # vs returning False ;) return False One disadvantage of returning a single value to represent both the success or failure of the quantification *and* the witness is that it leaves the caller vulnerable to this sort of bug: py> witness = any2([3, 0, 2, 4], lambda n: n%2==0) # get first even number py> if witness: ... print("first even number is,", witness) ... else: ... print("everything is odd") ... everything is odd I don't have a good solution for this. -- Steven From ncoghlan at gmail.com Sun Jan 15 03:27:10 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 15 Jan 2012 12:27:10 +1000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> Message-ID: On Sun, Jan 15, 2012 at 9:27 AM, Paul Moore wrote: > Agreed. Assignment-as-expression is nicer. But I have to say I've > never really missed it in Python until this discussion started. > > Question. Do you object to assignment-as-expression per se, or merely > as the default way of looking at assignment? Or to put it another way, > would something like my x := val (or maybe x <- val or something else) > be acceptable? Just to make sure I have the example right, I believe this is what the original example looks like without using comprehensions at all: for dec_slot, p1 in decisions: if dec_slot == slot_num: for prop_slot, p2 in proposals: if prop_slot == slot_num and p2 != p1: # Do something! You can merge the for loops and the if statements, and hence avoid the naming conflict for the slot variables with a couple of generator expressions (there's apparently no need to waste memory realising the set for this example). You can also trivially adopt a mathematics style ordering for any() and all() by using "True" as the body and moving the predicate to the comprehension's if clause. That means, the example can already be written as: for p1 in (p for slot, p in decisions if slot == slot_num): if any(True for slot, p2 in proposals if slot == slot_num and p2 != p1): # Do something! The tricks are: 1. Use generator expressions in order to get at the values as they're produced, rather than realising the sets in memory 2. Use a for loop instead of a while loop in order to capture those values 3. Use "True" with any()/all() to get a mathematical style quantification ordering of expressions Cheers, Nick. P.S. My reply appears in this part of the thread because it started out as a reference to my previous suite expressions concept [1] that supports embedded assignment statements in a way that allows the value saved and the value used as a predicate to differ. As I wrote it up though, I realised that the specific example given could be translated fairly neatly into *existing* Python constructs, as shown above. [1] http://ncoghlan_devs-python-notes.readthedocs.org/en/latest/pep_ideas/suite_expr.html -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sun Jan 15 03:36:22 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 15 Jan 2012 12:36:22 +1000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: <4F123374.5030402@pearwood.info> References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> Message-ID: On Sun, Jan 15, 2012 at 12:01 PM, Steven D'Aprano wrote: > One disadvantage of returning a single value to represent both the success > or failure of the quantification *and* the witness is that it leaves the > caller vulnerable to this sort of bug: > > py> witness = any2([3, 0, 2, 4], lambda n: n%2==0) ?# get first even number > py> if witness: > ... ? ? print("first even number is,", witness) > ... else: > ... ? ? print("everything is odd") > ... > everything is odd > > > I don't have a good solution for this. The way around it is to always return a tuple: empty if no answer was found, a singleton-tuple if it was. That way truth-testing will always give the right found/not-found answer, but in the found case you can access the original object. def any2(iterable, pred=bool): """Like any(pred(x) for x in iterable), but returns () if nothing matches the predicate, otherwise (obj,), a singleton-tuple holding the first value that matched it. """ for obj in iterable: if pred(obj): return (obj,) return () py> found = any2([3, 0, 2, 4], lambda n: n%2==0) # get first even number py> if found: ... print("first even number is,", found[0]) ... else: ... print("everything is odd") ... first even number is 0 Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From python at mrabarnett.plus.com Sun Jan 15 03:45:28 2012 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 15 Jan 2012 02:45:28 +0000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: <4F123374.5030402@pearwood.info> References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> Message-ID: <4F123DC8.5020603@mrabarnett.plus.com> On 15/01/2012 02:01, Steven D'Aprano wrote: > Guido van Rossum wrote: >> On Sat, Jan 14, 2012 at 1:38 PM, Paul Moore wrote: > [...] >>> So, going back to what Annie was referring to, there seem to be three >>> key concepts: >>> >>> Quantifications, which are covered in Python by any() and all() >>> Capturing a "witness", which can be done using assignment-as-expression >>> Tuple matching, which you have shown can be handled using tuple >>> unpacking plus the generator expression if clause, but could probably >>> gain from a more compact notation. >>> >> >> I'm not sure we need a new construct for tuple matching. Witness capturing >> seems the most important missing feature here. > > > If I recall correctly, there have been occasional discussions about changing > any() and all() to return the item found rather than a flag. Given the need > for backwards compatibility, I don't think we can or should do this, but a > hypothetical quant module, or a few new built-ins, could possibly help here. > I'm not sure that quantifications are quite important enough to justify new > syntax. > > For lack of better names: > > > def all2(iterable, pred=bool): > """Like all(pred(obj) for obj in iterable), returning True if it is true, > otherwise obj, the first witness that it false. > """ > for obj in iterable: > if not pred(obj): > return obj > return True > > def any2(iterable, pred=bool): > """Like any(pred(x) for x in iterable), returning False if it is false, > otherwise obj, the first witness that it is true. > """ > for obj in iterable: > if pred(obj): > return obj > # I look forward to the bike-shedding about returning None > # vs returning False ;) > return False > > > One disadvantage of returning a single value to represent both the success or > failure of the quantification *and* the witness is that it leaves the caller > vulnerable to this sort of bug: > > py> witness = any2([3, 0, 2, 4], lambda n: n%2==0) # get first even number > py> if witness: > ... print("first even number is,", witness) > ... else: > ... print("everything is odd") > ... > everything is odd > > > I don't have a good solution for this. > A 'default' parameter perhaps? >>> witness = any2([3, 0, 2, 4], lambda n: n % 2 == 0, default=None) >>> if witness is not None: ... print("first even number is", witness) ... else: ... print("everything is odd") ... first even number is 0 From nathan.alexander.rice at gmail.com Sun Jan 15 08:04:11 2012 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Sun, 15 Jan 2012 02:04:11 -0500 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: <1462421.BIX0U07Ti7@lixie> References: <1462421.BIX0U07Ti7@lixie> Message-ID: > The pattern you employ with SymbolicObject, is a nice way to mix code that is > executed immediately, with code that is executed later. Symbolic algebra > languages, like Maple and Mathematica, function this way too. I find it to be useful quite frequently. I think making generative SQL expressions with SQL Alchemy got me hooked. > I propose however a slightly different interface: > > There should be a special object "quoted" to create instances of > "SymbolicObject". The objects should be created in the "__getattribute__" > method. This way the objects know their name, which is quite useful. A > symbolic object would be created like this: > > ? ?X = quoted.X > > You could also write: > > ? ?const = Constraints(quoted.X * 2 + 1 >= 5, quoted.X % 2 != 0) The factory there seems to add complexity. The Class(args) constructor style is less magic, and I can imagine wanting to pass things to the symbolic object, like a name, default value, a docstring, etc. > The standard library should contain at least these symbolic algorithms: > > evalsym(tree, environment, return_type="any") > ? ?Evaluates an expression and returns the result. Works similar to: > > ? ?eval(compile(tree, "", "eval"), environment) > > ? ?This call would return 7: > > ? ? ? ?evalsym(quoted.X * 2 + 1, {"X":3}) > > ? ?However it should do partial evaluation if some symbols are > ? ?unknown, and return an ast.AST in this case (similar to Maple and > ? ?Mathematica). This expression should be True: > > ? ? ? ?evalsym(quoted.X + quoted.Y, {"Y":3}) == quoted.X + 3 > > ? ?The optional argument "return_type" specifies the type of the > ? ?result. This simplifies algorithms using "eval". Argument > ? ?"return_type" can be: "any", "ast", or "object". I think a method call leads to more readable code than a function, because they grow left to right and are structured like English. Something along the lines of: expr = X ** (M*(N+1)) expr.realize({N:2, M:3, X:4}) expr.realize((N,2)).realize((M, 3)).realize((X, 4)) Tasty curry. As a side note, perhaps PyPy is a good place to prototype something like this? :) Nathan From ncoghlan at gmail.com Sun Jan 15 08:44:30 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 15 Jan 2012 17:44:30 +1000 Subject: [Python-ideas] Special method to delegate PEP 3118 buffer access to another object? Message-ID: I'm still working on fixing CPython's operand precedence bug for pure sequence types implemented in C [1], and, while trying to test bytearray, came across the problem that there doesn't appear to be a way to write a class in pure Python that redirects the PEP 3118 buffer access slots to another object (e.g. a memoryview instance or a bytes object). Did I miss something? And, if not, would it make sense to have the buffer retrieval operations look for a __buffer__ method in the type dictionary as a last resort if the relevant C level slots aren't provided? Cheers, Nick. [1] http://bugs.python.org/issue11477 -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From techtonik at gmail.com Sun Jan 15 09:31:50 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Sun, 15 Jan 2012 11:31:50 +0300 Subject: [Python-ideas] Pythonic buffering in Py3 print() In-Reply-To: References: <20120112014017.GA14688@cskk.homeip.net> Message-ID: Thanks everybody for discussion. It took only 6 days between noticing the problem [1] and finding the cause [2]. I wish the process didn't require so intensive communication, because without holidays I'd probably gave up on this. =) It's good that I know English. 1. https://bitbucket.org/techtonik/python-pager/changeset/3e3d0700ba5f 2. https://bitbucket.org/techtonik/python-pager/changeset/ffe9da2d5316 -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Jan 15 09:59:43 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Jan 2012 08:59:43 +0000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> Message-ID: On 15 January 2012 02:27, Nick Coghlan wrote: > That means, the example can already be written as: > > ? ?for p1 in (p for slot, p in decisions if slot == slot_num): > ? ? ? ?if any(True for slot, p2 in proposals if slot == slot_num and p2 != p1): > ? ? ? ? ? ?# Do something! Actually, that may not work - the original requirement was to scan decisions on each iterations, and pick one item that satisfied the condition each time. If decisions is changing each iteration, you need to rescan each time, which your for loop doesn't do. I think that's why the (otherwise odd) "while there are any, pick one" construct is used. Of course, for an algorithm like that, I'd probably tend to choose a different data structure, maybe something like a work queue. Paul. From simon.sapin at kozea.fr Sun Jan 15 11:54:20 2012 From: simon.sapin at kozea.fr (Simon Sapin) Date: Sun, 15 Jan 2012 11:54:20 +0100 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: <4F123374.5030402@pearwood.info> References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> Message-ID: <4F12B05C.5090802@kozea.fr> Le 15/01/2012 03:01, Steven D'Aprano a ?crit : > def any2(iterable, pred=bool): > """Like any(pred(x) for x in iterable), returning False if it is false, > otherwise obj, the first witness that it is true. > """ > for obj in iterable: > if pred(obj): > return obj > # I look forward to the bike-shedding about returning None > # vs returning False;) > return False Hi, If the point is to get some value that matches a predicate, how about this? next(value for value in iterable if predicate(value)) If a "no value" marker is desired instead of StopIteration: next((value for value in iterable if predicate(value)), None) But yes, assignment-expression is still required to avoid the while True/if/break dance. Regards, -- Simon Sapin From solipsis at pitrou.net Sun Jan 15 16:26:37 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 15 Jan 2012 16:26:37 +0100 Subject: [Python-ideas] Special method to delegate PEP 3118 buffer access to another object? References: Message-ID: <20120115162637.3b13c4b4@pitrou.net> On Sun, 15 Jan 2012 17:44:30 +1000 Nick Coghlan wrote: > I'm still working on fixing CPython's operand precedence bug for pure > sequence types implemented in C [1], and, while trying to test > bytearray, came across the problem that there doesn't appear to be a > way to write a class in pure Python that redirects the PEP 3118 buffer > access slots to another object (e.g. a memoryview instance or a bytes > object). > > Did I miss something? And, if not, would it make sense to have the > buffer retrieval operations look for a __buffer__ method in the type > dictionary as a last resort if the relevant C level slots aren't > provided? Would make sense indeed. Regards Antoine. From liu at cs.stonybrook.edu Sun Jan 15 18:01:56 2012 From: liu at cs.stonybrook.edu (Annie Liu) Date: Sun, 15 Jan 2012 17:01:56 +0000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: <4F12B05C.5090802@kozea.fr> References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> <4F12B05C.5090802@kozea.fr> Message-ID: <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> Hi! Thanks for all the ideas and discussions! Here is a summary of my thinking, followed by some specific comments. 1. I think we should support proper quantifications, because there are no conceptual or technical difficulties, and it is win-win-win to do. No conceptual difficulties: . They are first-order logic, used in people's day-to-day reasoning. . They are in one of the first two CS courses, the other being programming. . They are used commonly in pseudocode. They are very far from requiring mathematicians or infinity. :-) It would be wonderful if students taking the first two CS courses can actually write quantifications in programs at the drop of a hat! No technical difficulties: . A straightforward implementation suffices, stopping at the first witness. . Alternatives/workarounds have many different kinds of difficulties. . They are in SETL, the first language that used sets for programming, developed over 40 years ago. They don't break any Python rules, as far as I know. :-) In fact, "some x in s" is better than "s.pop()" in terms of side effect, and it gives the programmer a test and a choice of not changing s. (not to mention that computing a comprehension to get s followed by a pop is less efficient than doing "some x in s" directly) It is win-win-win to do, because they help . make programming easier . make programs clearer . make programs more efficient This is a dream situation, considering the usual conflicts among these! 2. OK, I think we can ignore tuple patterns for now (they were why I used "!" for bound variables, for pattern matching). They are a much smaller conceptual and technical help. > Don't forget that in Python, assignments are > statements and don't get embedded in expressions. I like this principle dearly, but sometimes breaking it is better. :-) In fact, the alternative I had to use, s.pop(), is a good example of a statement that does get used in expressions. In general, any time we have a statement that also returns a value, it is the same as an expression that has side effect. There are plenty of those in Python. :-) > while True: > temp = {p1 for (s0,p1) in decisions if s0==slot_num} > if not temp: > break > p1 = temp.pop() > for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 != p1}: > > > The 5 tedious lines from "while" through "pop()" would be collapsed > into a single line if you could write > > while some s0, p1 in decisions if s0 == slot_num: > for p2 in {p2 for (s0,p2) in proposals if s0==slot_num if p2 != p1}: > Right!(with a little indentation in front of the "for") Thank you! In fact, as I mentioned, it happens that using "for" in place of "if" is ok here (in Robbert's algorithm), but in general, to capture the original "if" correctly, one would need another temp variable and some more tedious lines. > """Like all(pred(obj) for obj in iterable), returning True if it is true, > otherwise obj, the first witness that it false. Thanks for the idea of witness for "all"! I have actually not needed to use it, but I can see that the symmetry helps general uses, and we'd get it for free just like for "any". > > for p1 in (p for slot, p in decisions if slot == slot_num): > > if any(True for slot, p2 in proposals if slot == slot_num and p2 != p1): > > # Do something! > > Actually, that may not work - the original requirement was to scan > decisions on each iterations, and pick one item that satisfied the > condition each time. If decisions is changing each iteration, you need > to rescan each time, which your for loop doesn't do. Right! Thank you! > I think that's why the (otherwise odd) "while there are any, pick one" > construct is used. Of course, for an algorithm like that, I'd > probably tend to choose a different data structure, maybe something > like a work queue. In fact, "while some x in s" is exactly a best way to write a work list algorithm, where either a queue or stack could be used (in graph algorithms, a queue corresponds to breadth-first-search, and a stack corresponds to depth-first-search), but when the order of picking elements doesn't matter, "while some x in s" is easier and clearer. > Also note that SOME and EACH quantifiers were present in ABC (Python's > predecessor: http://homepages.cwi.nl/~steven/abc/qr.html#TESTS); I > dropped them for simplicity, not because I didn't like them. If we > wanted to we could have them back (except for the problems of > introducing new keywords). I hope new keywords are not the stopping factor. I remember some of my programs used "as" a lot (for many nouns starting with a, like arguments), and I avoided running them with py3 for several years, until I had to one day, and they were so easy to fix that I regretted not doing it sooner. I wonder what the best keywords (and separator) are to use: SETL: forall, exists e.g., exists x in s | pred html: &forall, &exist latex: \forall, \exists ABC: each, some e.g., some x in s has pred Ada 2012: for all, for some e.g., for some x in s => pred I had used "forall" and "exists", but quite liked "each" and "some" after I learned them from Lambert Meertens two years ago: besides being shorter, "each" gives the right grammar (singular) when read out. But I also feel conflicted, since "all" and "exist" are used for so long in history, to justify the reversed "A" and "E" in symbols. BTW, one could use "all" and "any", even shorter and has a reversed "A":-), but "any" is ambiguous when read out. For example, we say "some apple in the bag is red", not "any apple in the bag is red". For separator, I had used "if" (not "has") because it is consistent with uses of conditions in general in Python, and with uses of "|" in SETL. SETL has: comprehension {ret_exp: x in s | pred} quantification exists x in s | pred forall x in s | pred for loop for x in s | pred loop ... end loop and you get "while exists x in s | pred" for free. "| pred" can be omitted when "pred" is true. Interestingly, I noticed that Guido used "if" in his code (after the "5 tedious lines"). I feel that "has" reads well for single quantification, but not as well for nested (though "if" sounds a bit weird also, and I'd really like to use "|" if it doesn't break Python rules :-)). Compare: each x in s has (some y in t has pred) each x in s if (some y in t if pred) each x in s | (some y in t | pred) BTW, functions "all" and "any" are much harder to read here: all(any(pred for y in t) for x in s) and it is even harder when you have some real contents for pred, s, and t, not just the template here. This reminds me that Guy Steele had a popular lecture making fun of languages that have features that require people to swing their eyeballs back and forth on a line to read it. :-) In summary, again, I think we should support proper quantifications. BTW, some of the workarounds proposed here are looking like what a Haskell genius was trying to show me yesterday how to get around. :-) Thanks, Annie From p.f.moore at gmail.com Sun Jan 15 20:46:38 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 15 Jan 2012 19:46:38 +0000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> <4F12B05C.5090802@kozea.fr> <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> Message-ID: On 15 January 2012 17:01, Annie Liu wrote: > ? It is win-win-win to do, because they help > ? . make programming easier > ? . make programs clearer > ? . make programs more efficient > ? This is a dream situation, considering the usual conflicts among these! Hmm, while I'm coming to like the concept, I still think that "clearer" might be a bit strong. For simple uses where all() and any() work as replacements, those builtins are already pretty clear. For cases where they won't do, then quantifiers as you propose are certainly no worse than some of the other proposed alternatives, but I'm not sure how much better they are, either. But never mind, clarity is a matter of perspective at best :-) > ?> I think that's why the (otherwise odd) "while there are any, pick one" > ?> construct is used. Of course, for an algorithm like that, I'd > ?> probably tend to choose a different data structure, maybe something > ?> like a work queue. > > In fact, "while some x in s" is exactly a best way to write a work > list algorithm, where either a queue or stack could be used (in graph > algorithms, a queue corresponds to breadth-first-search, and a stack > corresponds to depth-first-search), but when the order of picking > elements doesn't matter, "while some x in s" is easier and clearer. Yes, I see your point here - in this case "while some x in s" reads nicely, and does what I'd expect. > I feel that "has" reads well for single quantification, but not as > well for nested (though "if" sounds a bit weird also, and I'd really > like to use "|" if it doesn't break Python rules :-)). ?Compare: > > ? each x in s has (some y in t has pred) > > ? each x in s if (some y in t if pred) > > ? each x in s | (some y in t | pred) > > BTW, functions "all" and "any" are much harder to read here: > > ? all(any(pred for y in t) for x in s) > > and it is even harder when you have some real contents for pred, s, > and t, not just the template here. OK, I'm willing to concede that the all/any formulation isn't as "clean looking" as the alternatives you propose. But I did understand it instantly, whereas I found that I couldn't work out the meaning of your form straight off, but had to "cheat" by looking at the all/any formulation, and work back from that. How much of that is simply because I'm more familiar with the conventional form, I'm not sure (it's certainly a reasonably substantial part of it - and if each/some become part of the language, they will become familiar too). But what I want to do is break things down into individual constructs - that's how my mind works (so sue me! :-)) With the all/any form, it goes: all(...) - function call, with a generator expression as argument ... for x in s - generator expression, argument to all any(...) - function call again pred for y in t - generator expression So we have 2 fundamental constructs, both used in many contexts throughout Python, and hence both familiar, combined in simple ways. For your each/some formulation we have: each x in s has ... - a new type of construct, and each expression some y in t has pred - another new type of construct, a some expression each and some expressions feel like they have structure in common with generator expressions, but they look slightly different. Actually, isn't it true that "each x in s has pred" is basically "all(x for x in s if pred)" except that no witness is captured? So syntactically the differences are: - you avoid the repetition of "x for x" (which is a known wart of generator expressions) - you're using has instead of if (it reads better, but otherwise is a bit of a spurious difference - you did propose if as an alternative) - you avoid the parentheses There are some subtler differences (you use "pred for x..." in the all/any version, whereas I'm equating each/some forms with "x for x" variations, for example). But basically this is my real concern - your proposed constructs are special case syntax that is close to, but subtly different from, existing forms. That "nearly but not quite the same" aspect is where I have concerns about readability (and teachability). The problem is that each/some can't build on generators, because if they do you need to explicitly specify the variable to use to capture the witness. And then if you use a generator expression, you end up with something like "some VAR GENERATOR", which becomes a monstrosity like "some x (x for x in ...)". So they have to build on generator *expressions* which are a subset of general generators. And we get to special case synax straight off. The advantage of any/all is that they work on any generator - whether it's a generator expression or some other form of generator). > In summary, again, I think we should support proper quantifications. I'm getting more comfortable with the idea, but I think it needs a bit more discussion to come up with something that fits in cleanly with the rest of Python, while still adding the value you want. Paul. From jeanpierreda at gmail.com Mon Jan 16 00:29:39 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 15 Jan 2012 18:29:39 -0500 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> <4F12B05C.5090802@kozea.fr> <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> Message-ID: On Sun, Jan 15, 2012 at 12:01 PM, Annie Liu wrote: > For separator, I had used "if" (not "has") because it is consistent > with uses of conditions in general in Python, and with uses of "|" in > SETL. ?SETL has: > > ? comprehension ? ? ? ?{ret_exp: x in s | pred} > ? quantification ? ? ? exists x in s | pred > ? ? ? ? ? ? ? ? ? ? ? ?forall x in s | pred > ? for loop ? ? ? ? ? ? for x in s | pred loop ... end loop > > ? and you get "while exists x in s | pred" for free. I hadn't been _really_ paying attention, [no active reading] but the "if" had confused me until I read this -- I didn't know what it meant. I'm so much more used to "|" and "s.t. (such that)". If reads like implication, which is something else. Of course, the same complaint applies to list comprehensions, so I expect I'll get used to it, but... with list comprehensions it seemed obvious because it replaced an idiom that literally used the if statement. Is the same true here? -- Devin From ncoghlan at gmail.com Mon Jan 16 15:15:05 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 17 Jan 2012 00:15:05 +1000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> <4F12B05C.5090802@kozea.fr> <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> Message-ID: On Mon, Jan 16, 2012 at 5:46 AM, Paul Moore wrote: >> In summary, again, I think we should support proper quantifications. > > I'm getting more comfortable with the idea, but I think it needs a bit > more discussion to come up with something that fits in cleanly with > the rest of Python, while still adding the value you want. Indeed. With the clarification that my original "long form" expansion was incorrect (since it only looped over the outer set once), I'll put forward a slightly shorter variant of Guido's expansion (as someone else already noted, next() can be a useful alternative to pop() if you aren't keeping the original set around) _sentinel = object() while True: p1 = next((p for (s0,p1) in decisions if s0==slot_num), _sentinel) if p1 is _sentinel: break for p2 in {p2 for (s0,p2) in proposals if s0==slot_num and p2 != p1}: ... So, let's consider this proposal: "some VAR in ITERABLE if COND" would be an expression that: 1. *deliberately* leaks the variable into the surrounding scope (embedded assignment ahoy) 2. Defines the expression value roughly as follows: _itr = (VAR for VAR in ITERABLE if COND) # i.e. it's an implicit generator expression try: VAR = next(_itr) except StopIteration: _expr_result = False else: _expr_result = True And, further suppose, that we finally cave on allowing a filter clause in for statements Then Annie's original example would look like this: while some s0, p1 in decisions if s0==slot_num: for s0, p2 in proposals if s0==slot_num and p2 != p1: ... And, since we *know* anything even vaguely resembling an embedded assignment syntax would almost immediately be adopted for any-and-all of the various embedded assignment requests we've seen over the years, here's how it would look for the regex use case: if some m in [pattern.search(data)] if m is not None: ... Hmm, still doesn't feel right to me - trying to shove a square peg (mathematical quantification) into a round hole (Python's syntax). In particular, the any()/all() way of doing things avoids the double-if problem when used in an if statement, whereas the version that captures the variable (and hence moves the predicate to the end) reads very strangely in that case. Removing the if clause doesn't help, since you can easily add it back by using a generator expression as your iterable. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jeanpierreda at gmail.com Mon Jan 16 15:47:39 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 16 Jan 2012 09:47:39 -0500 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> <4F12B05C.5090802@kozea.fr> <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> Message-ID: On Mon, Jan 16, 2012 at 9:15 AM, Nick Coghlan wrote: > And, since we *know* anything even vaguely resembling an embedded > assignment syntax would almost immediately be adopted for any-and-all > of the various embedded assignment requests we've seen over the years, > here's how it would look for the regex use case: > > ? ?if some m in [pattern.search(data)] if m is not None: > ? ? ? ?... > > Hmm, still doesn't feel right to me - trying to shove a square peg > (mathematical quantification) into a round hole (Python's syntax). In > particular, the any()/all() way of doing things avoids the double-if > problem when used in an if statement, whereas the version that > captures the variable (and hence moves the predicate to the end) reads > very strangely in that case. You can always add a full-on assignment-as-expression, if you're concerned about that. if (m := pat.search(data)) is not None: Also, the keyword doesn't have to be "if". I've always read the bar as "such that" or "where". -- Devin From p.f.moore at gmail.com Mon Jan 16 20:29:11 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 16 Jan 2012 19:29:11 +0000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> <4F12B05C.5090802@kozea.fr> <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> Message-ID: On 16 January 2012 14:47, Devin Jeanpierre wrote: > You can always add a full-on assignment-as-expression, if you're > concerned about that. > > ? ?if (m := pat.search(data)) is not None: > > Also, the keyword doesn't have to be "if". I've always read the bar as > "such that" or "where". I suggested that a while back in this thread (in the form of any(var := val for val in itr if pred)). It didn't get much traction. Paul. From jimjjewett at gmail.com Mon Jan 16 21:06:29 2012 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 16 Jan 2012 15:06:29 -0500 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: <4F123374.5030402@pearwood.info> References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> Message-ID: On Sat, Jan 14, 2012 at 9:01 PM, Steven D'Aprano wrote: >> On Sat, Jan 14, 2012 at 1:38 PM, Paul Moore wrote: >>> Capturing a (quantification) "witness", which can be done using >>> assignment-as-expression > If I recall correctly, there have been occasional discussions about > changing any() and all() to return the item found rather than a flag. Even in this thread, there have been multiple posts that would shown bugs when the witness itself had a boolean evaluation of False. What if any and all started to return a Witness object that evaluated to True/False, but made the value available? I could see an object like this being useful in other code as well. class Witness(object): def __bool__(self): return self.truth def __init__(self, truth, value): self.truth=truth self.value=value Of course, it might be nicer to inherit from type(True), and it still doesn't quite solve the laziness problem with for, or the assign-in-the-right-place problem with while. -jJ From tom at rothamel.us Tue Jan 17 02:57:07 2012 From: tom at rothamel.us (Tom Rothamel) Date: Mon, 16 Jan 2012 20:57:07 -0500 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns Message-ID: (Sorry about the formatting - I'm a former student of Annie's, who just joined the list.) Jim Jewett wrote: > Even in this thread, there have been multiple posts that would shown > bugs when the witness itself had a boolean evaluation of False. > What if any and all started to return a Witness object that evaluated > to True/False, but made the value available? I could see an object > like this being useful in other code as well. > class Witness(object): > def __bool__(self): return self.truth > def __init__(self, truth, value): > self.truth=truth > self.value=value > Of course, it might be nicer to inherit from type(True), and it still > doesn't quite solve the laziness problem with for, or the > assign-in-the-right-place problem with while. I'm wondering if would make sense to add an "as" clause to the while and if statements, similar to the as clause in the with statement. It would essentially retrieve the value field from the witness object described above. This would let one write the main loop in a topological sort (a classic workset algorithm) as: while any(v for v in vertices if v.incoming_count == 0) as v1: result.append(v) for v2 in v1.outgoing: v2.incoming_count -= 1 Another use of this could be for database queries that return 0 or 1 results: if db.query(page_name=page_name) as page: render_template(page) else: error404() This is a common pattern that often involves either exception handling or a sentinel value like None. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at 2sn.net Tue Jan 17 05:00:52 2012 From: python at 2sn.net (Alexander Heger) Date: Mon, 16 Jan 2012 22:00:52 -0600 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: Message-ID: <4F14F274.2070908@2sn.net> Dear Tom, > I'm wondering if would make sense to add an "as" clause to the while and if > statements, similar to the as clause in the with statement. It would > essentially retrieve the value field from the witness object described > above. > > This would let one write the main loop in a topological sort (a classic > workset algorithm) as: > > while any(v for v in vertices if v.incoming_count == 0) as v1: > result.append(v) > for v2 in v1.outgoing: > v2.incoming_count -= 1 I had the same thought going through the thread, but I think the problem here would be that any is a function that returns a Boolean value. You could add a key parameter to return something else, say a tuple of a truth value and a sample ( any( ..., sample = True) ) but that could break because 1) "any" could be another function in the context, 2) while would need to deal with special return values in this case A suggestion to replace the "while any" case is sample as x: # do something with x ("sample" may not be the best keyword choice) From the discussion so far I do not see how to easily avoid having a new keyword specifically, your example would become sample v for v in vertices if v.incoming_count == 0 as v1: # do things with v1 In case you only want one sample, you could add the break statement sample db.query(page_name=page_name) as page: render_template(page) break else: error404() -Alexander From ethan at stoneleaf.us Tue Jan 17 06:39:18 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 16 Jan 2012 21:39:18 -0800 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> <4F12B05C.5090802@kozea.fr> <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> Message-ID: <4F150986.303@stoneleaf.us> Paul Moore wrote: > On 16 January 2012 14:47, Devin Jeanpierre wrote: >> You can always add a full-on assignment-as-expression, if you're >> concerned about that. >> >> if (m := pat.search(data)) is not None: >> >> Also, the keyword doesn't have to be "if". I've always read the bar as >> "such that" or "where". > > I suggested that a while back in this thread (in the form of any(var > := val for val in itr if pred)). It didn't get much traction In the matter of := being used for assignment-as-expression... +1! ~Ethan~ From ncoghlan at gmail.com Tue Jan 17 12:16:39 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 17 Jan 2012 21:16:39 +1000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: <4F150986.303@stoneleaf.us> References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> <4F12B05C.5090802@kozea.fr> <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> <4F150986.303@stoneleaf.us> Message-ID: On Tue, Jan 17, 2012 at 3:39 PM, Ethan Furman wrote: > Paul Moore wrote: >> >> On 16 January 2012 14:47, Devin Jeanpierre wrote: >>> >>> You can always add a full-on assignment-as-expression, if you're >>> concerned about that. >>> >>> ? if (m := pat.search(data)) is not None: >>> >>> Also, the keyword doesn't have to be "if". I've always read the bar as >>> "such that" or "where". >> >> >> I suggested that a while back in this thread (in the form of any(var >> := val for val in itr if pred)). It didn't get much traction > > > In the matter of := being used for assignment-as-expression... +1! Assignments as expressions aren't all that easy to use in the "some" case that started the thread, and the reason is the same as the reason iteration variables in comprehensions don't leak into the surrounding scope any more: generator expressions and comprehensions create their own scope, separate from that of the containing function. If Python ever did get assignment expressions, the most likely candidate would actually be based on the existing use of "as" for naming (or renaming) things: if (pat.search(data) as m) is not None: ... while (next(p1 for (s0, p1) in decisions if s0==slot_num, None) as p1) is not None: ... However, assignments as expressions are still a long way from being shown to ever improve clarity enough to be worth the additional language complexity. Getting back specifically to the quantification question, I wonder if being able to explicitly export an iteration variable from a comprehension might work: while any(s0 == slot_num for (s0, some p1) in decisions): ... Abusing it for embedded assignments is still ugly, but at least vaguely comprehensible once you're familiar with the meaning of 'some': if any(m is not None for some m in [pattern.search(data)]): ... (As far as how that could actually work goes, with the bindings limited to simple names, as they are for iteration variables, you could probably just make the relevant mapping available inside the comprehension for class and module namespaces and make use of the nonlocal machinery for function namespaces) I still can't honestly say I like the idea, but I at least dislike it less than anything else I've seen in the thread so far :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Tue Jan 17 14:34:21 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 17 Jan 2012 13:34:21 +0000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> <4F12B05C.5090802@kozea.fr> <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> <4F150986.303@stoneleaf.us> Message-ID: On 17 January 2012 11:16, Nick Coghlan wrote: > Assignments as expressions aren't all that easy to use in the "some" > case that started the thread, and the reason is the same as the reason > iteration variables in comprehensions don't leak into the surrounding > scope any more: generator expressions and comprehensions create their > own scope, separate from that of the containing function. Ah. I tested the idea by setting an attribute of an existing object, which *does* work (but is ugly). I'd forgotten that scoping comes into the picture when you switch to assignment. I'll refrain from proposing "nonlocal x := var"... :-) Paul. From ncoghlan at gmail.com Tue Jan 17 14:39:40 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 17 Jan 2012 23:39:40 +1000 Subject: [Python-ideas] Fwd: quantifications, and tuple patterns In-Reply-To: References: <913240E0EEE49B48B1A9D5CFD10658FC3B79F1E6@mail2.cs.stonybrook.edu> <913240E0EEE49B48B1A9D5CFD10658FC3B79F21F@mail2.cs.stonybrook.edu> <4F123374.5030402@pearwood.info> <4F12B05C.5090802@kozea.fr> <913240E0EEE49B48B1A9D5CFD10658FC3B79F43E@mail2.cs.stonybrook.edu> <4F150986.303@stoneleaf.us> Message-ID: On Tue, Jan 17, 2012 at 11:34 PM, Paul Moore wrote: > I'll refrain from proposing "nonlocal x := var"... :-) That idea (or rather the "(var as nonlocal x)" equivalent) was actually the starting point for my train of thought that led to the suggestion of just using "some" as a marker for iteration variable export :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From nathan.alexander.rice at gmail.com Wed Jan 18 14:30:15 2012 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Wed, 18 Jan 2012 08:30:15 -0500 Subject: [Python-ideas] Symbolic expressions (or: partials and closures from the inside out) In-Reply-To: <1462421.BIX0U07Ti7@lixie> References: <1462421.BIX0U07Ti7@lixie> Message-ID: > There should be a special object "quoted" to create instances of > "SymbolicObject". The objects should be created in the "__getattribute__" > method. This way the objects know their name, which is quite useful. A > symbolic object would be created like this: > > ? ?X = quoted.X I did some additional thinking about how the interface could be designed, and I think there might be a good middle ground with regard to your quoted object. I still feel like creating variables should be independent, and use the standard class constructor method. I feel like a Context object would be a good place to keep variable values, and the expression evaluation method. thus, you might have: context = Context() expr = SymbolicObject(name="A") + SymbolicObject(name="B") * SymbolicObject(name="C", default=5) context.A = 1 context.B = 2 print context.eval(expr) 11 context.C = 3 print context.eval(expr) 7 > The (tree of) symbolic objects should have a method that translate it to an > abstract syntax tree (ast.AST from the standard library). Subsequent > algorithms should operate on this AST. This way symbolic algorithms can also > operate on all possible Python code (SymbolicObject can only create > expressions). > > ? ?my_ast = (quoted.X * 2 + 1).to_ast() > > For convenience I think that all symbolic algorithms should also accept > SymbolicObject, and convert them to ast.AST internally. I don't think it would be too bad to create an AST representation inside the symbolic expression while building it. > ? ?However it should do partial evaluation if some symbols are > ? ?unknown, and return an ast.AST in this case (similar to Maple and > ? ?Mathematica). This expression should be True: Having the evaluate method return an AST with incomplete variable information would be confusing to me. I would expect to get a new symbolic expression with the specified parameters replaced. > > substitute(tree, substitutions) > ? ?Replaces a symbol with a value (a sub-tree of ast.Ast). > ? ?This expression should be True: > > ? ? ? ?substitute(quoted.X + quoted.Y, {"Y": quoted.X * 2}) \ > ? ? ? ?== (quoted.X + quoted.X * 2).to_ast() > > ? ?Maybe this algorithm should also be able to match general fragments of > ? ?the tree and replace them. If you build up the AST representation while creating the symbolic expression, replacing a symbolic expression object should have the effect of updating the tree. I think having clearly delineated AST and SymbolicExpr representations might be easier (possibly via converter functions). The actual different may be negligible, but it would serve as an explicit way of declaring what your expectations are with regard to return values, etc. As a side note, it appears that between PyPy object spaces and continulets, it should be possible to create this without any weird interpreter hacking. Hopefully I'll have time to put something together this weekend. Nathan From mwm at mired.org Sat Jan 21 01:36:48 2012 From: mwm at mired.org (Mike Meyer) Date: Fri, 20 Jan 2012 16:36:48 -0800 Subject: [Python-ideas] Retrying EAFP without DRY Message-ID: <20120120163648.69288796@mikmeyer-vm-fedora> Just running another half-baked "easy" idea up to see what others think of it. One of the cases where EAFP is clumsy is if the "forgiveness" portion is actually retrying what failed after fixing things. I just ran into this in real code, and the resulting discussion suggested this change. Eiffel's exception-handling mechanism is built assuming that's the *standard* way to do things. Seems to work ok, but if I had to choose one of the two, I'll take try/except. So here's the proposal: A single new keyword/clause, "retry". It has the same syntax as an "except" clause, can be used anywhere "except" can be used, and can be intermingled with them in the same try statement. There's probably a better syntax, but this is easy to describe. The behavior change from except is that instead of exiting the "try" statement when the "retry" clause ends, it restarts the try clause. In python, this code: try: # try block except: # except block retry: # retry block else: # else block finally: # finally block Would provide the same behavior as this code: while True: try: # try block except: # except block retry: # retry block else: # else block finally: # finally block break Where except & retry can be repeated, include exceptions to check against, and intermixed with each other. The break would propagate up to the else block and any except blocks if there was no finally. If there's no else, the break from it winds up at the end of the try. The use case, as mentioned, is avoiding doing EAFP without repeating the code in the exception handler. I.e., the choices without retry are things like (LBYL): if not isinstance(x, my_class): x = fixup(x) mangle(x) other_stuff(x) or EAFP: try: mangle(x) except: x = fixup(x) mangle(x) other_stuff(x) or: while True: try: mangle(x) break except: x = fixup(x) other_stuff(x) with: try: mangle(x) retry: e = fixup(x) other_stuff(x) My gut reaction is cool, but not clear it's worth a new keyword. So maybe there's a tweak to the except clause that would have the same effect, but I don't see anything obvious. References: <20120120163648.69288796@mikmeyer-vm-fedora> Message-ID: On Fri, Jan 20, 2012 at 4:36 PM, Mike Meyer wrote: > A single new keyword/clause, "retry". It has the same syntax as an > "except" clause, can be used anywhere "except" can be used, and can be > intermingled with them in the same try statement. There's probably a > better syntax, but this is easy to describe. > > The behavior change from except is that instead of exiting the "try" > statement when the "retry" clause ends, it restarts the try > clause. In python, this code: > > try: > # try block > except: > # except block > retry: > # retry block > else: > # else block > finally: > # finally block > Can you write this in terms of current Python code? I don't understand exactly when a retry block would be executed. In Eiffel, retry is a statement, not a clause. Analogous to that would be: try: # block except: # block if condition: retry The equivalent of this in current Python is while True: try: # block except: # block if condition: continue # retry break FWIW, if this turns out to be a good idea, an alternate spelling that would not require a new keyword is 'try again'. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron3200 at gmail.com Sat Jan 21 03:24:30 2012 From: ron3200 at gmail.com (Ron Adam) Date: Fri, 20 Jan 2012 20:24:30 -0600 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <20120120163648.69288796@mikmeyer-vm-fedora> References: <20120120163648.69288796@mikmeyer-vm-fedora> Message-ID: <1327112670.2481.4.camel@Gutsy> On Fri, 2012-01-20 at 16:36 -0800, Mike Meyer wrote: > The behavior change from except is that instead of exiting the "try" > statement when the "retry" clause ends, it restarts the try > clause. In python, this code: > > try: > # try block > except: > # except block > retry: > # retry block > else: > # else block > finally: > # finally block > > Would provide the same behavior as this code: > > while True: > try: > # try block > except: > # except block > retry: > # retry block > else: > # else block > finally: > # finally block > break This looks like it would be very tricky to get right from a python programming stand point. Remember that python is full of one time iterators that won't repeat properly on a second try. Also many function calls mutate the arguments. So it's possible that a data structure could be changed on the first try, and give different results on a second try in an unexpected way. Cheers, Ron From benjamin at python.org Sat Jan 21 04:13:46 2012 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 21 Jan 2012 03:13:46 +0000 (UTC) Subject: [Python-ideas] Retrying EAFP without DRY References: <20120120163648.69288796@mikmeyer-vm-fedora> Message-ID: Bruce Leban writes: > > FWIW, if this turns out to be a good idea, an alternate spelling that would not require a new keyword is 'try again'. How is that not a new keyword? From steve at pearwood.info Sat Jan 21 05:16:36 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 Jan 2012 15:16:36 +1100 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> Message-ID: <4F1A3C24.9090508@pearwood.info> Benjamin Peterson wrote: > Bruce Leban writes: >> FWIW, if this turns out to be a good idea, an alternate spelling that would > not require a new keyword is 'try again'. > > How is that not a new keyword? "again" could be a not-actually-a-keyword keyword like "as" used to be. [steve at sylar ~]$ python2.3 Python 2.3.7 (#1, Aug 12 2010, 00:17:29) [GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> as = 20 >>> import math as m >>> m.sin(as) 0.91294525072762767 Not that I support that, but it is possible. -- Steven From pyideas at rebertia.com Sat Jan 21 05:20:51 2012 From: pyideas at rebertia.com (Chris Rebert) Date: Fri, 20 Jan 2012 20:20:51 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <20120120163648.69288796@mikmeyer-vm-fedora> References: <20120120163648.69288796@mikmeyer-vm-fedora> Message-ID: On Fri, Jan 20, 2012 at 4:36 PM, Mike Meyer wrote: > Just running another half-baked "easy" idea up to see what others > think of it. > > One of the cases where EAFP is clumsy is if the "forgiveness" portion > is actually retrying what failed after fixing things. I just ran into > this in real code, and the resulting discussion suggested this change. I would be very interested if you give at least the general gist of the part of your code in question. Real-world use cases are always useful data. > Eiffel's exception-handling mechanism is built assuming that's the > *standard* way to do things. Seems to work ok, but if I had to choose > one of the two, I'll take try/except. > > So here's the proposal: > > A single new keyword/clause, "retry". It has the same syntax as an > "except" clause, can be used anywhere "except" can be used, and can be > intermingled with them in the same try statement. There's probably a > better syntax, but this is easy to describe. > > The behavior change from except is that instead of exiting the "try" > statement when the "retry" clause ends, it restarts the try > clause. In python, this code: > > ? ?try: > ? ? ? ?# try block > ? ?except: > ? ? ? ?# except block > ? ?retry: > ? ? ? ?# retry block > ? ?else: > ? ? ? ?# else block > ? ?finally: > ? ? ? ?# finally block > The use case, as mentioned, is avoiding doing EAFP without repeating > the code in the exception handler. I.e., the choices without retry > are things like (LBYL): > or EAFP: > > ? ?try: > ? ? ? ?mangle(x) > ? ?except: > ? ? ? ?x = fixup(x) > ? ? ? ?mangle(x) > ? ?other_stuff(x) Right, that has evil code duplication. > or: > > ? ?while True: > ? ? ? ?try: > ? ? ? ? ? ?mangle(x) > ? ? ? ? ? ?break Personally, I would put the `break` in a separate `else` clause. > ? ? ? ?except: > ? ? ? ? ? ?x = fixup(x) > ? ?other_stuff(x) Your mention duplication as a gripe, yet there's no duplication in this version. I take then it that you just dislike this idiom? > with: > > ? ?try: > ? ? ? ?mangle(x) > ? ?retry: > ? ? ? ?e = fixup(x) > ? ?other_stuff(x) Do I misunderstand your proposal, or are you missing an "except: retry" here? If you aren't, then how is infinite-retrying avoided? I would think that in many (most?) cases, we'd like to cap the number of reattempts. Currently, this looks like: for _ in range(5): try: buy_off_ebay(item, price) except OutbidError: price = highest_bid_for(item) * 1.05 # continue else: break # Some fiddling can be had over where to put the `break` # and whether to include an explicit `continue`. With what I understand `retry` would be: # Not even bothering with the capping try: buy_off_ebay(item, price) except OutbidError: retry retry: price = highest_bid_for(item) * 1.05 # I'd have to maintain the count myself?! # That's easily going to squander the 1 line this saved. I grant you that this has saved 1 level of indentation, whatever that's worth (personally, I don't put enough premium on it in this case to justify new syntax). It's also slightly more explicit (once you know what `retry` does, of course). And, in more general cases, a very few lines can be saved. But when one gets to the more general, complicated cases, it seems to me that you're better off just writing and making use of your own try_N_times(make_bid, N) [or whatever your "keep retrying" heuristic is] function, which argues against adding new syntax for this. > My gut reaction is cool, but not clear it's worth a new keyword. So > maybe there's a tweak to the except clause that would have the same > effect, but I don't see anything obvious. My view on Python's choice of control structures is that it provides a near-minimal, simple (so long as you overlook the obscure and seldom-used `else` clause of `for` & `while`) set of them that you can then combine to achieve fancier behavior (e.g. while + try + break = retry). For example, there's no do-while (instead: while True + if-break); there's no unless/until (instead: while/if + not); there's no C-style `for` (instead: for+range(), or a custom while); there's no `redo` (instead: nested while True + if-break/continue); and thus, no `retry` (instead: nested try within a loop). This keeps the core language small, aesthetically clean, uniform, and easy-to-learn. If one understands `if`, `while`, `for`, `try`, `break`, and `continue` (and `not`, but that's merely a boolean operator, as opposed to a control structure), then one can easily work out from scratch what the idioms do, and one learns to recognize them with just a bit of practice/experience. Also, it's easy to tweak the idioms when your control flow changes or becomes more complicated, thus allowing for flexibility. On the other hand, some clarity/explicitness is definitely lost. There's a design trade-off here; I don't think Python has tended towards the side that would add `retry`. Cheers, Chris -- http://rebertia.com From steve at pearwood.info Sat Jan 21 08:47:46 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 Jan 2012 18:47:46 +1100 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <20120120163648.69288796@mikmeyer-vm-fedora> References: <20120120163648.69288796@mikmeyer-vm-fedora> Message-ID: <4F1A6DA2.5020807@pearwood.info> Mike Meyer wrote: > A single new keyword/clause, "retry". It has the same syntax as an > "except" clause, can be used anywhere "except" can be used, and can be > intermingled with them in the same try statement. There's probably a > better syntax, but this is easy to describe. > The behavior change from except is that instead of exiting the "try" > statement when the "retry" clause ends, it restarts the try This sounds like retry should be a flow-control statement, like continue or break, not a block. E.g.: try: something() except ValueError: if condition: retry else: raise "retry" will jump back to the start of the try block -- a limited form of GOTO, with all the pros and cons of this. > clause. In python, this code: > > try: > # try block > except: > # except block > retry: > # retry block > else: > # else block > finally: > # finally block > > Would provide the same behavior as this code: > > while True: > try: > # try block > except: > # except block > retry: > # retry block > else: > # else block > finally: > # finally block > break Um, you're trying to explain the behaviour of retry here, but your Python code includes a retry. This is one case where recursion is NOT your friend. > Where except & retry can be repeated, include exceptions to check > against, and intermixed with each other. The break would propagate up > to the else block and any except blocks if there was no finally. If > there's no else, the break from it winds up at the end of the try. None of your explanation is clear to me. Under what circumstances will the retry block be executed? E.g. given: try: len(None) except TypeError: print("spam spam spam") retry: print("a") what will happen? How about these? #1 try: len(None) retry: print("a") except TypeError: print("spam spam spam") #2 try: len(None) except TypeError: print("spam spam spam") retry: print("a") except ValueError: print("this never gets called") retry: print("b") #3 try: len(None) except TypeError: print("spam spam spam") retry: print("a") retry: print("b") > The use case, as mentioned, is avoiding doing EAFP without repeating > the code in the exception handler. I.e., the choices without retry > are things like (LBYL): I think that the idiom of a while or for loop is easy enough to read and write: for _ in range(5): # retry five times try: do_something(x) except SpamError: x = fix_up(x) else: break else: raise HamError("tried 5 times, giving up now") I just wish that break and continue could be written outside of a loop, so you can factor out common code: def do_the_thing(x): try: do_something(x) except SpamError: x = fix_up(x) else: break def try_repeatedly(n, func): for _ in range(n): func() else: raise HamError('tried %d times, giving up now" % n) try_repeatedly(5, do_the_thing) -- Steven From pyideas at rebertia.com Sat Jan 21 09:25:25 2012 From: pyideas at rebertia.com (Chris Rebert) Date: Sat, 21 Jan 2012 00:25:25 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <4F1A6DA2.5020807@pearwood.info> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> Message-ID: On Fri, Jan 20, 2012 at 11:47 PM, Steven D'Aprano wrote: > I think that the idiom of a while or for loop is easy enough to read and > write: > > for _ in range(5): ?# retry five times > ? ?try: > ? ? ? ?do_something(x) > ? ?except SpamError: > ? ? ? ?x = fix_up(x) > ? ?else: > ? ? ? ?break > else: > ? ?raise HamError("tried 5 times, giving up now") > > > I just wish that break and continue could be written outside of a loop, so > you can factor out common code: You also seem to have some shenanigans going on with `x`. > def do_the_thing(x): > ? ?try: > ? ? ? ?do_something(x) > ? ?except SpamError: > ? ? ? ?x = fix_up(x) > ? ?else: > ? ? ? ?break > > def try_repeatedly(n, func): > ? ?for _ in range(n): > ? ? ? ?func() > ? ?else: > ? ? ? ?raise HamError('tried %d times, giving up now" % n) > > try_repeatedly(5, do_the_thing) Easily accomplished: def do_the_thing(x): try: do_something(x) except SpamError: fix_up(x) return False else: return True def try_repeatedly(n, func, arg): for _ in range(n): if func(arg): break else: raise HamError('tried %d times, giving up now" % n) try_repeatedly(5, do_the_thing, y) Cheers, Chris From steve at pearwood.info Sat Jan 21 09:57:16 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 Jan 2012 19:57:16 +1100 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> Message-ID: <4F1A7DEC.40503@pearwood.info> Chris Rebert wrote: > On Fri, Jan 20, 2012 at 11:47 PM, Steven D'Aprano wrote: > >> I just wish that break and continue could be written outside of a loop, so >> you can factor out common code: [...] > Easily accomplished: > > def do_the_thing(x): > try: > do_something(x) > except SpamError: > fix_up(x) > return False > else: > return True > > def try_repeatedly(n, func, arg): > for _ in range(n): > if func(arg): break > else: > raise HamError('tried %d times, giving up now" % n) > > try_repeatedly(5, do_the_thing, y) Not so easily accomplished if you need the return result from do_the_thing. Naturally you can always return a tuple (result_I_actually_want, code_to_break_or_continue) but that's not exactly elegant. -- Steven From stephen at xemacs.org Sat Jan 21 10:11:48 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 21 Jan 2012 18:11:48 +0900 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <4F1A6DA2.5020807@pearwood.info> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> Message-ID: <87r4yt5siz.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > Mike Meyer wrote: > "retry" will jump back to the start of the try block This doesn't sound very intuitive to me. > -- a limited form of GOTO, with all the pros and cons of this. All control flow is a limited form of GOTO. But in this case, retry can only go to one place, about as unlike GOTO as I can imagine. It certainly doesn't have any cons of GOTO that "while" doesn't have! > > The use case, as mentioned, is avoiding doing EAFP without repeating > > the code in the exception handler. I.e., the choices without retry > > are things like (LBYL): > > I think that the idiom of a while or for loop is easy enough to read and write: > > for _ in range(5): # retry five times > try: > do_something(x) > except SpamError: > x = fix_up(x) > else: > break > else: > raise HamError("tried 5 times, giving up now") Agreed. I find this much easier to understand than the "retry" form. It seems to me that rather than a new keyword "retry", I'd want to use the existing "continue" here as with other looping constructs. The problem is that this would interfere with other uses of continue, not to mention that it would be inconsistent with break. While this thought experiment doesn't prove anything, it makes me wonder if the idea is really coherent (at least in the context of Python today). > I just wish that break and continue could be written outside of a > loop, so you can factor out common code: > > def do_the_thing(x): > try: > do_something(x) > except SpamError: > x = fix_up(x) > else: > break > > def try_repeatedly(n, func): > for _ in range(n): > func() > else: > raise HamError('tried %d times, giving up now" % n) > > try_repeatedly(5, do_the_thing) I really don't like the idea of using a nonlocal construct as part of ordinary control flow. I guess I could get used to this particular idiom, but it's not like the explicit loop is terribly burdensome, to write or to read. From cmjohnson.mailinglist at gmail.com Sat Jan 21 10:14:41 2012 From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson) Date: Fri, 20 Jan 2012 23:14:41 -1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <4F1A7DEC.40503@pearwood.info> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <4F1A7DEC.40503@pearwood.info> Message-ID: <56CBC685-6091-49EB-B2C3-865D89C3B4F4@gmail.com> On Jan 20, 2012, at 10:57 PM, Steven D'Aprano wrote: > Chris Rebert wrote: >> On Fri, Jan 20, 2012 at 11:47 PM, Steven D'Aprano wrote: >> >>> I just wish that break and continue could be written outside of a loop, so >>> you can factor out common code: > [...] >> Easily accomplished: >> def do_the_thing(x): >> try: >> do_something(x) >> except SpamError: >> fix_up(x) >> return False >> else: >> return True >> def try_repeatedly(n, func, arg): >> for _ in range(n): >> if func(arg): break >> else: >> raise HamError('tried %d times, giving up now" % n) >> try_repeatedly(5, do_the_thing, y) > > Not so easily accomplished if you need the return result from do_the_thing. Naturally you can always return a tuple > > (result_I_actually_want, code_to_break_or_continue) > > but that's not exactly elegant. I'd make the place I factor out code different: def try_repeatedly(times, exception, callback, args, kwargs): for _ in range(times): try: return callback(*args, **kwargs) except exception: continue break error_message = "Tried {} times but could not complete callback {}" error_message = error_message.format(times, repr(callback)) raise Exception(error_message) try_repeatedly(5, SpamError, do_something, [x]) From cmjohnson.mailinglist at gmail.com Sat Jan 21 10:17:50 2012 From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson) Date: Fri, 20 Jan 2012 23:17:50 -1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <56CBC685-6091-49EB-B2C3-865D89C3B4F4@gmail.com> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <4F1A7DEC.40503@pearwood.info> <56CBC685-6091-49EB-B2C3-865D89C3B4F4@gmail.com> Message-ID: <86BAF22E-86C8-4DCE-8E6B-BDE0E3EC502A@gmail.com> On Jan 20, 2012, at 11:14 PM, Carl M. Johnson wrote: > def try_repeatedly(times, exception, callback, args, kwargs): > for _ in range(times): > try: > return callback(*args, **kwargs) > except exception: > continue > break > error_message = "Tried {} times but could not complete callback {}" > error_message = error_message.format(times, repr(callback)) > raise Exception(error_message) > > try_repeatedly(5, SpamError, do_something, [x]) Oops, don't need the "break". From steve at pearwood.info Sat Jan 21 10:38:55 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 21 Jan 2012 20:38:55 +1100 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <87r4yt5siz.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <87r4yt5siz.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4F1A87AF.7060604@pearwood.info> Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > > Mike Meyer wrote: > > > "retry" will jump back to the start of the try block > > This doesn't sound very intuitive to me. No, not very. You might not guess what it does, but it's not that hard to learn: "retry jumps back to the beginning of the try block". > > -- a limited form of GOTO, with all the pros and cons of this. > > All control flow is a limited form of GOTO. But in this case, retry > can only go to one place, about as unlike GOTO as I can imagine. It > certainly doesn't have any cons of GOTO that "while" doesn't have! The main con that I can think of is that it isn't clear that you're looping until you reach the retry. try: ... except Spam: ... except Ham: ... else: if flag: ... else: retry # Ah, we're in a loop! I don't hate this, but nor am I going to champion it. If someone else wants to champion it, I'd vote +0. It just feels more natural than making retry a block clause (Mike's proposal), but I'm not really convinced it's necessary. [...] > It seems to me that rather than a new keyword "retry", I'd want to use > the existing "continue" here as with other looping constructs. The > problem is that this would interfere with other uses of continue, not > to mention that it would be inconsistent with break. While this > thought experiment doesn't prove anything, it makes me wonder if the > idea is really coherent (at least in the context of Python today). continue can't work, because it introduces ambiguity if you have a try inside a loop, or a loop inside a try. We could arbitrarily declare that try-continue wins over loop-continue, or visa versa, but whatever decision we make, it will leave half the would-be users unhappy. Besides, there's a neat symmetry: you try something, then you re-try it. (Believe it or not, I've only just noticed this.) -- Steven From p.f.moore at gmail.com Sat Jan 21 12:46:19 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 21 Jan 2012 11:46:19 +0000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <4F1A6DA2.5020807@pearwood.info> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> Message-ID: On 21 January 2012 07:47, Steven D'Aprano wrote: > This sounds like retry should be a flow-control statement, like continue or > break, not a block. E.g.: > > try: > ? ?something() > except ValueError: > ? ?if condition: retry > ? ?else: raise > > > "retry" will jump back to the start of the try block -- a limited form of > GOTO, with all the pros and cons of this. That's the way I would interpret a "retry" statement - and it's nothing like the OP's proposal as far as I can see (where retry introduces a suite). I'd be -1 on a retry keyword that worked any way other than this (simply because this is "clearly" the most obvious interpretation). Whether a retry statement is worth having at all, though, is something I'm not sure of - I'd like to see some real-world use cases first. I've never encountered the need for it myself. And I can't honestly imagine why the while True...try...break" idiom would ever not be sufficient. Paul. From ncoghlan at gmail.com Sat Jan 21 13:06:11 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 21 Jan 2012 22:06:11 +1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> Message-ID: On Sat, Jan 21, 2012 at 11:40 AM, Bruce Leban wrote: > while True: > ? ? try: > ? ? ? ? # block > ? ? except: > ? ? ? ? # block > ? ? ? ? if condition: continue # retry > ? ? break And, indeed, I'd consider "while True" to be a fairly idiomatic way to write "keep trying this until it works (and we exit the loop via break) or we give up (and raise an exception)" without repeating ourselves. Alternatively, I'd push the entire try/except block into a closure and call that from the loop until it reported it had worked. A nested generator could achieve much the same thing. I don't think there needs to be "one obvious way" to do this, since it's a rare need and the existing tools (while loops with break/continue, a loop with an appropriate flag, a loop plus a closure, a nested generator, etc) all provide ways to implement it reasonably cleanly. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Sat Jan 21 17:42:05 2012 From: guido at python.org (Guido van Rossum) Date: Sat, 21 Jan 2012 08:42:05 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <4F1A3C24.9090508@pearwood.info> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A3C24.9090508@pearwood.info> Message-ID: On Fri, Jan 20, 2012 at 8:16 PM, Steven D'Aprano wrote: > "again" could be a not-actually-a-keyword keyword like "as" used to be. Let's not do that again. It was a major pain. -- --Guido van Rossum (python.org/~guido) From sho at eikehein.com Sat Jan 21 23:23:20 2012 From: sho at eikehein.com (Eike Hein) Date: Sat, 21 Jan 2012 23:23:20 +0100 Subject: [Python-ideas] Combining test and assignment Message-ID: <4F1B3AD8.6030402@eikehein.com> Hello, this is very spur of the moment, and so I apologize if I am missing the obvious reasons for why this would be horribly dumb. I sometimes find myself annoyed that I cannot access the result of an expression in a conditional statememt. Consider: if spam(): ... frobulate the return value of spam() ... else: ... value too low to frobulate ... In order to frobulate spam(), either need to call it again in the indented block, or preassign: x = spam() if x: ... frobulate x ... The first option is usually inefficient, and the second feels ugly to me because something relevant to a block is happening outside it and its header. Instead, I'd love to be able to: if spam() as x: ... frobulate x ... Further, there are two reasons this seems a particularly good fit in Python: - There is a _syntactic_ precedent, or at least great similarity to the 'with' statement. - Because calls in Python return None if they don't ex- plicitly return anything else, there is no "can't assign void to x" to worry about. I haven't given much thought to scoping, particularly for else clauses, yet, at least not enough to make up my mind. The other question is if this reads good "as English" or needs to be 'if spam() then as x:', or if that's even an issue. Thoughts? Best regards, Eike From jkbbwr at gmail.com Sun Jan 22 00:30:49 2012 From: jkbbwr at gmail.com (Jakob Bowyer) Date: Sat, 21 Jan 2012 23:30:49 +0000 Subject: [Python-ideas] Combining test and assignment In-Reply-To: <4F1B3AD8.6030402@eikehein.com> References: <4F1B3AD8.6030402@eikehein.com> Message-ID: Say I have a function that is def test(): return sometimes_false if test() as x: print "Its not false" # what happens to x now? is it a NameError or is it None On Sat, Jan 21, 2012 at 10:23 PM, Eike Hein wrote: > > Hello, > > this is very spur of the moment, and so I apologize if > I am missing the obvious reasons for why this would be > horribly dumb. > > I sometimes find myself annoyed that I cannot access the > result of an expression in a conditional statememt. > > Consider: > > if spam(): > ... frobulate the return value of spam() ... > else: > ... value too low to frobulate ... > > In order to frobulate spam(), either need to call it > again in the indented block, or preassign: > > x = spam() > > if x: > ... frobulate x ... > > The first option is usually inefficient, and the second > feels ugly to me because something relevant to a block is > happening outside it and its header. > > Instead, I'd love to be able to: > > if spam() as x: > ... frobulate x ... > > Further, there are two reasons this seems a particularly > good fit in Python: > - There is a _syntactic_ precedent, or at least great > similarity to the 'with' statement. > - Because calls in Python return None if they don't ex- > plicitly return anything else, there is no "can't assign > void to x" to worry about. > > I haven't given much thought to scoping, particularly for > else clauses, yet, at least not enough to make up my mind. > > The other question is if this reads good "as English" or > needs to be 'if spam() then as x:', or if that's even an > issue. > > Thoughts? > > > Best regards, > Eike > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Jan 22 02:15:42 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 22 Jan 2012 12:15:42 +1100 Subject: [Python-ideas] Combining test and assignment In-Reply-To: References: <4F1B3AD8.6030402@eikehein.com> Message-ID: <4F1B633E.9060406@pearwood.info> Jakob Bowyer wrote: > Say I have a function that is > def test(): > return sometimes_false > > if test() as x: > print "Its not false" > # what happens to x now? is it a NameError or is it None Neither. It should be syntactic sugar for this: x = test() if x: print "Its not false" So x will always be sometimes_false, whatever that happens to be. -- Steven From ncoghlan at gmail.com Sun Jan 22 04:48:20 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Jan 2012 13:48:20 +1000 Subject: [Python-ideas] Combining test and assignment In-Reply-To: <4F1B3AD8.6030402@eikehein.com> References: <4F1B3AD8.6030402@eikehein.com> Message-ID: This suggestion (i.e. embedded assignment in while and if headers) has been made several times, and always come to a grinding halt on one simple problem: it isn't expressive enough. The current idiom is this : x = EXPR if predicate(x): # Do something Given a value and a predicate, this allows you to execute code conditionally based on whether or not the predicate is true. Simple embedded assignment, though, only works when the predicate is just "bool" - as soon as the condition differs from the value you want to access, you need to revert to the existing idiom *anyway*. Given the non-trivial costs of adding new syntax, that gets the idea put into the "not worth the hassle" bucket. The obvious escalation of the suggestion is to adopt "(EXPR as NAME)" as a general purpose embedded assignment expression. However, this idea has problems of its own: 1. In with statements and exception handlers, 'as' does *not* denote ordinary assignment. In the former case, the name is bound to the result of the EXPR.__enter__() call, in the latter to the exception instance that was actually caught. If assignment expressions were added, these could become a rather subtle trap (especially since many __enter__() methods just return self) 2. Generator expressions and the various forms of comprehension create new scopes to avoid leaking iteration variables, so, if embedded assignments were to do anything useful there, additional forms like "(EXPR as nonlocal NAME)" and "(EXPR as global NAME)" would be needed. (And, in the nonlocal case, would still require that the name already be defined in the containing scope) 3. Adding assignment expressions would mean having two ways to perform assignments at the statement level (either using the existing statement form or using the new expression form) To date, nobody has been interested enough in the latter idea to put together a formal PEP and reference implementation for python-dev's consideration, and the former idea has been informally rejected several times due to the lack of generality. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From sho at eikehein.com Sun Jan 22 05:05:48 2012 From: sho at eikehein.com (Eike Hein) Date: Sun, 22 Jan 2012 05:05:48 +0100 Subject: [Python-ideas] Combining test and assignment In-Reply-To: References: <4F1B3AD8.6030402@eikehein.com> Message-ID: <4F1B8B1C.8050106@eikehein.com> On 1/22/2012 4:48 AM, Nick Coghlan wrote: > Simple embedded assignment, though, only works when the predicate is > just "bool" - as soon as the condition differs from the value you want > to access, you need to revert to the existing idiom *anyway*. Given > the non-trivial costs of adding new syntax, that gets the idea put > into the "not worth the hassle" bucket. Actually, just to be clear, in the use case I had in mind spam() is returning a numeric value. That satisfies a truth test and is still useful in other ways (but I assume you are actually on the same page, hence the quotation marks around bool). Another case I come across frequently is with the re module, where re.match() returns either None or a MatchObject. if re.match(pattern, string) as m: ... make use of the match object ... Anyway, thank you for a comprehensive reply and recap of the history. > To date, nobody has been interested enough in the latter idea to put > together a formal PEP and reference implementation for python-dev's > consideration, and the former idea has been informally rejected > several times due to the lack of generality. I thought a bit about what Python offers today in terms of built-in tools to approximate what I want. My first thought was to try and abuse context managers, but aside from being horribly ugly, a with statement also can't prevent its body from being run, it seems. Another, possibly better way is to modify locals(), i.e.: if test(expr, 'var'): ... do stuff ... Where the test function returns the expression but also stores its results in locals()['var']. I wonder if a function like this (with a better name obviously) would have a place in the stdlib somewhere, or is it too much of a limited-use hack? > Cheers, > Nick. -- Best regards, Eike Hein From pyideas at rebertia.com Sun Jan 22 05:11:46 2012 From: pyideas at rebertia.com (Chris Rebert) Date: Sat, 21 Jan 2012 20:11:46 -0800 Subject: [Python-ideas] Combining test and assignment In-Reply-To: <4F1B8B1C.8050106@eikehein.com> References: <4F1B3AD8.6030402@eikehein.com> <4F1B8B1C.8050106@eikehein.com> Message-ID: On Sat, Jan 21, 2012 at 8:05 PM, Eike Hein wrote: > On 1/22/2012 4:48 AM, Nick Coghlan wrote: >> Simple embedded assignment, though, only works when the predicate is >> just "bool" - as soon as the condition differs from the value you want >> to access, you need to revert to the existing idiom *anyway*. Given >> the non-trivial costs of adding new syntax, that gets the idea put >> into the "not worth the hassle" bucket. > I thought a bit about what Python offers today in terms > of built-in tools to approximate what I want. My first > thought was to try and abuse context managers, but aside > from being horribly ugly, a with statement also can't > prevent its body from being run, it seems. > > Another, possibly better way is to modify locals(), i.e.: > > if test(expr, 'var'): > ? ... do stuff ... > > Where the test function returns the expression but also > stores its results in locals()['var']. That doesn't actually work (unless you happen to be writing code in module-level scope, where globals are locals): http://docs.python.org/library/functions.html#locals : "Note: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter." Cheers, Chris From sho at eikehein.com Sun Jan 22 05:18:24 2012 From: sho at eikehein.com (Eike Hein) Date: Sun, 22 Jan 2012 05:18:24 +0100 Subject: [Python-ideas] Combining test and assignment In-Reply-To: References: <4F1B3AD8.6030402@eikehein.com> <4F1B8B1C.8050106@eikehein.com> Message-ID: <4F1B8E10.9050609@eikehein.com> On 1/22/2012 5:11 AM, Chris Rebert wrote: > That doesn't actually work (unless you happen to be writing code in > module-level scope, where globals are locals): Right, brainfart - the locals() of test aren't the locals of the callsite obviously :). And nonlocal won't work due to the lack of an existing binding in the outer scope. Can anyone think of another way to do an assignment and a truth test in the same expression with today's tools? > http://docs.python.org/library/functions.html#locals : > "Note: The contents of this dictionary should not be modified; changes > may not affect the values of local and free variables used by the > interpreter." Yep, I was actually confusing this with vars() in terms of whether modifying its return value is considered kosher. Thanks. Sadly that makes the above harder ... > Cheers, > Chris -- Best regards, Eike Hein From steve at pearwood.info Sun Jan 22 05:51:13 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 22 Jan 2012 15:51:13 +1100 Subject: [Python-ideas] Combining test and assignment In-Reply-To: References: <4F1B3AD8.6030402@eikehein.com> Message-ID: <4F1B95C1.7020402@pearwood.info> Nick Coghlan wrote: > This suggestion (i.e. embedded assignment in while and if headers) has > been made several times, and always come to a grinding halt on one > simple problem: it isn't expressive enough. [...] > The obvious escalation of the suggestion is to adopt "(EXPR as NAME)" > as a general purpose embedded assignment expression. However, this > idea has problems of its own: I dislike assignment as an expression, and am glad that Python doesn't have it. Even though the form "expr as name" avoids the common C gotcha, I am -1 on adding it to Python. I believe it is an unnatural way of writing that obscures the code rather than simplifies it. (Sometimes, shorter is not better.) For example, when writing code, I might start off by thinking: if spam() != 'ham': frobulate(spam()) Then I would apply DRY and factor out the call to spam: x = spam() if x != 'ham': frobulate(x) This matches my thought processes and corresponds to how you might naturally describe the algorithm in English: Let x equal spam(). If x != 'ham', then frobulate(x). The important thing is that you name the thing you care about before using it. I think this is a very natural way of writing: first you give the thing you care about a name, then you refer to it by name. Assignment as an expression feels unnatural to me: if spam() as x != 'ham': frobulate(x) doesn't really correspond to any natural English order. The assignment is dropped in the middle of another clause: If -- let x = spam() -- x != 'ham', then frobulate(x). It saves a line, but is not a natural way of writing for me. I would not like to read code written that way. Assignment as an expression also lends itself to writing buggy code like this: while spam() as x and ham() as y: frobulate(x) glommify(y) which is wrong, because y won't be defined if x has a true value. The alternative would require defeating the short-circuit nature of "and", which would be bad. Perhaps the only thing going for it is that it would allow list comprehensions to not repeat themselves: # instead of this [frobulate(x) for x in seq if frobulate(x) > 0] # you could have this [y for x in seq if frobulate(x) as y > 0] Although I prefer to use an inner map: [y for y in map(frobulate, seq) if y > 0] # use itertools.imap in Python2 -- Steven From python at 2sn.net Sun Jan 22 06:33:23 2012 From: python at 2sn.net (Alexander Heger) Date: Sat, 21 Jan 2012 23:33:23 -0600 Subject: [Python-ideas] Combining test and assignment, quantifications, and tuple patterns In-Reply-To: <4F1B95C1.7020402@pearwood.info> References: <4F1B3AD8.6030402@eikehein.com> <4F1B95C1.7020402@pearwood.info> Message-ID: <4F1B9FA3.10108@2sn.net> I think one could use the same new construct I proposed for one of the recent lengthy discussions. For your convenience, I copy this below, in disregards of quoting style as it provides an overarching reply to this thread. You could use sample as x: # do something with x break else: # do something else here the generator expression would be "spam()" sample y for y in (spam(),) if cond(y) as x: (...) or allow a short form sample y for spam() if cond(y) as x: (...) I really think this expressions could add a lot to Python's expressiveness. As an important note, in the case below, the difference to just a "for loop" or "generator" would be that is re-evaluated from scratch every time the loop returns to the sample statement. This is what I think would set it apart besides providing extra uses otherwise. Admittedly, it is a very complicated construct. Longer than most good English sentences. (for which those above are not good examples) -Alexander -------- Original Message -------- Subject: Re: [Python-ideas] Fwd: quantifications, and tuple patterns Date: Mon, 16 Jan 2012 22:00:52 -0600 From: Alexander Heger To: python-ideas at python.org Dear Tom, > I'm wondering if would make sense to add an "as" clause to the while and if > statements, similar to the as clause in the with statement. It would > essentially retrieve the value field from the witness object described > above. > > This would let one write the main loop in a topological sort (a classic > workset algorithm) as: > > while any(v for v in vertices if v.incoming_count == 0) as v1: > result.append(v) > for v2 in v1.outgoing: > v2.incoming_count -= 1 I had the same thought going through the thread, but I think the problem here would be that any is a function that returns a Boolean value. You could add a key parameter to return something else, say a tuple of a truth value and a sample ( any( ..., sample = True) ) but that could break because 1) "any" could be another function in the context, 2) while would need to deal with special return values in this case A suggestion to replace the "while any" case is sample as x: # do something with x ("sample" may not be the best keyword choice) From the discussion so far I do not see how to easily avoid having a new keyword specifically, your example would become sample v for v in vertices if v.incoming_count == 0 as v1: # do things with v1 In case you only want one sample, you could add the break statement sample db.query(page_name=page_name) as page: render_template(page) break else: error404() -Alexander On 01/21/2012 10:51 PM, Steven D'Aprano wrote: > Nick Coghlan wrote: >> This suggestion (i.e. embedded assignment in while and if headers) has >> been made several times, and always come to a grinding halt on one >> simple problem: it isn't expressive enough. > [...] >> The obvious escalation of the suggestion is to adopt "(EXPR as NAME)" >> as a general purpose embedded assignment expression. However, this >> idea has problems of its own: > > > I dislike assignment as an expression, and am glad that Python doesn't have > it. Even though the form "expr as name" avoids the common C gotcha, I am -1 on > adding it to Python. > > I believe it is an unnatural way of writing that obscures the code rather than > simplifies it. (Sometimes, shorter is not better.) For example, when writing > code, I might start off by thinking: > > if spam() != 'ham': > frobulate(spam()) > > > Then I would apply DRY and factor out the call to spam: > > x = spam() > if x != 'ham': > frobulate(x) > > > This matches my thought processes and corresponds to how you might naturally > describe the algorithm in English: > > Let x equal spam(). > If x != 'ham', then frobulate(x). > > The important thing is that you name the thing you care about before using it. > I think this is a very natural way of writing: first you give the thing you > care about a name, then you refer to it by name. > > Assignment as an expression feels unnatural to me: > > if spam() as x != 'ham': > frobulate(x) > > doesn't really correspond to any natural English order. The assignment is > dropped in the middle of another clause: > > If -- let x = spam() -- x != 'ham', then frobulate(x). > > It saves a line, but is not a natural way of writing for me. I would not like > to read code written that way. > > Assignment as an expression also lends itself to writing buggy code like this: > > while spam() as x and ham() as y: > frobulate(x) > glommify(y) > > which is wrong, because y won't be defined if x has a true value. The > alternative would require defeating the short-circuit nature of "and", which > would be bad. > > Perhaps the only thing going for it is that it would allow list comprehensions > to not repeat themselves: > > # instead of this > [frobulate(x) for x in seq if frobulate(x)> 0] > # you could have this > [y for x in seq if frobulate(x) as y> 0] > > Although I prefer to use an inner map: > > [y for y in map(frobulate, seq) if y> 0] # use itertools.imap in Python2 > > > From cmjohnson.mailinglist at gmail.com Sun Jan 22 07:23:24 2012 From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson) Date: Sat, 21 Jan 2012 20:23:24 -1000 Subject: [Python-ideas] Combining test and assignment In-Reply-To: <4F1B95C1.7020402@pearwood.info> References: <4F1B3AD8.6030402@eikehein.com> <4F1B95C1.7020402@pearwood.info> Message-ID: On Jan 21, 2012, at 6:51 PM, Steven D'Aprano wrote: > I dislike assignment as an expression, and am glad that Python doesn't have it. Even though the form "expr as name" avoids the common C gotcha, I am -1 on adding it to Python. I find it interesting that Go-lang follows your intuition and so added the "assignment first" conditional. Here's an example from their docs: if x := f(); x < y { return x } else if x > z { return z } else { return y } In Python, we'd just put the x = f() first, but in Go, putting the assignment into the conditional affects the scoping of things. The x goes out of scope once the if is over. From p.f.moore at gmail.com Sun Jan 22 13:04:48 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 22 Jan 2012 12:04:48 +0000 Subject: [Python-ideas] Combining test and assignment In-Reply-To: <4F1B95C1.7020402@pearwood.info> References: <4F1B3AD8.6030402@eikehein.com> <4F1B95C1.7020402@pearwood.info> Message-ID: On 22 January 2012 04:51, Steven D'Aprano wrote: > Assignment as an expression feels unnatural to me: > > if spam() as x != 'ham': > ? ?frobulate(x) > > doesn't really correspond to any natural English order. While in general I agree with you, there *is* a natural reading of this: "if spam() isn't 'ham', frobulate it" The fact that you have to choose a name is because computer languages have to be more explicit than natural languages, and can't deal with the implicit referent involved in the English usage of "it". The need for assignment comes from the need for a name. You could actually deal with the if-or-while-expression-assignment case by defining a special variable __it__ to mean the value of the condition in the containing if or while expression, but it has all the same lack of generality issues as any proposal that limits itself to if/while conditions. And short of explicitly marking the expression somehow, you can't generalise any further. Paul. From ncoghlan at gmail.com Sun Jan 22 13:37:40 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Jan 2012 22:37:40 +1000 Subject: [Python-ideas] Combining test and assignment In-Reply-To: References: <4F1B3AD8.6030402@eikehein.com> <4F1B95C1.7020402@pearwood.info> Message-ID: On Sun, Jan 22, 2012 at 10:04 PM, Paul Moore wrote: > On 22 January 2012 04:51, Steven D'Aprano wrote: >> Assignment as an expression feels unnatural to me: >> >> if spam() as x != 'ham': >> ? ?frobulate(x) >> >> doesn't really correspond to any natural English order. > > While in general I agree with you, there *is* a natural reading of this: > > "if spam() isn't 'ham', frobulate it" Yeah, I agree characterising embedded assignments as a more formal equivalent of natural language pronouns is a useful way to think about them. However, one of the other reasons that these proposals tend not to go anywhere is that they ultimately boil down to some people hating the idea of having to write something they personally think of as one operation as a sequence of operations instead. Like multi-line lambdas, adding embedded assignments doesn't really increase the power and expressivity of the language as a whole all that much - it just moves the dividing line between what can be cleanly expressed in a single statement and what must be expressed as multiple statements a bit. Embedded assignments *do* address some instances of the loop-and-a-half problem, but even there, the more general idiom of "while True:" + "if exit_cond: break" can handle the situation quite happily. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From storchaka at gmail.com Sun Jan 22 19:02:40 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 22 Jan 2012 20:02:40 +0200 Subject: [Python-ideas] Combining test and assignment In-Reply-To: References: <4F1B3AD8.6030402@eikehein.com> <4F1B95C1.7020402@pearwood.info> Message-ID: 22.01.12 14:04, Paul Moore ???????(??): > While in general I agree with you, there *is* a natural reading of this: > > "if spam() isn't 'ham', frobulate it" > > The fact that you have to choose a name is because computer languages > have to be more explicit than natural languages, and can't deal with > the implicit referent involved in the English usage of "it". The need > for assignment comes from the need for a name. You could actually deal > with the if-or-while-expression-assignment case by defining a special > variable __it__ to mean the value of the condition in the containing > if or while expression, but it has all the same lack of generality > issues as any proposal that limits itself to if/while conditions. And > short of explicitly marking the expression somehow, you can't > generalise any further. Kotlin (http://confluence.jetbrains.net/display/Kotlin/Kotlin) has "it": names filter {it.startsWith("A")} sortby {it} map {it.toUpperCase()} foreach {print(it)} From pyideas at rebertia.com Sun Jan 22 19:36:05 2012 From: pyideas at rebertia.com (Chris Rebert) Date: Sun, 22 Jan 2012 10:36:05 -0800 Subject: [Python-ideas] Combining test and assignment In-Reply-To: References: <4F1B3AD8.6030402@eikehein.com> <4F1B95C1.7020402@pearwood.info> Message-ID: On Sun, Jan 22, 2012 at 4:04 AM, Paul Moore wrote: > On 22 January 2012 04:51, Steven D'Aprano wrote: >> Assignment as an expression feels unnatural to me: >> >> if spam() as x != 'ham': >> ? ?frobulate(x) >> >> doesn't really correspond to any natural English order. > > While in general I agree with you, there *is* a natural reading of this: > > "if spam() isn't 'ham', frobulate it" > > The fact that you have to choose a name is because computer languages > have to be more explicit than natural languages, and can't deal with > the implicit referent involved in the English usage of "it". The need > for assignment comes from the need for a name. You could actually deal > with the if-or-while-expression-assignment case by defining a special > variable __it__ to mean the value of the condition in the containing > if or while expression, but it has all the same lack of generality > issues as any proposal that limits itself to if/while conditions. And > short of explicitly marking the expression somehow, you can't > generalise any further. This is reminiscent of Perl's $_ variable, which represents the "current topic". http://www.perl.com/pub/2002/10/30/topic.html Cheers, Chris From mwm at mired.org Sun Jan 22 22:30:14 2012 From: mwm at mired.org (Mike Meyer) Date: Sun, 22 Jan 2012 13:30:14 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> Message-ID: <20120122133014.5e920a43@bhuda.mired.org> On Sat, 21 Jan 2012 11:46:19 +0000 Paul Moore wrote: > On 21 January 2012 07:47, Steven D'Aprano wrote: > > This sounds like retry should be a flow-control statement, like continue or > > break, not a block. E.g.: > > > > try: > > ? ?something() > > except ValueError: > > ? ?if condition: retry > > ? ?else: raise > > > > > > "retry" will jump back to the start of the try block -- a limited form of > > GOTO, with all the pros and cons of this. > > That's the way I would interpret a "retry" statement - and it's > nothing like the OP's proposal as far as I can see (where retry > introduces a suite). Yes, it's not much like what I proposed. But it does solve the same problem, and in a much better way than I proposed. Because of that, I'm not going to try and fix the error in the OP of not translating the proposal to proper Python :-(. > I'd be -1 on a retry keyword that worked any way other than this > (simply because this is "clearly" the most obvious interpretation). > Whether a retry statement is worth having at all, though, is something > I'm not sure of - I'd like to see some real-world use cases first. > I've never encountered the need for it myself. And I can't honestly > imagine why the while True...try...break" idiom would ever not be > sufficient. I saw a number of request for "real world uses cases". I thought I covered that in the OP. This ideas was *prompted* by a real world use case where we wanted to wrap an exception in our own private exception before handling it. Because the code that would raise the exception - other exceptions missing the attributes we added to ours - was the code we want to run after handling them, we're left with LBYL or DRY or wrapping a loop around the code when it wasn't really a loop. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From pyideas at rebertia.com Sun Jan 22 22:52:52 2012 From: pyideas at rebertia.com (Chris Rebert) Date: Sun, 22 Jan 2012 13:52:52 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <20120122133014.5e920a43@bhuda.mired.org> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> Message-ID: On Sun, Jan 22, 2012 at 1:30 PM, Mike Meyer wrote: > On Sat, 21 Jan 2012 11:46:19 +0000 > Paul Moore wrote: >> On 21 January 2012 07:47, Steven D'Aprano wrote: >> I'd be -1 on a retry keyword that worked any way other than this >> (simply because this is "clearly" the most obvious interpretation). >> Whether a retry statement is worth having at all, though, is something >> I'm not sure of - I'd like to see some real-world use cases first. >> I've never encountered the need for it myself. And I can't honestly >> imagine why the while True...try...break" idiom would ever not be >> sufficient. > > I saw a number of request for "real world uses cases". I thought I > covered that in the OP. This ideas was *prompted* by a real world use > case where we wanted to wrap an exception in our own private exception > before handling it. Because the code that would raise the exception > - other exceptions missing the attributes we added to ours - was the > code we want to run after handling them, we're left with LBYL or DRY > or wrapping a loop around the code when it wasn't really a loop. The thing is, until this message, you never described any concrete details about your specific use-case, other than that you had one. Now you have, but in prose (which is regrettably imprecise) rather than code; I'm still not entirely sure what your use case looks like. As best I can parse it, a nested try-except sounds like it would work. People like use-cases because they can reveal motivations or subtleties that often turn out to be significant, and they can more easily critique them than completely abstract/general cases. Cheers, Chris From pyideas at rebertia.com Sun Jan 22 22:59:47 2012 From: pyideas at rebertia.com (Chris Rebert) Date: Sun, 22 Jan 2012 13:59:47 -0800 Subject: [Python-ideas] Combining test and assignment In-Reply-To: References: <4F1B3AD8.6030402@eikehein.com> <4F1B95C1.7020402@pearwood.info> Message-ID: > On Sun, Jan 22, 2012 at 1:36 PM, Chris Rebert wrote: >> On Sun, Jan 22, 2012 at 4:04 AM, Paul Moore wrote: >> > On 22 January 2012 04:51, Steven D'Aprano wrote: >> >> Assignment as an expression feels unnatural to me: >> >> >> >> if spam() as x != 'ham': >> >> ? ?frobulate(x) >> >> >> >> doesn't really correspond to any natural English order. >> > >> > While in general I agree with you, there *is* a natural reading of this: >> > >> > "if spam() isn't 'ham', frobulate it" >> > >> > The fact that you have to choose a name is because computer languages >> > have to be more explicit than natural languages, and can't deal with >> > the implicit referent involved in the English usage of "it". The need >> > for assignment comes from the need for a name. You could actually deal >> > with the if-or-while-expression-assignment case by defining a special >> > variable __it__ to mean the value of the condition in the containing >> > if or while expression, but it has all the same lack of generality >> > issues as any proposal that limits itself to if/while conditions. And >> > short of explicitly marking the expression somehow, you can't >> > generalise any further. >> >> This is reminiscent of Perl's $_ variable, which represents the "current >> topic". >> http://www.perl.com/pub/2002/10/30/topic.html On Sun, Jan 22, 2012 at 1:24 PM, Edward Lesmes wrote: > why not make python do something like > _ = test() > if _: > ? ? ... > Automatically, so you can do something like > > if test(): > ? ? frobulate(_) > > > It seems natural to me, because in prompt, > python assigns the last result to the > _ variable I think "Explicit is better than implicit"[0] pretty much covers why that sort of thing is generally disfavored. Cheers, Chris -- [0]: http://www.python.org/dev/peps/pep-0020/ From mwm at mired.org Sun Jan 22 23:24:12 2012 From: mwm at mired.org (Mike Meyer) Date: Sun, 22 Jan 2012 14:24:12 -0800 Subject: [Python-ideas] Combining test and assignment In-Reply-To: References: <4F1B3AD8.6030402@eikehein.com> <4F1B95C1.7020402@pearwood.info> Message-ID: <20120122142412.772fc323@bhuda.mired.org> On Sun, 22 Jan 2012 12:04:48 +0000 Paul Moore wrote: > On 22 January 2012 04:51, Steven D'Aprano wrote: > > Assignment as an expression feels unnatural to me: > > > > if spam() as x != 'ham': > > ? ?frobulate(x) > > > > doesn't really correspond to any natural English order. > > While in general I agree with you, there *is* a natural reading of this: > > "if spam() isn't 'ham', frobulate it" > > The fact that you have to choose a name is because computer languages > have to be more explicit than natural languages, and can't deal with > the implicit referent involved in the English usage of "it". Others have pointed out that they can. My favorite example is Paul Graham's anaphoric macros, which bind the evaluated expression to "it" and then call the associated blocks. Two things of note in his discussion (http://dunsmor.com/lisp/onlisp/onlisp_18.html). First, he provides versions for the other conditional execution statements. Second, the one that is *most* used - he claims even more often than the regular version - is for while. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From steve at pearwood.info Sun Jan 22 23:58:26 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 23 Jan 2012 09:58:26 +1100 Subject: [Python-ideas] Combining test and assignment In-Reply-To: References: <4F1B3AD8.6030402@eikehein.com> <4F1B95C1.7020402@pearwood.info> Message-ID: <20120122225826.GC4148@ando> On Sun, Jan 22, 2012 at 12:04:48PM +0000, Paul Moore wrote: > On 22 January 2012 04:51, Steven D'Aprano wrote: > > Assignment as an expression feels unnatural to me: > > > > if spam() as x != 'ham': > > ? ?frobulate(x) > > > > doesn't really correspond to any natural English order. > > While in general I agree with you, there *is* a natural reading of this: > > "if spam() isn't 'ham', frobulate it" That's not quite the same, because there is an implicit assignment, which is ambiguous. Are you frobulating spam(), or 'ham'? Without having domain-specific knowledge, guessing is risky. E.g. this is easy to interpret: If the cat is on the mat, feed it. since we know that cats can eat but mats don't. But this is not: If the cat is on the mat, put it in the box. So I stand by my claim: there is no natural English analog of an explicit assignment in the middle of the clause. > The fact that you have to choose a name is because computer languages > have to be more explicit than natural languages, and can't deal with > the implicit referent involved in the English usage of "it". Actually they can. Hypertalk had at least two such implicit variables, "it" and "the result", with different rules for when each were set. I'm sure other languages have done similar. But really, it's not a great idea. The possibility of ambuiguity and confusion is too great. And what happens when you have two or more such variables? And in practice, you end up assigning "it" to a named variable anyway, otherwise it will be overwritten before you get around to using it. The CPython interactive interpreter already has one implicit variable, _ which holds the result of the previous command. I don't think Python the language should follow. -- Steven From mwm at mired.org Sun Jan 22 23:59:51 2012 From: mwm at mired.org (Mike Meyer) Date: Sun, 22 Jan 2012 14:59:51 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> Message-ID: <20120122145951.16abc641@bhuda.mired.org> On Sun, 22 Jan 2012 13:52:52 -0800 Chris Rebert wrote: > On Sun, Jan 22, 2012 at 1:30 PM, Mike Meyer wrote: > > On Sat, 21 Jan 2012 11:46:19 +0000 > > Paul Moore wrote: > >> On 21 January 2012 07:47, Steven D'Aprano wrote: > > >> I'd be -1 on a retry keyword that worked any way other than this > >> (simply because this is "clearly" the most obvious interpretation). > >> Whether a retry statement is worth having at all, though, is something > >> I'm not sure of - I'd like to see some real-world use cases first. > >> I've never encountered the need for it myself. And I can't honestly > >> imagine why the while True...try...break" idiom would ever not be > >> sufficient. > > > > I saw a number of request for "real world uses cases". I thought I > > covered that in the OP. This ideas was *prompted* by a real world use > > case where we wanted to wrap an exception in our own private exception > > before handling it. Because the code that would raise the exception > > - other exceptions missing the attributes we added to ours - was the > > code we want to run after handling them, we're left with LBYL or DRY > > or wrapping a loop around the code when it wasn't really a loop. > > The thing is, until this message, you never described any concrete > details about your specific use-case, other than that you had one. > Now you have, but in prose (which is regrettably imprecise) rather > than code; I'm still not entirely sure what your use case looks like. I can't give you the actual code (without having to deal with lawyers) because it comes from a client's proprietary product. I could make up code, but that's no better than the pseudo-code I gave in the OP. > As best I can parse it, a nested try-except sounds like it would work. I don't see how. Here's the pseudo-code example from the OP that most resembles the code we finally committed, doing LBYL instead of EAFP: if not isinstance(x, my_class): x = fixup(x) mangle(x) other_stuff(x) That doesn't show you where the exception would happen though, so here (also from the OP) is the version that violates DRY: try: mangle(x) except: x = fixup(x) mangle(x) other_stuff(x) The only thing the prose deception added to the above is that x is known to be an exception, and we wanted to make sure it had an appropriate attribute. The fact that it's an exception isn't really relevant, but that we needed it to have a specific attribute means those two might be better written as: if not isinstance(x, my_class): x = my_class(fixup(x)) x.mangle() other_stuff(x) and try: x.mangle() except: x = my_class(fixup(x)) x.mangle() other_stuff(x) If you've got a nested try/except solution that does EAFP and DRY, I'd be interested in seeing it. The OP included some loop-based variants as well. > People like use-cases because they can reveal motivations or > subtleties that often turn out to be significant, and they can more > easily critique them than completely abstract/general cases. Which is why I provided the best I could under the circumstances and why I'm one of the first to ask for them when they aren't present. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From jeanpierreda at gmail.com Mon Jan 23 00:15:49 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sun, 22 Jan 2012 18:15:49 -0500 Subject: [Python-ideas] Combining test and assignment In-Reply-To: <20120122225826.GC4148@ando> References: <4F1B3AD8.6030402@eikehein.com> <4F1B95C1.7020402@pearwood.info> <20120122225826.GC4148@ando> Message-ID: On Sun, Jan 22, 2012 at 5:58 PM, Steven D'Aprano wrote: > So I stand by my claim: there is no natural English analog of an > explicit assignment in the middle of the clause. There isn't a natural English analogue of explicit assignment at all. English doesn't have variables or anything like them. It has names, but these names cannot be redefined -- try telling someone "Bill is my cat", followed by, "Bill is my dad", and they'll give you a queer look. And then they'll interpret this as two different Bills, rather than just one that was "redefined" to mean something else, and if you refer to Bill again they will ask "which one?" There are unnatural phrases that work like assignment, and those same phrases can be used unnaturally to refer to assignment within an "if" as well. "Let the letter X refer to Y", "now let the letter X refer to Z", "if foo, which the letter X will now refer to, is a piece of ham, ..." But that's kind of silly. More generally, I'm not convinced programming has to fit into "natural" English. It should be expressible, yes, but it doesn't have to be idiomatic English. "Programming is an unnatural act. " (Perlis) (Perhaps it must be idiomatic in Dutch?) -- Devin From tjreedy at udel.edu Mon Jan 23 00:46:14 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 22 Jan 2012 18:46:14 -0500 Subject: [Python-ideas] Combining test and assignment In-Reply-To: References: <4F1B3AD8.6030402@eikehein.com> <4F1B95C1.7020402@pearwood.info> Message-ID: On 1/22/2012 7:04 AM, Paul Moore wrote: > On 22 January 2012 04:51, Steven D'Aprano wrote: >> Assignment as an expression feels unnatural to me: >> >> if spam() as x != 'ham': >> frobulate(x) >> >> doesn't really correspond to any natural English order. The best I can come up with is "if x, which is the result of calling 'spam', is not equal to 'ham', then frobulate x". But I am not going to call that 'natural' speech. More normal would be "Get the result of spam(). If it it not 'ham', then frobulate it." As others noted, the pronoun 'it' is a substitute for local names. But doing that for more than one object or value gets awkward. "Get ham() and spam(). If the first is less than the second, multiply the first by the second plus 3." "Former' and 'latter' are another pair of standardized pronoun-like names. The math device of temporary names is really handy. -- Terry Jan Reedy From jimjjewett at gmail.com Mon Jan 23 02:44:38 2012 From: jimjjewett at gmail.com (Jim Jewett) Date: Sun, 22 Jan 2012 20:44:38 -0500 Subject: [Python-ideas] Combining test and assignment In-Reply-To: <4F1B8B1C.8050106@eikehein.com> References: <4F1B3AD8.6030402@eikehein.com> <4F1B8B1C.8050106@eikehein.com> Message-ID: On Sat, Jan 21, 2012 at 11:05 PM, Eike Hein wrote: > On 1/22/2012 4:48 AM, Nick Coghlan wrote: >> Simple embedded assignment, though, only works when the >> predicate is just "bool" - as soon as the condition differs from the >> value you want to access, you need to revert to the existing idiom >> *anyway*. > Actually, just to be clear, in the use case I had in mind > spam() is returning a numeric value. That satisfies a truth > test and is still useful in other ways (but I assume you > are actually on the same page, hence the quotation marks > around bool). Perhaps. Does bool(spam()) produce the right answer? Or did you need to work with 0, or ignore -1? > Another, possibly better way is to modify locals(), i.e.: Well, not locals directly, but the following was inspired by the quantification thread. You still need to declare the Var before the test expression, but the declaration can be just a placeholder. x=Var() ... if Var(expr): -jJ -------------- next part -------------- class Var: """Var proxies another value; it allows assignment expressions Written by JimJJewett, inspired by Paul Moore's http://mail.python.org/pipermail/python-ideas/2012-January/013417.html >>> x=Var() >>> if x: ... print("True", x) ... else: ... print("False", x) False Var(None) >>> if x(5): ... print("True", x.value) ... else: ... print("False", x.value) True 5 >>> if x({}): ... print("True", x.value) ... else: ... print("False", x.value) False {} """ def __init__(self, value=None): self(value) def __call__(self, value): self.value=value; return self def __bool__(self): return bool(self.value) def __repr__(self): return "Var(%r)" % (self.value,) def _test(): import doctest doctest.testmod() if __name__ == "__main__": _test() From ncoghlan at gmail.com Mon Jan 23 03:18:37 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 23 Jan 2012 12:18:37 +1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <20120122133014.5e920a43@bhuda.mired.org> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> Message-ID: On Mon, Jan 23, 2012 at 7:30 AM, Mike Meyer wrote: > or wrapping a loop around the code when it wasn't really a loop. You want to do the same thing more than once: that's a loop. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From stephen at xemacs.org Mon Jan 23 05:34:44 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 23 Jan 2012 13:34:44 +0900 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> Message-ID: <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > On Mon, Jan 23, 2012 at 7:30 AM, Mike Meyer wrote: > > or wrapping a loop around the code when it wasn't really a loop. > > You want to do the same thing more than once: that's a loop. That's a question of point of view. If "thing" is thought of as a "try" (an operation that might fail), yes, he wants to do the same thing a nondeterministic number of times, and in general, more than once: it's a loop. If "thing" is thought of as a "block" (which isn't a try), then the "things" done with and without exception are different: it's not a loop, it's a conditional. Looking at his example "superficially" (I intend no deprecation by that word, just a point of view in looking at the code), I have some sympathy for Mike's claim that "it's not a loop and it violates DRY." I "know what he means" (and I bet you do, too!) However, when I try to define that, even informally, I arrive at the paragraph above, and I end up coming down on the side that you can't consistently claim that "it's not a loop" *and* claim that "it violates DRY", in some deeper sense. From ncoghlan at gmail.com Mon Jan 23 06:07:01 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 23 Jan 2012 15:07:01 +1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Jan 23, 2012 at 2:34 PM, Stephen J. Turnbull wrote: > Nick Coghlan writes: > ?> On Mon, Jan 23, 2012 at 7:30 AM, Mike Meyer wrote: > ?> > or wrapping a loop around the code when it wasn't really a loop. > ?> > ?> You want to do the same thing more than once: that's a loop. > > That's a question of point of view. ?If "thing" is thought of as a > "try" (an operation that might fail), yes, he wants to do the same > thing a nondeterministic number of times, and in general, more than > once: it's a loop. ?If "thing" is thought of as a "block" (which > isn't a try), then the "things" done with and without exception are > different: it's not a loop, it's a conditional. > > Looking at his example "superficially" (I intend no deprecation by > that word, just a point of view in looking at the code), I have some > sympathy for Mike's claim that "it's not a loop and it violates DRY." > I "know what he means" (and I bet you do, too!) ?However, when I try > to define that, even informally, I arrive at the paragraph above, and > I end up coming down on the side that you can't consistently claim > that "it's not a loop" *and* claim that "it violates DRY", in some > deeper sense. Exactly - in a very real sense, "retry once" is just a special case of a more general looping pattern: you execute the body either once or twice, perhaps with some massaging of state between the two attempts. One way to write it is: for attempt in (1, 2): # This iterable controls your number of attempts try: # Do whatever except ExpectedException: x = prevent_expected_exception(x) continue # continue == retry break # break == success (just like a search loop) else: raise Exception("Operation failed (made %d attempts)" % attempt) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From mwm at mired.org Mon Jan 23 19:26:06 2012 From: mwm at mired.org (Mike Meyer) Date: Mon, 23 Jan 2012 10:26:06 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20120123102606.3d4f5ff6@mikmeyer-vm-fedora> On Mon, 23 Jan 2012 15:07:01 +1000 Nick Coghlan wrote: > On Mon, Jan 23, 2012 at 2:34 PM, Stephen J. Turnbull > wrote: > > Nick Coghlan writes: > > ?> On Mon, Jan 23, 2012 at 7:30 AM, Mike Meyer > > wrote: > > or wrapping a loop around the code when it wasn't really > > a loop. > > > ?> You want to do the same thing more than once: that's a loop. > > > > That's a question of point of view. ?If "thing" is thought of as a > > "try" (an operation that might fail), yes, he wants to do the same > > thing a nondeterministic number of times, and in general, more than > > once: it's a loop. ?If "thing" is thought of as a "block" (which > > isn't a try), then the "things" done with and without exception are > > different: it's not a loop, it's a conditional. > > > > Looking at his example "superficially" (I intend no deprecation by > > that word, just a point of view in looking at the code), I have some > > sympathy for Mike's claim that "it's not a loop and it violates > > DRY." I "know what he means" (and I bet you do, too!) ?However, > > when I try to define that, even informally, I arrive at the > > paragraph above, and I end up coming down on the side that you > > can't consistently claim that "it's not a loop" *and* claim that > > "it violates DRY", in some deeper sense. > > Exactly - in a very real sense, "retry once" is just a special case of > a more general looping pattern: you execute the body either once or > twice, perhaps with some massaging of state between the two attempts. > > One way to write it is: > > for attempt in (1, 2): # This iterable controls your number of > attempts try: I think that highlights why it's not really a loop. While "retry once" is a general case of a looping pattern, it's *also* just a special case of a more general exception handling pattern. Because it fits under both special cases, it can be wrapped in a loop. But *any* sequence of instructions can be wrapped in a loop. All you need is a program counter (python is missing the tools to express it really cleanly, so I'm pretending): for pc in range(sys.maxsize): switch pc: 0: first_block 1: second_block 2, 3: third_block 4, 6: fourth_block 5: fifth_block 7: sixth_block if not done: pc -= 1 else: break pc += 1 So what of this is a real loop? The third block is in a two-count loop. The sixth block is in a while not done loop. The fourth block corresponds to the case here - you want to rerun one bit of code with another in the middle. You really don't want a general-purpose loop. So having to write one doesn't express the intent correctly. Of course, you can write a general-purpose loop this way. And there are probably cases where such a loop is the intent. Which lowers the value of a "retry" statement. References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1327345612.12382.51.camel@Gutsy> On Mon, 2012-01-23 at 13:34 +0900, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > On Mon, Jan 23, 2012 at 7:30 AM, Mike Meyer wrote: > > > or wrapping a loop around the code when it wasn't really a loop. > > > > You want to do the same thing more than once: that's a loop. > > That's a question of point of view. If "thing" is thought of as a > "try" (an operation that might fail), yes, he wants to do the same > thing a nondeterministic number of times, and in general, more than > once: it's a loop. If "thing" is thought of as a "block" (which > isn't a try), then the "things" done with and without exception are > different: it's not a loop, it's a conditional. > > Looking at his example "superficially" (I intend no deprecation by > that word, just a point of view in looking at the code), I have some > sympathy for Mike's claim that "it's not a loop and it violates DRY." > I "know what he means" (and I bet you do, too!) However, when I try > to define that, even informally, I arrive at the paragraph above, and > I end up coming down on the side that you can't consistently claim > that "it's not a loop" *and* claim that "it violates DRY", in some > deeper sense. You also have to consider the size and scope of the block. Anything more than one or two simple instructions will be difficult to achieve in a clean way. In the case of only one or to instruction, a try block works just fine. For larger scopes, I think it would require some form of "UNDO_BLOCK" opcode, so that the visited frame states can be restored before a retry attempt is done. It would break if any c code functions visits any frame that is outside the frames the byte code is executing in. I'd also be concerned about memory usage and speed, because it could lead to some very inefficient routines. Cheers, Ron From steve at pearwood.info Tue Jan 24 01:00:53 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 24 Jan 2012 11:00:53 +1100 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20120124000053.GC6693@ando> On Mon, Jan 23, 2012 at 01:34:44PM +0900, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > On Mon, Jan 23, 2012 at 7:30 AM, Mike Meyer wrote: > > > or wrapping a loop around the code when it wasn't really a loop. > > > > You want to do the same thing more than once: that's a loop. > > That's a question of point of view. If "thing" is thought of as a > "try" (an operation that might fail), yes, he wants to do the same > thing a nondeterministic number of times, and in general, more than > once: it's a loop. If "thing" is thought of as a "block" (which > isn't a try), then the "things" done with and without exception are > different: it's not a loop, it's a conditional. I'm afraid I don't understand this. What's a block (that isn't a try) here, and how does it differ from a thing, and what makes it a conditional? If all you are saying is that the caller may want to abstract out "repeat this thing until it succeeds (or fails permanently)" into a single operation without explicitly writing a for or while loop, that's what functions are for. map(func, seq) doesn't cease to be a loop just because you don't write it as an explicit loop. If you mean something different from this, I have no idea what you mean. > Looking at his example "superficially" (I intend no deprecation by > that word, just a point of view in looking at the code), I have some > sympathy for Mike's claim that "it's not a loop and it violates DRY." > I "know what he means" (and I bet you do, too!) I don't. To me, "retry this thing repeatedly" is fundamentally a loop. > However, when I try > to define that, even informally, I arrive at the paragraph above, and > I end up coming down on the side that you can't consistently claim > that "it's not a loop" *and* claim that "it violates DRY", in some > deeper sense. If it violates DRY, then by definition you must be repeating yourself. If you repeat yourself, then the obvious way to avoid repeating yourself is to place the repeated code inside a loop. Why is this a problem? What is especially confusing is that the proposed syntax is *defined* as looping back to the start of the try block, like a GOTO. If you draw the program flowchart of the construct, it loops backwards. Mike even stated that his proposal was exactly identical in behaviour to a try inside a while True loop. If he wants to argue that saving one indent level is so important that it's worth new syntax, that's one thing, but I am perplexed at claims that something which is identical to a loop isn't a loop. -- Steven From mwm at mired.org Tue Jan 24 02:06:33 2012 From: mwm at mired.org (Mike Meyer) Date: Mon, 23 Jan 2012 17:06:33 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <20120124000053.GC6693@ando> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> Message-ID: <20120123170633.1bfdbf11@mikmeyer-vm-fedora> On Tue, 24 Jan 2012 11:00:53 +1100 Steven D'Aprano wrote: > I don't. To me, "retry this thing repeatedly" is fundamentally a loop. What's being abstracted out isn't "retry this thing repeatedly". it's "I want to retry this thing after tweaking things if it fails." In particular, different ways of failure might require different tweaks before the retry, second failures would be handled differently from first failures, etc. References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> Message-ID: On Mon, Jan 23, 2012 at 8:06 PM, Mike Meyer wrote: > On Tue, 24 Jan 2012 11:00:53 +1100 > Steven D'Aprano wrote: >> I don't. To me, "retry this thing repeatedly" is fundamentally a loop. > What's being abstracted out isn't "retry this thing repeatedly". ?it's > "I want to retry this thing after tweaking things if it fails." In > particular, different ways of failure might require different tweaks > before the retry, second failures would be handled differently from > first failures, etc. So it doesn't seem like a loop because you hope to do it only once? Or because you're thinking of the thing-retried as the candidate loop, when that is just a constant function, and actual the loop is a loop over things-to-try-first (a loop over functions, rather than data)? -jJ From ncoghlan at gmail.com Tue Jan 24 02:20:18 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Jan 2012 11:20:18 +1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <20120123170633.1bfdbf11@mikmeyer-vm-fedora> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> Message-ID: On Tue, Jan 24, 2012 at 11:06 AM, Mike Meyer wrote: > On Tue, 24 Jan 2012 11:00:53 +1100 > Steven D'Aprano wrote: >> I don't. To me, "retry this thing repeatedly" is fundamentally a loop. > > What's being abstracted out isn't "retry this thing repeatedly". ?it's > "I want to retry this thing after tweaking things if it fails." In > particular, different ways of failure might require different tweaks > before the retry, second failures would be handled differently from > first failures, etc. But that's just normal loop-and-a-half behaviour, where the first half of the loop is consistent, but the second half depends on the results of the first half (including whether or not an exception is thrown). while attempts_remaining(): try: # attempt consistent operation except ExpectedException: # set up for next attempt based on result of current attempt # even the list of expected exceptions can be made dynamic! else: break # success! else: # All attempts failed! I'm not sure what it is about having an upper limit of 2 iterations, or having the loop exit criteria be "didn't throw an exception" rather than an ordinary conditional expression, that makes you feel like this construct isn't just an ordinary loop-and-a-half (and best thought about that way). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From stephen at xemacs.org Tue Jan 24 04:18:05 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 24 Jan 2012 12:18:05 +0900 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> Message-ID: <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> Jim Jewett writes: > So it doesn't seem like a loop because you hope to do it only once? With s/hope/expect/, that hits the nail on the head. I don't think syntax can express that cleanly, but I can't help thinking it's of good thing if somebody like Mike tries to find a way. He might succeed! From ncoghlan at gmail.com Tue Jan 24 04:56:38 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Jan 2012 13:56:38 +1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Jan 24, 2012 at 1:18 PM, Stephen J. Turnbull wrote: > I don't think syntax can express that cleanly, but I can't help > thinking it's of good thing if somebody like Mike tries to find a way. > He might succeed! Special casing "looping with at most two iterations" and "looping where the body is a single try statement" both seem like very poor ideas. OK, so some people apparently take issue with having to map "retry" to "loop", but how does that even come close to justifying making *everyone* learn a third looping construct? We can't even get consensus that PEP 315's generalised while loops (which allow you to write loop-and-a-half constructs without using break) would be a net win for the language over the existing idiom. I'll note that under PEP 315, the problem discussed in this thread could be handled as: do ... while retry: # '...' stands in for the suite below retry = False try: # attempt consistent operation except ExpectedException: # set up for next attempt based on result of current attempt # even the list of expected exceptions can be made dynamic! retry = True Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From mwm at mired.org Tue Jan 24 05:44:27 2012 From: mwm at mired.org (Mike Meyer) Date: Mon, 23 Jan 2012 20:44:27 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: "Stephen J. Turnbull" wrote: >Jim Jewett writes: > > > So it doesn't seem like a loop because you hope to do it only once? > >With s/hope/expect/, that hits the nail on the head. > >I don't think syntax can express that cleanly, but I can't help >thinking it's of good thing if somebody like Mike tries to find a way. Right. Doing it more than once is an exceptional case. And is more exceptional every time through. That using an LBYL idiom, or simply repeating yourself, means you don't need a loop says to me that it isn't really a loop. Which is why using loop to write it feels wrong. That I translated into a loop was because the only tools Python has for this is a loop. That no more makes it a loop than map being a function call makes it not a loop. I'd say that using retry to write what really is a loop (ie - you can't know a maximum number of iterations at compile time) would constitute abuse. A very attractive abuse, at that. Which certainly counts against out. >He might succeed! Or someone else will figure it out after I've correctly described the problem. -- Sent from my Android tablet with K-9 Mail. Please excuse my brevity. From ncoghlan at gmail.com Tue Jan 24 06:18:00 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Jan 2012 15:18:00 +1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Jan 24, 2012 at 2:44 PM, Mike Meyer wrote: > That using an LBYL idiom, or simply repeating yourself, means you don't need a loop says to me that it isn't really a loop. ?Which is why using loop to write it feels wrong. A retry is just a specific kind of loop that executes 1 or 2 times (or perhaps more if you're allowed to trigger the retry more than once). You don't actually need loops in general, since you can use recursion or repetition instead, so saying "look, i can rewrite it without the loop, so it's not really a loop!" doesn't mean all that much in an objective sense. The argument that it might be worth having dedicated syntax for a loop that runs 1 or 2 times is rather unconvincing when we don't even have dedicated syntax for a loop that runs 1 or more times (see PEP 315). We've survived this long with two variants of a loop that runs 0 or more times and using break as appropriate to handle all the other cases (e.g. while+break provides loop-and-a-half semantics and I've written for loops with an unconditional break at the end to get "0 or 1" iteration). Making the case that we need another looping construct is a fairly tall order (even PEP 315 doesn't suggest a completely new construct - it only proposes a generalisation of the existing while loops). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From mwm at mired.org Tue Jan 24 06:47:21 2012 From: mwm at mired.org (Mike Meyer) Date: Mon, 23 Jan 2012 21:47:21 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> Nick Coghlan wrote: >On Tue, Jan 24, 2012 at 2:44 PM, Mike Meyer wrote: >> That using an LBYL idiom, or simply repeating yourself, means you >don't need a loop says to me that it isn't really a loop. ?Which is why >using loop to write it feels wrong. >The argument that it might be worth having dedicated syntax for a loop >that runs 1 or 2 times is rather unconvincing when we don't even have >dedicated syntax for a loop that runs 1 or more times (see PEP 315). The argument isn't that we need a new syntax for a small set of loops, it's that the only ways to implement retrying after an exception leave a code smell. -- Sent from my Android tablet with K-9 Mail. Please excuse my brevity. From ncoghlan at gmail.com Tue Jan 24 08:00:28 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Jan 2012 17:00:28 +1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> Message-ID: On Tue, Jan 24, 2012 at 3:47 PM, Mike Meyer wrote: > The argument isn't that we need a new syntax for a small set of loops, it's that the only ways to implement retrying after an exception leave a code smell. Uh, saying "retrying is fundamentally a looping operation" is not a code smell. How do you plan to implement retry if not as a looping construct under the hood? No matter how many times you assert otherwise, "go back to this earlier point in the code and resume execution from there" is pretty much the *definition* of a loop. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From cmjohnson.mailinglist at gmail.com Tue Jan 24 08:19:08 2012 From: cmjohnson.mailinglist at gmail.com (Carl M. Johnson) Date: Mon, 23 Jan 2012 21:19:08 -1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> Message-ID: <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> On Jan 23, 2012, at 9:00 PM, Nick Coghlan wrote: > On Tue, Jan 24, 2012 at 3:47 PM, Mike Meyer wrote: >> The argument isn't that we need a new syntax for a small set of loops, it's that the only ways to implement retrying after an exception leave a code smell. > > Uh, saying "retrying is fundamentally a looping operation" is not a > code smell. I do think there's something code smelly about a retry--if it didn't work the first time, why should it work the second time after you give it a whack? Either whacking is good and you should do it in advance or it's bad and you should do something more sophisticated--but I don't see how creating language level support for retrying would remove the smell. Accessing deeply.nested.methods.and_.properties is a code smell too, even though it has language level support from Python. (You could imagine it being otherwise, if Python insisted that each attribute access get its own line, but that wouldn't remove the smell either.) The whole point of a "smell" is that it's not directly bad, but it's a sign that maybe you were thinking of something wrong at a different level, so it's time to re-architect a little. From wuwei23 at gmail.com Tue Jan 24 08:33:16 2012 From: wuwei23 at gmail.com (alex23) Date: Mon, 23 Jan 2012 23:33:16 -0800 (PST) Subject: [Python-ideas] Combining test and assignment In-Reply-To: <4F1B8E10.9050609@eikehein.com> References: <4F1B3AD8.6030402@eikehein.com> <4F1B8B1C.8050106@eikehein.com> <4F1B8E10.9050609@eikehein.com> Message-ID: <4c498c9b-9287-4eb6-ace0-3a761bc93701@pk8g2000pbb.googlegroups.com> On Jan 22, 2:18?pm, Eike Hein wrote: > Can anyone think of another way to do an assignment > and a truth test in the same expression with today's > tools? I've seen something similar to this used before: class Holder(object): def __call__(self, value): self.value = value return self def __eq__(self, val): return self.value == val __req__ = __eq__ from random import choice some_func = lambda: choice(['a','b','c']) holder = Holder() if holder(some_func()) == 'a': print 'A' elif holder == 'b': print 'B' elif holder == 'c': print 'C' As you can see, it still requires a value to be defined before the conditional, though :) From ncoghlan at gmail.com Tue Jan 24 08:34:37 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 24 Jan 2012 17:34:37 +1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> Message-ID: On Tue, Jan 24, 2012 at 5:19 PM, Carl M. Johnson wrote: > I don't see how creating language level support for retrying would remove the smell. I didn't even think of that aspect - very good point :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From p.f.moore at gmail.com Tue Jan 24 10:47:38 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Jan 2012 09:47:38 +0000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 24 January 2012 03:18, Stephen J. Turnbull wrote: > Jim Jewett writes: > > ?> So it doesn't seem like a loop because you hope to do it only once? > > With s/hope/expect/, that hits the nail on the head. > > I don't think syntax can express that cleanly, but I can't help > thinking it's of good thing if somebody like Mike tries to find a way. > He might succeed! If Python was like Haskell or Lisp, where you can define your own control structures, and this was a proposal to add a new control structure to the stdlib, I'd be fine with it. It *is* a somewhat unusual case, and from some people's viewpoint, certainly, the construct is not really a loop. So having "from exception_handling import retry" would be plausible. But Python doesn't have user defined control structures, by design, and indeed takes a strongly minimalist position on adding control structures - we don't even have do...while (yet), as Nick has pointed out. So in that context, I believe that this case is not sufficiently special, and the "not a loop" viewpoint is not sufficiently universal, to warrant a language change. Certainly, I wouldn't argue that this should be part of the language before do...while (which is more generally useful) gets in. Paul. From julien at tayon.net Tue Jan 24 11:19:08 2012 From: julien at tayon.net (julien tayon) Date: Tue, 24 Jan 2012 11:19:08 +0100 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> Message-ID: 2012/1/24 Carl M. Johnson : > > On Jan 23, 2012, at 9:00 PM, Nick Coghlan wrote: > >> On Tue, Jan 24, 2012 at 3:47 PM, Mike Meyer wrote: >>> The argument isn't that we need a new syntax for a small set of loops, it's that the only ways to implement retrying after an exception leave a code smell. >> >> Uh, saying "retrying is fundamentally a looping operation" is not a >> code smell. > > I do think there's something code smelly about a retry--if it didn't work the first time, why should it work the second time after you give it a whack? Either whacking is good and you should do it in advance or it's bad and you should do something more sophisticated--but I don't see how creating language level support for retrying would remove the smell. Accessing deeply.nested.methods.and_.properties is a code smell too, even though it has language level support from Python. (You could imagine it being otherwise, if Python insisted that each attribute access get its own line, but that wouldn't remove the smell either.) When could you need to retry a second time in a legitimate way? Mhhh, because you are trying to acquire a DHCP lease, you made a DHCPDISCOVER, and you have to retry on the same udp connexion you opened as long as you have no response, or you dont dont have a timeout, therefore you need to count the retry/time elapsed. Following the metaphor of the DHCP lease, where you need to retry with the same context than the try, you'd then claim that you may also need DHCPLEASE handling (a low level auth in DHCP) and retry all the way down from the DHCPDISCOVER if your IP is not accepted after a DHCPNAK. And then you discover that these problems of error handling with keeping context would best be handled in an evenemential fashion rather than with adding a weired hack in sequential programming. This case, and others are best treated with twisted or tornado, but do you really want to import tornado or twisted when you need a rough finite state automata ? (I don't have the answer) -- Regards, Jul From steve at pearwood.info Tue Jan 24 12:19:35 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 24 Jan 2012 22:19:35 +1100 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> Message-ID: <4F1E93C7.4050703@pearwood.info> Carl M. Johnson wrote: > On Jan 23, 2012, at 9:00 PM, Nick Coghlan wrote: > >> On Tue, Jan 24, 2012 at 3:47 PM, Mike Meyer wrote: >>> The argument isn't that we need a new syntax for a small set of loops, >>> it's that the only ways to implement retrying after an exception leave >>> a code smell. >> Uh, saying "retrying is fundamentally a looping operation" is not a code >> smell. > > I do think there's something code smelly about a retry--if it didn't work > the first time, why should it work the second time after you give it a > whack? Either whacking is good and you should do it in advance or it's bad > and you should do something more sophisticated "Whacking" is not necessarily bad or difficult, and is not necessarily a code smell. There are many common situations where "try again" is natural and expected. E.g. in response to a busy signal, you should wait a little while before retrying: delay = 5 # Initial delay between attempts in seconds. for _ in range(MAX_ATTEMPTS): try: response = urllib2.urlopen(url) except urllib2.HTTPError as e: if e.code == 503: # Service Unavailable. time.sleep(delay) delay *= 2 # Exponential back-off. else: raise else: break This could be written without the for-loop using a hypothetical retry statement, but it doesn't really gain us much: delay = 2 # Initial delay between attempts in seconds. count = 0 try: response = urllib2.urlopen(url) except urllib2.HTTPError as e: if e.code == 503 and count < MAX_ATTEMPTS: # Service Unavailable time.sleep(delay) delay *= 2 # Exponential back-off count += 1 retry else: raise Although you save one indent level, you don't save any lines of code, and it costs you the effort of handling the book-keeping that a for-loop would give you for free. I don't count this as a win. -- Steven From p.f.moore at gmail.com Tue Jan 24 12:36:29 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 24 Jan 2012 11:36:29 +0000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <4F1E93C7.4050703@pearwood.info> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> <4F1E93C7.4050703@pearwood.info> Message-ID: On 24 January 2012 11:19, Steven D'Aprano wrote: > "Whacking" is not necessarily bad or difficult, and is not necessarily a > code smell. There are many common situations where "try again" is natural > and expected. E.g. in response to a busy signal, you should wait a little > while before retrying: [...] > Although you save one indent level, you don't save any lines of code, and it > costs you the effort of handling the book-keeping that a for-loop would give > you for free. I don't count this as a win. Having read that, I agree that the "retry" doesn't buy you much. But I do think there's a pattern in there that it would be nice to be able to abstract out, namely the exponential back-off. The pattern of "try X, if Y happens, then do Z and wait, then retry" is probably common enough in certain types of application that it would be nice to encapsulate it in a library routine. (I haven't needed it myself, admittedly...) I can't quite see how to do it, though - I thought about clever uses of with statements, or abstract classes with defined callback methods you could use, but couldn't find anything obvious. (The big issue being that "Y" is that a particular exception is raised in this case, but may be something else in general). A construct that let end users abstract this type of pattern would probably be a far bigger win than a retry statement. (And it may be that it has the benefit of already existing, I just couldn't see it :-)) Paul. From ncoghlan at gmail.com Tue Jan 24 15:06:23 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Jan 2012 00:06:23 +1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> <4F1E93C7.4050703@pearwood.info> Message-ID: On Tue, Jan 24, 2012 at 9:36 PM, Paul Moore wrote: > A construct that let end users abstract this type of pattern would > probably be a far bigger win than a retry statement. (And it may be > that it has the benefit of already existing, I just couldn't see it > :-)) You just need to move the pause inside the iterator: def backoff(attempts, first_delay, scale=2): delay = first_delay for attempt in range(1, attempts+1): yield attempt time.sleep(delay) delay *= 2 for __ in backoff(MAX_ATTEMPTS, 5): try: response = urllib2.urlopen(url) except urllib2.HTTPError as e: if e.code == 503: # Service Unavailable. continue raise break You can also design smarter versions where the object yielded is mutable, making it easy to pass state back into the iterator. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jkbbwr at gmail.com Tue Jan 24 16:21:19 2012 From: jkbbwr at gmail.com (Jakob Bowyer) Date: Tue, 24 Jan 2012 15:21:19 +0000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> <4F1E93C7.4050703@pearwood.info> Message-ID: Would this not be better expressed as a context manager? with backoff(maxattempts, 5): # do stuff Sent from my iPad On 24 Jan 2012, at 14:06, Nick Coghlan wrote: > On Tue, Jan 24, 2012 at 9:36 PM, Paul Moore wrote: >> A construct that let end users abstract this type of pattern would >> probably be a far bigger win than a retry statement. (And it may be >> that it has the benefit of already existing, I just couldn't see it >> :-)) > > You just need to move the pause inside the iterator: > > def backoff(attempts, first_delay, scale=2): > delay = first_delay > for attempt in range(1, attempts+1): > yield attempt > time.sleep(delay) > delay *= 2 > > for __ in backoff(MAX_ATTEMPTS, 5): > try: > response = urllib2.urlopen(url) > except urllib2.HTTPError as e: > if e.code == 503: # Service Unavailable. > continue > raise > break > > You can also design smarter versions where the object yielded is > mutable, making it easy to pass state back into the iterator. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From mwm at mired.org Tue Jan 24 18:19:50 2012 From: mwm at mired.org (Mike Meyer) Date: Tue, 24 Jan 2012 09:19:50 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> Message-ID: <20120124091950.435c3519@mikmeyer-vm-fedora> On Tue, 24 Jan 2012 17:00:28 +1000 Nick Coghlan wrote: > On Tue, Jan 24, 2012 at 3:47 PM, Mike Meyer wrote: > > The argument isn't that we need a new syntax for a small set of > > loops, it's that the only ways to implement retrying after an > > exception leave a code smell. > Uh, saying "retrying is fundamentally a looping operation" is not a > code smell. No, the code smell is a loop that just gets run through once. I mean, I can write any if statement as a while loop, but it's probably a bad idea and a code smell. References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> <4F1E93C7.4050703@pearwood.info> Message-ID: > On 24 Jan 2012, at 14:06, Nick Coghlan wrote: >> On Tue, Jan 24, 2012 at 9:36 PM, Paul Moore wrote: >>> A construct that let end users abstract this type of pattern would >>> probably be a far bigger win than a retry statement. (And it may be >>> that it has the benefit of already existing, I just couldn't see it >>> :-)) >> >> You just need to move the pause inside the iterator: >> >> ? ?def backoff(attempts, first_delay, scale=2): >> ? ? ? ?delay = first_delay >> ? ? ? ?for attempt in range(1, attempts+1): >> ? ? ? ? ? ?yield attempt >> ? ? ? ? ? ?time.sleep(delay) >> ? ? ? ? ? ?delay *= 2 >> >> ? ?for __ in backoff(MAX_ATTEMPTS, 5): >> ? ? ? try: >> ? ? ? ? ? response = urllib2.urlopen(url) >> ? ? ? except urllib2.HTTPError as e: >> ? ? ? ? ? if e.code == 503: ?# Service Unavailable. >> ? ? ? ? ? ? ? continue >> ? ? ? ? ? raise >> ? ? ? break On Tue, Jan 24, 2012 at 7:21 AM, Jakob Bowyer wrote: > Would this not be better expressed as a context manager? > > with backoff(maxattempts, 5): > ? ?# do stuff It can't be. The `with` statement always executes its block exactly once; the context manager(s) have no say in the matter (unless perhaps you count raising an exception prior to the block's execution). Cheers, Chris From mwm at mired.org Tue Jan 24 19:09:34 2012 From: mwm at mired.org (Mike Meyer) Date: Tue, 24 Jan 2012 10:09:34 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> Message-ID: <20120124100934.063344a7@mikmeyer-vm-fedora> On Mon, 23 Jan 2012 21:19:08 -1000 "Carl M. Johnson" wrote: > On Jan 23, 2012, at 9:00 PM, Nick Coghlan wrote: > > On Tue, Jan 24, 2012 at 3:47 PM, Mike Meyer wrote: > >> The argument isn't that we need a new syntax for a small set of > >> loops, it's that the only ways to implement retrying after an > >> exception leave a code smell. > > Uh, saying "retrying is fundamentally a looping operation" is not a > > code smell. > I do think there's something code smelly about a retry--if it didn't > work the first time, why should it work the second time after you > give it a whack? Either whacking is good and you should do it in > advance That's the LBYL way of expressing the code. You know, the one that manages to express the "loop" without either repeating any code or using an actual loop. Except applying the fix without checking to see if it's needed is most likely a bug. You might want to argue that LBYL isn't a code smell. I've fixed enough bugs caused by it to disagree, but at this point it's a style argument. References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> Message-ID: <4F1EFA53.50008@pearwood.info> Mike Meyer wrote: > On Tue, 24 Jan 2012 17:00:28 +1000 > Nick Coghlan wrote: > >> On Tue, Jan 24, 2012 at 3:47 PM, Mike Meyer wrote: >>> The argument isn't that we need a new syntax for a small set of >>> loops, it's that the only ways to implement retrying after an >>> exception leave a code smell. >> Uh, saying "retrying is fundamentally a looping operation" is not a >> code smell. > > No, the code smell is a loop that just gets run through once. By this reasoning, "for i in range(n)" is a code smell, because n might happen to be 1. You can't know that the loop will run once until you actually try. The point of the retry idiom is that it could run twice, thrice, four times, ... up to whatever limit you impose (if any!). Even if you expect that 999 times out of a thousand it will only run once, you write it in a loop because you want to cover the 1 remaining time where it will run multiple times. That's not a code smell, it is the obvious way to write an operation that may need to be retried. > I mean, > I can write any if statement as a while loop, but it's probably a bad > idea and a code smell. There's no limit to the bizarre and obfuscatory code that a clever enough, or foolish enough, coder can write. But putting something that needs to run one OR MORE times inside a loop is neither bizarre nor obfuscatory. That's what loops are for. -- Steven From mwm at mired.org Tue Jan 24 19:45:36 2012 From: mwm at mired.org (Mike Meyer) Date: Tue, 24 Jan 2012 10:45:36 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <4F1EFA53.50008@pearwood.info> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> <4F1EFA53.50008@pearwood.info> Message-ID: <20120124104536.788b1186@mikmeyer-vm-fedora> On Wed, 25 Jan 2012 05:37:07 +1100 Steven D'Aprano wrote: > Mike Meyer wrote: > > On Tue, 24 Jan 2012 17:00:28 +1000 > > Nick Coghlan wrote: > > > >> On Tue, Jan 24, 2012 at 3:47 PM, Mike Meyer wrote: > >>> The argument isn't that we need a new syntax for a small set of > >>> loops, it's that the only ways to implement retrying after an > >>> exception leave a code smell. > >> Uh, saying "retrying is fundamentally a looping operation" is not a > >> code smell. > > > > No, the code smell is a loop that just gets run through once. > > By this reasoning, "for i in range(n)" is a code smell, because n > might happen to be 1. Not quite, because n might also happen to *not* be 1. You could even run it no times, if n were 0. Nothing wrong with any of that. > You can't know that the loop will run once until you actually try. Which is not a code smell. However, if you can tell by reading the code that it will only run once (or never run), like this one: for i in range(1): Then it's a code smell! References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> <4F1EFA53.50008@pearwood.info> <20120124104536.788b1186@mikmeyer-vm-fedora> Message-ID: <4F1F039F.5040406@stoneleaf.us> Mike Meyer wrote: > On Wed, 25 Jan 2012 05:37:07 +1100 > Steven D'Aprano wrote: >> You can't know that the loop will run once until you actually try. > > Which is not a code smell. However, if you can tell by reading the > code that it will only run once (or never run), like this one: > > for i in range(1): > > Then it's a code smell! Absolutely. But your retry may run through twice. :) ~Ethan~ From ncoghlan at gmail.com Wed Jan 25 01:04:37 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Jan 2012 10:04:37 +1000 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <20120124104536.788b1186@mikmeyer-vm-fedora> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> <4F1EFA53.50008@pearwood.info> <20120124104536.788b1186@mikmeyer-vm-fedora> Message-ID: On Wed, Jan 25, 2012 at 4:45 AM, Mike Meyer wrote: > Which is not a code smell. However, if you can tell by reading the > code that it will only run once (or never run), like this one: > > ? ? for i in range(1): > > Then it's a code smell! I agree specifically in regards to range() with a literal argument, but it really depends on the iterator. Using break to force a single iteration can be nicer than calling next() and catching StopIteration. For example, code like the following makes it easy to special case the first entry in an iterator: walk_iter = os.walk(topdir) for dirpath, files, subdirs in walk_iter: # Special case the top level directory break else: raise RuntimeError("No dir entry for {!r}".format(topdir)) for dirpath, files, subdirs in walk_iter: # Process the subdirectories Python has very powerful looping constructs already - we don't need more just because some folks have been trained to think that break and continue are evil and haven't considered whether or not these prejudices inherited from C and C++ are still relevant in Python. Like early returns, break and continue are potentially dangerous in C and C++ because having multiple exit points from a scope increases the chance of leaking memory (or some other resource). By contrast, garbage collection and context managers mean that making appropriate use of early returns, break and continue is quite easy and safe in Python (and often clearer than the alternatives that try to avoid them). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From mwm at mired.org Wed Jan 25 01:52:01 2012 From: mwm at mired.org (Mike Meyer) Date: Tue, 24 Jan 2012 16:52:01 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> <4F1EFA53.50008@pearwood.info> <20120124104536.788b1186@mikmeyer-vm-fedora> Message-ID: <20120124165201.1b91c620@mikmeyer-vm-fedora> On Wed, 25 Jan 2012 10:04:37 +1000 Nick Coghlan wrote: > On Wed, Jan 25, 2012 at 4:45 AM, Mike Meyer wrote: > > Which is not a code smell. However, if you can tell by reading the > > code that it will only run once (or never run), like this one: > > > > ? ? for i in range(1): > > > > Then it's a code smell! > > I agree specifically in regards to range() with a literal argument, > but it really depends on the iterator. Using break to force a single > iteration can be nicer than calling next() and catching StopIteration. > For example, code like the following makes it easy to special case the > first entry in an iterator: > > walk_iter = os.walk(topdir) > for dirpath, files, subdirs in walk_iter: > # Special case the top level directory > break > else: > raise RuntimeError("No dir entry for {!r}".format(topdir)) > for dirpath, files, subdirs in walk_iter: > # Process the subdirectories And we've now gotten to the level of picking apart real examples. I don't think a loop that never loops is a code smell because I think break or continue are evil (the opposite, in fact; I'd like a multi-level break/continue, but that's for another topic). I think it's a code smell because there's a good chance it can be refactored into straight-line code that would be better in a number of ways. For instance, I would write your example like so: try: walk_iter = os.walk(topdir) dirpath, files, subdirs = next(wi) # Special case the top level directory except StopIteration: raise RuntimeError("No dir entry for {!r}".format(topdir)) # other exception handling here, as appropriate. for dirpath, files, subdirs in walk_iter: # Process the subdirectories Exactly what winds up in the try block will vary depending on circumstances. Putting only the invocation of next in it would duplicate your code. os.walk isn't documented as returning exceptions, and ignores problems with listdir, so adding it seems to be a nop. Handling the special case for the top level directory with this try seems like a win, because all the exceptions from dealing with the top level directory get grouped together. It might be useful to handle exceptions from the subdir processing as well, but the choice is easy to make with this version. In any case, it's easier to change this version as needed to deal with exceptions than the original version. The other issue may be just me - I expect exiting a loop from the bottom to be a normal flow path. So the for version reads to me like raising RuntimeError is normal, not exceptional. > Python has very powerful looping constructs already - we don't need > more just because some folks have been trained to think that break and > continue are evil and haven't considered whether or not these > prejudices inherited from C and C++ are still relevant in Python. Likewise, the try statement is very powerful. That break/continue may or may not be evil is immaterial. The point is to make the try statement more powerful. So long as you incorrectly see this as "just another looping construct", your conclusions will be flawed. Not necessarily wrong, just flawed. It can also be used to fix LBYL and DRY code smells. References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> <4F1EFA53.50008@pearwood.info> <20120124104536.788b1186@mikmeyer-vm-fedora> <20120124165201.1b91c620@mikmeyer-vm-fedora> Message-ID: On Wed, Jan 25, 2012 at 10:52 AM, Mike Meyer wrote: > So long as you incorrectly see this as "just another looping > construct", your conclusions will be flawed. Not necessarily wrong, > just flawed. It can also be used to fix LBYL and DRY code smells. It *is* just another looping construct: "retry" = "go back to the start of the try block". It's really just proposing a weird way to spell while+continue. Exception handling and repeating a section of code are *not* the same thing - it doesn't make sense to bundle the two into a single mega-construct. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From mwm at mired.org Wed Jan 25 02:19:43 2012 From: mwm at mired.org (Mike Meyer) Date: Tue, 24 Jan 2012 17:19:43 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> <4F1EFA53.50008@pearwood.info> <20120124104536.788b1186@mikmeyer-vm-fedora> <20120124165201.1b91c620@mikmeyer-vm-fedora> Message-ID: <20120124171943.57860103@mikmeyer-vm-fedora> On Wed, 25 Jan 2012 11:13:18 +1000 Nick Coghlan wrote: > Exception handling and repeating a section of code are *not* the same > thing - it doesn't make sense to bundle the two into a single > mega-construct. Exception handling and retrying the code that caused the exception are part of the same concept. There is value in being able to express that concept directly with a single construct. References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> <4F1EFA53.50008@pearwood.info> <20120124104536.788b1186@mikmeyer-vm-fedora> <20120124165201.1b91c620@mikmeyer-vm-fedora> Message-ID: <87ehuo4lll.fsf@uwakimon.sk.tsukuba.ac.jp> Mike Meyer writes: > The point is to make the try statement more powerful. Not on python-ideas, it isn't. Here, the point is to make Python more expressive (including clarity in "expressive"). That may mean making some statements less expressive (in the extreme, eliminating them entirely, as in the case of "print"). > So long as you incorrectly see this as "just another looping > construct", your conclusions will be flawed. That's unfair. Nobody said "just". If anything, you were trying in earlier posts to maintain the opposite extreme ("not a looping construct"). From vince.vinet at gmail.com Wed Jan 25 03:53:49 2012 From: vince.vinet at gmail.com (Vince) Date: Tue, 24 Jan 2012 21:53:49 -0500 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <20120124171943.57860103@mikmeyer-vm-fedora> References: <20120120163648.69288796@mikmeyer-vm-fedora> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> <4F1EFA53.50008@pearwood.info> <20120124104536.788b1186@mikmeyer-vm-fedora> <20120124165201.1b91c620@mikmeyer-vm-fedora> <20120124171943.57860103@mikmeyer-vm-fedora> Message-ID: On Tue, Jan 24, 2012 at 8:19 PM, Mike Meyer wrote: > Exception handling and retrying the code that caused the exception are > part of the same concept. There is value in being able to express that > concept directly with a single construct. It seems to me that combining looping and exception handling in the same construct makes code harder to read. When you come across any looping construct, you know immediately that the following block could occur any number of times, depending on the line where the construct starts. In the case of try/except/retry, it may come as a surprise that the block can actually repeat. Also, you would probably, in most cases, need to add additional logic before the "retry" to ensure that you don't loop forever if the "fixup" you apply does not prevent the exception from happening again. This logic, instead of being tied to the loop itself, now ends up well hidden in one or many possible exception handlers. From mwm at mired.org Wed Jan 25 04:22:30 2012 From: mwm at mired.org (Mike Meyer) Date: Tue, 24 Jan 2012 19:22:30 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> <4F1EFA53.50008@pearwood.info> <20120124104536.788b1186@mikmeyer-vm-fedora> <20120124165201.1b91c620@mikmeyer-vm-fedora> <20120124171943.57860103@mikmeyer-vm-fedora> Message-ID: <20120124192230.65cd5c7f@bhuda.mired.org> On Tue, 24 Jan 2012 21:53:49 -0500 Vince wrote: > On Tue, Jan 24, 2012 at 8:19 PM, Mike Meyer wrote: > > Exception handling and retrying the code that caused the exception are > > part of the same concept. There is value in being able to express that > > concept directly with a single construct. > > It seems to me that combining looping and exception handling in the > same construct makes code harder to read. When you come across any > looping construct, you know immediately that the following block could > occur any number of times, depending on the line where the construct > starts. In the case of try/except/retry, it may come as a surprise that the > block can actually repeat. Except the block in the try *doesn't* repeat. That block is only executed once. It may be *started* multiple times if there are exceptional conditions, but once it finishes executing, it's over. That's sort of the point. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From mwm at mired.org Wed Jan 25 04:26:12 2012 From: mwm at mired.org (Mike Meyer) Date: Tue, 24 Jan 2012 19:26:12 -0800 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <87ehuo4lll.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20120120163648.69288796@mikmeyer-vm-fedora> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> <4F1EFA53.50008@pearwood.info> <20120124104536.788b1186@mikmeyer-vm-fedora> <20120124165201.1b91c620@mikmeyer-vm-fedora> <87ehuo4lll.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20120124192612.50edd619@bhuda.mired.org> On Wed, 25 Jan 2012 10:28:06 +0900 "Stephen J. Turnbull" wrote: > Mike Meyer writes: > > So long as you incorrectly see this as "just another looping > > construct", your conclusions will be flawed. > That's unfair. Nobody said "just". If anything, you were trying in > earlier posts to maintain the opposite extreme ("not a looping > construct"). I don't know that nobody said "just". What got said by Nick - repeatedly - was "We don't need another looping construct." I don't think it's reasonable to dismiss it without considering how it works in the other roles for which it's suitable. > > The point is to make the try statement more powerful. > Not on python-ideas, it isn't. Here, the point is to make Python more > expressive (including clarity in "expressive"). Didn't we just have a discussion about the ambiguity of English, and how amorphism isn't pythonic? The point of the *proposal* is to make the try statement more powerful. The point of the *list* is to make Python more expressive. The point of the *thread* is to see if the former does the latter. Given that intro, I think the conclusion is "not this proposal", with opinions ranging from "we don't need it at all" to "Might be nice, but costs to much for the expected use cases." In thinking about it, I think it's to special purpose. There are at least four interesting variants of "retry": 1) start the block from the top; 2) start the block somewhere internally (and where?); 3&4) Same two, disabling the except handler that ran the retry. It's not clear there's even a sane way to define #2. http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org From tjreedy at udel.edu Wed Jan 25 04:33:19 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 24 Jan 2012 22:33:19 -0500 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <20120124165201.1b91c620@mikmeyer-vm-fedora> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <20120124091950.435c3519@mikmeyer-vm-fedora> <4F1EFA53.50008@pearwood.info> <20120124104536.788b1186@mikmeyer-vm-fedora> <20120124165201.1b91c620@mikmeyer-vm-fedora> Message-ID: On 1/24/2012 7:52 PM, Mike Meyer wrote: > For instance, I would write your example like so: > > try: > walk_iter = os.walk(topdir) > dirpath, files, subdirs = next(wi) > # Special case the top level directory > except StopIteration: > raise RuntimeError("No dir entry for {!r}".format(topdir)) > # other exception handling here, as appropriate. > > for dirpath, files, subdirs in walk_iter: > # Process the subdirectories I would too, but ... > Likewise, the try statement is very powerful. That break/continue may > or may not be evil is immaterial. The point is to make the try > statement more powerful. I disagree with turning 'try' into a specialized loop construct by adding a version of continue, which is a version of goto. We currently have a general overt loop (while), the covert equivalent (tail recursion), and a 'specialized' loop (for) that covers the majority of loop needs, leaving the generalized loop (while) to cover everything else. A 'try' statement in itself it a glorified label statement that also sets up the context for exception statements. Exception statement are conditional statements where the condition is an exception state. A marked program position plus a conditional jump is what makes a loop. -- Terry Jan Reedy From aquavitae69 at gmail.com Wed Jan 25 07:36:02 2012 From: aquavitae69 at gmail.com (David Townshend) Date: Wed, 25 Jan 2012 08:36:02 +0200 Subject: [Python-ideas] Allow Profile() to be used as a context manager Message-ID: I often find it useful to profile small sections to code in a running application to pinpoint a known bottleneck. The cProfile and profile modules don't make this easy as they stand, requiring the section of code to be wrapped into a single statement with can be called as a string using exec(). This always feels a bit clumsy to me, so my idea is simply to provide __enter__ and __exit__ methods to cProfile.Profile() and profile.Profile(). From a quick look at the code it seems that this should be very easy for cProfile, and slightly less easy for profile (I can't quite figure out how to deal with the set_cmd() line, or how important it is - maybe someone can help?). Any major objections to this? David -------------- next part -------------- An HTML attachment was scrubbed... URL: From anacrolix at gmail.com Wed Jan 25 08:08:59 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Wed, 25 Jan 2012 18:08:59 +1100 Subject: [Python-ideas] -m pstats should combine all the profiles given as arguments Message-ID: Frequently when profiling multiple threads, I need to combine several dump stat files. Currently -m pstats reads the profiling data at only the first path given. It should merge all the profiling data from all the paths given. $ python3.3 -m pstats prof/5506-7f00f* http://bugs.python.org/issue13839 From ubershmekel at gmail.com Wed Jan 25 09:19:33 2012 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Wed, 25 Jan 2012 10:19:33 +0200 Subject: [Python-ideas] Allow Profile() to be used as a context manager In-Reply-To: References: Message-ID: On Wed, Jan 25, 2012 at 8:36 AM, David Townshend wrote: > [...] > Any major objections to this? > > At first this sounded like a good idea but 2 things pop to mind: 1. How does one get the pstat profiling output from the context manager? Will printing out the results be the only option? 2. Usually when I profile I don't want to touch the code at all so eventually if I'll use this context manager it'll be more akin to with profile.Profiler(): main() Which isn't much of a win over profile.run('main()') Now if you're talking about a reentrant profiler, that's interesting: profiler = profile.Profiler() while is_working: with profiler: do_hard_task() do_a_hard_but_i_dont_care_now_task() with profiler: do_a_diff_hard_task() and then I can call whatever method is there to get the pstat. If this is the vision then it sounds awesome. Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jan 25 09:45:16 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Jan 2012 18:45:16 +1000 Subject: [Python-ideas] Coroutines and PEP 380 In-Reply-To: References: <4F15F041.6010607@hotpy.org> <20DB36E8-2538-4FE8-9FBF-6B3DA67E3CD6@twistedmatrix.com> <4F168FA5.2000503@hotpy.org> <7F3B6F9E-A901-4FA5-939E-CDD7B1E6E5B5@twistedmatrix.com> <4F188DFD.6080401@canterbury.ac.nz> <4F1B48D0.3060309@canterbury.ac.nz> Message-ID: (redirecting to python-ideas - coroutine proposals are nowhere near mature enough for python-dev) On Wed, Jan 25, 2012 at 5:35 PM, Matt Joiner wrote: > If someone can explain what's stopping real coroutines being into > Python (3.3), that would be great. The general issues with that kind of idea: - the author hasn't offered the code for inclusion and relicensing under the PSF license (thus we legally aren't allowed to do it) - complexity - maintainability - platform support In the specific case of coroutines, you have the additional hurdle of convincing people whether or not they're a good idea at all. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From g.rodola at gmail.com Wed Jan 25 10:40:16 2012 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Wed, 25 Jan 2012 10:40:16 +0100 Subject: [Python-ideas] Allow Profile() to be used as a context manager In-Reply-To: References: Message-ID: Il 25 gennaio 2012 07:36, David Townshend ha scritto: > I often find it useful to profile small sections to code in a running > application to pinpoint a known bottleneck. The cProfile and profile modules > don't make this easy as they stand, requiring the section of code to be > wrapped into a single statement with can be called as a string using exec(). > ?This always feels a bit clumsy to me, so my idea?is simply to provide > __enter__ and __exit__ methods to cProfile.Profile() and profile.Profile(). > ?From a quick look at the code it seems that this should be very easy for > cProfile, and slightly less easy for profile (I can't quite figure out how > to deal with the set_cmd() line, or how important it is - maybe someone can > help?). > > Any major objections to this? Related: http://bugs.python.org/issue9285 --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/ From aquavitae69 at gmail.com Wed Jan 25 10:47:54 2012 From: aquavitae69 at gmail.com (David Townshend) Date: Wed, 25 Jan 2012 11:47:54 +0200 Subject: [Python-ideas] Allow Profile() to be used as a context manager In-Reply-To: References: Message-ID: On Wed, Jan 25, 2012 at 10:19 AM, Yuval Greenfield wrote: > On Wed, Jan 25, 2012 at 8:36 AM, David Townshend wrote: > >> [...] >> Any major objections to this? >> >> > > At first this sounded like a good idea but 2 things pop to mind: > > > 1. How does one get the pstat profiling output from the context > manager? Will printing out the results be the only option? > > The same way as currently: profile = cProfile.Profile() with profile: do_something() profile.dump_stats(filename) Perhaps it would be a good idea to provide a convenience function (like run()), so that: with cProfile.do(filename): do_something() I don't like calling the function do(), but I can't think of anything better off-hand, and it illustrates my point. > > 1. Usually when I profile I don't want to touch the code at all so > eventually if I'll use this context manager it'll be more akin to > > > with profile.Profiler(): > main() > > Which isn't much of a win over > > profile.run('main()') > I agree, but this isn't the use case. > Now if you're talking about a reentrant profiler, that's interesting: > > profiler = profile.Profiler() > > while is_working: > with profiler: > do_hard_task() > > do_a_hard_but_i_dont_care_now_task() > > with profiler: > do_a_diff_hard_task() > > > and then I can call whatever method is there to get the pstat. If this is > the vision then it sounds awesome. > > And this is the use case! I'm not entirely sure how re-entrancy would be dealt with by the profiler though. From what I read in the source code it looks like it would all accumulate, and this is certainly the way I would expect it to behave, but I'd need to run a few tests to be sure. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Jan 25 11:16:58 2012 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 25 Jan 2012 10:16:58 +0000 Subject: [Python-ideas] Allow Profile() to be used as a context manager In-Reply-To: References: Message-ID: On 1/25/12 6:36 AM, David Townshend wrote: > I often find it useful to profile small sections to code in a running > application to pinpoint a known bottleneck. The cProfile and profile modules > don't make this easy as they stand, requiring the section of code to be wrapped > into a single statement with can be called as a string using exec(). This > always feels a bit clumsy to me, so my idea is simply to provide __enter__ and > __exit__ methods to cProfile.Profile() and profile.Profile(). From a quick look > at the code it seems that this should be very easy for cProfile, and slightly > less easy for profile (I can't quite figure out how to deal with the set_cmd() > line, or how important it is - maybe someone can help?). > > Any major objections to this? I've done this in my kernprof.py script, which I use as a convenient generic profiling script. I subclass cProfile.Profile to add enable_by_count()/disable_by_count() methods that allow nesting. enable_by_count() increments a counter and only calls the enable() method the first time. disable_by_count() decrements the counter and only calls disable() when it hits 0 again. __enter__() and __exit__() just call these methods. I also add a __call__() method that lets a Profile instance act as a decorator, which I actually find somewhat more useful than the context manager. https://bitbucket.org/robertkern/line_profiler/src/tip/kernprof.py#cl-57 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ncoghlan at gmail.com Wed Jan 25 11:24:39 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Jan 2012 20:24:39 +1000 Subject: [Python-ideas] Allow Profile() to be used as a context manager In-Reply-To: References: Message-ID: On Wed, Jan 25, 2012 at 7:47 PM, David Townshend wrote: > The same way as currently: > > profile = cProfile.Profile() > with profile: > ? ? do_something() > > profile.dump_stats(filename) If __enter__() returns self (as is recommended when you don't have anything more context specific to return), then the following would also work: with cProfile.Profile() as profile: do_something() profile.dump_stats(filename) Assuming the re-entrancy question can be answered adequately, this sounds like a reasonable idea to me. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From aquavitae69 at gmail.com Wed Jan 25 11:28:46 2012 From: aquavitae69 at gmail.com (David Townshend) Date: Wed, 25 Jan 2012 12:28:46 +0200 Subject: [Python-ideas] Allow Profile() to be used as a context manager In-Reply-To: References: Message-ID: Would that stop the profiler between calles though? In Yuval's example, wouldn't do_a_hard_but_i_dont_care_now_task be included in the profile then? On Wed, Jan 25, 2012 at 12:16 PM, Robert Kern wrote: > On 1/25/12 6:36 AM, David Townshend wrote: > >> I often find it useful to profile small sections to code in a running >> application to pinpoint a known bottleneck. The cProfile and profile >> modules >> don't make this easy as they stand, requiring the section of code to be >> wrapped >> into a single statement with can be called as a string using exec(). This >> always feels a bit clumsy to me, so my idea is simply to provide >> __enter__ and >> __exit__ methods to cProfile.Profile() and profile.Profile(). From a >> quick look >> at the code it seems that this should be very easy for cProfile, and >> slightly >> less easy for profile (I can't quite figure out how to deal with the >> set_cmd() >> line, or how important it is - maybe someone can help?). >> >> Any major objections to this? >> > > I've done this in my kernprof.py script, which I use as a convenient > generic profiling script. I subclass cProfile.Profile to add > enable_by_count()/disable_by_**count() methods that allow nesting. > enable_by_count() increments a counter and only calls the enable() method > the first time. disable_by_count() decrements the counter and only calls > disable() when it hits 0 again. __enter__() and __exit__() just call these > methods. I also add a __call__() method that lets a Profile instance act as > a decorator, which I actually find somewhat more useful than the context > manager. > > https://bitbucket.org/**robertkern/line_profiler/src/** > tip/kernprof.py#cl-57 > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma > that is made terrible by our own mad attempt to interpret it as though it > had > an underlying truth." > -- Umberto Eco > > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jan 25 11:52:45 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Jan 2012 20:52:45 +1000 Subject: [Python-ideas] Allow Profile() to be used as a context manager In-Reply-To: References: Message-ID: On Wed, Jan 25, 2012 at 8:28 PM, David Townshend wrote: > Would that stop the profiler between calles though? ?In Yuval's example, > wouldn't? do_a_hard_but_i_dont_care_now_task?be included in the profile > then? No, because the two profiling blocks aren't overlapping - the count drops to zero between them so the profiler will be disabled for that period. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From aquavitae69 at gmail.com Wed Jan 25 12:03:04 2012 From: aquavitae69 at gmail.com (David Townshend) Date: Wed, 25 Jan 2012 13:03:04 +0200 Subject: [Python-ideas] Allow Profile() to be used as a context manager In-Reply-To: References: Message-ID: Oh yes, of course. Silly me! On Wed, Jan 25, 2012 at 12:52 PM, Nick Coghlan wrote: > On Wed, Jan 25, 2012 at 8:28 PM, David Townshend > wrote: > > Would that stop the profiler between calles though? In Yuval's example, > > wouldn't do_a_hard_but_i_dont_care_now_task be included in the profile > > then? > > No, because the two profiling blocks aren't overlapping - the count > drops to zero between them so the profiler will be disabled for that > period. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Wed Jan 25 12:37:46 2012 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 25 Jan 2012 11:37:46 +0000 Subject: [Python-ideas] Allow Profile() to be used as a context manager In-Reply-To: References: Message-ID: On 1/25/12 8:19 AM, Yuval Greenfield wrote: > On Wed, Jan 25, 2012 at 8:36 AM, David Townshend > > wrote: > > [...] > Any major objections to this? > > > > At first this sounded like a good idea but 2 things pop to mind: > > 1. How does one get the pstat profiling output from the context manager? Will > printing out the results be the only option? > 2. Usually when I profile I don't want to touch the code at all so eventually > if I'll use this context manager it'll be more akin to > > > with profile.Profiler(): > main() > > Which isn't much of a win over > > profile.run('main()') > > > Now if you're talking about a reentrant profiler, that's interesting: > > profiler = profile.Profiler() > > while is_working: > with profiler: > do_hard_task() > do_a_hard_but_i_dont_care_now_task() > > with profiler: > do_a_diff_hard_task() > > > and then I can call whatever method is there to get the pstat. If this is the > vision then it sounds awesome. You may want to take a look at my kernprof script mentioned elsewhere in the thread: http://packages.python.org/line_profiler/ http://pypi.python.org/pypi/line_profiler/ Besides adding the context manager to Profile, it is a generic "runner" script that will run your script under the profiler and dump the results to a file. In the normal use case of profiling the entire script, you don't have to modify your code at all: [line_profiler]$ kernprof.py -v pystone.py Pystone(1.1) time for 50000 passes = 0.918603 This machine benchmarks at 54430.5 pystones/second Wrote profile results to pystone.py.prof 1100070 function calls in 0.929 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.929 0.929 :1() 50000 0.140 0.000 0.322 0.000 pystone.py:137(Proc1) 50000 0.029 0.000 0.029 0.000 pystone.py:153(Proc2) 50000 0.036 0.000 0.048 0.000 pystone.py:164(Proc3) 50000 0.015 0.000 0.015 0.000 pystone.py:174(Proc4) 50000 0.013 0.000 0.013 0.000 pystone.py:181(Proc5) 50000 0.033 0.000 0.043 0.000 pystone.py:188(Proc6) 150000 0.034 0.000 0.034 0.000 pystone.py:207(Proc7) 50000 0.100 0.000 0.126 0.000 pystone.py:212(Proc8) 150000 0.035 0.000 0.035 0.000 pystone.py:225(Func1) ... But if you do want to use the context manager or the decorator, then there is a flag to the script that injects the Profile instance into the builtins. Then you can use "@profile" or "with profile:" to only enable the profiler at certain points (and multiple points as in your use case above). These are the only modifications you need to make. The stats dumping is handled by kernprof. [line_profiler]$ grin -A10 '@profile' pystone.py pystone.py: 164 : @profile 165 + def Proc3(PtrParOut): 166 + global IntGlob 167 + 168 + if PtrGlb is not None: 169 + PtrParOut = PtrGlb.PtrComp 170 + else: 171 + IntGlob = 100 172 + PtrGlb.IntComp = Proc7(10, IntGlob) 173 + return PtrParOut 174 + 182 : @profile 183 + def Proc5(): 184 + global Char1Glob 185 + global BoolGlob 186 + 187 + Char1Glob = 'A' 188 + BoolGlob = FALSE 189 + 190 + def Proc6(EnumParIn): 191 + EnumParOut = EnumParIn 192 + if not Func3(EnumParIn): [line_profiler]$ kernprof.py -v --builtin pystone.py Pystone(1.1) time for 50000 passes = 1.08921 This machine benchmarks at 45904.8 pystones/second Wrote profile results to pystone.py.prof 350000 function calls in 0.192 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 100000 0.116 0.000 0.123 0.000 kernprof.py:73(disable_by_count) 50000 0.042 0.000 0.055 0.000 pystone.py:164(Proc3) 50000 0.013 0.000 0.013 0.000 pystone.py:182(Proc5) 50000 0.013 0.000 0.013 0.000 pystone.py:209(Proc7) 100000 0.007 0.000 0.007 0.000 {method 'disable' of '_lsprof.Profiler' objects} -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From aquavitae69 at gmail.com Wed Jan 25 13:18:03 2012 From: aquavitae69 at gmail.com (David Townshend) Date: Wed, 25 Jan 2012 14:18:03 +0200 Subject: [Python-ideas] Allow Profile() to be used as a context manager In-Reply-To: References: Message-ID: shameless-advertisement noted! It looks like a useful tool, but I had something rather simpler in mind, and something included in the stdlib. I think adding a decorator too is a good idea, but as pointed out there is already a patch for this. David On Wed, Jan 25, 2012 at 1:37 PM, Robert Kern wrote: > On 1/25/12 8:19 AM, Yuval Greenfield wrote: > >> On Wed, Jan 25, 2012 at 8:36 AM, David Townshend >> > **> wrote: >> >> [...] >> Any major objections to this? >> >> >> >> At first this sounded like a good idea but 2 things pop to mind: >> >> 1. How does one get the pstat profiling output from the context manager? >> Will >> >> printing out the results be the only option? >> 2. Usually when I profile I don't want to touch the code at all so >> eventually >> >> if I'll use this context manager it'll be more akin to >> >> >> with profile.Profiler(): >> main() >> >> Which isn't much of a win over >> >> profile.run('main()') >> >> >> Now if you're talking about a reentrant profiler, that's interesting: >> >> profiler = profile.Profiler() >> >> while is_working: >> with profiler: >> do_hard_task() >> do_a_hard_but_i_dont_care_now_**task() >> >> with profiler: >> do_a_diff_hard_task() >> >> >> and then I can call whatever method is there to get the pstat. If this is >> the >> vision then it sounds awesome. >> > > > > You may want to take a look at my kernprof script mentioned elsewhere in > the thread: > > http://packages.python.org/**line_profiler/ > http://pypi.python.org/pypi/**line_profiler/ > > Besides adding the context manager to Profile, it is a generic "runner" > script that will run your script under the profiler and dump the results to > a file. In the normal use case of profiling the entire script, you don't > have to modify your code at all: > > [line_profiler]$ kernprof.py -v pystone.py > Pystone(1.1) time for 50000 passes = 0.918603 > This machine benchmarks at 54430.5 pystones/second > Wrote profile results to pystone.py.prof > 1100070 function calls in 0.929 seconds > > Ordered by: standard name > > ncalls tottime percall cumtime percall filename:lineno(function) > 1 0.000 0.000 0.929 0.929 :1() > 50000 0.140 0.000 0.322 0.000 pystone.py:137(Proc1) > 50000 0.029 0.000 0.029 0.000 pystone.py:153(Proc2) > 50000 0.036 0.000 0.048 0.000 pystone.py:164(Proc3) > 50000 0.015 0.000 0.015 0.000 pystone.py:174(Proc4) > 50000 0.013 0.000 0.013 0.000 pystone.py:181(Proc5) > 50000 0.033 0.000 0.043 0.000 pystone.py:188(Proc6) > 150000 0.034 0.000 0.034 0.000 pystone.py:207(Proc7) > 50000 0.100 0.000 0.126 0.000 pystone.py:212(Proc8) > 150000 0.035 0.000 0.035 0.000 pystone.py:225(Func1) > ... > > But if you do want to use the context manager or the decorator, then there > is a flag to the script that injects the Profile instance into the > builtins. Then you can use "@profile" or "with profile:" to only enable the > profiler at certain points (and multiple points as in your use case above). > These are the only modifications you need to make. The stats dumping is > handled by kernprof. > > [line_profiler]$ grin -A10 '@profile' pystone.py > pystone.py: > 164 : @profile > 165 + def Proc3(PtrParOut): > 166 + global IntGlob > 167 + > 168 + if PtrGlb is not None: > 169 + PtrParOut = PtrGlb.PtrComp > 170 + else: > 171 + IntGlob = 100 > 172 + PtrGlb.IntComp = Proc7(10, IntGlob) > 173 + return PtrParOut > 174 + > 182 : @profile > 183 + def Proc5(): > 184 + global Char1Glob > 185 + global BoolGlob > 186 + > 187 + Char1Glob = 'A' > 188 + BoolGlob = FALSE > 189 + > 190 + def Proc6(EnumParIn): > 191 + EnumParOut = EnumParIn > 192 + if not Func3(EnumParIn): > > [line_profiler]$ kernprof.py -v --builtin pystone.py > Pystone(1.1) time for 50000 passes = 1.08921 > This machine benchmarks at 45904.8 pystones/second > Wrote profile results to pystone.py.prof > 350000 function calls in 0.192 seconds > > Ordered by: standard name > > ncalls tottime percall cumtime percall filename:lineno(function) > 100000 0.116 0.000 0.123 0.000 kernprof.py:73(disable_by_* > *count) > 50000 0.042 0.000 0.055 0.000 pystone.py:164(Proc3) > 50000 0.013 0.000 0.013 0.000 pystone.py:182(Proc5) > 50000 0.013 0.000 0.013 0.000 pystone.py:209(Proc7) > 100000 0.007 0.000 0.007 0.000 {method 'disable' of > '_lsprof.Profiler' objects} > > > > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma > that is made terrible by our own mad attempt to interpret it as though it > had > an underlying truth." > -- Umberto Eco > > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan.alexander.rice at gmail.com Wed Jan 25 15:39:01 2012 From: nathan.alexander.rice at gmail.com (Nathan Rice) Date: Wed, 25 Jan 2012 09:39:01 -0500 Subject: [Python-ideas] Coroutines and PEP 380 In-Reply-To: References: <4F15F041.6010607@hotpy.org> <20DB36E8-2538-4FE8-9FBF-6B3DA67E3CD6@twistedmatrix.com> <4F168FA5.2000503@hotpy.org> <7F3B6F9E-A901-4FA5-939E-CDD7B1E6E5B5@twistedmatrix.com> <4F188DFD.6080401@canterbury.ac.nz> <4F1B48D0.3060309@canterbury.ac.nz> Message-ID: On Wed, Jan 25, 2012 at 3:45 AM, Nick Coghlan wrote: > (redirecting to python-ideas - coroutine proposals are nowhere near > mature enough for python-dev) > > On Wed, Jan 25, 2012 at 5:35 PM, Matt Joiner wrote: >> If someone can explain what's stopping real coroutines being into >> Python (3.3), that would be great. > > The general issues with that kind of idea: > - the author hasn't offered the code for inclusion and relicensing > under the PSF license (thus we legally aren't allowed to do it) > - complexity > - maintainability > - platform support > > In the specific case of coroutines, you have the additional hurdle of > convincing people whether or not they're a good idea at all. I think coroutines can be immensely useful. I am using continulets in PyPy to implement the flow control for symbolic expression capture, and it is by far the least complex method I have examined. I can't think of any other examples where I've really felt blocked by a standard stack though. Nathan From solipsis at pitrou.net Wed Jan 25 15:45:24 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 25 Jan 2012 15:45:24 +0100 Subject: [Python-ideas] Coroutines and PEP 380 References: <4F15F041.6010607@hotpy.org> <20DB36E8-2538-4FE8-9FBF-6B3DA67E3CD6@twistedmatrix.com> <4F168FA5.2000503@hotpy.org> <7F3B6F9E-A901-4FA5-939E-CDD7B1E6E5B5@twistedmatrix.com> <4F188DFD.6080401@canterbury.ac.nz> <4F1B48D0.3060309@canterbury.ac.nz> Message-ID: <20120125154524.74010b64@pitrou.net> On Wed, 25 Jan 2012 18:45:16 +1000 Nick Coghlan wrote: > On Wed, Jan 25, 2012 at 5:35 PM, Matt Joiner wrote: > > If someone can explain what's stopping real coroutines being into > > Python (3.3), that would be great. > > The general issues with that kind of idea: > - the author hasn't offered the code for inclusion and relicensing > under the PSF license (thus we legally aren't allowed to do it) > - complexity > - maintainability > - platform support > > In the specific case of coroutines, you have the additional hurdle of > convincing people whether or not they're a good idea at all. Do you mean preemptive coroutines as opposed to ("yield from"-based) cooperative coroutines? Because I don't see the point of PEP 380 if not for coroutines (tree-walking examples are academic). (I'd also like to point out the following paper which makes an useful distinction between "task management" and "stack management": http://www.usenix.org/event/usenix02/adyahowell.html ) Regards Antoine. From jimjjewett at gmail.com Wed Jan 25 16:32:35 2012 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 25 Jan 2012 10:32:35 -0500 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Mon, Jan 23, 2012 at 10:18 PM, Stephen J. Turnbull wrote: > Jim Jewett writes: > ?> So it doesn't seem like a loop because you hope to do it only once? > With s/hope/expect/, that hits the nail on the head. > I don't think syntax can express that cleanly, but I can't help > thinking it's of good thing if somebody like Mike tries to find a way. > He might succeed! Ah... I have often wanted a clean way to indicate "This branch can happen, but isn't normal." (Well, besides a comment that might be seen as condescending if it is *obviously* an edge case.) The retry proposal is just specializing that for when the weird branch includes a loop. Right now, the best I can do is hope that the special case (and possibly the normal case, if it is repeated) can be factored out, so that I can write if not doit(args): if not doit(tweak1(args)): if not doit(tweak2(args)): raise ReallyCant(args) or if oddcase(args): handle_oddcase(args) else: # Alternatively, make this suite much longer, so that it is # "obviously" the main point of the function. return _real_function(args) That said, I've wanted unusual-case annotation more when I thought the compiler might use the information. Without compiler support, I'm not sure how much takeup there would be for the resulting documentation-only construct. -jJ From jimjjewett at gmail.com Wed Jan 25 16:42:31 2012 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 25 Jan 2012 10:42:31 -0500 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Jan 24, 2012 at 12:18 AM, Nick Coghlan wrote: > ... I've written for loops with an unconditional break at the end to > get "0 or 1" iteration. Ouch ... That seems even worse than the C do {...} while (0); idiom. Why didn't you just use an "if"? (Or at least a try ... except StopIteration) -jJ From jimjjewett at gmail.com Wed Jan 25 17:18:49 2012 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 25 Jan 2012 11:18:49 -0500 Subject: [Python-ideas] Retrying EAFP without DRY In-Reply-To: References: <20120120163648.69288796@mikmeyer-vm-fedora> <4F1A6DA2.5020807@pearwood.info> <20120122133014.5e920a43@bhuda.mired.org> <871uqr595n.fsf@uwakimon.sk.tsukuba.ac.jp> <20120124000053.GC6693@ando> <20120123170633.1bfdbf11@mikmeyer-vm-fedora> <87obtt4wlu.fsf@uwakimon.sk.tsukuba.ac.jp> <669fa0fa-beca-4250-90ad-3a073f0d559c@email.android.com> <8F64C8BB-6033-404E-944E-F58E9830C584@gmail.com> <4F1E93C7.4050703@pearwood.info> Message-ID: On Tue, Jan 24, 2012 at 1:01 PM, Chris Rebert wrote: >> On 24 Jan 2012, at 14:06, Nick Coghlan wrote: >>> On Tue, Jan 24, 2012 at 9:36 PM, Paul Moore wrote: >>>> [ sees value in a RetryStrategy construct of some sort] >>> You just need to move the pause inside the iterator: >>> >>> ? ?def backoff(attempts, first_delay, scale=2): >>> ? ? ? ?delay = first_delay >>> ? ? ? ?for attempt in range(1, attempts+1): >>> ? ? ? ? ? ?yield attempt >>> ? ? ? ? ? ?time.sleep(delay) >>> ? ? ? ? ? ?delay *= 2 >>> >>> ? ?for __ in backoff(MAX_ATTEMPTS, 5): >>> ? ? ? try: >>> ? ? ? ? ? response = urllib2.urlopen(url) >>> ? ? ? except urllib2.HTTPError as e: >>> ? ? ? ? ? if e.code == 503: ?# Service Unavailable. >>> ? ? ? ? ? ? ? continue >>> ? ? ? ? ? raise >>> ? ? ? break > On Tue, Jan 24, 2012 at 7:21 AM, Jakob Bowyer wrote: >> Would this not be better expressed as a context manager? >> >> with backoff(maxattempts, 5): >> ? ?# do stuff > It can't be. It really should be, though. What do to on a fixable or temporary failure is pretty clearly an execution context. > The `with` statement always executes its block exactly once; > the context manager(s) have no say in the matter (unless perhaps > you count raising an exception prior to the block's execution). with RetryStrategy() as keepgoing: while keepgoing(): ... The catch is that keepgoing() is usually either simple enough to inline (even without the with statement) or sufficiently complicated that it needs to modify something the rest of the suite sees. Passing an object just to hold these changes adds its own code smell. -jJ From p.f.moore at gmail.com Wed Jan 25 19:47:06 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 25 Jan 2012 18:47:06 +0000 Subject: [Python-ideas] Coroutines and PEP 380 In-Reply-To: <20120125154524.74010b64@pitrou.net> References: <4F15F041.6010607@hotpy.org> <20DB36E8-2538-4FE8-9FBF-6B3DA67E3CD6@twistedmatrix.com> <4F168FA5.2000503@hotpy.org> <7F3B6F9E-A901-4FA5-939E-CDD7B1E6E5B5@twistedmatrix.com> <4F188DFD.6080401@canterbury.ac.nz> <4F1B48D0.3060309@canterbury.ac.nz> <20120125154524.74010b64@pitrou.net> Message-ID: On 25 January 2012 14:45, Antoine Pitrou wrote: > Do you mean preemptive coroutines as opposed to ("yield from"-based) > cooperative coroutines? My understanding (in general, and specifically of the various discussions that have happened on python-ideas) is that we're talking about co-operative coroutines. To be clear, what I mean by that is that a coroutine switch is invoked by an explicit call/statement (which may be "yield from" or some as-yet unspecified variation - a cocall statement, a call to a switch() method on a coroutine object, or whatever). The discussions seem to be more about whether incrementally extending generators (with send methods, yield from, etc) to turn them into coroutines is better or worse than starting with a new approach which is closer to "conventional" approaches from other languages with first-class coroutines (for example Lua, which uses method calls on coroutine objects). I presume that "preemptive coroutines" means that a coroutine switch can occur at any point. I can't see how that is in any practical sense different from a thread... Paul. From anacrolix at gmail.com Thu Jan 26 03:08:41 2012 From: anacrolix at gmail.com (Matt Joiner) Date: Thu, 26 Jan 2012 13:08:41 +1100 Subject: [Python-ideas] Coroutines and PEP 380 In-Reply-To: References: <4F15F041.6010607@hotpy.org> <20DB36E8-2538-4FE8-9FBF-6B3DA67E3CD6@twistedmatrix.com> <4F168FA5.2000503@hotpy.org> <7F3B6F9E-A901-4FA5-939E-CDD7B1E6E5B5@twistedmatrix.com> <4F188DFD.6080401@canterbury.ac.nz> <4F1B48D0.3060309@canterbury.ac.nz> <20120125154524.74010b64@pitrou.net> Message-ID: On Thu, Jan 26, 2012 at 5:47 AM, Paul Moore wrote: > My understanding (in general, and specifically of the various > discussions that have happened on python-ideas) is that we're talking > about co-operative coroutines. > > To be clear, what I mean by that is that a coroutine switch is invoked > by an explicit call/statement (which may be "yield from" or some > as-yet unspecified variation - a cocall statement, a call to a > switch() method on a coroutine object, or whatever). The discussions > seem to be more about whether incrementally extending generators (with > send methods, yield from, etc) to turn them into coroutines is better > or worse than starting with a new approach which is closer to > "conventional" approaches from other languages with first-class > coroutines (for example Lua, which uses method calls on coroutine > objects). > > I presume that "preemptive coroutines" means that a coroutine switch > can occur at any point. I can't see how that is in any practical sense > different from a thread... I'm not sure that preemptive coroutines as you've described them are what people are aiming for. Without better terminology I've taken to calling them implicit and explicit coroutines: Implicit coroutines can arbitrary switch without changes in syntax, calling convention, or traversing back up the stack. Explicit coroutines must be "driven" (yield from), are called differently (yield from), and send values back up the stack before they can switch cleanly (yield from). Clearly all coroutines share several fantastic properties, they: * Are stored on the heap, * Are not preempted, * They can execute within a single native thread, * Store state without tying up a native stack for this purpose. Used for concurrency with an eventing scheduler in Python they give: * Massive concurrency with minimal system impact (try 20k threads instead of 20k coroutines), * Side-step the GIL, * Ridiculously cheap context switching, * Remove the need for callbacks and all that rubbish (I saw Guido was keen on this one), * Allow existing synchronous code to remain unchanged by modifying the underlying blocking calls (implicit coroutines only). From ubershmekel at gmail.com Thu Jan 26 08:53:02 2012 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Thu, 26 Jan 2012 09:53:02 +0200 Subject: [Python-ideas] Coroutines and PEP 380 In-Reply-To: References: <4F15F041.6010607@hotpy.org> <20DB36E8-2538-4FE8-9FBF-6B3DA67E3CD6@twistedmatrix.com> <4F168FA5.2000503@hotpy.org> <7F3B6F9E-A901-4FA5-939E-CDD7B1E6E5B5@twistedmatrix.com> <4F188DFD.6080401@canterbury.ac.nz> <4F1B48D0.3060309@canterbury.ac.nz> <20120125154524.74010b64@pitrou.net> Message-ID: On Thu, Jan 26, 2012 at 4:08 AM, Matt Joiner wrote: > Used for concurrency with an eventing scheduler in Python they give: > > * Massive concurrency with minimal system impact (try 20k threads > instead of 20k coroutines), > * Side-step the GIL, > > How would the eventing scheduler and coroutines side-step the GIL? Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Jan 26 09:02:21 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 26 Jan 2012 08:02:21 +0000 Subject: [Python-ideas] Coroutines and PEP 380 In-Reply-To: References: <4F15F041.6010607@hotpy.org> <20DB36E8-2538-4FE8-9FBF-6B3DA67E3CD6@twistedmatrix.com> <4F168FA5.2000503@hotpy.org> <7F3B6F9E-A901-4FA5-939E-CDD7B1E6E5B5@twistedmatrix.com> <4F188DFD.6080401@canterbury.ac.nz> <4F1B48D0.3060309@canterbury.ac.nz> <20120125154524.74010b64@pitrou.net> Message-ID: On 26 January 2012 02:08, Matt Joiner wrote: > Implicit coroutines can arbitrary switch without changes in syntax, > calling convention, or traversing back up the stack. [...] > * Are not preempted, That's what confuses me. If an implicit coroutine can arbitrarily switch without changes in syntax, etc, how is it not pre-empted? How does the switch happen, if not by pre-emption or by some explicit request/syntax? An example of what you mean would probably help. Paul From simon.sapin at kozea.fr Thu Jan 26 09:19:29 2012 From: simon.sapin at kozea.fr (Simon Sapin) Date: Thu, 26 Jan 2012 09:19:29 +0100 Subject: [Python-ideas] Coroutines and PEP 380 In-Reply-To: References: <4F15F041.6010607@hotpy.org> <20DB36E8-2538-4FE8-9FBF-6B3DA67E3CD6@twistedmatrix.com> <4F168FA5.2000503@hotpy.org> <7F3B6F9E-A901-4FA5-939E-CDD7B1E6E5B5@twistedmatrix.com> <4F188DFD.6080401@canterbury.ac.nz> <4F1B48D0.3060309@canterbury.ac.nz> <20120125154524.74010b64@pitrou.net> Message-ID: <4F210C91.303@kozea.fr> Le 26/01/2012 03:08, Matt Joiner a ?crit : > Implicit coroutines can arbitrary switch without changes in syntax, > calling convention, or traversing back up the stack. Do you mean that greenlets? some_greenlet.switch() looks like a method call and does not required a special syntax like "yield from" so it can be hidden in some_file.read() like gevent does. Regards, -- Simon Sapin From techtonik at gmail.com Thu Jan 26 15:18:55 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 26 Jan 2012 16:18:55 +0200 Subject: [Python-ideas] Module field in tracker Message-ID: Is it a good idea to have a list of modules in tracker? So that you can have a summary of all bugs per component similar to Trac's http://trac.edgewall.org/milestone/0.12.3 -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Thu Jan 26 17:04:40 2012 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Thu, 26 Jan 2012 17:04:40 +0100 Subject: [Python-ideas] Module field in tracker In-Reply-To: References: Message-ID: Il 26 gennaio 2012 15:18, anatoly techtonik ha scritto: > Is it a good idea to have a list of modules in tracker? > So that you can have a summary of all bugs per component similar to Trac's > http://trac.edgewall.org/milestone/0.12.3 > -- > anatoly t. +1. I think there's already an issue opened on the bug tracker's bug tracker for this. :-) --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/ From techtonik at gmail.com Thu Jan 26 18:25:40 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 26 Jan 2012 19:25:40 +0200 Subject: [Python-ideas] Dict-like object with property access Message-ID: I expected to find the answer to this question in FAQ, but because there is no FAQ I ask it anyway. How about adding a new standard dict-like container type that allows access using . (dot) to its members instead of ['index']? Why? It is convenient to write options.help instead of options['halp'] etc. Example: >>> mydict = container(someprop=somevalue) >>> mydict['someprop'] somevalue >>> mydict.someprop somevalue >>> mydict.otherprop Exception KeyError ... I know that it is easy to implement, but wouldn't it be nice to make it available by default? A side benefit of having this in stdlib is that newbies will be aware of the behaviour of derived classes without having to understand the mechanics of magic methods. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mwm at mired.org Thu Jan 26 18:38:15 2012 From: mwm at mired.org (Mike Meyer) Date: Thu, 26 Jan 2012 09:38:15 -0800 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: Message-ID: <20120126093815.54f8d5b9@mikmeyer-vm-fedora> On Thu, 26 Jan 2012 19:25:40 +0200 anatoly techtonik wrote: > I expected to find the answer to this question in FAQ, but because > there is no FAQ I ask it anyway. Better to have searched the python-ideas mail list archive. > How about adding a new standard dict-like container type that allows > access using . (dot) to its members instead of ['index']? > Why? It is convenient to write options.help instead of > options['halp'] etc. Because it doesn't work in general. There are strings that can be used as an index, but not as an attribute. There are existing attributes that you have to avoid, etc. The only way this really works in practice is if you start with a fixed set of names you want to access, in which case collections.namedtuple will probably do the job. References: Message-ID: On Thu, Jan 26, 2012 at 9:25 AM, anatoly techtonik wrote: > I expected to find the answer to this question in FAQ, but because there is > no FAQ I ask it anyway. > > How about adding a new standard dict-like container type that allows access > using . (dot) to its members instead of ['index']? > Why? It is convenient to write options.help instead of options['halp'] etc. > > Example: >>>> mydict = container(someprop=somevalue) >>>> mydict['someprop'] > somevalue >>>> mydict.someprop > somevalue >>>> mydict.otherprop > Exception KeyError ... > > I know that it is easy to implement, but wouldn't it be nice to make it > available by default? > A side benefit of having this in stdlib is that newbies will be aware of the > behaviour of derived classes without having to understand the mechanics of > magic methods. That is pretty much JavaScript's 'object', and I hate this ambiguity. If your keys are always constants, define a proper options class so you can say options.help instead of options['help']. You can also write a generic subclass of dict that works this way, if you really think you like it so much. But please keep it out of the stdlib. It leads to confused users, not happy users. An example of the problems that arise: If d['a'] == d.a, then how come d['clear'] != d.clear ? -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Thu Jan 26 22:28:06 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 26 Jan 2012 16:28:06 -0500 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <20120126093815.54f8d5b9@mikmeyer-vm-fedora> References: <20120126093815.54f8d5b9@mikmeyer-vm-fedora> Message-ID: On 1/26/2012 12:38 PM, Mike Meyer wrote: > On Thu, 26 Jan 2012 19:25:40 +0200 > anatoly techtonik wrote: >> How about adding a new standard dict-like container type that allows >> access using . (dot) to its members instead of ['index']? We already have them: any subclass of *object*. >>> class Option: pass >>> options = Option >>> options.help = 'be happy' >>> options.help 'be happy' Or did you mean 'instead of' to mean 'in addition to'? That is actually possible too. >>> options.__dict__['help'] 'be happy' >>> options_d = options.__dict__ # dict view of options >>> options.bye = 'Nice to know you!' >>> options_d['bye'] 'Nice to know you!' >> Why? It is convenient to write options.help instead of >> options['halp'] etc. > > Because it doesn't work in general. There are strings that can be used > as an index, but not as an attribute. There are existing attributes > that you have to avoid, etc. > > The only way this really works in practice is if you start with a > fixed set of names you want to access, in which case > collections.namedtuple will probably do the job. An advantage of collections.namedtuple is that all the pre-existing attributes start with '_' so as to avoid name clashes. Another is that name subscripts do *not* work, so there is no ambiguity there either. -- Terry Jan Reedy From techtonik at gmail.com Mon Jan 30 15:43:06 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 30 Jan 2012 17:43:06 +0300 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: Message-ID: On Thu, Jan 26, 2012 at 8:47 PM, Guido van Rossum wrote: > On Thu, Jan 26, 2012 at 9:25 AM, anatoly techtonik > wrote: > > I expected to find the answer to this question in FAQ, but because there > is > > no FAQ I ask it anyway. > > > > How about adding a new standard dict-like container type that allows > access > > using . (dot) to its members instead of ['index']? > > Why? It is convenient to write options.help instead of options['halp'] > etc. > > > > Example: > >>>> mydict = container(someprop=somevalue) > >>>> mydict['someprop'] > > somevalue > >>>> mydict.someprop > > somevalue > >>>> mydict.otherprop > > Exception KeyError ... > > > > I know that it is easy to implement, but wouldn't it be nice to make it > > available by default? > > A side benefit of having this in stdlib is that newbies will be aware of > the > > behaviour of derived classes without having to understand the mechanics > of > > magic methods. > > That is pretty much JavaScript's 'object', and I hate this ambiguity. > If your keys are always constants, define a proper options class so > you can say options.help instead of options['help']. You can also > write a generic subclass of dict that works this way, if you really > think you like it so much. But please keep it out of the stdlib. It > leads to confused users, not happy users. An example of the problems > that arise: If d['a'] == d.a, then how come d['clear'] != d.clear ? In which case d['clear'] != d.clear can be true? I've found a MIT licensed library that implements just that: http://pypi.python.org/pypi/bunch/ Q. Why it is better than subclass of object? A. Because it is implicit. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From simon.sapin at kozea.fr Mon Jan 30 15:48:15 2012 From: simon.sapin at kozea.fr (Simon Sapin) Date: Mon, 30 Jan 2012 15:48:15 +0100 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: Message-ID: <4F26ADAF.7010907@kozea.fr> Le 30/01/2012 15:43, anatoly techtonik a ?crit : > In which case d['clear'] != d.clear can be true? dict.clear is a method: >>> import bunch >>> b = bunch.Bunch() >>> b['clear'] = 4 >>> b['clear'] 4 >>> b.clear >>> b.clear.__doc__ 'D.clear() -> None. Remove all items from D.' Regards, -- Simon Sapin From massimo.dipierro at gmail.com Mon Jan 30 15:58:46 2012 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Mon, 30 Jan 2012 08:58:46 -0600 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: Message-ID: In web2py we have a class called Storage. (web.py has a similar class too). works exactly like this except that if you do mydict.someprop and someprop does not exist returns None (which plays the role of JS undefined) instead of raining an exception. Users like this a lot because they can do: a = mydict.somevalue or 'somedefault' which new users find more readable than a = mydict.get('somevalue','somedefault') mydict.__getattr__ is the single most called method in web2py and it does affect performance. It it were supported natively by the language we would benefit from it. Massimo On Jan 26, 2012, at 11:47 AM, Guido van Rossum wrote: > On Thu, Jan 26, 2012 at 9:25 AM, anatoly techtonik wrote: >> I expected to find the answer to this question in FAQ, but because there is >> no FAQ I ask it anyway. >> >> How about adding a new standard dict-like container type that allows access >> using . (dot) to its members instead of ['index']? >> Why? It is convenient to write options.help instead of options['halp'] etc. >> >> Example: >>>>> mydict = container(someprop=somevalue) >>>>> mydict['someprop'] >> somevalue >>>>> mydict.someprop >> somevalue >>>>> mydict.otherprop >> Exception KeyError ... >> >> I know that it is easy to implement, but wouldn't it be nice to make it >> available by default? >> A side benefit of having this in stdlib is that newbies will be aware of the >> behaviour of derived classes without having to understand the mechanics of >> magic methods. > > That is pretty much JavaScript's 'object', and I hate this ambiguity. > If your keys are always constants, define a proper options class so > you can say options.help instead of options['help']. You can also > write a generic subclass of dict that works this way, if you really > think you like it so much. But please keep it out of the stdlib. It > leads to confused users, not happy users. An example of the problems > that arise: If d['a'] == d.a, then how come d['clear'] != d.clear ? > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From solipsis at pitrou.net Mon Jan 30 16:06:56 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 30 Jan 2012 16:06:56 +0100 Subject: [Python-ideas] Dict-like object with property access References: Message-ID: <20120130160656.204dd1d1@pitrou.net> On Mon, 30 Jan 2012 08:58:46 -0600 Massimo Di Pierro wrote: > In web2py we have a class called Storage. (web.py has a similar class too). > > works exactly like this except that if you do mydict.someprop and someprop does not exist returns None (which plays the role of JS undefined) instead of raining an exception. Users like this a lot because they can do: > > a = mydict.somevalue or 'somedefault' > > which new users find more readable than > > a = mydict.get('somevalue','somedefault') > > mydict.__getattr__ is the single most called method in web2py and it does affect performance. It it were supported natively by the language we would benefit from it. The easy and Pythonic way to benefit from native performance is to use a dict instead... Regards Antoine. From masklinn at masklinn.net Mon Jan 30 17:01:31 2012 From: masklinn at masklinn.net (Masklinn) Date: Mon, 30 Jan 2012 17:01:31 +0100 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: Message-ID: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> On 2012-01-30, at 15:58 , Massimo Di Pierro wrote: > In web2py we have a class called Storage. (web.py has a similar class too). > > works exactly like this except that if you do mydict.someprop and someprop does not exist returns None (which plays the role of JS undefined) instead of raining an exception. Users like this a lot because they can do: > > a = mydict.somevalue or 'somedefault' > > which new users find more readable than > > a = mydict.get('somevalue','somedefault') And with completely different semantics, as a falsy value (empty string, 0, empty collection, None, etc?) will yield the default for the first case, but not for the second case. From massimo.dipierro at gmail.com Mon Jan 30 17:22:26 2012 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Mon, 30 Jan 2012 10:22:26 -0600 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> Message-ID: I do not think the issue is whether the people who use that semantic understand it or not. I can assure you they do and they know when it is appropriate to use it or not. The issue is whether there is any value is making it faster by including it in python or not. Because of the increasing popularity of JS I think new users are starting to expect something like it out of the box. Anyway, Guido's object about the "clear" method (and other methods) is the most serious problem. One way around could be preventing keys which conflicts with the method names. Massimo On Jan 30, 2012, at 10:01 AM, Masklinn wrote: > On 2012-01-30, at 15:58 , Massimo Di Pierro wrote: >> In web2py we have a class called Storage. (web.py has a similar class too). >> >> works exactly like this except that if you do mydict.someprop and someprop does not exist returns None (which plays the role of JS undefined) instead of raining an exception. Users like this a lot because they can do: >> >> a = mydict.somevalue or 'somedefault' >> >> which new users find more readable than >> >> a = mydict.get('somevalue','somedefault') > > And with completely different semantics, as a falsy value (empty string, 0, empty collection, None, etc?) will yield the default for the first case, but not for the second case. From arnodel at gmail.com Mon Jan 30 18:14:11 2012 From: arnodel at gmail.com (Arnaud Delobelle) Date: Mon, 30 Jan 2012 17:14:11 +0000 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> Message-ID: On Jan 30, 2012 4:23 PM, "Massimo Di Pierro" wrote: > > I do not think the issue is whether the people who use that semantic understand it or not. I can assure you they do and they know when it is appropriate to use it or not. The issue is whether there is any value is making it faster by including it in python or not. Because of the increasing popularity of JS I think new users are starting to expect something like it out of the box. But this design decision in JavaScript is at the heart of many problems (e.g. simply looping over keys is a pain). That it is widely used doesn't make it desirable. My experience with JavaScript is that we should keep this 'feature' out of Python. If people want it they can implement it very easily but encouraging them would be wrong. -- Arnaud -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Mon Jan 30 18:28:01 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 30 Jan 2012 10:28:01 -0700 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> Message-ID: On Mon, Jan 30, 2012 at 10:14 AM, Arnaud Delobelle wrote: > > On Jan 30, 2012 4:23 PM, "Massimo Di Pierro" > wrote: >> >> I do not think the issue is whether the people who use that semantic >> understand it or not. I can assure you they do and they know when it is >> appropriate to use it or not. The issue is whether there is any value is >> making it faster by including it in python or not. Because of the increasing >> popularity of JS I think new users are starting to expect something like it >> out of the box. > > But this design decision in JavaScript is at the heart of many problems > (e.g. simply looping over keys is a pain).? That it is widely used doesn't > make it desirable. My experience with JavaScript is that we should keep this > 'feature' out of Python. If people want it they can implement it very easily > but encouraging them would be wrong. +1 From aquavitae69 at gmail.com Mon Jan 30 18:50:11 2012 From: aquavitae69 at gmail.com (David Townshend) Date: Mon, 30 Jan 2012 19:50:11 +0200 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> Message-ID: I actually implemented this while ago in a dictionary I created. And a few weeks ago I went through all my code and removed it because it just wasn't worth the problems it gave. It's really just sugar for something that's pretty neat already. On Jan 30, 2012 7:28 PM, "Eric Snow" wrote: > On Mon, Jan 30, 2012 at 10:14 AM, Arnaud Delobelle > wrote: > > > > On Jan 30, 2012 4:23 PM, "Massimo Di Pierro" > > > wrote: > >> > >> I do not think the issue is whether the people who use that semantic > >> understand it or not. I can assure you they do and they know when it is > >> appropriate to use it or not. The issue is whether there is any value is > >> making it faster by including it in python or not. Because of the > increasing > >> popularity of JS I think new users are starting to expect something > like it > >> out of the box. > > > > But this design decision in JavaScript is at the heart of many problems > > (e.g. simply looping over keys is a pain). That it is widely used > doesn't > > make it desirable. My experience with JavaScript is that we should keep > this > > 'feature' out of Python. If people want it they can implement it very > easily > > but encouraging them would be wrong. > > +1 > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From massimo.dipierro at gmail.com Mon Jan 30 19:49:54 2012 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Mon, 30 Jan 2012 12:49:54 -0600 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> Message-ID: <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> Trying to make sure I understand where we disagree and perhaps explain my problem better. For me this has very little to do with dictionaries. STEP 1) Today we can do this: class Dummy(object): pass d = Dummy() d.something = 5 print d.something Is anybody calling this un-pythonic? STEP 2) We can now overload the [] to make the dynamic attributes accessible in an alternative syntax: class Dummy(object): def __getitem__(self,key): return getattr(self,key) def __setitem__(self,key,value): return setattr(self,key,value) d = Dummy() d.something = 5 d['something'] = 5 print d.something print d['something'] STEP 3) Is anybody calling this un-pythonic? We can add redefine __getattr__ so that it never raises an exception: STEP 4) class Dummy(object): def __getitem__(self,key): return getattr(self,key) def __setitem__(self,key,value): return setattr(self,key,value) def __getattr__(self,key): return object.__getattr__(self,key) if hasattr(self,key) else Dummy() Is this un-pythonic? I do not think so but I do have a problem with it: class Dummy(object): def __getitem__(self,key): return getattr(self,key) def __setitem__(self,key,value): return setattr(self,key,value) def __getattr__(self,key): print 'wtf' return object.__getattr__(self,key) if hasattr(self,key) else None >>> d=Dummy() >>> print d.somethingelse wtf ... 334 wtf times with python2.7, 999 times with python2.5... wtf None whatever this does internally, it makes some programs slower then I would like them to be. Why is it calling itself 334 times? STEP 5) We can add methods to make this object behave like a dictionary by redefining d.keys() in terms of d.__dict__.keys() etc. STEP 6) we can re-factor it a bit so that actually class Dummy is derived from dict. Is this the part that people do not like? I would be happy if Python provided an efficient way to do something like STEP 4 without the problem I mentioned. Perhaps there is one and I ignore it. I do not necessarily require STEP5 and STEP6. Use case: settings = Dummy() settings.parameter1 = 'a' settings.parameter2 = 'b' etc. if not settings.parameter1: do something ... Massimo On Jan 30, 2012, at 11:28 AM, Eric Snow wrote: > On Mon, Jan 30, 2012 at 10:14 AM, Arnaud Delobelle wrote: >> >> On Jan 30, 2012 4:23 PM, "Massimo Di Pierro" >> wrote: >>> >>> I do not think the issue is whether the people who use that semantic >>> understand it or not. I can assure you they do and they know when it is >>> appropriate to use it or not. The issue is whether there is any value is >>> making it faster by including it in python or not. Because of the increasing >>> popularity of JS I think new users are starting to expect something like it >>> out of the box. >> >> But this design decision in JavaScript is at the heart of many problems >> (e.g. simply looping over keys is a pain). That it is widely used doesn't >> make it desirable. My experience with JavaScript is that we should keep this >> 'feature' out of Python. If people want it they can implement it very easily >> but encouraging them would be wrong. > > +1 From ckaynor at zindagigames.com Mon Jan 30 20:00:30 2012 From: ckaynor at zindagigames.com (Chris Kaynor) Date: Mon, 30 Jan 2012 11:00:30 -0800 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> Message-ID: On Mon, Jan 30, 2012 at 10:49 AM, Massimo Di Pierro < massimo.dipierro at gmail.com> wrote: > STEP 4) > class Dummy(object): > def __getitem__(self,key): return getattr(self,key) > def __setitem__(self,key,value): return setattr(self,key,value) > def __getattr__(self,key): > return object.__getattr__(self,key) if hasattr(self,key) > else Dummy() > > Is this un-pythonic? > > I do not think so but I do have a problem with it: > > class Dummy(object): > def __getitem__(self,key): return getattr(self,key) > def __setitem__(self,key,value): return setattr(self,key,value) > def __getattr__(self,key): > print 'wtf' > return object.__getattr__(self,key) if hasattr(self,key) else > None > > >>> d=Dummy() > >>> print d.somethingelse > wtf > ... 334 wtf times with python2.7, 999 times with python2.5... > wtf > None > getattr calls hasattr, which calls getattr again. You are hitting the recursion limit before failing the test, due to hasattr failing with a recursion error, thus returning false. You could refactor this to use exceptions, such as: def __getattr__(self, key): try: return object.__getattr__(self, key) except Exception: return None > > whatever this does internally, it makes some programs slower then I would > like them to be. Why is it calling itself 334 times? > > STEP 5) > We can add methods to make this object behave like a dictionary by > redefining d.keys() in terms of d.__dict__.keys() etc. > > STEP 6) > we can re-factor it a bit so that actually class Dummy is derived from > dict. > > Is this the part that people do not like? > The general discontent with the idea is step 4. Doing so leads to abnormalities, such as d['clear'] != d.clear. The only ways to resolve such abnormalities are to make using any keys/attributes which conflict illegal, or to use different access methods for the two forms. > > > I would be happy if Python provided an efficient way to do something like > STEP 4 without the problem I mentioned. > Perhaps there is one and I ignore it. I do not necessarily require STEP5 > and STEP6. > > Use case: > > settings = Dummy() > settings.parameter1 = 'a' > settings.parameter2 = 'b' > etc. > > if not settings.parameter1: do something ... > > Massimo -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Jan 30 20:02:26 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 30 Jan 2012 20:02:26 +0100 Subject: [Python-ideas] Dict-like object with property access References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> Message-ID: <20120130200226.0a5ab1d8@pitrou.net> On Mon, 30 Jan 2012 12:49:54 -0600 Massimo Di Pierro wrote: > Trying to make sure I understand where we disagree and perhaps explain my problem better. For me this has very little to do with dictionaries. > > STEP 1) Today we can do this: > > class Dummy(object): pass > > d = Dummy() > d.something = 5 > print d.something > > Is anybody calling this un-pythonic? Depends what you're doing with it, but having a custom class which serves as nothing but a plain container is quite contrived in my opinion. > STEP 2) We can now overload the [] to make the dynamic attributes accessible in an alternative syntax: > > class Dummy(object): > def __getitem__(self,key): return getattr(self,key) > def __setitem__(self,key,value): return setattr(self,key,value) > d = Dummy() > d.something = 5 > d['something'] = 5 > print d.something > print d['something'] > > STEP 3) Is anybody calling this un-pythonic? Yes. You don't need both kinds of accesses. Regards Antoine. From massimo.dipierro at gmail.com Mon Jan 30 20:05:26 2012 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Mon, 30 Jan 2012 13:05:26 -0600 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> Message-ID: Let me add that one can do this: class Dummy(object): def __getitem__(self,key): return getattr(self,key) def __setitem__(self,key,value): return setattr(self,key,value) def __getattr__(self,key): return self.__dict__.get(key,None) d=Dummy() d.something = 5 print d.something print d.somethingelse which is more or less efficient and accomplished 4. Not as fast as a dict anyway. On Jan 30, 2012, at 12:49 PM, Massimo Di Pierro wrote: > Trying to make sure I understand where we disagree and perhaps explain my problem better. For me this has very little to do with dictionaries. > > STEP 1) Today we can do this: > > class Dummy(object): pass > > d = Dummy() > d.something = 5 > print d.something > > Is anybody calling this un-pythonic? > > STEP 2) We can now overload the [] to make the dynamic attributes accessible in an alternative syntax: > > class Dummy(object): > def __getitem__(self,key): return getattr(self,key) > def __setitem__(self,key,value): return setattr(self,key,value) > d = Dummy() > d.something = 5 > d['something'] = 5 > print d.something > print d['something'] > > STEP 3) Is anybody calling this un-pythonic? > > We can add redefine __getattr__ so that it never raises an exception: > > STEP 4) > class Dummy(object): > def __getitem__(self,key): return getattr(self,key) > def __setitem__(self,key,value): return setattr(self,key,value) > def __getattr__(self,key): > return object.__getattr__(self,key) if hasattr(self,key) else Dummy() > > Is this un-pythonic? > > I do not think so but I do have a problem with it: > > class Dummy(object): > def __getitem__(self,key): return getattr(self,key) > def __setitem__(self,key,value): return setattr(self,key,value) > def __getattr__(self,key): > print 'wtf' > return object.__getattr__(self,key) if hasattr(self,key) else None > >>>> d=Dummy() >>>> print d.somethingelse > wtf > ... 334 wtf times with python2.7, 999 times with python2.5... > wtf > None > > whatever this does internally, it makes some programs slower then I would like them to be. Why is it calling itself 334 times? > > STEP 5) > We can add methods to make this object behave like a dictionary by redefining d.keys() in terms of d.__dict__.keys() etc. > > STEP 6) > we can re-factor it a bit so that actually class Dummy is derived from dict. > > Is this the part that people do not like? > > > I would be happy if Python provided an efficient way to do something like STEP 4 without the problem I mentioned. > Perhaps there is one and I ignore it. I do not necessarily require STEP5 and STEP6. > > Use case: > > settings = Dummy() > settings.parameter1 = 'a' > settings.parameter2 = 'b' > etc. > > if not settings.parameter1: do something ... > > Massimo > > > > On Jan 30, 2012, at 11:28 AM, Eric Snow wrote: > >> On Mon, Jan 30, 2012 at 10:14 AM, Arnaud Delobelle wrote: >>> >>> On Jan 30, 2012 4:23 PM, "Massimo Di Pierro" >>> wrote: >>>> >>>> I do not think the issue is whether the people who use that semantic >>>> understand it or not. I can assure you they do and they know when it is >>>> appropriate to use it or not. The issue is whether there is any value is >>>> making it faster by including it in python or not. Because of the increasing >>>> popularity of JS I think new users are starting to expect something like it >>>> out of the box. >>> >>> But this design decision in JavaScript is at the heart of many problems >>> (e.g. simply looping over keys is a pain). That it is widely used doesn't >>> make it desirable. My experience with JavaScript is that we should keep this >>> 'feature' out of Python. If people want it they can implement it very easily >>> but encouraging them would be wrong. >> >> +1 > From phd at phdru.name Mon Jan 30 20:11:36 2012 From: phd at phdru.name (Oleg Broytman) Date: Mon, 30 Jan 2012 23:11:36 +0400 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> Message-ID: <20120130191136.GA1555@iskra.aviel.ru> On Mon, Jan 30, 2012 at 12:49:54PM -0600, Massimo Di Pierro wrote: > STEP 2) We can now overload the [] to make the dynamic attributes accessible in an alternative syntax: > > class Dummy(object): > def __getitem__(self,key): return getattr(self,key) > def __setitem__(self,key,value): return setattr(self,key,value) > d = Dummy() > d.something = 5 > d['something'] = 5 > print d.something > print d['something'] > > STEP 3) Is anybody calling this un-pythonic? I do. The object has two different interfaces for the same values and that's IMO unpythonic. The Zen says "There should be one-- and preferably only one --obvious way to do it." Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ethan at stoneleaf.us Mon Jan 30 20:23:35 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 30 Jan 2012 11:23:35 -0800 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <20120130200226.0a5ab1d8@pitrou.net> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <20120130200226.0a5ab1d8@pitrou.net> Message-ID: <4F26EE37.2040104@stoneleaf.us> Antoine Pitrou wrote: > On Mon, 30 Jan 2012 12:49:54 -0600 > Massimo Di Pierro > wrote: >> Trying to make sure I understand where we disagree and perhaps explain my problem better. For me this has very little to do with dictionaries. > >> STEP 2) We can now overload the [] to make the dynamic attributes accessible in an alternative syntax: >> >> class Dummy(object): >> def __getitem__(self,key): return getattr(self,key) >> def __setitem__(self,key,value): return setattr(self,key,value) >> d = Dummy() >> d.something = 5 >> d['something'] = 5 >> print d.something >> print d['something'] >> >> STEP 3) Is anybody calling this un-pythonic? > > Yes. You don't need both kinds of accesses. Sure you do -- as soon as 'something' can be passed in via a variable: def some_func(name): print(d.name) # uh, no print(d[name]) # okay, this works And when you are working with known objects (not passed in names): def some_other_func(): flam(d.something) sniggle(d.somethingelse) Okay, *need* might be too strong, as you could get by with [] access -- but . access is so much nicer when possible (saves three characters, which can make a difference for those of us with wimpy wrists!). ~Ethan~ From masklinn at masklinn.net Mon Jan 30 20:39:08 2012 From: masklinn at masklinn.net (Masklinn) Date: Mon, 30 Jan 2012 20:39:08 +0100 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <4F26EE37.2040104@stoneleaf.us> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <20120130200226.0a5ab1d8@pitrou.net> <4F26EE37.2040104@stoneleaf.us> Message-ID: On 2012-01-30, at 20:23 , Ethan Furman wrote: > Sure you do -- as soon as 'something' can be passed in via a variable: That's what getattr is for, isn't it? The issue I have is that, to me, string keys say "arbitrary" and attributes say "enumerated set". Using one for the other? Wreaks that information. I like that Python (and most languages) makes the distinction, and I don't think Javascript and Lua should be emulated on this point. From massimo.dipierro at gmail.com Mon Jan 30 20:49:01 2012 From: massimo.dipierro at gmail.com (Massimo Di Pierro) Date: Mon, 30 Jan 2012 13:49:01 -0600 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <20120130200226.0a5ab1d8@pitrou.net> <4F26EE37.2040104@stoneleaf.us> Message-ID: <4EC01939-F741-4192-9781-E39387AFA11F@gmail.com> On Jan 30, 2012, at 1:39 PM, Masklinn wrote: > On 2012-01-30, at 20:23 , Ethan Furman wrote: >> Sure you do -- as soon as 'something' can be passed in via a variable: > That's what getattr is for, isn't it? > > The issue I have is that, to me, string keys say "arbitrary" and attributes say "enumerated set". I do not understand the distinciton. In fact getattr(x,...) functionally just delegates to getitem(x.__dict__, ...). I do not see why this delegation should not be achieved using the same operator __getitem__. > Using one for the other? Wreaks that information. I like that Python (and most languages) makes the distinction, and I don't think Javascript and Lua should be emulated on this point. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From ethan at stoneleaf.us Mon Jan 30 20:25:12 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 30 Jan 2012 11:25:12 -0800 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <20120130191136.GA1555@iskra.aviel.ru> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <20120130191136.GA1555@iskra.aviel.ru> Message-ID: <4F26EE98.90001@stoneleaf.us> Oleg Broytman wrote: > On Mon, Jan 30, 2012 at 12:49:54PM -0600, Massimo Di Pierro wrote: >> STEP 2) We can now overload the [] to make the dynamic attributes accessible in an alternative syntax: >> >> class Dummy(object): >> def __getitem__(self,key): return getattr(self,key) >> def __setitem__(self,key,value): return setattr(self,key,value) >> d = Dummy() >> d.something = 5 >> d['something'] = 5 >> print d.something >> print d['something'] >> >> STEP 3) Is anybody calling this un-pythonic? > > I do. The object has two different interfaces for the same values and > that's IMO unpythonic. The Zen says > > "There should be one-- and preferably only one --obvious way to do it." Indeed. When you have the attribute name stored in a variable, then the one obvious way is [] access. When you already know the attribute name, the one obvious way is . access. ~Ethan~ From arnodel at gmail.com Mon Jan 30 20:56:41 2012 From: arnodel at gmail.com (Arnaud Delobelle) Date: Mon, 30 Jan 2012 19:56:41 +0000 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <4F26EE98.90001@stoneleaf.us> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <20120130191136.GA1555@iskra.aviel.ru> <4F26EE98.90001@stoneleaf.us> Message-ID: On 30 January 2012 19:25, Ethan Furman wrote: >> ? I do. The object has two different interfaces for the same values and >> that's IMO unpythonic. The Zen says >> >> ? "There should be one-- and preferably only one --obvious way to do it." > > > Indeed. > > When you have the attribute name stored in a variable, then the one obvious > way is [] access. If it's an attribute, the obvious way is getattr(obj, attrname) > When you already know the attribute name, the one obvious way is . access. Yes. -- Arnaud From phd at phdru.name Mon Jan 30 21:06:32 2012 From: phd at phdru.name (Oleg Broytman) Date: Tue, 31 Jan 2012 00:06:32 +0400 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <4F26EE98.90001@stoneleaf.us> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <20120130191136.GA1555@iskra.aviel.ru> <4F26EE98.90001@stoneleaf.us> Message-ID: <20120130200632.GA4588@iskra.aviel.ru> On Mon, Jan 30, 2012 at 11:25:12AM -0800, Ethan Furman wrote: > Oleg Broytman wrote: > >On Mon, Jan 30, 2012 at 12:49:54PM -0600, Massimo Di Pierro wrote: > >>STEP 2) We can now overload the [] to make the dynamic attributes accessible in an alternative syntax: > >> > >> class Dummy(object): > >> def __getitem__(self,key): return getattr(self,key) > >> def __setitem__(self,key,value): return setattr(self,key,value) > >> d = Dummy() > >> d.something = 5 > >> d['something'] = 5 > >> print d.something > >> print d['something'] > >> > >>STEP 3) Is anybody calling this un-pythonic? > > > > I do. The object has two different interfaces for the same values and > >that's IMO unpythonic. The Zen says > > > > "There should be one-- and preferably only one --obvious way to do it." > > Indeed. > > When you have the attribute name stored in a variable, then the one > obvious way is [] access. > > When you already know the attribute name, the one obvious way is . access. Sure, for attributes. But a dict contents ain't attributes, it's (key => value) mapping. So for a dict, getitem interface give access to dict's content, while getattr interface give access to dict's services - all those .keys(), .items(), .values(), .clear(), .update() et al. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ethan at stoneleaf.us Mon Jan 30 21:11:44 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 30 Jan 2012 12:11:44 -0800 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <20120130191136.GA1555@iskra.aviel.ru> <4F26EE98.90001@stoneleaf.us> Message-ID: <4F26F980.1040404@stoneleaf.us> Arnaud Delobelle wrote: > On 30 January 2012 19:25, Ethan Furman wrote: >>> I do. The object has two different interfaces for the same values and >>> that's IMO unpythonic. The Zen says >>> >>> "There should be one-- and preferably only one --obvious way to do it." >> >> Indeed. >> >> When you have the attribute name stored in a variable, then the one obvious >> way is [] access. > > If it's an attribute, the obvious way is getattr(obj, attrname) Fair point. I guess I'll be happy that Python will let me do it the way that works better for me. :) I think the thing to keep in mind is that this Bunch type object is not general purpose, like a dict is: it is specialized -- only string keys, only keys that don't conflict with methods (or no methods, depending on the needs), etc., etc. My custom object that works this way has attributes/methods divided by size: 10 or less and it's an attribute; more and it's a method. Works well for me. (10 because that's the maximum size for a field name in a dbf table (version 5 and below, anyway).) ~Ethan~ From ncoghlan at gmail.com Mon Jan 30 21:56:13 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 31 Jan 2012 06:56:13 +1000 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> Message-ID: On Tue, Jan 31, 2012 at 4:49 AM, Massimo Di Pierro wrote: > Trying to make sure I understand where we disagree and perhaps explain my problem better. For me this has very little to do with dictionaries. One general challenge with "arbitrary keys as attributes" data stores in Python is that they can get into messy namespace conflicts, because you want to allow arbitrary keys, but you also want access to ordinary methods. collections.namedtuple gets around this by leaving the attribute namespace largely free for end users and prefixing method names and class information with underscores. Ultimately, what ends up happening as far as the standard library goes is that "explicit is better than implicit" wins. If you don't know what keys you're going to get, then the appropriate data structure is a dict, not an object. If you *do* know which keys you're going to get, then the appropriate data structure is a predefined class (perhaps generated programmatically once the full set of permitted attributes is determined - e.g. when creating an appropriate namedtuple definition from a set of column headers). *Outside* the standard library, it's "practicality beats purity" that wins. People *like* being able to type ".attr" instead of "['attr']" when a field name happens to be a legal identifier. The core problem is that it's so easy to write a "good enough" version of such a class for yourself that nobody has ever bothered to standardise on a particular way of handling this that is suitable for stdlib inclusion (particularly when there are so many people that object to the idea *in principle*, regardless of the details of how it is implemented) Cheers, Nick. > STEP 4) > class Dummy(object): > def __getitem__(self,key): return getattr(self,key) > def __setitem__(self,key,value): return setattr(self,key,value) > def __getattr__(self,key): > return object.__getattr__(self,key) if hasattr(self,key) else Dummy() P.S. Tip for fast autovivification in Python: from collections import defaultdict def autodict(): return defaultdict(autodict) >>> store['a']['b']['c'] defaultdict(, {}) If you want this behaviour in instances of a particular class, set the instance __dict__ attributes to one of those in __new__ instead of overriding __getattr__. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Mon Jan 30 22:17:24 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 30 Jan 2012 16:17:24 -0500 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> Message-ID: On 1/30/2012 1:49 PM, Massimo Di Pierro wrote: > Trying to make sure I understand where we disagree... > class Dummy(object): pass > > d = Dummy() > d.something = 5 > print d.something > > Is anybody calling this un-pythonic? I already suggested that that is the way to make a pure *name* to data mapping object. If you want a default data object, add your corrected method: def __getattr__(self,key): return self.__dict__.get(key,None) To make Dummy instances key iterable like dicts, add def __iter__(self): return iter(self.__dict__) If you prefer to iterate by name-object pairs: def __iter__(self): return iter(self.__dict__.items()) One could define a few other special methods, like __eq__, to tie into other syntax. > STEP 2) We can now overload the [] to make the dynamic attributes accessible in an alternative syntax: > > class Dummy(object): > def __getitem__(self,key): return getattr(self,key) > def __setitem__(self,key,value): return setattr(self,key,value) > Is anybody calling this un-pythonic? Guido already did, insofar as he defines 'pythonic'. Anyway, skip that word. This is where many of us 'disagree'. This design redefines Dummy as a *string* to data mapping object. You add a second access method that makes the first access method only partial. To me, it is a conceptually crazy object. And it gets worse when you try to make it a dict, with some names now reserved for methods. Dicts, lists, and tuples have a clean separation between contents, accessed by subscript, and methods to work on contents, accessed as attributes. Many of us consider that a virtue and a feature. -- Terry Jan Reedy From cs at zip.com.au Mon Jan 30 22:22:10 2012 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 31 Jan 2012 08:22:10 +1100 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: Message-ID: <20120130212210.GA9664@cskk.homeip.net> On 30Jan2012 10:28, Eric Snow wrote: | On Mon, Jan 30, 2012 at 10:14 AM, Arnaud Delobelle wrote: | > On Jan 30, 2012 4:23 PM, "Massimo Di Pierro" | > wrote: | >> I do not think the issue is whether the people who use that semantic | >> understand it or not. I can assure you they do and they know when it is | >> appropriate to use it or not. The issue is whether there is any value is | >> making it faster by including it in python or not. Because of the increasing | >> popularity of JS I think new users are starting to expect something like it | >> out of the box. | > | > But this design decision in JavaScript is at the heart of many problems | > (e.g. simply looping over keys is a pain).? That it is widely used doesn't | > make it desirable. My experience with JavaScript is that we should keep this | > 'feature' out of Python. If people want it they can implement it very easily | > but encouraging them would be wrong. | | +1 +1 Like a few others, I have implemented this kind of class. The only one of mine that survives only supports .UPPERCASE attribute->key access and is quite special purpose. I'm very much against including such a facility in the language directly. -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ Excuse me, do you know what time it is? -About Noon. I'd prefer something more exact . . . -About Noon, Stupid. From steve at pearwood.info Tue Jan 31 01:28:53 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 31 Jan 2012 11:28:53 +1100 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> Message-ID: <4F2735C5.4000307@pearwood.info> Nick Coghlan wrote: > *Outside* the standard library, it's "practicality beats purity" that > wins. People *like* being able to type ".attr" instead of "['attr']" > when a field name happens to be a legal identifier. Yes, but people like all sorts of things that aren't good for them, and while Python can and does allow people to shoot themselves in the foot, I don't think we should be providing a standard foot-shooting class :) As I see it, x.name is best used for attributes of the object, which typically means that (1) in some sense, they are inherently part of the object; (2) they come from a relatively small enumerated set of names which is usually fixed; and (3) you are very unlikely to be added or deleting them from an object (usually a static set rather than dynamic). x['name'] is for the opposite cases: (1) the key/value pairs are not inherently part of the object, but in some sense are possessions of the object; (2) they come from a potentially arbitrarily large set of names which could grow indefinitely large; and (3) are usually dynamic rather than static. In my experience, there are very few use-cases which fall between those two extremes and don't see the need for a standard Javascript-like dotted-access dict. But I think the first place to put one would be a recipe on (say) ActiveState's cookbook, and see what sort of feedback and use it gets. -- Steven From ncoghlan at gmail.com Tue Jan 31 01:49:33 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 31 Jan 2012 10:49:33 +1000 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <4F2735C5.4000307@pearwood.info> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <4F2735C5.4000307@pearwood.info> Message-ID: On Tue, Jan 31, 2012 at 10:28 AM, Steven D'Aprano wrote: > Nick Coghlan wrote: > >> *Outside* the standard library, it's "practicality beats purity" that >> wins. People *like* being able to type ".attr" instead of "['attr']" >> when a field name happens to be a legal identifier. > > > Yes, but people like all sorts of things that aren't good for them, and > while Python can and does allow people to shoot themselves in the foot, I > don't think we should be providing a standard foot-shooting class :) You don't have to persuade *me* of that. I spent a bunch of time years ago working with Steven Bethard on the "namespaces" proto-PEP and package, and the idea is logically incoherent enough that it's simply hard to cover all the use case variants in a single class. You either end up with a baroque monstrosity that handles everything, or you have people still rolling their own because the "official" one doesn't behave exactly the way they want. About the only variant of the idea that I *could* get behind these days is a collections.record class factory that was basically a variant of collections.namedtuple that produced mutable objects instead of tuples. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From stephen at xemacs.org Tue Jan 31 02:03:11 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 31 Jan 2012 10:03:11 +0900 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <4EC01939-F741-4192-9781-E39387AFA11F@gmail.com> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <20120130200226.0a5ab1d8@pitrou.net> <4F26EE37.2040104@stoneleaf.us> <4EC01939-F741-4192-9781-E39387AFA11F@gmail.com> Message-ID: <87mx941y5s.fsf@uwakimon.sk.tsukuba.ac.jp> Massimo Di Pierro writes: > On Jan 30, 2012, at 1:39 PM, Masklinn wrote: > > The issue I have is that, to me, string keys say "arbitrary" and attributes say "enumerated set". > > I do not understand the distinciton. In fact getattr(x,...) functionally just delegates to getitem(x.__dict__, ...). I do not see why this delegation should not be achieved using the same operator __getitem__. That's because you're looking at it through the lens of the implementation. But you're suggesting changing the language. The argument for that cannot depend on the implementation. From ethan at stoneleaf.us Tue Jan 31 02:09:51 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 30 Jan 2012 17:09:51 -0800 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <4F2735C5.4000307@pearwood.info> Message-ID: <4F273F5F.2050406@stoneleaf.us> Nick Coghlan wrote: > On Tue, Jan 31, 2012 at 10:28 AM, Steven D'Aprano wrote: >> Nick Coghlan wrote: >> >>> *Outside* the standard library, it's "practicality beats purity" that >>> wins. People *like* being able to type ".attr" instead of "['attr']" >>> when a field name happens to be a legal identifier. >> >> Yes, but people like all sorts of things that aren't good for them, and >> while Python can and does allow people to shoot themselves in the foot, I >> don't think we should be providing a standard foot-shooting class :) > > You don't have to persuade *me* of that. I spent a bunch of time years > ago working with Steven Bethard on the "namespaces" proto-PEP and > package, and the idea is logically incoherent enough that it's simply > hard to cover all the use case variants in a single class. You either > end up with a baroque monstrosity that handles everything, or you have > people still rolling their own because the "official" one doesn't > behave exactly the way they want. > > About the only variant of the idea that I *could* get behind these > days is a collections.record class factory that was basically a > variant of collections.namedtuple that produced mutable objects > instead of tuples. I agree that making a general purpose object is not feasible, but I am very glad that Python doesn't put giant roadblocks in the way so we can make special purpose ones that meet our own needs and tradeoffs. ~Ethan~ From g.brandl at gmx.net Tue Jan 31 08:10:03 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 31 Jan 2012 08:10:03 +0100 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: <4F26EE37.2040104@stoneleaf.us> References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <20120130200226.0a5ab1d8@pitrou.net> <4F26EE37.2040104@stoneleaf.us> Message-ID: Am 30.01.2012 20:23, schrieb Ethan Furman: > Antoine Pitrou wrote: >> On Mon, 30 Jan 2012 12:49:54 -0600 >> Massimo Di Pierro >> wrote: >>> Trying to make sure I understand where we disagree and perhaps explain my problem better. For me this has very little to do with dictionaries. >> >>> STEP 2) We can now overload the [] to make the dynamic attributes accessible in an alternative syntax: >>> >>> class Dummy(object): >>> def __getitem__(self,key): return getattr(self,key) >>> def __setitem__(self,key,value): return setattr(self,key,value) >>> d = Dummy() >>> d.something = 5 >>> d['something'] = 5 >>> print d.something >>> print d['something'] >>> >>> STEP 3) Is anybody calling this un-pythonic? >> >> Yes. You don't need both kinds of accesses. > > Sure you do -- as soon as 'something' can be passed in via a variable: > > def some_func(name): > print(d.name) # uh, no > print(d[name]) # okay, this works If anything, the d.[name] (short for getattr(d, name)) proposal should be reconsidered. But IIRC it was concluded when discussed last time that the syntax is too hard to quickly distinguish from item access, and a better one couldn't be found. Georg From eric at trueblade.com Tue Jan 31 15:31:50 2012 From: eric at trueblade.com (Eric Smith) Date: Tue, 31 Jan 2012 09:31:50 -0500 (EST) Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <4F2735C5.4000307@pearwood.info> Message-ID: <83d6e47f0afc488953529cce99197350.squirrel@mail.trueblade.com> > About the only variant of the idea that I *could* get behind these > days is a collections.record class factory that was basically a > variant of collections.namedtuple that produced mutable objects > instead of tuples. There are many of these. My own http://pypi.python.org/pypi/recordtype is one such. It's primary use cases are: - mutability - default values Eric. From ericsnowcurrently at gmail.com Tue Jan 31 19:06:43 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 31 Jan 2012 11:06:43 -0700 Subject: [Python-ideas] Dict-like object with property access In-Reply-To: References: <17E44F92-A978-4E93-9713-7D542E61ED10@masklinn.net> <3DFDD08E-D82B-4706-8DAB-9F06B9E1F403@gmail.com> <20120130200226.0a5ab1d8@pitrou.net> <4F26EE37.2040104@stoneleaf.us> Message-ID: On Tue, Jan 31, 2012 at 12:10 AM, Georg Brandl wrote: > If anything, the d.[name] (short for getattr(d, name)) proposal should be > reconsidered. ?But IIRC it was concluded when discussed last time that the > syntax is too hard to quickly distinguish from item access, and a better > one couldn't be found. The last discussion I recall for that syntax was less than a year ago[1] and got lost amid other proposals. The idea did come up (and floundered) in 2007, though. It would be worth having another look. As Raymond put it[2]: IIRC, the idea for a __getattr__ syntax was favorably received at first, but it then drowned in a sea of syntax bikeshedding which precluded any serious discussion of use cases and benefits. Also remember that not all dead proposals have to stay dead. When generator expressions were first proposed, the PEP was rejected. The same was true for generator exceptions and for pushing data into running generators, yey these were ultimately accepted in the form of throw() and send(). +1 for reconsidering the d.[name] / d.(name) / d!name syntax. -eric [1] blow-by-blow compilation from March 2011: * Raymond brought it up -- http://mail.python.org/pipermail/python-ideas/2011-March/009265.html * ...and remained relatively neutral -- http://mail.python.org/pipermail/python-ideas/2011-March/009269.html * Greg Ewing approved -- http://mail.python.org/pipermail/python-ideas/2011-March/009271.html * Guido was thinking about it -- http://mail.python.org/pipermail/python-ideas/2011-March/009290.html * Larry Hastings on the 2007 thread -- http://mail.python.org/pipermail/python-ideas/2011-March/009284.html * Raymond in response -- http://mail.python.org/pipermail/python-ideas/2011-March/009286.html [2] http://mail.python.org/pipermail/python-ideas/2011-March/009286.html