A large part of the problem in list comprehensions (I think) is that we need to allow more then one "for" term, so we can iterate over the product of several lists. Why not solve that the same way we solved the parallel iteration problem, by using a function, called, say, "product". Then list comprehensions can limited to be (in pseudo formal grammar): '[' expr 'for' var 'in' seq [ 'if' pred ] ']' Such that "var" can be anything that can today be in a for loop, e.g., (x, (y, z)) Then we wouldn't have the irksome problem of how for's and if's intermix: the predicate at the end can be an and'ed sequence of predicates. Reference implemntation: class _Producter: def __init__(self, lists): if not lists: raise TypeError("must receive at least one list") self.lists = lists self.lengths = map(len, lists) self.n_lists = len(lists) def __getitem__(self, i): if i<0: raise ValueError("only non-negative indices supported") ret = [None] * self.n_lists for j in range(self.n_lists): i, i_j = divmod(i, self.lengths[j]) ret[j] = self.lists[j][i_j] if i != 0: raise IndexError("not that many items") return tuple(ret) def product(*lists): return _Producter(lists) -- Moshe Zadka <moshez@math.huji.ac.il> There is no IGLU cabal. http://advogato.org/person/moshez
"MZ" == Moshe Zadka <moshez@math.huji.ac.il> writes:
MZ> A large part of the problem in list comprehensions (I think) MZ> is that we need to allow more then one "for" term, so we can MZ> iterate over the product of several lists. All this belongs in PEP 202. Can we please stop discussing it here until that PEP is done? Tim Peters owns it, so until he assigns it to someone else, I suggest that if you have an opinion on list comprehensions, send your comments to him. Let Tim collect all the comments and write the PEP. didn't-mean-to-pick-on-moshe-ly y'rs, -Barry
[Moshe]
A large part of the problem in list comprehensions (I think) is that we need to allow more then one "for" term, so we can iterate over the product of several lists. ...
[Barry A. Warsaw]
All this belongs in PEP 202. Can we please stop discussing it here until that PEP is done?
But closed discussions are not the way to build a PEP: it's a PEP champion's job to *summarize* significant opposing viewpoints in the rationale, but there's no way to judge "significant" without intense community involvement. Vigorous open discussion is vital to any stds process.
Tim Peters owns it, so until he assigns it to someone else, I suggest that if you have an opinion on list comprehensions, send your comments to him. Let Tim collect all the comments and write the PEP.
This is not how I want to do it. In the particular case of list comprehensions, everyone (well, everyone both conscious and sane <wink>) is sick of the debate because it's been going on for *years*, and was almost purely repetitive long before Greg Ewing made his first patch. In this case I happen to think Moshe is trying to solve "a problem" likely nobody else sees as a problem, but I can't tell without seeing how others react to his (bizarre <wink>) suggestion. A PEP isn't a dumping ground for every random suggestion that comes along, either -- open discussion is needed to narrow things down to the truly serious alternatives. A PEP writer's job should be more editor than arbiter: Guido is the only BDFL here! All that said, I have an enormous backlog of listcomp msgs still to plow through, and I'm going to flatly ignore any in the future unless they have PEP 202 in the Subject line (as this retitled msg has). A pep-dev mailing list is clearly a very good idea, but until one is set up I'm afraid python-dev is the only natural place to bear the unbearable (to some) pep-dev traffic.
[Tim]
In this case I happen to think Moshe is trying to solve "a problem" likely nobody else sees as a problem, but I can't tell without seeing how others react to his (bizarre <wink>) suggestion.
FWIW, I like the original Greg Ewing syntax best. Reason: list comprehensions are declarative, and his syntax *looks* declarative. Procedural syntax inside those [...] is wacko, just as d = { ['one'] = 1, ['two'] = 2 } would be. Also, extra punctuation is just noise. PEP202-now-closed-ly y'rs - Gordon
(Added hiphens in 2-0-2 to keep out of the timbot's radar) On Tue, 25 Jul 2000, Tim Peters wrote:
A pep-dev mailing list is clearly a very good idea, but until one is set up I'm afraid python-dev is the only natural place to bear the unbearable (to some) pep-dev traffic.
It seems that the correct solution is not a big "pep-dev", but rather a "pep-[1-9][0-9]*-dev" lists, one for each pep. So I'll just sit back quietly and wait for ?!in to suggest roundup for this. Then we'll Barry see how he can easily make mailman do this. <202 wink> last-man-coding-ly y'rs, Z. -- Moshe Zadka <moshez@math.huji.ac.il> There is no IGLU cabal. http://advogato.org/person/moshez
Moshe Zadka <moshez@math.huji.ac.il>:
A large part of the problem in list comprehensions (I think) is that we need to allow more then one "for" term, so we can iterate over the product of several lists. Why not solve that the same way we solved the parallel iteration problem, by using a function, called, say, "product".
Then list comprehensions can limited to be (in pseudo formal grammar):
'[' expr 'for' var 'in' seq [ 'if' pred ] ']'
Such that "var" can be anything that can today be in a for loop, e.g.,
(x, (y, z))
Then we wouldn't have the irksome problem of how for's and if's intermix: the predicate at the end can be an and'ed sequence of predicates.
*KLANNGGG!!!* <-- the sound of the sacred two-by-four of enlightenment belatedly whacking Eric upside the head Dammit, I should have seen this sooner; the whole debate about parallel loops and zip() was a flare-lit clue. Gentlemen, we've been going entirely down the wrong path trying to express list comprehensions via a syntax in imperative style. What they want to be is an applicative sublanguage in functional style -- and mostly about data composition rather than the expression of iteration per se. zip() is part of this. Finding a way to handle conditionals ain't hard; in fact, it's already in the language! The right way to say [x**2 for x in range(1, 99) if isprime(x)] is like this: [x**2 for x in filter(isprime, range(1, 99)] Look at the implications: there is no longer any need for special syntax (or for python-dev debates over special syntax, or newbie confusion over special syntax) on the right side of the for. We get all the power of list comprehensions from one simple [for ... in ...] construct exactly parallel to the for ... in that Python programmers already understand. So instead of trying to fix list comprehensions with syntactic hacks, we can extend it by having a properly rich and orthogonal set of constructor functions. filter, zip, product, and lambda are good enough to cover anything the presently proposed comprehension syntax can handle. But there's a difference, because unfreezing and extending a syntax is a heckuva lot harder than inventing new functional constructors. If you notice that this looks a lot like LISP or F or Haskell, you win. These languages got comprehensions right on the basis of a lot of experience in depth. We can learn from their example. -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a> An armed society is a polite society. Manners are good when one may have to back up his acts with his life. -- Robert A. Heinlein, "Beyond This Horizon", 1942
On Tue, Jul 25, 2000 at 03:28:32PM -0400, Eric S. Raymond wrote:
[x**2 for x in filter(isprime, range(1, 99)] ... filter, zip, product, and lambda are good enough to cover anything the presently proposed comprehension syntax can handle.
... but not so elegantly, if the filter needs variables from the current scope. Consider removing all numbers in that range that are a multiple of a variable 'n'; you'd have to write: [x**2 for x in filter(lambda x, number = n: (x%n)!=0, range(1,99)) ] This uses the default argument hack to pass the value of 'n' into the scope of the lambda-created function as the parameter 'number'. Icky... I'd rather see extra syntax than more use of this kludge! --amk
Andrew Kuchling <akuchlin@mems-exchange.org>:
On Tue, Jul 25, 2000 at 03:28:32PM -0400, Eric S. Raymond wrote:
[x**2 for x in filter(isprime, range(1, 99)] ... filter, zip, product, and lambda are good enough to cover anything the presently proposed comprehension syntax can handle.
... but not so elegantly, if the filter needs variables from the current scope. Consider removing all numbers in that range that are a multiple of a variable 'n'; you'd have to write:
[x**2 for x in filter(lambda x, number = n: (x%n)!=0, range(1,99)) ]
This uses the default argument hack to pass the value of 'n' into the scope of the lambda-created function as the parameter 'number'. Icky... I'd rather see extra syntax than more use of this kludge!
Wrong answer. The right answer is to fix lambda (or some variant of lambda) to be a true lexical closure. -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a> Question with boldness even the existence of a God; because, if there be one, he must more approve the homage of reason, than that of blindfolded fear.... Do not be frightened from this inquiry from any fear of its consequences. If it ends in the belief that there is no God, you will find incitements to virtue in the comfort and pleasantness you feel in its exercise... -- Thomas Jefferson, in a 1787 letter to his nephew
On Tue, Jul 25, 2000 at 06:08:18PM -0400, Eric S. Raymond wrote:
Wrong answer. The right answer is to fix lambda (or some variant of lambda) to be a true lexical closure.
Hm... hmmmm... fixing this has been suggested before, but always foundered on the fact that creating a closure required a cycle, which would leak memory. Now we have an optional GC that should handle this, so maybe fixing it can be revisited. (But this would mean that GC is essentially no longer optional -- maybe too radical a thing to do before we're sure about the new GC. 2.1, maybe?) --amk
"AMK" == Andrew Kuchling <akuchlin@mems-exchange.org> writes:
AMK> On Tue, Jul 25, 2000 at 06:08:18PM -0400, Eric S. Raymond AMK> wrote:
Wrong answer. The right answer is to fix lambda (or some variant of lambda) to be a true lexical closure.
AMK> Hm... hmmmm... fixing this has been suggested before, but AMK> always foundered on the fact that creating a closure required a AMK> cycle, which would leak memory. Now we have an optional GC AMK> that should handle this, so maybe fixing it can be revisited. AMK> (But this would mean that GC is essentially no longer optional AMK> -- maybe too radical a thing to do before we're sure about the AMK> new GC. 2.1, maybe?) I was thinking this would be a post 2.0 PEP. Jeremy
Andrew Kuchling <akuchlin@mems-exchange.org>:
On Tue, Jul 25, 2000 at 06:08:18PM -0400, Eric S. Raymond wrote:
Wrong answer. The right answer is to fix lambda (or some variant of lambda) to be a true lexical closure.
Hm... hmmmm... fixing this has been suggested before, but always foundered on the fact that creating a closure required a cycle, which would leak memory. Now we have an optional GC that should handle this, so maybe fixing it can be revisited. (But this would mean that GC is essentially no longer optional -- maybe too radical a thing to do before we're sure about the new GC. 2.1, maybe?)
I think there's enough controversy about comprehensions at this point to warrant not rushing out an implementation we might regret. (Heh. Look at me telling *Guido* to go slow and careful...heh.) -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a> To make inexpensive guns impossible to get is to say that you're putting a money test on getting a gun. It's racism in its worst form. -- Roy Innis, president of the Congress of Racial Equality (CORE), 1988
On Tue, Jul 25, 2000 at 06:01:16PM -0400, Andrew Kuchling wrote:
On Tue, Jul 25, 2000 at 06:08:18PM -0400, Eric S. Raymond wrote:
Wrong answer. The right answer is to fix lambda (or some variant of lambda) to be a true lexical closure.
Hm... hmmmm... fixing this has been suggested before, but always foundered on the fact that creating a closure required a cycle, which would leak memory. Now we have an optional GC that should handle this, so maybe fixing it can be revisited. (But this would mean that GC is essentially no longer optional -- maybe too radical a thing to do before we're sure about the new GC. 2.1, maybe?)
I would like to get the memory consumption down before making GC standard. Hopefully Vladimir and I can team up to accomplish this. still-waiting-for-my-home-machine-ly Neil
[Andrew Kuchling]
Hm... hmmmm... fixing this has been suggested before, but always foundered on the fact that creating a closure required a cycle, which would leak memory. Now we have an optional GC that should handle this, so maybe fixing it can be revisited. (But this would mean that GC is essentially no longer optional -- maybe too radical a thing to do before we're sure about the new GC. 2.1, maybe?)
Definitely PEP material: let's please, please, please not have this entire debate again. BTW, I'm having trouble believing the release schedule for 1.6 and 2.0 without this, so definitely post-2.0 on those grounds alone.
On Tue, 25 Jul 2000, Andrew Kuchling wrote:
On Tue, Jul 25, 2000 at 06:08:18PM -0400, Eric S. Raymond wrote:
Wrong answer. The right answer is to fix lambda (or some variant of lambda) to be a true lexical closure.
Hm... hmmmm... fixing this has been suggested before, but always foundered on the fact that creating a closure required a cycle, which would leak memory. Now we have an optional GC that should handle this, so maybe fixing it can be revisited. (But this would mean that GC is essentially no longer optional -- maybe too radical a thing to do before we're sure about the new GC. 2.1, maybe?)
I think I'd rather have this solved by weak references then by building cycles. Cycles are evil even in the face of GC -- you have undetermined finalization. -- Moshe Zadka <moshez@math.huji.ac.il> There is no IGLU cabal. http://advogato.org/person/moshez
I think I'd rather have this solved by weak references then by building cycles. Cycles are evil even in the face of GC -- you have undetermined finalization.
It's worth pointing out that Python actually guarantees you may have undetermined finalization - it's just not an implementation requirement that it be that way. But-it-phrases-it-a-little-differently-than-that-<wink> ly, Mark.
Hm... hmmmm... fixing this has been suggested before, but always foundered on the fact that creating a closure required a cycle, which would leak memory. Now we have an optional GC that should handle this, so maybe fixing it can be revisited. (But this would mean that GC is essentially no longer optional -- maybe too radical a thing to do before we're sure about the new GC. 2.1, maybe?)
I think I'd rather have this solved by weak references then by building cycles. Cycles are evil even in the face of GC -- you have undetermined finalization.
But in this case, there aren't any custom finalizers involved in the cycle (although there may be some *hanging off* the cycle). E.g. after: def f(): def g(): pass f() we have (taking some notational liberties and simplifications): globals = {'f': f} f.func_globals = globals locals = {'g': g} g.func_globals = locals # or maybe g.func_parent = locals In other words the cycles are between a dictionary and a function object: globals <--> f locals <--> g I don't see any finalization issues. --Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)
On Wed, 26 Jul 2000, Guido van Rossum wrote:
I think I'd rather have this solved by weak references then by building cycles. Cycles are evil even in the face of GC -- you have undetermined finalization.
But in this case, there aren't any custom finalizers involved in the cycle (although there may be some *hanging off* the cycle).
Yes, but those finalizers hanging off the cycle would occur in some random time in the future, not right now, hence rendering the def f(): fp = open("fff") pass idiom dangerous, since you're not sure how long fp will remain open. -- Moshe Zadka <moshez@math.huji.ac.il> There is no IGLU cabal. http://advogato.org/person/moshez
But in this case, there aren't any custom finalizers involved in the cycle (although there may be some *hanging off* the cycle).
Yes, but those finalizers hanging off the cycle would occur in some random time in the future, not right now, hence rendering the
def f(): fp = open("fff") pass
idiom dangerous, since you're not sure how long fp will remain open.
Aha. You're complaining about the finalizers being postponed arbitrarily. I thought you were complaining about the issue of finalization order *within* the cycle. Since this is still an improvement over current practice (where a finalizer hanging off a cycle may never be called) and there are no serialization issues, I don't see the problem. If you want to use weak references, please try to specify *in detail* which objects would have weak references to which other objects and how the system would resolve these... I'm never sure that things can be made 100% safe using this technique, so it may help to spell it out for me! --Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)
On Wed, 26 Jul 2000, Guido van Rossum wrote:
Since this is still an improvement over current practice (where a finalizer hanging off a cycle may never be called) and there are no serialization issues, I don't see the problem.
Today inner functions don't create cycles involving local variables. Lexical scoping would change that.
If you want to use weak references, please try to specify *in detail* which objects would have weak references to which other objects and how the system would resolve these... I'm never sure that things can be made 100% safe using this technique, so it may help to spell it out for me!
OK, how's that: inner functions have weak reference to variables in the same scope def f(): x = 1 def g(): pass g would have only a weak reference to "x". I'm changing "no reference" to "weak reference", so I'm not incrementing effective reference counting. -- Moshe Zadka <moshez@math.huji.ac.il> There is no IGLU cabal. http://advogato.org/person/moshez
Today inner functions don't create cycles involving local variables. Lexical scoping would change that.
Fair enough. To resolve this, an inner function should only have a reference to the outer function if it actually references the outer function's locals -- which can be determined at compile time.
If you want to use weak references, please try to specify *in detail* which objects would have weak references to which other objects and how the system would resolve these... I'm never sure that things can be made 100% safe using this technique, so it may help to spell it out for me!
OK, how's that: inner functions have weak reference to variables in the same scope
def f(): x = 1 def g(): pass
Insert "print x" in the body of g, and "return g" in the body of f.
g would have only a weak reference to "x".
I'm changing "no reference" to "weak reference", so I'm not incrementing effective reference counting.
Ehm, I'm always confused by the semantics of weak references (you could say that I haven't a clue about them), so you'd have to do better than this. In any case, in your proposal the weak reference to x would go away when f returns, so if g is exported from f, it breaks! def f(): x = 1 def g(): print x g() return g gee = f() gee() This would first print 1 (the call to g() inside f()) and then raise a NameError (probably a new subclass e.g. UnboundOuterNameError). This behavior is unacceptable, at least to the people who have been clamoring most for this feature (the Schemers): it is expected that x is kept alive as long as g lives. So, try again! --Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)
On Wed, 26 Jul 2000, Guido van Rossum wrote:
OK, how's that: inner functions have weak reference to variables in the same scope
def f(): x = 1 def g(): pass
Insert "print x" in the body of g, and "return g" in the body of f.
What are you trying to do, simulate classes via closures? I don't see any reason for this to work. However, this should work def powers_of(n): return map(lambda x: n**x, range(10))
Ehm, I'm always confused by the semantics of weak references (you could say that I haven't a clue about them), so you'd have to do better than this.
In any case, in your proposal the weak reference to x would go away when f returns, so if g is exported from f, it breaks!
Exactly. But breaks cleanly, of course (what yo suggested works great)
This behavior is unacceptable, at least to the people who have been clamoring most for this feature (the Schemers): it is expected that x is kept alive as long as g lives.
Ummmm....to put it harshly f**k Schemers (I'm speaking as a latent schemer myself, so I can be harsh): I don't want lexical scoping to simulate classes, Python's got a wonderful object system. I want lexical scoping to make the common case of map(lambda) or def _(...): .... map(_, ....) less horrible (IOW, without the lambda x,n=n,y=y,something=other: hack) The only question about lexical scoping is "what are we trying to do?". I'm trying to have a working map(). You (seem to be) trying to simulate objects via closures. -- Moshe Zadka <moshez@math.huji.ac.il> There is no IGLU cabal. http://advogato.org/person/moshez
Moshe Zadka writes:
On Wed, 26 Jul 2000, Guido van Rossum wrote:
OK, how's that: inner functions have weak reference to variables in the same scope
def f(): x = 1 def g(): pass
Insert "print x" in the body of g, and "return g" in the body of f.
What are you trying to do, simulate classes via closures? I don't see any reason for this to work.
There's not simulation going on. He's just expecting closures to work. I think that any proposal for changed scoping needs to do lexical scoping properly, including closures. If it doesn't, I vote -1. The argument about finalization is specious. You should not write code that depends on current finalization semantics to do things like closing files. It's relying on an implementation-dependent feature that is not part of the language spec. (Just try it on JPython.) Jeremy
Jeremy Hylton <jeremy@beopen.com>:
The argument about finalization is specious. You should not write code that depends on current finalization semantics to do things like closing files. It's relying on an implementation-dependent feature that is not part of the language spec. (Just try it on JPython.)
I strongly agree. -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a> Morality is always the product of terror; its chains and strait-waistcoats are fashioned by those who dare not trust others, because they dare not trust themselves, to walk in liberty. -- Aldous Huxley
On Wed, 26 Jul 2000, Jeremy Hylton wrote:
The argument about finalization is specious. You should not write code that depends on current finalization semantics to do things like closing files. It's relying on an implementation-dependent feature that is not part of the language spec. (Just try it on JPython.)
I know, and I'm not. But the thing is, there are plenty of users of CPython which do rely on this feature -- so you're going to break people's code. Not nice. -- Moshe Zadka <moshez@math.huji.ac.il> There is no IGLU cabal. http://advogato.org/person/moshez
I know, and I'm not. But the thing is, there are plenty of users of CPython which do rely on this feature -- so you're going to break people's code. Not nice.
No - we will simply be pointing out their already broken code. We arent breaking anything! I have no problem with this at all. The sooner we point out the broken code the better. The last thing we want is for obviously and documented broken code to suddenly be considered non-broken simply by the volume of poor code out there... Either way, I think you are overstating the problem. The main scenario is closing resources, and for most users these are unlikely to be so scarce that delaying their finalization will cause problems. If someone is writing a web-server, for example, and assumes that files or sockets are automatically closed, and they open them at such a rate that true GC causes them a problem, then IMO they have got what they deserve ;-) Mark.
Hi all, This rant is about automatically reclaimed resources like file descriptors. First let me cite a few sentences from the ancient book "Programming Python" written by Mark Lutz (p.122). Since this was the very first available book on Python, it could be considered as some kind of Python Programmers Guide. Much existing code today relies on the lessons teached there: """The short answer is that all files in this program are closed automatically: there's no need to call the file object "close" method. [...] More specifically, Python uses a reference count garbage collection strategy to reclaim memory behind the scenes. [...] each time we set 'input' to a new file in the for loop above, its previous value can be reclaimed. [...] When a file object is reclaimed, the file is automatically closed [...] We usually don't need to close such files manually; [...] """ A typical code example (simplified from a real application) :
import os, rfc822 posters = {} for arti in os.listdir('/var/spool/news/comp/lang/python'): ... poster = rfc822.Message(open(arti)).get("From") ... posters[poster] = posters.get(poster, 0) + 1 ... for poster, howoften in posters.items(): ... print poster, "posted", howoften, "times to clp." ...
I'm in no way convinced to consider code like above as broken. This simple automatic and deterministic behaviour of C-Python is part of the attractive elegance of Python and one of the major advantages over other languages like Java, that IMO suck beans. I've *NOT* tried scripts like the example above with JPython. But if JPython would run out of available file descriptors with such a simple application, then I think JPython or the Java VM are broken. Jeremy Hylton <jeremy@beopen.com>:
The argument about finalization is specious. You should not write code that depends on current finalization semantics to do things like closing files. It's relying on an implementation-dependent feature that is not part of the language spec. (Just try it on JPython.)
Eric S. Raymond" <esr@thyrsus.com>:
I strongly agree.
Mark Hammond <mhammond@skippinet.com.au> (answering to Moshe):
No - we will simply be pointing out their already broken code. We arent breaking anything!
I have no problem with this at all. The sooner we point out the broken code the better. The last thing we want is for obviously and documented broken code to suddenly be considered non-broken simply by the volume of poor code out there...
Either way, I think you are overstating the problem. The main scenario is closing resources, and for most users these are unlikely to be so scarce that delaying their finalization will cause problems. If someone is writing a web-server, for example, and assumes that files or sockets are automatically closed, and they open them at such a rate that true GC causes them a problem, then IMO they have got what they deserve ;-)
no no no. This time I have to agree with Moshe and disagree with Mark, Jeremy, Eric. Or I've missed the point. Regards, Peter -- Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260 office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen)
This rant is about automatically reclaimed resources like file descriptors.
First let me cite a few sentences from the ancient book "Programming Python" written by Mark Lutz (p.122). Since this was the very first available book on Python, it could be considered as some kind of Python Programmers Guide. Much existing code today relies on the lessons teached there:
"""The short answer is that all files in this program are closed automatically: there's no need to call the file object "close" method. [...] More specifically, Python uses a reference count garbage collection strategy to reclaim memory behind the scenes. [...] each time we set 'input' to a new file in the for loop above, its previous value can be reclaimed. [...] When a file object is reclaimed, the file is automatically closed [...] We usually don't need to close such files manually; [...] """
At the time Mark wrote, this was true. When JPython was introduced, I realized that I couldn't guarantee this in all implementations, so I removed that part of the language spec (at least in my head, and in the collective memory of the Python community). In any case, Moshe used this argument to defend weak binding to references to outer variables from inside inner functions; but that breaks closures, which is unacceptable. --Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)
"PF" == Peter Funk <pf@artcom-gmbh.de> writes:
PF> Hi all, This rant is about automatically reclaimed resources PF> like file descriptors. PF> First let me cite a few sentences from the ancient book PF> "Programming Python" written by Mark Lutz (p.122). Since this PF> was the very first available book on Python, it could be PF> considered as some kind of Python Programmers Guide. Feel free to think of it as a guide, but don't think of it as a specification! (It is, as you mention, fairly old.) The only official word on reclaiming externel resource is the Python Language Reference -- http://www.python.org/doc/current/ref/.
Some objects contain references to "external" resources such as open files or windows. It is understood that these resources are freed when the object is garbage-collected, but since garbage collection is not guaranteed to happen, such objects also provide an explicit way to release the external resource, usually a close() method. Programs are strongly recommended to explicitly close such objects. The `try...finally' statement provides a convenient way to do this.
If this isn't clear enough, we could revise it for the next release of the documentation. As Guido said, the code is broken, not JPython. Jeremy
Hi, Jeremy Hylton wrote: [...]
Feel free to think of it as a guide, but don't think of it as a specification! (It is, as you mention, fairly old.) The only ------------------------------------------^^^^^^^^^^^ This was my argument to harden Moshes point, that *MUCH* code written until today relies on this advertised behaviour!
official word on reclaiming externel resource is the Python Language Reference -- http://www.python.org/doc/current/ref/.
Some objects contain references to "external" resources such as open files or windows. It is understood that these resources are freed when the object is garbage-collected, but since garbage collection is not guaranteed to happen, such objects also provide an explicit way to release the external resource, usually a close() method. Programs are strongly recommended to explicitly close such objects. The `try...finally' statement provides a convenient way to do this.
If this isn't clear enough, we could revise it for the next release of the documentation. As Guido said, the code is broken, not JPython.
Thank you: This paragraph is indeed clear enough. :-( And I must admit, that the above cited paragraph is at least as old as Mark Lutz book (I've just restored Python1.2/Doc to check this out, since I couldn't believe that in the first place: damned time machine!) I must have overlooked this particular paragraph in the past and so may many other users of Python (At least Mark Lutz ;-) ?) However I disagree with this paragraph. IMO one of the major strenghts of Python (in its reference implementation C-Python) has today compared to other languages (Java) is this particular behaviour. The argument made in the last sentence of the paragraph cited from the reference manual is bogus: In order to be able to call a close() method you have to keep an explicit reference to the object in question. This usually requires many lines of source code where previously only one line was needed: func(constructor("foo").method()) has to be recoded as try: ref = constructor("foo") func(ref.method()) finally: ref.close() if constructor happens to keep external resources. This is a factor of five (in LOC) of code bloat and is absolutely unacceptable. And in GUI programs you usually have *many* objects, that keep external resources like windows, files and the like. If this kind of coding is required to make existing Python apps running reliable under 'JPython', then this will render 'JPython' (and any other Python implementation without proper reliable reclaiming) pretty useless for me. (and may be to other members of the Python community) For me all those nifty new features currently discussed here (augmented ass., list comprehensions...) are not all *that* useful, compared to this important convience of the current C-Python implementation. Regards, Peter -- Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260 office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen)
IMO one of the major strenghts of Python (in its reference implementation C-Python) has today compared to other languages (Java) is this particular behaviour.
["this behavior": closing files as soon as they go out of scope]
The argument made in the last sentence of the paragraph cited from the reference manual is bogus: In order to be able to call a close() method you have to keep an explicit reference to the object in question. This usually requires many lines of source code where previously only one line was needed:
func(constructor("foo").method()) has to be recoded as try: ref = constructor("foo") func(ref.method()) finally: ref.close() if constructor happens to keep external resources. This is a factor of five (in LOC) of code bloat and is absolutely unacceptable.
Only if you expect that func() may fail *and* that there may be an outer try/except that keeps the program alive for much longer. Normally it becomes the totally acceptable ref = constructor("foo") func(ref.method()) ref.close() I would recommend the try/finally version only if func() is *expected* to fail for some I/O related reason. And in much Python code (e.g. most scripts and throw-away programming) even the one-liner is totally acceptable.
If this kind of coding is required to make existing Python apps running reliable under 'JPython', then this will render 'JPython' (and any other Python implementation without proper reliable reclaiming) pretty useless for me. (and may be to other members of the Python community)
This sounds like a bit of an exaggeration to me. The fact of the matter is that we can't guarantee this, and in fact there are lots of gotcha's with expecting this behavior (e.g. when you catch an exception while a file object is local, the exception handling may easily keep it alive longer than expected. It's also very easy to forget other references to the object. Listen, I'm sorry for causing your worldview on this point to collapse, but it's no big deal, okay?! --Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)
Hi, Guido van Rossum:
IMO one of the major strenghts of Python (in its reference implementation C-Python) has today compared to other languages (Java) is this particular behaviour.
["this behavior": closing files as soon as they go out of scope]
The argument made in the last sentence of the paragraph cited from the reference manual is bogus: In order to be able to call a close() method you have to keep an explicit reference to the object in question. This usually requires many lines of source code where previously only one line was needed:
func(constructor("foo").method()) has to be recoded as try: ref = constructor("foo") func(ref.method()) finally: ref.close() if constructor happens to keep external resources. This is a factor of five (in LOC) of code bloat and is absolutely unacceptable.
Only if you expect that func() may fail *and* that there may be an outer try/except that keeps the program alive for much longer.
In a GUI this is usually the case: For example Pmw contains such a default handler, which catches all otherwise uncaught exceptions and prevents the application from bombing.
Normally it becomes the totally acceptable
ref = constructor("foo") func(ref.method()) ref.close()
Sigh. This is still a factor of three. But LOCs are not my major point: An application programmer has to know, whether the object in question uses external resources or not, or he has to use the inconvinient 'close()' or 'destroy()' anyway, if he want's to make sure. [...]
And in much Python code (e.g. most scripts and throw-away programming) even the one-liner is totally acceptable.
Only if the "constructor().method()" idiom is not contained in a loop. Ancient Unix systems allowed only 20 files open at the same time. Although this has been increased to 120 or so in the mean time, you will easily ran out of file descriptors with the simple throw-away script posted in my first rant in this thread:
import os, rfc822 posters = {} for arti in os.listdir('/var/spool/news/comp/lang/python'): ... poster = rfc822.Message(open(arti)).get("From") ... posters[poster] = posters.get(poster, 0) + 1 ... for poster, howoften in posters.items(): ... print poster, "posted", howoften, "times to clp." ...
I believe scripts like this one are very common in the industry, since this idioms were advertised by several books (I still have to look up Martin v.Loewis "Das Pythonbuch", but IRC it contained subsection comparing Pythons garbage collection with Java GC and advertised the advantages of ref count GC)
If this kind of coding is required to make existing Python apps running reliable under 'JPython', then this will render 'JPython' (and any other Python implementation without proper reliable reclaiming) pretty useless for me. (and may be to other members of the Python community)
This sounds like a bit of an exaggeration to me.
May be. Since Tkinter has been ported to JPython, I theoretically could give this platform at least a try. But now I know in advance, where I've to expect serious problems. I believe a lot of my GUI code based on Tkinter/Pmw makes use of this behaviour which simply works as advertised by Mark Lutz (and possibly others). [...]
Listen, I'm sorry for causing your worldview on this point to collapse, but it's no big deal, okay?!
If it is only *my* worldview: sure. But their might be others, who have learned Python 3 or 4 years ago. May be by using C-Python, may be by using Mark Lutz book "Programming Python". Since C-Python is available on all major platforms and the implementation in fact guarantees this behaviour (Pheeew!), this is indeed no big deal. No need to use 'JPython'. ;-) Regards, Peter -- Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260 office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen)
[Peter Funk]
... Only if the "constructor().method()" idiom is not contained in a loop. Ancient Unix systems allowed only 20 files open at the same time. Although this has been increased to 120 or so in the mean time, you will easily ran out of file descriptors with the simple throw-away script posted in my first rant in this thread: ... I believe scripts like this one are very common in the industry, since this idioms were advertised by several books (I still have to look up Martin v.Loewis "Das Pythonbuch", but IRC it contained subsection comparing Pythons garbage collection with Java GC and advertised the advantages of ref count GC)
You don't have to dig that deep: they're also common in the standard distribution, including at least two of mine (checkappend.py and tabnanny.py). There's a long copy/paste/modify tradition of not bothering to close file objects in multi-file tools, dating back at least to Guido's 1991 eptags.py <wink>. These aren't hard to "fix", but that isn't the point. A lot of code out there would start failing in data- and platform-dependent ways if CPython stopped cleaning up dead file objects "real soon" after they die, and there's no easy way to identify in advance which code that may be. It would piss off a lot of people! But I don't know why we're arguing about it. Nobody (AFAIK) has announced plans to take refcounting out of CPython, but that you can't *rely* on refcounting across Python implementations is ancient news (and a reality in practice since JPython). Guido certainly can't *require* that all implementations use refcounting! The Reference Manual has been clear about this for years, and other implementations already rely on what they were promised there: it's already too late to stop this from being a basis on which implementations will compete. as-if-microsoft-had-the-resources-to-take-on-beopen-anyway<wink>-ly y'rs - tim
Hi, Tim Peters:
[about Class().method()-idiom] [...] You don't have to dig that deep: they're also common in the standard distribution, including at least two of mine (checkappend.py and tabnanny.py). There's a long copy/paste/modify tradition of not bothering to close file objects in multi-file tools, dating back at least to Guido's 1991 eptags.py <wink>.
These aren't hard to "fix", but that isn't the point. A lot of code out there would start failing in data- and platform-dependent ways if CPython stopped cleaning up dead file objects "real soon" after they die, and there's no easy way to identify in advance which code that may be. It would piss off a lot of people!
Thank you! Pheew. This applies in particular not only to file descriptors. This hits other limited resources also: color table entries, windows, network connections and so on.
But I don't know why we're arguing about it. Nobody (AFAIK) has announced plans to take refcounting out of CPython, but that you can't *rely* on refcounting across Python implementations is ancient news (and a reality in practice since JPython).
This topic was initially raised in the recent discussion about weak references which had a completely different subject line: "Product iteration". Mark Hammond wrote in this thread: MH> I have no problem with this at all. The sooner we point out the broken MH> code the better. The last thing we want is for obviously and documented MH> broken code to suddenly be considered non-broken simply by the volume of MH> poor code out there... Here I would say : "too late!" The volume of code is already there and from my POV this code simply doesn't look broken. I think it is but elegant, because it is usually shorter and was easier to write, because the programmer hasn't to bother too much about any external resources, which may have been used by some temp objects under the hood.
Guido certainly can't *require* that all implementations use refcounting!
Okay. But this is "(for language lawyers)" anyway ;-). Anybody who want's to develop a seriously competing Python implementation is well advised to implement a similar "real soon" object recycling strategy regardless of what the Language Reference may tell them and what Guido "require". The existing code requires it. The current reliable automatic recycling behaviour in C-Python is so very very convenient, that I wouldn't want to do without it in practice. This is like having to use the staircase to go upstairs to the 10th floor in a building if there is an elevator with an open door just beside the first step.
The Reference Manual has been clear about this for years, and other implementations already rely on what they were promised there: it's already too late to stop this from being a basis on which implementations will compete.
Competing Python implementations? Hmmm.... Hmmm.... there is "stackless" but AFAIK it also uses refcounting... <duck>. ;-) Happy with C-Python, Peter
Peter, what would you say if someone told you your Python programs would run, oh, 30% faster, but in return for that you could no longer rely on refcount semantics? What do you think *most* people would say? I don't know, but I'm certain I'd hear different answers from different people. Don't fall into the trap of believing that refcounting is a pure win! JPython and Vyper and the MS runtime aren't doing "real gc" just because they want to irritate people <wink>. There are tradeoffs here, and new Python code is probably more interesting to MS than old Python code. I personally don't think you'd find this a big deal if you had *always* needed to do explicit .close()-like calls in Python, and most programmers (across the world and across languages) have never used a language in which they *didn't* need to do explicit closes. Python has plenty of other attractions for them. MS may very well be delighted to leave the legacy Python market to BeOpen <0.9 wink>. remind-me-to-call-uncle-bill-in-the-morning-ly y'rs - tim
Don't fall into the trap of believing that refcounting is a pure win! JPython and Vyper and the MS runtime aren't doing "real gc" just because they want to irritate people <wink>. There are tradeoffs here, and new Python code is probably more interesting to MS than old Python code.
This aspect of GC is, however, irritating some C++ programmers who are an important MS customer group. This can be used to block C# and 'managed' C++ inside .NET with the argument "C# is /less/ safe because it is harder to ensure clean up so you'll leak resources". The base problem is that the only resource GC knows about is memory. There should be a way of attaching other resources such as file handles and data base connections to a similar mechanism. Neil
Good morning Tim, Tim Peters:
Peter, what would you say if someone told you your Python programs would run, oh, 30% faster, but in return for that you could no longer rely on refcount semantics?
Personally I wouldn't go for that trade. Waiting three or four more months and buying new hardware usually gives the same speedup anyway. What really matters is programmer time, not program run time. At least most of the time. When I started programming (with punch cards 25 yrs ago) the situation was clearly different. But today? Computing power is so incredible cheap today...
What do you think *most* people would say? [...] Hard to say.
But I believe people looking at Python are more interested in programmer productivity than in highly optimized performance. And don't having to worry about tedious and clumsy cleanup actions is an undeniable win in programmer productivity. This win is twofold: 1. you don't have to check, whether the objects provide a close() or a destroy() method. 2. you don't have to invent silly names for these otherwise anonymous objects. Regards, Peter -- Peter Funk, Oldenburger Str.86, D-27777 Ganderkesee, Germany, Fax:+49 4222950260 office: +49 421 20419-0 (ArtCom GmbH, Grazer Str.8, D-28359 Bremen)
On Sun, 30 Jul 2000, Tim Peters wrote:
But I don't know why we're arguing about it. Nobody (AFAIK) has announced plans to take refcounting out of CPython, but that you can't *rely* on refcounting across Python implementations is ancient news (and a reality in practice since JPython).
Well, to bring this back to the original: I proposed a variant of lexical scoping where an inner function has only a weak reference to its surroundin environment, so that def f(x): def g(): return x return g f(1)() Would fail with "UnboundOuterVariable", but the more commonly used def f(x): return map(lambda i,x=x: i*x, range(10)) Would turn into def f(x): return map(lambda i: i*x, range(10)) And work. Guido seemed to want to simulate classes by closures <wink>, and I tried to convince him that a) Python has a perfectly good class mechanism <3000 wink> b) non-weak references would cause cycles, which would mean that the currently working code: def f(file): file = open(file) def g(x): print x g(file.read()) Would fail is called in a loop if GC is not called too often. lexical-scoping-is-too-powerful-for-its-own-good-ly y'rs, Z. -- Moshe Zadka <moshez@math.huji.ac.il> There is no IGLU cabal. http://advogato.org/person/moshez
Plea: could we try to get an accurate subject line on at least 1 msg in 10? Thanks. [Moshe]
I know, and I'm not. But the thing is, there are plenty of users of CPython which do rely on this feature -- so you're going to break people's code. Not nice.
[Mark Hammond]
No - we will simply be pointing out their already broken code. We arent breaking anything!
I have no problem with this at all. The sooner we point out the broken code the better. The last thing we want is for obviously and documented broken code to suddenly be considered non-broken simply by the volume of poor code out there...
Spoken like a man who has never maintained a language implementation used by hordes of unhappy users <0.5 wink>. The bitching about "breaking" .append(1,2,3) was nothing compared to what this would stir up, and when the first "important" customer seriously threatens to walk, vendors usually back down out of a healthy sense of self-preservation. Been there so many times it even makes me want to puke <wink>.
Either way, I think you are overstating the problem.
I agree with that too. I don't think CPython could get away with this, but you have a new implementation and users will see this as a basis on which they compete. The *convenience* of CPython's (accidental but real!) semantics here is undeniable. Still, I don't see it coming up much except wrt temp files, and if that becomes An Issue for you, you can always special-case the snot out of them. users-don't-read-manuals-but-they-do-pay-the-bills-ly y'rs - tim
"MZ" == Moshe Zadka <moshez@math.huji.ac.il> writes:
MZ> I know, and I'm not. But the thing is, there are plenty of MZ> users of CPython which do rely on this feature -- so you're MZ> going to break people's code. Not nice. Nope, the code is already broken. They just don't know it yet. -Barry
moshe wrote:
On Wed, 26 Jul 2000, Jeremy Hylton wrote:
The argument about finalization is specious. You should not write code that depends on current finalization semantics to do things like closing files. It's relying on an implementation-dependent feature that is not part of the language spec. (Just try it on JPython.)
I know, and I'm not. But the thing is, there are plenty of users of CPython which do rely on this feature -- so you're going to break people's code. Not nice.
one problem is that there are many places when you don't know if you can close the file or not -- whoever handed you the file handle might expect it to remain open after you've done with it. That's why people use stuff like: self.fp = None # close if I'm the last user ::: But wouldn't fp = fopen(...); if (!fp && (errno == EMFILE || errno == ENFILE)) { py_force_gc(); fp = fopen(...); } solve this, at least for simple scripts? </F>
Eric> Dammit, I should have seen this sooner; the whole debate about Eric> parallel loops and zip() was a flare-lit clue. Gentlemen, we've Eric> been going entirely down the wrong path trying to express list Eric> comprehensions via a syntax in imperative style. What they want Eric> to be is an applicative sublanguage in functional style -- and Eric> mostly about data composition rather than the expression of Eric> iteration per se. Part of the motivation for list comprehensions as I recall is that we'd like to get away from such a strong reliance on map, reduce and filter. Although powerful concepts, as implemented in Python they tend to be inefficient in many cases, are not powerful enough (incomplete set of functions perhaps?) for the Lispers and are too confusing for non-Lispers, especially when used in complex ways (more than two of them used in the same expression constitutes complex usage to my feeble brain). (I PEP'd the subject so Guido and Tim can find it easily.) Skip
Skip Montanaro <skip@mojam.com>:
Part of the motivation for list comprehensions as I recall is that we'd like to get away from such a strong reliance on map, reduce and filter.
In that case, the the lesson is that this is a bad road to go down. The cost of moving away from a functional syntax, in this case, seems to be dubious thickets of syntactic ugliness, Mind you, I'm not a functional-language bigot; I'm not after turning Python into LISP. Python has taught me the limits of the LISP approach to the world. But I think the lesson from this particular debate is that it's not really possible to do better than functional syntax for expressing the kinds of things that want to go on the right side of a comprehension. -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a> Live free or die; death is not the worst of evils. -- General George Stark.
[Eric S. Raymond]
... The right way to say
[x**2 for x in range(1, 99) if isprime(x)]
is like this:
[x**2 for x in filter(isprime, range(1, 99)] ... If you notice that this looks a lot like LISP or F or Haskell, you win.
Win some, lose some: the original looks much more like Haskell, from which Greg Ewing borrowed list comprehensions: [x**2, x <- [1..98], isprime(x)] (Note: that's from memory, as I don't have Haskell on my laptop; while a detail may be wrong (in particular, I don't recall how Haskell spells "square"), the thrust is not: Haskell allows an arbitrary sequence of iterators and predicates, exactly as does the proposal. It's formally defined in terms of expansion to masses of lambdas and applications, but the whole point to the Haskell spelling is to raise the expressiveness above the LISP level]
These languages got comprehensions right on the basis of a lot of experience in depth. We can learn from their example.
Indeed, we're trying to <wink>. This is much more a Haskell than a LISP feature, and follows Haskell in using special syntax to relieve the monotony of relentlessly uniform function composition. It derives in turn from SETL, which followed common set-theoretic notation. This "grabbed" Guido; LISP spellings never did, and I bet never will. Indeed, your latter spelling is so close to the current map(lambda x: x**2, filter(isprime, range(1, 99)) as to make little difference.
It derives in turn from SETL, which followed common set-theoretic notation. This "grabbed" Guido; LISP spellings never did, and I bet never will.
I've never used or even seen SETL, but there is a connection: SETL had a profound influence on the designers of ABC, in particular Lambert Meertens, who spent a sabbatical year working with the folks at New York University who wrote the first validated Ada compiler in SETL. (http://www.faqs.org/faqs/computer-lang/Ada/comp-lang-ada/part2/ question 4.2.2, to save you the Google search.) I happen to have a standing invitation to come give a talk there... Since I'm posting in this thread anyway, I am with Tim -- I like Greg Ewing's proposed syntax best, for the reasons Tim stated. --Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)
guido wrote:
It derives in turn from SETL, which followed common set-theoretic notation. This "grabbed" Guido; LISP spellings never did, and I bet never will.
I've never used or even seen SETL, but there is a connection: SETL had a profound influence on the designers of ABC, in particular Lambert Meertens, who spent a sabbatical year working with the folks at New York University who wrote the first validated Ada compiler in SETL.
found this one: http://www-robotics.eecs.lehigh.edu/~bacon/setlprog.ps.gz interesting reading. SETL sure has a pythonic feel... with a few improvements, of course (like the [start next end] range literals... too late to fix that in python, I suppose...) and personally, I prefer their "tuple former" syntax over the the current PEP202 proposal: [expression : iterator] [n : n in [1:100]] [(x**2, x) : x in [1:5]] [a : a in y if a > 5] (all examples are slightly pythonified; most notably, they use "|" or "st" (such that) instead of "if") the expression can be omitted if it's the same thing as the loop variable, *and* there's at least one "if" clause: [a in y if a > 5] also note that their "for-in" statement can take qualifiers: for a in y if a > 5: ... is there any special reason why we cannot use colon instead of "for"? </F>
Fredrik Lundh wrote:
SETL sure has a pythonic feel...
with a few improvements, of course (like the [start next end] range literals... too late to fix that in python, I suppose...)
and personally, I prefer their "tuple former" syntax over the the current PEP202 proposal:
[expression : iterator]
[n : n in [1:100]] [(x**2, x) : x in [1:5]] [a : a in y if a > 5]
(all examples are slightly pythonified; most notably, they use "|" or "st" (such that) instead of "if")
the expression can be omitted if it's the same thing as the loop variable, *and* there's at least one "if" clause:
[a in y if a > 5]
also note that their "for-in" statement can take qualifiers:
for a in y if a > 5: ...
is there any special reason why we cannot use colon instead of "for"?
For mathematicians the [<expr> : <condition>] or [<expr> | <condition>] looks natural since it's their every day style of writing sets, but I doubt that CP4E users get the grasp of this. Anyway, whatever syntax you choose, please make sure that the <expr> and the <condition> parts are very well visibily separated. [<expr> for <vars> in <list>] doesn't fall in that category, IMHO. Hehe, this is starting to get the feel of SQL with their horrible SELECT syntax: SELECT <vars> FROM <list> WHERE <condition> ORDER BY <vars> etc. pp. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
[Fredrik Lundh]
and personally, I prefer their "tuple former" syntax over the the current PEP202 proposal:
[expression : iterator]
[n : n in [1:100]] [(x**2, x) : x in [1:5]] [a : a in y if a > 5]
Funny. I send a mail to Tim this morning with a similar proposal: [Peter]
Not being too happy with the proposed syntaxes I came up with a new one.
I think it is much better to parse (for human readers), but I am open about ';' as the delimiter. ':' would feel very fine to me, too.
Syntax: [<expr>; <cond>, <cond>, ...]
Example for semantics: [x+y; x in [1, 2, 3], x <= 2, y in [10, 20, 30], y in [5, 10, 20]] == [11, 21, 12, 22]
How to implement this example: (only a sketch, this will probably be too slow to really do it like this, but maybe you get the idea that it is at least possible to do it) 1) set baselist_for_x and baselist_for_y to something special (None?) indicating all possible values for x and y (they are unconstrained) 2) on encountering "x in [1, 2, 3]" set baselist_for_x to [1, 2, 3] 3) on encountering "x <= 2" filter baselist_for_x and get [1, 2] 4) on encountering "y in [10, 20, 30]" set baselist_for_y to [10, 20, 30] 5) on encountering "y in [5, 10, 20] see that baselist_for_y != special value (None?) and so filter baselist_for_y and get [10, 20] 6) produce cross product of baselist_for_x and baselist_for_y 7) apply expression to elements of cross product
The general idea is to start with lists of all possible values (represented by a special value) and use the conditions in the list comprehension to constrain that list. If we don't want to deal with infinite lists (and therefore lazy stuff) we could through an ConstraintError on the following:
[x; x > 10000] ConstraintError: missing sequence constraint for x
The difference (besides ';' against ':') is the use of commas for seperation and not using if. [Fredrik Lundh]
is there any special reason why we cannot use colon instead of "for"?
There seem to be some reasons, but I am not sure I have understood them. Probably I'll best quote from Tim's reply: [Tim Peters]
That said, I can guarantee Guido will hate that in
[x+y; x in [1, 2, 3], x <= 2, y in [10, 20, 30], y in [5, 10, 20]]
you have y in [...] meaning iteration in one position but have it meaning a filter in another.
No. I have it mean filter in all cases. That's just implementation. [Tim Peters]
The syntax is also deadly ambiguous: x <= 2 is a legit Python expression today, but so is x <= 2, y and so is x <= 2, y in [10, 20, 30] and so is x <= 2, y in [10, 20, 30], y etc. This syntax is simply unparsable given the uses for commas in Python's existing expression syntax. That's one reason "for" and "if" have been in every other proposal for at least the last year <wink>.
Wouldn't it be possible to just parse the first part (before the delimiter - be it ';' or ':') as an expression and allow a tuple of boolean expressions after the delimiter? Picture list comprehensions as list constraints. With [x,y: x in list1, y in list2, x > 10, odd(y)] I say that I want to have all tuples of x and y who satisfy the following constraints: - x is an element of list1 and greater than 10 - y is an element of list2 and odd For the parser the following cases would both be valid: [x+y: x in list1, y in list2] [(x+y): (x in list1, y in list2)] The latter would be what Python sees, but given that parentheses are optional for expressions and tuple building the former would be legal python. [Tim Peters]
Don't mean to be discouraging, but looks like you missed the first year of this debate!
Actually I think I did. But then I didn't even know about Python 4 months ago. newbie-ly y'rs Peter -- Peter Schneider-Kamp ++47-7388-7331 Herman Krags veg 51-11 mailto:peter@schneider-kamp.de N-7050 Trondheim http://schneider-kamp.de
is there any special reason why we cannot use colon instead of "for"?
Would conflict with the proposed syntax for range literals. Both [0:10] and [x : x in seq] have the same syntactical form. --Guido van Rossum (home page: http://www.pythonlabs.com/~guido/)
On Wed, Jul 26, 2000 at 08:00:33AM -0500, Guido van Rossum wrote:
is there any special reason why we cannot use colon instead of "for"?
Would conflict with the proposed syntax for range literals. Both
[0:10]
and
[x : x in seq]
have the same syntactical form.
Much, much worse: the latter *is* a range literal. It's not likely to produce anything useful, depending on 'x', but it is a valid range literal. (It'll create a range from 'x' to '0', if 'x' is not in 'seq', or '1', if 'x' is in 'seq', using step 1.) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!
"Eric S. Raymond" <esr@thyrsus.com> said:
What they want to be is an applicative sublanguage in functional style
Um, that might be what *you* want them to be, but it's the complete opposite of what *I* want them to be, which is a way of getting away from all those convoluted combinator constructs! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
Greg Ewing <greg@cosc.canterbury.ac.nz>:
What they want to be is an applicative sublanguage in functional style
Um, that might be what *you* want them to be, but it's the complete opposite of what *I* want them to be, which is a way of getting away from all those convoluted combinator constructs!
So your theory is that non-intuitive procedural syntax so complex that it *needs* usability testing is better. Riiiight... -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a> "Those who make peaceful revolution impossible will make violent revolution inevitable." -- John F. Kennedy
So your theory is that non-intuitive procedural syntax so complex that it *needs* usability testing is better.
It seems that our brains work in different ways here. What's clear to you is obscure to me, and vice versa. (I'm not the one that thinks it needs usability testing, by the way - that's a different Greg.) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+
"ESR" == Eric S Raymond <esr@thyrsus.com> writes:
ESR> Greg Ewing <greg@cosc.canterbury.ac.nz>: [>> ESR wrote:]
What they want to be is an applicative sublanguage in functional style
Um, that might be what *you* want them to be, but it's the complete opposite of what *I* want them to be, which is a way of getting away from all those convoluted combinator constructs!
ESR> So your theory is that non-intuitive procedural syntax so ESR> complex that it *needs* usability testing is better. ESR> Riiiight... Wow! There's so much jargon being slung here I don't know what's going on. (For the second time in the last few weeks, I feel like a bit player in a movie where all the major roles are played by Peter Sellers. <wink>) Seriously, I don't understand how to apply terms like "applicative sublanguage in functional style" or "procedural syntax" to a concept like list comprehensions. The basic idea is to express the elements of list using a set-like notation similar to the one used by mathematicians. I don't know whether to call that applicative, functional, or procedural; perhaps none of them applu. Maybe it's just late, but I suspect that these high-level terms don't inform the debate much. I think we can all agree on two things: 1. Greg and Eric have different goals, which is fine. 2. Usability testing is always a good thing. To paraphrase Fred Brooks, even the best language designers aren't so omniscient as to get it right the first time. Jeremy
Jeremy> 2. Usability testing is always a good thing. To paraphrase Fred Jeremy> Brooks, even the best language designers aren't so omniscient Jeremy> as to get it right the first time. Though Guido came damn close... ;-) S
Jeremy Hylton <jeremy@beopen.com>:
Seriously, I don't understand how to apply terms like "applicative sublanguage in functional style" or "procedural syntax" to a concept like list comprehensions. The basic idea is to express the elements of list using a set-like notation similar to the one used by mathematicians. I don't know whether to call that applicative, functional, or procedural; perhaps none of them applu. Maybe it's just late, but I suspect that these high-level terms don't inform the debate much.
Partly this is my error; I should have said "conventional" or "infix" where I said "procedural". I was writing too fast -- procedural languages are conventional in a different way, but I was referring to conventional syntax. More generally, I guess I have to apologize for excessive use of jargon without enough context. The crowd on this list is so fearsomely competent that I tend to forget that most of you don't have a technical specialty in language design. Here's a brief glossary of relevant language-designer jargon: functional style: Refers to a language in which everything is pure function calls, without side-effects or variables. Backus's F is one of the better-known examples. LISP would be functional if there were no (setq ...). The advantage of functional language is the same as their disadvantage: no side-effects. Makes them easy to reason about, but you have to jump through hoops to do statefulness. procedural style: A procedural language has procedures that operate on variables. Explicit statefulness and side-effects are basic to such languages. applicative sublanguage: This is more about syntax. To say a syntax is applicative is to say that it *looks* like cascaded function calls, as opposed (for example) to using infix syntax or keywords. (This is oversimplifying a little; Haskell combines keywords with a syntax that could be called applicative, but Haskell is weird by design that way.) imperative vs. declarative: In an imperative language, you tell the machine to do stuff. In a declarative language you specify a desired end state and let the language implementation deduce how to generate it. A good example of a declarative sublanguage familiar to most Python programers is Tk widget layout. List comprehensions are declarative in style, even though they analyze down to a series of imperative actions. Most of Python is much more straightforwardly imperative. One of the heuristics of good language design is that boundaries between imperative and declarative sublanguages are always tricky -- users have a strong tendency to try to port concepts across the boundary and get confused. This is behind some of the confusion about comprehensions. conventional: Means different things in different contexts. When speaking of syntax, it means pseudo-algebraic syntax with infix notation and keyords. In other contexts, it can mean procedural rather than functional, or manual memory management rather than garbage collection, or fixed-extent vs. unlimited-extent. The archetypal conventional language is Algol-60. For you young 'uns out there, think Pascal. -- <a href="http://www.tuxedo.org/~esr">Eric S. Raymond</a>
On Wed, 26 Jul 2000, Eric S. Raymond wrote:
functional style:
Refers to a language in which everything is pure function calls, without side-effects or variables. Backus's F is one of the better-known examples. LISP would be functional if there were no (setq ...).
Typo, i think: that would probably be "FP". Quite cool language, from the guy also primary in bringing us fortran.
The advantage of functional language is the same as their disadvantage: no side-effects. Makes them easy to reason about, but you have to jump through hoops to do statefulness.
I think "easy to reason about" in this case is mostly in the line of rigorous mathematic reasoning - algebraic/provability - than it is about approachability for novices/non-mathematicians! My impression is that in functional programming, you use functions to manipulate functions to get results - combinators, which are extremely powerful, expressively and mathematically, are also quite difficult to approach for the uninitiated. (I always thought APL presented similar challenges, but in the medium of programs as arrays/matrices.) I think one big advantage of the procedural style is how much easier it is for people to get what loops do, where applicative (map/reduce/apply) and combinatory logic operators are a lot harder. And list comprehensions as proposed capitalize on the greater approachability of loops. However, i do think a function like zip()/pairwise()/whatever can help for the tricky bit where the loop approach breaks down... (And i would be surprised if jeremy, who was steeped in MIT's scheme-based programming fundamentals curriculum, was asking for definitions. ?) Ken klm@digicool.com
participants (19)
-
Andrew Kuchling -
bwarsaw＠beopen.com -
Eric S. Raymond -
Fredrik Lundh -
Gordon McMillan -
Greg Ewing -
Guido van Rossum -
Jeremy Hylton -
Ken Manheimer -
M.-A. Lemburg -
Mark Hammond -
Moshe Zadka -
Neil Hodgson -
Neil Schemenauer -
Peter Schneider-Kamp -
pf＠artcom-gmbh.de -
Skip Montanaro -
Thomas Wouters -
Tim Peters