From steve at pearwood.info Wed Oct 1 02:51:57 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Oct 2014 10:51:57 +1000 Subject: [Python-ideas] Python 2's re module should take longs In-Reply-To: <20140930215744.6967bfbd@fsol> References: <20140930215744.6967bfbd@fsol> Message-ID: <20141001005156.GU19757@ando.pearwood.info> On Tue, Sep 30, 2014 at 09:57:44PM +0200, Antoine Pitrou wrote: > On Tue, 30 Sep 2014 12:16:13 -0500 > Ryan Gonzalez wrote: > > This works: > > > > re.search('(abc)', 'abc').group(1) > > > > but this doesn't: > > > > re.search('(abc)', 'abc').group(1L) > > > > The latter raises "IndexError: no such group". Shouldn't that technically > > work? > > Yes, it's a bug. Feel free to open an issue. I'm not so sure that it's a bug. Should .group(1.0) work? That also is numerically equal to 1. MatchObject.group does not necessarily have to obey the semantics of list.index or dict.__getitem__. Just because ['a', 'b'].index(1L) and {1: 'b'}[1.0] both return 'b' doesn't force .group() to do the same. The documentation for .group is underspecified, and perhaps TypeError would be a more appropriate error rather than IndexError, but IndexError is consistent with other bad arguments: py> re.search('(abc)', 'abc').group([]) Traceback (most recent call last): File "", line 1, in IndexError: no such group So I don't think this is a bug, I think it is just an unfortunate choice of misleading exception type. The re module goes back to at least 1.5, and as far as I can tell, .group has never accepted longs. (I have tested it on 2.4 through 2.7, and 1.5, and it fails with all of them.) So this is a long-established restriction on the argument. (Another restriction is that a maximum of 99 groups are supported, so there are no cases where a long is needed.) Allowing longs is a new feature, not a bug fix. Since 2.7 is bug-fix only mode, and this fix is unneeded in 3.x, there is no point in raising an issue to the tracker. -- Steven From nad at acm.org Wed Oct 1 04:31:24 2014 From: nad at acm.org (Ned Deily) Date: Tue, 30 Sep 2014 19:31:24 -0700 Subject: [Python-ideas] Python 2's re module should take longs References: <20140930215744.6967bfbd@fsol> <20141001005156.GU19757@ando.pearwood.info> Message-ID: In article <20141001005156.GU19757 at ando.pearwood.info>, Steven D'Aprano wrote: > Since 2.7 is bug-fix only mode, and this fix is unneeded in 3.x, there > is no point in raising an issue to the tracker. It's a moot point now: http://bugs.python.org/issue22530 https://hg.python.org/cpython/rev/30f72ed73c3b -- Ned Deily, nad at acm.org From steve at pearwood.info Wed Oct 1 04:35:13 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Oct 2014 12:35:13 +1000 Subject: [Python-ideas] Python 2's re module should take longs In-Reply-To: References: <20140930215744.6967bfbd@fsol> <20141001005156.GU19757@ando.pearwood.info> Message-ID: <20141001023513.GZ19757@ando.pearwood.info> On Tue, Sep 30, 2014 at 07:31:24PM -0700, Ned Deily wrote: > In article <20141001005156.GU19757 at ando.pearwood.info>, > Steven D'Aprano wrote: > > Since 2.7 is bug-fix only mode, and this fix is unneeded in 3.x, there > > is no point in raising an issue to the tracker. > > It's a moot point now: Ack; Guido has spoken, and his logic is impecable. -- Steven From solipsis at pitrou.net Wed Oct 1 11:03:56 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 1 Oct 2014 11:03:56 +0200 Subject: [Python-ideas] Python 2's re module should take longs References: <20140930215744.6967bfbd@fsol> <20141001005156.GU19757@ando.pearwood.info> Message-ID: <20141001110356.1a29693b@fsol> On Wed, 1 Oct 2014 10:51:57 +1000 Steven D'Aprano wrote: > On Tue, Sep 30, 2014 at 09:57:44PM +0200, Antoine Pitrou wrote: > > On Tue, 30 Sep 2014 12:16:13 -0500 > > Ryan Gonzalez wrote: > > > This works: > > > > > > re.search('(abc)', 'abc').group(1) > > > > > > but this doesn't: > > > > > > re.search('(abc)', 'abc').group(1L) > > > > > > The latter raises "IndexError: no such group". Shouldn't that technically > > > work? > > > > Yes, it's a bug. Feel free to open an issue. > > I'm not so sure that it's a bug. Should .group(1.0) work? That also is > numerically equal to 1. It's not about them being numerically equal, it's about them being integers (interchangeable, as Guido points out). We have been fixing many such bugs over the years. > So this is a long-established restriction on the argument. Please don't try to second-guess the documentation when deciding what is a "long-established restriction". Regards Antoine. From rob.cliffe at btinternet.com Wed Oct 1 13:16:16 2014 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 01 Oct 2014 12:16:16 +0100 Subject: [Python-ideas] Python 2's re module should take longs In-Reply-To: <20141001023513.GZ19757@ando.pearwood.info> References: <20140930215744.6967bfbd@fsol> <20141001005156.GU19757@ando.pearwood.info> <20141001023513.GZ19757@ando.pearwood.info> Message-ID: <542BE280.7030303@btinternet.com> On 01/10/2014 03:35, Steven D'Aprano wrote: > On Tue, Sep 30, 2014 at 07:31:24PM -0700, Ned Deily wrote: >> In article <20141001005156.GU19757 at ando.pearwood.info>, >> Steven D'Aprano wrote: >>> Since 2.7 is bug-fix only mode, and this fix is unneeded in 3.x, there >>> is no point in raising an issue to the tracker. >> It's a moot point now: > Ack; Guido has spoken, and his logic is impecable. > > Unlike your spelling. From Stephan.Sahm at gmx.de Thu Oct 2 19:39:21 2014 From: Stephan.Sahm at gmx.de (Stephan Sahm) Date: Thu, 2 Oct 2014 19:39:21 +0200 Subject: [Python-ideas] possible extension to how the new StopIteration is handled (python >=3.3) Message-ID: Hi all, I just tried the new possibilities in writing generators which were included since python3.3 into the python standard. Concretely, using return 1 in a generator is equal to raise StopIteration(1) Thus the following works: >>> def f(): ... yield 0 ... return 1 >>> g = f() >>> next(g) 0 >>> next(g) Traceback (most recent call last): File "", line 1, in StopIteration: 1 however calling next(g) once again gives the following: >>> next(g) Traceback (most recent call last): File "", line 1, in StopIteration mind the missing StopIteration: 1 . In my impression this is not like I intuitively think about a generator. If there is a StopIteration Exception connected to a generator, then there should be only ONE such. This is not only theoretical, but would have a real application using such generator in for loops. At the moment the following happens: >>> g = f() >>> for e in g: ... print(e) ... 0 >>> next(g) Traceback (most recent call last): File "", line 1, in StopIteration Remind again the missing StopIteration: 1 . If the same StopIteration would be thrown at every call when g is empty, one could actually extract the return value (without using some maybe possible but more complicated return_value = yield from work arrounds). Any feedbacks? Are there others thinking that this would be a straightforward extension to the newly introduced raise StopIteration(return_value) feature? looking forward to your responses, with best wishes, Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Oct 2 19:50:37 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 3 Oct 2014 03:50:37 +1000 Subject: [Python-ideas] possible extension to how the new StopIteration is handled (python >=3.3) In-Reply-To: References: Message-ID: On Fri, Oct 3, 2014 at 3:39 AM, Stephan Sahm wrote: > Any feedbacks? > Are there others thinking that this would be a straightforward extension to > the newly introduced raise StopIteration(return_value) feature? So basically, what you're saying is that the rule should change from "once StopIteration has been raised, any call to next() should also raise StopIteration" to "... raise StopIteration with the same payload". I think that's not unreasonable in the simple form; but it would mean that the generator would have to retain a reference to its return value, which is contrary to what most people will expect of function return values. It might make for unnecessary object longevity. Is there a backward compatibility issue here? It's theoretically possible for code to actively expect that a repeated StopIteration won't have a payload (eg to distinguish between the initial return and any attempt to re-next() the generator), but is that at all a reasonable thing to have done? ChrisA From greg.ewing at canterbury.ac.nz Fri Oct 3 00:15:05 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Oct 2014 11:15:05 +1300 Subject: [Python-ideas] possible extension to how the new StopIteration is handled (python >=3.3) In-Reply-To: References: Message-ID: <542DCE69.9030105@canterbury.ac.nz> Chris Angelico wrote: > it > would mean that the generator would have to retain a reference to its > return value, which is contrary to what most people will expect of > function return values. It might make for unnecessary object > longevity. This was debated during the yield-from discussions and decided against. If I remember rightly, unexpected longevity of return values was one of the main objections. -- Greg From Stephan.Sahm at gmx.de Fri Oct 3 08:36:43 2014 From: Stephan.Sahm at gmx.de (Stephan Sahm) Date: Fri, 3 Oct 2014 08:36:43 +0200 Subject: [Python-ideas] possible extension to how the new StopIteration is handled (python >=3.3) In-Reply-To: <542DCE69.9030105@canterbury.ac.nz> References: <542DCE69.9030105@canterbury.ac.nz> Message-ID: Thank you both very much, I see that this longevity problem is really crucial. A pity for me ;) best wishes, Stephan On 3 October 2014 00:15, Greg Ewing wrote: > Chris Angelico wrote: > >> it >> would mean that the generator would have to retain a reference to its >> return value, which is contrary to what most people will expect of >> function return values. It might make for unnecessary object >> longevity. >> > > This was debated during the yield-from discussions and > decided against. If I remember rightly, unexpected longevity > of return values was one of the main objections. > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Oct 3 09:01:59 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 3 Oct 2014 17:01:59 +1000 Subject: [Python-ideas] possible extension to how the new StopIteration is handled (python >=3.3) In-Reply-To: References: <542DCE69.9030105@canterbury.ac.nz> Message-ID: On Fri, Oct 3, 2014 at 4:36 PM, Stephan Sahm wrote: > Thank you both very much, > > I see that this longevity problem is really crucial. A pity for me ;) > best wishes, > Stephan In the general case, it is. But if you want it just for your own generators, it ought to be possible to write a decorator that yields-from your original function, retains the return value, and then reraises the exception repeatedly. Here's a simple version: def gen(): yield 0 return 1 class repeater: def __init__(self): self.g=gen() def __iter__(self): return self def __next__(self): if hasattr(self, "done"): raise StopIteration(self.done) try: return next(self.g) except StopIteration as exc: self.done=exc.value raise Generalizing this is left as an exercise for the reader. ChrisA From Stephan.Sahm at gmx.de Fri Oct 3 09:36:47 2014 From: Stephan.Sahm at gmx.de (Stephan Sahm) Date: Fri, 3 Oct 2014 09:36:47 +0200 Subject: [Python-ideas] possible extension to how the new StopIteration is handled (python >=3.3) In-Reply-To: References: <542DCE69.9030105@canterbury.ac.nz> Message-ID: Thank you very much Chris! works like a charm. I already know a bit about generator, but I haven't thought that such is also possible. Impressively powerful language best, Stephan On 3 October 2014 09:01, Chris Angelico wrote: > On Fri, Oct 3, 2014 at 4:36 PM, Stephan Sahm wrote: > > Thank you both very much, > > > > I see that this longevity problem is really crucial. A pity for me ;) > > best wishes, > > Stephan > > In the general case, it is. But if you want it just for your own > generators, it ought to be possible to write a decorator that > yields-from your original function, retains the return value, and then > reraises the exception repeatedly. Here's a simple version: > > def gen(): > yield 0 > return 1 > class repeater: > def __init__(self): > self.g=gen() > def __iter__(self): return self > def __next__(self): > if hasattr(self, "done"): raise StopIteration(self.done) > try: > return next(self.g) > except StopIteration as exc: > self.done=exc.value > raise > > Generalizing this is left as an exercise for the reader. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.chaumeny at gmail.com Fri Oct 3 17:09:20 2014 From: t.chaumeny at gmail.com (Thomas Chaumeny) Date: Fri, 3 Oct 2014 17:09:20 +0200 Subject: [Python-ideas] Make len() usable on a generator Message-ID: Hi! I have just come across some code counting a generator comprehension expression by doing len([foo for foo in bar if some_condition]) and I realized it might be better if we could just use len(foo for foo in bar if some_condition) as it would avoid a list allocation in memory. Another possibility is to write sum(1 for foo in bar if some_condition), but that is not optimal either as it generates a lot of intermediate additions which should not be needed. Sure, len(generator) might lead to an infinite loop but since sum(generator) is allowed in Python I see no reason why len(generator) isn't. What do you think ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Oct 3 17:17:25 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 4 Oct 2014 01:17:25 +1000 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: On Sat, Oct 4, 2014 at 1:09 AM, Thomas Chaumeny wrote: > I have just come across some code counting a generator comprehension > expression by doing len([foo for foo in bar if some_condition]) and I > realized it might be better if we could just use len(foo for foo in bar if > some_condition) as it would avoid a list allocation in memory. > > Another possibility is to write sum(1 for foo in bar if some_condition), but > that is not optimal either as it generates a lot of intermediate additions > which should not be needed. > > Sure, len(generator) might lead to an infinite loop but since sum(generator) > is allowed in Python I see no reason why len(generator) isn't. I think len() would be confusing, as it has to exhaust the generator to find out the length. The easiest would be to roll your own: def genlen(gen): len = 0 for len, _ in enumerate(gen, 1): pass return len At least then it's clear that it'll iterate over it completely. ChrisA From marky1991 at gmail.com Fri Oct 3 17:20:15 2014 From: marky1991 at gmail.com (Mark Young) Date: Fri, 3 Oct 2014 11:20:15 -0400 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: To be fair, sum has to do the same. -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.chaumeny at gmail.com Fri Oct 3 17:21:49 2014 From: t.chaumeny at gmail.com (Thomas Chaumeny) Date: Fri, 3 Oct 2014 17:21:49 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: Yes, it has to exhaust the generator to find the length, but that also is what list(generator) or sum(generator) have to do and yet they are allowed constructions. Actually I don't think that calling len(generator) would be very useful with a generator variable, but with an "anonymous" generator using a comprehension like: valid_customers = len(customer for customer in customers if customer.is_valid()) that would be useful. On Fri, Oct 3, 2014 at 5:17 PM, Chris Angelico wrote: > On Sat, Oct 4, 2014 at 1:09 AM, Thomas Chaumeny > wrote: > > I have just come across some code counting a generator comprehension > > expression by doing len([foo for foo in bar if some_condition]) and I > > realized it might be better if we could just use len(foo for foo in bar > if > > some_condition) as it would avoid a list allocation in memory. > > > > Another possibility is to write sum(1 for foo in bar if some_condition), > but > > that is not optimal either as it generates a lot of intermediate > additions > > which should not be needed. > > > > Sure, len(generator) might lead to an infinite loop but since > sum(generator) > > is allowed in Python I see no reason why len(generator) isn't. > > I think len() would be confusing, as it has to exhaust the generator > to find out the length. The easiest would be to roll your own: > > def genlen(gen): > len = 0 > for len, _ in enumerate(gen, 1): pass > return len > > At least then it's clear that it'll iterate over it completely. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Oct 3 17:32:59 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 Oct 2014 01:32:59 +1000 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: On 4 October 2014 01:21, Thomas Chaumeny wrote: > Yes, it has to exhaust the generator to find the length, but that also is > what list(generator) or sum(generator) have to do and yet they are allowed > constructions. It's appropriate to look at the big-O algorithmic complexity when deciding whether an iterator is an appropriate input to an operation. list(itr) and sum(itr) are both O(n) operations - the time they take is proportional to the length of the input. len(container) by contrast, is an O(1) operation - the time it takes is independent of the length of the input. There's no length independent way of calculating the length of an arbitrary generator - it's an inherently O(n) operation. "sum(1 for x in seq)" conveys that accurately, "len(x for x in seq)" does not. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From marky1991 at gmail.com Fri Oct 3 17:36:41 2014 From: marky1991 at gmail.com (Mark Young) Date: Fri, 3 Oct 2014 11:36:41 -0400 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: Why do you think len is an inherently O(1) operation? Sure, on array-based things it is, but for an arbitrary collection, the only logical assertion is that it's O(n). (For some containers, like arrays, it might be O(1) as well, but you can't assume that in general)? -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Fri Oct 3 17:40:18 2014 From: toddrjen at gmail.com (Todd) Date: Fri, 3 Oct 2014 17:40:18 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: On Oct 3, 2014 5:22 PM, "Thomas Chaumeny" wrote: > > Yes, it has to exhaust the generator to find the length, but that also is what list(generator) or sum(generator) have to do and yet they are allowed constructions. > > Actually I don't think that calling len(generator) would be very useful with a generator variable, but with an "anonymous" generator using a comprehension like: > valid_customers = len(customer for customer in customers if customer.is_valid()) > > that would be useful. You could just do: sum(customer.is_valid() for customer in customers) -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Fri Oct 3 17:48:06 2014 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Fri, 3 Oct 2014 10:48:06 -0500 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: On Fri, Oct 3, 2014 at 10:36 AM, Mark Young wrote: > Why do you think len is an inherently O(1) operation? Almost all containers (dicts and sets look a bit different at casual glance, but are still O(1)) in Snake store their current length as the ob_size attribute. It's thus O(1) to "calculate" it. You need not actually count the elements in the container. For example, for lists (this is from 2.7-ish, so 3.x is perhaps subtly different), omitting comments: typedef struct { PyObject_VAR_HEAD PyObject **ob_item; Py_ssize_t allocated; } PyListObject; where PyObject_VAR_HEAD is: #define PyObject_VAR_HEAD \ PyObject_HEAD \ Py_ssize_t ob_size; All len() does is return the ob_size attribute. Again, except for dicts and sets, which do things slightly differently, but are still O(1) in this regard. Skip From t.chaumeny at gmail.com Fri Oct 3 17:49:27 2014 From: t.chaumeny at gmail.com (Thomas Chaumeny) Date: Fri, 3 Oct 2014 17:49:27 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: In that case it works because boolean values will be converted to 0/1 but you have to force the construction to generate booleans. Also, that generates intermediate additions at each step which might make it slower than a fast iteration until exhaustion. On Fri, Oct 3, 2014 at 5:40 PM, Todd wrote: > > On Oct 3, 2014 5:22 PM, "Thomas Chaumeny" wrote: > > > > Yes, it has to exhaust the generator to find the length, but that also > is what list(generator) or sum(generator) have to do and yet they are > allowed constructions. > > > > Actually I don't think that calling len(generator) would be very useful > with a generator variable, but with an "anonymous" generator using a > comprehension like: > > valid_customers = len(customer for customer in customers if > customer.is_valid()) > > > > that would be useful. > > You could just do: > sum(customer.is_valid() for customer in customers) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri Oct 3 17:43:50 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 04 Oct 2014 00:43:50 +0900 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: <87oatttayx.fsf@uwakimon.sk.tsukuba.ac.jp> Thomas Chaumeny writes: > Yes, it has to exhaust the generator to find the length, but that also is > what list(generator) or sum(generator) have to do and yet they are allowed > constructions. sum(1 for x in generator) I'm on the fence about whether I prefer that to len(generator), on the grounds that I normally would not expect len to use up the sequence, while I sort of expect sum to do so (for values of "sort of expect" that acknowledge that s = [1, 2, 3] print(sum(s)) doesn't "use up" s in fact, but it does "use" all the values). From cathalgarvey at cathalgarvey.me Fri Oct 3 17:48:33 2014 From: cathalgarvey at cathalgarvey.me (Cathal Garvey) Date: Fri, 03 Oct 2014 16:48:33 +0100 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: <542EC551.1080404@cathalgarvey.me> Well, he said for _containers_, for which we assume a length attribute has been tallied all along; calling len() on a container should just access this tally, making it minimally intensive. For iteration-enabled classes, you can implement a __len__() magic method which returns whatever you like, so if you can make predictions about the length of an iterable class, even if the values are generated on the fly, then you can implement this to make it return the value without much computation. Some builtin generators appear to do just this: In [10]: %timeit len(range(100)) 1000000 loops, best of 3: 1.27 ?s per loop In [11]: %timeit len(range(1000)) 1000000 loops, best of 3: 1.51 ?s per loop In [12]: %timeit len(range(100000)) 1000000 loops, best of 3: 1.59 ?s per loop In [13]: %timeit len(range(10000000)) 1000000 loops, best of 3: 1.58 ?s per loop On 03/10/14 16:36, Mark Young wrote: > Why do you think len is an inherently O(1) operation? Sure, on array-based > things it is, but for an arbitrary collection, the only logical assertion > is that it's O(n). (For some containers, like arrays, it might be O(1) as > well, but you can't assume that in general)? > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Twitter: @onetruecathal, @formabiolabs Phone: +353876363185 Blog: http://indiebiotech.com miniLock.io: JjmYYngs7akLZUjkvFkuYdsZ3PyPHSZRBKNm6qTYKZfAM -------------- next part -------------- A non-text attachment was scrubbed... Name: 0x988B9099.asc Type: application/pgp-keys Size: 6176 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From alexander.belopolsky at gmail.com Fri Oct 3 18:20:31 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 3 Oct 2014 12:20:31 -0400 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: <542EC551.1080404@cathalgarvey.me> References: <542EC551.1080404@cathalgarvey.me> Message-ID: On Fri, Oct 3, 2014 at 11:48 AM, Cathal Garvey wrote: > Some builtin generators appear to do just this: > > In [10]: %timeit len(range(100)) > Builtin range is not a generator. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Fri Oct 3 18:30:05 2014 From: ram at rachum.com (Ram Rachum) Date: Fri, 3 Oct 2014 19:30:05 +0300 Subject: [Python-ideas] Delegating `BoundArguments.__getitem__` to `BoundArguments.arguments.__getitem__` Message-ID: Hi, I suggest delegating `BoundArguments.__getitem__` to `BoundArguments.arguments.__getitem__`, so instead of doing `bound_arguments.arguments['foo']` we could do `bound_arguments['foo']`. Since there isn't that much more to a `BoundArguments` object than its arguments I think it's nice to save some typing and be able to access the arguments directly. Thanks, Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Oct 3 18:32:14 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 3 Oct 2014 12:32:14 -0400 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <542EC551.1080404@cathalgarvey.me> Message-ID: On Fri, Oct 3, 2014 at 12:27 PM, Ian Cordasco wrote: > > Builtin range is not a generator. > > It is on Python 3 (but not Python 2): > > $ python3 > >>> range(100000) > range(0, 100000) > > And I assumed that most discussion for Python-ideas pertained to > features for Python 3, am I incorrect in that? You are correct about python-ideas, but range is not a generator in python 3 either: Python 3.4.0a0 (default:e9cecb612ff7+, Oct 3 2014, 12:07:42) >>> type(range(10)) # not a generator >>> type(x for x in range(10)) # a generator -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Fri Oct 3 18:27:51 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Fri, 3 Oct 2014 11:27:51 -0500 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <542EC551.1080404@cathalgarvey.me> Message-ID: On Fri, Oct 3, 2014 at 11:20 AM, Alexander Belopolsky wrote: > > On Fri, Oct 3, 2014 at 11:48 AM, Cathal Garvey > wrote: >> >> Some builtin generators appear to do just this: >> >> In [10]: %timeit len(range(100)) > > > Builtin range is not a generator. It is on Python 3 (but not Python 2): $ python3 >>> range(100000) range(0, 100000) And I assumed that most discussion for Python-ideas pertained to features for Python 3, am I incorrect in that? From graffatcolmingov at gmail.com Fri Oct 3 18:45:00 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Fri, 3 Oct 2014 11:45:00 -0500 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <542EC551.1080404@cathalgarvey.me> Message-ID: On Fri, Oct 3, 2014 at 11:32 AM, Alexander Belopolsky wrote: > > On Fri, Oct 3, 2014 at 12:27 PM, Ian Cordasco > wrote: >> >> > Builtin range is not a generator. >> >> It is on Python 3 (but not Python 2): >> >> $ python3 >> >>> range(100000) >> range(0, 100000) >> >> And I assumed that most discussion for Python-ideas pertained to >> features for Python 3, am I incorrect in that? > > > You are correct about python-ideas, but range is not a generator in python 3 > either: > > Python 3.4.0a0 (default:e9cecb612ff7+, Oct 3 2014, 12:07:42) >>>> type(range(10)) # not a generator > >>>> type(x for x in range(10)) # a generator > Ah yeah. All too often I conflate lazy iterators with generators. Nevermind From ram at rachum.com Fri Oct 3 18:51:51 2014 From: ram at rachum.com (Ram Rachum) Date: Fri, 3 Oct 2014 19:51:51 +0300 Subject: [Python-ideas] Delegating `BoundArguments.__getitem__` to `BoundArguments.arguments.__getitem__` In-Reply-To: References: Message-ID: And while we're at it, if the user tries to access an argument that isn't given but has a default, let's return the default. On Fri, Oct 3, 2014 at 7:30 PM, Ram Rachum wrote: > Hi, > > I suggest delegating `BoundArguments.__getitem__` to > `BoundArguments.arguments.__getitem__`, so instead of doing > `bound_arguments.arguments['foo']` we could do `bound_arguments['foo']`. > Since there isn't that much more to a `BoundArguments` object than its > arguments I think it's nice to save some typing and be able to access the > arguments directly. > > > Thanks, > Ram. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Oct 3 19:05:26 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Oct 2014 19:05:26 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: Thomas Chaumeny schrieb am 03.10.2014 um 17:09: > What do you think ? You can read the answers in the previous discussion threads on this list (and on python-list, and IIRC the py3k list, and maybe python-dev, too) that deal with exactly this proposal. Stefan From joshua at landau.ws Fri Oct 3 19:13:20 2014 From: joshua at landau.ws (Joshua Landau) Date: Fri, 3 Oct 2014 18:13:20 +0100 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <542EC551.1080404@cathalgarvey.me> Message-ID: On 3 October 2014 17:45, Ian Cordasco wrote: > On Fri, Oct 3, 2014 at 11:32 AM, Alexander Belopolsky > wrote: >> On Fri, Oct 3, 2014 at 12:27 PM, Ian Cordasco >> wrote: >>> >>> > Builtin range is not a generator. >> >> You are correct about python-ideas, but range is not a generator in python 3 >> either: > > Ah yeah. All too often I conflate lazy iterators with generators. Nevermind It's not an iterator either: from collections.abc import Iterator isinstance(range(10), Iterator) #>>> False It's just an iterable container. From abarnert at yahoo.com Fri Oct 3 19:26:05 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 3 Oct 2014 19:26:05 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <542EC551.1080404@cathalgarvey.me> Message-ID: On Oct 3, 2014, at 19:13, Joshua Landau wrote: > On 3 October 2014 17:45, Ian Cordasco wrote: >> On Fri, Oct 3, 2014 at 11:32 AM, Alexander Belopolsky >> wrote: >>> On Fri, Oct 3, 2014 at 12:27 PM, Ian Cordasco >>> wrote: >>>> >>>>> Builtin range is not a generator. >>> >>> You are correct about python-ideas, but range is not a generator in python 3 >>> either: >> >> Ah yeah. All too often I conflate lazy iterators with generators. Nevermind > > It's not an iterator either: > > from collections.abc import Iterator > isinstance(range(10), Iterator) > #>>> False > > It's just an iterable container. In a sense, it's like the dict views, or any other sequence, set, or similar container that doesn't directly store its objects but instead created them lazily. It might be nice if we had a generic term for such things. On the other hand, maybe it's not so critical. I think most of the confusion comes from people who are Python 3 novices but not Python 2 novices (who expect that if range, dict.keys, etc. no longer produce lists, they must instead be producing iterators; new programmers, transplants from Java or Perl, etc. won't have such expectations), and the number of those is dwindling and will continue to do so. From t.chaumeny at gmail.com Fri Oct 3 19:43:23 2014 From: t.chaumeny at gmail.com (Thomas Chaumeny) Date: Fri, 3 Oct 2014 19:43:23 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: I searched this list before I asked and couldn't find anything. Anyway, I understand from Nick's response that len() is conceived for collections which have a size computable in constant time (I suppose structures that maintain some inner attribute to store their length). I also believe that "length" sounds more like an attribute name than a name for something which really does some computation ("count" sounds more appropriate for that). On the other hand, it doesn't seem to be a general concern not to provide construction which can lead to hidden complexity cost (like "in" used on a list vs a set). On Fri, Oct 3, 2014 at 7:05 PM, Stefan Behnel wrote: > Thomas Chaumeny schrieb am 03.10.2014 um 17:09: > > What do you think ? > > You can read the answers in the previous discussion threads on this list > (and on python-list, and IIRC the py3k list, and maybe python-dev, too) > that deal with exactly this proposal. > > Stefan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Oct 3 20:10:40 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 3 Oct 2014 20:10:40 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: <2C5BE530-0931-40F8-9FE8-C9812615982E@yahoo.com> On Oct 3, 2014, at 19:43, Thomas Chaumeny wrote: > I searched this list before I asked and couldn't find anything. > > Anyway, I understand from Nick's response that len() is conceived for collections which have a size computable in constant time (I suppose structures that maintain some inner attribute to store their length). I also believe that "length" sounds more like an attribute name than a name for something which really does some computation ("count" sounds more appropriate for that). On the other hand, it doesn't seem to be a general concern not to provide construction which can lead to hidden complexity cost (like "in" used on a list vs a set). The collections.abc module attempted to retroactively codify the standards for these things. All containers support in, which implies that you can't expect better than linear (even though some subtypes might provide logarithmic or amortized constant). Only sized containers support len, which implies that it should be constant time. It might have been better if the 1.x designers thought through all of these issues ahead of time (as the designers of C++'s STL did), but what they came up with semi-accidentally (of course not really accidental, because Guido and friends had experience to guide their choices even when they didn't rigorously think things through) seems pretty good to me. At any rate, it seems far too late to change things that have been consistent since before 1.0 even if alternatives _would_ be more logical a priori. If you design a new language, you might want to provide separate operators/methods/functions for a log-or-better contains and a may-be-linear contains, and likewise you might want an operator to directly do may-be-linear indexing instead of having to write next(islice(?)), or you may not, but I don't think you can expect such things in Python. > > On Fri, Oct 3, 2014 at 7:05 PM, Stefan Behnel wrote: >> Thomas Chaumeny schrieb am 03.10.2014 um 17:09: >> > What do you think ? >> >> You can read the answers in the previous discussion threads on this list >> (and on python-list, and IIRC the py3k list, and maybe python-dev, too) >> that deal with exactly this proposal. >> >> Stefan >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From flying-sheep at web.de Fri Oct 3 23:10:59 2014 From: flying-sheep at web.de (Philipp A.) Date: Fri, 3 Oct 2014 23:10:59 +0200 Subject: [Python-ideas] Extend unicodedata with a name search Message-ID: I noticed that the excellent perl utility unum uses an obsolete unicode database. Since I?m a Pythonista, i recalled hearing about the stdlib unicodedata module, using which I either wanted to rewrite unum or extend its database. Unfortunately, unicodedata is very limited. Partly rightfully so, since you can convert codepoints and chars with chr() and ord(), and str.upper() and friends are unicode-aware. But the name database is only queryable using full names! I want to do unicodedata.search('clock') and get a list of dozens of glyphs with names like CLOCKWISE RIGHTWARDS AND LEFTWARDS OPEN CIRCLE ARROWS and CLOCK FACE THREE-THIRTY. Maybe this should spit out a list of (name, char) tuples? or a {name: char} dict? What do you mean? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Oct 3 23:15:29 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 03 Oct 2014 23:15:29 +0200 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: References: Message-ID: <542F11F1.5070607@egenix.com> On 03.10.2014 23:10, Philipp A. wrote: > I noticed that the excellent perl utility unum > uses an obsolete unicode database. > > Since I?m a Pythonista, i recalled hearing about the stdlib unicodedata > module, using which I either wanted to rewrite unum or extend its database. > > Unfortunately, unicodedata is very limited. Partly rightfully so, since you > can convert codepoints and chars with chr() and ord(), and str.upper() and > friends are unicode-aware. > > But the name database is only queryable using full names! I want to do > unicodedata.search('clock') and get a list of dozens of glyphs with names > like CLOCKWISE RIGHTWARDS AND LEFTWARDS OPEN CIRCLE ARROWS > and CLOCK FACE THREE-THIRTY. > > Maybe this should spit out a list of (name, char) tuples? or a {name: char} > dict? > > What do you mean? You should be able to code this as a PyPI package. I don't think it's a use case that warrants making the unicodedata module more complex. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 03 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From yselivanov.ml at gmail.com Fri Oct 3 23:34:20 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 03 Oct 2014 17:34:20 -0400 Subject: [Python-ideas] Delegating `BoundArguments.__getitem__` to `BoundArguments.arguments.__getitem__` In-Reply-To: References: Message-ID: <542F165C.3060703@gmail.com> Hi, On 2014-10-03, 12:51 PM, Ram Rachum wrote: > And while we're at it, if the user tries to access an argument that isn't > given but has a default, let's return the default. No. There is a note on this in python docs re BoundArguments.arguments: "Contains only explicitly bound arguments." This is an important feature, because it gives you information of what exact set of arguments was bound in the first place. Also, when you use BoundArguments for RPC, you're interested in passing the minimal set of information for each call. If you want to change this behaviour, you're free to extend Signature class and return a customized BA instance. In 3.5 there is Signature.from_callable for making this easy, and there is a 3 line example in docs on how to bind defaults. > > On Fri, Oct 3, 2014 at 7:30 PM, Ram Rachum wrote: > >> Hi, >> >> I suggest delegating `BoundArguments.__getitem__` to >> `BoundArguments.arguments.__getitem__`, so instead of doing >> `bound_arguments.arguments['foo']` we could do `bound_arguments['foo']`. You'll also need to delegate __setitem__, keys(), __len__ and all other Mapping methods. And to be honest, I see absolutely no point in doing this. >> Since there isn't that much more to a `BoundArguments` object than its >> arguments For now yes. In future we might want to extend the API. Yury From scott+python-ideas at scottdial.com Sat Oct 4 02:33:30 2014 From: scott+python-ideas at scottdial.com (Scott Dial) Date: Fri, 03 Oct 2014 20:33:30 -0400 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: <542F405A.6060101@scottdial.com> On 2014-10-03 11:32 AM, Nick Coghlan wrote: > list(itr) and sum(itr) are both O(n) operations - the time they take > is proportional to the length of the input. > len(container) by contrast, is an O(1) operation - the time it takes > is independent of the length of the input. What about collections.ChainMap (computes a union)? https://hg.python.org/cpython/file/3.4/Lib/collections/__init__.py#l795 What about mailbox.Maildir (calls os.listdir)? https://hg.python.org/cpython/file/3.4/Lib/mailbox.py#l408 I think there are plenty of virtual containers that fail your rule. -Scott -- Scott Dial scott at scottdial.com From stephen at xemacs.org Sat Oct 4 05:17:58 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 04 Oct 2014 12:17:58 +0900 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: <542F11F1.5070607@egenix.com> References: <542F11F1.5070607@egenix.com> Message-ID: <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> M.-A. Lemburg writes: > On 03.10.2014 23:10, Philipp A. wrote: > > Unfortunately, unicodedata is very limited. Phillip, do you really mean *very* limited? If so, I wonder what else you think is missing besides "fuzzy" name lookup. The UCD is defined by the standard, and AFAICS access to all properties is provided. > > But the name database is only queryable using full names! I want > > to do unicodedata.search('clock') and get a list of dozens of glyphs > You should be able to code this as a PyPI package. I don't think > it's a use case that warrants making the unicodedata module more > complex. I think it's unfortunate that unicodedata is limited in this particular way, since the database is in C, and as you point out hardly extensible. For example, as a native English speaker who enjoys wordplay I was able to guess which euphemism is the source of the name of U+1F4A9 without looking it up, but I doubt a non-native would be able to. A builtin ability to do fuzzy searches ("unicodenames.startswith('PILE OF')") would be useful. OTOH, a little thought convinced me that I don't know the TOOWTDI for fuzzy search here: - regexp: database will be a huge string or similar - startswith, endswith, contains: probably sufficient, but I suppose one would like at least conjunction and disjunction operations: unicodematch.contains('GREEK', 'SMALL', 'ALPHA', op='and') unicodematch.startswith('PIECE OF', 'PILE OF', op='or') (OK, that's pretty horrible, but it gives an idea.) - something else? From stephen at xemacs.org Sat Oct 4 05:36:42 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 04 Oct 2014 12:36:42 +0900 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: <87lhowtsj9.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > len(container) by contrast, is an O(1) operation - the time it takes > is independent of the length of the input. I think a better way to put this is that len(container) is *designed* to return an attribute that is computed as the container is (de-) populated, so is O(1) for access. The fact that some "containers" that wrap external objects (eg, Scott's example of Maildir) have to use O(N) algorithms for accuracy (since these objects are shared with other processes) doesn't really contradict the argument from "design", it just suggests one should be careful when giving those containers a __len__. Also, at least for the Maildir example, len(maildir) doesn't access the Maildir; it accesses the OS's directory listing facility, which has a "C" so small that I suppose it's effectively O(1) compared to sum(1 for m in maildir). I don't know if that (pseudo?) logic applies to other such examples, though. From rosuav at gmail.com Sat Oct 4 07:50:33 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 4 Oct 2014 15:50:33 +1000 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Sat, Oct 4, 2014 at 1:17 PM, Stephen J. Turnbull wrote: > - startswith, endswith, contains: probably sufficient, but I suppose > one would like at least conjunction and disjunction operations: > unicodematch.contains('GREEK', 'SMALL', 'ALPHA', op='and') > unicodematch.startswith('PIECE OF', 'PILE OF', op='or') > (OK, that's pretty horrible, but it gives an idea.) There's an easier way, though it would take a bit of setup work. Start by building up an actual list in RAM of [unicodedata.name(chr(i)) for i in range(sys.maxunicode+1)] and then do regular string operations. I'm fairly sure most Python programmers can figure out how to search a list of strings according to whatever rules they like - maybe using contains/startswith/endswith, or maybe regexps, or whatever. ChrisA From steve at pearwood.info Sat Oct 4 08:21:36 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 4 Oct 2014 16:21:36 +1000 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20141004062136.GE19757@ando.pearwood.info> On Sat, Oct 04, 2014 at 12:17:58PM +0900, Stephen J. Turnbull wrote: > M.-A. Lemburg writes: > > On 03.10.2014 23:10, Philipp A. wrote: > > > > Unfortunately, unicodedata is very limited. > > Phillip, do you really mean *very* limited? If so, I wonder what else > you think is missing besides "fuzzy" name lookup. The UCD is defined > by the standard, and AFAICS access to all properties is provided. Hmmm. There's a lot of properties in Unicode, and I'm pretty sure that unicodedata does not give access to *all* of them. Here's a line from UnicodeData.txt: http://www.unicode.org/Public/UNIDATA/UnicodeData.txt 04BF;CYRILLIC SMALL LETTER ABKHASIAN CHE WITH DESCENDER;Ll;0;L;;;;;N;CYRILLIC SMALL LETTER IE HOOK OGONEK;;04BE;;04BE There are 15 semi-colon separated fields. The zeroth is the code point, the others are described here: http://www.unicode.org/Public/5.1.0/ucd/UCD.html#UnicodeData.txt I don't believe that there is any way to get access to all 14 (excluding the code point itself) fields. E.g. how do I find out the "Unicode_1_Name"? And UnicodeData.txt is only one of many Unicode databases. See the UCD.html link above. > > > But the name database is only queryable using full names! I want > > > to do unicodedata.search('clock') and get a list of dozens of glyphs > > > You should be able to code this as a PyPI package. I don't think > > it's a use case that warrants making the unicodedata module more > > complex. > > I think it's unfortunate that unicodedata is limited in this > particular way, since the database is in C, and as you point out > hardly extensible. For example, as a native English speaker who > enjoys wordplay I was able to guess which euphemism is the source of > the name of U+1F4A9 without looking it up, but I doubt a non-native > would be able to. A builtin ability to do fuzzy searches > ("unicodenames.startswith('PILE OF')") would be useful. I would love it if unicodedata exposed the full UnicodeData.txt database in some efficient format. That would allow people to scratch their own itch without having to duplicate the UnicodeData.txt database. Failing that, the two features I miss the most are: (1) fuzzy_lookup(glob): Return iterator which yields (ordinal, name) for each unicode code point which matches the glob. Names beginning with a substring: fuzzy_lookup("SPAM*") Names ending with a substring: fuzzy_lookup("*SPAM") Names containing a substring: fuzzy_lookup("SPAM") (2) get_data(ordinal_or_character): Return a namedtuple with 15 fields, taken directly from the UnicodeData.txt database. The first function solves the very common problem of "I kind of know what the character is called, but not exactly", the second would allow people to code their own arbitrary lookups. -- Steven From steve at pearwood.info Sat Oct 4 08:29:24 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 4 Oct 2014 16:29:24 +1000 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20141004062923.GF19757@ando.pearwood.info> On Sat, Oct 04, 2014 at 03:50:33PM +1000, Chris Angelico wrote: > On Sat, Oct 4, 2014 at 1:17 PM, Stephen J. Turnbull wrote: > > - startswith, endswith, contains: probably sufficient, but I suppose > > one would like at least conjunction and disjunction operations: > > unicodematch.contains('GREEK', 'SMALL', 'ALPHA', op='and') > > unicodematch.startswith('PIECE OF', 'PILE OF', op='or') > > (OK, that's pretty horrible, but it gives an idea.) > > There's an easier way, though it would take a bit of setup work. Start > by building up an actual list in RAM of [unicodedata.name(chr(i)) for > i in range(sys.maxunicode+1)] and then do regular string operations. > I'm fairly sure most Python programmers can figure out how to search a > list of strings according to whatever rules they like - maybe using > contains/startswith/endswith, or maybe regexps, or whatever. py> x = [unicodedata.name(chr(i)) for i in range(sys.maxunicode+1)] Traceback (most recent call last): File "", line 1, in File "", line 1, in ValueError: no such name There are 1114112 such code points, and most of them are unused. Some of the used ones don't have names: py> unicodedata.name('\0') Traceback (most recent call last): File "", line 1, in ValueError: no such name But even once you deal with those complications, you'll end up duplicating information which (I presume) Python already has, and still end up needing to do a linear search in slow Python code looking for what you want. I think there are probably better solutions. Or at least, I hope there are better solutions :-) -- Steven From stephen at xemacs.org Sat Oct 4 08:47:57 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 04 Oct 2014 15:47:57 +0900 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87eguotjoi.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Angelico writes: > Start by building up an actual list in RAM of > [unicodedata.name(chr(i)) for i in range(sys.maxunicode+1)] and > then do regular string operations. I'm fairly sure most Python > programmers can figure out how to search a list of strings > according to whatever rules they like - maybe using > contains/startswith/endswith, or maybe regexps, or whatever. OK. Times are quite imprecise, but after importing re, sys, unicodedata >>> names = [unicodedata.name(chr(i)) for i in range(sys.maxunicode+1)] Traceback (most recent call last): File "", line 1, in File "", line 1, in ValueError: no such name oops, although you didn't actually claim that would work. :-) (BTW, chr(0) has no name. At least it was instantaneous. :-) Then >>> for i in range(sys.maxunicode+1): ... try: ... names.append(unicodedata.name(chr(i))) ... except ValueError: ... pass ... takes between 1 and 2 seconds, while >>> names.index("PILE OF POO") 61721 >>> "PILE OF POO" in names True is instantaneous. Note: 61721 is *much* smaller than 0x1F4A9. And now >>> pops = [name for name in names if re.match("^P\\S* O.* P", name)] >>> pops ['PILE OF POO'] takes just noticable time (250ms, maybe?) This on a 4-year-old 2.7GHz i7 MacBook Pro running "Mavericks". Plenty good for my use cases. From rosuav at gmail.com Sat Oct 4 09:13:18 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 4 Oct 2014 17:13:18 +1000 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: <87eguotjoi.fsf@uwakimon.sk.tsukuba.ac.jp> References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> <87eguotjoi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Sat, Oct 4, 2014 at 4:47 PM, Stephen J. Turnbull wrote: >>>> names = [unicodedata.name(chr(i)) for i in range(sys.maxunicode+1)] > Traceback (most recent call last): > File "", line 1, in > File "", line 1, in > ValueError: no such name > > oops, although you didn't actually claim that would work. :-) (BTW, > chr(0) has no name. At least it was instantaneous. :-) Oops, forgot about that. Yet another case where the absence of PEP 463 forces the function to have an additional argument: names = [unicodedata.name(chr(i), '') for i in range(sys.maxunicode+1)] Now it works. Sorry for the omission, this is what happens when code is typed straight into the email without testing :) > Then > >>>> for i in range(sys.maxunicode+1): > ... try: > ... names.append(unicodedata.name(chr(i))) > ... except ValueError: > ... pass > ... I would recommend appending a shim in the ValueError branch, to allow the indexing to be correct. Which would look something like this: names = [unicodedata.name(chr(i)) except ValueError: '' for i in range(sys.maxunicode+1)] Or, since name() does indeed have a 'default' parameter, the code from above. :) > takes between 1 and 2 seconds, while > >>>> names.index("PILE OF POO") > 61721 >>>> "PILE OF POO" in names > True > > is instantaneous. Note: 61721 is *much* smaller than 0x1F4A9. >>> names.index("PILE OF POO") 128169 >>> hex(_).upper() '0X1F4A9' And still instantaneous. Of course, a prefix search is a bit slower: >>> [i for i,s in enumerate(names) if s.startswith("PILE")] [128169] Takes about 1s on my aging Windows laptop, where the building of the list takes about 4s, so it should be quicker on your system. The big downside, I guess, is the RAM usage. >>> sys.getsizeof(names) 4892352 >>> sum(sys.getsizeof(n) for n in names) 30698194 That's ~32MB of stuff stored, just to allow these lookups. ChrisA From steve at pearwood.info Sat Oct 4 10:18:23 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 4 Oct 2014 18:18:23 +1000 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> <87eguotjoi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20141004081823.GG19757@ando.pearwood.info> On Sat, Oct 04, 2014 at 05:13:18PM +1000, Chris Angelico wrote: [...] > The big downside, I guess, is the RAM usage. > > >>> sys.getsizeof(names) > 4892352 > >>> sum(sys.getsizeof(n) for n in names) > 30698194 > > That's ~32MB of stuff stored, just to allow these lookups. And presumably it is already stored, to support \N{} and unicodedata.lookup(). For reference, UnicodeData.txt is a 1.4MB text file. -- Steven From abarnert at yahoo.com Sat Oct 4 11:28:52 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 4 Oct 2014 11:28:52 +0200 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: <20141004062136.GE19757@ando.pearwood.info> References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> <20141004062136.GE19757@ando.pearwood.info> Message-ID: On Oct 4, 2014, at 8:21, Steven D'Aprano wrote: > 1) fuzzy_lookup(glob): > Return iterator which yields (ordinal, name) for > each unicode code point which matches the glob. > > Names beginning with a substring: > fuzzy_lookup("SPAM*") > > Names ending with a substring: > fuzzy_lookup("*SPAM") > > Names containing a substring: > fuzzy_lookup("SPAM") Surely that last one is "*SPAM*", right? Otherwise this is a weird sort of glob where * doesn't match anything on this end, it instead constrains the opposite end or something. At any rate, why would you expect glob here? There's really nothing else in Python that uses glob patterns except for glob/fnmatch, which are explicitly matching equivalent OS services. It doesn't seem any more natural to think of the database as a directory of files than as a file of text or a database of key values, so why not a regex, or a SQL like pattern, or something else? From steve at pearwood.info Sat Oct 4 12:26:10 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 4 Oct 2014 20:26:10 +1000 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> <20141004062136.GE19757@ando.pearwood.info> Message-ID: <20141004102609.GJ19757@ando.pearwood.info> On Sat, Oct 04, 2014 at 11:28:52AM +0200, Andrew Barnert wrote: > On Oct 4, 2014, at 8:21, Steven D'Aprano wrote: > > > 1) fuzzy_lookup(glob): > > Return iterator which yields (ordinal, name) for > > each unicode code point which matches the glob. > > > > Names beginning with a substring: > > fuzzy_lookup("SPAM*") > > > > Names ending with a substring: > > fuzzy_lookup("*SPAM") > > > > Names containing a substring: > > fuzzy_lookup("SPAM") > > Surely that last one is "*SPAM*", right? It's a fuzzy lookup, not an exact lookup, so by default it matches the substring anywhere in the string. (If you want an exact name lookup, unicodedata already supports that.) You could write "*SPAM*" of course, but the stars would be redundant. I'm not trying to match the full range of shell globs, I'm just suggesting the minimum set of features I want. The only metacharacter I can see a practical use for is *. If you can think of uses for other metacharacters, feel free to propose them. > Otherwise this is a weird sort of glob where * doesn't match anything > on this end, it instead constrains the opposite end or something. I don't quite understand what you are trying to say here. > At any rate, why would you expect glob here? There's really nothing > else in Python that uses glob patterns except for glob/fnmatch, which > are explicitly matching equivalent OS services. It doesn't seem any > more natural to think of the database as a directory of files than as > a file of text or a database of key values, so why not a regex, or a > SQL like pattern, or something else? Because globs are simpler than regexes, and easier to use. They support the most common (or at least what I think will be the most common) use-cases: matching something that contains, ends with or starts with a substring. (Globbing may be most well-known from shells, but there is nothing about glob syntax that is limited to matching file names. It's a string matching language, which the shell happens to use to match file names.) I don't see a use for supporting the full range of regexes. As far as I am concerned, globbing is complicated enough for what I need, and full support for arbitrary regexes is YAGNI. -- Steven From ncoghlan at gmail.com Sat Oct 4 13:39:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 Oct 2014 21:39:07 +1000 Subject: [Python-ideas] Delegating `BoundArguments.__getitem__` to `BoundArguments.arguments.__getitem__` In-Reply-To: <542F165C.3060703@gmail.com> References: <542F165C.3060703@gmail.com> Message-ID: On 4 October 2014 07:34, Yury Selivanov wrote: > You'll also need to delegate __setitem__, keys(), __len__ and all other > Mapping methods. And to be honest, I see absolutely no point in doing this. I don't see a benefit either. Ram, please try to remember that shorter isn't better if it comes at a cost in clarity, and that *every single change to Python* comes at a substantial cost in terms of the long term ripple effect it has on the ecosystem. This is at the heart of the "one - and preferably only one - obvious way to do it" philosophy - avoiding redundancy wherever possible promotes consistency, which reduces the number of things that folks need to learn how to *read*. As a result, adding a redundant spelling of an existing feature has to offer an *extraordinarily* compelling readability benefit for it to ever be worthwhile. Saving a few characters when typing is almost never going to be sufficiently compelling. In this particular case, it's also worth noting that we explicitly moved *away* from a similar "implicit subcontainer access" model for exception arguments in order to eliminate the redundancy: # Python 2 >>> Exception(1, 2, 3).args[2] 3 >>> Exception(1, 2, 3)[2] 3 # Python 3 >>> Exception(1, 2, 3).args[2] 3 >>> Exception(1, 2, 3)[2] Traceback (most recent call last): File "", line 1, in TypeError: 'Exception' object is not subscriptable Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Oct 4 13:42:34 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 Oct 2014 21:42:34 +1000 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: On 4 October 2014 01:36, Mark Young wrote: > Why do you think len is an inherently O(1) operation? Sure, on array-based > things it is, but for an arbitrary collection, the only logical assertion is > that it's O(n). (For some containers, like arrays, it might be O(1) as well, > but you can't assume that in general) It's not a hard requirement (it can't be), but it *is* a rather strong convention that checking something's length should be an O(1) operation. A non-specialised container that doesn't implement len() as an O(1) operation in Python would quite likely end up with users asking for the length to be cached to make it O(1). In particular, it's generally expected that "if obj:" should be O(1), and that's likely to fall back to __len__() for most container types. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Oct 4 13:48:59 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 Oct 2014 21:48:59 +1000 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <542EC551.1080404@cathalgarvey.me> Message-ID: On 4 October 2014 02:45, Ian Cordasco wrote: > On Fri, Oct 3, 2014 at 11:32 AM, Alexander Belopolsky > wrote: >> You are correct about python-ideas, but range is not a generator in python 3 >> either: >> >> Python 3.4.0a0 (default:e9cecb612ff7+, Oct 3 2014, 12:07:42) >>>>> type(range(10)) # not a generator >> >>>>> type(x for x in range(10)) # a generator >> > > Ah yeah. All too often I conflate lazy iterators with generators. Nevermind Python 3 range isn't a lazy iterator either - it's a full sequence type (specifically, a calculated tuple: https://docs.python.org/3/library/stdtypes.html#ranges). The only immutable sequence operations it doesn't support are concatenation and repetition (since concatenating or repeating a range can't be represented using the simple "start + n*step" calculation that range uses internally). We'd have gotten a *lot* more complaints if we just dropped xrange in as a substitute for the list objects returned by Python 2's range builtin :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Sat Oct 4 14:39:30 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 4 Oct 2014 14:39:30 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <542EC551.1080404@cathalgarvey.me> Message-ID: <1D3C91B8-E830-41CE-BFBC-BD5078FA41CA@yahoo.com> On Oct 4, 2014, at 13:48, Nick Coghlan wrote: > On 4 October 2014 02:45, Ian Cordasco wrote: >> On Fri, Oct 3, 2014 at 11:32 AM, Alexander Belopolsky >> wrote: >>> You are correct about python-ideas, but range is not a generator in python 3 >>> either: >>> >>> Python 3.4.0a0 (default:e9cecb612ff7+, Oct 3 2014, 12:07:42) >>>>>> type(range(10)) # not a generator >>> >>>>>> type(x for x in range(10)) # a generator >>> >> >> Ah yeah. All too often I conflate lazy iterators with generators. Nevermind > > Python 3 range isn't a lazy iterator either - it's a full sequence > type (specifically, a calculated tuple: > https://docs.python.org/3/library/stdtypes.html#ranges). The only > immutable sequence operations it doesn't support are concatenation and > repetition (since concatenating or repeating a range can't be > represented using the simple "start + n*step" calculation that range > uses internally). > > We'd have gotten a *lot* more complaints if we just dropped xrange in > as a substitute for the list objects returned by Python 2's range > builtin :) For some reason, most of the 3.x early adopters understood this, but later switchers seem to often believe that this is exactly what Python 3 did. For example, at least half a dozen times, I've written an answer on StackOverflow showing someone that `n in range(start, stop)` is what they were trying to write, and explaining that it means the same thing as `start <= n < stop` (with various caveats about Python 2, and `step`, etc.) only for someone to edit or comment on my answer claiming that range.__contents__ has to be linear and should never be used. I explain that it's a sequence, and can be indexed, and show them a quick timing test, and they still insist that's just an accidental optimization that shouldn't be relied on, because it's really just meant to be Python 2's xrange. Once someone insisted that if you want this functionality you should use a multi-interval class off PyPI, even though the docs for that class explicitly say that a single interval is just an empty subclass of range in Py 3. People who stick with 2.x just really want to believe you got this wrong, I guess. I've seen similar things with, e.g., people not believing that dict.keys() returns a set that's a view into the dict's storage rather than some kind of magical iterator that knows how to be used multiple times. (I'm not even sure how that would work; how could __iter__ know when you were trying to get self and when you were trying to start over?) From steve at pearwood.info Sat Oct 4 15:27:41 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 4 Oct 2014 23:27:41 +1000 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: Message-ID: <20141004132741.GK19757@ando.pearwood.info> On Fri, Oct 03, 2014 at 05:09:20PM +0200, Thomas Chaumeny wrote: > Hi! > > I have just come across some code counting a generator comprehension > expression by doing len([foo for foo in bar if some_condition]) and I > realized it might be better if we could just use len(foo for foo in bar if > some_condition) as it would avoid a list allocation in memory. > > Another possibility is to write sum(1 for foo in bar if some_condition), > but that is not optimal either as it generates a lot of intermediate > additions which should not be needed. I don't understand this reasoning. Why do you think that they are unnecessary? I believe that, in the general case of an arbitrary generator expression, there are only two ways to tell what the length will be. The first is to produce a list (or other sequence), then return the length of the list: len([obj for obj in generator if condition]). The second is to count each item as it is produced, but without storing them all: sum(1 for obj in generator if condition). The first is optimized for readability, the second for memory. If I have missed a third way, please tell me. I don't believe that there is any other general way to work out the length of an arbitrary generator. (Apart from trivial, or obfuscated, variations on the above two, of course.) How would you tell what the length of this generator should be, apart from actually running it to exhaustion? def generator(): while True: if random.random() < 0.5: return yield "spam" Since there is no general way to know what the length of an arbitrary generator will be, it is better to be explicit that it has to be calculated by running through the generator and exhausting it. -- Steven From p.f.moore at gmail.com Sat Oct 4 16:23:53 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 4 Oct 2014 15:23:53 +0100 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: <20141004102609.GJ19757@ando.pearwood.info> References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> <20141004062136.GE19757@ando.pearwood.info> <20141004102609.GJ19757@ando.pearwood.info> Message-ID: On 4 October 2014 11:26, Steven D'Aprano wrote: > I don't see a use for supporting the full range of regexes. As far as I > am concerned, globbing is complicated enough for what I need, and full > support for arbitrary regexes is YAGNI. I don't know how unicodedata is implemented, but would it be practical to simply expose a function that iterates over every name in the database? Then you could simply do (name for name in unicodedata.names() if name.startswith(prefix)) Paul. From abarnert at yahoo.com Sat Oct 4 16:45:01 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 4 Oct 2014 16:45:01 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: <20141004132741.GK19757@ando.pearwood.info> References: <20141004132741.GK19757@ando.pearwood.info> Message-ID: On Oct 4, 2014, at 15:27, Steven D'Aprano wrote: > On Fri, Oct 03, 2014 at 05:09:20PM +0200, Thomas Chaumeny wrote: >> Hi! >> >> I have just come across some code counting a generator comprehension >> expression by doing len([foo for foo in bar if some_condition]) and I >> realized it might be better if we could just use len(foo for foo in bar if >> some_condition) as it would avoid a list allocation in memory. >> >> Another possibility is to write sum(1 for foo in bar if some_condition), >> but that is not optimal either as it generates a lot of intermediate >> additions which should not be needed. > > I don't understand this reasoning. Why do you think that they are > unnecessary? > > I believe that, in the general case of an arbitrary generator > expression, there are only two ways to tell what the length will be. The > first is to produce a list (or other sequence), then return the > length of the list: len([obj for obj in generator if condition]). The > second is to count each item as it is produced, but without storing them > all: sum(1 for obj in generator if condition). The first is optimized > for readability, the second for memory. Also more that the itertools recipes include a function for doing exactly this: def quantify(iterable, pred=bool): "Count how many times the predicate is true" return sum(map(pred, iterable)) The more-itertools library on PyPI has this, plus an even simpler `ilen` function that just does, IIRC, sum(1 for _ in iterable). Of course they're both trivial one-liners, but maybe calling more_itertools.ilen is a nice way to clarify that you're intentionally consuming an iterator just to get its length. > I don't believe that there is any other general way to work out the > length of an arbitrary generator. (Apart from trivial, or obfuscated, > variations on the above two, of course.) Would teeing the iterator and consuming the extra copy count as an obfuscated variation? In practice, it's kind of the worst of both worlds--you're constructing a sequence (a deque instead of a list) in memory, but can only access it as a one-shot iterator. Conceptually, it looks somewhat nice--you're copying the iterator to count it without destroying it--but I think that's only an illusion for those who don't understand lists as iterables. > How would you tell what the > length of this generator should be, apart from actually running it to > exhaustion? Obviously we just need to rewrite Python around lazy dataflow variables, so whatever you assign len(generator()) to doesn't consume the generator until necessary, meaning you could still use the iterator's vales elsewhere in the mean time. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Oct 4 16:52:37 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 4 Oct 2014 16:52:37 +0200 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> <20141004062136.GE19757@ando.pearwood.info> <20141004102609.GJ19757@ando.pearwood.info> Message-ID: <6ABE7EB6-53A9-4969-A950-29163B62D0A3@yahoo.com> On Oct 4, 2014, at 16:23, Paul Moore wrote: > On 4 October 2014 11:26, Steven D'Aprano wrote: >> I don't see a use for supporting the full range of regexes. As far as I >> am concerned, globbing is complicated enough for what I need, and full >> support for arbitrary regexes is YAGNI. > > I don't know how unicodedata is implemented, but would it be practical > to simply expose a function that iterates over every name in the > database? Then you could simply do > > (name for name in unicodedata.names() if name.startswith(prefix)) IIRC, the perl UCD CPAN package and the ruby unicodedata gem expose the name to code and code to name mappings as hashes. Doing the equivalent in Python would allow you to do anything you want (including exactly that same line of code). From t.chaumeny at gmail.com Sun Oct 5 01:28:33 2014 From: t.chaumeny at gmail.com (Thomas Chaumeny) Date: Sun, 5 Oct 2014 01:28:33 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: <20141004132741.GK19757@ando.pearwood.info> References: <20141004132741.GK19757@ando.pearwood.info> Message-ID: On Sat, Oct 4, 2014 at 3:27 PM, Steven D'Aprano wrote: > > I don't understand this reasoning. Why do you think that they are > unnecessary? Sorry, that was a bad formulation. Obvisouly you need to increment a counter at some point when iterating over the generator. Now, if you have a builtin function len (or other), you can define it to run a fast iterating loop using a C integer to maintain the number of elements during the loop. If you use sum(1 for a in ...), I suppose there is some overhead due do the fact that you have to deal with Python objects when doing the additions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Oct 5 02:37:01 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 4 Oct 2014 17:37:01 -0700 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <20141004132741.GK19757@ando.pearwood.info> Message-ID: Support for len() on iterators isn't going to happen. The "gut feeling" reason is that len() shouldn't "consume" anything. It would lure people into thinking they can first call len() on the iterator and then iterate over it -- while, if it is an actual iterator (like an I/O stream), after calling len(), everything is consumed. Yes, it's possible that you only want to count the number of lines, but that's unusual, and the idiom for that is readily available (e.g. sum(1 for ...)). If the idiom occurs frequently in your code you can define a helper function count(). The speed benefit of doing the loop in C would be minimal in most cases. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Oct 5 05:35:46 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 5 Oct 2014 13:35:46 +1000 Subject: [Python-ideas] O(1) containment testing with Py3 range objects (was Re: Make len() usable on a generator) Message-ID: On 4 October 2014 22:39, Andrew Barnert wrote: > On Oct 4, 2014, at 13:48, Nick Coghlan wrote: > >> On 4 October 2014 02:45, Ian Cordasco wrote: >>> On Fri, Oct 3, 2014 at 11:32 AM, Alexander Belopolsky >>> wrote: >>>> You are correct about python-ideas, but range is not a generator in python 3 >>>> either: >>>> >>>> Python 3.4.0a0 (default:e9cecb612ff7+, Oct 3 2014, 12:07:42) >>>>>>> type(range(10)) # not a generator >>>> >>>>>>> type(x for x in range(10)) # a generator >>>> >>> >>> Ah yeah. All too often I conflate lazy iterators with generators. Nevermind >> >> Python 3 range isn't a lazy iterator either - it's a full sequence >> type (specifically, a calculated tuple: >> https://docs.python.org/3/library/stdtypes.html#ranges). The only >> immutable sequence operations it doesn't support are concatenation and >> repetition (since concatenating or repeating a range can't be >> represented using the simple "start + n*step" calculation that range >> uses internally). >> >> We'd have gotten a *lot* more complaints if we just dropped xrange in >> as a substitute for the list objects returned by Python 2's range >> builtin :) > > For some reason, most of the 3.x early adopters understood this, but later switchers seem to often believe that this is exactly what Python 3 did. I've seen a few transition guides that oversimplify the situation as "Python 3 range is the same as Python 2 xrange". Those transition guides are wrong (xrange has many limitations compared to the more full featured Python 3 range implementation), but we don't have the capacity to chase down every occurrence and ask people to fix it. One nice trick with Py3 range for example, is to work with ranges that Py2 could never hope to handle: $ python3 -m timeit -s "seq = range(-(10**24), 10**24)" "0 in seq" 1000000 loops, best of 3: 0.295 usec per loop $ python -m timeit -s "seq = range(-(10**24), 10**24)" "0 in seq" Traceback (most recent call last): File "/usr/lib64/python2.7/timeit.py", line 300, in main x = t.timeit(number) File "/usr/lib64/python2.7/timeit.py", line 195, in timeit timing = self.inner(it, self.timer) File "", line 3, in inner seq = range(-(10**24), 10**24) OverflowError: range() result has too many items There's still an unfortunate technical limitation in Py3 where *len* is restricted to 64 bit integers, but start/stop/step are exposed on range objects these days, so you can do the calculation directly if you really need to. (That workaround severely reduces the motivation for anyone to put together a backwards compatible proposal for an alternative C level length API that uses a Python object rather than a C sssize_t value) > For example, at least half a dozen times, I've written an answer on StackOverflow showing someone that `n in range(start, stop)` is what they were trying to write, and explaining that it means the same thing as `start <= n < stop` (with various caveats about Python 2, and `step`, etc.) only for someone to edit or comment on my answer claiming that range.__contents__ has to be linear and should never be used. I explain that it's a sequence, and can be indexed, and show them a quick timing test, and they still insist that's just an accidental optimization that shouldn't be relied on, because it's really just meant to be Python 2's xrange. Once someone insisted that if you want this functionality you should use a multi-interval class off PyPI, even though the docs for that class explicitly say that a single interval is just an empty subclass of range in Py 3. People who stick with 2.x just really want to believe you got this wrong, I guess. Containment tests on range() are guaranteed to be O(1) when suing CPython to test against true int and bool objects. Other implementations may not offer the same guarantee (although it's a simple enough optimisation that I believe they should). CPython itself will currently fall back to O(n) behaviour for int subclasses and other objects to handle backwards compatibility issues in relation to custom __eq__ methods (see http://bugs.python.org/issue1766304 for history and details - note that this *was* under consideration for 2.7 xrange as well, and merely didn't make the release deadline). For those that haven't seen the kind of timing examples Andrew mentioned, one easy way to see the difference is by comparing integer lookup performance with a numeric type that isn't an integer at all: $ python3 -m timeit -s "seq = range(10**6)" "10e5 in seq" 10 loops, best of 3: 118 msec per loop $ python3 -m timeit -s "seq = range(10**6)" "int(10e5) in seq" 1000000 loops, best of 3: 0.348 usec per loop You cop a small constant-time hit for the float->int conversion, but gain back 6 orders of magnitude (in this particular example) through the switch to an O(1) containment check. This optimisation is particularly useful in cases that can't easily be converted to comparison operations, such as when the range has a step magnitude other than 1. I thought I had an open issue proposing to expand the O(1) optimisation to any type that implements index (as well as making it a documented guarantee all implementations are expected to provide), but it looks like I may have only thought about filing that rather than actually doing it. (Or perhaps it's hiding in the python-ideas archives somewhere) > I've seen similar things with, e.g., people not believing that dict.keys() returns a set that's a view into the dict's storage rather than some kind of magical iterator that knows how to be used multiple times. (I'm not even sure how that would work; how could __iter__ know when you were trying to get self and when you were trying to start over?) That at least has an authoritative reference in PEP 3106 (along with the standard library docs). Unfortunately, the more convinced someone is that they already understand a change, the less likely they are to read the references to the relevant design discussions and decisions :P Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Sun Oct 5 15:04:26 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 05 Oct 2014 16:04:26 +0300 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> <20141004062136.GE19757@ando.pearwood.info> <20141004102609.GJ19757@ando.pearwood.info> Message-ID: > I don't know how unicodedata is implemented, but would it be practical > to simply expose a function that iterates over every name in the > database? Then you could simply do > > (name for name in unicodedata.names() if name.startswith(prefix)) (unicodedata.name(c, '') for c in map(chr, range(sys.maxunicode)) if unicodedata.name(c, '').startswith(prefix)) From mistersheik at gmail.com Sun Oct 5 19:27:16 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 5 Oct 2014 10:27:16 -0700 (PDT) Subject: [Python-ideas] Add __parent__ to all classes, functions, and modules Message-ID: Many classes, functions, and modules are defined within the context of another class, function, or module thereby forming a mathematical forest of declarations. It is possible to walk the descendants using __dict__ (for classes and modules), but not the ancestors. I propose adding __parent__ that would be filled at the same time that __qualname__ is filled in. One concrete use case for __parent__ is allowing a method decorator to call super, which is currently impossible because the class in which the method has been defined is not available to the decorator. This way, one could write: def ensure_finished(iterator): try: next(iterator) except StopIteration: return else: raise RuntimeError def derived_generator(method): def new_method(self, *args, **kwargs): x = method(self, *args, **kwargs) y = getattr(super(method.__parent__, self), method.__name__)\ (*args, **kwargs) for a, b in zip(x, y): assert a is None and b is None yield ensure_finished(x) ensure_finished(y) This is currently impossible to implement without restating the class name every time the decorator is used as far as I know. Best, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sun Oct 5 20:09:15 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 5 Oct 2014 18:09:15 +0000 (UTC) Subject: [Python-ideas] =?utf-8?q?Add_=5F=5Fparent=5F=5F_to_all_classes=2C?= =?utf-8?q?_functions=2C_and_modules?= References: Message-ID: Neil Girdhar writes: > > Many classes, functions, and modules are defined within the context of another class, function, or module thereby forming a mathematical forest of declarations. ?It is possible to walk the descendants using __dict__ (for classes and modules), but not the ancestors. ?I propose adding __parent__ that would be filled at the same time that __qualname__ is filled in. This is unlikely to work. 1) It turns basically everything into a cycle. 2) __qualname__ is determined strictly from syntax, whereas __parent__ could not be. For example, what happens if I take a method from one class and set it on another? __parent__ would not be well-defined. Regards, Benjamin From ram at rachum.com Sun Oct 5 20:40:47 2014 From: ram at rachum.com (Ram Rachum) Date: Sun, 5 Oct 2014 21:40:47 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) Message-ID: Hi guys, I was just going to test whether a mapping is dict-like or an OrderedDict-like object by doing this: isinstance(x, collections.Sequence) But then I checked and saw that this is False: issubclass(collections.OrderedDict, collections.Sequence) is False Why isn't an OrderedDict a Sequence? -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sun Oct 5 21:52:11 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 05 Oct 2014 22:52:11 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: > Why isn't an OrderedDict a Sequence? http://docs.python.org/3/glossary.html#term-sequence From mistersheik at gmail.com Sun Oct 5 22:51:44 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 5 Oct 2014 16:51:44 -0400 Subject: [Python-ideas] Add __parent__ to all classes, functions, and modules In-Reply-To: References: Message-ID: On Sun, Oct 5, 2014 at 2:09 PM, Benjamin Peterson wrote: > Neil Girdhar writes: > > > > > Many classes, functions, and modules are defined within the context of > another class, function, or module thereby forming a mathematical forest of > declarations. It is possible to walk the descendants using __dict__ (for > classes and modules), but not the ancestors. I propose adding __parent__ > that would be filled at the same time that __qualname__ is filled in. > > This is unlikely to work. > > 1) It turns basically everything into a cycle. > Why a cycle? 2) __qualname__ is determined strictly from syntax, whereas __parent__ could > not be. For example, what happens if I take a method from one class and set > it on another? __parent__ would not be well-defined. > I'm suggesting that parent be determined purely from declaration. If you copy something, neither qualname nor parent would change unless you change them. > Regards, > Benjamin > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/94fTkAkjhCo/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Sun Oct 5 22:56:40 2014 From: ram at rachum.com (Ram Rachum) Date: Sun, 5 Oct 2014 23:56:40 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: Ok, makes sense. Though it still would be nice to have a general way to check whether a collection is ordered or not, regardless of index access. Perhaps a `collections.Ordered`? On Sun, Oct 5, 2014 at 11:53 PM, Guido van Rossum wrote: > On Sun, Oct 5, 2014 at 11:40 AM, Ram Rachum wrote: > >> I was just going to test whether a mapping is dict-like or an >> OrderedDict-like object by doing this: >> >> isinstance(x, collections.Sequence) >> >> But then I checked and saw that this is False: >> >> issubclass(collections.OrderedDict, collections.Sequence) is False >> >> Why isn't an OrderedDict a Sequence? >> > > Check the interface of Sequence. It requires indexing behavior that > OrdedDict doesn't have -- a[i] is a key lookup in the underlying dict, not > an indexed lookup in the ordered list of elements. I don't think it would > be wise to attempt to add that behavior either (given that the keys may > well be integers). > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Oct 5 22:53:10 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 5 Oct 2014 13:53:10 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: On Sun, Oct 5, 2014 at 11:40 AM, Ram Rachum wrote: > I was just going to test whether a mapping is dict-like or an > OrderedDict-like object by doing this: > > isinstance(x, collections.Sequence) > > But then I checked and saw that this is False: > > issubclass(collections.OrderedDict, collections.Sequence) is False > > Why isn't an OrderedDict a Sequence? > Check the interface of Sequence. It requires indexing behavior that OrdedDict doesn't have -- a[i] is a key lookup in the underlying dict, not an indexed lookup in the ordered list of elements. I don't think it would be wise to attempt to add that behavior either (given that the keys may well be integers). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Sun Oct 5 23:11:57 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 5 Oct 2014 21:11:57 +0000 (UTC) Subject: [Python-ideas] =?utf-8?q?Add_=5F=5Fparent=5F=5F_to_all_classes=2C?= =?utf-8?q?_functions=2C=09and_modules?= References: Message-ID: Neil Girdhar writes: > > > > On Sun, Oct 5, 2014 at 2:09 PM, Benjamin Peterson python.org> wrote: > Neil Girdhar ...> writes: > > > > Many classes, functions, and modules are defined within the context of > another class, function, or module thereby forming a mathematical forest of > declarations.? It is possible to walk the descendants using __dict__ (for > classes and modules), but not the ancestors.? I propose adding __parent__ > that would be filled at the same time that __qualname__ is filled in.This is unlikely to work. > 1) It turns basically everything into a cycle. > > > Why a cycle?? Because, for example, the class would reference methods, which would reference the class. > > > 2) __qualname__ is determined strictly from syntax, whereas __parent__ could > not be. For example, what happens if I take a method from one class and set > it on another? __parent__ would not be well-defined. > > > I'm suggesting that parent be determined purely from declaration.? If you copy something, neither qualname nor parent would change unless you change them.? It would mean your trick with super would only work for some methods. From mistersheik at gmail.com Sun Oct 5 23:18:51 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 5 Oct 2014 17:18:51 -0400 Subject: [Python-ideas] Add __parent__ to all classes, functions, and modules In-Reply-To: References: Message-ID: On Sun, Oct 5, 2014 at 5:11 PM, Benjamin Peterson wrote: > Neil Girdhar writes: > > > > > > > > > On Sun, Oct 5, 2014 at 2:09 PM, Benjamin Peterson > python.org> wrote: > > Neil Girdhar ...> writes: > > > > > > Many classes, functions, and modules are defined within the context of > > another class, function, or module thereby forming a mathematical forest > of > > declarations. It is possible to walk the descendants using __dict__ (for > > classes and modules), but not the ancestors. I propose adding __parent__ > > that would be filled at the same time that __qualname__ is filled in.This > is unlikely to work. > > 1) It turns basically everything into a cycle. > > > > > > Why a cycle? > > Because, for example, the class would reference methods, which would > reference the class. > Right. I don't see what's wrong with that. This already happens with the __module__ member and the module's immediate children. > > > > > > > 2) __qualname__ is determined strictly from syntax, whereas __parent__ > could > > not be. For example, what happens if I take a method from one class and > set > > it on another? __parent__ would not be well-defined. > > > > > > I'm suggesting that parent be determined purely from declaration. If you > copy something, neither qualname nor parent would change unless you change > them. > > It would mean your trick with super would only work for some methods. > It would only work for that are decorated in the usual way using @blah on methods defined in the class. If someone wants to copy that method somewhere else then they'll have to update __parent__ themselves. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/94fTkAkjhCo/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Oct 5 23:43:13 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 5 Oct 2014 14:43:13 -0700 Subject: [Python-ideas] Add __parent__ to all classes, functions, and modules In-Reply-To: References: Message-ID: This is a clever idea but I doubt it's worth adding. The pattern proposed in the example in the first post doesn't strike me as very readable -- though you don't give any examples of *using* the decorator from the example, so it's a little difficult to imagine what the use case would be. Adding __parent__ would have to happen as an extra pass once the class is defined -- the decorator (at least its outer part) runs at the time that the method is being defined as a plain function object, and at that time the class object hasn't been created yet. This also means that some decorators can't use the __parent__ attribute (because the outer function of the decorator runs before the object to be stored in __parent__ has been created). If you really want to explore this idea further, you can probably define a metaclass that adds a __parent__ attribute to the function objects in the class __dict__ when the class is being created. Then you can experiment with using the pattern in some real code. Please report back here with some examples where the presence of __parent__ lets you write cleaner code. An alternative that wouldn't even require a metaclass would be to write a helper function that looks the class object up by name after parsing __qualname__. There are some limitations to this, e.g. classes defined inside functions -- but that's already a suspect pattern anyway. On Sun, Oct 5, 2014 at 2:18 PM, Neil Girdhar wrote: > > > On Sun, Oct 5, 2014 at 5:11 PM, Benjamin Peterson > wrote: > >> Neil Girdhar writes: >> >> > >> > >> > >> > On Sun, Oct 5, 2014 at 2:09 PM, Benjamin Peterson >> python.org> wrote: >> > Neil Girdhar ...> writes: >> > > >> > > Many classes, functions, and modules are defined within the context of >> > another class, function, or module thereby forming a mathematical >> forest of >> > declarations. It is possible to walk the descendants using __dict__ >> (for >> > classes and modules), but not the ancestors. I propose adding >> __parent__ >> > that would be filled at the same time that __qualname__ is filled >> in.This >> is unlikely to work. >> > 1) It turns basically everything into a cycle. >> > >> > >> > Why a cycle? >> >> Because, for example, the class would reference methods, which would >> reference the class. >> > > Right. I don't see what's wrong with that. This already happens with the > __module__ member and the module's immediate children. > >> >> > >> > >> > 2) __qualname__ is determined strictly from syntax, whereas __parent__ >> could >> > not be. For example, what happens if I take a method from one class and >> set >> > it on another? __parent__ would not be well-defined. >> > >> > >> > I'm suggesting that parent be determined purely from declaration. If >> you >> copy something, neither qualname nor parent would change unless you change >> them. >> >> It would mean your trick with super would only work for some methods. >> > > It would only work for that are decorated in the usual way using @blah on > methods defined in the class. If someone wants to copy that method > somewhere else then they'll have to update __parent__ themselves. > >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "python-ideas" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/python-ideas/94fTkAkjhCo/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> python-ideas+unsubscribe at googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Oct 5 23:53:35 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 5 Oct 2014 14:53:35 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: Can I ask you to mathematically define the property you are looking for here? In some other languages "Ordered" refers to being sorted or sortable, but the use of this word in OrderedDict does not refer to that at all. I tried coming up with a definition, but I got stuck with the rather vague "insertion order is preserved". Unfortunately, for the other collection types that (presumably) would qualify for this predicate, like list and tuple, the "insertion" operation is spelled differently than for OrderedDict -- OrderedDict's rule seems to be "new elements inserted with __setitem__ are appended to the end of the underlying sequence", while for list you'd have to write something about the insert() and append() methods (plus other operations like slice assignment and extend() that can be reduced to these), and for tuple you'd have to refer to the constructor. I don't think that definition is sufficiently crisp and reusable to warrant defining a collection ABC for it. But perhaps you can come up with a better specification for your proposed collections.Ordered ABC? On Sun, Oct 5, 2014 at 1:56 PM, Ram Rachum wrote: > Ok, makes sense. Though it still would be nice to have a general way to > check whether a collection is ordered or not, regardless of index access. > Perhaps a `collections.Ordered`? > > On Sun, Oct 5, 2014 at 11:53 PM, Guido van Rossum > wrote: > >> On Sun, Oct 5, 2014 at 11:40 AM, Ram Rachum wrote: >> >>> I was just going to test whether a mapping is dict-like or an >>> OrderedDict-like object by doing this: >>> >>> isinstance(x, collections.Sequence) >>> >>> But then I checked and saw that this is False: >>> >>> issubclass(collections.OrderedDict, collections.Sequence) is False >>> >>> Why isn't an OrderedDict a Sequence? >>> >> >> Check the interface of Sequence. It requires indexing behavior that >> OrdedDict doesn't have -- a[i] is a key lookup in the underlying dict, not >> an indexed lookup in the ordered list of elements. I don't think it would >> be wise to attempt to add that behavior either (given that the keys may >> well be integers). >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From edk141 at gmail.com Mon Oct 6 00:05:12 2014 From: edk141 at gmail.com (Ed Kellett) Date: Sun, 5 Oct 2014 23:05:12 +0100 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: I think the definition of ordered here would be that the container's iteration order carries any meaning (compare: dicts and sets' having iteration order is an incidental effect of one thing having to be iterated out first, and one shouldn't depend on that order; lists and ordereddicts' having iteration order is deliberate). I don't think there's a mathematical definition possible - it's just a question of whether whoever wrote the class left the iteration order documented or unspecified. edk From ram at rachum.com Mon Oct 6 00:07:59 2014 From: ram at rachum.com (Ram Rachum) Date: Mon, 6 Oct 2014 01:07:59 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: Yep, Ed said exactly what I meant. On Oct 6, 2014 1:05 AM, "Ed Kellett" wrote: > I think the definition of ordered here would be that the container's > iteration order carries any meaning (compare: dicts and sets' having > iteration order is an incidental effect of one thing having to be > iterated out first, and one shouldn't depend on that order; lists and > ordereddicts' having iteration order is deliberate). > > I don't think there's a mathematical definition possible - it's just a > question of whether whoever wrote the class left the iteration order > documented or unspecified. > > > edk > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zuo at kaliszewski.net Mon Oct 6 00:42:07 2014 From: zuo at kaliszewski.net (Jan Kaliszewski) Date: Mon, 6 Oct 2014 00:42:07 +0200 Subject: [Python-ideas] collections.abc.BytesLike or ...? [was: [Python-Dev] bytes-like objects] In-Reply-To: References: <20141005161158.9A52F250069@webabinitio.net> Message-ID: <20141006004207.06fdd3cd@grzmot> 2014-10-05 Victor Stinner dixit: > I prefer "bytes-like" than "buffer protocol". By the way, is there a > documentation in Python doc which explains "bytes-like" and maybe > list most compatible types? > > I'm not sure that the term has an unique definition. In some parts of > Python, I saw explicit checks on the type: bytes or bytearray, > sometimes memoryview is accepted. The behaviour is different in C > functions using PyArg API. It probably depends if the function relies > on the content (read bytes) or on methods (ex: call .find). Maybe an ABC would be useful for isinstance checks? Or some kind of almost-an-ABC-which-does-not-allow-to-register- -arbitrary-virtual-subclasses? Cheers. *j From guido at python.org Mon Oct 6 01:15:45 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 5 Oct 2014 16:15:45 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: OK, so collections.Ordered would be a subclass of Iterable that adds no new methods but makes a promise about the author's intent. But what kind of use could a program make of this? The only things I can think of would have to be combined with some other specific container type that isn't always ordered (Sequence is ordered, Set and Mapping aren't). I can think of plenty of situations where it would be useful to say "use an OrderedDict, otherwise X will happen", but I can't think of a case where I could say "use any container type you like as long as it is Ordered". What was your use case? Do you have one? Or are you just interested in asking questions? On Sun, Oct 5, 2014 at 3:07 PM, Ram Rachum wrote: > Yep, Ed said exactly what I meant. > On Oct 6, 2014 1:05 AM, "Ed Kellett" wrote: > >> I think the definition of ordered here would be that the container's >> iteration order carries any meaning (compare: dicts and sets' having >> iteration order is an incidental effect of one thing having to be >> iterated out first, and one shouldn't depend on that order; lists and >> ordereddicts' having iteration order is deliberate). >> >> I don't think there's a mathematical definition possible - it's just a >> question of whether whoever wrote the class left the iteration order >> documented or unspecified. >> >> >> edk >> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Mon Oct 6 03:26:01 2014 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 5 Oct 2014 18:26:01 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: <945D8D7D-7F43-488D-AD34-3117BB369B41@gmail.com> On Oct 5, 2014, at 11:40 AM, Ram Rachum wrote: > Why isn't an OrderedDict a Sequence? Because wasn't designed that way (it doesn't have or need count(), index(), slicing, etc. Nor is it designed in a way that makes any of those operations efficient). If you want a list, it is simple to make one: s = list(od). Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Oct 6 04:21:32 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 5 Oct 2014 19:21:32 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: <29983D67-3EA9-416A-BEA5-42DB43C3DFED@yahoo.com> On Oct 5, 2014, at 15:05, Ed Kellett wrote: > I think the definition of ordered here would be that the container's > iteration order carries any meaning (compare: dicts and sets' having > iteration order is an incidental effect of one thing having to be > iterated out first, and one shouldn't depend on that order; lists and > ordereddicts' having iteration order is deliberate). So this also implies that your favorite sorted dict class would also be Ordered. And that all of the views of OrderedDict (and SortedDict and rbtree and so on) would also be Ordered. But I'm not sure what exactly you can do with that either... (Having the values view of an ordered or sorted dict be a Sequence, I can sort of imagine a use case for, but just an Ordered Container MappingView, I don't know.) From steve at pearwood.info Mon Oct 6 04:36:29 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 6 Oct 2014 13:36:29 +1100 Subject: [Python-ideas] Classes inside functions [was Re: Add __parent__ to all classes, functions, and modules] In-Reply-To: References: Message-ID: <20141006023628.GN19757@ando.pearwood.info> On Sun, Oct 05, 2014 at 02:43:13PM -0700, Guido van Rossum wrote: > An alternative that wouldn't even require a metaclass would be to write a > helper function that looks the class object up by name after parsing > __qualname__. There are some limitations to this, e.g. classes defined > inside functions -- but that's already a suspect pattern anyway. I understand that "suspect pattern" doesn't mean "don't ever do this", but I wonder what aspect of defining classes inside a function is suspect. I think the suspect part is that each invocation of the function creates a new class which (at least in the naive case) has the same name, same functionality, and looks identical to the casual glance but actually isn't. So given: def maker(): class MyClass: pass return MyClass x = maker()() y = maker()() assert isinstance(x, type(y)) assert isinstance(y, type(x)) both assertions will fail, which may confuse some people. There's probably also implications for pickling as well. Are these the only "suspect" parts of defining classes inside functions, or have I missed something else? -- Steven From mistersheik at gmail.com Mon Oct 6 04:51:07 2014 From: mistersheik at gmail.com (Neil Girdhar) Date: Sun, 5 Oct 2014 19:51:07 -0700 (PDT) Subject: [Python-ideas] Add __parent__ to all classes, functions, and modules In-Reply-To: References: Message-ID: On Sunday, October 5, 2014 5:43:33 PM UTC-4, Guido van Rossum wrote: > > This is a clever idea but I doubt it's worth adding. > > The pattern proposed in the example in the first post doesn't strike me as > very readable -- though you don't give any examples of *using* the > decorator from the example, so it's a little difficult to imagine what the > use case would be. > The idea is that I have a caller who is making changes to a network of nodes in steps. The caller tells the link to do phase 1, then makes some changes, then tells the link to do phase 2, then makes some changes, etc. Each time, the link can memoize some values from the network, send signals, etc. Therefore, I did not want to have the phases be different methods since the memos would then have to be stored on the object. Instead the phases are implemented using the generator pattern (that is, each phase is separated by yield) and the caller simply iterates over the generator as it makes its changes. The problem is that in addition to what is implemented by the method, the method wants to also delegate to super. So we need to iterate over the generator returned by the superclass. This pattern is easy to follow, but it would be nice to wrap that up in a decorator so that the method-writer can forget about this detail. > Adding __parent__ would have to happen as an extra pass once the class is > defined -- the decorator (at least its outer part) runs at the time that > the method is being defined as a plain function object, and at that time > the class object hasn't been created yet. This also means that some > decorators can't use the __parent__ attribute (because the outer function > of the decorator runs before the object to be stored in __parent__ has been > created). > Couldn't __parent__ be filled in when the class is complete by the default metaclass (is that just "type"?) The generator doesn't need to actually do anything with the parent member. The lookup is happening in the created method, so I think that's at call time, at which point the attribute will be filled in. I could have done the same thing with qualname, right? > > If you really want to explore this idea further, you can probably define a > metaclass that adds a __parent__ attribute to the function objects in the > class __dict__ when the class is being created. Then you can experiment > with using the pattern in some real code. Please report back here with some > examples where the presence of __parent__ lets you write cleaner code. > The problem is that such a metaclass would add the member to the decorated method, and unfortunately, methods and functions don't have access to themselves and their attribtes at runtime as far as I know. So I'd have to look inside the members and add the __parent__ attribute to "method" attributes of the decorated methods, which is a bit weird. > > An alternative that wouldn't even require a metaclass would be to write a > helper function that looks the class object up by name after parsing > __qualname__. There are some limitations to this, e.g. classes defined > inside functions -- but that's already a suspect pattern anyway. > Yes, that works, but it really is very ugly to parse a string and then eval it. > > > On Sun, Oct 5, 2014 at 2:18 PM, Neil Girdhar > wrote: > >> >> >> On Sun, Oct 5, 2014 at 5:11 PM, Benjamin Peterson > > wrote: >> >>> Neil Girdhar writes: >>> >>> > >>> > >>> > >>> > On Sun, Oct 5, 2014 at 2:09 PM, Benjamin Peterson >>> python.org> wrote: >>> > Neil Girdhar ...> writes: >>> > > >>> > > Many classes, functions, and modules are defined within the context >>> of >>> > another class, function, or module thereby forming a mathematical >>> forest of >>> > declarations. It is possible to walk the descendants using __dict__ >>> (for >>> > classes and modules), but not the ancestors. I propose adding >>> __parent__ >>> > that would be filled at the same time that __qualname__ is filled >>> in.This >>> is unlikely to work. >>> > 1) It turns basically everything into a cycle. >>> > >>> > >>> > Why a cycle? >>> >>> Because, for example, the class would reference methods, which would >>> reference the class. >>> >> >> Right. I don't see what's wrong with that. This already happens with >> the __module__ member and the module's immediate children. >> >>> >>> > >>> > >>> > 2) __qualname__ is determined strictly from syntax, whereas __parent__ >>> could >>> > not be. For example, what happens if I take a method from one class >>> and set >>> > it on another? __parent__ would not be well-defined. >>> > >>> > >>> > I'm suggesting that parent be determined purely from declaration. If >>> you >>> copy something, neither qualname nor parent would change unless you >>> change >>> them. >>> >>> It would mean your trick with super would only work for some methods. >>> >> >> It would only work for that are decorated in the usual way using @blah on >> methods defined in the class. If someone wants to copy that method >> somewhere else then they'll have to update __parent__ themselves. >> >>> >>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python... at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "python-ideas" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/python-ideas/94fTkAkjhCo/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> python-ideas... at googlegroups.com . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Oct 6 05:04:25 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 5 Oct 2014 20:04:25 -0700 Subject: [Python-ideas] Classes inside functions [was Re: Add __parent__ to all classes, functions, and modules] In-Reply-To: <20141006023628.GN19757@ando.pearwood.info> References: <20141006023628.GN19757@ando.pearwood.info> Message-ID: On Sun, Oct 5, 2014 at 7:36 PM, Steven D'Aprano wrote: > On Sun, Oct 05, 2014 at 02:43:13PM -0700, Guido van Rossum wrote: > > > An alternative that wouldn't even require a metaclass would be to write a > > helper function that looks the class object up by name after parsing > > __qualname__. There are some limitations to this, e.g. classes defined > > inside functions -- but that's already a suspect pattern anyway. > > I understand that "suspect pattern" doesn't mean "don't ever do this", > but I wonder what aspect of defining classes inside a function is > suspect. I think the suspect part is that each invocation of the > function creates a new class which (at least in the naive case) has the > same name, same functionality, and looks identical to the casual > glance but actually isn't. So given: > > def maker(): > class MyClass: pass > return MyClass > > x = maker()() > y = maker()() > > assert isinstance(x, type(y)) > assert isinstance(y, type(x)) > > > both assertions will fail, which may confuse some people. There's > probably also implications for pickling as well. Are these the only > "suspect" parts of defining classes inside functions, or have I missed > something else? > Those (and their variations, e.g. using these in except clauses) are the main concerns; it also generally bothers me when something "smallish" like a function contains something "larger" like a class, especially when the class is substantial. It's another case of "flat is better than nested". The only non-evil use of this pattern that I can recall is in unit tests -- even when it is possible (as it usually is) to move the class definition to a spot outside the test method, that doesn't make the test code any clearer. Such classes are usually very small (e.g. overriding just one or two methods) and are only used within the unit test. (That latter property is really the saving grace.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Oct 6 05:14:17 2014 From: guido at python.org (Guido van Rossum) Date: Sun, 5 Oct 2014 20:14:17 -0700 Subject: [Python-ideas] Add __parent__ to all classes, functions, and modules In-Reply-To: References: Message-ID: On Sun, Oct 5, 2014 at 7:51 PM, Neil Girdhar wrote: > > > On Sunday, October 5, 2014 5:43:33 PM UTC-4, Guido van Rossum wrote: >> >> This is a clever idea but I doubt it's worth adding. >> >> The pattern proposed in the example in the first post doesn't strike me >> as very readable -- though you don't give any examples of *using* the >> decorator from the example, so it's a little difficult to imagine what the >> use case would be. >> > > The idea is that I have a caller who is making changes to a network of > nodes in steps. The caller tells the link to do phase 1, then makes some > changes, then tells the link to do phase 2, then makes some changes, etc. > Each time, the link can memoize some values from the network, send signals, > etc. Therefore, I did not want to have the phases be different methods > since the memos would then have to be stored on the object. Instead the > phases are implemented using the generator pattern (that is, each phase is > separated by yield) and the caller simply iterates over the generator as it > makes its changes. The problem is that in addition to what is implemented > by the method, the method wants to also delegate to super. So we need to > iterate over the generator returned by the superclass. This pattern is > easy to follow, but it would be nice to wrap that up in a decorator so that > the method-writer can forget about this detail. > I'm sorry, you'd have to point to the actual code if you want anyone to make sense of that description. You are combining so many abstractions in one paragraph that it feels like there must be lots of entirely different ways to map the ideas to code, and you may well just have picked the wrong way. > Adding __parent__ would have to happen as an extra pass once the class is >> defined -- the decorator (at least its outer part) runs at the time that >> the method is being defined as a plain function object, and at that time >> the class object hasn't been created yet. This also means that some >> decorators can't use the __parent__ attribute (because the outer function >> of the decorator runs before the object to be stored in __parent__ has been >> created). >> > > Couldn't __parent__ be filled in when the class is complete by the default > metaclass (is that just "type"?) > I think that's what I was saying. :-) > The generator doesn't need to actually do anything with the parent > member. The lookup is happening in the created method, so I think that's > at call time, at which point the attribute will be filled in. I could have > done the same thing with qualname, right? > Eh? Anyway, __qualname__ does not have this problem -- the class *name* is known by the time we get to 'def'. > If you really want to explore this idea further, you can probably define a >> metaclass that adds a __parent__ attribute to the function objects in the >> class __dict__ when the class is being created. Then you can experiment >> with using the pattern in some real code. Please report back here with some >> examples where the presence of __parent__ lets you write cleaner code. >> > > The problem is that such a metaclass would add the member to the decorated > method, and unfortunately, methods and functions don't have access to > themselves and their attribtes at runtime as far as I know. So I'd have to > look inside the members and add the __parent__ attribute to "method" > attributes of the decorated methods, which is a bit weird. > That's a fair point. But the default metaclass (i.e. type) would encounter the same problem. So how *do* you imagine this should work? The fact remains that the class isn't constructed until all the method definitions have been executed, and the namespace in which the methods are defined is what's passed to type() to construct the class object (with the class name and the list of base classes). > An alternative that wouldn't even require a metaclass would be to write a >> helper function that looks the class object up by name after parsing >> __qualname__. There are some limitations to this, e.g. classes defined >> inside functions -- but that's already a suspect pattern anyway. >> > > Yes, that works, but it really is very ugly to parse a string and then > eval it. > Which you should take as a hint that your problem is looking for a different solution. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Mon Oct 6 10:10:50 2014 From: ram at rachum.com (Ram Rachum) Date: Mon, 6 Oct 2014 11:10:50 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: Here are a couple: - I'm making a combinatorics package, and a combination space needs to have a __contains__ method that takes a combination and returns whether it's part of a set. Since a combination, unlike a permutation, has no order, I like to have my combinations be canonicalized in a sorted order. For example, in a combination space of 3 items on range(4), (0, 1, 3) would be a combination in that space, but (3, 1, 0) would not because it's not sorted. (It's a k-permutation but not a combination.) However, if the user does `{0, 1, 3} in comb_space` I still want to return True, regardless of whether the set iterator happens to give these items in order or not. - For the same package, I'm defining `Tally` and `OrderedTally` classes. (Simliar to `Counter` except without having an identity crisis between being a dict subclass and counter; mine are strictly counters.) In the tests I want to easily see whether the class I'm testing is the ordered one or not, so I'll know to run appropriate tests. (There are also `FrozenTally` and `FrozenOrderedTally` so it's not just one class.) I could set `is_ordered = True` on them or give them some base class, but I think a general `collections.Ordered` abstract base class would be the best solution. On Mon, Oct 6, 2014 at 2:15 AM, Guido van Rossum wrote: > OK, so collections.Ordered would be a subclass of Iterable that adds no > new methods but makes a promise about the author's intent. > > But what kind of use could a program make of this? The only things I can > think of would have to be combined with some other specific container type > that isn't always ordered (Sequence is ordered, Set and Mapping aren't). I > can think of plenty of situations where it would be useful to say "use an > OrderedDict, otherwise X will happen", but I can't think of a case where I > could say "use any container type you like as long as it is Ordered". > > What was your use case? Do you have one? Or are you just interested in > asking questions? > > On Sun, Oct 5, 2014 at 3:07 PM, Ram Rachum wrote: > >> Yep, Ed said exactly what I meant. >> On Oct 6, 2014 1:05 AM, "Ed Kellett" wrote: >> >>> I think the definition of ordered here would be that the container's >>> iteration order carries any meaning (compare: dicts and sets' having >>> iteration order is an incidental effect of one thing having to be >>> iterated out first, and one shouldn't depend on that order; lists and >>> ordereddicts' having iteration order is deliberate). >>> >>> I don't think there's a mathematical definition possible - it's just a >>> question of whether whoever wrote the class left the iteration order >>> documented or unspecified. >>> >>> >>> edk >>> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon Oct 6 14:02:03 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 06 Oct 2014 15:02:03 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: On 06.10.14 02:15, Guido van Rossum wrote: > But what kind of use could a program make of this? The only things I can > think of would have to be combined with some other specific container > type that isn't always ordered (Sequence is ordered, Set and Mapping > aren't). I can think of plenty of situations where it would be useful to > say "use an OrderedDict, otherwise X will happen", but I can't think of > a case where I could say "use any container type you like as long as it > is Ordered". > > What was your use case? Do you have one? Or are you just interested in > asking questions? There is one use case. When output a set or mapping in human readable way, it is handy to sort them by elements or keys. Except when it is a sequence or OrderedDict. This is very specific use case however. From storchaka at gmail.com Mon Oct 6 14:30:33 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 06 Oct 2014 15:30:33 +0300 Subject: [Python-ideas] collections.abc.BytesLike or ...? [was: [Python-Dev] bytes-like objects] In-Reply-To: <20141006004207.06fdd3cd@grzmot> References: <20141005161158.9A52F250069@webabinitio.net> <20141006004207.06fdd3cd@grzmot> Message-ID: On 06.10.14 01:42, Jan Kaliszewski wrote: > 2014-10-05 Victor Stinner dixit: >> I prefer "bytes-like" than "buffer protocol". By the way, is there a >> documentation in Python doc which explains "bytes-like" and maybe >> list most compatible types? >> >> I'm not sure that the term has an unique definition. In some parts of >> Python, I saw explicit checks on the type: bytes or bytearray, >> sometimes memoryview is accepted. The behaviour is different in C >> functions using PyArg API. It probably depends if the function relies >> on the content (read bytes) or on methods (ex: call .find). > > Maybe an ABC would be useful for isinstance checks? > > Or some kind of almost-an-ABC-which-does-not-allow-to-register- > -arbitrary-virtual-subclasses? Just use an object in functions which requires bytes-like argument (e.g. memoryview or two-arguments str). From abarnert at yahoo.com Mon Oct 6 17:07:13 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 6 Oct 2014 08:07:13 -0700 Subject: [Python-ideas] Classes inside functions [was Re: Add __parent__ to all classes, functions, and modules] In-Reply-To: References: <20141006023628.GN19757@ando.pearwood.info> Message-ID: On Oct 5, 2014, at 20:04, Guido van Rossum wrote: > On Sun, Oct 5, 2014 at 7:36 PM, Steven D'Aprano wrote: >> On Sun, Oct 05, 2014 at 02:43:13PM -0700, Guido van Rossum wrote: >> >> > An alternative that wouldn't even require a metaclass would be to write a >> > helper function that looks the class object up by name after parsing >> > __qualname__. There are some limitations to this, e.g. classes defined >> > inside functions -- but that's already a suspect pattern anyway. >> >> I understand that "suspect pattern" doesn't mean "don't ever do this", >> but I wonder what aspect of defining classes inside a function is >> suspect. I think the suspect part is that each invocation of the >> function creates a new class which (at least in the naive case) has the >> same name, same functionality, and looks identical to the casual >> glance but actually isn't. >> ? >> There's >> probably also implications for pickling as well. Are these the only >> "suspect" parts of defining classes inside functions, or have I missed >> something else? > > Those (and their variations, e.g. using these in except clauses) are the main concerns; it also generally bothers me when something "smallish" like a function contains something "larger" like a class, especially when the class is substantial. It's another case of "flat is better than nested". > > The only non-evil use of this pattern that I can recall is in unit tests I agree that returning a class from a function is an uncommon special case, and any code that does so ought to have a good and obvious reason. But to say that such functions are evil doesn't seem right. What about the functional constructors for namedtuple, or the two kinds of enums? Or a custom kind of enum (strings, bitsets, ...), which the PEP specifically suggests building on top of the stdlib support? Or something more inherently dynamic like a bridge library? Even in cases like generating an ORM from a SQL dump or a client from an RPC discovery call, which could be moved to a separate "pre-compilation" phase of building your application, is a function that generates source code for a class less suspect than one that just generates a class? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Oct 6 17:43:10 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 06 Oct 2014 08:43:10 -0700 Subject: [Python-ideas] Classes inside functions [was Re: Add __parent__ to all classes, functions, and modules] In-Reply-To: References: <20141006023628.GN19757@ando.pearwood.info> Message-ID: <5432B88E.2060709@stoneleaf.us> On 10/06/2014 08:07 AM, Andrew Barnert wrote: > On Oct 5, 2014, at 20:04, Guido van Rossum wrote: >> >> The only non-evil use of this pattern that I can recall is in unit tests > > I agree that returning a class from a function is an uncommon special case, and any code that does so ought to have a > good and obvious reason. But to say that such functions are evil doesn't seem right. > > What about [the] the two kinds of enums? Or a custom kind of enum (strings, > bitsets, ...), which the PEP specifically suggests building on top of the stdlib support? What do you mean? Enums are classes, and the standard way to make new Enum types is by subclassing. -- ~Ethan~ From guido at python.org Mon Oct 6 19:23:42 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 6 Oct 2014 10:23:42 -0700 Subject: [Python-ideas] Classes inside functions [was Re: Add __parent__ to all classes, functions, and modules] In-Reply-To: References: <20141006023628.GN19757@ando.pearwood.info> Message-ID: On Mon, Oct 6, 2014 at 8:07 AM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > On Oct 5, 2014, at 20:04, Guido van Rossum wrote: > > On Sun, Oct 5, 2014 at 7:36 PM, Steven D'Aprano > wrote: > >> On Sun, Oct 05, 2014 at 02:43:13PM -0700, Guido van Rossum wrote: >> >> > An alternative that wouldn't even require a metaclass would be to write >> a >> > helper function that looks the class object up by name after parsing >> > __qualname__. There are some limitations to this, e.g. classes defined >> > inside functions -- but that's already a suspect pattern anyway. >> >> I understand that "suspect pattern" doesn't mean "don't ever do this", >> but I wonder what aspect of defining classes inside a function is >> suspect. I think the suspect part is that each invocation of the >> function creates a new class which (at least in the naive case) has the >> same name, same functionality, and looks identical to the casual >> glance but actually isn't. > > ? > > There's >> probably also implications for pickling as well. Are these the only >> "suspect" parts of defining classes inside functions, or have I missed >> something else? >> > > Those (and their variations, e.g. using these in except clauses) are the > main concerns; it also generally bothers me when something "smallish" like > a function contains something "larger" like a class, especially when the > class is substantial. It's another case of "flat is better than nested". > > The only non-evil use of this pattern that I can recall is in unit tests > > > I agree that returning a class from a function is an uncommon special > case, and any code that does so ought to have a good and obvious reason. > But to say that such functions are evil doesn't seem right. > I didn't say *all* uses (or all uses outside unit tests) are evil. I still maintain that most non-test code doing this has a bad smell and is difficult to maintain. > > What about the functional constructors for namedtuple, or the two kinds of > enums? Or a custom kind of enum (strings, bitsets, ...), which the PEP > specifically suggests building on top of the stdlib support? Or something > more inherently dynamic like a bridge library? Even in cases like > generating an ORM from a SQL dump or a client from an RPC discovery > call, which could be moved to a separate "pre-compilation" phase of > building your application, is a function that generates source code for a > class less suspect than one that just generates a class? > How namedtuple is implemented should be nobody's business, except Raymond's, and it certainly isn't a pattern to be recommended. That's why it's in the stdlib -- so you don't have to write such code yourself. Same for enums. Yes, it *can* be done. But it takes superhuman skills to get it right and it still won't be maintainable (TBH, every time I see the namedtuple implementation I have to resist the urge to rewrite it. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Oct 6 19:35:36 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 6 Oct 2014 10:35:36 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: On Mon, Oct 6, 2014 at 1:10 AM, Ram Rachum wrote: > Here are a couple: > > - I'm making a combinatorics package, and a combination space needs to > have a __contains__ method that takes a combination and returns whether > it's part of a set. Since a combination, unlike a permutation, has no > order, I like to have my combinations be canonicalized in a sorted order. > For example, in a combination space of 3 items on range(4), (0, 1, 3) would > be a combination in that space, but (3, 1, 0) would not because it's not > sorted. (It's a k-permutation but not a combination.) However, if the user > does `{0, 1, 3} in comb_space` I still want to return True, regardless of > whether the set iterator happens to give these items in order or not. > So how are you writing this code today? In the following case, what's in the then or else branch? if not isinstance(x, collections.Ordered): else: Even if you could write this, how would you know that an ordered argument is in the *canonical* order? - For the same package, I'm defining `Tally` and `OrderedTally` classes. > (Simliar to `Counter` except without having an identity crisis between > being a dict subclass and counter; mine are strictly counters.) In the > tests I want to easily see whether the class I'm testing is the ordered one > or not, so I'll know to run appropriate tests. (There are also > `FrozenTally` and `FrozenOrderedTally` so it's not just one class.) I could > set `is_ordered = True` on them or give them some base class, but I think a > general `collections.Ordered` abstract base class would be the best > solution. > Same question. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Oct 6 19:47:14 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 6 Oct 2014 13:47:14 -0400 Subject: [Python-ideas] Better stdlib support for Path objects Message-ID: <20141006134714.640fc3ca@anarchist.wooz.org> Over in issue 22570, I lament the fact that while pathlib is awesome, its wider support in the stdlib is pretty sparse. I've tried to convert parts of a medium sized Python 3 application from os.path to pathlib and found this lack of support rather demotivating. Yes, it's fairly easy to wrap Path objects in str() to pass them to stdlib methods that expect only strings, but it's a lot of work in user code and I find that the resulting str()s are distracting. It's a disincentive. Antoine provided a link to a previous discussion[*] but that didn't go very far. One simple solution would be to sprinkle str() calls in various stdlib methods, but I'm not sure if that would fail miserably in the face of bytes paths (if the original stdlib API even accepts bytes paths). The suggestion in the issue is to add a "path protocol" and the referenced article suggests .strpath and .bytespath. OTOH, isn't str() and bytes() enough? I don't have any other brilliant ideas, but I opened the issue and am posting here to see if we can jump start another discussion for Python 3.5. I'd *like* to use more Paths, but not at the expense of my own code's readability. Yes, I'd sacrifice a bit of readability in the stdlib, especially if that would cover more use cases. Cheers, -Barry [*] https://mail.python.org/pipermail/python-ideas/2014-May/027869.html -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From donald at stufft.io Mon Oct 6 19:53:47 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 6 Oct 2014 13:53:47 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141006134714.640fc3ca@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: > > On Oct 6, 2014, at 1:47 PM, Barry Warsaw wrote: > > Over in issue 22570, I lament the fact that while pathlib is awesome, its > wider support in the stdlib is pretty sparse. I've tried to convert parts of > a medium sized Python 3 application from os.path to pathlib and found this > lack of support rather demotivating. Yes, it's fairly easy to wrap Path > objects in str() to pass them to stdlib methods that expect only strings, but > it's a lot of work in user code and I find that the resulting str()s are > distracting. It's a disincentive. > > Antoine provided a link to a previous discussion[*] but that didn't go very > far. > > One simple solution would be to sprinkle str() calls in various stdlib > methods, but I'm not sure if that would fail miserably in the face of bytes > paths (if the original stdlib API even accepts bytes paths). The suggestion > in the issue is to add a "path protocol" and the referenced article suggests > .strpath and .bytespath. OTOH, isn't str() and bytes() enough? > > I don't have any other brilliant ideas, but I opened the issue and am posting > here to see if we can jump start another discussion for Python 3.5. I'd > *like* to use more Paths, but not at the expense of my own code's readability. > Yes, I'd sacrifice a bit of readability in the stdlib, especially if that > would cover more use cases. > > Cheers, > -Barry > > [*] https://mail.python.org/pipermail/python-ideas/2014-May/027869.html > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ Previous attempts at a pathlib like thing outside of the stdlib had the object inherit from str so it?d still work just fine in all those APIs. It even makes a certain bit of sense since a path really is just a specialized string. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From solipsis at pitrou.net Mon Oct 6 19:58:13 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 6 Oct 2014 19:58:13 +0200 Subject: [Python-ideas] Better stdlib support for Path objects References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: <20141006195813.284d7cd3@fsol> On Mon, 6 Oct 2014 13:47:14 -0400 Barry Warsaw wrote: > > One simple solution would be to sprinkle str() calls in various stdlib > methods, but I'm not sure if that would fail miserably in the face of bytes > paths (if the original stdlib API even accepts bytes paths). The suggestion > in the issue is to add a "path protocol" and the referenced article suggests > .strpath and .bytespath. OTOH, isn't str() and bytes() enough? str() works on anything (including enums and distutils compilers), so it's pretty bad IMO. Regards Antoine. From barry at python.org Mon Oct 6 19:59:19 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 6 Oct 2014 13:59:19 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: <20141006135919.387865d8@anarchist.wooz.org> On Oct 06, 2014, at 01:53 PM, Donald Stufft wrote: >Previous attempts at a pathlib like thing outside of the stdlib >had the object inherit from str so it?d still work just fine in >all those APIs. It even makes a certain bit of sense since a >path really is just a specialized string. Yeah, except: http://bugs.python.org/issue22570#msg228716 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From guido at python.org Mon Oct 6 20:04:16 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 6 Oct 2014 11:04:16 -0700 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141006134714.640fc3ca@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: On Mon, Oct 6, 2014 at 10:47 AM, Barry Warsaw wrote: > Over in issue 22570, I lament the fact that while pathlib is awesome, its > wider support in the stdlib is pretty sparse. I've tried to convert parts > of > a medium sized Python 3 application from os.path to pathlib and found this > lack of support rather demotivating. Yes, it's fairly easy to wrap Path > objects in str() to pass them to stdlib methods that expect only strings, > but > it's a lot of work in user code and I find that the resulting str()s are > distracting. It's a disincentive. > > Antoine provided a link to a previous discussion[*] but that didn't go very > far. > > One simple solution would be to sprinkle str() calls in various stdlib > methods, but I'm not sure if that would fail miserably in the face of bytes > paths (if the original stdlib API even accepts bytes paths). The > suggestion > in the issue is to add a "path protocol" and the referenced article > suggests > .strpath and .bytespath. OTOH, isn't str() and bytes() enough? > > I don't have any other brilliant ideas, but I opened the issue and am > posting > here to see if we can jump start another discussion for Python 3.5. I'd > *like* to use more Paths, but not at the expense of my own code's > readability. > Yes, I'd sacrifice a bit of readability in the stdlib, especially if that > would cover more use cases. > > Cheers, > -Barry > > [*] https://mail.python.org/pipermail/python-ideas/2014-May/027869.html > I'd turn it around. You can construct a Path from an argument that can be either a string or another Path. Example: >>> from pathlib import Path >>> p = Path('/etc/passwd') >>> q = Path(p) >>> p == q True >>> So you could start refactoring stdlib code to use Path internally without forcing the callers to use Path, but still *allow* the callers to pass a Path. Though I'm not sure how this would work for return values without breaking backwards compatibility -- you'd have to keep returning strings and the callers would have to use the same mechanism to go back to using Paths. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Oct 6 20:24:31 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 6 Oct 2014 14:24:31 -0400 Subject: [Python-ideas] Better stdlib support for Path objects References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: <20141006142431.56ddba78@anarchist.wooz.org> On Oct 06, 2014, at 11:04 AM, Guido van Rossum wrote: >You can construct a Path from an argument that can be either a string or >another Path. Example: > >>>> from pathlib import Path >>>> p = Path('/etc/passwd') >>>> q = Path(p) >>>> p == q >True >>>> > >So you could start refactoring stdlib code to use Path internally without >forcing the callers to use Path, but still *allow* the callers to pass a >Path. Though I'm not sure how this would work for return values without >breaking backwards compatibility -- you'd have to keep returning strings >and the callers would have to use the same mechanism to go back to using >Paths. That's a very interesting perspective, and one I'd like to pursue further. I wonder if we can take a polymorphic approach similar to some bytes/str APIs, namely that if the argument is a pathlib object, a pathlib object could be returned, and if a str was passed, a str would be returned. An example is ConfigParser.read() which currently accepts only strings. I want to pass it a Path. It would be really useful if this method returned Paths when passed paths (they're used to verify which arguments were actually opened and read). There's probably a gazillion APIs that *could* be pathlib-ified, and I'm not looking to do a comprehensive expansion across the entire stdlib. But I think we could probably take a per-module approach, at least at first, to see if there are any hidden gotchas. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From guido at python.org Mon Oct 6 20:29:40 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 6 Oct 2014 11:29:40 -0700 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141006142431.56ddba78@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> Message-ID: On Mon, Oct 6, 2014 at 11:24 AM, Barry Warsaw wrote: > On Oct 06, 2014, at 11:04 AM, Guido van Rossum wrote: > > >You can construct a Path from an argument that can be either a string or > >another Path. Example: > > > >>>> from pathlib import Path > >>>> p = Path('/etc/passwd') > >>>> q = Path(p) > >>>> p == q > >True > >>>> > > > >So you could start refactoring stdlib code to use Path internally without > >forcing the callers to use Path, but still *allow* the callers to pass a > >Path. Though I'm not sure how this would work for return values without > >breaking backwards compatibility -- you'd have to keep returning strings > >and the callers would have to use the same mechanism to go back to using > >Paths. > > That's a very interesting perspective, and one I'd like to pursue further. > > I wonder if we can take a polymorphic approach similar to some bytes/str > APIs, > namely that if the argument is a pathlib object, a pathlib object could be > returned, and if a str was passed, a str would be returned. > > An example is ConfigParser.read() which currently accepts only strings. I > want to pass it a Path. It would be really useful if this method returned > Paths when passed paths (they're used to verify which arguments were > actually > opened and read). > > There's probably a gazillion APIs that *could* be pathlib-ified, and I'm > not > looking to do a comprehensive expansion across the entire stdlib. But I > think > we could probably take a per-module approach, at least at first, to see if > there are any hidden gotchas. > Sounds like a plan. I'd like to see how hard it is in practice to do the polymorphic return value thing. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Oct 6 20:33:25 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 6 Oct 2014 14:33:25 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> Message-ID: <20141006143325.0b9cdda6@anarchist.wooz.org> On Oct 06, 2014, at 11:29 AM, Guido van Rossum wrote: >Sounds like a plan. I'd like to see how hard it is in practice to do the >polymorphic return value thing. I'll do a limited experiment and post the results. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From tjreedy at udel.edu Mon Oct 6 21:09:38 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 06 Oct 2014 15:09:38 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: On 10/6/2014 2:04 PM, Guido van Rossum wrote: > On Mon, Oct 6, 2014 at 10:47 AM, Barry Warsaw > > wrote: > > Over in issue 22570, I lament the fact that while pathlib is > awesome, its > wider support in the stdlib is pretty sparse. I've tried to convert > parts of > a medium sized Python 3 application from os.path to pathlib and > found this > lack of support rather demotivating. Yes, it's fairly easy to wrap Path > objects in str() to pass them to stdlib methods that expect only > strings, but > it's a lot of work in user code and I find that the resulting str()s are > distracting. It's a disincentive. > > Antoine provided a link to a previous discussion[*] but that didn't > go very > far. > > One simple solution would be to sprinkle str() calls in various stdlib > methods, but I'm not sure if that would fail miserably in the face > of bytes > paths (if the original stdlib API even accepts bytes paths). The > suggestion > in the issue is to add a "path protocol" and the referenced article > suggests > .strpath and .bytespath. OTOH, isn't str() and bytes() enough? > > I don't have any other brilliant ideas, but I opened the issue and > am posting > here to see if we can jump start another discussion for Python 3.5. I'd > *like* to use more Paths, but not at the expense of my own code's > readability. > Yes, I'd sacrifice a bit of readability in the stdlib, especially if > that > would cover more use cases. > > Cheers, > -Barry > > [*] https://mail.python.org/pipermail/python-ideas/2014-May/027869.html To me, the first question to me is whether we 'believe' in pathlib enough to really support it in the stdlib and encourage its use. > I'd turn it around. > > You can construct a Path from an argument that can be either a string or > another Path. Example: > > >>> from pathlib import Path > >>> p = Path('/etc/passwd') > >>> q = Path(p) > >>> p == q > True > >>> > > So you could start refactoring stdlib code to use Path internally > without forcing the callers to use Path, but still *allow* the callers > to pass a Path. If yes (which the above seems to hint), the second question is how to enlarge apis while remaining back compatible. For functions that take a pathstring in and do not return a pathstring, just allow a Path as alternate input type. There are already functions that take either a pathstring or open file-like object. > Though I'm not sure how this would work for return > values without breaking backwards compatibility -- you'd have to keep > returning strings and the callers would have to use the same mechanism > to go back to using Paths. Some of the os functions that take a pathstring and return a pathstring are already 'duplicated' as pathlib functions or Path methods that map Path to Path. For others, there is a choice of duplicating the function in pathlib or making the output type depend on the input type. I do not remember the current scandir spec, but it should when introduced at least optionally accept and produce paths and perhaps live in pathlib instead of os. I suspect that functions that produce a pathstring without a pathstring input, such as int file descriptor to filename, are rare and low-level enough to leave as is. But that should be specified in any transition-plan pep. -- Terry Jan Reedy From python at mrabarnett.plus.com Mon Oct 6 21:19:59 2014 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 06 Oct 2014 20:19:59 +0100 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141006143325.0b9cdda6@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> <20141006143325.0b9cdda6@anarchist.wooz.org> Message-ID: <5432EB5F.7090005@mrabarnett.plus.com> On 2014-10-06 19:33, Barry Warsaw wrote: > On Oct 06, 2014, at 11:29 AM, Guido van Rossum wrote: > >> Sounds like a plan. I'd like to see how hard it is in practice to >> do the polymorphic return value thing. > > I'll do a limited experiment and post the results. > I wonder whether it might be cleaner to use a simple function for it, something like: def pathify(result_path, like=original_path): if isinstance(original_path, Path): return Path(result_path) if isinstance(original_path, str): return str(result_path) raise TypeError('original path must be a path or a string, not {!r}'.format(type(original_path))) From yselivanov.ml at gmail.com Mon Oct 6 22:00:36 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 06 Oct 2014 16:00:36 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <5432EB5F.7090005@mrabarnett.plus.com> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> <20141006143325.0b9cdda6@anarchist.wooz.org> <5432EB5F.7090005@mrabarnett.plus.com> Message-ID: <5432F4E4.7060003@gmail.com> On 2014-10-06, 3:19 PM, MRAB wrote: > On 2014-10-06 19:33, Barry Warsaw wrote: >> On Oct 06, 2014, at 11:29 AM, Guido van Rossum wrote: >> >>> Sounds like a plan. I'd like to see how hard it is in practice to >>> do the polymorphic return value thing. >> >> I'll do a limited experiment and post the results. >> > I wonder whether it might be cleaner to use a simple function for it, > something like: > > def pathify(result_path, like=original_path): > if isinstance(original_path, Path): > return Path(result_path) > > if isinstance(original_path, str): > return str(result_path) > > raise TypeError('original path must be a path or a string, not > {!r}'.format(type(original_path))) It's also possible to make a decorator for this, something along the lines of: @pathify(args=['path1', 'path2'], return_value=True) def method(path1, path2, option1): ... Yury From cathalgarvey at cathalgarvey.me Mon Oct 6 22:01:03 2014 From: cathalgarvey at cathalgarvey.me (Cathal Garvey) Date: Mon, 06 Oct 2014 21:01:03 +0100 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> Message-ID: <5432F4FF.5080004@cathalgarvey.me> > I wonder if we can take a polymorphic approach similar to some > bytes/str APIs, namely that if the argument is a pathlib object, a > pathlib object could be returned, and if a str was passed, a str > would be returned. This is a strong solution for new code. A quick hack to fix old code would be to simply cast the return values from anything returning a "path" to a string. If it's already a string, it'll be the same string. If it's a pathlib path, it'll end up a string. That minimises the effort needed to bring an old codebase up to speed with pathlib while implementing new code in pathlib, right? And you can work downwards from there; as long as all callers expecting a path do an explicit str-cast, you're safe either way? On 06/10/14 19:29, Guido van Rossum wrote: > On Mon, Oct 6, 2014 at 11:24 AM, Barry Warsaw wrote: > >> On Oct 06, 2014, at 11:04 AM, Guido van Rossum wrote: >> >>> You can construct a Path from an argument that can be either a string or >>> another Path. Example: >>> >>>>>> from pathlib import Path >>>>>> p = Path('/etc/passwd') >>>>>> q = Path(p) >>>>>> p == q >>> True >>>>>> >>> >>> So you could start refactoring stdlib code to use Path internally without >>> forcing the callers to use Path, but still *allow* the callers to pass a >>> Path. Though I'm not sure how this would work for return values without >>> breaking backwards compatibility -- you'd have to keep returning strings >>> and the callers would have to use the same mechanism to go back to using >>> Paths. >> >> That's a very interesting perspective, and one I'd like to pursue further. >> >> I wonder if we can take a polymorphic approach similar to some bytes/str >> APIs, >> namely that if the argument is a pathlib object, a pathlib object could be >> returned, and if a str was passed, a str would be returned. >> >> An example is ConfigParser.read() which currently accepts only strings. I >> want to pass it a Path. It would be really useful if this method returned >> Paths when passed paths (they're used to verify which arguments were >> actually >> opened and read). >> >> There's probably a gazillion APIs that *could* be pathlib-ified, and I'm >> not >> looking to do a comprehensive expansion across the entire stdlib. But I >> think >> we could probably take a per-module approach, at least at first, to see if >> there are any hidden gotchas. >> > > Sounds like a plan. I'd like to see how hard it is in practice to do the > polymorphic return value thing. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Twitter: @onetruecathal, @formabiolabs Phone: +353876363185 Blog: http://indiebiotech.com miniLock.io: JjmYYngs7akLZUjkvFkuYdsZ3PyPHSZRBKNm6qTYKZfAM -------------- next part -------------- A non-text attachment was scrubbed... Name: 0x988B9099.asc Type: application/pgp-keys Size: 6176 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From apalala at gmail.com Mon Oct 6 22:20:59 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Mon, 6 Oct 2014 15:50:59 -0430 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: On Mon, Oct 6, 2014 at 2:39 PM, Terry Reedy wrote: > To me, the first question to me is whether we 'believe' in pathlib > enough to really support it in the stdlib and encourage its use. > os.path is cumbersome (horrible?) when compared to pathlib. The issue is prevalent throughout stdlib. For example, json won't take a pathlib.Path, so you have to pass str(mypath). Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Mon Oct 6 22:41:55 2014 From: ram at rachum.com (Ram Rachum) Date: Mon, 6 Oct 2014 23:41:55 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: On Mon, Oct 6, 2014 at 8:35 PM, Guido van Rossum wrote: > On Mon, Oct 6, 2014 at 1:10 AM, Ram Rachum wrote: > >> Here are a couple: >> >> - I'm making a combinatorics package, and a combination space needs to >> have a __contains__ method that takes a combination and returns whether >> it's part of a set. Since a combination, unlike a permutation, has no >> order, I like to have my combinations be canonicalized in a sorted order. >> For example, in a combination space of 3 items on range(4), (0, 1, 3) would >> be a combination in that space, but (3, 1, 0) would not because it's not >> sorted. (It's a k-permutation but not a combination.) However, if the user >> does `{0, 1, 3} in comb_space` I still want to return True, regardless of >> whether the set iterator happens to give these items in order or not. >> > > So how are you writing this code today? In the following case, what's in > the then or else branch? > > if not isinstance(x, collections.Ordered): > > else: > > > Even if you could write this, how would you know that an ordered argument > is in the *canonical* order? > That part isn't written yet (the package is still work-in-progress), but I'm not sure what the problem is. I'll have the code look at `x`. If it's marked as ordered, then I'd iterate on it. If it has the correct items (meaning the items of the sequence that this is a combination space of) and the items in x are in the same order as they are in the sequence, and it has the correct number of items, then we have a match. If we have `not isinstance(x, collections.Ordered)` then I do the same thing except I ignore the order of the items. What's the problem? > > - For the same package, I'm defining `Tally` and `OrderedTally` classes. >> (Simliar to `Counter` except without having an identity crisis between >> being a dict subclass and counter; mine are strictly counters.) In the >> tests I want to easily see whether the class I'm testing is the ordered one >> or not, so I'll know to run appropriate tests. (There are also >> `FrozenTally` and `FrozenOrderedTally` so it's not just one class.) I could >> set `is_ordered = True` on them or give them some base class, but I think a >> general `collections.Ordered` abstract base class would be the best >> solution. >> > > Same question. > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Oct 6 22:42:43 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 6 Oct 2014 13:42:43 -0700 Subject: [Python-ideas] Classes inside functions [was Re: Add __parent__ to all classes, functions, and modules] In-Reply-To: References: <20141006023628.GN19757@ando.pearwood.info> Message-ID: <64A896D3-F66C-4B25-A4D1-77BC856109AE@yahoo.com> Sent from a random iPhone On Oct 6, 2014, at 10:23, Guido van Rossum wrote: > On Mon, Oct 6, 2014 at 8:07 AM, Andrew Barnert wrote: >> On Oct 5, 2014, at 20:04, Guido van Rossum wrote: >>> On Sun, Oct 5, 2014 at 7:36 PM, Steven D'Aprano wrote: >>>> On Sun, Oct 05, 2014 at 02:43:13PM -0700, Guido van Rossum wrote: >>>> >>>> > An alternative that wouldn't even require a metaclass would be to write a >>>> > helper function that looks the class object up by name after parsing >>>> > __qualname__. There are some limitations to this, e.g. classes defined >>>> > inside functions -- but that's already a suspect pattern anyway. >>>> >>>> I understand that "suspect pattern" doesn't mean "don't ever do this", >>>> but I wonder what aspect of defining classes inside a function is >>>> suspect. I think the suspect part is that each invocation of the >>>> function creates a new class which (at least in the naive case) has the >>>> same name, same functionality, and looks identical to the casual >>>> glance but actually isn't. >>>> ? >>>> There's >>>> probably also implications for pickling as well. Are these the only >>>> "suspect" parts of defining classes inside functions, or have I missed >>>> something else? >>> >>> Those (and their variations, e.g. using these in except clauses) are the main concerns; it also generally bothers me when something "smallish" like a function contains something "larger" like a class, especially when the class is substantial. It's another case of "flat is better than nested". >>> >>> The only non-evil use of this pattern that I can recall is in unit tests >> >> I agree that returning a class from a function is an uncommon special case, and any code that does so ought to have a good and obvious reason. But to say that such functions are evil doesn't seem right. > > I didn't say *all* uses (or all uses outside unit tests) are evil. I still maintain that most non-test code doing this has a bad smell and is difficult to maintain. OK, I think I just misinterpreted you here. If you're just saying that this is enough of a red flag that you need to pause and prove to yourself (and ideally your code's readers) that it's a "necessary evil" whenever you find yourself writing a function that returns a class, then that makes total sense. Apologies. >> What about the functional constructors for namedtuple, or the two kinds of enums? Or a custom kind of enum (strings, bitsets, ...), which the PEP specifically suggests building on top of the stdlib support? Or something more inherently dynamic like a bridge library? Even in cases like generating an ORM from a SQL dump or a client from an RPC discovery call, which could be moved to a separate "pre-compilation" phase of building your application, is a function that generates source code for a class less suspect than one that just generates a class? > > How namedtuple is implemented should be nobody's business, except Raymond's, and it certainly isn't a pattern to be recommended. That's why it's in the stdlib -- so you don't have to write such code yourself. Sure, but I don't think you want every bridge library, dynamic RMDB/GUI/RPC/etc. binding library, etc. to be in the stdlib. _Someone_ has to write them, and the fact that Python makes such code writable (if not easy) is a strength of the language. Still, I take your point; I wouldn't trust a dynamic binding library that isn't being used by multiple independent projects, because it's so hard to verify by reading that it's gotten all the details right. And a proposal that makes such things slightly easier to write (while still leaving them very hard to get right) probably doesn't add much to the language. -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Mon Oct 6 23:19:37 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 06 Oct 2014 23:19:37 +0200 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: On 10/06/2014 10:20 PM, Juancarlo A?ez wrote: > > On Mon, Oct 6, 2014 at 2:39 PM, Terry Reedy > > wrote: > > To me, the first question to me is whether we 'believe' in pathlib > enough to really support it in the stdlib and encourage its use. > > > os.path is cumbersome (horrible?) when compared to pathlib. Horrible? Please. > The issue is prevalent throughout stdlib. For example, json won't take a > pathlib.Path, so you have to pass str(mypath). That it is prevalent is exactly the problem with fixing it, in my opinion. This is like the situation with context managers. We've taken about 3-4 minor releases to add "with" support to objects that can logically support it. Nobody remembers this, so people have to refer to the docs (or the code) to see if and when e.g. smtplib.SMTP gained "with" support. However, the context managers are a few dozen classes at most. With paths, there are hundreds of APIs that would have to be updated to take Paths in the stdlib alone. Granted, a good portion would probably work fine since they only pass through paths to lower level APIs, but still every one has to be checked. Going by precedent, that's not something that we would be able to do consistently, even throughout several releases. (Another precedent is Argument Clinic.) cheers, Georg From random832 at fastmail.us Mon Oct 6 23:55:47 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Mon, 06 Oct 2014 17:55:47 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: <1412632547.2422407.175865077.0C99C066@webmail.messagingengine.com> On Mon, Oct 6, 2014, at 14:04, Guido van Rossum wrote: > I'd turn it around. > > You can construct a Path from an argument that can be either a string or > another Path. Example: > > >>> from pathlib import Path > >>> p = Path('/etc/passwd') > >>> q = Path(p) > >>> p == q > True > >>> > > So you could start refactoring stdlib code to use Path internally without > forcing the callers to use Path, but still *allow* the callers to pass a > Path. Though I'm not sure how this would work for return values without > breaking backwards compatibility -- you'd have to keep returning strings > and the callers would have to use the same mechanism to go back to using > Paths. Should Path accept a bytes? It doesn't appear to now. From apalala at gmail.com Tue Oct 7 01:26:58 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Mon, 6 Oct 2014 18:56:58 -0430 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: On Mon, Oct 6, 2014 at 4:49 PM, Georg Brandl wrote: > However, the context managers are a few dozen classes at most. With paths, > there are hundreds of APIs that would have to be updated to take Paths > in the stdlib alone. Granted, a good portion would probably work fine > since > they only pass through paths to lower level APIs, but still every one has > to > be checked. Going by precedent, that's not something that we would be able > to do consistently, even throughout several releases. (Another precedent > is > Argument Clinic.) > What do you think would be the nastier impacts of making pathlib.Path inherit from str? Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From apalala at gmail.com Tue Oct 7 01:38:04 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Mon, 6 Oct 2014 19:08:04 -0430 Subject: [Python-ideas] Classes returned from functions Message-ID: Grako is a PEG parser builder in Python: https://bitbucket.org/apalala/grako In grako, there's code to synthesize AST node types according to annotations in the grammar whenever a type is not previously provided. This is the code: def _get_nodetype(self, typename): typename = str(typename) if typename in self.nodetypes: return self.nodetypes[typename] # synthethize a new type nodetype = type(typename, (self.baseType,), {}) self._register_nodetype(nodetype) return nodetype It works, but the classes thus created are "weird". I don't remember the details right now, but I think they don't have a proper module or qualname. This particular feature (exploiting this Python capability) is very useful in the early stages of developing a parser, while most of the work goes into debugging the grammar. Having a nice AST produced cheaply is of great value. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Tue Oct 7 01:40:54 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 6 Oct 2014 19:40:54 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: On Mon, Oct 6, 2014 at 7:26 PM, Juancarlo A?ez wrote: > What do you think would be the nastier impacts of making pathlib.Path > inherit from str? Duplication of storage. Currently, pathlib.Path objects store a list of path components. To inherit meaningfully from str, they would have to store joined path string as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Tue Oct 7 01:52:55 2014 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 07 Oct 2014 00:52:55 +0100 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: <54332B57.9020300@mrabarnett.plus.com> On 2014-10-07 00:40, Alexander Belopolsky wrote: > > On Mon, Oct 6, 2014 at 7:26 PM, Juancarlo A?ez > wrote: > > What do you think would be the nastier impacts of making > pathlib.Path inherit from str? > > > Duplication of storage. Currently, pathlib.Path objects store a list of > path components. To inherit meaningfully from str, they would have to > store joined path string as well. > Wouldn't it also mean that it would inherit str's methods, but they would return instances of path, and it would need to determine the new path components? From solipsis at pitrou.net Tue Oct 7 01:58:21 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Oct 2014 01:58:21 +0200 Subject: [Python-ideas] Better stdlib support for Path objects References: <20141006134714.640fc3ca@anarchist.wooz.org> <54332B57.9020300@mrabarnett.plus.com> Message-ID: <20141007015821.2ff84b7d@fsol> Hi, Not inheriting from built-in classes such as str, list or tuple was one of the design points of pathlib. It will not change in the future ;-) PEP 428 outlines this, but you can probably find a more detailed discussion in the python-ideas archive. Regards Antoine. On Tue, 07 Oct 2014 00:52:55 +0100 MRAB wrote: > On 2014-10-07 00:40, Alexander Belopolsky wrote: > > > > On Mon, Oct 6, 2014 at 7:26 PM, Juancarlo A?ez > > wrote: > > > > What do you think would be the nastier impacts of making > > pathlib.Path inherit from str? > > > > > > Duplication of storage. Currently, pathlib.Path objects store a list of > > path components. To inherit meaningfully from str, they would have to > > store joined path string as well. > > > Wouldn't it also mean that it would inherit str's methods, but they > would return instances of path, and it would need to determine the new > path components? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Tue Oct 7 02:39:41 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 6 Oct 2014 17:39:41 -0700 Subject: [Python-ideas] Classes inside functions [was Re: Add __parent__ to all classes, functions, and modules] In-Reply-To: <64A896D3-F66C-4B25-A4D1-77BC856109AE@yahoo.com> References: <20141006023628.GN19757@ando.pearwood.info> <64A896D3-F66C-4B25-A4D1-77BC856109AE@yahoo.com> Message-ID: On Mon, Oct 6, 2014 at 1:42 PM, Andrew Barnert < abarnert at yahoo.com.dmarc.invalid> wrote: > > > Sent from a random iPhone > > On Oct 6, 2014, at 10:23, Guido van Rossum wrote: > > On Mon, Oct 6, 2014 at 8:07 AM, Andrew Barnert < > abarnert at yahoo.com.dmarc.invalid> wrote: > >> On Oct 5, 2014, at 20:04, Guido van Rossum wrote: >> >> On Sun, Oct 5, 2014 at 7:36 PM, Steven D'Aprano >> wrote: >> >>> On Sun, Oct 05, 2014 at 02:43:13PM -0700, Guido van Rossum wrote: >>> >>> > An alternative that wouldn't even require a metaclass would be to >>> write a >>> > helper function that looks the class object up by name after parsing >>> > __qualname__. There are some limitations to this, e.g. classes defined >>> > inside functions -- but that's already a suspect pattern anyway. >>> >>> I understand that "suspect pattern" doesn't mean "don't ever do this", >>> but I wonder what aspect of defining classes inside a function is >>> suspect. I think the suspect part is that each invocation of the >>> function creates a new class which (at least in the naive case) has the >>> same name, same functionality, and looks identical to the casual >>> glance but actually isn't. >> >> ? >> >> There's >>> probably also implications for pickling as well. Are these the only >>> "suspect" parts of defining classes inside functions, or have I missed >>> something else? >>> >> >> Those (and their variations, e.g. using these in except clauses) are the >> main concerns; it also generally bothers me when something "smallish" like >> a function contains something "larger" like a class, especially when the >> class is substantial. It's another case of "flat is better than nested". >> >> The only non-evil use of this pattern that I can recall is in unit tests >> >> >> I agree that returning a class from a function is an uncommon special >> case, and any code that does so ought to have a good and obvious reason. >> But to say that such functions are evil doesn't seem right. >> > > I didn't say *all* uses (or all uses outside unit tests) are evil. I still > maintain that most non-test code doing this has a bad smell and is > difficult to maintain. > > > OK, I think I just misinterpreted you here. If you're just saying that > this is enough of a red flag that you need to pause and prove to yourself > (and ideally your code's readers) that it's a "necessary evil" whenever you > find yourself writing a function that returns a class, then that makes > total sense. Apologies. > No problem! That's what I meant. (Note also that I didn't threaten to remove the feature. :-) > > What about the functional constructors for namedtuple, or the two kinds of >> enums? Or a custom kind of enum (strings, bitsets, ...), which the PEP >> specifically suggests building on top of the stdlib support? Or something >> more inherently dynamic like a bridge library? Even in cases like >> generating an ORM from a SQL dump or a client from an RPC discovery >> call, which could be moved to a separate "pre-compilation" phase of >> building your application, is a function that generates source code for a >> class less suspect than one that just generates a class? >> > > How namedtuple is implemented should be nobody's business, except > Raymond's, and it certainly isn't a pattern to be recommended. That's why > it's in the stdlib -- so you don't have to write such code yourself. > > > Sure, but I don't think you want every bridge library, dynamic > RMDB/GUI/RPC/etc. binding library, etc. to be in the stdlib. _Someone_ has > to write them, and the fact that Python makes such code writable (if not > easy) is a strength of the language. > Yup. But that code rarely takes the simple form def make_a_class(parameters): ... class ClassTemplate: def method1(self): ... def method2(self): ... ... ... return C Still, I take your point; I wouldn't trust a dynamic binding library that > isn't being used by multiple independent projects, because it's so hard to > verify by reading that it's gotten all the details right. And a proposal > that makes such things slightly easier to write (while still leaving them > very hard to get right) probably doesn't add much to the language. > Right. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Oct 7 02:40:59 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 6 Oct 2014 17:40:59 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: I think I just lost interest in this thread. On Mon, Oct 6, 2014 at 1:41 PM, Ram Rachum wrote: > On Mon, Oct 6, 2014 at 8:35 PM, Guido van Rossum wrote: > >> On Mon, Oct 6, 2014 at 1:10 AM, Ram Rachum wrote: >> >>> Here are a couple: >>> >>> - I'm making a combinatorics package, and a combination space needs to >>> have a __contains__ method that takes a combination and returns whether >>> it's part of a set. Since a combination, unlike a permutation, has no >>> order, I like to have my combinations be canonicalized in a sorted order. >>> For example, in a combination space of 3 items on range(4), (0, 1, 3) would >>> be a combination in that space, but (3, 1, 0) would not because it's not >>> sorted. (It's a k-permutation but not a combination.) However, if the user >>> does `{0, 1, 3} in comb_space` I still want to return True, regardless of >>> whether the set iterator happens to give these items in order or not. >>> >> >> So how are you writing this code today? In the following case, what's in >> the then or else branch? >> >> if not isinstance(x, collections.Ordered): >> >> else: >> >> >> Even if you could write this, how would you know that an ordered argument >> is in the *canonical* order? >> > > That part isn't written yet (the package is still work-in-progress), but > I'm not sure what the problem is. I'll have the code look at `x`. If it's > marked as ordered, then I'd iterate on it. If it has the correct items > (meaning the items of the sequence that this is a combination space of) and > the items in x are in the same order as they are in the sequence, and it > has the correct number of items, then we have a match. If we have `not > isinstance(x, collections.Ordered)` then I do the same thing except I > ignore the order of the items. What's the problem? > > >> >> - For the same package, I'm defining `Tally` and `OrderedTally` classes. >>> (Simliar to `Counter` except without having an identity crisis between >>> being a dict subclass and counter; mine are strictly counters.) In the >>> tests I want to easily see whether the class I'm testing is the ordered one >>> or not, so I'll know to run appropriate tests. (There are also >>> `FrozenTally` and `FrozenOrderedTally` so it's not just one class.) I could >>> set `is_ordered = True` on them or give them some base class, but I think a >>> general `collections.Ordered` abstract base class would be the best >>> solution. >>> >> >> Same question. >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Oct 7 03:38:24 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 6 Oct 2014 21:38:24 -0400 Subject: [Python-ideas] Better stdlib support for Path objects References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: <20141006213824.46066a22@anarchist.wooz.org> On Oct 06, 2014, at 11:19 PM, Georg Brandl wrote: >This is like the situation with context managers. We've taken about 3-4 >minor releases to add "with" support to objects that can logically support >it. Nobody remembers this, so people have to refer to the docs (or the code) >to see if and when e.g. smtplib.SMTP gained "with" support. > >However, the context managers are a few dozen classes at most. With paths, >there are hundreds of APIs that would have to be updated to take Paths >in the stdlib alone. Granted, a good portion would probably work fine since >they only pass through paths to lower level APIs, but still every one has to >be checked. Going by precedent, that's not something that we would be able >to do consistently, even throughout several releases. (Another precedent is >Argument Clinic.) I appreciate that this is a problem with such transitions. Is it an argument for never doing so though? Is it better that the stdlib prohibit adoption of advanced features and libraries than to do so piecemeal? I'm not so sure; even though it's admittedly annoying when such support is missing, it's really nice when they're there. How useful is pathlib if it can't be used with the stdlib? Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From brianvanderburg2 at aim.com Tue Oct 7 03:34:16 2014 From: brianvanderburg2 at aim.com (Brian Allen Vanderburg II) Date: Mon, 6 Oct 2014 21:34:16 -0400 Subject: [Python-ideas] Idea to support lazy loaded names. Message-ID: <20141006213416.21bdb347@allen-pc> When developing Python libraries (and any other language for that matter), items are separated into modules for organization and other purpose. The following is an example fake framework to illustrate this: mini-framework/ __init__.py app.py class BaseApplication class Application class ProxyApplication config.py class Config class ConfigLoader template.py class Template class TemplateCompiler class TemplateManager In order to be used externally, one must reference the module: from miniframework.app import Application from miniframework.template import TemplateManager This can be solved by importing these values directly in __init__.py. However this requires fully loading those imported modules: __init__.py: from .app import BaseApplication, Application, ProxyApplication from .config import Config, ConfigLoader from .template import Template, TemplateCompiler, TemplateManager One idea I had was to support lazy imported names. I've seen some frameworks implement this in various means, and figured the idea is good to be part Python. The idea is that for a new module attribute to exist: __lazyload__. During the access of any attribute for a module, the following would happen: * If the named attribute already exists, use it * If the named attribute does not already exist: * If a lazy load of the name has already been attempted, result in a NameError * If a lazy load of the name has not yet been attempted: * Check the __lazyload__ module attribute for the name, perform the loading operation and assign the module attribute the value if found or result in a NameError * Even if not found, set a flag that lazy load has been attempted so it will not be attempted again for the same name The __lazyload__ attribute is intended to be a dictionary. The key of the dictionary is the name of the attribute that would be set/tested for in the module. The value of the dictionary is a string that represents either the module name to load or the module name and attribute to load. If the value starts with a dot, then it is treated as a relative import relative to the module/package containing the __lazyload__ value. With this idea, the packages __init__.py file could look like this: __lazyload__ = { 'BaseApplication': '.app.BaseApplication', 'Application': '.app.Application', ... } The end use of the package (and even the developer) can then perform an import as follows: from miniframework import Application instead of: from miniframework.app import Application This allows the public api be be cleaner, while still being efficient by not loading all modules in __init__.py until the value is actually accessed. Brian Allen Vanderburg II From donald at stufft.io Tue Oct 7 05:26:32 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 6 Oct 2014 23:26:32 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141006134714.640fc3ca@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: <441EC747-0836-4DEB-81B4-D68E262B10AA@stufft.io> > On Oct 6, 2014, at 1:47 PM, Barry Warsaw wrote: > > Over in issue 22570, I lament the fact that while pathlib is awesome, its > wider support in the stdlib is pretty sparse. I've tried to convert parts of > a medium sized Python 3 application from os.path to pathlib and found this > lack of support rather demotivating. Yes, it's fairly easy to wrap Path > objects in str() to pass them to stdlib methods that expect only strings, but > it's a lot of work in user code and I find that the resulting str()s are > distracting. It's a disincentive. > > Antoine provided a link to a previous discussion[*] but that didn't go very > far. > > One simple solution would be to sprinkle str() calls in various stdlib > methods, but I'm not sure if that would fail miserably in the face of bytes > paths (if the original stdlib API even accepts bytes paths). The suggestion > in the issue is to add a "path protocol" and the referenced article suggests > .strpath and .bytespath. OTOH, isn't str() and bytes() enough? > > I don't have any other brilliant ideas, but I opened the issue and am posting > here to see if we can jump start another discussion for Python 3.5. I'd > *like* to use more Paths, but not at the expense of my own code's readability. > Yes, I'd sacrifice a bit of readability in the stdlib, especially if that > would cover more use cases. > > Suggested by Alex on IRC: Path(???).path instead of str(Path(???)). Tiny change but it?s a bit easier to type? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Tue Oct 7 05:30:01 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 6 Oct 2014 23:30:01 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141006213824.46066a22@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> Message-ID: <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> > On Oct 6, 2014, at 9:38 PM, Barry Warsaw wrote: > > On Oct 06, 2014, at 11:19 PM, Georg Brandl wrote: > >> This is like the situation with context managers. We've taken about 3-4 >> minor releases to add "with" support to objects that can logically support >> it. Nobody remembers this, so people have to refer to the docs (or the code) >> to see if and when e.g. smtplib.SMTP gained "with" support. >> >> However, the context managers are a few dozen classes at most. With paths, >> there are hundreds of APIs that would have to be updated to take Paths >> in the stdlib alone. Granted, a good portion would probably work fine since >> they only pass through paths to lower level APIs, but still every one has to >> be checked. Going by precedent, that's not something that we would be able >> to do consistently, even throughout several releases. (Another precedent is >> Argument Clinic.) > > I appreciate that this is a problem with such transitions. Is it an argument > for never doing so though? Is it better that the stdlib prohibit adoption of > advanced features and libraries than to do so piecemeal? I'm not so sure; > even though it's admittedly annoying when such support is missing, it's really > nice when they're there. > > How useful is pathlib if it can't be used with the stdlib? What about C functions? Will they be reasonable to convert them to accept Path instances too? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From steve at pearwood.info Tue Oct 7 05:30:19 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 7 Oct 2014 14:30:19 +1100 Subject: [Python-ideas] Idea to support lazy loaded names. In-Reply-To: <20141006213416.21bdb347@allen-pc> References: <20141006213416.21bdb347@allen-pc> Message-ID: <20141007033019.GA24603@ando.pearwood.info> On Mon, Oct 06, 2014 at 09:34:16PM -0400, Brian Allen Vanderburg II wrote: > > When developing Python libraries (and any other language for that > matter), items are separated into modules for organization and other > purpose. The following is an example fake framework to illustrate this: [...] > One idea I had was to support lazy imported names. I've seen some > frameworks implement this in various means, and figured the idea is good > to be part Python. > > The idea is that for a new module attribute to exist: __lazyload__. > During the access of any attribute for a module, the following would > happen: This is a specific example of a more general technique, namely computed attributes. Rather than a solution to the specific example, I would rather have a general solution to the problem of having computed attributes in modules. We have half a solution already: the descriptor protocol, the most obvious example being property(). Well, perhaps a quarter of a solution. Assuming property(), or some equivalent, worked in modules, the solution to lazy loading is simple: have a getter that returns the attribute if it already exists, otherwise initialise it (importing it from elsewhere) then return it. The problem then becomes, how do we get descriptors to be run when accessing them via module attributes? The normal process is that if a name is found in the instance __dict__, the descriptor protocol is not followed: py> class Test(object): ... spam = property(lambda self: 23) ... def __init__(self): ... self.eggs = property(lambda self: 42) ... py> x = Test() py> x.spam 23 py> x.eggs This is relevant because modules are instances, and module attributes live in the instance __dict__. > * If the named attribute already exists, use it > * If the named attribute does not already exist: > * If a lazy load of the name has already been attempted, result in > a NameError > * If a lazy load of the name has not yet been attempted: > * Check the __lazyload__ module attribute for the name, perform > the loading operation and assign the module attribute the > value if found or result in a NameError > * Even if not found, set a flag that lazy load has been > attempted so it will not be attempted again for the same name I don't think that you should record that the import has been tried and failed. Regular imports cache successes, not failures, and lazy imports should do the same: just because an import failed a minute ago doesn't mean that it will fail now. -- Steven From tjreedy at udel.edu Tue Oct 7 05:51:45 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 06 Oct 2014 23:51:45 -0400 Subject: [Python-ideas] Idea to support lazy loaded names. In-Reply-To: <20141007033019.GA24603@ando.pearwood.info> References: <20141006213416.21bdb347@allen-pc> <20141007033019.GA24603@ando.pearwood.info> Message-ID: On 10/6/2014 11:30 PM, Steven D'Aprano wrote: > This is a specific example of a more general technique, namely computed > attributes. Rather than a solution to the specific example, I would > rather have a general solution to the problem of having computed > attributes in modules. We have half a solution already: the descriptor > protocol, the most obvious example being property(). Well, perhaps a > quarter of a solution. > > Assuming property(), or some equivalent, worked in modules, the solution > to lazy loading is simple: have a getter that returns the attribute if > it already exists, otherwise initialise it (importing it from elsewhere) > then return it. > > The problem then becomes, how do we get descriptors to be run when > accessing them via module attributes? The normal process is that if a > name is found in the instance __dict__, the descriptor protocol is not > followed: > > py> class Test(object): > ... spam = property(lambda self: 23) > ... def __init__(self): > ... self.eggs = property(lambda self: 42) > ... > py> x = Test() > py> x.spam > 23 > py> x.eggs > > > > This is relevant because modules are instances, and module attributes > live in the instance __dict__. So a 'solution' might be to make modules be instances (but with no __new__ or __init__) of a module metaclass, so that module dicts could act like class dicts with respect to descriptors. I have no idea how much code this would break ;-). -- Terry Jan Reedy From steve at pearwood.info Tue Oct 7 05:54:27 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 7 Oct 2014 14:54:27 +1100 Subject: [Python-ideas] Classes inside functions [was Re: Add __parent__ to all classes, functions, and modules] In-Reply-To: References: <20141006023628.GN19757@ando.pearwood.info> Message-ID: <20141007035427.GB24603@ando.pearwood.info> On Mon, Oct 06, 2014 at 10:23:42AM -0700, Guido van Rossum wrote: > How namedtuple is implemented should be nobody's business, except > Raymond's, and it certainly isn't a pattern to be recommended. That's why > it's in the stdlib -- so you don't have to write such code yourself. Same > for enums. Yes, it *can* be done. But it takes superhuman skills to get it > right and it still won't be maintainable (TBH, every time I see the > namedtuple implementation I have to resist the urge to rewrite it. :-) Heh :-) I was curious to see how much code in namedtuple actually needed to be run through exec, so I forked Raymond's recipe on ActiveState, and re-wrote with the Inner class as a regular nested class, and only a two-line __new__ method needing to be run through exec. http://code.activestate.com/recipes/578918-yet-another-namedtuple/ It passes the minimal tests in the recipe. I haven't taken it any further. -- Steven From steve at pearwood.info Tue Oct 7 06:11:13 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 7 Oct 2014 15:11:13 +1100 Subject: [Python-ideas] Idea to support lazy loaded names. In-Reply-To: References: <20141006213416.21bdb347@allen-pc> <20141007033019.GA24603@ando.pearwood.info> Message-ID: <20141007041113.GC24603@ando.pearwood.info> On Mon, Oct 06, 2014 at 11:51:45PM -0400, Terry Reedy wrote: > On 10/6/2014 11:30 PM, Steven D'Aprano wrote: [...] > >This is relevant because modules are instances, and module attributes > >live in the instance __dict__. > > So a 'solution' might be to make modules be instances (but with no > __new__ or __init__) of a module metaclass, so that module dicts could > act like class dicts with respect to descriptors. I have no idea how > much code this would break ;-). Probably lots :-) It's not common to write something like this outside of a class: x = property(some_function) but if you did, you would normally expect lookups on x to return the property object itself, not call the __get__ method. That's what happens now, and so by backward compatibility that can't change. More importantly, defining top-level functions is ubiquitous in Python code. And functions obey the descriptor protocol! That's how functions magically turn into methods when put inside a class. So if modules called __get__ by default, every single function lookup would break. That would be bad. So any change to the behaviour of module lookups would need to be opt-in. -- Steven From abarnert at yahoo.com Tue Oct 7 06:12:56 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 6 Oct 2014 21:12:56 -0700 Subject: [Python-ideas] Idea to support lazy loaded names. In-Reply-To: References: <20141006213416.21bdb347@allen-pc> <20141007033019.GA24603@ando.pearwood.info> Message-ID: <5A211262-A3DB-4A7F-924D-7C54F96B5F3A@yahoo.com> On Oct 6, 2014, at 20:51, Terry Reedy wrote: > On 10/6/2014 11:30 PM, Steven D'Aprano wrote: > >> This is a specific example of a more general technique, namely computed >> attributes. Rather than a solution to the specific example, I would >> rather have a general solution to the problem of having computed >> attributes in modules. We have half a solution already: the descriptor >> protocol, the most obvious example being property(). Well, perhaps a >> quarter of a solution. >> >> Assuming property(), or some equivalent, worked in modules, the solution >> to lazy loading is simple: have a getter that returns the attribute if >> it already exists, otherwise initialise it (importing it from elsewhere) >> then return it. >> >> The problem then becomes, how do we get descriptors to be run when >> accessing them via module attributes? The normal process is that if a >> name is found in the instance __dict__, the descriptor protocol is not >> followed: >> >> py> class Test(object): >> ... spam = property(lambda self: 23) >> ... def __init__(self): >> ... self.eggs = property(lambda self: 42) >> ... >> py> x = Test() >> py> x.spam >> 23 >> py> x.eggs >> >> >> >> This is relevant because modules are instances, and module attributes >> live in the instance __dict__. > > So a 'solution' might be to make modules be instances (but with no __new__ or __init__) of a module metaclass, so that module dicts could act like class dicts with respect to descriptors. I have no idea how much code this would break ;-). Didn't we just have this discussion a few weeks ago, in the context of making lazy loading of subpackages easier to implement? IIRC, the not-obviously-unreasonable options suggested were: 1) Analogy with __new__: For packages only, if there's a __new__.py, that gets executed first. If it "returns" (not sure how that was defined) an instance of a subclass of ModuleType, that instance is used to run __init__.py instead of a normal module instance. 2) Analogy with metaclass= (or with 2.x __metaclass__): If a module (or a package's __init__.py) does some new syntax or magic comment before any non-comment code, it can specify a custom type in place of ModuleType (not sure how that type gets imported and made available). 3) Analogy with re-classing object instances: Just allow modules to set __class__ during execution (or after, if you want). Variations on this include allowing that for all non-heap types, or even getting rid of the concept of non-heap types. 4) Make it easier to write import hooks for this special purpose. 5) Make it easier for a module to construct a ModuleType-subclass instance with a copy of the same dict and replace itself in sys.modules. It seems like any of these would allow defining properties, other descriptors, and special methods on modules. From ram at rachum.com Tue Oct 7 09:00:07 2014 From: ram at rachum.com (Ram Rachum) Date: Tue, 7 Oct 2014 10:00:07 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: Use case not relevant enough for you? On Tue, Oct 7, 2014 at 3:40 AM, Guido van Rossum wrote: > I think I just lost interest in this thread. > > On Mon, Oct 6, 2014 at 1:41 PM, Ram Rachum wrote: > >> On Mon, Oct 6, 2014 at 8:35 PM, Guido van Rossum >> wrote: >> >>> On Mon, Oct 6, 2014 at 1:10 AM, Ram Rachum wrote: >>> >>>> Here are a couple: >>>> >>>> - I'm making a combinatorics package, and a combination space needs to >>>> have a __contains__ method that takes a combination and returns whether >>>> it's part of a set. Since a combination, unlike a permutation, has no >>>> order, I like to have my combinations be canonicalized in a sorted order. >>>> For example, in a combination space of 3 items on range(4), (0, 1, 3) would >>>> be a combination in that space, but (3, 1, 0) would not because it's not >>>> sorted. (It's a k-permutation but not a combination.) However, if the user >>>> does `{0, 1, 3} in comb_space` I still want to return True, regardless of >>>> whether the set iterator happens to give these items in order or not. >>>> >>> >>> So how are you writing this code today? In the following case, what's in >>> the then or else branch? >>> >>> if not isinstance(x, collections.Ordered): >>> >>> else: >>> >>> >>> Even if you could write this, how would you know that an ordered >>> argument is in the *canonical* order? >>> >> >> That part isn't written yet (the package is still work-in-progress), but >> I'm not sure what the problem is. I'll have the code look at `x`. If it's >> marked as ordered, then I'd iterate on it. If it has the correct items >> (meaning the items of the sequence that this is a combination space of) and >> the items in x are in the same order as they are in the sequence, and it >> has the correct number of items, then we have a match. If we have `not >> isinstance(x, collections.Ordered)` then I do the same thing except I >> ignore the order of the items. What's the problem? >> >> >>> >>> - For the same package, I'm defining `Tally` and `OrderedTally` >>>> classes. (Simliar to `Counter` except without having an identity crisis >>>> between being a dict subclass and counter; mine are strictly counters.) In >>>> the tests I want to easily see whether the class I'm testing is the ordered >>>> one or not, so I'll know to run appropriate tests. (There are also >>>> `FrozenTally` and `FrozenOrderedTally` so it's not just one class.) I could >>>> set `is_ordered = True` on them or give them some base class, but I think a >>>> general `collections.Ordered` abstract base class would be the best >>>> solution. >>>> >>> >>> Same question. >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 7 11:34:30 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Oct 2014 11:34:30 +0200 Subject: [Python-ideas] Better stdlib support for Path objects References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> Message-ID: <20141007113430.7af4ae2d@fsol> On Mon, 6 Oct 2014 23:30:01 -0400 Donald Stufft wrote: > > > On Oct 6, 2014, at 9:38 PM, Barry Warsaw wrote: > > > > On Oct 06, 2014, at 11:19 PM, Georg Brandl wrote: > > > >> This is like the situation with context managers. We've taken about 3-4 > >> minor releases to add "with" support to objects that can logically support > >> it. Nobody remembers this, so people have to refer to the docs (or the code) > >> to see if and when e.g. smtplib.SMTP gained "with" support. > >> > >> However, the context managers are a few dozen classes at most. With paths, > >> there are hundreds of APIs that would have to be updated to take Paths > >> in the stdlib alone. Granted, a good portion would probably work fine since > >> they only pass through paths to lower level APIs, but still every one has to > >> be checked. Going by precedent, that's not something that we would be able > >> to do consistently, even throughout several releases. (Another precedent is > >> Argument Clinic.) > > > > I appreciate that this is a problem with such transitions. Is it an argument > > for never doing so though? Is it better that the stdlib prohibit adoption of > > advanced features and libraries than to do so piecemeal? I'm not so sure; > > even though it's admittedly annoying when such support is missing, it's really > > nice when they're there. > > > > How useful is pathlib if it can't be used with the stdlib? > > What about C functions? Will they be reasonable to convert them to accept Path instances too? IMO it's reasonable, but assuming we devise a dedicated protocol for getting a path's representation (e.g. __strpath__). Concrete type checking should be avoided. I hope someone (Barry? :-)) can take the time to think it out. Regards Antoine. From ncoghlan at gmail.com Tue Oct 7 11:48:23 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Oct 2014 19:48:23 +1000 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141006142431.56ddba78@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> Message-ID: On 7 Oct 2014 04:26, "Barry Warsaw" wrote: > > On Oct 06, 2014, at 11:04 AM, Guido van Rossum wrote: > > >You can construct a Path from an argument that can be either a string or > >another Path. Example: > > > >>>> from pathlib import Path > >>>> p = Path('/etc/passwd') > >>>> q = Path(p) > >>>> p == q > >True > >>>> > > > >So you could start refactoring stdlib code to use Path internally without > >forcing the callers to use Path, but still *allow* the callers to pass a > >Path. Though I'm not sure how this would work for return values without > >breaking backwards compatibility -- you'd have to keep returning strings > >and the callers would have to use the same mechanism to go back to using > >Paths. > > That's a very interesting perspective, and one I'd like to pursue further. pathlib is quite high level, so there's a chance of introducing undesirable circular dependencies in introducing it too broadly. With ipaddress and, as far as I am aware, pathlib, the intent was for it to be useful as a manipulation library, but to drop back to a serialised representation for transfer to other libraries (including the rest of the stdlib). This helps avoid the monolithic object model coupling that tends to pervade large Java applications. If the current spelling of that is too verbose/awkward/error prone, then I'd prefer to see that tackled directly (e.g. by introducing some appropriate calculated properties), rather than switching to the highly coupled all pervasive standard library change we were trying to avoid. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Oct 7 11:54:36 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Oct 2014 19:54:36 +1000 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> Message-ID: On 7 Oct 2014 19:48, "Nick Coghlan" wrote: > > > On 7 Oct 2014 04:26, "Barry Warsaw" wrote: > > > > On Oct 06, 2014, at 11:04 AM, Guido van Rossum wrote: > > > > >You can construct a Path from an argument that can be either a string or > > >another Path. Example: > > > > > >>>> from pathlib import Path > > >>>> p = Path('/etc/passwd') > > >>>> q = Path(p) > > >>>> p == q > > >True > > >>>> > > > > > >So you could start refactoring stdlib code to use Path internally without > > >forcing the callers to use Path, but still *allow* the callers to pass a > > >Path. Though I'm not sure how this would work for return values without > > >breaking backwards compatibility -- you'd have to keep returning strings > > >and the callers would have to use the same mechanism to go back to using > > >Paths. > > > > That's a very interesting perspective, and one I'd like to pursue further. > > pathlib is quite high level, so there's a chance of introducing undesirable circular dependencies in introducing it too broadly. > > With ipaddress and, as far as I am aware, pathlib, the intent was for it to be useful as a manipulation library, but to drop back to a serialised representation for transfer to other libraries (including the rest of the stdlib). This helps avoid the monolithic object model coupling that tends to pervade large Java applications. > > If the current spelling of that is too verbose/awkward/error prone, then I'd prefer to see that tackled directly (e.g. by introducing some appropriate calculated properties), rather than switching to the highly coupled all pervasive standard library change we were trying to avoid. Note that a path protocol (with appropriate C API support) would also address this concern with excessive coupling to a specific concrete type. A single dispatch generic function as an adapter API would be another option, but would likely pose bootstrapping problems for the lowest level interfaces like os.path and the open builtin. Cheers, Nick. > > Regards, > Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cathalgarvey at cathalgarvey.me Tue Oct 7 12:02:01 2014 From: cathalgarvey at cathalgarvey.me (Cathal Garvey) Date: Tue, 07 Oct 2014 11:02:01 +0100 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: <5433BA19.1090005@cathalgarvey.me> If you haven't written the code yet, how can you say there's something missing in Python that needs to be implemented, at significant time and cost, by the Python developers? You need to provide your best attempt at implementing this in Python as-is, and identify where the language's design or gaps in the standard libraries are creating a problem for you and others that's significant enough to warrant a change to Python's stdlib. Or, show how solving your problem would create a language that's better, more exciting, or more fun for the rest of us. Right now, your use-case seems really specific, and is likely something that you can solve just by exhaustively listing the types you want to consider "ordered" as a global tuple and using that tuple as your isinstance() check. So, `isinstance(foo, (collections.OrderedDict, tuple, list))` or whatever. Although, actually it sounds like you need to know more than whether an object has a defined order, but also whether it's in the order you expect, so really this calls for a dedicated checking function that does all the magic you want at once. On 07/10/14 08:00, Ram Rachum wrote: > Use case not relevant enough for you? > > On Tue, Oct 7, 2014 at 3:40 AM, Guido van Rossum wrote: > >> I think I just lost interest in this thread. >> >> On Mon, Oct 6, 2014 at 1:41 PM, Ram Rachum wrote: >> >>> On Mon, Oct 6, 2014 at 8:35 PM, Guido van Rossum >>> wrote: >>> >>>> On Mon, Oct 6, 2014 at 1:10 AM, Ram Rachum wrote: >>>> >>>>> Here are a couple: >>>>> >>>>> - I'm making a combinatorics package, and a combination space needs to >>>>> have a __contains__ method that takes a combination and returns whether >>>>> it's part of a set. Since a combination, unlike a permutation, has no >>>>> order, I like to have my combinations be canonicalized in a sorted order. >>>>> For example, in a combination space of 3 items on range(4), (0, 1, 3) would >>>>> be a combination in that space, but (3, 1, 0) would not because it's not >>>>> sorted. (It's a k-permutation but not a combination.) However, if the user >>>>> does `{0, 1, 3} in comb_space` I still want to return True, regardless of >>>>> whether the set iterator happens to give these items in order or not. >>>>> >>>> >>>> So how are you writing this code today? In the following case, what's in >>>> the then or else branch? >>>> >>>> if not isinstance(x, collections.Ordered): >>>> >>>> else: >>>> >>>> >>>> Even if you could write this, how would you know that an ordered >>>> argument is in the *canonical* order? >>>> >>> >>> That part isn't written yet (the package is still work-in-progress), but >>> I'm not sure what the problem is. I'll have the code look at `x`. If it's >>> marked as ordered, then I'd iterate on it. If it has the correct items >>> (meaning the items of the sequence that this is a combination space of) and >>> the items in x are in the same order as they are in the sequence, and it >>> has the correct number of items, then we have a match. If we have `not >>> isinstance(x, collections.Ordered)` then I do the same thing except I >>> ignore the order of the items. What's the problem? >>> >>> >>>> >>>> - For the same package, I'm defining `Tally` and `OrderedTally` >>>>> classes. (Simliar to `Counter` except without having an identity crisis >>>>> between being a dict subclass and counter; mine are strictly counters.) In >>>>> the tests I want to easily see whether the class I'm testing is the ordered >>>>> one or not, so I'll know to run appropriate tests. (There are also >>>>> `FrozenTally` and `FrozenOrderedTally` so it's not just one class.) I could >>>>> set `is_ordered = True` on them or give them some base class, but I think a >>>>> general `collections.Ordered` abstract base class would be the best >>>>> solution. >>>>> >>>> >>>> Same question. >>>> >>>> -- >>>> --Guido van Rossum (python.org/~guido) >>>> >>> >>> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Twitter: @onetruecathal, @formabiolabs Phone: +353876363185 Blog: http://indiebiotech.com miniLock.io: JjmYYngs7akLZUjkvFkuYdsZ3PyPHSZRBKNm6qTYKZfAM -------------- next part -------------- A non-text attachment was scrubbed... Name: 0x988B9099.asc Type: application/pgp-keys Size: 6176 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From mal at egenix.com Tue Oct 7 12:28:01 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Oct 2014 12:28:01 +0200 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> Message-ID: <5433C031.1030008@egenix.com> On 07.10.2014 11:48, Nick Coghlan wrote: > On 7 Oct 2014 04:26, "Barry Warsaw" wrote: >> >> On Oct 06, 2014, at 11:04 AM, Guido van Rossum wrote: >> >>> You can construct a Path from an argument that can be either a string or >>> another Path. Example: >>> >>>>>> from pathlib import Path >>>>>> p = Path('/etc/passwd') >>>>>> q = Path(p) >>>>>> p == q >>> True >>>>>> >>> >>> So you could start refactoring stdlib code to use Path internally without >>> forcing the callers to use Path, but still *allow* the callers to pass a >>> Path. Though I'm not sure how this would work for return values without >>> breaking backwards compatibility -- you'd have to keep returning strings >>> and the callers would have to use the same mechanism to go back to using >>> Paths. >> >> That's a very interesting perspective, and one I'd like to pursue further. > > pathlib is quite high level, so there's a chance of introducing undesirable > circular dependencies in introducing it too broadly. > > With ipaddress and, as far as I am aware, pathlib, the intent was for it to > be useful as a manipulation library, but to drop back to a serialised > representation for transfer to other libraries (including the rest of the > stdlib). This helps avoid the monolithic object model coupling that tends > to pervade large Java applications. > > If the current spelling of that is too verbose/awkward/error prone, then > I'd prefer to see that tackled directly (e.g. by introducing some > appropriate calculated properties), rather than switching to the highly > coupled all pervasive standard library change we were trying to avoid. The approach to use pathlib internally in the stdlib while making sure that callers will get strings as return values should work fine. We've been using a similar approach with mxURL in some of our application server code. mxURL which provides a parsed URL object that implements common tasks such as joining URLs, rebuilding, etc. The approach makes code more readable, you get the option of passing in a string or an already parsed URL object (saving some overhead) and code using the APIs get strings which prevents other code from complaining about a wrong type. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Tue Oct 7 13:02:21 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Oct 2014 21:02:21 +1000 Subject: [Python-ideas] Classes inside functions [was Re: Add __parent__ to all classes, functions, and modules] In-Reply-To: <20141007035427.GB24603@ando.pearwood.info> References: <20141006023628.GN19757@ando.pearwood.info> <20141007035427.GB24603@ando.pearwood.info> Message-ID: On 7 October 2014 13:54, Steven D'Aprano wrote: > On Mon, Oct 06, 2014 at 10:23:42AM -0700, Guido van Rossum wrote: > >> How namedtuple is implemented should be nobody's business, except >> Raymond's, and it certainly isn't a pattern to be recommended. That's why >> it's in the stdlib -- so you don't have to write such code yourself. Same >> for enums. Yes, it *can* be done. But it takes superhuman skills to get it >> right and it still won't be maintainable (TBH, every time I see the >> namedtuple implementation I have to resist the urge to rewrite it. :-) > > Heh :-) > > I was curious to see how much code in namedtuple actually needed to be > run through exec, so I forked Raymond's recipe on ActiveState, and > re-wrote with the Inner class as a regular nested class, and only a > two-line __new__ method needing to be run through exec. > > http://code.activestate.com/recipes/578918-yet-another-namedtuple/ > > It passes the minimal tests in the recipe. I haven't taken it any > further. One nice thing about the current impl is you can actually dump the full source code of the underlying class implementation. That ties in to another nice property, which is that anyone that understands class definitions and exec could maintain most of the current implementation. It would be tough to write it from scratch (since there are some subtle details to get right), but it's still a much lower knowledge barrier to get over than is the case with the corresponding dynamic type creation code. By contrast, writing it out directly means reimplementing several things the interpreter will otherwise take care of, which also means you may not automatically benefit from changes to the way class definitions work. For example, that recipe version doesn't appear to adjust __qualname__ at all, while namedtuple (without any changes to the namedtuple implementation) automatically sets it to match __name__. Maintainability is a complicated thing, and our instincts don't always get the trade-offs right :) Cheers, Nick. P.S. Despite everything I said above, in most cases, dynamic type creation is still going to be a better idea. namedtuple itself just isn't one of those cases. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Oct 7 13:26:14 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Oct 2014 21:26:14 +1000 Subject: [Python-ideas] Idea to support lazy loaded names. In-Reply-To: References: <20141006213416.21bdb347@allen-pc> <20141007033019.GA24603@ando.pearwood.info> Message-ID: On 7 October 2014 13:51, Terry Reedy wrote: > On 10/6/2014 11:30 PM, Steven D'Aprano wrote: > >> This is relevant because modules are instances, and module attributes >> live in the instance __dict__. > > So a 'solution' might be to make modules be instances (but with no __new__ > or __init__) of a module metaclass, so that module dicts could act like > class dicts with respect to descriptors. I have no idea how much code this > would break ;-). As Steven noted, making such a change by default would actually break the world, since every module level function would break when it started being treated as a method instead: >>> def f(): pass ... >>> hasattr(f, "__get__") True Anything related to changing module attribute lookup also needs to deal with the fact that manipulating the module namespace via globals() and the "global" directive is a fully supported feature. This is why models which put something that *isn't* a normal module (such as an ordinary class object) in sys.modules tend to be preferred. That approach is already fully supported, since the import system *retrieves the result from sys.modules*, rather than trusting what the import hooks return - that gives the module the ability to replace itself in sys.modules as a side effect of the import process. For example, here's a rough (untested) sketch of one possible way to do a lazily calculated module attribute today: # rest of the module ... class _ThisModule: @property def calculated_attribute(self): return 42 _ThisModule.__dict__.update(globals()) for k, v in _ThisModule.__dict__.items(): if callable(v): setattr(_ThisModule, k, staticmethod(v)) import sys sys.modules[__name__] = _ThisModule() You could potentially wrap most of that dance up in a "replace_module" helper function and stick it somewhere in importlib. Cheers, Nick. P.S. This is a variant of approach #5 from Andrew's list. Approach #3 would allow code along the lines of "__class__ = _ThisModule", potentially simplifying the situation even further than a helper function could. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wolfgang.maier at biologie.uni-freiburg.de Tue Oct 7 14:33:49 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 07 Oct 2014 14:33:49 +0200 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141006195813.284d7cd3@fsol> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006195813.284d7cd3@fsol> Message-ID: <5433DDAD.20904@biologie.uni-freiburg.de> On 10/06/2014 07:58 PM, Antoine Pitrou wrote: > On Mon, 6 Oct 2014 13:47:14 -0400 > Barry Warsaw wrote: >> >> One simple solution would be to sprinkle str() calls in various stdlib >> methods, but I'm not sure if that would fail miserably in the face of bytes >> paths (if the original stdlib API even accepts bytes paths). The suggestion >> in the issue is to add a "path protocol" and the referenced article suggests >> .strpath and .bytespath. OTOH, isn't str() and bytes() enough? > > str() works on anything (including enums and distutils compilers), so > it's pretty bad IMO. > If that's a concern, couldn't that be solved with an abstract base class "Path" ? pathlib path classes could be inheriting from it, while modules like os, json and others that wish to accept also strings could register str, then call str() on all input that passes an isinstance(input, Path) check, raise an error otherwise. The remaining question then would be where the abstract base class should live. Wolfgang From barry at python.org Tue Oct 7 15:17:31 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 7 Oct 2014 09:17:31 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> Message-ID: <20141007091731.5b414848@anarchist.wooz.org> On Oct 07, 2014, at 07:54 PM, Nick Coghlan wrote: >A single dispatch generic function as an adapter API would be another >option, but would likely pose bootstrapping problems for the lowest level >interfaces like os.path and the open builtin. I wouldn't expect low level APIs like os.path and built-in open to accept Path objects. pathlib already covers most of those use cases, and whatever is missing from there can probably be easily added. It's higher level libraries accepting Path objects that is more interesting I think. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From donald at stufft.io Tue Oct 7 15:20:11 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 7 Oct 2014 09:20:11 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141007091731.5b414848@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> <20141007091731.5b414848@anarchist.wooz.org> Message-ID: <47963623-4702-4F07-A65C-37B394CAF12B@stufft.io> > On Oct 7, 2014, at 9:17 AM, Barry Warsaw wrote: > > On Oct 07, 2014, at 07:54 PM, Nick Coghlan wrote: > >> A single dispatch generic function as an adapter API would be another >> option, but would likely pose bootstrapping problems for the lowest level >> interfaces like os.path and the open builtin. > > I wouldn't expect low level APIs like os.path and built-in open to accept Path > objects. pathlib already covers most of those use cases, and whatever is > missing from there can probably be easily added. It's higher level libraries > accepting Path objects that is more interesting I think. > > Cheers, > -Barry TBH I find a path library that I can?t use anywhere I use paths to be rather neutered. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From random832 at fastmail.us Tue Oct 7 15:25:47 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Tue, 07 Oct 2014 09:25:47 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <47963623-4702-4F07-A65C-37B394CAF12B@stufft.io> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> <20141007091731.5b414848@anarchist.wooz.org> <47963623-4702-4F07-A65C-37B394CAF12B@stufft.io> Message-ID: <1412688347.2708323.176114633.7B901459@webmail.messagingengine.com> On Tue, Oct 7, 2014, at 09:20, Donald Stufft wrote: > > On Oct 7, 2014, at 9:17 AM, Barry Warsaw wrote: > > On Oct 07, 2014, at 07:54 PM, Nick Coghlan wrote: > >> A single dispatch generic function as an adapter API would be another > >> option, but would likely pose bootstrapping problems for the lowest level > >> interfaces like os.path and the open builtin. > > > > I wouldn't expect low level APIs like os.path and built-in open to accept Path > > objects. pathlib already covers most of those use cases, and whatever is > > missing from there can probably be easily added. It's higher level libraries > > accepting Path objects that is more interesting I think. > > TBH I find a path library that I can?t use anywhere I use paths to be > rather neutered. It makes about as much sense for os.open to accept a path as it would for it to return a TextIOWrapper instead of an int. The same applies, in reverse, to built-in open. From donald at stufft.io Tue Oct 7 15:29:20 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 7 Oct 2014 09:29:20 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <1412688347.2708323.176114633.7B901459@webmail.messagingengine.com> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> <20141007091731.5b414848@anarchist.wooz.org> <47963623-4702-4F07-A65C-37B394CAF12B@stufft.io> <1412688347.2708323.176114633.7B901459@webmail.messagingengine.com> Message-ID: > On Oct 7, 2014, at 9:25 AM, random832 at fastmail.us wrote: > > On Tue, Oct 7, 2014, at 09:20, Donald Stufft wrote: >>> On Oct 7, 2014, at 9:17 AM, Barry Warsaw wrote: >>> On Oct 07, 2014, at 07:54 PM, Nick Coghlan wrote: >>>> A single dispatch generic function as an adapter API would be another >>>> option, but would likely pose bootstrapping problems for the lowest level >>>> interfaces like os.path and the open builtin. >>> >>> I wouldn't expect low level APIs like os.path and built-in open to accept Path >>> objects. pathlib already covers most of those use cases, and whatever is >>> missing from there can probably be easily added. It's higher level libraries >>> accepting Path objects that is more interesting I think. >> >> TBH I find a path library that I can?t use anywhere I use paths to be >> rather neutered. > > It makes about as much sense for os.open to accept a path as it would > for it to return a TextIOWrapper instead of an int. The same applies, in > reverse, to built-in open. I might agree with you if there wasn?t 20 years of code that expects to be able to pass a ?path? in to various places. Having to keep a mental note, or worse look up in the documentation every time, where I?m expected to pass a Path object and where I?m expected to pass a str is just about the worst UX I can imagine. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From barry at python.org Tue Oct 7 15:37:10 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 7 Oct 2014 09:37:10 -0400 Subject: [Python-ideas] Better stdlib support for Path objects References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> Message-ID: <20141007093710.6cb7a44b@anarchist.wooz.org> On Oct 07, 2014, at 11:34 AM, Antoine Pitrou wrote: >IMO it's reasonable, but assuming we devise a dedicated protocol for >getting a path's representation (e.g. __strpath__). Concrete type >checking should be avoided. What would __strpath__ do that __str__ wouldn't do? Or do you think it's better to explicitly disallow str-like objects that aren't path objects? What I'm trying to understand is whether str(argument) is that "path protocol" or whether there's a good reason that something else that's specifically not str-ification is required. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From apalala at gmail.com Tue Oct 7 15:25:11 2014 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Tue, 7 Oct 2014 08:55:11 -0430 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <5433DDAD.20904@biologie.uni-freiburg.de> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006195813.284d7cd3@fsol> <5433DDAD.20904@biologie.uni-freiburg.de> Message-ID: On Tue, Oct 7, 2014 at 8:03 AM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > then call str() on all input that passes an isinstance(input, Path) check, > raise an error otherwise. Calling: path=str(path) within all API functions that accept file paths is a good enough solution, and one that doesn't change dependencies in stdlib. Most of the API functions I've had to call with str(path) don't return another path, but some more-structured else (json, csv, configparser, ...). Maintainers could silently start adding the path=str(path) prolog to the API so declaring official support for Path can be postponed until it is certified that there's 100% coverage in stdlib. Cheers, -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 7 15:44:59 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Oct 2014 15:44:59 +0200 Subject: [Python-ideas] Better stdlib support for Path objects References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> Message-ID: <20141007154459.6dbbf05a@fsol> On Tue, 7 Oct 2014 09:37:10 -0400 Barry Warsaw wrote: > On Oct 07, 2014, at 11:34 AM, Antoine Pitrou wrote: > > >IMO it's reasonable, but assuming we devise a dedicated protocol for > >getting a path's representation (e.g. __strpath__). Concrete type > >checking should be avoided. > > What would __strpath__ do that __str__ wouldn't do? Or do you think it's > better to explicitly disallow str-like objects that aren't path objects? The latter. > What I'm trying to understand is whether str(argument) is that "path protocol" > or whether there's a good reason that something else that's specifically not > str-ification is required. Every object str-ifies successfully. IMO that's a good enough reason ;) Regards Antoine. From rosuav at gmail.com Tue Oct 7 15:45:41 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 8 Oct 2014 00:45:41 +1100 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141007093710.6cb7a44b@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> Message-ID: On Wed, Oct 8, 2014 at 12:37 AM, Barry Warsaw wrote: > On Oct 07, 2014, at 11:34 AM, Antoine Pitrou wrote: > >>IMO it's reasonable, but assuming we devise a dedicated protocol for >>getting a path's representation (e.g. __strpath__). Concrete type >>checking should be avoided. > > What would __strpath__ do that __str__ wouldn't do? Or do you think it's > better to explicitly disallow str-like objects that aren't path objects? Currently, __str__ will happily return a string representation of basically anything. More generally than __strpath__, maybe what's needed is a method "give me a str that accurately represents this object", to be provided only by those types which are "virtually strings". It'd be an alternative to subclassing str, as a means of saying "I'm a string!". Or maybe this should be done as a collections.abc - there's ByteString, maybe there should be TextString. Then anything that registers itself as a TextString should be assumed to function as a string, and a path could just register itself. Would that make more sense than __strpath__ does? ChrisA From edk141 at gmail.com Tue Oct 7 15:45:59 2014 From: edk141 at gmail.com (Ed Kellett) Date: Tue, 7 Oct 2014 14:45:59 +0100 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141007093710.6cb7a44b@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> Message-ID: On 7 October 2014 14:37, Barry Warsaw wrote: > What would __strpath__ do that __str__ wouldn't do? Or do you think it's > better to explicitly disallow str-like objects that aren't path objects? What is a str-like object? A lot of objects are acceptable to str(); most of them aren't "str-like" in any reasonable sense of the term (e.g. function, int), and probably shouldn't be acceptable as paths. edk From barry at python.org Tue Oct 7 15:47:31 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 7 Oct 2014 09:47:31 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> Message-ID: <20141007094731.445d0936@anarchist.wooz.org> On Oct 07, 2014, at 02:45 PM, Ed Kellett wrote: >On 7 October 2014 14:37, Barry Warsaw wrote: >> What would __strpath__ do that __str__ wouldn't do? Or do you think it's >> better to explicitly disallow str-like objects that aren't path objects? > >What is a str-like object? A lot of objects are acceptable to str(); >most of them aren't "str-like" in any reasonable sense of the term >(e.g. function, int), and probably shouldn't be acceptable as paths. Sorry, str-able is probably a better term to use in the above. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From p.f.moore at gmail.com Tue Oct 7 15:55:58 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Oct 2014 14:55:58 +0100 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141006134714.640fc3ca@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: On 6 October 2014 18:47, Barry Warsaw wrote: > I've tried to convert parts of > a medium sized Python 3 application from os.path to pathlib and found this > lack of support rather demotivating. Yes, it's fairly easy to wrap Path > objects in str() to pass them to stdlib methods that expect only strings, but > it's a lot of work in user code and I find that the resulting str()s are > distracting. It's a disincentive. I find it worse than a disincentive, it makes understanding the code perceptibly harder, which is a maintenance issue. Having an attribute that returns the string representation would be a substantial improvement (as it's the extra parentheses from the str call that I find the most distracting, that and the code smell that an "explicit cast" involves). Having more things accept Path objects would be good, but there will always be 3rd party libraries, as well as places where you want to pass a pathname that really *shouldn't* have to expect them. Paul From wolfgang.maier at biologie.uni-freiburg.de Tue Oct 7 16:00:38 2014 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 07 Oct 2014 16:00:38 +0200 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> Message-ID: On 07.10.2014 15:45, Chris Angelico wrote: > On Wed, Oct 8, 2014 at 12:37 AM, Barry Warsaw wrote: >> On Oct 07, 2014, at 11:34 AM, Antoine Pitrou wrote: >> >>> IMO it's reasonable, but assuming we devise a dedicated protocol for >>> getting a path's representation (e.g. __strpath__). Concrete type >>> checking should be avoided. >> >> What would __strpath__ do that __str__ wouldn't do? Or do you think it's >> better to explicitly disallow str-like objects that aren't path objects? > > Currently, __str__ will happily return a string representation of > basically anything. More generally than __strpath__, maybe what's > needed is a method "give me a str that accurately represents this > object", to be provided only by those types which are "virtually > strings". It'd be an alternative to subclassing str, as a means of > saying "I'm a string!". > > Or maybe this should be done as a collections.abc - there's > ByteString, maybe there should be TextString. Then anything that > registers itself as a TextString should be assumed to function as a > string, and a path could just register itself. > > Would that make more sense than __strpath__ does? > I think so, but I wouldn't call that ABC TextString, but Path since that's what it is and modules wishing to accept str in addition to pathlib objects should register str as a Path. Wolfgang From p.f.moore at gmail.com Tue Oct 7 16:08:38 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Oct 2014 15:08:38 +0100 Subject: [Python-ideas] Enhanced stat_result objects (was: Better stdlib support for Path objects) Message-ID: On 6 October 2014 18:47, Barry Warsaw wrote: > Over in issue 22570, I lament the fact that while pathlib is awesome, its > wider support in the stdlib is pretty sparse. The other thing that bugs me whenever I try to use pathlib is the non-caching behaviour of the stat-related methods. I know why this is as it is, and I agree with the reasons, but for some reason, I still find that I expect code like p = Path(...) if not p.exists(): print("Nonexistent") elif p.is_dir(): print("Directory") elif p.is_file(): print("File") else: print("Something else") to only call stat once - even though I *don't* expect that of the equivalent os.path based code. I would find it really useful if stat_result objects supported the various is_XXX methods from pathlib. Then I could write code like: p = Path(...) st = p.stat() if st.exists(): ... elif st.is_dir(): ... which explicitly shows that stat is only called once, but doesn't involve verbose and ugly code like if stat.S_ISDIR(st.st_mode): ... Would this be a worthwhile addition? Paul From barry at python.org Tue Oct 7 16:24:54 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 7 Oct 2014 10:24:54 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: <20141007102454.0a730024@anarchist.wooz.org> On Oct 07, 2014, at 02:55 PM, Paul Moore wrote: >I find it worse than a disincentive, it makes understanding the code >perceptibly harder, which is a maintenance issue. Having an attribute >that returns the string representation would be a substantial >improvement (as it's the extra parentheses from the str call that I >find the most distracting, that and the code smell that an "explicit >cast" involves). I realize there's another thing that bugs me about sprinkling str() calls all over the place, and this relates to my other question about whether str()-ability is "the path protocol". The problem is that if I'm looking at some random code and see: my_parser.load(str(path)) I really don't have any idea what 'path' is. Maybe that's a good thing, but in the few cases where I did this, it seemed bad. ;) OTOH, if I saw this, it would be a strong clue that path were a pathlib object: my_parser.load(path.string_path) substituting .string_path for whatever color the shed gets painted. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From edk141 at gmail.com Tue Oct 7 16:33:53 2014 From: edk141 at gmail.com (Ed Kellett) Date: Tue, 7 Oct 2014 15:33:53 +0100 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141007094731.445d0936@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> <20141007094731.445d0936@anarchist.wooz.org> Message-ID: On 7 October 2014 14:47, Barry Warsaw wrote: > On Oct 07, 2014, at 02:45 PM, Ed Kellett wrote: > >>On 7 October 2014 14:37, Barry Warsaw wrote: >>> What would __strpath__ do that __str__ wouldn't do? Or do you think it's >>> better to explicitly disallow str-like objects that aren't path objects? >> >>What is a str-like object? A lot of objects are acceptable to str(); >>most of them aren't "str-like" in any reasonable sense of the term >>(e.g. function, int), and probably shouldn't be acceptable as paths. > > Sorry, str-able is probably a better term to use in the above. > Yeah - but whatever the term, I think being able to str() something is too weak a predicate for using it as a path. If there were a way for an object to declare itself str-like in the sense that Paths are, that'd be much more reasonable, imv. From ncoghlan at gmail.com Tue Oct 7 16:40:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Oct 2014 00:40:07 +1000 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141007093710.6cb7a44b@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> Message-ID: On 7 October 2014 23:37, Barry Warsaw wrote: > On Oct 07, 2014, at 11:34 AM, Antoine Pitrou wrote: > >>IMO it's reasonable, but assuming we devise a dedicated protocol for >>getting a path's representation (e.g. __strpath__). Concrete type >>checking should be avoided. > > What would __strpath__ do that __str__ wouldn't do? Or do you think it's > better to explicitly disallow str-like objects that aren't path objects? > > What I'm trying to understand is whether str(argument) is that "path protocol" > or whether there's a good reason that something else that's specifically not > str-ification is required. It's mostly a matter of offering better failure modes - the reasons would be similar to why we added operator.index and __index__ in order to expand various APIs from "only int or long objects" without expanding them so far that they also accepted float objects or strings the way coercing via "int" would. Using str(x) implicitly allows lots of nonsense that should throw an immediate TypeError to instead throw OSError later on (or, worse, perhaps even appear to work). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From alexander.belopolsky at gmail.com Tue Oct 7 16:48:04 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 7 Oct 2014 10:48:04 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> Message-ID: On Tue, Oct 7, 2014 at 10:40 AM, Nick Coghlan wrote: > Using str(x) implicitly allows lots of nonsense that should throw an > immediate TypeError to instead throw OSError later on (or, worse, > perhaps even appear to work). > +1 It's no fun trying to figure out how a file named '' came about or how to remove it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Oct 7 16:49:12 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Oct 2014 00:49:12 +1000 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> Message-ID: On 8 October 2014 00:40, Nick Coghlan wrote: > On 7 October 2014 23:37, Barry Warsaw wrote: >> On Oct 07, 2014, at 11:34 AM, Antoine Pitrou wrote: >> >>>IMO it's reasonable, but assuming we devise a dedicated protocol for >>>getting a path's representation (e.g. __strpath__). Concrete type >>>checking should be avoided. >> >> What would __strpath__ do that __str__ wouldn't do? Or do you think it's >> better to explicitly disallow str-like objects that aren't path objects? >> >> What I'm trying to understand is whether str(argument) is that "path protocol" >> or whether there's a good reason that something else that's specifically not >> str-ification is required. > > It's mostly a matter of offering better failure modes - the reasons > would be similar to why we added operator.index and __index__ in order > to expand various APIs from "only int or long objects" without > expanding them so far that they also accepted float objects or strings > the way coercing via "int" would. > > Using str(x) implicitly allows lots of nonsense that should throw an > immediate TypeError to instead throw OSError later on (or, worse, > perhaps even appear to work). Combining this thought with Chris Angelico's reply, I actually wonder if the index vs int analogy is even more applicable than I first thought. What if the protocol was __text__ with a new text() builtin (or at least an operator.text() function), and it was advised to only be implemented by types where they were, at least conceptually, truly representable as strings? That's basically what was done with the __index__ method in http://www.python.org/dev/peps/pep-0357/ to introduce ducktyping to several APIs that previously only worked with builtin int/long objects. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Tue Oct 7 16:59:47 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 07 Oct 2014 23:59:47 +0900 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> <20141007091731.5b414848@anarchist.wooz.org> <47963623-4702-4F07-A65C-37B394CAF12B@stufft.io> <1412688347.2708323.176114633.7B901459@webmail.messagingengine.com> Message-ID: <87bnporkm4.fsf@uwakimon.sk.tsukuba.ac.jp> Donald Stufft writes: > > On Oct 7, 2014, at 9:25 AM, random832 at fastmail.us wrote: > > It makes about as much sense for os.open to accept a path as it would > > for it to return a TextIOWrapper instead of an int. The same applies, in > > reverse, to built-in open. > > I might agree with you if there wasn?t 20 years of code that > expects to be able to pass a ?path? in to various places. Having to > keep a mental note, or worse look up in the documentation every > time, where I?m expected to pass a Path object and where I?m > expected to pass a str is just about the worst UX I can imagine. You wouldn't need a mental memo. Just do "import os" rather than "from os import *" and the "os." prefix will tell you you need a str. Otherwise use a Path. That would be the goal, I expect. How to get there, I'm not sure. From abarnert at yahoo.com Tue Oct 7 17:16:24 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 7 Oct 2014 08:16:24 -0700 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> Message-ID: <9B204E1C-752A-48D9-8D88-1A8458197300@yahoo.com> On Oct 7, 2014, at 7:40, Nick Coghlan wrote: > On 7 October 2014 23:37, Barry Warsaw wrote: >> On Oct 07, 2014, at 11:34 AM, Antoine Pitrou wrote: >> >>> IMO it's reasonable, but assuming we devise a dedicated protocol for >>> getting a path's representation (e.g. __strpath__). Concrete type >>> checking should be avoided. >> >> What would __strpath__ do that __str__ wouldn't do? Or do you think it's >> better to explicitly disallow str-like objects that aren't path objects? >> >> What I'm trying to understand is whether str(argument) is that "path protocol" >> or whether there's a good reason that something else that's specifically not >> str-ification is required. > > It's mostly a matter of offering better failure modes - the reasons > would be similar to why we added operator.index and __index__ in order > to expand various APIs from "only int or long objects" without > expanding them so far that they also accepted float objects or strings > the way coercing via "int" would. > > Using str(x) implicitly allows lots of nonsense that should throw an > immediate TypeError to instead throw OSError later on (or, worse, > perhaps even appear to work). The "worse" case would be pretty common. Any API that writes files would just try to create a file named str(path), and on most POSIX platforms that would succeed for almost anything--bytes, int, TextIOWrapper, list of paths that you forgot to open in a listcomp, sequence of chars that you forgot to ''.join, function that you meant to call instead of just referencing, arbitrary generic-repr'd or constructor-repr'd instance, ... Most of my memories of Alpha (a Tcl-programmable classic Mac text editor) are of debugging and cleaning up after errors exactly like this. It would be even more fun for novices trying to figure out how to pass names with angle brackets and quotes to find or rm in a shell they barely know how to use... From donald at stufft.io Tue Oct 7 17:35:34 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 7 Oct 2014 11:35:34 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <87bnporkm4.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> <20141007091731.5b414848@anarchist.wooz.org> <47963623-4702-4F07-A65C-37B394CAF12B@stufft.io> <1412688347.2708323.176114633.7B901459@webmail.messagingengine.com> <87bnporkm4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <7CBAF05E-7E22-40D1-802B-0641C5074DFC@stufft.io> > On Oct 7, 2014, at 10:59 AM, Stephen J. Turnbull wrote: > > Donald Stufft writes: >>> On Oct 7, 2014, at 9:25 AM, random832 at fastmail.us wrote: > >>> It makes about as much sense for os.open to accept a path as it would >>> for it to return a TextIOWrapper instead of an int. The same applies, in >>> reverse, to built-in open. >> >> I might agree with you if there wasn?t 20 years of code that >> expects to be able to pass a ?path? in to various places. Having to >> keep a mental note, or worse look up in the documentation every >> time, where I?m expected to pass a Path object and where I?m >> expected to pass a str is just about the worst UX I can imagine. > > You wouldn't need a mental memo. Just do "import os" rather than > "from os import *" and the "os." prefix will tell you you need a str. > Otherwise use a Path. > > That would be the goal, I expect. How to get there, I'm not sure. I never use star imports. The problem is that not only does tons of stuff in the stdlib currently accept only str, but so do tons of things on PyPI. Without looking at the implementation/docs for each one of these items I won?t know if I can pass it a str or a Path (or more likely I won?t know if I need to coerce my Path to a str since i doubt anyone is going to make Path only APIs). For me personally this means that pathlib will likely never be a useful tool because I find the need to coerce all over the place far worse than using the relevant os.path functions. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From solipsis at pitrou.net Tue Oct 7 17:39:43 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Oct 2014 17:39:43 +0200 Subject: [Python-ideas] Better stdlib support for Path objects References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> Message-ID: <20141007173943.2989dd6c@fsol> On Wed, 8 Oct 2014 00:49:12 +1000 Nick Coghlan wrote: > > Combining this thought with Chris Angelico's reply, I actually wonder > if the index vs int analogy is even more applicable than I first > thought. > > What if the protocol was __text__ with a new text() builtin (or at > least an operator.text() function), and it was advised to only be > implemented by types where they were, at least conceptually, truly > representable as strings? What does "truly representable, at least conceptually" mean? Do HTML trees apply? Configuration files? Command-line arguments? Regards Antoine. From rosuav at gmail.com Tue Oct 7 17:48:25 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 8 Oct 2014 02:48:25 +1100 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141007173943.2989dd6c@fsol> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> <20141007173943.2989dd6c@fsol> Message-ID: On Wed, Oct 8, 2014 at 2:39 AM, Antoine Pitrou wrote: > On Wed, 8 Oct 2014 00:49:12 +1000 > Nick Coghlan wrote: >> >> Combining this thought with Chris Angelico's reply, I actually wonder >> if the index vs int analogy is even more applicable than I first >> thought. >> >> What if the protocol was __text__ with a new text() builtin (or at >> least an operator.text() function), and it was advised to only be >> implemented by types where they were, at least conceptually, truly >> representable as strings? > > What does "truly representable, at least conceptually" mean? > Do HTML trees apply? Configuration files? Command-line arguments? > I like the comparison with __index__. It's up to the class to decide whether it wants to be "truly representable as a string". Command-line arguments probably would be, in some canonical form. Config files probably not, as they need their structure, but if you want your config object to act exactly like a text string, then you define __text__. ChrisA From solipsis at pitrou.net Tue Oct 7 17:55:38 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Oct 2014 17:55:38 +0200 Subject: [Python-ideas] Better stdlib support for Path objects References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> <20141007173943.2989dd6c@fsol> Message-ID: <20141007175538.1c7eb495@fsol> On Wed, 8 Oct 2014 02:48:25 +1100 Chris Angelico wrote: > On Wed, Oct 8, 2014 at 2:39 AM, Antoine Pitrou wrote: > > On Wed, 8 Oct 2014 00:49:12 +1000 > > Nick Coghlan wrote: > >> > >> Combining this thought with Chris Angelico's reply, I actually wonder > >> if the index vs int analogy is even more applicable than I first > >> thought. > >> > >> What if the protocol was __text__ with a new text() builtin (or at > >> least an operator.text() function), and it was advised to only be > >> implemented by types where they were, at least conceptually, truly > >> representable as strings? > > > > What does "truly representable, at least conceptually" mean? > > Do HTML trees apply? Configuration files? Command-line arguments? > > > > I like the comparison with __index__. It's up to the class to decide > whether it wants to be "truly representable as a string". Command-line > arguments probably would be, in some canonical form. Config files > probably not, as they need their structure, but if you want your > config object to act exactly like a text string, then you define > __text__. And what's the use case that would be fulfilled by this exactly? From rosuav at gmail.com Tue Oct 7 18:04:05 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 8 Oct 2014 03:04:05 +1100 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141007175538.1c7eb495@fsol> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> <20141007173943.2989dd6c@fsol> <20141007175538.1c7eb495@fsol> Message-ID: On Wed, Oct 8, 2014 at 2:55 AM, Antoine Pitrou wrote: >> I like the comparison with __index__. It's up to the class to decide >> whether it wants to be "truly representable as a string". Command-line >> arguments probably would be, in some canonical form. Config files >> probably not, as they need their structure, but if you want your >> config object to act exactly like a text string, then you define >> __text__. > > And what's the use case that would be fulfilled by this exactly? You pass a Path object to anything whatsoever. If it wants a Path, it uses it. As soon as that Path gets picked up by something that was expecting a string, it'll easily and conveniently coerce to string. But if you pass something other than a string or string-equivalent, it'll throw an error rather than calling str() on it. Where is the definition of "able to convert to str implicitly"? >>> "asdf"+[1,2,3] Traceback (most recent call last): File "", line 1, in TypeError: Can't convert 'list' object to str implicitly Because this could just call Path.__text__(), get back a string, and use that. Basically, the use-case is that you could create something that's not a string, but can be used like a string, and APIs unaware of it don't need to be made aware of it. Downside: These "text-aware" objects would require Python 3.whatever, and would be hard to backport. On older versions, they wouldn't implicitly become strings. ChrisA From solipsis at pitrou.net Tue Oct 7 18:11:52 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Oct 2014 18:11:52 +0200 Subject: [Python-ideas] Better stdlib support for Path objects References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006213824.46066a22@anarchist.wooz.org> <2B1821D2-82E9-4858-A403-4B0CD10ECB00@stufft.io> <20141007113430.7af4ae2d@fsol> <20141007093710.6cb7a44b@anarchist.wooz.org> <20141007173943.2989dd6c@fsol> <20141007175538.1c7eb495@fsol> Message-ID: <20141007181152.5191e69b@fsol> On Wed, 8 Oct 2014 03:04:05 +1100 Chris Angelico wrote: > >>> "asdf"+[1,2,3] > Traceback (most recent call last): > File "", line 1, in > TypeError: Can't convert 'list' object to str implicitly > > Because this could just call Path.__text__(), get back a string, and use that. This won't happen, sorry. From wes.turner at gmail.com Tue Oct 7 20:25:44 2014 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 7 Oct 2014 13:25:44 -0500 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141006134714.640fc3ca@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: Does path.py have different performance in this respect? I like the Path.isdir(), Path.walk() methods; but maybe not for everyone. Way OT, but similar support for URIs (e.g. URLObject) would likely need to take a similar approach: http://www.reddit.com/r/Python/comments/1r7h1t/python_objects_for_working_with_urls_and_uris/ On Oct 6, 2014 12:50 PM, "Barry Warsaw" wrote: > Over in issue 22570, I lament the fact that while pathlib is awesome, its > wider support in the stdlib is pretty sparse. I've tried to convert parts > of > a medium sized Python 3 application from os.path to pathlib and found this > lack of support rather demotivating. Yes, it's fairly easy to wrap Path > objects in str() to pass them to stdlib methods that expect only strings, > but > it's a lot of work in user code and I find that the resulting str()s are > distracting. It's a disincentive. > > Antoine provided a link to a previous discussion[*] but that didn't go very > far. > > One simple solution would be to sprinkle str() calls in various stdlib > methods, but I'm not sure if that would fail miserably in the face of bytes > paths (if the original stdlib API even accepts bytes paths). The > suggestion > in the issue is to add a "path protocol" and the referenced article > suggests > .strpath and .bytespath. OTOH, isn't str() and bytes() enough? > > I don't have any other brilliant ideas, but I opened the issue and am > posting > here to see if we can jump start another discussion for Python 3.5. I'd > *like* to use more Paths, but not at the expense of my own code's > readability. > Yes, I'd sacrifice a bit of readability in the stdlib, especially if that > would cover more use cases. > > Cheers, > -Barry > > [*] https://mail.python.org/pipermail/python-ideas/2014-May/027869.html > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Oct 7 20:37:50 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 07 Oct 2014 11:37:50 -0700 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141007102454.0a730024@anarchist.wooz.org> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141007102454.0a730024@anarchist.wooz.org> Message-ID: <543432FE.9000408@stoneleaf.us> On 10/07/2014 07:24 AM, Barry Warsaw wrote: > On Oct 07, 2014, at 02:55 PM, Paul Moore wrote: >> >> I find it worse than a disincentive, it makes understanding the code >> perceptibly harder, which is a maintenance issue. Having an attribute >> that returns the string representation would be a substantial >> improvement (as it's the extra parentheses from the str call that I >> find the most distracting, that and the code smell that an "explicit >> cast" involves). > > I realize there's another thing that bugs me about sprinkling str() calls all > over the place, and this relates to my other question about whether > str()-ability is "the path protocol". > > The problem is that if I'm looking at some random code and see: > > my_parser.load(str(path)) > > I really don't have any idea what 'path' is. Maybe that's a good thing, but > in the few cases where I did this, it seemed bad. ;) > > OTOH, if I saw this, it would be a strong clue that path were a pathlib > object: > > my_parser.load(path.string_path) > > substituting .string_path for whatever color the shed gets painted. Neither should be needed: my_parser.load(path) should do the trick. What was the point of adding pathlib to the stdlib if we cannot use it with the stdlib? Having a __strpath__ and/or __bytespath__ would also allow third-party path libraries to be utilized. -- ~Ethan~ From encukou at gmail.com Tue Oct 7 20:46:00 2014 From: encukou at gmail.com (Petr Viktorin) Date: Tue, 7 Oct 2014 20:46:00 +0200 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> Message-ID: On Tue, Oct 7, 2014 at 8:25 PM, Wes Turner wrote: > Does path.py have different performance in this respect? > > I like the Path.isdir(), Path.walk() methods; but maybe not for everyone. > > Way OT, but similar support for URIs (e.g. URLObject) would likely need to > take a similar approach: > http://www.reddit.com/r/Python/comments/1r7h1t/python_objects_for_working_with_urls_and_uris/ URLObject is actually a good reason against __strpath__: path.open() should work whether path is a filesystem path, or a URL object (with pathlib API), or any other path-like object; while open(path) would not. I think path.open() should be encouraged. From donald at stufft.io Tue Oct 7 20:48:22 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 7 Oct 2014 14:48:22 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <543432FE.9000408@stoneleaf.us> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141007102454.0a730024@anarchist.wooz.org> <543432FE.9000408@stoneleaf.us> Message-ID: <8573E6A5-0A58-4CAD-B82B-72E408435A05@stufft.io> > On Oct 7, 2014, at 2:37 PM, Ethan Furman wrote: > > On 10/07/2014 07:24 AM, Barry Warsaw wrote: >> On Oct 07, 2014, at 02:55 PM, Paul Moore wrote: >>> >>> I find it worse than a disincentive, it makes understanding the code >>> perceptibly harder, which is a maintenance issue. Having an attribute >>> that returns the string representation would be a substantial >>> improvement (as it's the extra parentheses from the str call that I >>> find the most distracting, that and the code smell that an "explicit >>> cast" involves). >> >> I realize there's another thing that bugs me about sprinkling str() calls all >> over the place, and this relates to my other question about whether >> str()-ability is "the path protocol". >> >> The problem is that if I'm looking at some random code and see: >> >> my_parser.load(str(path)) >> >> I really don't have any idea what 'path' is. Maybe that's a good thing, but >> in the few cases where I did this, it seemed bad. ;) >> >> OTOH, if I saw this, it would be a strong clue that path were a pathlib >> object: >> >> my_parser.load(path.string_path) >> >> substituting .string_path for whatever color the shed gets painted. > > Neither should be needed: > > my_parser.load(path) > > should do the trick. > > What was the point of adding pathlib to the stdlib if we cannot use it with the stdlib? That would be ideal, however very few (any?) APIs in Python stdlib supports pathlib as it stands. Passing paths around is sprinkled all over the stdlib and history suggests it?ll be a fairly lengthy process to actually get all of those stdlib locations figured out. That doesn?t even begin to touch on all of the places *not* in the stdlib that expects str based paths, some of which are C code so making them use a Path object isn?t the easiest thing ever. > > Having a __strpath__ and/or __bytespath__ would also allow third-party path libraries to be utilized. > --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From g.brandl at gmx.net Tue Oct 7 20:53:12 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 07 Oct 2014 20:53:12 +0200 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <87bnporkm4.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20141006134714.640fc3ca@anarchist.wooz.org> <20141006142431.56ddba78@anarchist.wooz.org> <20141007091731.5b414848@anarchist.wooz.org> <47963623-4702-4F07-A65C-37B394CAF12B@stufft.io> <1412688347.2708323.176114633.7B901459@webmail.messagingengine.com> <87bnporkm4.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 10/07/2014 04:59 PM, Stephen J. Turnbull wrote: > Donald Stufft writes: > > > On Oct 7, 2014, at 9:25 AM, random832 at fastmail.us wrote: > > > > It makes about as much sense for os.open to accept a path as it would > > > for it to return a TextIOWrapper instead of an int. The same applies, in > > > reverse, to built-in open. > > > > I might agree with you if there wasn?t 20 years of code that > > expects to be able to pass a ?path? in to various places. Having to > > keep a mental note, or worse look up in the documentation every > > time, where I?m expected to pass a Path object and where I?m > > expected to pass a str is just about the worst UX I can imagine. That's exactly what I'd be worried about. It would require a big effort to convert enough APIs to make the few that don't take Paths insignificant. That would also signal a strong urge to third-party libs to become Path-aware. However, I'm skeptical that python-dev can muster enough energy for this effort. I believe that a .path attribute (name to be discussed) is probably as good as we can do. In reverse, it means that Path should grow many utility methods for common operations. > You wouldn't need a mental memo. Just do "import os" rather than > "from os import *" and the "os." prefix will tell you you need a str. > Otherwise use a Path. Note that Barry said: "I wouldn't expect low level APIs like os.path and built-in open to accept Path objects." which refers to open(), not os.open(). Georg From alexander.belopolsky at gmail.com Tue Oct 7 20:57:03 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 7 Oct 2014 14:57:03 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: <20141007015821.2ff84b7d@fsol> References: <20141006134714.640fc3ca@anarchist.wooz.org> <54332B57.9020300@mrabarnett.plus.com> <20141007015821.2ff84b7d@fsol> Message-ID: On Mon, Oct 6, 2014 at 7:58 PM, Antoine Pitrou wrote: > Not inheriting from built-in classes such as str, list or tuple > was one of the design points of pathlib. It will not change in the > future ;-) > > PEP 428 outlines this, but you can probably find a more detailed > discussion in the python-ideas archive. > See also rejected PEP 355: "Subclassing from str is a particularly bad idea; many string operations make no sense when applied to a path." http://legacy.python.org/dev/peps/pep-0355/ (I would add that many str operations make no sense - period, so propagating them into newer designs would be a mistake.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Oct 7 20:59:26 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Oct 2014 20:59:26 +0200 Subject: [Python-ideas] Better stdlib support for Path objects References: <20141006134714.640fc3ca@anarchist.wooz.org> <54332B57.9020300@mrabarnett.plus.com> <20141007015821.2ff84b7d@fsol> Message-ID: <20141007205926.7677a88f@fsol> On Tue, 7 Oct 2014 14:57:03 -0400 Alexander Belopolsky wrote: > On Mon, Oct 6, 2014 at 7:58 PM, Antoine Pitrou wrote: > > > Not inheriting from built-in classes such as str, list or tuple > > was one of the design points of pathlib. It will not change in the > > future ;-) > > > > PEP 428 outlines this, but you can probably find a more detailed > > discussion in the python-ideas archive. > > > > See also rejected PEP 355: "Subclassing from str is a particularly bad > idea; many string > operations make no sense when applied to a path." > > http://legacy.python.org/dev/peps/pep-0355/ > > (I would add that many str operations make no sense - period, so > propagating them into newer designs would be a mistake.) Thanks for the reference. It is indeed useful to know where we are coming from here. Trying to mimick str has already been tried in the past. Regards Antoine. From donald at stufft.io Tue Oct 7 21:02:36 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 7 Oct 2014 15:02:36 -0400 Subject: [Python-ideas] Better stdlib support for Path objects In-Reply-To: References: <20141006134714.640fc3ca@anarchist.wooz.org> <54332B57.9020300@mrabarnett.plus.com> <20141007015821.2ff84b7d@fsol> Message-ID: <4C471296-5D87-4AC0-8535-DF30A670E9DA@stufft.io> > On Oct 7, 2014, at 2:57 PM, Alexander Belopolsky wrote: > > > On Mon, Oct 6, 2014 at 7:58 PM, Antoine Pitrou > wrote: > Not inheriting from built-in classes such as str, list or tuple > was one of the design points of pathlib. It will not change in the > future ;-) > > PEP 428 outlines this, but you can probably find a more detailed > discussion in the python-ideas archive. > > See also rejected PEP 355: "Subclassing from str is a particularly bad idea; many string > operations make no sense when applied to a path." > > http://legacy.python.org/dev/peps/pep-0355/ > > (I would add that many str operations make no sense - period, so propagating them into newer designs would be a mistake.) Many str operations make no sense when applied to lots of different types of strings, even moreso many bytes methods make no sense when applied to any kind of bytes except a very narrow subset. I?m pretty sure that ?does every single method make sense in every scenario? it not a useful distinction to make. However the sheer weight of APIs out there that expect str, and only str means that either pathlib isn?t very useful without wrapping every call in a str() or using some attribute/method on it to convert to str. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Oct 7 21:05:02 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 07 Oct 2014 12:05:02 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: Message-ID: <5434395E.6070909@stoneleaf.us> On 10/07/2014 12:00 AM, Ram Rachum wrote: > > Use case not relevant enough for you? No, your attitude stinks -- or maybe it's just your communication style. Phrases such as, "I don't see what the problem is" does not credit anybody else with intelligence, or we would also 'not see what the problem is", right? But we are not you, don't have your experience, aren't writing your software, and most of us will not see your issues unless you explain them thoroughly. "This would be cool" is not a thorough explanation, does not list use-cases, provides no list of pro's and con's, and provides no evidence that you have thoroughly thought through your proposal. At least, that's my take on the matter. -- ~Ethan~ From ram at rachum.com Tue Oct 7 21:45:46 2014 From: ram at rachum.com (Ram Rachum) Date: Tue, 7 Oct 2014 22:45:46 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: <5434395E.6070909@stoneleaf.us> References: <5434395E.6070909@stoneleaf.us> Message-ID: I'm sorry that I offended. I didn't mean to imply anything about anyone's intelligence by saying "I don't see what the problem is", I was just saying I don't see what the problem is, and if I someone sees a problem I didn't, I'll be happy to have it pointed out to me so we can discuss it, and if the other person agrees that there isn't a problem, we can move on. I implemented classes for this in my project and I'll see whether it's useful (I'm already getting the feeling that a `DefinitelyUnordered` base class is more useful as it's easier to capture the unordered than the ordered.) If it proves useful, I'll write back and if there'll be interest to put something like this into Python, that'll be cool. On Tue, Oct 7, 2014 at 10:05 PM, Ethan Furman wrote: > On 10/07/2014 12:00 AM, Ram Rachum wrote: > >> >> Use case not relevant enough for you? >> > > No, your attitude stinks -- or maybe it's just your communication style. > > Phrases such as, "I don't see what the problem is" does not credit anybody > else with intelligence, or we would also 'not see what the problem is", > right? But we are not you, don't have your experience, aren't writing your > software, and most of us will not see your issues unless you explain them > thoroughly. > > "This would be cool" is not a thorough explanation, does not list > use-cases, provides no list of pro's and con's, and provides no evidence > that you have thoroughly thought through your proposal. > > At least, that's my take on the matter. > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Oct 7 21:54:21 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 7 Oct 2014 12:54:21 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> Message-ID: For me, the main problem is that time and again you responded to very precise questions with vague answers. On Tue, Oct 7, 2014 at 12:45 PM, Ram Rachum wrote: > I'm sorry that I offended. I didn't mean to imply anything about anyone's > intelligence by saying "I don't see what the problem is", I was just saying > I don't see what the problem is, and if I someone sees a problem I didn't, > I'll be happy to have it pointed out to me so we can discuss it, and if the > other person agrees that there isn't a problem, we can move on. > > I implemented classes for this in my project and I'll see whether it's > useful (I'm already getting the feeling that a `DefinitelyUnordered` base > class is more useful as it's easier to capture the unordered than the > ordered.) If it proves useful, I'll write back and if there'll be interest > to put something like this into Python, that'll be cool. > > > On Tue, Oct 7, 2014 at 10:05 PM, Ethan Furman wrote: > >> On 10/07/2014 12:00 AM, Ram Rachum wrote: >> >>> >>> Use case not relevant enough for you? >>> >> >> No, your attitude stinks -- or maybe it's just your communication style. >> >> Phrases such as, "I don't see what the problem is" does not credit >> anybody else with intelligence, or we would also 'not see what the problem >> is", right? But we are not you, don't have your experience, aren't writing >> your software, and most of us will not see your issues unless you explain >> them thoroughly. >> >> "This would be cool" is not a thorough explanation, does not list >> use-cases, provides no list of pro's and con's, and provides no evidence >> that you have thoroughly thought through your proposal. >> >> At least, that's my take on the matter. >> >> -- >> ~Ethan~ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Oct 7 22:00:25 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 07 Oct 2014 13:00:25 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> Message-ID: <54344659.9090000@stoneleaf.us> On 10/07/2014 12:45 PM, Ram Rachum wrote: > I'm sorry that I offended. I didn't mean to imply anything about anyone's intelligence by saying "I don't see what the > problem is", I was just saying I don't see what the problem is, and if I someone sees a problem I didn't, I'll be happy > to have it pointed out to me so we can discuss it, and if the other person agrees that there isn't a problem, we can > move on. Text communication is difficult, considering a large portion of human communication is via body language and intonation. > I implemented classes for this in my project and I'll see whether it's useful (I'm already getting the feeling that a > `DefinitelyUnordered` base class is more useful as it's easier to capture the unordered than the ordered.) If it proves > useful, I'll write back and if there'll be interest to put something like this into Python, that'll be cool. The big hurdles for getting anything new into Python are: - general applicability - additional expressive power - easing of hard-to-get-right tasks - etc. Claiming that something is so (such as Enum serial numbers) is not sufficient to make it so. You have to provide examples either from other languages or actual code that prove it, and hopefully be able to demonstrate that the need is greater than just your code base. -- ~Ethan~ From ram at rachum.com Tue Oct 7 22:01:20 2014 From: ram at rachum.com (Ram Rachum) Date: Tue, 7 Oct 2014 23:01:20 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> Message-ID: Thanks for giving me feedback. I'm trying to be easier to deal with. I looked back at the thread but couldn't see where I was asked a precise question and gave a vague answer. The closest thing to it was when you asked for the meaning of when something is ordered. Ed answered something and I said I meant exactly what he said, but maybe I should have been more explicit: I meant that it's guaranteed that `tuple(x) == tuple(x)`. If there are any more cases where I was vague, I'll be happy to be more specific. On Tue, Oct 7, 2014 at 10:54 PM, Guido van Rossum wrote: > For me, the main problem is that time and again you responded to very > precise questions with vague answers. > > On Tue, Oct 7, 2014 at 12:45 PM, Ram Rachum wrote: > >> I'm sorry that I offended. I didn't mean to imply anything about anyone's >> intelligence by saying "I don't see what the problem is", I was just saying >> I don't see what the problem is, and if I someone sees a problem I didn't, >> I'll be happy to have it pointed out to me so we can discuss it, and if the >> other person agrees that there isn't a problem, we can move on. >> >> I implemented classes for this in my project and I'll see whether it's >> useful (I'm already getting the feeling that a `DefinitelyUnordered` base >> class is more useful as it's easier to capture the unordered than the >> ordered.) If it proves useful, I'll write back and if there'll be interest >> to put something like this into Python, that'll be cool. >> >> >> On Tue, Oct 7, 2014 at 10:05 PM, Ethan Furman wrote: >> >>> On 10/07/2014 12:00 AM, Ram Rachum wrote: >>> >>>> >>>> Use case not relevant enough for you? >>>> >>> >>> No, your attitude stinks -- or maybe it's just your communication style. >>> >>> Phrases such as, "I don't see what the problem is" does not credit >>> anybody else with intelligence, or we would also 'not see what the problem >>> is", right? But we are not you, don't have your experience, aren't writing >>> your software, and most of us will not see your issues unless you explain >>> them thoroughly. >>> >>> "This would be cool" is not a thorough explanation, does not list >>> use-cases, provides no list of pro's and con's, and provides no evidence >>> that you have thoroughly thought through your proposal. >>> >>> At least, that's my take on the matter. >>> >>> -- >>> ~Ethan~ >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Tue Oct 7 22:10:51 2014 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 7 Oct 2014 16:10:51 -0400 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> Message-ID: On Tue, Oct 7, 2014 at 4:01 PM, Ram Rachum wrote: > you asked for the meaning of when something is ordered. Ed answered > something and I said I meant exactly what he said, but maybe I should have > been more explicit: I meant that it's guaranteed that `tuple(x) == > tuple(x)`. I don't think this is a very useful definition: >>> x = iter('abc') >>> tuple(x) == tuple(x) False >>> x = set('abc') >>> tuple(x) == tuple(x) True -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Tue Oct 7 22:10:56 2014 From: ram at rachum.com (Ram Rachum) Date: Tue, 7 Oct 2014 23:10:56 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: <54344659.9090000@stoneleaf.us> References: <5434395E.6070909@stoneleaf.us> <54344659.9090000@stoneleaf.us> Message-ID: On Tue, Oct 7, 2014 at 11:00 PM, Ethan Furman wrote: > > I implemented classes for this in my project and I'll see whether it's >> useful (I'm already getting the feeling that a >> `DefinitelyUnordered` base class is more useful as it's easier to capture >> the unordered than the ordered.) If it proves >> useful, I'll write back and if there'll be interest to put something like >> this into Python, that'll be cool. >> > > The big hurdles for getting anything new into Python are: > - general applicability > - additional expressive power > - easing of hard-to-get-right tasks > - etc. > > Claiming that something is so (such as Enum serial numbers) is not > sufficient to make it so. You have to provide examples either from other > languages or actual code that prove it, and hopefully be able to > demonstrate that the need is greater than just your code base. > > On one hand I understand the reasoning of looking for concrete use cases, but on the other hand I think it has its drawbacks. There's lots of things, like for example tuple.count, which if I remember correctly was added in a pretty late version of Python (2.6 or something) and I needed it in a previous version and it annoyed me that it wasn't there. You don't need to wait for someone to say "Hey, I need `tuple.count`", if you have it on `list` then people are probably gonna need it on `tuple` at one point or another, so if it's a reasonable feature, I think it's good to provide it early to save all that time, because it's obvious there are going to be use cases. Another example is PEP 3155. Instead of waiting for people to get frustrated that they can't pass references to methods to `multiprocessing.Process` and then having to wait for Python 3.3 to do that, we could have preempted and implemented it before based on common sense (i.e. if I have a method object, it should have information on it telling where to find it.) Of course, this approach has its disadvantages as well, and we always need to be careful when adding anything to Python because it's a mostly unreversible process. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Tue Oct 7 22:13:57 2014 From: ram at rachum.com (Ram Rachum) Date: Tue, 7 Oct 2014 23:13:57 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> Message-ID: You're right, I didn't think of that case. So the best definition I can come up with is: "The iteration order is defined and meaningful, and not random." Is this specific enough? I know it's something which isn't testable programmatically (same as `tuple(x) == tuple(x)` which is impractical to test.) On Tue, Oct 7, 2014 at 11:10 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Tue, Oct 7, 2014 at 4:01 PM, Ram Rachum wrote: > >> you asked for the meaning of when something is ordered. Ed answered >> something and I said I meant exactly what he said, but maybe I should have >> been more explicit: I meant that it's guaranteed that `tuple(x) == >> tuple(x)`. > > > I don't think this is a very useful definition: > > >>> x = iter('abc') > >>> tuple(x) == tuple(x) > False > >>> x = set('abc') > >>> tuple(x) == tuple(x) > True > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Oct 7 22:16:00 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 07 Oct 2014 13:16:00 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> <54344659.9090000@stoneleaf.us> Message-ID: <54344A00.8030401@stoneleaf.us> On 10/07/2014 01:10 PM, Ram Rachum wrote: > > On one hand I understand the reasoning of looking for concrete use cases, but on the other hand I think it has its > drawbacks. There's lots of things, like for example tuple.count, which if I remember correctly was added in a pretty > late version of Python (2.6 or something) and I needed it in a previous version and it annoyed me that it wasn't there. > You don't need to wait for someone to say "Hey, I need `tuple.count`", if you have it on `list` then people are probably > gonna need it on `tuple` at one point or another, so if it's a reasonable feature, I think it's good to provide it early > to save all that time, because it's obvious there are going to be use cases. The trouble with obvious is it isn't. ;) > Another example is PEP 3155. Instead of waiting for people to get frustrated that they can't pass references to methods > to `multiprocessing.Process` and then having to wait for Python 3.3 to do that, we could have preempted and implemented > it before based on common sense (i.e. if I have a method object, it should have information on it telling where to find > it.) This why we have a public PEP process, to try and feedback from as many folk as possible. (see my previous point) > Of course, this approach has its disadvantages as well, and we always need to be careful when adding anything to Python > because it's a mostly unreversible process. Yup -- it's easier to add something once it's known to be needed, than to live with bloat because we added something and it turns out nobody uses it, or a better method was found and added, or ... -- ~Ethan~ From guido at python.org Tue Oct 7 22:20:17 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 7 Oct 2014 13:20:17 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> Message-ID: But your use case (as I understand it) requires that the iteration order is the same as the canonical order that your app defines. On Tue, Oct 7, 2014 at 1:13 PM, Ram Rachum wrote: > You're right, I didn't think of that case. So the best definition I can > come up with is: "The iteration order is defined and meaningful, and not > random." Is this specific enough? I know it's something which isn't > testable programmatically (same as `tuple(x) == tuple(x)` which is > impractical to test.) > > > On Tue, Oct 7, 2014 at 11:10 PM, Alexander Belopolsky < > alexander.belopolsky at gmail.com> wrote: > >> >> On Tue, Oct 7, 2014 at 4:01 PM, Ram Rachum wrote: >> >>> you asked for the meaning of when something is ordered. Ed answered >>> something and I said I meant exactly what he said, but maybe I should have >>> been more explicit: I meant that it's guaranteed that `tuple(x) == >>> tuple(x)`. >> >> >> I don't think this is a very useful definition: >> >> >>> x = iter('abc') >> >>> tuple(x) == tuple(x) >> False >> >>> x = set('abc') >> >>> tuple(x) == tuple(x) >> True >> >> >> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ram at rachum.com Tue Oct 7 22:27:50 2014 From: ram at rachum.com (Ram Rachum) Date: Tue, 7 Oct 2014 23:27:50 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> Message-ID: Ah, perhaps I haven't explained clearly enough. I did not mean that the result of `isinstance(x, Ordered)` would be somehow determined by the canonical order in my combinatorics package (which would be absurd as my package is just my own thing and shouldn't affect a type in `collections`). I meant that my combinatorics package, when asked whether a certain iterable belongs in a combination space, would check its order only if the iterable type is `Ordered`. (i.e. it's communicating "my order is meaningful".) For example, someone could have a combination space "3 items taken from range(5)", which has members like (0, 1, 2) and (1, 3, 4), but not (4, 2, 1) because it's not in canoncial order. If someone were to check `(4, 2, 1) in comb_space` he should get `False`, but if he's checking `{1, 2, 4} in comb_space` he should get `True`, regardless of iteration order, because it's a `set` which isn't supposed to be ordered so it's assumed the user is not asking a question about order. On Tue, Oct 7, 2014 at 11:20 PM, Guido van Rossum wrote: > But your use case (as I understand it) requires that the iteration order > is the same as the canonical order that your app defines. > > On Tue, Oct 7, 2014 at 1:13 PM, Ram Rachum wrote: > >> You're right, I didn't think of that case. So the best definition I can >> come up with is: "The iteration order is defined and meaningful, and not >> random." Is this specific enough? I know it's something which isn't >> testable programmatically (same as `tuple(x) == tuple(x)` which is >> impractical to test.) >> >> >> On Tue, Oct 7, 2014 at 11:10 PM, Alexander Belopolsky < >> alexander.belopolsky at gmail.com> wrote: >> >>> >>> On Tue, Oct 7, 2014 at 4:01 PM, Ram Rachum wrote: >>> >>>> you asked for the meaning of when something is ordered. Ed answered >>>> something and I said I meant exactly what he said, but maybe I should have >>>> been more explicit: I meant that it's guaranteed that `tuple(x) == >>>> tuple(x)`. >>> >>> >>> I don't think this is a very useful definition: >>> >>> >>> x = iter('abc') >>> >>> tuple(x) == tuple(x) >>> False >>> >>> x = set('abc') >>> >>> tuple(x) == tuple(x) >>> True >>> >>> >>> >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Tue Oct 7 22:28:31 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Tue, 07 Oct 2014 16:28:31 -0400 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> Message-ID: <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> On Tue, Oct 7, 2014, at 16:20, Guido van Rossum wrote: > But your use case (as I understand it) requires that the iteration order > is > the same as the canonical order that your app defines. No, that is the condition he is testing for. i.e. (pseudo-code, and probably not the order in which these tests would normally take place) if the elements of object A are a subset those of list B: if object A has a meaningful order: if the order of object A's elements is the same as that of list B: return True else: return False else: return True else: return False From ram at rachum.com Tue Oct 7 22:31:23 2014 From: ram at rachum.com (Ram Rachum) Date: Tue, 7 Oct 2014 23:31:23 +0300 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> Message-ID: Yep, this is what I meant. On Tue, Oct 7, 2014 at 11:28 PM, wrote: > On Tue, Oct 7, 2014, at 16:20, Guido van Rossum wrote: > > But your use case (as I understand it) requires that the iteration order > > is > > the same as the canonical order that your app defines. > > No, that is the condition he is testing for. > > i.e. (pseudo-code, and probably not the order in which these tests would > normally take place) > > if the elements of object A are a subset those of list B: > if object A has a meaningful order: > if the order of object A's elements is the same as that of list > B: > return True > else: > return False > else: > return True > else: > return False > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Tue Oct 7 22:41:54 2014 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 8 Oct 2014 07:41:54 +1100 Subject: [Python-ideas] Enhanced stat_result objects (was: Better stdlib support for Path objects) In-Reply-To: References: Message-ID: On 8 October 2014 01:08, Paul Moore wrote: > I would find it really useful if stat_result objects supported the > various is_XXX methods from pathlib. Then I could write code like: > > p = Path(...) > st = p.stat() > if st.exists(): > ... > elif st.is_dir(): > ... > > which explicitly shows that stat is only called once, but doesn't > involve verbose and ugly code like > > if stat.S_ISDIR(st.st_mode): > ... > > Would this be a worthwhile addition? > +1 Then Path, DirEntry (from scandir) and stat_result objects would all support the same API. DirEntry.is_dir() and .is_file() methods have an optional follow_symlinks parameter - I'm thinking Path should follow suit. Obviously stat_result would not. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Oct 8 01:50:17 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 8 Oct 2014 10:50:17 +1100 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> Message-ID: <20141007235016.GC27477@ando.pearwood.info> On Tue, Oct 07, 2014 at 11:13:57PM +0300, Ram Rachum wrote: > You're right, I didn't think of that case. So the best definition I can > come up with is: "The iteration order is defined and meaningful, and not > random." Is this specific enough? "Meaningful" isn't specific. Meaningful to whom, in what way? Some people might say that the iteration order of a dict or set is "meaningful" because it reflects the order of entries in the underlying hash table. And "not random" is too specific, since there could be many things which have arbitrary but deterministic and non-random order. Like sets and dicts. -- Steven From steve at pearwood.info Wed Oct 8 02:14:43 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 8 Oct 2014 11:14:43 +1100 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> Message-ID: <20141008001443.GD27477@ando.pearwood.info> On Tue, Oct 07, 2014 at 11:27:50PM +0300, Ram Rachum wrote: [...] > I meant that my combinatorics package, when asked whether a > certain iterable belongs in a combination space, would check its order only > if the iterable type is `Ordered`. (i.e. it's communicating "my order is > meaningful".) > > For example, someone could have a combination space "3 items taken from > range(5)", which has members like (0, 1, 2) and (1, 3, 4), but not (4, 2, > 1) because it's not in canoncial order. If someone were to check `(4, 2, 1) > in comb_space` he should get `False`, but if he's checking `{1, 2, 4} in > comb_space` he should get `True`, regardless of iteration order, because > it's a `set` which isn't supposed to be ordered so it's assumed the user is > not asking a question about order. If I have understood you correctly, you have a single function which does two different tests, an ordered test and an unordered test, with the decision of which to perform being implicitly decided by the input type. I think the Zen of Python has a few things to say about that sort of API: Explicit is better than implicit. In the face of ambiguity, refuse the temptation to guess. It sounds to me that you would be better served by two functions, one which checks order and one which does not. But if you insist on having a single function which decides which test to perform based on the type of the input argument, you are under no obligation to attempt to support every imaginable sequence type "correctly". You can offer a firm guarantee that your function will guess correctly if the type is a list, tuple, set or frozenset, and for everything else you offer a "best effort" guess as to whether the input sequence is ordered or not. -- Steven From stephen at xemacs.org Wed Oct 8 03:09:45 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 08 Oct 2014 10:09:45 +0900 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> Message-ID: <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> Ram Rachum writes: > Yep, this is what I meant. (And you'll answer in this precision in the future, of course. :) > On Tue, Oct 7, 2014 at 11:28 PM, wrote: > > i.e. (pseudo-code, and probably not the order in which these tests would > > normally take place) > > > > if the elements of object A are a subset those of list B: > > if object A has a meaningful order: > > if the order of object A's elements is the same as that of list B: But this requires a comparison of the two orders, and "ordering relation" is not a type available in the stdlib. So I don't see how you propose to do this comparison. This is why you need concrete examples, preferably with running code. OTOH, in the kind of context you're working with, I'd probably use a specific ordered type to contain the combinations (which might be a tree or other implicitly ordered type, not necessarily a Sequence) rather than generic lists etc., and define __eq__ on it. The need for collections.Ordered evaporates! From random832 at fastmail.us Wed Oct 8 05:54:31 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Tue, 07 Oct 2014 23:54:31 -0400 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> On Tue, Oct 7, 2014, at 21:09, Stephen J. Turnbull wrote: > But this requires a comparison of the two orders, and "ordering > relation" is not a type available in the stdlib. This has nothing to do with comparable types - the only comparison being done on the members themselves is equality. i.e. if the main sequence is [5, 1, 3, 2, 4], it's true for [5, 2, 4] but not [5, 4, 1]. Think of it as being the question of whether "51234" matches "5.*2.*4" (it does) or "5.*4.*1" (it does not). But if the argument is a set instead he wants it to be whether it's a subset rather than a subsequence. From ben+python at benfinney.id.au Wed Oct 8 06:07:31 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 08 Oct 2014 15:07:31 +1100 Subject: [Python-ideas] Enhanced stat_result objects References: Message-ID: <85siiztda4.fsf@benfinney.id.au> Paul Moore writes: > I would find it really useful if stat_result objects supported the > various is_XXX methods from pathlib. Better, IMO, if those were data attributes of the ?stat_result? object, not explicit methods: p = Path(...) st = p.stat() if st.exists: ... elif st.is_dir: ... > which explicitly shows that stat is only called once The above style IMO signals even stronger that no further filesystem calls are expected. > Would this be a worthwhile addition? I'd support the above enhancement, yes. -- \ ?The fact that I have no remedy for all the sorrows of the | `\ world is no reason for my accepting yours. It simply supports | _o__) the strong probability that yours is a fake.? ?Henry L. Mencken | Ben Finney From guido at python.org Wed Oct 8 06:00:46 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 7 Oct 2014 21:00:46 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> Message-ID: On Tue, Oct 7, 2014 at 8:54 PM, wrote: > On Tue, Oct 7, 2014, at 21:09, Stephen J. Turnbull wrote: > > But this requires a comparison of the two orders, and "ordering > > relation" is not a type available in the stdlib. > > This has nothing to do with comparable types - the only comparison being > done on the members themselves is equality. > > i.e. if the main sequence is [5, 1, 3, 2, 4], it's true for [5, 2, 4] > but not [5, 4, 1]. > > Think of it as being the question of whether "51234" matches "5.*2.*4" > (it does) or "5.*4.*1" (it does not). But if the argument is a set > instead he wants it to be whether it's a subset rather than a > subsequence. > This is the first understandable description of Ram's use case that I have seen. Thank you! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Wed Oct 8 06:37:13 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 08 Oct 2014 13:37:13 +0900 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> Message-ID: <87k34bqirq.fsf@uwakimon.sk.tsukuba.ac.jp> random832 at fastmail.us writes: > This has nothing to do with comparable types "if ordering is the same" is what you wrote. How do you propose to determine that without an OrderRelation type or similar? From oreilldf at gmail.com Wed Oct 8 07:24:27 2014 From: oreilldf at gmail.com (Dan O'Reilly) Date: Wed, 8 Oct 2014 01:24:27 -0400 Subject: [Python-ideas] Better integration of multiprocessing with asyncio In-Reply-To: <6D00A7DF-A35D-4608-8AEA-5C21376F909B@ryanhiebert.com> References: <6D00A7DF-A35D-4608-8AEA-5C21376F909B@ryanhiebert.com> Message-ID: For what it's worth, I ended up writing a module that takes the "threads on top of regular multiprocessing" approach to integrate the two: https://github.com/dano/aioprocessing. Re-implementing multiprocessing to use asyncio internally, while an interesting exercise, would require a very large amount of effort both to implement and maintain alongside the current multiprocessing module. I'm not sure it's really worth it when using threads on top of multiprocessing gives you the same effect without requiring you to basically re-implement large parts of the multiprocessing module. Anyway, we'll see if it ends up getting much use... On Sat, Jul 26, 2014 at 11:48 PM, Ryan Hiebert wrote: > > On Jul 26, 2014, at 10:43 PM, Guido van Rossum wrote: > > I'm going to go out on a limb here and say that it feels too early to me. > First someone has to actually solve this problem well as a 3rd party > package before we can talk about adding it to the asyncio package. It > doesn't actually sound like Billiards has adapted to asyncio yet (not that > I have any idea what Billiards is -- it sounds like a fork of > multiprocessing actually?). > > > Yep, Billiard is a fork of multiprocessing: > https://pypi.python.org/pypi/billiard > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 8 07:29:30 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Oct 2014 15:29:30 +1000 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> Message-ID: On 8 Oct 2014 14:09, "Guido van Rossum" wrote: > > On Tue, Oct 7, 2014 at 8:54 PM, wrote: >> >> On Tue, Oct 7, 2014, at 21:09, Stephen J. Turnbull wrote: >> > But this requires a comparison of the two orders, and "ordering >> > relation" is not a type available in the stdlib. >> >> This has nothing to do with comparable types - the only comparison being >> done on the members themselves is equality. >> >> i.e. if the main sequence is [5, 1, 3, 2, 4], it's true for [5, 2, 4] >> but not [5, 4, 1]. >> >> Think of it as being the question of whether "51234" matches "5.*2.*4" >> (it does) or "5.*4.*1" (it does not). But if the argument is a set >> instead he wants it to be whether it's a subset rather than a >> subsequence. > > > This is the first understandable description of Ram's use case that I have seen. Thank you! And now that I also understand it, I can note that this kind of thing *does* occasionally come up in testing polymorphic filtering operations. If the original container has a non-arbitrary ordering, the filtering operation should preserve it. If the container ordering is arbitrary (e.g. set, dict) then filtering may also change the relative ordering of the retained elements. That observation also gives a possible concrete semantic definition as to what "Ordered" might mean: adding or removing a new element never changes the relative ordering of the other elements in the container. Under that definition, set and dict aren't ordered, since the order of iteration can be affected by the current number of items in the container. As an example of the difference mattering in practice, "assertSameElements" is specifically designed to help deal with cases where you're comparing an unordered result to ordered reference data. Cheers, Nick. > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From edk141 at gmail.com Wed Oct 8 07:53:11 2014 From: edk141 at gmail.com (Ed Kellett) Date: Wed, 8 Oct 2014 06:53:11 +0100 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: <87k34bqirq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> <87k34bqirq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 8 October 2014 05:37, Stephen J. Turnbull wrote: > random832 at fastmail.us writes: > > > This has nothing to do with comparable types > > "if ordering is the same" is what you wrote. How do you propose to > determine that without an OrderRelation type or similar? > If the order of the elements in the ordered container is the same - which is easy to test for. In the interest of being exactly clear, here's some code that does so (assuming I understand correctly). This probably should be cleaned up, but gets the job done: def subordered(subset, superset): si = iter(superset) for x in subset: for y in si: if x == y: break else: return False return True print(subordered([1, 4, 8, 9], range(10))) # True print(subordered([1, 4, 9, 8], range(10))) # False edk From stephen at xemacs.org Wed Oct 8 09:31:17 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 08 Oct 2014 16:31:17 +0900 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> <87k34bqirq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87d2a3qapm.fsf@uwakimon.sk.tsukuba.ac.jp> Ed Kellett writes: > On 8 October 2014 05:37, Stephen J. Turnbull wrote: > > random832 at fastmail.us writes: > > > > > This has nothing to do with comparable types > > > > "if ordering is the same" is what you wrote. How do you propose to > > determine that without an OrderRelation type or similar? > > > > If the order of the elements in the ordered container is the same - > which is easy to test for. In the interest of being exactly clear, > here's some code that does so Well, sure, if it's computable you can write Python code to do it. My point is that isinstance.(collections.Ordered) isn't enough. You must do additional work (worst case O(N) for the largest possible N = #superset) or put additional information into the type. I just don't see much valued added in this ABC or whatever is being proposed for addition to the stdlib. From edk141 at gmail.com Wed Oct 8 12:07:22 2014 From: edk141 at gmail.com (Ed Kellett) Date: Wed, 8 Oct 2014 11:07:22 +0100 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: <87d2a3qapm.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> <87k34bqirq.fsf@uwakimon.sk.tsukuba.ac.jp> <87d2a3qapm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 8 Oct 2014 08:31, "Stephen J. Turnbull" wrote: > Well, sure, if it's computable you can write Python code to do it. My > point is that isinstance.(collections.Ordered) isn't enough. You must > do additional work (worst case O(N) for the largest possible N = > #superset) or put additional information into the type. > > I just don't see much valued added in this ABC or whatever is being > proposed for addition to the stdlib. You're missing the point. The use case for this ABC (or whatever? Shouldn't you be sure what a proposal is before shooting it down?) is knowing whether to do that test at all. edk -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Oct 9 05:24:21 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 9 Oct 2014 04:24:21 +0100 Subject: [Python-ideas] metamodules (was: Re: Idea to support lazy loaded names.) Message-ID: On Tue, Oct 7, 2014 at 5:12 AM, Andrew Barnert wrote: > On Oct 6, 2014, at 20:51, Terry Reedy wrote: > >> So a 'solution' might be to make modules be instances (but with no __new__ or __init__) of a module metaclass, so that module dicts could act like class dicts with respect to descriptors. I have no idea how much code this would break ;-). > > Didn't we just have this discussion a few weeks ago, in the context of making lazy loading of subpackages easier to implement? Yeah, and having had some time to think about that discussion and do some prototyping, I'm going to argue below that allowing assignment to module instances' __class__ really is the best path forward. (For those who find the below TLDR, check this out instead: https://github.com/njsmith/metamodule) > IIRC, the not-obviously-unreasonable options suggested were: Great list! I've rearranged a bit to make my argument clearer. > 1) Analogy with __new__: For packages only, if there's a __new__.py, that gets executed first. If it "returns" (not sure how that was defined) an instance of a subclass of ModuleType, that instance is used to run __init__.py instead of a normal module instance. This is very similar to the current approach of having __init__.py reassign sys.modules[__name__]. The advantages are: - It gives a way to enforce the rule that you have to do this assignment as the very first thing inside your module, before allowing arbitrary code to run (e.g. by importing other modules which might recursively import your module in turn, and access sys.modules[__name__] before you've modified it). - Because __new__.py would run *before* the It avoids the headache of having to juggle two module objects, one of whose __dict__'s is already being used as the execution environment for the code that is trying to do the switcheroo. But: - It's a pretty complicated way to accomplish the stated goals. - The restriction to packages is unfortunate. - The backcompat story is terrible -- faking __new__.py support in old versions of python would be really difficult, and the main reason I care about this stuff in the first place is because I want to be able to e.g. deprecate module attributes that are in the public API of old, widely-used packages. It will be many years before such packages can require 3.5. So I think we should probably try to make the existing sys.modules[__name__] = ... strategy work before considering this. > 2) Analogy with metaclass= (or with 2.x __metaclass__): If a module (or a package's __init__.py) does some new syntax or magic comment before any non-comment code, it can specify a custom type in place of ModuleType (not sure how that type gets imported and made available). I don't see any way to solve this import problem you refer to at the end -- in most cases the code implementing the metamodule type will be defined *inside* the module/package which wants to use the metamodule, so we have a chicken-and-egg problem. > 4) Make it easier to write import hooks for this special purpose. This has the same problem as the previous -- who imports the importer? > 5) Make it easier for a module to construct a ModuleType-subclass instance with a copy of the same dict and replace itself in sys.modules. So, trying to *copy* the dict is just not going to work. Consider the package foo, with a foo/__init__.py that looks like: orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) a = 1 from .submod import b c = 3 sys.modules[__name__].__dict__.update(orig_dict) and where foo/submod.py looks like: import foo b = foo.a + 1 def c(): return foo.a + 2 This won't work, because at the time we import .submod, sys.modules["foo"].__dict__ does not contain an entry for "a" -- only the original module's dict has that. There are a bunch of different ways we could try writing our __init__.py. We might try putting the sys.module assignment at the end: a = 1 from .submod import b, c d = 4 orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) sys.modules[__name__].__dict__.update(orig_dict) Now when .submod re-imports the top-level module, it ends up with a reference to the original module object, which has an "a" entry, so the definition of "b" works. But, now .submod.foo will continue to refer to the original module object, even after we substitute in the metamodule object. If we do 'foo.a = 5' later on, then foo.c() will continue to use the original binding of 'a'; this mutation will be invisible to it. I guess the only way to make it work in this case is to do multiple copies, one before every nested-import: orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) a = 1 sys.modules[__name__].__dict__.update(orig_dict) from .submod import b, c d = 4 sys.modules[__name__].__dict__.update(orig_dict) ...but this is incredibly ugly and error-prone. What we really want to do instead is to make our new metamodule object refer directly to the original module's __dict__: orig_dict = sys.modules[__name__].__dict__ sys.modules[__name__] = MyModule(__name__, __doc__) sys.modules[__name__].__dict__ = orig_dict a = 1 from .submod import b, c d = 4 That way they will always be in sync. This looks like it should work great! But it has a few problems: - Trying to assign to a module's __dict__ attribute raises "TypeError: readonly attribute". - So I actually implemented a fix for that, and ran into a new problem: module's take jealous ownership of their __dict__. In particular, they assume that they are when they are deallocated they should wipe their dict clean (https://www.python.org/doc/essays/cleanup/). Obviously this is bad for us because we are still using that dict! - Also, in modern Python module objects contain more state besides __dict__ -- in particular, PEP 3121-related state. There's no public API to get at this. - Possibly future version of Python will add more state fields again, who knows. The easiest way to solve all these problems is to *swap* all of the internal fields between the old module object and the new metamodule object. This can be done hackishly using ctypes; this requires knowing about CPython's struct layouts, but that's okay for prototyping and for backwards compatibility hacks (which only have to support specific known versions). To do it non-hackishly, I was at first thinking that we should provide an official API for swapping module object states. But then I realized that at that point, we're basically back to... > 3) Analogy with re-classing object instances: Just allow modules to set __class__ during execution (or after, if you want). Variations on this include allowing that for all non-heap types, or even getting rid of the concept of non-heap types. ...this proposal after all. And allowing __class__ assignment on modules strikes me as more asthetic than having a sys.swap_module_contents function. I implemented a prototype of this functionality here: https://github.com/njsmith/metamodule The implementation is here: https://github.com/njsmith/metamodule/blob/master/metamodule.py That file has 3 parts: - A fancy metamodule class that handles implicit-imports and warn-on-attribute-access. - A utility function for setting up a metamodule; it tries __class__ assignment, and if that doesn't work falls back on ctypes hackery. - The aforementioned ctypes hackery. It's pretty ugly, but if we add __class__ assignment then it will become unnecessary on future Python versions, woohoo! -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From guido at python.org Thu Oct 9 06:03:19 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 8 Oct 2014 21:03:19 -0700 Subject: [Python-ideas] Better integration of multiprocessing with asyncio In-Reply-To: References: <6D00A7DF-A35D-4608-8AEA-5C21376F909B@ryanhiebert.com> Message-ID: This looks neat. I skimmed the README, and I noticed you marked up most "yield from" epressions with "# non blocking". That feels confusing to me, because when I read asyncio code, I think fo "yield from" as blocking (the task, if not the world :-). What do you think? --Guido On Tuesday, October 7, 2014, Dan O'Reilly wrote: > For what it's worth, I ended up writing a module that takes the "threads > on top of regular multiprocessing" approach to integrate the two: > https://github.com/dano/aioprocessing. > > Re-implementing multiprocessing to use asyncio internally, while an > interesting exercise, would require a very large amount of effort both to > implement and maintain alongside the current multiprocessing module. I'm > not sure it's really worth it when using threads on top of multiprocessing > gives you the same effect without requiring you to basically re-implement > large parts of the multiprocessing module. > > Anyway, we'll see if it ends up getting much use... > > On Sat, Jul 26, 2014 at 11:48 PM, Ryan Hiebert > wrote: > >> >> On Jul 26, 2014, at 10:43 PM, Guido van Rossum > > wrote: >> >> I'm going to go out on a limb here and say that it feels too early to me. >> First someone has to actually solve this problem well as a 3rd party >> package before we can talk about adding it to the asyncio package. It >> doesn't actually sound like Billiards has adapted to asyncio yet (not that >> I have any idea what Billiards is -- it sounds like a fork of >> multiprocessing actually?). >> >> >> Yep, Billiard is a fork of multiprocessing: >> https://pypi.python.org/pypi/billiard >> > > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From oreilldf at gmail.com Thu Oct 9 06:09:04 2014 From: oreilldf at gmail.com (Dan O'Reilly) Date: Thu, 9 Oct 2014 00:09:04 -0400 Subject: [Python-ideas] Better integration of multiprocessing with asyncio In-Reply-To: References: <6D00A7DF-A35D-4608-8AEA-5C21376F909B@ryanhiebert.com> Message-ID: Ah, right. I find the terminology confusing myself, and I probably got it wrong. I meant "non-blocking" as in - "this won't block the event loop", not "this won't block the coroutine". I'll try to clarify that. Thanks! On Thu, Oct 9, 2014 at 12:03 AM, Guido van Rossum wrote: > This looks neat. I skimmed the README, and I noticed you marked up most > "yield from" epressions with "# non blocking". That feels confusing to me, > because when I read asyncio code, I think fo "yield from" as blocking (the > task, if not the world :-). What do you think? > > --Guido > > > On Tuesday, October 7, 2014, Dan O'Reilly wrote: > >> For what it's worth, I ended up writing a module that takes the "threads >> on top of regular multiprocessing" approach to integrate the two: >> https://github.com/dano/aioprocessing. >> >> Re-implementing multiprocessing to use asyncio internally, while an >> interesting exercise, would require a very large amount of effort both to >> implement and maintain alongside the current multiprocessing module. I'm >> not sure it's really worth it when using threads on top of multiprocessing >> gives you the same effect without requiring you to basically re-implement >> large parts of the multiprocessing module. >> >> Anyway, we'll see if it ends up getting much use... >> >> On Sat, Jul 26, 2014 at 11:48 PM, Ryan Hiebert >> wrote: >> >>> >>> On Jul 26, 2014, at 10:43 PM, Guido van Rossum wrote: >>> >>> I'm going to go out on a limb here and say that it feels too early to >>> me. First someone has to actually solve this problem well as a 3rd party >>> package before we can talk about adding it to the asyncio package. It >>> doesn't actually sound like Billiards has adapted to asyncio yet (not that >>> I have any idea what Billiards is -- it sounds like a fork of >>> multiprocessing actually?). >>> >>> >>> Yep, Billiard is a fork of multiprocessing: >>> https://pypi.python.org/pypi/billiard >>> >> >> > > -- > --Guido van Rossum (on iPad) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Oct 9 06:35:49 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 8 Oct 2014 21:35:49 -0700 Subject: [Python-ideas] Better integration of multiprocessing with asyncio In-Reply-To: References: <6D00A7DF-A35D-4608-8AEA-5C21376F909B@ryanhiebert.com> Message-ID: It takes time to get used to it, but eventually you don't have to think about the event loop. There really is no need for any comment of that kind at all. The beauty is that *nothing* blocks the event loop... On Wednesday, October 8, 2014, Dan O'Reilly wrote: > Ah, right. I find the terminology confusing myself, and I probably got it > wrong. I meant "non-blocking" as in - "this won't block the event loop", > not "this won't block the coroutine". I'll try to clarify that. Thanks! > > On Thu, Oct 9, 2014 at 12:03 AM, Guido van Rossum > wrote: > >> This looks neat. I skimmed the README, and I noticed you marked up most >> "yield from" epressions with "# non blocking". That feels confusing to me, >> because when I read asyncio code, I think fo "yield from" as blocking (the >> task, if not the world :-). What do you think? >> >> --Guido >> >> >> On Tuesday, October 7, 2014, Dan O'Reilly > > wrote: >> >>> For what it's worth, I ended up writing a module that takes the "threads >>> on top of regular multiprocessing" approach to integrate the two: >>> https://github.com/dano/aioprocessing. >>> >>> Re-implementing multiprocessing to use asyncio internally, while an >>> interesting exercise, would require a very large amount of effort both to >>> implement and maintain alongside the current multiprocessing module. I'm >>> not sure it's really worth it when using threads on top of multiprocessing >>> gives you the same effect without requiring you to basically re-implement >>> large parts of the multiprocessing module. >>> >>> Anyway, we'll see if it ends up getting much use... >>> >>> On Sat, Jul 26, 2014 at 11:48 PM, Ryan Hiebert >>> wrote: >>> >>>> >>>> On Jul 26, 2014, at 10:43 PM, Guido van Rossum >>>> wrote: >>>> >>>> I'm going to go out on a limb here and say that it feels too early to >>>> me. First someone has to actually solve this problem well as a 3rd party >>>> package before we can talk about adding it to the asyncio package. It >>>> doesn't actually sound like Billiards has adapted to asyncio yet (not that >>>> I have any idea what Billiards is -- it sounds like a fork of >>>> multiprocessing actually?). >>>> >>>> >>>> Yep, Billiard is a fork of multiprocessing: >>>> https://pypi.python.org/pypi/billiard >>>> >>> >>> >> >> -- >> --Guido van Rossum (on iPad) >> > > -- --Guido van Rossum (on iPad) -------------- next part -------------- An HTML attachment was scrubbed... URL: From oreilldf at gmail.com Thu Oct 9 06:51:43 2014 From: oreilldf at gmail.com (Dan O'Reilly) Date: Thu, 9 Oct 2014 00:51:43 -0400 Subject: [Python-ideas] Better integration of multiprocessing with asyncio In-Reply-To: References: <6D00A7DF-A35D-4608-8AEA-5C21376F909B@ryanhiebert.com> Message-ID: Right - I was just calling it out in the example to highlight that the aioprocessing calls wouldn't block the event loop, as opposed to the equivalent multiprocessing calls, which would. Though I suppose that should be obvious from the use of "yield from". I'll just remove the comments from the example and add a sentence or two afterwards that clarifies the event loop is never blocked. On Thu, Oct 9, 2014 at 12:35 AM, Guido van Rossum wrote: > It takes time to get used to it, but eventually you don't have to think > about the event loop. There really is no need for any comment of that kind > at all. The beauty is that *nothing* blocks the event loop... > > > On Wednesday, October 8, 2014, Dan O'Reilly wrote: > >> Ah, right. I find the terminology confusing myself, and I probably got it >> wrong. I meant "non-blocking" as in - "this won't block the event loop", >> not "this won't block the coroutine". I'll try to clarify that. Thanks! >> >> On Thu, Oct 9, 2014 at 12:03 AM, Guido van Rossum >> wrote: >> >>> This looks neat. I skimmed the README, and I noticed you marked up most >>> "yield from" epressions with "# non blocking". That feels confusing to me, >>> because when I read asyncio code, I think fo "yield from" as blocking (the >>> task, if not the world :-). What do you think? >>> >>> --Guido >>> >>> >>> On Tuesday, October 7, 2014, Dan O'Reilly wrote: >>> >>>> For what it's worth, I ended up writing a module that takes the >>>> "threads on top of regular multiprocessing" approach to integrate the two: >>>> https://github.com/dano/aioprocessing. >>>> >>>> Re-implementing multiprocessing to use asyncio internally, while an >>>> interesting exercise, would require a very large amount of effort both to >>>> implement and maintain alongside the current multiprocessing module. I'm >>>> not sure it's really worth it when using threads on top of multiprocessing >>>> gives you the same effect without requiring you to basically re-implement >>>> large parts of the multiprocessing module. >>>> >>>> Anyway, we'll see if it ends up getting much use... >>>> >>>> On Sat, Jul 26, 2014 at 11:48 PM, Ryan Hiebert >>>> wrote: >>>> >>>>> >>>>> On Jul 26, 2014, at 10:43 PM, Guido van Rossum >>>>> wrote: >>>>> >>>>> I'm going to go out on a limb here and say that it feels too early to >>>>> me. First someone has to actually solve this problem well as a 3rd party >>>>> package before we can talk about adding it to the asyncio package. It >>>>> doesn't actually sound like Billiards has adapted to asyncio yet (not that >>>>> I have any idea what Billiards is -- it sounds like a fork of >>>>> multiprocessing actually?). >>>>> >>>>> >>>>> Yep, Billiard is a fork of multiprocessing: >>>>> https://pypi.python.org/pypi/billiard >>>>> >>>> >>>> >>> >>> -- >>> --Guido van Rossum (on iPad) >>> >> >> > > -- > --Guido van Rossum (on iPad) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Thu Oct 9 08:04:54 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 09 Oct 2014 15:04:54 +0900 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> <87k34bqirq.fsf@uwakimon.sk.tsukuba.ac.jp> <87d2a3qapm.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87zjd5pym1.fsf@uwakimon.sk.tsukuba.ac.jp> Ed Kellett writes: > You're missing the point. The use case for this ABC (or whatever? > Shouldn't you be sure what a proposal is before shooting it down?) > is knowing whether to do that test at all. No, I didn't miss that at all. I just have a different point that I think is far more important. I think you, random, and Ram are wasting a lot of your own time defending this idea in the terms you are using, and I'm trying to explain why I believe that. I'll happily stop, right here, since I've been entirely unsuccessful so far. From edk141 at gmail.com Thu Oct 9 13:46:53 2014 From: edk141 at gmail.com (Ed Kellett) Date: Thu, 9 Oct 2014 12:46:53 +0100 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: <87zjd5pym1.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> <87k34bqirq.fsf@uwakimon.sk.tsukuba.ac.jp> <87d2a3qapm.fsf@uwakimon.sk.tsukuba.ac.jp> <87zjd5pym1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 9 October 2014 07:04, Stephen J. Turnbull wrote: > Ed Kellett writes: > > > You're missing the point. The use case for this ABC (or whatever? > > Shouldn't you be sure what a proposal is before shooting it down?) > > is knowing whether to do that test at all. > > No, I didn't miss that at all. I just have a different point that I > think is far more important. I think you, random, and Ram are wasting > a lot of your own time defending this idea in the terms you are using, > and I'm trying to explain why I believe that. I'll happily stop, > right here, since I've been entirely unsuccessful so far. I don't actually believe I have defended this idea once. I don't care whether it is added to the stdlib - I just didn't want it to get shut down because of a miscommunication, rather than because people didn't think its uses were useful enough. edk From encukou at gmail.com Thu Oct 9 14:13:50 2014 From: encukou at gmail.com (Petr Viktorin) Date: Thu, 9 Oct 2014 14:13:50 +0200 Subject: [Python-ideas] metamodules (was: Re: Idea to support lazy loaded names.) In-Reply-To: References: Message-ID: On Thu, Oct 9, 2014 at 5:24 AM, Nathaniel Smith wrote: [...] >> 1) Analogy with __new__: For packages only, if there's a __new__.py, that gets executed first. If it "returns" (not sure how that was defined) an instance of a subclass of ModuleType, that instance is used to run __init__.py instead of a normal module instance. > > This is very similar to the current approach of having __init__.py > reassign sys.modules[__name__]. The advantages are: > - It gives a way to enforce the rule that you have to do this > assignment as the very first thing inside your module, before allowing > arbitrary code to run (e.g. by importing other modules which might > recursively import your module in turn, and access > sys.modules[__name__] before you've modified it). > - Because __new__.py would run *before* the It avoids the headache of > having to juggle two module objects, one of whose __dict__'s is > already being used as the execution environment for the code that is > trying to do the switcheroo. > > But: > - It's a pretty complicated way to accomplish the stated goals. As in, a non-obvious way to do a non-obvious thing, from the user's point of view? On the implementation side it doesn't strike me as particularly complicated, am I wrong? > - The restriction to packages is unfortunate. True, but it seems to me that you'd usually want it for the __init__.py ? after all if you need to do `from mypackage import submodule` anyway, and submodule isn't a package itself, you can usually can just make `submodule` a class directly. Or have the top-level package define an import hook. Seems to me that this would really only hurt single-file top-level modules, but those are easily converted to a package. > - The backcompat story is terrible -- faking __new__.py support in old > versions of python would be really difficult, and the main reason I > care about this stuff in the first place is because I want to be able > to e.g. deprecate module attributes that are in the public API of old, > widely-used packages. It will be many years before such packages can > require 3.5. That seems like a terrible reason to me ? if it should work nicely on older Pythons, it means a 3rd party module would be enough. Why bother adding it to the language? Valid reasons would be to make it easier for alternative interpreters, or to catch edge cases (e.g. make it work nicely with custom importers, zip files, etc). But at that point, we can just do it the "right way"*, have the backcompat story to a third-party shim. * i.e. "not paying attention to older Pythons" ? I'm not saying __new__.py is necessarily the right way, I'm criticizing this reason against it From random832 at fastmail.us Thu Oct 9 14:49:13 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Thu, 09 Oct 2014 08:49:13 -0400 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: References: <5434395E.6070909@stoneleaf.us> <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> <87k34bqirq.fsf@uwakimon.sk.tsukuba.ac.jp> <87d2a3qapm.fsf@uwakimon.sk.tsukuba.ac.jp> <87zjd5pym1.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1412858953.4020216.177019361.49EBCFF0@webmail.messagingengine.com> On Thu, Oct 9, 2014, at 07:46, Ed Kellett wrote: > On 9 October 2014 07:04, Stephen J. Turnbull wrote: > > No, I didn't miss that at all. I just have a different point that I > > think is far more important. I think you, random, and Ram are wasting > > a lot of your own time defending this idea in the terms you are using, > > and I'm trying to explain why I believe that. I'll happily stop, > > right here, since I've been entirely unsuccessful so far. > > I don't actually believe I have defended this idea once. I don't care > whether it is added to the stdlib - I just didn't want it to get shut > down because of a miscommunication, rather than because people didn't > think its uses were useful enough. Same here, personally I think his use case would be better served by two functions, or by a flag that's passed in by the caller. There's no particular reason to _not_ want to do the unordered subset comparison on a list. From njs at pobox.com Thu Oct 9 16:04:24 2014 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 9 Oct 2014 15:04:24 +0100 Subject: [Python-ideas] metamodules (was: Re: Idea to support lazy loaded names.) In-Reply-To: References: Message-ID: On 9 Oct 2014 13:14, "Petr Viktorin" wrote: > > On Thu, Oct 9, 2014 at 5:24 AM, Nathaniel Smith wrote: > [...] > >> 1) Analogy with __new__: For packages only, if there's a __new__.py, that gets executed first. If it "returns" (not sure how that was defined) an instance of a subclass of ModuleType, that instance is used to run __init__.py instead of a normal module instance. > > > > This is very similar to the current approach of having __init__.py > > reassign sys.modules[__name__]. The advantages are: > > - It gives a way to enforce the rule that you have to do this > > assignment as the very first thing inside your module, before allowing > > arbitrary code to run (e.g. by importing other modules which might > > recursively import your module in turn, and access > > sys.modules[__name__] before you've modified it). > > - Because __new__.py would run *before* the It avoids the headache of > > having to juggle two module objects, one of whose __dict__'s is > > already being used as the execution environment for the code that is > > trying to do the switcheroo. > > > > But: > > - It's a pretty complicated way to accomplish the stated goals. > > As in, a non-obvious way to do a non-obvious thing, from the user's > point of view? > On the implementation side it doesn't strike me as particularly > complicated, am I wrong? Mostly I mean that it requires a much heavier-weight design discussion. It would add qualitatively new concepts to the language, require its own PEP, we'd have to have the debate about whether the cost of extra stat calls on every import was worth it, we'd have to figure out all the fiddly details about how exactly it should work (are variables defined in __new__.py visible in __init__.py?), etc. Usually one requirement in such debates is to demonstrate that there is no acceptable lighter-weight alternative. I think that's a hard argument to make here. The __class__ assignment approach requires minimal changes to python itself, and works pretty much just as well as the __new__.py approach; maybe better in some ways (as per below). So it's hard to justify the effort it would require to get consensus. > > - The restriction to packages is unfortunate. > > True, but it seems to me that you'd usually want it for the > __init__.py ? after all if you need to do `from mypackage import > submodule` anyway, and submodule isn't a package itself, you can > usually can just make `submodule` a class directly. Or have the > top-level package define an import hook. > Seems to me that this would really only hurt single-file top-level > modules, but those are easily converted to a package. Yeah, this is not a showstopper, but given that we have an alternative that *doesn't* have such fiddly restrictions, it's worth mentioning. > > - The backcompat story is terrible -- faking __new__.py support in old > > versions of python would be really difficult, and the main reason I > > care about this stuff in the first place is because I want to be able > > to e.g. deprecate module attributes that are in the public API of old, > > widely-used packages. It will be many years before such packages can > > require 3.5. > > That seems like a terrible reason to me ? if it should work nicely on > older Pythons, it means a 3rd party module would be enough. Why bother > adding it to the language? I posted an implementation that works fine on older pythons. If you look at the code though then I think you will agree that the way it works is OK for a backcompat shim that only has to target a specific range of python versions, but... it is also something that we should feel embarrassed to recommend as a long-term solution. > Valid reasons would be to make it easier for alternative interpreters, > or to catch edge cases (e.g. make it work nicely with custom > importers, zip files, etc). But at that point, we can just do it the > "right way"*, have the backcompat story to a third-party shim. > > * i.e. "not paying attention to older Pythons" ? I'm not saying > __new__.py is necessarily the right way, I'm criticizing this reason > against it My point is just that it matters whether a backcompat shim is doable. Probably there is some way to implement a backcompat shim for __new__.py, but I don't immediately know how to do it. (E.g., how do you read a file that's a sibling of the current source file in a compatible way across python versions, taking into account zipimport etc.? Is it possible to get the right namespace semantics when executing __new__.py *after* the package module object has been created? Of course it depends on what those semantics are, which is currently underspecified....) And again, this is to be compared to the __class__ assignment approach, where we know very well that a backcompat shim is doable because it is done :-). -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Oct 9 17:20:09 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 09 Oct 2014 08:20:09 -0700 Subject: [Python-ideas] issubclass(collections.OrderedDict, collections.Sequence) In-Reply-To: <1412858953.4020216.177019361.49EBCFF0@webmail.messagingengine.com> References: <1412713711.2847879.176293293.031E9C7A@webmail.messagingengine.com> <87vbnvqsdi.fsf@uwakimon.sk.tsukuba.ac.jp> <1412740471.1168526.176418849.1B9957D4@webmail.messagingengine.com> <87k34bqirq.fsf@uwakimon.sk.tsukuba.ac.jp> <87d2a3qapm.fsf@uwakimon.sk.tsukuba.ac.jp> <87zjd5pym1.fsf@uwakimon.sk.tsukuba.ac.jp> <1412858953.4020216.177019361.49EBCFF0@webmail.messagingengine.com> Message-ID: <5436A7A9.2010601@stoneleaf.us> On 10/09/2014 05:49 AM, random832 at fastmail.us wrote: > > > On Thu, Oct 9, 2014, at 07:46, Ed Kellett wrote: >> On 9 October 2014 07:04, Stephen J. Turnbull wrote: >>> No, I didn't miss that at all. I just have a different point that I >>> think is far more important. I think you, random, and Ram are wasting >>> a lot of your own time defending this idea in the terms you are using, >>> and I'm trying to explain why I believe that. I'll happily stop, >>> right here, since I've been entirely unsuccessful so far. >> >> I don't actually believe I have defended this idea once. I don't care >> whether it is added to the stdlib - I just didn't want it to get shut >> down because of a miscommunication, rather than because people didn't >> think its uses were useful enough. > > Same here, personally I think his use case would be better served by two > functions, or by a flag that's passed in by the caller. There's no > particular reason to _not_ want to do the unordered subset comparison on > a list. Well, my thanks to all three of you! Between the back-and-forth a clear picture of what Ram wanted emerged, and that's a good thing. :) -- ~Ethan~ From ncoghlan at gmail.com Fri Oct 10 00:21:06 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Oct 2014 08:21:06 +1000 Subject: [Python-ideas] metamodules (was: Re: Idea to support lazy loaded names.) In-Reply-To: References: Message-ID: On 10 Oct 2014 00:05, "Nathaniel Smith" wrote: > > > Valid reasons would be to make it easier for alternative interpreters, > > or to catch edge cases (e.g. make it work nicely with custom > > importers, zip files, etc). But at that point, we can just do it the > > "right way"*, have the backcompat story to a third-party shim. > > > > * i.e. "not paying attention to older Pythons" ? I'm not saying > > __new__.py is necessarily the right way, I'm criticizing this reason > > against it > > My point is just that it matters whether a backcompat shim is doable. Probably there is some way to implement a backcompat shim for __new__.py, but I don't immediately know how to do it. Eric Snow already backported the Python 3.4 importlib to Python 2.7 as importlib2, and I know of at least one large company that is planning to deploy that in order to benefit from the directory caching feature. It's a reasonably safe assumption that any future Python 3 import system changes will be made available for Python 2.7 the same way. That said... > And again, this is to be compared to the __class__ assignment approach, where we know very well that a backcompat shim is doable because it is done :-). I actually agree making modules a proxy type that allows setting __class__ on instances is likely the simpler solution here. You won't get full magic method support, but it *will* enable use of the descriptor protocol for module level attributes. Cheers, Nick. > > -n > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Fri Oct 10 01:02:46 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 9 Oct 2014 16:02:46 -0700 Subject: [Python-ideas] metamodules (was: Re: Idea to support lazy loaded names.) In-Reply-To: References: Message-ID: On Oct 9, 2014, at 15:21, Nick Coghlan wrote: > On 10 Oct 2014 00:05, "Nathaniel Smith" wrote: > > > > > Valid reasons would be to make it easier for alternative interpreters, > > > or to catch edge cases (e.g. make it work nicely with custom > > > importers, zip files, etc). But at that point, we can just do it the > > > "right way"*, have the backcompat story to a third-party shim. > > > > > > * i.e. "not paying attention to older Pythons" ? I'm not saying > > > __new__.py is necessarily the right way, I'm criticizing this reason > > > against it > > > > My point is just that it matters whether a backcompat shim is doable. Probably there is some way to implement a backcompat shim for __new__.py, but I don't immediately know how to do it. > > Eric Snow already backported the Python 3.4 importlib to Python 2.7 as importlib2, and I know of at least one large company that is planning to deploy that in order to benefit from the directory caching feature. It's a reasonably safe assumption that any future Python 3 import system changes will be made available for Python 2.7 the same way. > > That said... > > > And again, this is to be compared to the __class__ assignment approach, where we know very well that a backcompat shim is doable because it is done :-). > > I actually agree making modules a proxy type that allows setting __class__ on instances is likely the simpler solution here. > By "proxy type", you mean that instead of turning off the non-heap flag for ModuleType, we'd just add a (pure Python, or C but not non-heap) ModuleType that delegates to the real one? If so, that certainly sounds just as easy to backport to 2.7 with importlib2 as any of the other solutions; no need for ctypes hackery. > You won't get full magic method support, but it *will* enable use of the descriptor protocol for module level attributes. > > Cheers, > Nick. > > > > > -n > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri Oct 10 02:16:55 2014 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 10 Oct 2014 01:16:55 +0100 Subject: [Python-ideas] metamodules (was: Re: Idea to support lazy loaded names.) In-Reply-To: References: Message-ID: On Thu, Oct 9, 2014 at 11:21 PM, Nick Coghlan wrote: > > On 10 Oct 2014 00:05, "Nathaniel Smith" wrote: >> >> > Valid reasons would be to make it easier for alternative interpreters, >> > or to catch edge cases (e.g. make it work nicely with custom >> > importers, zip files, etc). But at that point, we can just do it the >> > "right way"*, have the backcompat story to a third-party shim. >> > >> > * i.e. "not paying attention to older Pythons" ? I'm not saying >> > __new__.py is necessarily the right way, I'm criticizing this reason >> > against it >> >> My point is just that it matters whether a backcompat shim is doable. >> Probably there is some way to implement a backcompat shim for __new__.py, >> but I don't immediately know how to do it. > > Eric Snow already backported the Python 3.4 importlib to Python 2.7 as > importlib2, and I know of at least one large company that is planning to > deploy that in order to benefit from the directory caching feature. It's a > reasonably safe assumption that any future Python 3 import system changes > will be made available for Python 2.7 the same way. I don't really see how importlib2 can help here. I mean, we can't reasonably tell people "okay, this is very important -- before you run 'import numpy' (or whatever) you first have to run 'import importlib2; importlib2.hook.install()' or else it won't work". The bootstrapping problem here is nasty, because the whole idea is that __new__.py would be executed before *any* other code distributed with the package. > > That said... > >> And again, this is to be compared to the __class__ assignment approach, >> where we know very well that a backcompat shim is doable because it is done >> :-). > > I actually agree making modules a proxy type that allows setting __class__ > on instances is likely the simpler solution here. > > You won't get full magic method support, but it *will* enable use of the > descriptor protocol for module level attributes. Like Andrew, I'm not sure what you mean here. Note that there's various code (e.g. reload()) which insists that module objects must be instances of the built-in ModuleType class. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org From Stephan.Sahm at gmx.de Fri Oct 10 11:11:36 2014 From: Stephan.Sahm at gmx.de (Stephan Sahm) Date: Fri, 10 Oct 2014 11:11:36 +0200 Subject: [Python-ideas] reduce(func, seq, initial=0) Message-ID: Hi folk, I stumbled upon the builtin reduce function. It has an optional parameter initial, however if trying to set it like import operator as op reduce(op.add, range(10), initial=0) I get TypeError: reduce() takes no keyword arguments I think it would be really straightforward to also add keyword possibilities, at least to the parameter initial. What do you think? best, Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From flying-sheep at web.de Fri Oct 10 12:05:20 2014 From: flying-sheep at web.de (Philipp A.) Date: Fri, 10 Oct 2014 12:05:20 +0200 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> <87eguotjoi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: you?re right, all of this works. iterating over all of unicode simply looked to big a task for me, so i didn?t consider it, but apparently it works well enough. yet one puzzle piece is missing: blocks. python has no built-in information about unicode blocks (which are basically range()s with associated names). an API involving blocks would need a way to enumerate them, to get the range for a name, and the name for a char/codepoint. 2014-10-04 9:13 GMT+02:00 Chris Angelico : > On Sat, Oct 4, 2014 at 4:47 PM, Stephen J. Turnbull > wrote: > >>>> names = [unicodedata.name(chr(i)) for i in range(sys.maxunicode+1)] > > Traceback (most recent call last): > > File "", line 1, in > > File "", line 1, in > > ValueError: no such name > > > > oops, although you didn't actually claim that would work. :-) (BTW, > > chr(0) has no name. At least it was instantaneous. :-) > > Oops, forgot about that. Yet another case where the absence of PEP 463 > forces the function to have an additional argument: > > names = [unicodedata.name(chr(i), '') for i in range(sys.maxunicode+1)] > > Now it works. Sorry for the omission, this is what happens when code > is typed straight into the email without testing :) > > > Then > > > >>>> for i in range(sys.maxunicode+1): > > ... try: > > ... names.append(unicodedata.name(chr(i))) > > ... except ValueError: > > ... pass > > ... > > I would recommend appending a shim in the ValueError branch, to allow > the indexing to be correct. Which would look something like this: > > names = [unicodedata.name(chr(i)) except ValueError: '' for i in > range(sys.maxunicode+1)] > > Or, since name() does indeed have a 'default' parameter, the code from > above. :) > > > takes between 1 and 2 seconds, while > > > >>>> names.index("PILE OF POO") > > 61721 > >>>> "PILE OF POO" in names > > True > > > > is instantaneous. Note: 61721 is *much* smaller than 0x1F4A9. > > >>> names.index("PILE OF POO") > 128169 > >>> hex(_).upper() > '0X1F4A9' > > And still instantaneous. Of course, a prefix search is a bit slower: > > >>> [i for i,s in enumerate(names) if s.startswith("PILE")] > [128169] > > Takes about 1s on my aging Windows laptop, where the building of the > list takes about 4s, so it should be quicker on your system. > > The big downside, I guess, is the RAM usage. > > >>> sys.getsizeof(names) > 4892352 > >>> sum(sys.getsizeof(n) for n in names) > 30698194 > > That's ~32MB of stuff stored, just to allow these lookups. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Oct 10 12:07:08 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 10 Oct 2014 21:07:08 +1100 Subject: [Python-ideas] reduce(func, seq, initial=0) In-Reply-To: References: Message-ID: On Fri, Oct 10, 2014 at 8:11 PM, Stephan Sahm wrote: > I stumbled upon the builtin reduce function. It has an optional parameter > initial, however if trying to set it like > > import operator as op > reduce(op.add, range(10), initial=0) > > I get > > TypeError: reduce() takes no keyword arguments > > > I think it would be really straightforward to also add keyword > possibilities, at least to the parameter initial. The first thing to note is that reduce(), in Python 3, is now imported from functools; if you're working with Python 2, you won't see this changed unless you can convince people that this is a bug to be fixed. However, prefixing your code with "from functools import reduce" produces the exact same result in Python 3. The question is, though: What's the advantage? The first two arguments are mandatory, and there's only one optional argument. Do you need to say "initial=" on that? ChrisA From steve at pearwood.info Fri Oct 10 12:10:10 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 10 Oct 2014 21:10:10 +1100 Subject: [Python-ideas] reduce(func, seq, initial=0) In-Reply-To: References: Message-ID: <20141010101008.GA23419@ando.pearwood.info> On Fri, Oct 10, 2014 at 11:11:36AM +0200, Stephan Sahm wrote: > Hi folk, > > I stumbled upon the builtin reduce function. It has an optional parameter > initial, however if trying to set it like > > import operator as op > reduce(op.add, range(10), initial=0) > > I get > > TypeError: reduce() takes no keyword arguments > > > I think it would be really straightforward to also add keyword > possibilities, at least to the parameter initial. Unfortunately, this is a limitation of many built-in functions and methods, possibly the majority. E.g.: Help on built-in function getattr in module builtins: getattr(...) getattr(object, name[, default]) -> value but: py> getattr(None, name='spam', default=42) Traceback (most recent call last): File "", line 1, in TypeError: getattr() takes no keyword arguments I seem to recall that there was a reason for this, but I don't quite remember what... searching the archives may give a clue. Personally, I have no objections *in principle* to adding support for keyword arguments to built-ins, but it will depend on the details of why they weren't already supported. -- Steven From ncoghlan at gmail.com Fri Oct 10 12:24:11 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Oct 2014 20:24:11 +1000 Subject: [Python-ideas] metamodules (was: Re: Idea to support lazy loaded names.) In-Reply-To: References: Message-ID: On 10 Oct 2014 10:16, "Nathaniel Smith" wrote: > > On Thu, Oct 9, 2014 at 11:21 PM, Nick Coghlan wrote: > > > > On 10 Oct 2014 00:05, "Nathaniel Smith" wrote: > >> > >> > Valid reasons would be to make it easier for alternative interpreters, > >> > or to catch edge cases (e.g. make it work nicely with custom > >> > importers, zip files, etc). But at that point, we can just do it the > >> > "right way"*, have the backcompat story to a third-party shim. > >> > > >> > * i.e. "not paying attention to older Pythons" ? I'm not saying > >> > __new__.py is necessarily the right way, I'm criticizing this reason > >> > against it > >> > >> My point is just that it matters whether a backcompat shim is doable. > >> Probably there is some way to implement a backcompat shim for __new__.py, > >> but I don't immediately know how to do it. > > > > Eric Snow already backported the Python 3.4 importlib to Python 2.7 as > > importlib2, and I know of at least one large company that is planning to > > deploy that in order to benefit from the directory caching feature. It's a > > reasonably safe assumption that any future Python 3 import system changes > > will be made available for Python 2.7 the same way. > > I don't really see how importlib2 can help here. I mean, we can't > reasonably tell people "okay, this is very important -- before you run > 'import numpy' (or whatever) you first have to run 'import importlib2; > importlib2.hook.install()' or else it won't work". Yes, that's exactly what the Python 2 usage model would be if relying on an import system feature. > The bootstrapping > problem here is nasty, because the whole idea is that __new__.py would > be executed before *any* other code distributed with the package. If folks want Python 3 import system features in Python 2.7, importlib2 is how to get them. Most single source packages won't want to do that because of the bootstrapping challenge, but that's not the same thing as being entirely incompatible with Python 2. > > That said... > > > >> And again, this is to be compared to the __class__ assignment approach, > >> where we know very well that a backcompat shim is doable because it is done > >> :-). > > > > I actually agree making modules a proxy type that allows setting __class__ > > on instances is likely the simpler solution here. > > > > You won't get full magic method support, but it *will* enable use of the > > descriptor protocol for module level attributes. > > Like Andrew, I'm not sure what you mean here. > > Note that there's various code (e.g. reload()) which insists that > module objects must be instances of the built-in ModuleType class. I'm agreeing with you. One of the main characteristics of proxy types is that type(x) and x.__class__ give different answers, which is how I interpreted your metamodule proposal. Cheers, Nick. > > -n > > -- > Nathaniel J. Smith > Postdoctoral researcher - Informatics - University of Edinburgh > http://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Oct 10 12:25:39 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 Oct 2014 12:25:39 +0200 Subject: [Python-ideas] reduce(func, seq, initial=0) In-Reply-To: <20141010101008.GA23419@ando.pearwood.info> References: <20141010101008.GA23419@ando.pearwood.info> Message-ID: Steven D'Aprano schrieb am 10.10.2014 um 12:10: > Personally, I have no objections *in principle* to adding support for > keyword arguments to built-ins, but it will depend on the details of why > they weren't already supported. There was a major move towards proper argument parsing for builtins and C implemented stdlib modules in Py3.4, but it's still not finished (as this example shows). AFAICT, help is always appreciated on python-dev side. Stefan From ncoghlan at gmail.com Fri Oct 10 12:27:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Oct 2014 20:27:01 +1000 Subject: [Python-ideas] reduce(func, seq, initial=0) In-Reply-To: References: Message-ID: On 10 Oct 2014 20:07, "Chris Angelico" wrote: > > On Fri, Oct 10, 2014 at 8:11 PM, Stephan Sahm wrote: > > I stumbled upon the builtin reduce function. It has an optional parameter > > initial, however if trying to set it like > > > > import operator as op > > reduce(op.add, range(10), initial=0) > > > > I get > > > > TypeError: reduce() takes no keyword arguments > > > > > > I think it would be really straightforward to also add keyword > > possibilities, at least to the parameter initial. > > The first thing to note is that reduce(), in Python 3, is now imported > from functools; if you're working with Python 2, you won't see this > changed unless you can convince people that this is a bug to be fixed. > However, prefixing your code with "from functools import reduce" > produces the exact same result in Python 3. > > The question is, though: What's the advantage? The first two arguments > are mandatory, and there's only one optional argument. Do you need to > say "initial=" on that? Converting functools to argument clinic for 3.5 should fix it. I don't believe anyone has tackled that yet. Cheers, Nick. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at the-compiler.org Fri Oct 10 13:05:59 2014 From: me at the-compiler.org (Florian Bruhin) Date: Fri, 10 Oct 2014 13:05:59 +0200 Subject: [Python-ideas] reduce(func, seq, initial=0) In-Reply-To: <20141010101008.GA23419@ando.pearwood.info> References: <20141010101008.GA23419@ando.pearwood.info> Message-ID: <20141010110559.GD23422@lupin> * Steven D'Aprano [2014-10-10 21:10:10 +1100]: > On Fri, Oct 10, 2014 at 11:11:36AM +0200, Stephan Sahm wrote: > > Hi folk, > > > > I stumbled upon the builtin reduce function. It has an optional parameter > > initial, however if trying to set it like > > > > import operator as op > > reduce(op.add, range(10), initial=0) > > > > I get > > > > TypeError: reduce() takes no keyword arguments > > > > > > I think it would be really straightforward to also add keyword > > possibilities, at least to the parameter initial. > > Unfortunately, this is a limitation of many built-in functions and > methods, possibly the majority. E.g.: > > > Help on built-in function getattr in module builtins: > getattr(...) > getattr(object, name[, default]) -> value > > > but: > > py> getattr(None, name='spam', default=42) > Traceback (most recent call last): > File "", line 1, in > TypeError: getattr() takes no keyword arguments > > > I seem to recall that there was a reason for this, but I don't quite > remember what... searching the archives may give a clue. > > Personally, I have no objections *in principle* to adding support for > keyword arguments to built-ins, but it will depend on the details of why > they weren't already supported. FWIW, I'd also really like to see keyword arguments for these builtins. I always have to keep myself from doing things like "foo bar baz".split(count=1) But as said, I suspect there is *some* reason this is not possible. Florian -- http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP) GPG 0xFD55A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From stefan_ml at behnel.de Fri Oct 10 13:29:51 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 Oct 2014 13:29:51 +0200 Subject: [Python-ideas] reduce(func, seq, initial=0) In-Reply-To: <20141010110559.GD23422@lupin> References: <20141010101008.GA23419@ando.pearwood.info> <20141010110559.GD23422@lupin> Message-ID: Florian Bruhin schrieb am 10.10.2014 um 13:05: > FWIW, I'd also really like to see keyword arguments for these > builtins. I always have to keep myself from doing things like > > "foo bar baz".split(count=1) > > But as said, I suspect there is *some* reason this is not possible. You only have to use the right argument name (and Py3.4): >>> "abc def".split(maxsplit=1) ['abc', 'def'] Stefan From flying-sheep at web.de Fri Oct 10 14:43:15 2014 From: flying-sheep at web.de (Philipp A.) Date: Fri, 10 Oct 2014 14:43:15 +0200 Subject: [Python-ideas] Extend unicodedata with a name search In-Reply-To: References: <542F11F1.5070607@egenix.com> <87mw9ctteh.fsf@uwakimon.sk.tsukuba.ac.jp> <87eguotjoi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: oh, and name aliases may be supported ? unicodedata.lookup('BEL') works, but there?s no way for the reverse operation. so i suggest to introduce: 1. everything from https://github.com/nagisa/unicodeblocks 2. unicodedata.names(chr) ? list of primary name and all aliases, possibly empty (therefore no default) 2014-10-10 12:05 GMT+02:00 Philipp A. : > you?re right, all of this works. > > iterating over all of unicode simply looked to big a task for me, so i > didn?t consider it, but apparently it works well enough. > > yet one puzzle piece is missing: blocks. > > python has no built-in information about unicode blocks (which are > basically range()s with associated names). > > an API involving blocks would need a way to enumerate them, to get the > range for a name, and the name for a char/codepoint. > > 2014-10-04 9:13 GMT+02:00 Chris Angelico : > >> On Sat, Oct 4, 2014 at 4:47 PM, Stephen J. Turnbull >> wrote: >> >>>> names = [unicodedata.name(chr(i)) for i in range(sys.maxunicode+1)] >> > Traceback (most recent call last): >> > File "", line 1, in >> > File "", line 1, in >> > ValueError: no such name >> > >> > oops, although you didn't actually claim that would work. :-) (BTW, >> > chr(0) has no name. At least it was instantaneous. :-) >> >> Oops, forgot about that. Yet another case where the absence of PEP 463 >> forces the function to have an additional argument: >> >> names = [unicodedata.name(chr(i), '') for i in range(sys.maxunicode+1)] >> >> Now it works. Sorry for the omission, this is what happens when code >> is typed straight into the email without testing :) >> >> > Then >> > >> >>>> for i in range(sys.maxunicode+1): >> > ... try: >> > ... names.append(unicodedata.name(chr(i))) >> > ... except ValueError: >> > ... pass >> > ... >> >> I would recommend appending a shim in the ValueError branch, to allow >> the indexing to be correct. Which would look something like this: >> >> names = [unicodedata.name(chr(i)) except ValueError: '' for i in >> range(sys.maxunicode+1)] >> >> Or, since name() does indeed have a 'default' parameter, the code from >> above. :) >> >> > takes between 1 and 2 seconds, while >> > >> >>>> names.index("PILE OF POO") >> > 61721 >> >>>> "PILE OF POO" in names >> > True >> > >> > is instantaneous. Note: 61721 is *much* smaller than 0x1F4A9. >> >> >>> names.index("PILE OF POO") >> 128169 >> >>> hex(_).upper() >> '0X1F4A9' >> >> And still instantaneous. Of course, a prefix search is a bit slower: >> >> >>> [i for i,s in enumerate(names) if s.startswith("PILE")] >> [128169] >> >> Takes about 1s on my aging Windows laptop, where the building of the >> list takes about 4s, so it should be quicker on your system. >> >> The big downside, I guess, is the RAM usage. >> >> >>> sys.getsizeof(names) >> 4892352 >> >>> sum(sys.getsizeof(n) for n in names) >> 30698194 >> >> That's ~32MB of stuff stored, just to allow these lookups. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From me at the-compiler.org Fri Oct 10 14:58:39 2014 From: me at the-compiler.org (Florian Bruhin) Date: Fri, 10 Oct 2014 14:58:39 +0200 Subject: [Python-ideas] reduce(func, seq, initial=0) In-Reply-To: References: <20141010101008.GA23419@ando.pearwood.info> <20141010110559.GD23422@lupin> Message-ID: <20141010125839.GF23422@lupin> * Stefan Behnel [2014-10-10 13:29:51 +0200]: > Florian Bruhin schrieb am 10.10.2014 um 13:05: > > FWIW, I'd also really like to see keyword arguments for these > > builtins. I always have to keep myself from doing things like > > > > "foo bar baz".split(count=1) > > > > But as said, I suspect there is *some* reason this is not possible. > > You only have to use the right argument name (and Py3.4): > > >>> "abc def".split(maxsplit=1) > ['abc', 'def'] > > Stefan Oh, thanks for the heads-up! The name was from memory, and I guess the last time I tested this was with 3.3. Florian -- http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP) GPG 0xFD55A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From ncoghlan at gmail.com Fri Oct 10 16:05:19 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Oct 2014 00:05:19 +1000 Subject: [Python-ideas] reduce(func, seq, initial=0) In-Reply-To: <20141010110559.GD23422@lupin> References: <20141010101008.GA23419@ando.pearwood.info> <20141010110559.GD23422@lupin> Message-ID: On 10 Oct 2014 21:07, "Florian Bruhin" wrote: > > FWIW, I'd also really like to see keyword arguments for these > builtins. I always have to keep myself from doing things like > > "foo bar baz".split(count=1) > > But as said, I suspect there is *some* reason this is not possible. The technical reason is that many older CPython functions written in C use PyArg_ParseTuple, which only supports positional arguments. When there's a concrete readability gain from adding keyword argument support to a particular function, patches will often be accepted. Switching to argument clinic also offers improved introspection support, so that can be a good way to pursue the enhancement. Regards, Nick. > > Florian > > -- > http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP) > GPG 0xFD55A072 | http://the-compiler.org/pubkey.asc > I love long mails! | http://email.is-not-s.ms/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.jorgensen.za at gmail.com Fri Oct 10 17:06:19 2014 From: adam.jorgensen.za at gmail.com (Adam Jorgensen) Date: Fri, 10 Oct 2014 17:06:19 +0200 Subject: [Python-ideas] Query about proposed Multi-line Anonymous Function syntaxes Message-ID: Hey all, my first time posting here. So, as I understand it, the main reason we don't have multi-line anonymous functions is because there has been no consensus about the syntax. I was just wondering if someone has a document detailing the various syntaxes that have been proposed in the the past. Adam -------------- next part -------------- An HTML attachment was scrubbed... URL: From adam.jorgensen.za at gmail.com Fri Oct 10 17:09:42 2014 From: adam.jorgensen.za at gmail.com (Adam Jorgensen) Date: Fri, 10 Oct 2014 17:09:42 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <20141004132741.GK19757@ando.pearwood.info> Message-ID: I don't think it makes much sense for len() to work on generators and the fact that sum() works isn't a good argument. Summing the contents of a generator can make sense whereas attempting to obtain the length of something which specifically does not define a length seems a little nonsensical to me... On 5 October 2014 02:37, Guido van Rossum wrote: > Support for len() on iterators isn't going to happen. > > The "gut feeling" reason is that len() shouldn't "consume" anything. It > would lure people into thinking they can first call len() on the iterator > and then iterate over it -- while, if it is an actual iterator (like an I/O > stream), after calling len(), everything is consumed. > > Yes, it's possible that you only want to count the number of lines, but > that's unusual, and the idiom for that is readily available (e.g. sum(1 for > ...)). If the idiom occurs frequently in your code you can define a helper > function count(). The speed benefit of doing the loop in C would be minimal > in most cases. > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Fri Oct 10 20:06:20 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 10 Oct 2014 14:06:20 -0400 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <20141004132741.GK19757@ando.pearwood.info> Message-ID: <1412964380.393885.177565373.2EBD85EB@webmail.messagingengine.com> On Fri, Oct 10, 2014, at 11:09, Adam Jorgensen wrote: > I don't think it makes much sense for len() to work on generators and the > fact that sum() works isn't a good argument. > > Summing the contents of a generator can make sense whereas attempting to > obtain the length of something which specifically does not define a > length > seems a little nonsensical to me... Why doesn't it define a length? No, hear me out. Is there any reason that, for example, generator expressions on lists or ranges shouldn't be read-only views instead of generators? From rosuav at gmail.com Fri Oct 10 20:10:52 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 11 Oct 2014 05:10:52 +1100 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: <1412964380.393885.177565373.2EBD85EB@webmail.messagingengine.com> References: <20141004132741.GK19757@ando.pearwood.info> <1412964380.393885.177565373.2EBD85EB@webmail.messagingengine.com> Message-ID: On Sat, Oct 11, 2014 at 5:06 AM, wrote: > On Fri, Oct 10, 2014, at 11:09, Adam Jorgensen wrote: >> I don't think it makes much sense for len() to work on generators and the >> fact that sum() works isn't a good argument. >> >> Summing the contents of a generator can make sense whereas attempting to >> obtain the length of something which specifically does not define a >> length >> seems a little nonsensical to me... > > Why doesn't it define a length? No, hear me out. Is there any reason > that, for example, generator expressions on lists or ranges shouldn't be > read-only views instead of generators? For a start, you'd have to preclude anything with an 'if' in it. What's the length of this generator? gen = (sin(x*.0314159) for x in range(100) if f(x)) Without actually exhausting the iterator, there's no way to predict how many values it'll produce. The only way to do it would be to restrict this to generators that basically map a function/expression over an iterable of known length. ChrisA From jmcs at jsantos.eu Fri Oct 10 20:36:04 2014 From: jmcs at jsantos.eu (=?UTF-8?B?Sm/Do28gU2FudG9z?=) Date: Fri, 10 Oct 2014 20:36:04 +0200 Subject: [Python-ideas] Query about proposed Multi-line Anonymous Function syntaxes In-Reply-To: References: Message-ID: >From what I understand there was also no consensus about having a multi line anonymous function syntax actually being an advantage over having to define a "normal" function for that use case. On 10 October 2014 17:06, Adam Jorgensen wrote: > Hey all, my first time posting here. > > So, as I understand it, the main reason we don't have multi-line anonymous > functions is because there has been no consensus about > the syntax. > > I was just wondering if someone has a document detailing the various > syntaxes that have been proposed in the the past. > > Adam > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.us Fri Oct 10 20:39:54 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 10 Oct 2014 14:39:54 -0400 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <20141004132741.GK19757@ando.pearwood.info> <1412964380.393885.177565373.2EBD85EB@webmail.messagingengine.com> Message-ID: <1412966394.402982.177576957.449BA897@webmail.messagingengine.com> On Fri, Oct 10, 2014, at 14:10, Chris Angelico wrote: > On Sat, Oct 11, 2014 at 5:06 AM, wrote: > > Why doesn't it define a length? No, hear me out. Is there any reason > > that, for example, generator expressions on lists or ranges shouldn't be > > read-only views instead of generators? > > For a start, you'd have to preclude anything with an 'if' in it. Okay, and? (I couldn't remember if the "if" feature was a real feature or a proposed on, or I would have mentioned it.) From guido at python.org Fri Oct 10 20:42:56 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 10 Oct 2014 11:42:56 -0700 Subject: [Python-ideas] Query about proposed Multi-line Anonymous Function syntaxes In-Reply-To: References: Message-ID: On Fri, Oct 10, 2014 at 8:06 AM, Adam Jorgensen wrote: > > So, as I understand it, the main reason we don't have multi-line anonymous > functions is because there has been no consensus about > the syntax. > Well, it's worse -- it's hard to propose a syntax that actually works without huge changes to the way Python's parser treats indentation: inside parentheses, indentation is currently *not* significant. > I was just wondering if someone has a document detailing the various > syntaxes that have been proposed in the the past. > No, but there have been many discussions. If you have a taste for mailing list archaeology it would be nice of you could dig up the various threads and summarize them for posterity, so at least we might make progress rather than going around in circles. As for myself, I have encountered a number of languages that do support multi-line anonymous functions (Java, JavaScript, CoffeeScript) and I have to say that for some reason they do not lead to very readable code. Especially in the latter two, the prevalent use for anonymous functions is for asynchronous I/O callbacks. I find Python 3.4's "yield from" as used by the asyncio library leads to much more readable code, without he need for callback functions at all. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Oct 10 20:52:10 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 10 Oct 2014 11:52:10 -0700 Subject: [Python-ideas] Query about proposed Multi-line Anonymous Function syntaxes In-Reply-To: References: Message-ID: <54382ADA.3000804@stoneleaf.us> On 10/10/2014 11:42 AM, Guido van Rossum wrote: > > As for myself, I have encountered a number of languages that do support multi-line anonymous functions (Java, > JavaScript, CoffeeScript) and I have to say that for some reason they do not lead to very readable code. My take on the less-readable-code angle is that one of the main reasons for writing procedures is for abstraction: def some_func(spammer, target): """ fraggle the target with spammer """ You can look at that signature and have a rough idea of whats happening. On the other hand, anonymous blocks have the serious drawback of blowing your abstraction to bits: def some_func( multi_lambda: { /* many lines to define spammer /* } multi_lambda: { /* many lines to retrieve the target /* } ): { /* many lines to implement some_func /* } What a readability nightmare. -- ~Ethan~ From random832 at fastmail.us Fri Oct 10 21:25:25 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Fri, 10 Oct 2014 15:25:25 -0400 Subject: [Python-ideas] Query about proposed Multi-line Anonymous Function syntaxes In-Reply-To: References: Message-ID: <1412969125.417619.177588237.18521841@webmail.messagingengine.com> On Fri, Oct 10, 2014, at 14:42, Guido van Rossum wrote: > Well, it's worse -- it's hard to propose a syntax that actually works > without huge changes to the way Python's parser treats indentation: > inside > parentheses, indentation is currently *not* significant. What about forgetting about indentation entirely, and just using semicolons. You could use an extra semicolon to "break out" of the scope created by e.g. an if statement. lambda a: (if a: b; c;; else d; e;) equivalent to def f(a): if a: b return c else: d return e Basically, is the real reason people ask for multi-line lambdas because they want multiple lines, or just because they want multiple statements? And conversely, if you don't like the implications of allowing an unlimited number of statements of any kind on one line / within an expression, do you really want to give them multi-line lambdas, even if the syntax issue magically resolved itself? From ncoghlan at gmail.com Sat Oct 11 00:11:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Oct 2014 08:11:20 +1000 Subject: [Python-ideas] Query about proposed Multi-line Anonymous Function syntaxes In-Reply-To: References: Message-ID: On 11 Oct 2014 04:44, "Guido van Rossum" wrote: > > On Fri, Oct 10, 2014 at 8:06 AM, Adam Jorgensen < adam.jorgensen.za at gmail.com> wrote: >> >> >> So, as I understand it, the main reason we don't have multi-line anonymous functions is because there has been no consensus about >> the syntax. > > > Well, it's worse -- it's hard to propose a syntax that actually works without huge changes to the way Python's parser treats indentation: inside parentheses, indentation is currently *not* significant. > >> >> I was just wondering if someone has a document detailing the various syntaxes that have been proposed in the the past. > > > No, but there have been many discussions. If you have a taste for mailing list archaeology it would be nice of you could dig up the various threads and summarize them for posterity, so at least we might make progress rather than going around in circles. There's also my 3 different documented suggestions aimed at this problem space: PEP 403 (overriding the name binding step for function and class definitions): http://www.python.org/dev/peps/pep-0403/ PEP 3150 (forward references to a trailing statement local namespace): http://www.python.org/dev/peps/pep-3150/ Suite expressions: http://python-notes.curiousefficiency.org/en/latest/pep_ideas/suite_expr.html The main stumbling block for all of them isn't getting agreement on syntax as such, but rather finding compelling examples that can't already be written more clearly by refactoring them to remove the need for a multi-line inline function. I don't have such examples myself, which is why both PEPs are deferred, and the last idea was never even submitted in the first place. Without such compelling examples, it's hard to judge the value of the syntax proposals themselves, as it's difficult to counter the "well, don't write it like that" argument. I asked various folks at SciPy earlier this year to look into coming up with such examples (as that was one of the best aspects of the matrix multiplication PEP), but haven't heard anything back as yet. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From t.chaumeny at gmail.com Sat Oct 11 00:15:55 2014 From: t.chaumeny at gmail.com (Thomas) Date: Sat, 11 Oct 2014 00:15:55 +0200 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: References: <20141004132741.GK19757@ando.pearwood.info> Message-ID: Well, I'm already convinced actually :) The thing is I brought that question based on the fact that "sum(1 for ...)" was slower then "len([foo for ...])" and I thought it was a shame there were no builtin way to count quickly an iterable. I realized afterwards that this fact is no longer true, as "sum(1 for ...)" has become faster with some recent Python version (it looks like it will not use Python objects addition if all the operands are numbers). On Fri, Oct 10, 2014 at 5:09 PM, Adam Jorgensen wrote: > I don't think it makes much sense for len() to work on generators and the > fact that sum() works isn't a good argument. > > Summing the contents of a generator can make sense whereas attempting to > obtain the length of something which specifically does not define a length > seems a little nonsensical to me... > > On 5 October 2014 02:37, Guido van Rossum wrote: > >> Support for len() on iterators isn't going to happen. >> >> The "gut feeling" reason is that len() shouldn't "consume" anything. It >> would lure people into thinking they can first call len() on the iterator >> and then iterate over it -- while, if it is an actual iterator (like an I/O >> stream), after calling len(), everything is consumed. >> >> Yes, it's possible that you only want to count the number of lines, but >> that's unusual, and the idiom for that is readily available (e.g. sum(1 for >> ...)). If the idiom occurs frequently in your code you can define a helper >> function count(). The speed benefit of doing the loop in C would be minimal >> in most cases. >> >> -- >> --Guido van Rossum (python.org/~guido) >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Oct 11 01:26:20 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 10 Oct 2014 16:26:20 -0700 Subject: [Python-ideas] Query about proposed Multi-line Anonymous Function syntaxes In-Reply-To: References: Message-ID: On Oct 10, 2014, at 11:42, Guido van Rossum wrote: > On Fri, Oct 10, 2014 at 8:06 AM, Adam Jorgensen wrote: >> >> So, as I understand it, the main reason we don't have multi-line anonymous functions is because there has been no consensus about >> the syntax. > > Well, it's worse -- it's hard to propose a syntax that actually works without huge changes to the way Python's parser treats indentation: inside parentheses, indentation is currently *not* significant. As a somewhat-related data point, compare the indentation rules in the standard Apple style for Objective C (as strongly encouraged by Xcode) before and after the addition of blocks (which are close enough to multi-line lambdas for this purpose). I don't know a single person who sat down at Xcode 3, or emacs ObjC-mode, or any of a half-dozen other assistive IDEs that all applied the same rules, and didn't pick up those rules within a few minutes of coding. But since blocks, there are full-time ObjC developers who are now on their third generation of Xcode versions and still fighting with it trying to start their code at column 70, and the rules in all those different IDEs are no longer consistent. Throwing suites into the middle of expressions inherently makes indentation more complicated. That's not to say that someone from the Python community couldn't solve this problem that Apple, Eclipse, JetBrains, etc. have failed at, just that the presumption going in shouldn't be that it's a trivial issue that's being used to unfairly dismiss a good idea. > As for myself, I have encountered a number of languages that do support multi-line anonymous functions (Java, JavaScript, CoffeeScript) and I have to say that for some reason they do not lead to very readable code. Especially in the latter two, the prevalent use for anonymous functions is for asynchronous I/O callbacks. I find Python 3.4's "yield from" as used by the asyncio library leads to much more readable code, without he need for callback functions at all. A lot of code in these languages uses different techniques to flatten out the "callback hell". The most prevalent is probably deferred/promise objects. Although these are borrowed from Twisted from the Python world, the way they interact with JS-like syntax arguably makes them more useful there: Every callback is defined as the argument to a method on a promise, which means they start in a consistent place, toward the left edge of the screen, etc., and they rarely nest, which avoids most of the problems with inline function definitions while still allowing the benefits (not having to define the functions out of order, or give them meaningless names). Of course you could easily come up with a syntax based on one of Nick's two PEPs that's even nicer. And the fact that Python has an even better solution already (asyncio-style) makes it less interesting. Also, JS definitely encourages novices to write unreadable horrible code, even if it allows experienced developers to write it more cleanly, and I don't think that's a balance Python wants. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Oct 11 01:34:37 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 10 Oct 2014 16:34:37 -0700 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: <1412964380.393885.177565373.2EBD85EB@webmail.messagingengine.com> References: <20141004132741.GK19757@ando.pearwood.info> <1412964380.393885.177565373.2EBD85EB@webmail.messagingengine.com> Message-ID: <79DD2445-D129-496F-9465-1D743BC739DD@yahoo.com> On Oct 10, 2014, at 11:06, random832 at fastmail.us wrote: > On Fri, Oct 10, 2014, at 11:09, Adam Jorgensen wrote: >> I don't think it makes much sense for len() to work on generators and the >> fact that sum() works isn't a good argument. >> >> Summing the contents of a generator can make sense whereas attempting to >> obtain the length of something which specifically does not define a >> length >> seems a little nonsensical to me... > > Why doesn't it define a length? No, hear me out. Is there any reason > that, for example, generator expressions on lists or ranges shouldn't be > read-only views instead of generators? When I first started playing with Swift, I was impressed by the fact that they tried to make as many things as possible into views instead of iterators. (I'm going to use Python terminology here, because for some reason they came up with new terms for everything, making sure that every word was a false cognate with something in either Python, C++, or Haskell...) Map returns a view which can be indexed and in general acts like a sequence. Filter returns a view that can be indexed by non-random-accessible indices (you can start with begin or end and ++ or -- from there). And so on. But when you start trying to build your own views, iterators, and sequences, or just trying to chain together the existing ones, everything gets a lot harder. It's nowhere near as nice as Python, C++, or Haskell in practice, even though it combines the best of all three in toy examples. That's not to say that a good design for this stuff isn't possible, just that it's obviously not trivial. The if example is a good illustration of this. If a genexpr is a sequence iff it has exactly 1 for clause and 0 if clauses, becoming a non-random-accessible view if it has a second for or any if, that makes it a lot harder to read, document, or teach code that uses genexprs (unless that code just ignores all view functionality and treats everything as a generic iterable). From steve at pearwood.info Sat Oct 11 07:11:40 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 11 Oct 2014 16:11:40 +1100 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: <1412964380.393885.177565373.2EBD85EB@webmail.messagingengine.com> References: <20141004132741.GK19757@ando.pearwood.info> <1412964380.393885.177565373.2EBD85EB@webmail.messagingengine.com> Message-ID: <20141011051140.GC23419@ando.pearwood.info> On Fri, Oct 10, 2014 at 02:06:20PM -0400, random832 at fastmail.us wrote: > On Fri, Oct 10, 2014, at 11:09, Adam Jorgensen wrote: > > I don't think it makes much sense for len() to work on generators and the > > fact that sum() works isn't a good argument. > > > > Summing the contents of a generator can make sense whereas attempting to > > obtain the length of something which specifically does not define a > > length > > seems a little nonsensical to me... > > Why doesn't it define a length? No, hear me out. Is there any reason > that, for example, generator expressions on lists or ranges shouldn't be > read-only views instead of generators? The important question is not "Why *doesn't* it define a length?" but "Why *should* it define a length?". What advantage does it give you? Let's take the case where you are both the producer and consumer of the generator. You might like to write something like this: it = (func(x) for x in some_list) n = len(it) consume(it, n) But that's no better or easier than: it = (func(x) for x in some_list) n = len(some_list) consume(it, n) so there is no benefit to having the generator have a length. It does no harm either. However, it does require a very specific special case. It only works when you walk directly over a fixed-length sequence, and can't be used in cases like these: it = (func(x) for x in some_list if condition(x)) it = (func(x) for x in some_iterator_of_unpredictable_length) So from the perspective of the producer, generators cannot always be given a length, and if they can, since Python can determine the length, so can you. There's no advantage to having the generator type do so that I can see. Now consider from the perspective of a consumer of an iterator. You don't know where it comes from or how it is produced, so you don't know if it has a predictable length or not. Since you can't rely on it having a length, it doesn't actually give you any benefit. Perhaps you're already writing code that supports sequences and iterators, with a separate branch for sequences based on the fact that they have a known length: def function(it): try: n = len(it) except TypeError: # process the cases with no known length else: # process cases with a known length It might be nice if some generators will be processed by the "known length" branch, but it doesn't save you from having to write the "unknown length" branch. Again, the benefit is minimal or zero. There may be cases where you *require* a predictable length, in which case you probably should support only sequences. So there is no real benefit as far as I can see why generators should support len() even when they could. -- Steven From jsbfox at gmail.com Sat Oct 11 22:34:33 2014 From: jsbfox at gmail.com (Thomas Allen) Date: Sun, 12 Oct 2014 00:34:33 +0400 Subject: [Python-ideas] Negative limit for traceback.* In-Reply-To: References: Message-ID: I'm still interested in this proposal. If I write a patch, how do I go about getting it accepted? The previous reply was wrong, sorry about that. On Aug 24, 2014 9:53 PM, "Guido van Rossum" wrote: > I like this. It's kind of sad that when the limit causes truncation of the > traceback it drops the most recent frames, which might give a hint as to > what happens, and keeps the oldest frames, which are usually less > interesting (I know my program started at main() :-). > > > On Sun, Aug 24, 2014 at 10:48 AM, Thomas Allen wrote: > >> I'd like to propose adding the ability to extract last n stack trace >> entries from a traceback using functions from traceback module. Simply >> put, passing limit=-n would make functions to return (or print or >> yield) last abs(n) entries. No-brainer implementation for extract_tb: >> >> extracted = list(_extract_tb_iter(tb, limit=(limit, None)[limit < 0])) >> if limit is not None and limit < 0: >> return extracted[limit:] >> return extracted >> >> The motivation: http://stackoverflow.com/q/25472412/2301450 >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Oct 12 00:39:58 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 11 Oct 2014 15:39:58 -0700 Subject: [Python-ideas] Negative limit for traceback.* In-Reply-To: References: Message-ID: <5439B1BE.4050800@stoneleaf.us> On 10/11/2014 01:34 PM, Thomas Allen wrote: > On Aug 24, 2014 9:53 PM, "Guido van Rossum" wrote: >> On Sun, Aug 24, 2014 at 10:48 AM, Thomas Allen wrote: >>> >>> I'd like to propose adding the ability to extract last n stack trace >>> entries from a traceback using functions from traceback module. Simply >>> put, passing limit=-n would make functions to return (or print or >>> yield) last abs(n) entries. > > >> I like this. It's kind of sad that when the limit causes truncation of the traceback it drops the most recent >> frames, which might give a hint as to what happens, and keeps the oldest frames, which are usually less interesting >> I know my program started at main() :-). > > I'm still interested in this proposal.? If I write a patch, how do I go about getting it accepted? Create an issue at bugs.python.org, write your patch and submit to the issue -- if nobody takes a look within three to four weeks, ping back here or on python-dev. -- ~Ethan~ From ncoghlan at gmail.com Sun Oct 12 01:53:55 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Oct 2014 09:53:55 +1000 Subject: [Python-ideas] Query about proposed Multi-line Anonymous Function syntaxes In-Reply-To: References: Message-ID: On 11 October 2014 09:26, Andrew Barnert wrote: > A lot of code in these languages uses different techniques to flatten out > the "callback hell". The most prevalent is probably deferred/promise > objects. Although these are borrowed from Twisted from the Python world, the > way they interact with JS-like syntax arguably makes them more useful there: > Every callback is defined as the argument to a method on a promise, which > means they start in a consistent place, toward the left edge of the screen, > etc., and they rarely nest, which avoids most of the problems with inline > function definitions while still allowing the benefits (not having to define > the functions out of order, or give them meaningless names). > > Of course you could easily come up with a syntax based on one of Nick's two > PEPs that's even nicer. And the fact that Python has an even better solution > already (asyncio-style) makes it less interesting. Also, JS definitely > encourages novices to write unreadable horrible code, even if it allows > experienced developers to write it more cleanly, and I don't think that's a > balance Python wants. Note that it's also *not* a coincidence that some of my own suggestions are closer to Ruby's blocks than they are to multi-line lambdas. The exact way blocks work in Ruby is heavily reliant on the "accept a block as the last parameter to your function" convention, which makes them hard to adapt to an existing ecosystem like Python's, but they're still an elegant design pattern worth studying. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From random832 at fastmail.us Sun Oct 12 06:22:21 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Sun, 12 Oct 2014 00:22:21 -0400 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: <79DD2445-D129-496F-9465-1D743BC739DD@yahoo.com> References: <20141004132741.GK19757@ando.pearwood.info> <1412964380.393885.177565373.2EBD85EB@webmail.messagingengine.com> <79DD2445-D129-496F-9465-1D743BC739DD@yahoo.com> Message-ID: <1413087741.3023486.177932929.66313FD3@webmail.messagingengine.com> On Fri, Oct 10, 2014, at 19:34, Andrew Barnert wrote: > The if example is a good illustration of this. If a genexpr is a sequence > iff it has exactly 1 for clause and 0 if clauses Why not 2 for clause? Won't its length always be the product of the two inputs, then? (And you forgot if its inputs are sequences) More to the point, since python isn't statically typed, there's no reason that user code would have to produce a view (though I'd suggest it should at least produce an "iterable" that next can't be called directly on without iter - a paradigm that has worked well for Java and C#) if it doesn't want to. From random832 at fastmail.us Sun Oct 12 06:24:40 2014 From: random832 at fastmail.us (random832 at fastmail.us) Date: Sun, 12 Oct 2014 00:24:40 -0400 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: <20141011051140.GC23419@ando.pearwood.info> References: <20141004132741.GK19757@ando.pearwood.info> <1412964380.393885.177565373.2EBD85EB@webmail.messagingengine.com> <20141011051140.GC23419@ando.pearwood.info> Message-ID: <1413087880.3023917.177934105.7F1D1711@webmail.messagingengine.com> On Sat, Oct 11, 2014, at 01:11, Steven D'Aprano wrote: > It might be nice if some generators will be processed by the "known > length" branch, but it doesn't save you from having to write the > "unknown length" branch. You don't have to if you don't want to. Demanding that your callers pass in an object on which __len__ is defined is no different than demanding objects with any other method defined. That's how duck typing works. From tjreedy at udel.edu Sun Oct 12 07:38:53 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 12 Oct 2014 01:38:53 -0400 Subject: [Python-ideas] Make len() usable on a generator In-Reply-To: <1413087880.3023917.177934105.7F1D1711@webmail.messagingengine.com> References: <20141004132741.GK19757@ando.pearwood.info> <1412964380.393885.177565373.2EBD85EB@webmail.messagingengine.com> <20141011051140.GC23419@ando.pearwood.info> <1413087880.3023917.177934105.7F1D1711@webmail.messagingengine.com> Message-ID: On 10/12/2014 12:24 AM, random832 at fastmail.us wrote: > On Sat, Oct 11, 2014, at 01:11, Steven D'Aprano wrote: >> It might be nice if some generators will be processed by the "known >> length" branch, but it doesn't save you from having to write the >> "unknown length" branch. > > You don't have to if you don't want to. Demanding that your callers pass > in an object on which __len__ is defined is no different than demanding > objects with any other method defined. That's how duck typing works. Iterators, including generators, can optionally have a __length_hint__ method, which can be used by list, etc, for better initial space allocation. "object.__length_hint__(self) Called to implement operator.length_hint(). Should return an estimated length for the object (which may be greater or less than the actual length). The length must be an integer >= 0. This method is purely an optimization and is never required for correctness." I believe this was added when Raymond H. noticed that some of the itertools generators could sometimes give accurate length information. Guido did not want to add __len__, just sometimes, lest it incorrectly become regarded as part of the iterator protocol. -- Terry Jan Reedy From victor.stinner at gmail.com Mon Oct 13 23:40:11 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 13 Oct 2014 23:40:11 +0200 Subject: [Python-ideas] including psutil in the standard library? In-Reply-To: <542319B1.2090606@ferrara.linux.it> References: <542319B1.2090606@ferrara.linux.it> Message-ID: 2014-09-24 21:21 GMT+02:00 Stefano Borini : > I am wondering if it would be possible to include psutil > (https://pypi.python.org/pypi/psutil ) in the standard library, and if not, > what would be needed. The problem is that Python only releases a new major version every 18 months (or something like that), so adding a single new function will take so much time. Supporting a new platform, architecture, major version of an OS, etc. may also need to wait a new Python major release. Operating systems are evolving quite fast. For example, there is the new architecture AArch64, containers are more and more popular, systemd can also be used to provide features similar to psutil and it supports containers (running systemd, it even supports recursive containers....). Network becomes more virtual with Software Defined Networks (SDN, NFV, etc.). The Linux kernel is also extended at each release. Example of "recent" addition: /proc/pid/fdinfo/ directory. Does psutil provide NUMA information? NUMA also becomes more information nowadays for performances. The psutil also still evolves. For example, I see that the version 2.1 released at April, 2014 adds "netstat-like functionalities". The version 2.0 was only released one month before (March, 2014). The API of psutil changed a lot between 1.x and 2.0. The version 2.0 was only released a few months ago. Is it enough to consider that the API is now stable enough (was enough tested)? Giampaolo (author of the module) doesn't look to be confident in its own API ;-) He wrote "I still want to work on a couple of new features first (...) and be 100% sure that I'm happy with the API as it is right now". Maybe I'm wrong and the psutil is stable and can be "frozen" in the standard library. Maybe it's possible to extract a subpart of the psutil to keep the most stable part? I'm not strongly opposed to the integration of the psutil module into the stdlib. Is it hard to install the psutil module today? I see wheel packages and .exe installers for Windows. They are Linux packages (Debian, Ubuntu, Fedora, ArchLinux, etc.) Victor From abarnert at yahoo.com Tue Oct 14 00:49:30 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 13 Oct 2014 15:49:30 -0700 Subject: [Python-ideas] including psutil in the standard library? In-Reply-To: References: <542319B1.2090606@ferrara.linux.it> Message-ID: <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com> On Oct 13, 2014, at 14:40, Victor Stinner wrote: > 2014-09-24 21:21 GMT+02:00 Stefano Borini : >> I am wondering if it would be possible to include psutil >> (https://pypi.python.org/pypi/psutil ) in the standard library, and if not, >> what would be needed. > > The problem is that Python only releases a new major version every 18 > months (or something like that), so adding a single new function will > take so much time. > > Supporting a new platform, architecture, major version of an OS, etc. > may also need to wait a new Python major release. > > Operating systems are evolving quite fast. For example, there is the > new architecture AArch64, containers are more and more popular, > systemd can also be used to provide features similar to psutil and it > supports containers (running systemd, it even supports recursive > containers....). Network becomes more virtual with Software Defined > Networks (SDN, NFV, etc.). The Linux kernel is also extended at each > release. Example of "recent" addition: /proc/pid/fdinfo/ directory. > Does psutil provide NUMA information? NUMA also becomes more > information nowadays for performances. > > The psutil also still evolves. For example, I see that the version 2.1 > released at April, 2014 adds "netstat-like functionalities". The > version 2.0 was only released one month before (March, 2014). > > The API of psutil changed a lot between 1.x and 2.0. The version 2.0 > was only released a few months ago. Is it enough to consider that the > API is now stable enough (was enough tested)? Giampaolo (author of the > module) doesn't look to be confident in its own API ;-) He wrote "I > still want to work on a couple of new features first (...) and be 100% > sure that I'm happy with the API as it is right now". > > Maybe I'm wrong and the psutil is stable and can be "frozen" in the > standard library. Maybe it's possible to extract a subpart of the > psutil to keep the most stable part? > > I'm not strongly opposed to the integration of the psutil module into > the stdlib. > > Is it hard to install the psutil module today? I see wheel packages > and .exe installers for Windows. They are Linux packages (Debian, > Ubuntu, Fedora, ArchLinux, etc.) This might be a good idea. It seems to me that there are two common uses for psutil. Some people want just a little bit more than Python makes available, and they want it in a cross-platform (at least gnu vs. bsd vs. sysv, if not unix vs. windows) way. That doesn't change that much over time, and adding that much from psutil to the stdlib might make sense. Other people want all kinds of detailed process information, in a platform-specific way, without having to grub through low-level APIs. That changes frequently, and for those people, pip install psutil should probably remain the right answer. (Of course there are also people who use psutil to do the exact same things subprocess and os can already do better; for those people, the answer is to stop doing that...) I guess the real question is, what are the use cases for the "stable core"? Once we know that, we can identify whether it's stable enough and simple enough to extract. From adam.jorgensen.za at gmail.com Tue Oct 14 07:30:40 2014 From: adam.jorgensen.za at gmail.com (Adam Jorgensen) Date: Tue, 14 Oct 2014 07:30:40 +0200 Subject: [Python-ideas] Query about proposed Multi-line Anonymous Function syntaxes In-Reply-To: References: Message-ID: On 10 October 2014 20:42, Guido van Rossum wrote: > On Fri, Oct 10, 2014 at 8:06 AM, Adam Jorgensen < > adam.jorgensen.za at gmail.com> wrote: > >> >> So, as I understand it, the main reason we don't have multi-line >> anonymous functions is because there has been no consensus about >> the syntax. >> > > Well, it's worse -- it's hard to propose a syntax that actually works > without huge changes to the way Python's parser treats indentation: inside > parentheses, indentation is currently *not* significant. > Aaaaaah, this was exactly the area I figured would be an issue when I thought about the problem for more than 10 seconds :-) > > >> I was just wondering if someone has a document detailing the various >> syntaxes that have been proposed in the the past. >> > > No, but there have been many discussions. If you have a taste for mailing > list archaeology it would be nice of you could dig up the various threads > and summarize them for posterity, so at least we might make progress rather > than going around in circles. > > As for myself, I have encountered a number of languages that do support > multi-line anonymous functions (Java, JavaScript, CoffeeScript) and I have > to say that for some reason they do not lead to very readable code. > Especially in the latter two, the prevalent use for anonymous functions is > for asynchronous I/O callbacks. I find Python 3.4's "yield from" as used by > the asyncio library leads to much more readable code, without he need for > callback functions at all. > I agree that anonymous functions do have the potential to produce illegible code, although I think this might be more a product of those languages general ability to enable said illegibility :-) In the context of python the only strong use-case I personally have for anonymous functions is a (extant) scenario where I'm writing nested decorators that don't use the decorator package to preserve signatures (There are some instances where signature preservation is actually undesirable :-). In this scenario the nested defs *might* be cleaner written in using anonymous functions, although there's no guarantee and it's not exactly a common scenario anyway... Maybe if I have some time I'll do the archaeology and cobble together a summary of the various proposals... > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Oct 14 09:56:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Oct 2014 17:56:18 +1000 Subject: [Python-ideas] including psutil in the standard library? In-Reply-To: <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com> References: <542319B1.2090606@ferrara.linux.it> <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com> Message-ID: On 14 October 2014 08:49, Andrew Barnert wrote: > On Oct 13, 2014, at 14:40, Victor Stinner wrote: >> Maybe I'm wrong and the psutil is stable and can be "frozen" in the >> standard library. Maybe it's possible to extract a subpart of the >> psutil to keep the most stable part? >> >> I'm not strongly opposed to the integration of the psutil module into >> the stdlib. >> >> Is it hard to install the psutil module today? I see wheel packages >> and .exe installers for Windows. They are Linux packages (Debian, >> Ubuntu, Fedora, ArchLinux, etc.) > > This might be a good idea. > > It seems to me that there are two common uses for psutil. > > Some people want just a little bit more than Python makes available, and they want it in a cross-platform (at least gnu vs. bsd vs. sysv, if not unix vs. windows) way. That doesn't change that much over time, and adding that much from psutil to the stdlib might make sense. > > Other people want all kinds of detailed process information, in a platform-specific way, without having to grub through low-level APIs. That changes frequently, and for those people, pip install psutil should probably remain the right answer. > > (Of course there are also people who use psutil to do the exact same things subprocess and os can already do better; for those people, the answer is to stop doing that...) > > I guess the real question is, what are the use cases for the "stable core"? Once we know that, we can identify whether it's stable enough and simple enough to extract. Right, "use case driven design" is the way to go. In particular, some of the key drivers for standard libraries additions in recent times have been: 1. Adding a module because we want to use it in the standard library (enum) 2. Adding a module because it smooths the learning curve for folks learning programming and other computing concepts (ipaddress, statistics) 3. Adding a module as part of a broader Python ecosystem interoperability effort (asyncio) 4. Adding a module because it is closely coupled to internal interpreter details (tracemalloc) A "minimal psutil" strikes me as something that could be useful for learning about computers by way of learning to program in Python - listing processes, users, network connections, memory usage, etc. Having that kind of basic capability in the standard library could be quite beneficial to a broad audience. The more sophisticated capabilities in psutil around resource limiting, and providing alternative interfaces to standard library functionality would likely make less sense in a standard library module (especially as these are the ones more likely to evolve over time). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Tue Oct 14 15:38:21 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 15 Oct 2014 00:38:21 +1100 Subject: [Python-ideas] Query about proposed Multi-line Anonymous Function syntaxes In-Reply-To: References: Message-ID: On Tue, Oct 14, 2014 at 4:30 PM, Adam Jorgensen wrote: > Maybe if I have some time I'll do the archaeology and cobble together a > summary of the various proposals... Possibly the best format for such a summary would be a PEP, which can then be rejected. Then anyone else who wants to suggest multi-line lambdas can go to the PEP and try to come up with answers to all the objections. ChrisA From tshepang at gmail.com Tue Oct 14 22:32:23 2014 From: tshepang at gmail.com (Tshepang Lekhonkhobe) Date: Tue, 14 Oct 2014 22:32:23 +0200 Subject: [Python-ideas] including psutil in the standard library? In-Reply-To: <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com> References: <542319B1.2090606@ferrara.linux.it> <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com> Message-ID: On Tue, Oct 14, 2014 at 12:49 AM, Andrew Barnert wrote: > (Of course there are also people who use psutil to do the exact same things subprocess and os can already do better; for those people, the answer is to stop doing that...) I was of the thought that psutil is better in everything it does compared to anything comparable in the stdlib. The only example I have in mind is its (far more featureful) Popen. What did I miss? From jim.baker at python.org Tue Oct 14 23:15:12 2014 From: jim.baker at python.org (Jim Baker) Date: Tue, 14 Oct 2014 15:15:12 -0600 Subject: [Python-ideas] including psutil in the standard library? In-Reply-To: References: <542319B1.2090606@ferrara.linux.it> <59DEA399-5915-4562-8042-1372CF9D4353@yahoo.com> Message-ID: What are the implications of placing psutil in the stdlib for cross implementation support, eg Jython/IronPython? FWIW, we are planning to start work on Jython 3.5 at the PyCon sprints in Montreal. On Tue, Oct 14, 2014 at 2:32 PM, Tshepang Lekhonkhobe wrote: > On Tue, Oct 14, 2014 at 12:49 AM, Andrew Barnert > wrote: > > (Of course there are also people who use psutil to do the exact same > things subprocess and os can already do better; for those people, the > answer is to stop doing that...) > > I was of the thought that psutil is better in everything it does > compared to anything comparable in the stdlib. The only example I have > in mind is its (far more featureful) Popen. What did I miss? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- - Jim jim.baker@{colorado.edu|python.org|rackspace.com|zyasoft.com} twitter.com/jimbaker github.com/jimbaker bitbucket.com/jimbaker -------------- next part -------------- An HTML attachment was scrubbed... URL: From julien at palard.fr Mon Oct 20 21:42:47 2014 From: julien at palard.fr (Julien Palard) Date: Mon, 20 Oct 2014 21:42:47 +0200 Subject: [Python-ideas] Need feedback about exposing an operator overloading in a module Message-ID: <544565B7.8040100@palard.fr> Hi everybody, Long long time ago, I wrote thehttps://pypi.python.org/pypi/pipe Python module, TL;DR: a 6 lines of code decorator enabling infix syntax. Long time ago someone came to speak to me on IRC advising me to speak about it here, but I was still blocked in my mind about the exposed/leaked `|` operator overload. For me, a module should never expose an operator overload, because it may lead to unpredicted/unnatural/surprising behavior. But I may be wrong. Recently I was thinking about my pipe module and got an idea: What about exposing a single function instead of a whole bunch of operator-overloaded functions ? Like Pipe (because I can't spot a real name for the moment but need one for the example) and translate from: fib() | pp.where(lambda x: x % 2 == 0) | pp.take_while(lambda x: x < 4000000) | pp.add to Pipe(fib()).where(lambda x: x % 2 == 0) .take_while(lambda x: x < 4000000) .add().data It's a bit more dense but still far more readable than the classic suffix notation: pp.add(pp.take_while(lambdax: x < 4000000, pp.where(lambda x: x % 2 == 0, fib()))) (I explicitly prefixed by `pp.` to explicityly show that using the `|` operator we have to import either the module or every used functions, but using Pipe we do not have to) For clarity, where are the two implementations (Doctests permiting you to compare usability of both): Actual Pipe module: https://github.com/JulienPalard/Pipe/blob/master/pipe.py Pipe module with a function instead of operator overloading: https://github.com/JulienPalard/Pipe/blob/less_intrusive/pipe.py So a few questions: 1) Do you think, like me, that exposing a `|` overload is bad ? 2) In the less intrusive version, do you think, like me, that .register, Pipe(...), and .data are too much ? 3) Do you want to see something permitting infix notation in Python standard library ? Bests, -- Julien Palard From ethan at stoneleaf.us Mon Oct 20 22:10:22 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 20 Oct 2014 13:10:22 -0700 Subject: [Python-ideas] Need feedback about exposing an operator overloading in a module In-Reply-To: <544565B7.8040100@palard.fr> References: <544565B7.8040100@palard.fr> Message-ID: <54456C2E.10604@stoneleaf.us> On 10/20/2014 12:42 PM, Julien Palard wrote: > > fib() | pp.where(lambda x: x % 2 == 0) > | pp.take_while(lambda x: x < 4000000) > | pp.add I find this very readable. > Pipe(fib()).where(lambda x: x % 2 == 0) > .take_while(lambda x: x < 4000000) > .add().data I think this is also readable, although you need to surround the whole thing with parens or end the first two lines with '\'. > It's a bit more dense but still far more readable than the > classic suffix notation: > > pp.add(pp.take_while(lambdax: x < 4000000, > pp.where(lambda x: x % 2 == 0, fib()))) Complete agreement here. :) > So a few questions: > > 1) Do you think, like me, that exposing a `|` overload is bad ? I do not see that as a problem, but without using it I can't know for sure. > 2) In the less intrusive version, do you think, like me, that .register, Pipe(...), and .data are too much ? I don't see a `.register` in the example, but I don't think Pipe and .data are inappropriate. > 3) Do you want to see something permitting infix notation in Python > standard library ? No opinion. Although method chaining isn't strongly supported in the stdlib. -- ~Ethan~ From greg.ewing at canterbury.ac.nz Mon Oct 20 23:25:49 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 21 Oct 2014 10:25:49 +1300 Subject: [Python-ideas] Need feedback about exposing an operator overloading in a module In-Reply-To: <544565B7.8040100@palard.fr> References: <544565B7.8040100@palard.fr> Message-ID: <54457DDD.4080800@canterbury.ac.nz> Julien Palard wrote: > For me, a module should never > expose an operator overload, because it may lead to > unpredicted/unnatural/surprising behavior. Modules don't expose operator overloads, types do. The idea that different types respond to operators in different ways is fundamental to Python, and in general I don't see anything wrong with making use of that. I do agree it's something that should be used sparingly, and most of the time a named method is much better. But I wouldn't say it should *never* be done. -- Greg From robertc at robertcollins.net Tue Oct 21 03:18:54 2014 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 21 Oct 2014 14:18:54 +1300 Subject: [Python-ideas] namespace for backported stdlib modules? Message-ID: I want to make the improvements in unittest (which transitively includes mock and traceback and perhaps inspect ...) available for folk that can't use 3.5 yet. We've currently got the unittest2 and mock as modules on pypi, and clearly we can add another e.g. traceback2 as a way to get the traceback changes out there. I'm wondering though if perhaps putting up a namespace package on pypi for things in the stdlib - e.g. 'stdlib' (or stdlib2 or something) would make sense. In particular that might permit relative imports to automatically pick up the other also backported modules without source changes. So we'd have e.g. stdlib.unittest stdlib.traceback stdlib.mock And then using that would be done through the normal try: from stdlib import unitest except ImportError: import unittest dance. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From tjreedy at udel.edu Tue Oct 21 03:29:59 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 20 Oct 2014 21:29:59 -0400 Subject: [Python-ideas] Need feedback about exposing an operator overloading in a module In-Reply-To: <544565B7.8040100@palard.fr> References: <544565B7.8040100@palard.fr> Message-ID: On 10/20/2014 3:42 PM, Julien Palard wrote: > Hi everybody, > > Long long time ago, I wrote thehttps://pypi.python.org/pypi/pipe > Python module, TL;DR: a 6 lines of code decorator enabling infix > syntax. > > Long time ago someone came to speak to me on IRC advising me to speak > about it here, but I was still blocked in my mind about the > exposed/leaked `|` operator overload. For me, a module should never > expose an operator overload, because it may lead to > unpredicted/unnatural/surprising behavior. But I may be wrong. > > Recently I was thinking about my pipe module and got an idea: What > about exposing a single function instead of a whole bunch of > operator-overloaded functions ? Like Pipe (because I can't spot a real > name for the moment but need one for the example) and translate from: > > fib() | pp.where(lambda x: x % 2 == 0) > | pp.take_while(lambda x: x < 4000000) > | pp.add > > to > > Pipe(fib()).where(lambda x: x % 2 == 0) > .take_while(lambda x: x < 4000000) > .add().data > > It's a bit more dense but still far more readable than the > classic suffix notation: > > pp.add(pp.take_while(lambdax: x < 4000000, > pp.where(lambda x: x % 2 == 0, fib()))) I think the best is current idiomatic python: import itertools as it def fibg(): a,b = 0, 1 while True: yield a a,b = b, a+b evenfib = filter(lambda x: not x % 2, fibg()) print(sum(it.takewhile(lambda x: x < 4000000, evenfib))) >>> 4613732 Two lines instead of three, with meaning clear. I am a fan of abstraction by naming and like breaking paragraphs up into sentences, and suites into statements. > 1) Do you think, like me, that exposing a `|` overload is bad ? No. > 3) Do you want to see something permitting infix notation in Python > standard library ? Not especially. More persuasive use case needed. -- Terry Jan Reedy From tjreedy at udel.edu Tue Oct 21 03:40:07 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 20 Oct 2014 21:40:07 -0400 Subject: [Python-ideas] namespace for backported stdlib modules? In-Reply-To: References: Message-ID: On 10/20/2014 9:18 PM, Robert Collins wrote: > I want to make the improvements in unittest (which transitively > includes mock and traceback and perhaps inspect ...) available for > folk that can't use 3.5 yet. > > We've currently got the unittest2 and mock as modules on pypi, and > clearly we can add another e.g. traceback2 as a way to get the > traceback changes out there. > > I'm wondering though if perhaps putting up a namespace package on pypi > for things in the stdlib - e.g. 'stdlib' (or stdlib2 or something) I would go with stdlib2 as less confusing, and then omit '2' from everything inside -- unless you intend the package to also be used on earlier versions of 3.x. -- Terry Jan Reedy From graffatcolmingov at gmail.com Tue Oct 21 03:41:27 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Mon, 20 Oct 2014 20:41:27 -0500 Subject: [Python-ideas] namespace for backported stdlib modules? In-Reply-To: References: Message-ID: On Oct 20, 2014 8:25 PM, "Robert Collins" wrote: > > I want to make the improvements in unittest (which transitively > includes mock and traceback and perhaps inspect ...) available for > folk that can't use 3.5 yet. > > We've currently got the unittest2 and mock as modules on pypi, and > clearly we can add another e.g. traceback2 as a way to get the > traceback changes out there. > > I'm wondering though if perhaps putting up a namespace package on pypi > for things in the stdlib - e.g. 'stdlib' (or stdlib2 or something) > would make sense. In particular that might permit relative imports to > automatically pick up the other also backported modules without source > changes. > > So we'd have e.g. > stdlib.unittest > stdlib.traceback > stdlib.mock > > And then using that would be done through the normal > > try: > from stdlib import unitest > except ImportError: > import unittest > > dance. > > -Rob There are several packages already named backports.{{module}}. I think they all use the same namespace too. They're also all maintained by people not working on the stdlib. For example, backports.ssl is an attempt to backport python 3's ssl features to python 2 using pyOpenSSL -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Tue Oct 21 03:49:03 2014 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 21 Oct 2014 14:49:03 +1300 Subject: [Python-ideas] namespace for backported stdlib modules? In-Reply-To: References: Message-ID: On 21 October 2014 14:40, Terry Reedy wrote: > On 10/20/2014 9:18 PM, Robert Collins wrote: >> >> I want to make the improvements in unittest (which transitively >> includes mock and traceback and perhaps inspect ...) available for >> folk that can't use 3.5 yet. >> >> We've currently got the unittest2 and mock as modules on pypi, and >> clearly we can add another e.g. traceback2 as a way to get the >> traceback changes out there. >> >> I'm wondering though if perhaps putting up a namespace package on pypi >> for things in the stdlib - e.g. 'stdlib' (or stdlib2 or something) > > > I would go with stdlib2 as less confusing, and then omit '2' from everything > inside -- unless you intend the package to also be used on earlier versions > of 3.x. I intend it to be used on still-supported earlier versions of 3.x. What do you think of the idea itself? E.g. is the cost of transition from unittest2 -> $NAMESPACE.unittest worth doing? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From solipsis at pitrou.net Tue Oct 21 08:51:13 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 Oct 2014 08:51:13 +0200 Subject: [Python-ideas] namespace for backported stdlib modules? References: Message-ID: <20141021085113.72afad42@fsol> On Tue, 21 Oct 2014 14:49:03 +1300 Robert Collins wrote: > On 21 October 2014 14:40, Terry Reedy wrote: > > On 10/20/2014 9:18 PM, Robert Collins wrote: > >> > >> I want to make the improvements in unittest (which transitively > >> includes mock and traceback and perhaps inspect ...) available for > >> folk that can't use 3.5 yet. > >> > >> We've currently got the unittest2 and mock as modules on pypi, and > >> clearly we can add another e.g. traceback2 as a way to get the > >> traceback changes out there. > >> > >> I'm wondering though if perhaps putting up a namespace package on pypi > >> for things in the stdlib - e.g. 'stdlib' (or stdlib2 or something) > > > > > > I would go with stdlib2 as less confusing, and then omit '2' from everything > > inside -- unless you intend the package to also be used on earlier versions > > of 3.x. > > I intend it to be used on still-supported earlier versions of 3.x. > > What do you think of the idea itself? E.g. is the cost of transition > from unittest2 -> $NAMESPACE.unittest worth doing? That doesn't sound terribly useful. Regards Antoine. From victor.stinner at gmail.com Tue Oct 21 10:09:02 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 21 Oct 2014 10:09:02 +0200 Subject: [Python-ideas] namespace for backported stdlib modules? In-Reply-To: References: Message-ID: Hi, 2014-10-21 3:18 GMT+02:00 Robert Collins : > We've currently got the unittest2 and mock as modules on pypi, and > clearly we can add another e.g. traceback2 as a way to get the > traceback changes out there. Why not contributing to the unittest2 module instead of creating yet another project? Would it be possible to work on unittest2 and mock to make them "compatible"? > I'm wondering though if perhaps putting up a namespace package on pypi > for things in the stdlib - e.g. 'stdlib' (or stdlib2 or something) > would make sense. In particular that might permit relative imports to > automatically pick up the other also backported modules without source > changes. In my experience, namespaces with Python < 3.3 only means pain. I only got issues with the "oslo" namespace created with a hack in a .pth file for oslo.config, oslo.rootwrap, etc. FYI there a list on the Python wiki: https://wiki.python.org/moin/StandardLibraryBackports Victor From g.brandl at gmx.net Tue Oct 21 11:34:03 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 21 Oct 2014 11:34:03 +0200 Subject: [Python-ideas] namespace for backported stdlib modules? In-Reply-To: References: Message-ID: On 10/21/2014 03:18 AM, Robert Collins wrote: > I want to make the improvements in unittest (which transitively > includes mock and traceback and perhaps inspect ...) available for > folk that can't use 3.5 yet. > > We've currently got the unittest2 and mock as modules on pypi, and > clearly we can add another e.g. traceback2 as a way to get the > traceback changes out there. > > I'm wondering though if perhaps putting up a namespace package on pypi > for things in the stdlib - e.g. 'stdlib' (or stdlib2 or something) > would make sense. In particular that might permit relative imports to > automatically pick up the other also backported modules without source > changes. > > So we'd have e.g. > stdlib.unittest > stdlib.traceback > stdlib.mock > > And then using that would be done through the normal > > try: > from stdlib import unitest > except ImportError: > import unittest Seeing this as a newcomer, I'd be wondering which one is the one that's actually coming from the standard library. Please don't commandeer "stdlib" for something that's *not* the stdlib. Georg From robertc at robertcollins.net Tue Oct 21 11:48:18 2014 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 21 Oct 2014 22:48:18 +1300 Subject: [Python-ideas] namespace for backported stdlib modules? In-Reply-To: References: Message-ID: On 21 October 2014 21:09, Victor Stinner wrote: > Hi, > > 2014-10-21 3:18 GMT+02:00 Robert Collins : >> We've currently got the unittest2 and mock as modules on pypi, and >> clearly we can add another e.g. traceback2 as a way to get the >> traceback changes out there. > > Why not contributing to the unittest2 module instead of creating yet > another project? Huh, thats exactly what I'm doing. Remember: unittest2 *is* unittest, just broken out and shipped on PyPI for older Python releases. I'm talking purely about the mechanics of doing that distribution and examining ways I might make it cheaper: unittest2 hasn't been updated in ~2 years and if we're going to keep it up to date it needs to be low overhead. > Would it be possible to work on unittest2 and mock to make them "compatible"? Not sure what you mean here. >> I'm wondering though if perhaps putting up a namespace package on pypi >> for things in the stdlib - e.g. 'stdlib' (or stdlib2 or something) >> would make sense. In particular that might permit relative imports to >> automatically pick up the other also backported modules without source >> changes. > > In my experience, namespaces with Python < 3.3 only means pain. I only > got issues with the "oslo" namespace created with a hack in a .pth > file for oslo.config, oslo.rootwrap, etc. Yeah, 3.3 makes it lovely, and I put some patches into importlib2 (same as unittest2 - a backport of importlib for older pythons) to make that better, but its got a bit of a thorny issue to resolve w.r.t. relative imports before it can be used as the default importer. > FYI there a list on the Python wiki: > https://wiki.python.org/moin/StandardLibraryBackports Thanks. Turns out thats incomplete, I'll figure out a login for that later and see about adding importlib2. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From barry at python.org Tue Oct 21 17:39:13 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 21 Oct 2014 11:39:13 -0400 Subject: [Python-ideas] namespace for backported stdlib modules? References: Message-ID: <20141021113913.55fdec6a@anarchist> On Oct 21, 2014, at 11:34 AM, Georg Brandl wrote: >> try: >> from stdlib import unitest >> except ImportError: >> import unittest > >Seeing this as a newcomer, I'd be wondering which one is the one that's >actually coming from the standard library. > >Please don't commandeer "stdlib" for something that's *not* the stdlib. Agreed. But 'backports' would be pretty obvious I think. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From ethan at stoneleaf.us Tue Oct 21 17:43:12 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 21 Oct 2014 08:43:12 -0700 Subject: [Python-ideas] namespace for backported stdlib modules? In-Reply-To: References: Message-ID: <54467F10.3070906@stoneleaf.us> On 10/21/2014 02:34 AM, Georg Brandl wrote: > On 10/21/2014 03:18 AM, Robert Collins wrote: >> >> try: >> from stdlib import unitest >> except ImportError: >> import unittest > > Seeing this as a newcomer, I'd be wondering which one is the one that's > actually coming from the standard library. > > Please don't commandeer "stdlib" for something that's *not* the stdlib. Indeed. `backports` is a much better name for things that have been, um, backported. From robertc at robertcollins.net Tue Oct 21 21:48:00 2014 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 22 Oct 2014 08:48:00 +1300 Subject: [Python-ideas] namespace for backported stdlib modules? In-Reply-To: <54467F10.3070906@stoneleaf.us> References: <54467F10.3070906@stoneleaf.us> Message-ID: https://pypi.python.org/pypi/backports/1.0 Question now is whether its active :) On 22 October 2014 04:43, Ethan Furman wrote: > On 10/21/2014 02:34 AM, Georg Brandl wrote: >> >> On 10/21/2014 03:18 AM, Robert Collins wrote: >>> >>> >>> try: >>> from stdlib import unitest >>> except ImportError: >>> import unittest >> >> >> Seeing this as a newcomer, I'd be wondering which one is the one that's >> actually coming from the standard library. >> >> Please don't commandeer "stdlib" for something that's *not* the stdlib. > > > Indeed. `backports` is a much better name for things that have been, um, > backported. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Robert Collins Distinguished Technologist HP Converged Cloud From skreft at gmail.com Tue Oct 21 23:49:58 2014 From: skreft at gmail.com (Sebastian Kreft) Date: Tue, 21 Oct 2014 23:49:58 +0200 Subject: [Python-ideas] namespace for backported stdlib modules? In-Reply-To: References: <54467F10.3070906@stoneleaf.us> Message-ID: I think the decentralized namespace idea is doomed, for several reasons: - The namespace is not private and anyone can create a backport there without adhering to any standards. This has the problem of false security for potential users and code that is not thoroughly tested. - Some projects flatten the name of the backport, specially when they just want to backport one function. Which result in awfully long names. - You need to import all the namespace boilerplate in your backport project. - Licensing: Not all people put their backports with the PSF license. - it is not clear from which version/revision things were backported. - at the end, one still needs to have this conditional imports that force you to have this compatibility layer, so naming is not that important. What I would do is instead of having a namespace is to actually have just one project (no strong opinions about the name) with community curated backports. The project should/could: - enforce the PSF license - define different namespaces for the different backports, like backport27, backport33, backport34 - keep track of revision used in the latest backport and notify to maintainers of changes - be tested in all platforms specified - have guidelines for backporters - provide diffs of what was changed and limitations of the backport - generate different packages for different python versions or maybe even allow people to select their own bundle. - it could even provide some import magic like: import backports33 backports33.install('shutil.wich') import shutil shutil.wich(...) Note: I am maintaining two backports: https://pypi.python.org/pypi/backport_ipaddress and https://pypi.python.org/pypi/backport_collections. On Tue, Oct 21, 2014 at 9:48 PM, Robert Collins wrote: > https://pypi.python.org/pypi/backports/1.0 > > Question now is whether its active :) > > On 22 October 2014 04:43, Ethan Furman wrote: > > On 10/21/2014 02:34 AM, Georg Brandl wrote: > >> > >> On 10/21/2014 03:18 AM, Robert Collins wrote: > >>> > >>> > >>> try: > >>> from stdlib import unitest > >>> except ImportError: > >>> import unittest > >> > >> > >> Seeing this as a newcomer, I'd be wondering which one is the one that's > >> actually coming from the standard library. > >> > >> Please don't commandeer "stdlib" for something that's *not* the stdlib. > > > > > > Indeed. `backports` is a much better name for things that have been, um, > > backported. > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Sebastian Kreft -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Wed Oct 22 00:43:56 2014 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 22 Oct 2014 11:43:56 +1300 Subject: [Python-ideas] namespace for backported stdlib modules? In-Reply-To: References: Message-ID: Sorry for not seeing this one - it went to my spam folder - I've hopefully sorted that out for future messages. -Rob On 21 October 2014 14:41, Ian Cordasco wrote: > > On Oct 20, 2014 8:25 PM, "Robert Collins" wrote: >> >> I want to make the improvements in unittest (which transitively >> includes mock and traceback and perhaps inspect ...) available for >> folk that can't use 3.5 yet. >> >> We've currently got the unittest2 and mock as modules on pypi, and >> clearly we can add another e.g. traceback2 as a way to get the >> traceback changes out there. >> >> I'm wondering though if perhaps putting up a namespace package on pypi >> for things in the stdlib - e.g. 'stdlib' (or stdlib2 or something) >> would make sense. In particular that might permit relative imports to >> automatically pick up the other also backported modules without source >> changes. >> >> So we'd have e.g. >> stdlib.unittest >> stdlib.traceback >> stdlib.mock >> >> And then using that would be done through the normal >> >> try: >> from stdlib import unitest >> except ImportError: >> import unittest >> >> dance. >> >> -Rob > > There are several packages already named backports.{{module}}. I think they > all use the same namespace too. They're also all maintained by people not > working on the stdlib. For example, backports.ssl is an attempt to backport > python 3's ssl features to python 2 using pyOpenSSL -- Robert Collins Distinguished Technologist HP Converged Cloud From russell at keith-magee.com Sat Oct 25 04:20:05 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Sat, 25 Oct 2014 10:20:05 +0800 Subject: [Python-ideas] Adding iOS/Android support to Python Message-ID: Hi all, tl;dr - I'd like to add iOS and Android to the list of platforms that Python supports out-of-the-box, and I'm looking for guidance on the way forward (and assistance, if anyone is keen). Longer version: For the last 9 months or so, I've been working on tools to enable developers to write native mobile apps using Python. My motivation is to get Toga (http://pybee.org/toga) to the point where it's a viable way to write mobile apps. However, I'm not alone in the "Python on mobile" goal - Kivy has been around for a few years, with similar goals (although they've got some different ideas and priorities). To date, I've been using the groundwork established by Kivy, which takes the Python 2.7.1 (for iOS) and Python 2.7.2 (for Android) source code, applies a bunch of patches, and wraps those in a set of build scripts to yield Python libraries that can be used on iOS and Android. I'd like to start supporting more recent versions of Python - most importantly, Python 3. While I could certainly continue maintaining patches externally (and I imagine I'll have to if I want to maintain Python 2.7 support), I'd like to see that effort integrated into Python's repository. In particular, there are four areas where I can see changes required: 1) At least 2 new sys.platform definitions - "ios" and "android" (a third "iossimulator" platform might also be needed - the iOS simulator has enough differences that it can be helpful to be able to identify the platform) 2) Modifications to the build system to support cross-platform builds for mobile - AFAICT, cross platform builds currently only work on Linux and require readelf; iOS libraries can only be built on Mac, and building Android libraries don't require readelf (AFAICT). 3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. 4) Support for building fat libraries for iOS. Xcode uses a tool called lipo to mash multiple platform libraries into a single library file. I don't imagine 1 & 2 are especially controversial. However, I would anticipate that 3 might raise some concerns, as the general Python "batteries included" philosophy would be modified to "unless you're on one of these platforms". Any guidance on whether this approach would be palatable? Is there any precedent for "module not supported on platform X" that I'm not aware of? Any platforms where Python is *only* available as a library? I have no idea how to tackle 4. To create a complete iOS build, you have to do at least 4 complete Python builds - a native system build (so you can run Python scripts that are part of the build), an i386 build (for the simulator), an armv7 build, and an arm64 build - and then the build products of the last three need to be combined into a single framework directory. Given three independent platform-specific build directories, it's relatively easy to construct a "fat" framework, but it isn't clear to me how that would fit into Python's build system. I can see that the build system has support for "universal" builds, which looks like an artefact of the OS X PPC->i386 conversion, but the build approach appears to be different to that required for iOS. Is there any documentation of the build process itself? Or is there anyone who can point me in the right direction? I'm also not clear how to tackle the testing and CI problems. Since you can't run iOS code on a non-iOS machine, it isn't entirely clear to me what an acceptance test would look like for a mobile build. So - am I completely wasting my time? Are patches for mobile platforms likely to be accepted into Python's repository? Or should I stick with the "external patches" approach? Any pointers, hints, guidance or assistance would be gratefully accepted. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Oct 25 05:06:26 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Oct 2014 20:06:26 -0700 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: This idea seems to be in the air. Just yesterday someone posted a set of patches and suggestions to python-dev about making cross-compilation for Android easier. I expect most people in the core Python team might not be eager to jump on this bandwagon because they have no immediate need, but these mobile platforms are only getting faster more popular, and I think it would be a good idea to be prepared for when suddenly everybody wants to develop mobile apps in Python. On Fri, Oct 24, 2014 at 7:20 PM, Russell Keith-Magee < russell at keith-magee.com> wrote: > Hi all, > > tl;dr - I'd like to add iOS and Android to the list of platforms that > Python supports out-of-the-box, and I'm looking for guidance on the way > forward (and assistance, if anyone is keen). > > Longer version: For the last 9 months or so, I've been working on tools to > enable developers to write native mobile apps using Python. My motivation > is to get Toga (http://pybee.org/toga) to the point where it's a viable > way to write mobile apps. However, I'm not alone in the "Python on mobile" > goal - Kivy has been around for a few years, with similar goals (although > they've got some different ideas and priorities). > > To date, I've been using the groundwork established by Kivy, which takes > the Python 2.7.1 (for iOS) and Python 2.7.2 (for Android) source code, > applies a bunch of patches, and wraps those in a set of build scripts to > yield Python libraries that can be used on iOS and Android. > > I'd like to start supporting more recent versions of Python - most > importantly, Python 3. While I could certainly continue maintaining patches > externally (and I imagine I'll have to if I want to maintain Python 2.7 > support), I'd like to see that effort integrated into Python's repository. > > In particular, there are four areas where I can see changes required: > > 1) At least 2 new sys.platform definitions - "ios" and "android" (a third > "iossimulator" platform might also be needed - the iOS simulator has enough > differences that it can be helpful to be able to identify the platform) > > 2) Modifications to the build system to support cross-platform builds for > mobile - AFAICT, cross platform builds currently only work on Linux and > require readelf; iOS libraries can only be built on Mac, and building > Android libraries don't require readelf (AFAICT). > > 3) Disabling certain modules on mobile platforms. Supporting modules like > linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile > platforms doesn't make much sense; modules likes bsddb and bz2 are > difficult to support due to library dependencies; and the need for modules > like multiprocessing is arguable (and difficult to support on mobile). Even > providing a Python executable/shell is arguable for these platforms. > > 4) Support for building fat libraries for iOS. Xcode uses a tool called > lipo to mash multiple platform libraries into a single library file. > > I don't imagine 1 & 2 are especially controversial. > > However, I would anticipate that 3 might raise some concerns, as the > general Python "batteries included" philosophy would be modified to "unless > you're on one of these platforms". Any guidance on whether this approach > would be palatable? Is there any precedent for "module not supported on > platform X" that I'm not aware of? Any platforms where Python is *only* > available as a library? > > I have no idea how to tackle 4. To create a complete iOS build, you have > to do at least 4 complete Python builds - a native system build (so you can > run Python scripts that are part of the build), an i386 build (for the > simulator), an armv7 build, and an arm64 build - and then the build > products of the last three need to be combined into a single framework > directory. > > Given three independent platform-specific build directories, it's > relatively easy to construct a "fat" framework, but it isn't clear to me > how that would fit into Python's build system. I can see that the build > system has support for "universal" builds, which looks like an artefact of > the OS X PPC->i386 conversion, but the build approach appears to be > different to that required for iOS. > > Is there any documentation of the build process itself? Or is there anyone > who can point me in the right direction? > > I'm also not clear how to tackle the testing and CI problems. Since you > can't run iOS code on a non-iOS machine, it isn't entirely clear to me what > an acceptance test would look like for a mobile build. > > So - am I completely wasting my time? Are patches for mobile platforms > likely to be accepted into Python's repository? Or should I stick with the > "external patches" approach? Any pointers, hints, guidance or assistance > would be gratefully accepted. > > Yours, > Russ Magee %-) > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Sat Oct 25 09:36:30 2014 From: toddrjen at gmail.com (Todd) Date: Sat, 25 Oct 2014 09:36:30 +0200 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" wrote: > 3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. I would definitely be extremely interested in a python shell in android. One thing I feel are lacking on android are good advanced mathematical tools and and python shell with appropriate libraries could make a very powerful open-source tool for that. There have been some attempts at that already. I would also differentiate android and iOs more. Android seems to be betting on multi-core performance while iOs seems to be betting on single-chore performance. So while multiprocessing may not make much sense on iOs, I think it may be more sense on Android, especially if they move from 4 to 8 cores. Similarly, ultimately android is Linux-based, so things like readline should work, and it seems others have been able to get it to work, and bz2 seems to work fine on third-party android file managers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From masklinn at masklinn.net Sat Oct 25 10:15:13 2014 From: masklinn at masklinn.net (Masklinn) Date: Sat, 25 Oct 2014 10:15:13 +0200 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On 2014-10-25, at 04:20 , Russell Keith-Magee wrote: > 3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. > > However, I would anticipate that 3 might raise some concerns, as the general Python "batteries included" philosophy would be modified to "unless you're on one of these platforms". Any guidance on whether this approach would be palatable? Is there any precedent for "module not supported on platform X" that I'm not aware of? * IIRC, Python will not setup e.g. bz2 if there's no libbz2 on the system, likewise ssl and libssl, so the underlying library is not available on a platform it wouldn't be entirely surprising for it to be missing in the corresponding Python * there are a number of modules which are only available on a restricted number of platforms, though for the most part they're either windows-specific[0] or unix-specific[1]. There are also limitations within modules e.g. os.path.relpath is documented as only available on Unix and Windows, samepath and sameopenfile used to be Unix-only and are now Unix+Windows * some modules also have very different contents or behaviors based on the platform e.g. ssl, signal or mmap So there seems to be a precedent for stdlib modules entirely missing on some platforms, or having restricted or platform-specific content. [0] https://docs.python.org/3/library/windows.html [1] https://docs.python.org/3/library/unix.html From ncoghlan at gmail.com Sat Oct 25 17:25:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Oct 2014 01:25:05 +1000 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On 25 October 2014 12:20, Russell Keith-Magee wrote: > I'd like to start supporting more recent versions of Python - most > importantly, Python 3. While I could certainly continue maintaining patches > externally (and I imagine I'll have to if I want to maintain Python 2.7 > support), I'd like to see that effort integrated into Python's repository. Maintaining support for new versions of existing platforms is already within scope for Python 2.7, so if you get this working for Python 3, it *might* be possible to make the case for 2.x. However, it would only be feasible to make that determination after we had a better idea as to the magnitude of the changes involved, and the risks introduced for other platforms. > In particular, there are four areas where I can see changes required: > 3) Disabling certain modules on mobile platforms. Supporting modules like > linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile > platforms doesn't make much sense; modules likes bsddb and bz2 are difficult > to support due to library dependencies; and the need for modules like > multiprocessing is arguable (and difficult to support on mobile). Even > providing a Python executable/shell is arguable for these platforms. > > However, I would anticipate that 3 might raise some concerns, as the general > Python "batteries included" philosophy would be modified to "unless you're > on one of these platforms". Any guidance on whether this approach would be > palatable? Is there any precedent for "module not supported on platform X" > that I'm not aware of? Yes, platform specific features are actually quite common (search the docs for "Availability:"). It's just the fact that folks writing cross-platform code tend to avoid those modules/features, so they're largely only known to folks writing lower level (or otherwise platform specific) code. > Any platforms where Python is *only* available as a > library? The CPython main.c is a fairly thin shim around Py_Main anyway, so I don't see it as a big deal whether the standard CLI is included as part of the mobile builds. > So - am I completely wasting my time? Are patches for mobile platforms > likely to be accepted into Python's repository? I think both iOS and Android are well and truly established enough now as platforms to be worth targeting directly. For the CI problem, it may be worth approaching Xamarin, as setting up the infrastructure for doing that ourselves looks like a formidable engineering challenge we don't currently have the relevant skillsets to provide. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat Oct 25 17:42:50 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 25 Oct 2014 17:42:50 +0200 Subject: [Python-ideas] Adding iOS/Android support to Python References: Message-ID: <20141025174250.4a7edb60@fsol> On Sat, 25 Oct 2014 10:20:05 +0800 Russell Keith-Magee wrote: > > In particular, there are four areas where I can see changes required: > > 1) At least 2 new sys.platform definitions - "ios" and "android" (a third > "iossimulator" platform might also be needed - the iOS simulator has enough > differences that it can be helpful to be able to identify the platform) "ios" and "android" sound ok to me. A specific platform for the simulator sounds wrong, though. If it simulates iOS, it should certainly simulate its platform identification. > 2) Modifications to the build system to support cross-platform builds for > mobile - AFAICT, cross platform builds currently only work on Linux and > require readelf; iOS libraries can only be built on Mac, and building > Android libraries don't require readelf (AFAICT). I don't know anything about cross-compiling. The best is that you try to whip up a patch and submit it on the tracker. Also, I suggest you tackle one system at a time. It will be easier to review and accept patches if they target only Android or only iOS, as opposed to both at the same time. > 3) Disabling certain modules on mobile platforms. Supporting modules like > linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile > platforms doesn't make much sense; modules likes bsddb and bz2 are > difficult to support due to library dependencies; and the need for modules > like multiprocessing is arguable (and difficult to support on mobile). Special support for this isn't necessary. If setup.py doesn't find the necessary external dependencies, it will skip building those modules. If OTOH those modules are built, you can remove them when creating your installer (I imagine you won't use "setup.py install" on those platforms). > I have no idea how to tackle 4. To create a complete iOS build, you have to > do at least 4 complete Python builds - a native system build (so you can > run Python scripts that are part of the build), an i386 build (for the > simulator), an armv7 build, and an arm64 build - and then the build > products of the last three need to be combined into a single framework > directory. No idea. Perhaps start by tackling the first 3 points. When you arrive at 4, open an issue and our resident Mac experts can help you :-) > I'm also not clear how to tackle the testing and CI problems. Since you > can't run iOS code on a non-iOS machine, it isn't entirely clear to me what > an acceptance test would look like for a mobile build. We don't have acceptance tests for all platforms. You'll have to check that Python still builds on a regular basis. Regards Antoine. From tjreedy at udel.edu Sat Oct 25 19:14:14 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 25 Oct 2014 13:14:14 -0400 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: <544BDA66.40804@udel.edu> On 10/24/2014 11:06 PM, Guido van Rossum wrote: > This idea seems to be in the air. Just yesterday someone posted a set of > patches and suggestions to python-dev about making cross-compilation for > Android easier. Matt Frank, "Cross Compiling Python for Android", or close to that. He experimented with 3.4.1. > I expect most people in the core Python team might not > be eager to jump on this bandwagon because they have no immediate need, > but these mobile platforms are only getting faster more popular, and I > think it would be a good idea to be prepared for when suddenly everybody > wants to develop mobile apps in Python. > > On Fri, Oct 24, 2014 at 7:20 PM, Russell Keith-Magee > > wrote: > tl;dr - I'd like to add iOS and Android to the list of platforms > that Python supports out-of-the-box, and I'm looking for guidance on > the way forward (and assistance, if anyone is keen). You (freakboy3742) and / or Matt Frank (WanderingLogic) should read the developer guide (https://docs.python.org/devguide), submit patches to the tracker for 3.5*, put each other as nosy (using the tracker ids above), review each other's patches, respond to other issue participants, especially core developers, and modify patches as needed to get them committed. Hopefully, one of you would have the goal of joining the developer team with commit privileges at least for android or ios development. * The first beta for 3.5 is scheduled for late May, 2015. At least a working prototype of any new feature for 3.5 should be in that release. > 3) Disabling certain modules on mobile platforms. Supporting > modules like linuxaudiodev, ossaudiodev, readline, curses, idle and > tkinter on mobile platforms doesn't make much sense; modules likes > bsddb and bz2 are difficult to support due to library dependencies; > and the need for modules like multiprocessing is arguable (and > difficult to support on mobile). To add to what other said, many tests in the test suite are limited to certain platforms already -- skip if windows, skip if not windows, etc. Part of your patching would consist of adding skips for tests. We would also need to expand the possible Availability notes in the doc, adding something like 'Availability - desktop systems' and 'Availablity - mobile (Android/IOS)' if mobile-only features were needed. -- Terry Jan Reedy From abarnert at yahoo.com Sat Oct 25 22:58:16 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 25 Oct 2014 13:58:16 -0700 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On Oct 24, 2014, at 19:20, Russell Keith-Magee wrote: > I'd like to start supporting more recent versions of Python - most importantly, Python 3. While I could certainly continue maintaining patches externally (and I imagine I'll have to if I want to maintain Python 2.7 support), I'd like to see that effort integrated into Python's repository. People have already been making Python 3 builds for iOS, wrapping them in mini-IDEs, and selling them on the App Store for quite some time. Have you talked to Xiaowen Huang or any of the others? As far as I know, most of them build Python as a static lib, and link in the C extensions from the stdlib, to get around the issues with shared-lib frameworks before iOS 8. And I'm not sure how they deal with third-party modules (at least one of them includes numpy, for example). You also may want to talk to whoever maintains the ports for jailbroken iOS, because that's just a standard-as-possible build, with the command line interpreter and pip and everything. > In particular, there are four areas where I can see changes required: > > 1) At least 2 new sys.platform definitions - "ios" and "android" (a third "iossimulator" platform might also be needed - the iOS simulator has enough differences that it can be helpful to be able to identify the platform) There are other fields in platform and sys that can be used to distinguish iOS from the simulator. And, since you need to check iOS simulator or real iOS vs. everything else at least as often as you need to distinguish them from each other, I'm not sure it's important whether it's treated as a separate platform or a different version of the same platform--as long as there's some documented and guaranteed way to distinguish them. > 3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. How is multiprocessing difficult to provide on mobile? iOS has had fork, posix_spawn, semaphores, etc. since the first release with an SDK, although you'd get rejected from the App Store for using them in the early days. Of course Apple would prefer people to use XPC via grand central, but that doesn't mean you need to make it impossible to use fork instead; at most, you just need to warn people to read the App Store guidelines. In the opposite direction, it might be worth looking at the "extra batteries" in Apple's OS X builds. At least PyObjC seems like something you'd want to get working on iOS. > I have no idea how to tackle 4. To create a complete iOS build, you have to do at least 4 complete Python builds - a native system build (so you can run Python scripts that are part of the build), an i386 build (for the simulator), an armv7 build, and an arm64 build - and then the build products of the last three need to be combined into a single framework directory. > > Given three independent platform-specific build directories, it's relatively easy to construct a "fat" framework, but it isn't clear to me how that would fit into Python's build system. I can see that the build system has support for "universal" builds, which looks like an artefact of the OS X PPC->i386 conversion, but the build approach appears to be different to that required for iOS. It's still perfectly possible to build Python universal on OS X. In fact, both the python.org builds and Apple's pre-installed builds are fat i386/x86_64. iOS does put some limitations on things, especially when building shared libs, but if you need to fall back to building each target separately and lipo'ing them together manually, I'm pretty sure there are a few third-party libs that already use that workaround. Didn't PyObjC used to build that way? From russell at keith-magee.com Sun Oct 26 01:32:08 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Sun, 26 Oct 2014 07:32:08 +0800 Subject: [Python-ideas] Fwd: Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On Sat, Oct 25, 2014 at 9:27 PM, Hernan Grecco wrote: > > El oct 24, 2014 11:22 PM, "Russell Keith-Magee" > escribi?: > > > > Hi all, > > > > tl;dr - I'd like to add iOS and Android to the list of platforms that > Python supports out-of-the-box, and I'm looking for guidance on the way > forward (and assistance, if anyone is keen). > > Have you looked a QPython? They provide Python 2 and 3 consoles in > android. I am not sure how large is the stdlib support, I have only tried > small scripts and it works quite nicely. > Yes, I've tried it, but I'm in a similar boat in that I didn't exhaustively exercise the full stdlib. However, iOS and Android are both Posix. I'm sure *many* of the extension modules - especially the ones that are purely data-based, like bz2 - could be compiled. However anything with a UI element (like tkinter) doesn't make any sense. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at keith-magee.com Sun Oct 26 01:32:30 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Sun, 26 Oct 2014 07:32:30 +0800 Subject: [Python-ideas] Fwd: Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On Sat, Oct 25, 2014 at 3:36 PM, Todd wrote: > On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" > wrote: > > 3) Disabling certain modules on mobile platforms. Supporting modules > like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on > mobile platforms doesn't make much sense; modules likes bsddb and bz2 are > difficult to support due to library dependencies; and the need for modules > like multiprocessing is arguable (and difficult to support on mobile). Even > providing a Python executable/shell is arguable for these platforms. > > I would definitely be extremely interested in a python shell in android. > One thing I feel are lacking on android are good advanced mathematical > tools and and python shell with appropriate libraries could make a very > powerful open-source tool for that. There have been some attempts at that > already. > Yes - and (to the best of my knowledge) none of them provide the default Python shell. They're custom user interfaces, using native system controls, that provide a shell-like UI. What I'm talking about here is the literal "python.exe" build target - the one that is an executable that starts and expects to attach to stdin/stdout. *That* executable isn't practical on Android *or* iOS, because neither platform has the concept of a "console" in the traditional Unix sense of the word. > I would also differentiate android and iOs more. Android seems to be > betting on multi-core performance while iOs seems to be betting on > single-chore performance. So while multiprocessing may not make much sense > on iOs, I think it may be more sense on Android, especially if they move > from 4 to 8 cores. > Firstly - I don't know what gave you the impression Apple devices aren't multicore - every Apple processor since the A5 (introduced in the iPhone 4S and iPad 2) has been at least dual core, and the A8X in the newest iPads is triple core. Secondly, if you're assuming "multicore" automatically means "mathematical powerhouse", you're mistaken. If you're planning on doing serious mathematical heavy lifting on a phone... well, you've already made your first mistake :-) > Similarly, ultimately android is Linux-based, so things like readline > should work, and it seems others have been able to get it to work, and bz2 > seems to work fine on third-party android file managers. > iOS is Posix under the hood as well. My point was that readline and curses make a whole lot of assumptions about the type of terminal you're using, and if you're writing a halfway decent UI for a mobile platform, you'd probably be better served throwing those tools overboard in preference for something that takes native UI controls into account. As for bz2 - I can't think of any reason it wouldn't work on Android (or iOS either, for that matter); I was more flagging the fact that it has binary dependencies which need to be resolved, and external dependencies complicate the cross-compilation process. More broadly, as a result of both these points, I was trying to gauge the extent to which "complete availability of all Python modules" would be considered a requirement of any official effort to port Python to Android and iOS. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at keith-magee.com Sun Oct 26 01:36:32 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Sun, 26 Oct 2014 07:36:32 +0800 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: Hi Nick, On Sat, Oct 25, 2014 at 11:25 PM, Nick Coghlan wrote: > On 25 October 2014 12:20, Russell Keith-Magee > wrote: > > I'd like to start supporting more recent versions of Python - most > > importantly, Python 3. While I could certainly continue maintaining > patches > > externally (and I imagine I'll have to if I want to maintain Python 2.7 > > support), I'd like to see that effort integrated into Python's > repository. > > Maintaining support for new versions of existing platforms is already > within scope for Python 2.7, so if you get this working for Python 3, > it *might* be possible to make the case for 2.x. However, it would > only be feasible to make that determination after we had a better idea > as to the magnitude of the changes involved, and the risks introduced > for other platforms. > Good to know - I'll keep this in mind. > > In particular, there are four areas where I can see changes required: > > 3) Disabling certain modules on mobile platforms. Supporting modules > like > > linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile > > platforms doesn't make much sense; modules likes bsddb and bz2 are > difficult > > to support due to library dependencies; and the need for modules like > > multiprocessing is arguable (and difficult to support on mobile). Even > > providing a Python executable/shell is arguable for these platforms. > > > > However, I would anticipate that 3 might raise some concerns, as the > general > > Python "batteries included" philosophy would be modified to "unless > you're > > on one of these platforms". Any guidance on whether this approach would > be > > palatable? Is there any precedent for "module not supported on platform > X" > > that I'm not aware of? > > Yes, platform specific features are actually quite common (search the > docs for "Availability:"). It's just the fact that folks writing > cross-platform code tend to avoid those modules/features, so they're > largely only known to folks writing lower level (or otherwise platform > specific) code. Good point - I hadn't thought about the OS-specific bits of the existing std lib. > > Any platforms where Python is *only* available as a > > library? > > The CPython main.c is a fairly thin shim around Py_Main anyway, so I > don't see it as a big deal whether the standard CLI is included as > part of the mobile builds. Ok - I'll keep that in mind as I work on my patches. > > So - am I completely wasting my time? Are patches for mobile platforms > > likely to be accepted into Python's repository? > > I think both iOS and Android are well and truly established enough now > as platforms to be worth targeting directly. For the CI problem, it > may be worth approaching Xamarin, as setting up the infrastructure for > doing that ourselves looks like a formidable engineering challenge we > don't currently have the relevant skillsets to provide. > Is this said in the context of "we (Python) have an existing relationship with Xamarin, and we should lean on those contacts", or "we should really develop a relationship with Xamarin"? Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at keith-magee.com Sun Oct 26 01:44:49 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Sun, 26 Oct 2014 07:44:49 +0800 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: <544BDA66.40804@udel.edu> References: <544BDA66.40804@udel.edu> Message-ID: Hi Terry, On Sun, Oct 26, 2014 at 1:14 AM, Terry Reedy wrote: > On 10/24/2014 11:06 PM, Guido van Rossum wrote: > >> This idea seems to be in the air. Just yesterday someone posted a set of >> patches and suggestions to python-dev about making cross-compilation for >> Android easier. >> > > Matt Frank, "Cross Compiling Python for Android", or close to that. He > experimented with 3.4.1. > I've seen that thread - thanks. > I expect most people in the core Python team might not >> be eager to jump on this bandwagon because they have no immediate need, >> but these mobile platforms are only getting faster more popular, and I >> think it would be a good idea to be prepared for when suddenly everybody >> wants to develop mobile apps in Python. >> >> On Fri, Oct 24, 2014 at 7:20 PM, Russell Keith-Magee >> > > wrote: >> > > tl;dr - I'd like to add iOS and Android to the list of platforms >> that Python supports out-of-the-box, and I'm looking for guidance on >> the way forward (and assistance, if anyone is keen). >> > > You (freakboy3742) and / or Matt Frank (WanderingLogic) should read the > developer guide (https://docs.python.org/devguide), submit patches to the > tracker for 3.5*, put each other as nosy (using the tracker ids above), > review each other's patches, respond to other issue participants, > especially core developers, and modify patches as needed to get them > committed. Hopefully, one of you would have the goal of joining the > developer team with commit privileges at least for android or ios > development. > > * The first beta for 3.5 is scheduled for late May, 2015. At least a > working prototype of any new feature for 3.5 should be in that release. > Good to know the timeline. If I get to the point where it looks like I have a viable patch, I'll open a ticket. > 3) Disabling certain modules on mobile platforms. Supporting >> modules like linuxaudiodev, ossaudiodev, readline, curses, idle and >> tkinter on mobile platforms doesn't make much sense; modules likes >> bsddb and bz2 are difficult to support due to library dependencies; >> and the need for modules like multiprocessing is arguable (and >> difficult to support on mobile). >> > > To add to what other said, many tests in the test suite are limited to > certain platforms already -- skip if windows, skip if not windows, etc. > Part of your patching would consist of adding skips for tests. > > We would also need to expand the possible Availability notes in the doc, > adding something like 'Availability - desktop systems' and 'Availablity - > mobile (Android/IOS)' if mobile-only features were needed. > Ok - thanks for the pointers. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at keith-magee.com Sun Oct 26 01:40:47 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Sun, 26 Oct 2014 07:40:47 +0800 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: <20141025174250.4a7edb60@fsol> References: <20141025174250.4a7edb60@fsol> Message-ID: On Sat, Oct 25, 2014 at 11:42 PM, Antoine Pitrou wrote: > On Sat, 25 Oct 2014 10:20:05 +0800 > Russell Keith-Magee > wrote: > > > > In particular, there are four areas where I can see changes required: > > > > 1) At least 2 new sys.platform definitions - "ios" and "android" (a > third > > "iossimulator" platform might also be needed - the iOS simulator has > enough > > differences that it can be helpful to be able to identify the platform) > > "ios" and "android" sound ok to me. A specific platform for the > simulator sounds wrong, though. If it simulates iOS, it should > certainly simulate its platform identification. > The iOS simulator is an odd beast. Yes, it's *mostly* like the actual device, but there are a couple of places where you need to be aware that you're on a simulator, and deal with it accordingly. For one thing, it's an i386 "device", rather than ARM; there are also some subtle differences in the way it deals with device "hardware", like the camera, or interacts with built-in apps (like the Photo library). However, for the sake of simplicity, I'll start by assuming that there's just one "iOS" platform. > > 2) Modifications to the build system to support cross-platform builds > for > > mobile - AFAICT, cross platform builds currently only work on Linux and > > require readelf; iOS libraries can only be built on Mac, and building > > Android libraries don't require readelf (AFAICT). > > I don't know anything about cross-compiling. The best is that you try > to whip up a patch and submit it on the tracker. > > Also, I suggest you tackle one system at a time. It will be easier to > review and accept patches if they target only Android or only iOS, as > opposed to both at the same time. > Understood. > 3) Disabling certain modules on mobile platforms. Supporting modules like > > linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile > > platforms doesn't make much sense; modules likes bsddb and bz2 are > > difficult to support due to library dependencies; and the need for > modules > > like multiprocessing is arguable (and difficult to support on mobile). > > Special support for this isn't necessary. If setup.py doesn't find the > necessary external dependencies, it will skip building those modules. > If OTOH those modules are built, you can remove them when creating your > installer (I imagine you won't use "setup.py install" on those > platforms). > Good point - I hadn't thought of that. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Sun Oct 26 01:52:51 2014 From: nad at acm.org (Ned Deily) Date: Sat, 25 Oct 2014 16:52:51 -0700 Subject: [Python-ideas] Adding iOS/Android support to Python References: Message-ID: In article , Andrew Barnert wrote: > On Oct 24, 2014, at 19:20, Russell Keith-Magee > wrote: > > Given three independent platform-specific build directories, it's > > relatively easy to construct a "fat" framework, but it isn't clear to me > > how that would fit into Python's build system. I can see that the build > > system has support for "universal" builds, which looks like an artefact of > > the OS X PPC->i386 conversion, but the build approach appears to be > > different to that required for iOS. > > It's still perfectly possible to build Python universal on OS X. In fact, > both the python.org builds and Apple's pre-installed builds are fat > i386/x86_64. > > iOS does put some limitations on things, especially when building shared > libs, but if you need to fall back to building each target separately and > lipo'ing them together manually, I'm pretty sure there are a few third-party > libs that already use that workaround. Didn't PyObjC used to build that way? I have no personal experience with iOS builds but I *think* the complexity of multi-arch builds is handled by the Apple-supplied compiler tools in the same way for iOS as is done for OS X builds. I assume that, like for OS X builds, the compiler driver handles the multiple builds and lipos automatically under the covers based on the requested archs, the SDK, and platform tools in use. As Andrew notes, we have long fully supported SDK builds and all of the universal archs for OS X (again using Apple's compiler chain distributed via Xcode and/or the Command Line Tools). I wouldn't expect that to be a big problem to adapt for iOS. (Cross building on platforms other than OS X is another matter.) The main issue, as always, is getting buy-in to "standardize" support for these platforms and all of the issues of long-term maintenance. Applying a set of changes is just the first step. We need to have subject experts for code and documentation. We'd need to have buildbots to give some assurance that we aren't breaking things. We need to have people willing to test pre-releases. All these things are doable and probably desirable. But they don't happen automatically. The changes that have already been added to the Python build system to support various kinds of cross-builds are today largely undocumented, unsupported by python-dev, and untested by our buildbots. We need to solve these problems for them as well before we end up with more bitrot. -- Ned Deily, nad at acm.org From russell at keith-magee.com Sun Oct 26 01:58:36 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Sun, 26 Oct 2014 07:58:36 +0800 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On Sun, Oct 26, 2014 at 4:58 AM, Andrew Barnert wrote: > On Oct 24, 2014, at 19:20, Russell Keith-Magee > wrote: > > > I'd like to start supporting more recent versions of Python - most > importantly, Python 3. While I could certainly continue maintaining patches > externally (and I imagine I'll have to if I want to maintain Python 2.7 > support), I'd like to see that effort integrated into Python's repository. > > People have already been making Python 3 builds for iOS, wrapping them in > mini-IDEs, and selling them on the App Store for quite some time. Have you > talked to Xiaowen Huang or any of the others? > > As far as I know, most of them build Python as a static lib, and link in > the C extensions from the stdlib, to get around the issues with shared-lib > frameworks before iOS 8. And I'm not sure how they deal with third-party > modules (at least one of them includes numpy, for example). > > You also may want to talk to whoever maintains the ports for jailbroken > iOS, because that's just a standard-as-possible build, with the command > line interpreter and pip and everything. > To clarify - this isn't a theoretical thing - I've got a Python 2.7 static library running on non-jailbroken iOS and Android devices, and I've published the tools (and builds) to let others do the same: https://github.com/pybee/Python-iOS-support https://github.com/pybee/Python-Android-support And I've published cookiecutter templates for iOS and Android projects if you want to write your apps in Python: https://github.com/pybee/Python-iOS-template https://github.com/pybee/Python-Android-template The support libraries are mostly patches and build system tweaks, and at present, they're all for stale versions of Python 2.7. I'm on Python-ideas because I want to move as much as possible back upstream into Python's sources. > > In particular, there are four areas where I can see changes required: > > > > 1) At least 2 new sys.platform definitions - "ios" and "android" (a > third "iossimulator" platform might also be needed - the iOS simulator has > enough differences that it can be helpful to be able to identify the > platform) > > There are other fields in platform and sys that can be used to distinguish > iOS from the simulator. And, since you need to check iOS simulator or real > iOS vs. everything else at least as often as you need to distinguish them > from each other, I'm not sure it's important whether it's treated as a > separate platform or a different version of the same platform--as long as > there's some documented and guaranteed way to distinguish them. > That's a good point; I'll investigate what can be done using other fields in platform and sys. > 3) Disabling certain modules on mobile platforms. Supporting modules > like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on > mobile platforms doesn't make much sense; modules likes bsddb and bz2 are > difficult to support due to library dependencies; and the need for modules > like multiprocessing is arguable (and difficult to support on mobile). Even > providing a Python executable/shell is arguable for these platforms. > > How is multiprocessing difficult to provide on mobile? iOS has had fork, > posix_spawn, semaphores, etc. since the first release with an SDK, although > you'd get rejected from the App Store for using them in the early days. Of > course Apple would prefer people to use XPC via grand central, but that > doesn't mean you need to make it impossible to use fork instead; at most, > you just need to warn people to read the App Store guidelines. > > In the opposite direction, it might be worth looking at the "extra > batteries" in Apple's OS X builds. At least PyObjC seems like something > you'd want to get working on iOS. > I looked at PyObjC; I was unsatisfied with the performance, and it has a native compiled component. I ended up writing my own ObjC bindings, based off an initial collection of code from Pyglet: https://github.com/pybee/rubicon-objc This uses ctypes, so there's no compiled component. > I have no idea how to tackle 4. To create a complete iOS build, you have > to do at least 4 complete Python builds - a native system build (so you can > run Python scripts that are part of the build), an i386 build (for the > simulator), an armv7 build, and an arm64 build - and then the build > products of the last three need to be combined into a single framework > directory. > > > > Given three independent platform-specific build directories, it's > relatively easy to construct a "fat" framework, but it isn't clear to me > how that would fit into Python's build system. I can see that the build > system has support for "universal" builds, which looks like an artefact of > the OS X PPC->i386 conversion, but the build approach appears to be > different to that required for iOS. > > It's still perfectly possible to build Python universal on OS X. In fact, > both the python.org builds and Apple's pre-installed builds are fat > i386/x86_64. > Yes, but AFAICT, this is done by leveraging the ability of clang to compile a binary that supports multiple architectures at once.. The iOS toolchain doesn't do that (AFAICT) - you need to build separately and lipo the products together. OSX only needs a single compiler pass; iOS needs multiple passes. iOS does put some limitations on things, especially when building shared > libs, but if you need to fall back to building each target separately and > lipo'ing them together manually, I'm pretty sure there are a few > third-party libs that already use that workaround. Didn't PyObjC used to > build that way? I'll take a closer look at PyObjC's build process and see what I can work out. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at keith-magee.com Sun Oct 26 02:56:57 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Sun, 26 Oct 2014 08:56:57 +0800 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On Sun, Oct 26, 2014 at 7:52 AM, Ned Deily wrote: > In article , > Andrew Barnert > wrote: > > On Oct 24, 2014, at 19:20, Russell Keith-Magee > > wrote: > > > Given three independent platform-specific build directories, it's > > > relatively easy to construct a "fat" framework, but it isn't clear to > me > > > how that would fit into Python's build system. I can see that the build > > > system has support for "universal" builds, which looks like an > artefact of > > > the OS X PPC->i386 conversion, but the build approach appears to be > > > different to that required for iOS. > > > > It's still perfectly possible to build Python universal on OS X. In fact, > > both the python.org builds and Apple's pre-installed builds are fat > > i386/x86_64. > > > > iOS does put some limitations on things, especially when building shared > > libs, but if you need to fall back to building each target separately and > > lipo'ing them together manually, I'm pretty sure there are a few > third-party > > libs that already use that workaround. Didn't PyObjC used to build that > way? > > I have no personal experience with iOS builds but I *think* the > complexity of multi-arch builds is handled by the Apple-supplied > compiler tools in the same way for iOS as is done for OS X builds. That would be a nice if it were true, but unfortunately, it isn't AFAICT. Under OS/X, to compile a single .c file, you make a single call to clang; if you reference an external library, you only need to point at a single library file. The system libraries provided by Apple are "fat", and contain multiple architectures. If there's an analogous set of options for iOS, I haven't been able to find them. In particular, there's a very distinct split between the x86 simulator libraries, and the ARM device libraries. In order to compile a fat iOS library, you have to do *at least* 2 compiles - one for i386, and one for ARM - because there are two completely separate - and incompatible - system libraries. On top of that, the OS/X build is self hosting - once you've compiled python.exe, you can use that executable to compile the python sources and to run the setup.py script that compiles module extensions. Netiher of the iOS builds are self hosting - you have to have a working native python compiler before you . It looks like it *might* be possible to compile the multiple *device* architectures (armv7, armv7s, arm64) on a single compile pass, but I haven't quite worked out the incantation yet. In the meantime, I'm having to do 5 complete builds (native, i386 simulator, armv7, armv7s, and arm64) to get a single library that can be used in a standalone project. I > assume that, like for OS X builds, the compiler driver handles the > multiple builds and lipos automatically under the covers based on the > requested archs, the SDK, and platform tools in use. As Andrew notes, > we have long fully supported SDK builds and all of the universal archs > for OS X (again using Apple's compiler chain distributed via Xcode > and/or the Command Line Tools). I wouldn't expect that to be a big > problem to adapt for iOS. (Cross building on platforms other than OS X > is another matter.) > > The main issue, as always, is getting buy-in to "standardize" support > for these platforms and all of the issues of long-term maintenance. > Applying a set of changes is just the first step. We need to have > subject experts for code and documentation. We'd need to have buildbots > to give some assurance that we aren't breaking things. We need to have > people willing to test pre-releases. All these things are doable and > probably desirable. But they don't happen automatically. The changes > that have already been added to the Python build system to support > various kinds of cross-builds are today largely undocumented, > unsupported by python-dev, and untested by our buildbots. We need to > solve these problems for them as well before we end up with more bitrot. ... and that's what I've been trying to gauge with my post. I have a personal need for "Python on Mobile" support, and I don't think I'm alone in this. I can maintain a set of my own build tools and patch sets that provide Python support to my own satisfaction, but I think there's value in having that capability in Python's trunk. However, I've got enough experience maintaining a large Open Source Python project to know that the story doesn't stop with submitting a patch that scratches my own itch - there's the long term maintenance to consider. And in this case, there's all the additional complications of getting testing, CI etc working on a mobile platform. So - at the core of all of this is the big question: Should I bother? If I was to do the work, is the core team interested? And if so, what constitutes "the work" from Python's perspective? Is it just adding platform libraries and some patches to the build system to allow cross-platform builds for iOS and Android? Or do I need to clean up and document the entire cross-platform build system? And a quick meta-question - Is there *any* documentation of the current build system? I've had a poke around, and I can't find anything that seems on topic. The "Setup and Usage' guide (or maybe the Windows FAW) is the closest I can find. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Sun Oct 26 03:14:57 2014 From: nad at acm.org (Ned Deily) Date: Sat, 25 Oct 2014 19:14:57 -0700 Subject: [Python-ideas] Adding iOS/Android support to Python References: Message-ID: In article , Russell Keith-Magee wrote: > On Sun, Oct 26, 2014 at 7:52 AM, Ned Deily > wrote: > > I have no personal experience with iOS builds but I *think* the > > complexity of multi-arch builds is handled by the Apple-supplied > > compiler tools in the same way for iOS as is done for OS X builds. > That would be a nice if it were true, but unfortunately, it isn't AFAICT. > > Under OS/X, to compile a single .c file, you make a single call to clang; > if you reference an external library, you only need to point at a single > library file. The system libraries provided by Apple are "fat", and contain > multiple architectures. > > If there's an analogous set of options for iOS, I haven't been able to find > them. In particular, there's a very distinct split between the x86 > simulator libraries, and the ARM device libraries. In order to compile a > fat iOS library, you have to do *at least* 2 compiles - one for i386, and > one for ARM - because there are two completely separate - and incompatible > - system libraries. [...] > It looks like it *might* be possible to compile the multiple *device* > architectures (armv7, armv7s, arm64) on a single compile pass, but I > haven't quite worked out the incantation yet. In the meantime, I'm having > to do 5 complete builds (native, i386 simulator, armv7, armv7s, and arm64) > to get a single library that can be used in a standalone project. This is all really off-topic but, after a little bit of experimentation, I think it is the case that you can compile and link multiple iOS archs (armv7, armv7s, arm64) in a single cc/clang call just as you can for OS X archs; you just need to specify the proper sdk via xcrun (or the other supported methods) and the proper -arch values to clang. One way: xcrun --sdk iphoneos8.1 cc -arch arm64 -arch armv7 -arch armv7s -c test.c But I think you are correct that it is not possible to do iOS builds and iOS simulators builds in one step as they are definitely different platforms from the tool chain perspective (just os OS X is a different platforms); I'm not sure how important that is. $ xcodebuild -showsdks OS X SDKs: OS X 10.9 -sdk macosx10.9 OS X 10.10 -sdk macosx10.10 iOS SDKs: iOS 8.1 -sdk iphoneos8.1 iOS Simulator SDKs: Simulator - iOS 8.1 -sdk iphonesimulator8.1 > On top of that, the OS/X build is self hosting - once you've compiled > python.exe, you can use that executable to compile the python sources and > to run the setup.py script that compiles module extensions. Netiher of the > iOS builds are self hosting - you have to have a working native python > compiler before you . As I mentioned, there already is some, mostly undocumented support for cross-building with separation of build host tools from target host tools. Presumably that could be extended as needed. > > The main issue, as always, is getting buy-in to "standardize" support > > for these platforms and all of the issues of long-term maintenance. > > Applying a set of changes is just the first step. We need to have > > subject experts for code and documentation. We'd need to have buildbots > > to give some assurance that we aren't breaking things. We need to have > > people willing to test pre-releases. All these things are doable and > > probably desirable. But they don't happen automatically. The changes > > that have already been added to the Python build system to support > > various kinds of cross-builds are today largely undocumented, > > unsupported by python-dev, and untested by our buildbots. We need to > > solve these problems for them as well before we end up with more bitrot. > ... and that's what I've been trying to gauge with my post. I have a > personal need for "Python on Mobile" support, and I don't think I'm alone > in this. I can maintain a set of my own build tools and patch sets that > provide Python support to my own satisfaction, but I think there's value in > having that capability in Python's trunk. > > However, I've got enough experience maintaining a large Open Source Python > project to know that the story doesn't stop with submitting a patch that > scratches my own itch - there's the long term maintenance to consider. And > in this case, there's all the additional complications of getting testing, > CI etc working on a mobile platform. > > So - at the core of all of this is the big question: Should I bother? If I > was to do the work, is the core team interested? And if so, what > constitutes "the work" from Python's perspective? Is it just adding > platform libraries and some patches to the build system to allow > cross-platform builds for iOS and Android? Or do I need to clean up and > document the entire cross-platform build system? That's a really good question. I'm not sure how you get an answer. Perhaps one approach is through producing a PEP and getting at least one core developer to champion it. Guido's positive response is important. There would need to be some sort to consensus over on python-dev as well, I think. > And a quick meta-question - Is there *any* documentation of the current > build system? I've had a poke around, and I can't find anything that seems > on topic. The "Setup and Usage' guide (or maybe the Windows FAW) is the > closest I can find. The Python Developer's Guide is probably the best place to start. But you'll find very little about cross-building since, AFAIK, there are few core developers who are familiar with the topic. Matthias Klose added the most recent cross-build changes. https://docs.python.org/devguide/ -- Ned Deily, nad at acm.org From stefan_ml at behnel.de Sun Oct 26 09:18:06 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 26 Oct 2014 09:18:06 +0100 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: Russell Keith-Magee schrieb am 25.10.2014 um 04:20: > Kivy has been around for a few years, with similar goals (although > they've got some different ideas and priorities). > > To date, I've been using the groundwork established by Kivy, which takes > the Python 2.7.1 (for iOS) and Python 2.7.2 (for Android) source code, > applies a bunch of patches, and wraps those in a set of build scripts to > yield Python libraries that can be used on iOS and Android. I assume you're in touch with the Kivy developers to join forces? Stefan From ncoghlan at gmail.com Sun Oct 26 12:07:34 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Oct 2014 21:07:34 +1000 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On 26 October 2014 09:36, Russell Keith-Magee wrote: > On Sat, Oct 25, 2014 at 11:25 PM, Nick Coghlan wrote: >> >> I think both iOS and Android are well and truly established enough now >> as platforms to be worth targeting directly. For the CI problem, it >> may be worth approaching Xamarin, as setting up the infrastructure for >> doing that ourselves looks like a formidable engineering challenge we >> don't currently have the relevant skillsets to provide. > > Is this said in the context of "we (Python) have an existing relationship > with Xamarin, and we should lean on those contacts", or "we should really > develop a relationship with Xamarin"? The latter - although as Antoine noted, we *do* have some nominally supported platforms that don't have buildbots. That configuration just means that *other* core devs aren't likely to be too worried about breaking the platform - more of the burden will fall back on the folks actually keeping that platform running. However, it did occur to me that running in the iOS and Android simulators should be feasible on a normal x86_64 system, assuming you can figure out a way to run the regression test suite without a normal console. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From toddrjen at gmail.com Sun Oct 26 13:18:47 2014 From: toddrjen at gmail.com (Todd) Date: Sun, 26 Oct 2014 13:18:47 +0100 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On Oct 25, 2014 10:13 AM, "Russell Keith-Magee" wrote: > > > On Sat, Oct 25, 2014 at 3:36 PM, Todd wrote: >> >> On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" wrote: >> > 3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. >> >> I would definitely be extremely interested in a python shell in android. One thing I feel are lacking on android are good advanced mathematical tools and and python shell with appropriate libraries could make a very powerful open-source tool for that. There have been some attempts at that already. > > Yes - and (to the best of my knowledge) none of them provide the default Python shell. They're custom user interfaces, using native system controls, that provide a shell-like UI. What I'm talking about here is the literal "python.exe" build target - the one that is an executable that starts and expects to attach to stdin/stdout. *That* executable isn't practical on Android *or* iOS, because neither platform has the concept of a "console" in the traditional Unix sense of the word. Perhaps no console by default, but it is possible to have a traditional console on android. I have one and many ROMs install one by default. So although it may not be part of the default configuration I think it would be good to support it for the people who do have a console. Further, with rooted users, python could be set up to be used with the built-in adb shell. It is unclear from the discussion where things ultimately came out on this issue. If there still a possibility it might removed, although I understand that consoles are not the primary use-case, I think is still a valid use-case that should supported. >> I would also differentiate android and iOs more. Android seems to be betting on multi-core performance while iOs seems to be betting on single-chore performance. So while multiprocessing may not make much sense on iOs, I think it may be more sense on Android, especially if they move from 4 to 8 cores. > > Firstly - I don't know what gave you the impression Apple devices aren't multicore - every Apple processor since the A5 (introduced in the iPhone 4S and iPad 2) has been at least dual core, and the A8X in the newest iPads is triple core. I was referring to the benchmarks where corresponding iOs and android devices generally have better single and multi-core performance, respectively, but you right that isn't that important. > Secondly, if you're assuming "multicore" automatically means "mathematical powerhouse", you're mistaken. If you're planning on doing serious mathematical heavy lifting on a phone... well, you've already made your first mistake :-) No, on the contrary, I was thinking that on devices with limited performance, being able to divide components between processes, such as UI and logic, is all the more important. It probably would not be any use for the sort of calculator I am thinking about. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Sun Oct 26 16:44:09 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sun, 26 Oct 2014 16:44:09 +0100 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: <368032DF-857D-46E2-9BC3-4AF87F1EB2F5@mac.com> > On 25 Oct 2014, at 22:58, Andrew Barnert wrote: > > On Oct 24, 2014, at 19:20, Russell Keith-Magee wrote: > > >> I have no idea how to tackle 4. To create a complete iOS build, you have to do at least 4 complete Python builds - a native system build (so you can run Python scripts that are part of the build), an i386 build (for the simulator), an armv7 build, and an arm64 build - and then the build products of the last three need to be combined into a single framework directory. >> >> Given three independent platform-specific build directories, it's relatively easy to construct a "fat" framework, but it isn't clear to me how that would fit into Python's build system. I can see that the build system has support for "universal" builds, which looks like an artefact of the OS X PPC->i386 conversion, but the build approach appears to be different to that required for iOS. > > It's still perfectly possible to build Python universal on OS X. In fact, both the python.org builds and Apple's pre-installed builds are fat i386/x86_64. > > iOS does put some limitations on things, especially when building shared libs, but if you need to fall back to building each target separately and lipo'ing them together manually, I'm pretty sure there are a few third-party libs that already use that workaround. Didn't PyObjC used to build that way? Nope. PyObjC directly builds fat binaries without explicitly using lipo. That?s for PyObjC on OSX, I have no idea if there still is a port for PyObjC to iOS and how that?s build. Ronald From wes.turner at gmail.com Sun Oct 26 19:50:19 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 26 Oct 2014 13:50:19 -0500 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: > It probably would not be any use for the sort of calculator I am thinking about. * "Hacker's Keyboard, a 5-row keyboard using full PC key layout for Android tablets or phones " https://code.google.com/p/hackerskeyboard/ * IPython Notebook requires pyzmq and libzmq: http://zeromq.org/build:android * IPython qtconsole requires Qt: http://qt-project.org/doc/qt-5/android-support.html * SPyder require PySide or PyQt: https://code.google.com/p/spyderlib/ * http://continuum.io/blog/raspberry ( http://repo.continuum.io/miniconda/Miniconda-3.5.5-Linux-armv6l.sh ) On Sun, Oct 26, 2014 at 7:18 AM, Todd wrote: > > On Oct 25, 2014 10:13 AM, "Russell Keith-Magee" > wrote: > > > > > > On Sat, Oct 25, 2014 at 3:36 PM, Todd wrote: > >> > >> On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" > wrote: > >> > 3) Disabling certain modules on mobile platforms. Supporting modules > like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on > mobile platforms doesn't make much sense; modules likes bsddb and bz2 are > difficult to support due to library dependencies; and the need for modules > like multiprocessing is arguable (and difficult to support on mobile). Even > providing a Python executable/shell is arguable for these platforms. > >> > >> I would definitely be extremely interested in a python shell in > android. One thing I feel are lacking on android are good advanced > mathematical tools and and python shell with appropriate libraries could > make a very powerful open-source tool for that. There have been some > attempts at that already. > > > > Yes - and (to the best of my knowledge) none of them provide the default > Python shell. They're custom user interfaces, using native system controls, > that provide a shell-like UI. What I'm talking about here is the literal > "python.exe" build target - the one that is an executable that starts and > expects to attach to stdin/stdout. *That* executable isn't practical on > Android *or* iOS, because neither platform has the concept of a "console" > in the traditional Unix sense of the word. > > Perhaps no console by default, but it is possible to have a traditional > console on android. I have one and many ROMs install one by default. So > although it may not be part of the default configuration I think it would > be good to support it for the people who do have a console. > > Further, with rooted users, python could be set up to be used with the > built-in adb shell. > > It is unclear from the discussion where things ultimately came out on this > issue. If there still a possibility it might removed, although I understand > that consoles are not the primary use-case, I think is still a valid > use-case that should supported. > > >> I would also differentiate android and iOs more. Android seems to be > betting on multi-core performance while iOs seems to be betting on > single-chore performance. So while multiprocessing may not make much sense > on iOs, I think it may be more sense on Android, especially if they move > from 4 to 8 cores. > > > > Firstly - I don't know what gave you the impression Apple devices aren't > multicore - every Apple processor since the A5 (introduced in the iPhone 4S > and iPad 2) has been at least dual core, and the A8X in the newest iPads is > triple core. > > I was referring to the benchmarks where corresponding iOs and android > devices generally have better single and multi-core performance, > respectively, but you right that isn't that important. > > > Secondly, if you're assuming "multicore" automatically means > "mathematical powerhouse", you're mistaken. If you're planning on doing > serious mathematical heavy lifting on a phone... well, you've already made > your first mistake :-) > > No, on the contrary, I was thinking that on devices with limited > performance, being able to divide components between processes, such as UI > and logic, is all the more important. It probably would not be any use for > the sort of calculator I am thinking about. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Wes Turner https://westurner.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sun Oct 26 19:58:21 2014 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 26 Oct 2014 13:58:21 -0500 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: As far as tagging multi-platform builds: * wheel: http://legacy.python.org/dev/peps/pep-0427/#file-name-convention {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl. * conda: http://conda.pydata.org/docs/build.html#preprocessing-selectors On Sun, Oct 26, 2014 at 1:50 PM, Wes Turner wrote: > > It probably would not be any use for the sort of calculator I am > thinking about. > > * "Hacker's Keyboard, a 5-row keyboard using full PC key layout for > Android tablets or phones " > https://code.google.com/p/hackerskeyboard/ > * IPython Notebook requires pyzmq and libzmq: > http://zeromq.org/build:android > * IPython qtconsole requires Qt: > http://qt-project.org/doc/qt-5/android-support.html > * SPyder require PySide or PyQt: https://code.google.com/p/spyderlib/ > * http://continuum.io/blog/raspberry ( > http://repo.continuum.io/miniconda/Miniconda-3.5.5-Linux-armv6l.sh ) > > > On Sun, Oct 26, 2014 at 7:18 AM, Todd wrote: > >> >> On Oct 25, 2014 10:13 AM, "Russell Keith-Magee" >> wrote: >> > >> > >> > On Sat, Oct 25, 2014 at 3:36 PM, Todd wrote: >> >> >> >> On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" < >> russell at keith-magee.com> wrote: >> >> > 3) Disabling certain modules on mobile platforms. Supporting >> modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter >> on mobile platforms doesn't make much sense; modules likes bsddb and bz2 >> are difficult to support due to library dependencies; and the need for >> modules like multiprocessing is arguable (and difficult to support on >> mobile). Even providing a Python executable/shell is arguable for these >> platforms. >> >> >> >> I would definitely be extremely interested in a python shell in >> android. One thing I feel are lacking on android are good advanced >> mathematical tools and and python shell with appropriate libraries could >> make a very powerful open-source tool for that. There have been some >> attempts at that already. >> > >> > Yes - and (to the best of my knowledge) none of them provide the >> default Python shell. They're custom user interfaces, using native system >> controls, that provide a shell-like UI. What I'm talking about here is the >> literal "python.exe" build target - the one that is an executable that >> starts and expects to attach to stdin/stdout. *That* executable isn't >> practical on Android *or* iOS, because neither platform has the concept of >> a "console" in the traditional Unix sense of the word. >> >> Perhaps no console by default, but it is possible to have a traditional >> console on android. I have one and many ROMs install one by default. So >> although it may not be part of the default configuration I think it would >> be good to support it for the people who do have a console. >> >> Further, with rooted users, python could be set up to be used with the >> built-in adb shell. >> >> It is unclear from the discussion where things ultimately came out on >> this issue. If there still a possibility it might removed, although I >> understand that consoles are not the primary use-case, I think is still a >> valid use-case that should supported. >> >> >> I would also differentiate android and iOs more. Android seems to be >> betting on multi-core performance while iOs seems to be betting on >> single-chore performance. So while multiprocessing may not make much sense >> on iOs, I think it may be more sense on Android, especially if they move >> from 4 to 8 cores. >> > >> > Firstly - I don't know what gave you the impression Apple devices >> aren't multicore - every Apple processor since the A5 (introduced in the >> iPhone 4S and iPad 2) has been at least dual core, and the A8X in the >> newest iPads is triple core. >> >> I was referring to the benchmarks where corresponding iOs and android >> devices generally have better single and multi-core performance, >> respectively, but you right that isn't that important. >> >> > Secondly, if you're assuming "multicore" automatically means >> "mathematical powerhouse", you're mistaken. If you're planning on doing >> serious mathematical heavy lifting on a phone... well, you've already made >> your first mistake :-) >> >> No, on the contrary, I was thinking that on devices with limited >> performance, being able to divide components between processes, such as UI >> and logic, is all the more important. It probably would not be any use for >> the sort of calculator I am thinking about. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Wes Turner > https://westurner.github.io/ > -- Wes Turner https://westurner.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at keith-magee.com Mon Oct 27 00:27:46 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Mon, 27 Oct 2014 07:27:46 +0800 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On Sun, Oct 26, 2014 at 7:07 PM, Nick Coghlan wrote: > On 26 October 2014 09:36, Russell Keith-Magee > wrote: > > On Sat, Oct 25, 2014 at 11:25 PM, Nick Coghlan > wrote: > >> > >> I think both iOS and Android are well and truly established enough now > >> as platforms to be worth targeting directly. For the CI problem, it > >> may be worth approaching Xamarin, as setting up the infrastructure for > >> doing that ourselves looks like a formidable engineering challenge we > >> don't currently have the relevant skillsets to provide. > > > > Is this said in the context of "we (Python) have an existing relationship > > with Xamarin, and we should lean on those contacts", or "we should really > > develop a relationship with Xamarin"? > > The latter - although as Antoine noted, we *do* have some nominally > supported platforms that don't have buildbots. That configuration just > means that *other* core devs aren't likely to be too worried about > breaking the platform - more of the burden will fall back on the folks > actually keeping that platform running. > Understood. That's essentially the status quo anyway, just with the code outside the Python repository. However, it did occur to me that running in the iOS and Android > simulators should be feasible on a normal x86_64 system, assuming you > can figure out a way to run the regression test suite without a normal > console. > Both iOS and Android have a console that could be used to display test output - it's just not visible on the phone itself. The "Hello world" you get at the end of the template project I linked to earlier produces a couple of lines on the debug console. In theory, I guess it should be possible to build a full iOS/Android "App" whose only purpose is to run the Python test suite. I haven't tried this; I'll put it on my todo list. Regarding the test process: It's certainly possible to run the iOS simulator and Android emulator on x86_64 machines -- but the iOS simulator is a completely different CPU architecture to that used on a device. It might be useful as a sanity check, but you'd still want to run the actual iOS build on an actual iOS device to confirm it was fully working as expected. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From russell at keith-magee.com Mon Oct 27 00:35:44 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Mon, 27 Oct 2014 07:35:44 +0800 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: Hi Todd, On Sun, Oct 26, 2014 at 8:18 PM, Todd wrote: > > On Oct 25, 2014 10:13 AM, "Russell Keith-Magee" > wrote: > > > > > > On Sat, Oct 25, 2014 at 3:36 PM, Todd wrote: > >> > >> On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" > wrote: > >> > 3) Disabling certain modules on mobile platforms. Supporting modules > like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on > mobile platforms doesn't make much sense; modules likes bsddb and bz2 are > difficult to support due to library dependencies; and the need for modules > like multiprocessing is arguable (and difficult to support on mobile). Even > providing a Python executable/shell is arguable for these platforms. > >> > >> I would definitely be extremely interested in a python shell in > android. One thing I feel are lacking on android are good advanced > mathematical tools and and python shell with appropriate libraries could > make a very powerful open-source tool for that. There have been some > attempts at that already. > > > > Yes - and (to the best of my knowledge) none of them provide the default > Python shell. They're custom user interfaces, using native system controls, > that provide a shell-like UI. What I'm talking about here is the literal > "python.exe" build target - the one that is an executable that starts and > expects to attach to stdin/stdout. *That* executable isn't practical on > Android *or* iOS, because neither platform has the concept of a "console" > in the traditional Unix sense of the word. > > Perhaps no console by default, but it is possible to have a traditional > console on android. I have one and many ROMs install one by default. So > although it may not be part of the default configuration I think it would > be good to support it for the people who do have a console. > > Further, with rooted users, python could be set up to be used with the > built-in adb shell. > It is unclear from the discussion where things ultimately came out on this > issue. If there still a possibility it might removed, although I understand > that consoles are not the primary use-case, I think is still a valid > use-case that should supported. > Supporting rooted devices and "unix console on mobile device" features are of no interest to me personally; so I suppose the question is whether providing support for these use cases would be an impediment to a set of patches being added to Python's trunk. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Oct 27 02:25:27 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 26 Oct 2014 21:25:27 -0400 Subject: [Python-ideas] Fwd: Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On 10/25/2014 7:32 PM, Russell Keith-Magee wrote: > > On Sat, Oct 25, 2014 at 3:36 PM, Todd > > wrote: > > On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" > > wrote: > > 3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. > > I would definitely be extremely interested in a python shell in > android. One thing I feel are lacking on android are good advanced > mathematical tools and and python shell with appropriate libraries > could make a very powerful open-source tool for that. There have > been some attempts at that already. > > Yes - and (to the best of my knowledge) none of them provide the default > Python shell. They're custom user interfaces, using native system > controls, that provide a shell-like UI. What I'm talking about here is > the literal "python.exe" build target - the one that is an executable > that starts and expects to attach to stdin/stdout. *That* executable > isn't practical on Android *or* iOS, because neither platform has the > concept of a "console" in the traditional Unix sense of the word. > > I would also differentiate android and iOs more. Android seems to > be betting on multi-core performance while iOs seems to be betting > on single-chore performance. So while multiprocessing may not make > much sense on iOs, I think it may be more sense on Android, > especially if they move from 4 to 8 cores. > > Firstly - I don't know what gave you the impression Apple devices aren't > multicore - every Apple processor since the A5 (introduced in the iPhone > 4S and iPad 2) has been at least dual core, and the A8X in the newest > iPads is triple core. > > Secondly, if you're assuming "multicore" automatically means > "mathematical powerhouse", you're mistaken. If you're planning on doing > serious mathematical heavy lifting on a phone... well, you've already > made your first mistake :-) > > Similarly, ultimately android is Linux-based, so things like > readline should work, and it seems others have been able to get it > to work, and bz2 seems to work fine on third-party android file > managers. > > iOS is Posix under the hood as well. My point was that readline and > curses make a whole lot of assumptions about the type of terminal you're > using, and if you're writing a halfway decent UI for a mobile platform, > you'd probably be better served throwing those tools overboard in > preference for something that takes native UI controls into account. > > As for bz2 - I can't think of any reason it wouldn't work on Android (or > iOS either, for that matter); I was more flagging the fact that it has > binary dependencies which need to be resolved, and external dependencies > complicate the cross-compilation process. > > More broadly, as a result of both these points, I was trying to gauge > the extent to which "complete availability of all Python modules" would > be considered a requirement of any official effort to port Python to > Android and iOS. > > Yours, > Russ Magee %-) > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Terry Jan Reedy From tjreedy at udel.edu Mon Oct 27 02:30:17 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 26 Oct 2014 21:30:17 -0400 Subject: [Python-ideas] Fwd: Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On 10/26/2014 9:25 PM, Terry Reedy wrote: nothing -- hit wrong button, sorry. > On 10/25/2014 7:32 PM, Russell Keith-Magee wrote: stuff -- Terry Jan Reedy From russell at keith-magee.com Mon Oct 27 02:53:55 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Mon, 27 Oct 2014 09:53:55 +0800 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: Hi Stefan, On Sun, Oct 26, 2014 at 4:18 PM, Stefan Behnel wrote: > Russell Keith-Magee schrieb am 25.10.2014 um 04:20: > > Kivy has been around for a few years, with similar goals (although > > they've got some different ideas and priorities). > > > > To date, I've been using the groundwork established by Kivy, which takes > > the Python 2.7.1 (for iOS) and Python 2.7.2 (for Android) source code, > > applies a bunch of patches, and wraps those in a set of build scripts to > > yield Python libraries that can be used on iOS and Android. > > I assume you're in touch with the Kivy developers to join forces? > We've had some initial introductory conversations on Twitter, but we're not working closely on anything at the moment. There's certainly potential for collaboration, though. Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Oct 27 14:11:46 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Oct 2014 23:11:46 +1000 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: On 27 October 2014 11:53, Russell Keith-Magee wrote: > Hi Stefan, > > On Sun, Oct 26, 2014 at 4:18 PM, Stefan Behnel wrote: >> >> Russell Keith-Magee schrieb am 25.10.2014 um 04:20: >> > Kivy has been around for a few years, with similar goals (although >> > they've got some different ideas and priorities). >> > >> > To date, I've been using the groundwork established by Kivy, which takes >> > the Python 2.7.1 (for iOS) and Python 2.7.2 (for Android) source code, >> > applies a bunch of patches, and wraps those in a set of build scripts to >> > yield Python libraries that can be used on iOS and Android. >> >> I assume you're in touch with the Kivy developers to join forces? > > > We've had some initial introductory conversations on Twitter, but we're not > working closely on anything at the moment. There's certainly potential for > collaboration, though. Some other folks potentially worth contacting: * Jonathan Hosmer (creator of http://pythonforios.com/) * Ole Zorn (creator of http://omz-software.com/pythonista/) * Karl Traunm?ller (creator of http://computableapp.com/) If working on better iOS and Android upstream happens to help bring those apps to Android as well, that would be a rather nice outcome :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From sturla.molden at gmail.com Tue Oct 28 01:52:00 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Tue, 28 Oct 2014 00:52:00 +0000 (UTC) Subject: [Python-ideas] Adding iOS/Android support to Python References: Message-ID: <562838779436149584.465371sturla.molden-gmail.com@news.gmane.org> Nick Coghlan wrote: > I think both iOS and Android are well and truly established enough now > as platforms to be worth targeting directly. Does Apple allow Python apps in AppStore now? I thought they had a ban on anything not written in Objective-C or Swift? Sturla From russell at keith-magee.com Tue Oct 28 03:00:53 2014 From: russell at keith-magee.com (Russell Keith-Magee) Date: Tue, 28 Oct 2014 10:00:53 +0800 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: <562838779436149584.465371sturla.molden-gmail.com@news.gmane.org> References: <562838779436149584.465371sturla.molden-gmail.com@news.gmane.org> Message-ID: On Tue, Oct 28, 2014 at 8:52 AM, Sturla Molden wrote: > Nick Coghlan wrote: > > > I think both iOS and Android are well and truly established enough now > > as platforms to be worth targeting directly. > > Does Apple allow Python apps in AppStore now? I thought they had a ban on > anything not written in Objective-C or Swift? > Hi Sturla, That restriction was lifted a while back. Apple now allows apps that embed other languages, with some restrictions about the source of the scripts being run - for example, your app must contain all the scripts it's going to use, you can't download new scripts from the web. Evidence of this is Kivy - there are a number of apps written using Kivy on the App Store: https://github.com/kivy/kivy/wiki/List-of-Kivy-Projects Of course, there's no guarantee this won't change back in the future. Working at the whims and fancies of the AppStore T&C's will always be a risk. :-) Yours, Russ Magee %-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Oct 28 05:00:31 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 27 Oct 2014 21:00:31 -0700 Subject: [Python-ideas] Adding iOS/Android support to Python In-Reply-To: References: Message-ID: <684C5A2D-6796-448F-9096-8C51712020A9@yahoo.com> On Oct 26, 2014, at 16:35, Russell Keith-Magee wrote: > Hi Todd, > > On Sun, Oct 26, 2014 at 8:18 PM, Todd wrote: >> >> On Oct 25, 2014 10:13 AM, "Russell Keith-Magee" wrote: >> > >> > >> > On Sat, Oct 25, 2014 at 3:36 PM, Todd wrote: >> >> >> >> On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" wrote: >> >> > 3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. >> >> >> >> I would definitely be extremely interested in a python shell in android. One thing I feel are lacking on android are good advanced mathematical tools and and python shell with appropriate libraries could make a very powerful open-source tool for that. There have been some attempts at that already. >> > >> > Yes - and (to the best of my knowledge) none of them provide the default Python shell. They're custom user interfaces, using native system controls, that provide a shell-like UI. What I'm talking about here is the literal "python.exe" build target - the one that is an executable that starts and expects to attach to stdin/stdout. *That* executable isn't practical on Android *or* iOS, because neither platform has the concept of a "console" in the traditional Unix sense of the word. >> >> Perhaps no console by default, but it is possible to have a traditional console on android. I have one and many ROMs install one by default. So although it may not be part of the default configuration I think it would be good to support it for the people who do have a console. >> >> Further, with rooted users, python could be set up to be used with the built-in adb shell. >> >> It is unclear from the discussion where things ultimately came out on this issue. If there still a possibility it might removed, although I understand that consoles are not the primary use-case, I think is still a valid use-case that should supported. >> > Supporting rooted devices and "unix console on mobile device" features are of no interest to me personally; so I suppose the question is whether providing support for these use cases would be an impediment to a set of patches being added to Python's trunk. I don't think it's so much people wanted to use the Unix console directly on their device, as wanting to use it when ssh'd into their device. I built 3.2 or 3.3 for iOS (on the device, using the tool chain from Cydia) just to make it usable in test scripts for an app I was working on. But that seems like a very different use case from people who want a cross-compilable framework that they can use in an app, that runs in the simulator, that has GUI libraries, etc. If it's that hard to support the interpreter, I think it would be reasonable to disable it, and let people who want that create a Cydia package separate from yours. -------------- next part -------------- An HTML attachment was scrubbed... URL: From antony.lee at berkeley.edu Wed Oct 29 23:39:59 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Wed, 29 Oct 2014 15:39:59 -0700 Subject: [Python-ideas] operator.call / operator.__call__ Message-ID: A simple suggestion: add "operator.call" and "operator.__call__", which would provide a function like Python2's "apply" (but it'd be called with already unpacked arguments, i.e. operator.call(f, *args, **kwargs)). Why? To be able to pass it as an argument to other function, just like the other functions defined in the "operator" module. Antony -------------- next part -------------- An HTML attachment was scrubbed... URL: From kaiser.yann at gmail.com Thu Oct 30 00:23:03 2014 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Wed, 29 Oct 2014 23:23:03 +0000 Subject: [Python-ideas] operator.call / operator.__call__ References: Message-ID: How would this be different from using functools.partial? On Wed, Oct 29, 2014, 23:40 Antony Lee wrote: > A simple suggestion: add "operator.call" and "operator.__call__", which > would provide a function like Python2's "apply" (but it'd be called with > already unpacked arguments, i.e. operator.call(f, *args, **kwargs)). > Why? To be able to pass it as an argument to other function, just like > the other functions defined in the "operator" module. > > Antony > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Oct 30 01:53:22 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 30 Oct 2014 11:53:22 +1100 Subject: [Python-ideas] Deprecate dunder functions from operator module Message-ID: <20141030005321.GC26866@ando.pearwood.info> (For the avoidance of all doubt, the following has nothing to do with dunder methods like int.__add__ etc., but only the functions in the operator module.) The operator module contains various function versions of operators, e.g. operator.add for + , operator.sub for - , etc. There are also dunder versions of each function, and according to the documentation: The function names are those used for special class methods; variants without leading and trailing __ are also provided for convenience. https://docs.python.org/3/library/operator.html (except that's not quite right, because there are no functions __radd__ etc. matching the special class methods). The existence of redundant dunder functions came as a complete surprise to me when I recently discovered it, and I daresay it will come as a surprise to many people. Having two versions in the operator module has been known to lead to confusion, e.g.: https://mail.python.org/pipermail/tutor/2014-October/102877.html All the third party documentation I've seen for the operator module refers to the dunderless versions. I doubt that there is anyone who prefers to write operator.__add__ instead of operator.add, and Terry Reedy has kindly determined that (aside from tests) the standard library doesn't use any of the dunder versions. https://mail.python.org/pipermail/python-list/2014-October/679226.html I propose a few things: * institute a policy that, in the event of a new function being added to the operator module, only the dunderless version will be added; * change the documentation to make it clear that the dunderless versions should be used, rather than merely being "convenience" functions; * add a prominent note that the dunder versions exist for backwards compatibility only and should not be used in new code. At this point, I do NOT propose having the dunder versions of the functions raise a depreciation warning. I would be content for them to merely be documented as discouraged, and defer actual depreciation until Python 5000 or so. (But if there is a strong preference for actual deprecation, I won't argue against it.) Thoughts? -- Steven From steve at pearwood.info Thu Oct 30 01:57:05 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 30 Oct 2014 11:57:05 +1100 Subject: [Python-ideas] operator.call / operator.__call__ In-Reply-To: References: Message-ID: <20141030005705.GD26866@ando.pearwood.info> On Wed, Oct 29, 2014 at 03:39:59PM -0700, Antony Lee wrote: > A simple suggestion: add "operator.call" and "operator.__call__", which > would provide a function like Python2's "apply" (but it'd be called with > already unpacked arguments, i.e. operator.call(f, *args, **kwargs)). > Why? To be able to pass it as an argument to other function, just like the > other functions defined in the "operator" module. I think you want something like this? def call(callable, *args, **kwargs): return callable(*args, **kwargs) Can you give an example of how you might use this call method? It doesn't have to be a real world use-case, just an illustration. Perhaps I'm being a bit dim-witted today, but I'm having trouble thinking of where I would use this operator.call rather than just directly applying *args, **kwargs to the callable. -- Steven From ethan at stoneleaf.us Thu Oct 30 02:38:15 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 29 Oct 2014 18:38:15 -0700 Subject: [Python-ideas] Deprecate dunder functions from operator module In-Reply-To: <20141030005321.GC26866@ando.pearwood.info> References: <20141030005321.GC26866@ando.pearwood.info> Message-ID: <54519687.6030708@stoneleaf.us> On 10/29/2014 05:53 PM, Steven D'Aprano wrote: > > I propose a few things: > > * institute a policy that, in the event of a new function being added > to the operator module, only the dunderless version will be added; > > * change the documentation to make it clear that the dunderless > versions should be used, rather than merely being "convenience" > functions; > > * add a prominent note that the dunder versions exist for backwards > compatibility only and should not be used in new code. +1 -- ~Ethan~ From tjreedy at udel.edu Thu Oct 30 06:01:03 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 30 Oct 2014 01:01:03 -0400 Subject: [Python-ideas] Deprecate dunder functions from operator module In-Reply-To: <20141030005321.GC26866@ando.pearwood.info> References: <20141030005321.GC26866@ando.pearwood.info> Message-ID: On 10/29/2014 8:53 PM, Steven D'Aprano wrote: > (For the avoidance of all doubt, the following has nothing to do with > dunder methods like int.__add__ etc., but only the functions in the > operator module.) > > The operator module contains various function versions of operators, > e.g. operator.add for + , operator.sub for - , etc. > > There are also dunder versions of each function, and according to the > documentation: > > The function names are those used for special class methods; > variants without leading and trailing __ are also provided for > convenience. > > https://docs.python.org/3/library/operator.html > > (except that's not quite right, because there are no functions __radd__ > etc. matching the special class methods). > > The existence of redundant dunder functions came as a complete surprise > to me when I recently discovered it, and I daresay it will come as a > surprise to many people. Having two versions in the operator module has > been known to lead to confusion, e.g.: > > https://mail.python.org/pipermail/tutor/2014-October/102877.html > > All the third party documentation I've seen for the operator module > refers to the dunderless versions. > > I doubt that there is anyone who prefers to write operator.__add__ > instead of operator.add, and Terry Reedy has kindly determined that > (aside from tests) the standard library doesn't use any of the dunder > versions. I only determined before that 'operator.__' is never used except for 6 tests. I just grepped Lib/*.py for 'from operator import' and none of the 28 line imports a dunder version. > https://mail.python.org/pipermail/python-list/2014-October/679226.html > > > I propose a few things: > > * institute a policy that, in the event of a new function being added > to the operator module, only the dunderless version will be added; > > * change the documentation to make it clear that the dunderless > versions should be used, rather than merely being "convenience" > functions; > > * add a prominent note that the dunder versions exist for backwards > compatibility only and should not be used in new code. > > At this point, I do NOT propose having the dunder versions of the > functions raise a depreciation warning. I would be content for them to > merely be documented as discouraged, and defer actual depreciation until > Python 5000 or so. (But if there is a strong preference for actual > deprecation, I won't argue against it.) > > Thoughts? I think the dunder versions should at least be deprecated in the doc, and possibly in the functions also. Could we also augment 2to3. I went to https://code.openhub.net/search and searched for "from operator import" and in the first 160 Python hits, there were no dunders. (Anyone else could continue checking from page 17.) Not counting 'attrgetter' and 'itemgetter' hits might leave 100 non-dunder imports. "from operator import mul" has 150 hits, while same with "__mul__" has one hit in a 'pythran' test/test_operator.py file. For 'add' and '__add__', the numbers are 330 and 8. For operator.add and operator.__add__, dunder use is more common: 2079 versus 187, but still under 10% -- Terry Jan Reedy From antony.lee at berkeley.edu Thu Oct 30 09:07:26 2014 From: antony.lee at berkeley.edu (Antony Lee) Date: Thu, 30 Oct 2014 01:07:26 -0700 Subject: [Python-ideas] operator.call / operator.__call__ In-Reply-To: <20141030005705.GD26866@ando.pearwood.info> References: <20141030005705.GD26866@ando.pearwood.info> Message-ID: I have two use cases below. Yes, it is true that just defining the "call" function myself would only take me two lines, but on the other hand the same is true for most functions in the operator module... - Sometimes, depending on a switch, you want either to call a function directly, or to pass it to a specialized "caller", e.g. def run_in_thread(func, *args, **kwargs): Thread(target=func, args=args, kwargs=kwargs).start() def run_here_or_in_thread(in_thread, func, *args, **kwargs): (run_in_thread if in_thread else operator.call)(func, *args, **kwargs) or to optionally use a ContextDecorator: def run_in_ctx(ctx, func, *args, **kwargs): (ctx or operator.call)(func) - In Qt programs (although I assume similar ideas apply to other GUI toolkits), where functions that operate on the GUI can only be called from the main thread, but "signals" (in the Qt sense of the term) can be used to communicate between threads, I find it useful to define a "call this in the GUI thread" generic signal. The implementation is essentially class Main(QMainWindow): any_callable_signal = pyqtSignal(object) def __init__(self, *args, **kwargs): <...> self.any_callable_signal.connect(lambda f: f()) # <- operator.call seems to express this better. def call_in_gui_thread(self, func): self.any_callable_signal.emit(func) Antony 2014-10-29 17:57 GMT-07:00 Steven D'Aprano : > On Wed, Oct 29, 2014 at 03:39:59PM -0700, Antony Lee wrote: > > A simple suggestion: add "operator.call" and "operator.__call__", which > > would provide a function like Python2's "apply" (but it'd be called with > > already unpacked arguments, i.e. operator.call(f, *args, **kwargs)). > > Why? To be able to pass it as an argument to other function, just like > the > > other functions defined in the "operator" module. > > I think you want something like this? > > def call(callable, *args, **kwargs): > return callable(*args, **kwargs) > > > Can you give an example of how you might use this call method? It > doesn't have to be a real world use-case, just an illustration. > > Perhaps I'm being a bit dim-witted today, but I'm having trouble > thinking of where I would use this operator.call rather than just > directly applying *args, **kwargs to the callable. > > > -- > Steven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From j.wielicki at sotecware.net Thu Oct 30 09:23:58 2014 From: j.wielicki at sotecware.net (Jonas Wielicki) Date: Thu, 30 Oct 2014 09:23:58 +0100 Subject: [Python-ideas] operator.call / operator.__call__ In-Reply-To: References: <20141030005705.GD26866@ando.pearwood.info> Message-ID: <5451F59E.30608@sotecware.net> (only referring to the second use case here) On 30.10.2014 09:07, Antony Lee wrote: > [snip] > - In Qt programs (although I assume similar ideas apply to other GUI > toolkits), where functions that operate on the GUI can only be called from > the main thread, but "signals" (in the Qt sense of the term) can be used to > communicate between threads, I find it useful to define a "call this in the > GUI thread" generic signal. The implementation is essentially > > class Main(QMainWindow): > any_callable_signal = pyqtSignal(object) > def __init__(self, *args, **kwargs): > <...> > self.any_callable_signal.connect(lambda f: f()) # <- operator.call > seems to express this better. > def call_in_gui_thread(self, func): > self.any_callable_signal.emit(func) > How is that different from directly passing f? Or use functools.partial in case you need to pass additional, fixed arguments to f. regards, jwi > Antony > > 2014-10-29 17:57 GMT-07:00 Steven D'Aprano : > >> On Wed, Oct 29, 2014 at 03:39:59PM -0700, Antony Lee wrote: >>> A simple suggestion: add "operator.call" and "operator.__call__", which >>> would provide a function like Python2's "apply" (but it'd be called with >>> already unpacked arguments, i.e. operator.call(f, *args, **kwargs)). >>> Why? To be able to pass it as an argument to other function, just like >> the >>> other functions defined in the "operator" module. >> >> I think you want something like this? >> >> def call(callable, *args, **kwargs): >> return callable(*args, **kwargs) >> >> >> Can you give an example of how you might use this call method? It >> doesn't have to be a real world use-case, just an illustration. >> >> Perhaps I'm being a bit dim-witted today, but I'm having trouble >> thinking of where I would use this operator.call rather than just >> directly applying *args, **kwargs to the callable. >> >> >> -- >> Steven >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From abarnert at yahoo.com Thu Oct 30 10:03:59 2014 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 30 Oct 2014 02:03:59 -0700 Subject: [Python-ideas] operator.call / operator.__call__ In-Reply-To: <5451F59E.30608@sotecware.net> References: <20141030005705.GD26866@ando.pearwood.info> <5451F59E.30608@sotecware.net> Message-ID: <773DC4E0-8DEB-4589-97F5-727FA931B81F@yahoo.com> On Oct 30, 2014, at 1:23, Jonas Wielicki wrote: > (only referring to the second use case here) > > On 30.10.2014 09:07, Antony Lee wrote: >> [snip] >> self.any_callable_signal.connect(lambda f: f()) # <- operator.call >> seems to express this better. >> def call_in_gui_thread(self, func): >> self.any_callable_signal.emit(func) > > How is that different from directly passing f? Or use functools.partial > in case you need to pass additional, fixed arguments to f. He's passing a function that takes any function f and calls it. That's not the same as passing any particular function. Maybe it's more obvious if you compare: lambda f: f() lambda: f() From j.wielicki at sotecware.net Thu Oct 30 10:08:10 2014 From: j.wielicki at sotecware.net (Jonas Wielicki) Date: Thu, 30 Oct 2014 10:08:10 +0100 Subject: [Python-ideas] operator.call / operator.__call__ In-Reply-To: <773DC4E0-8DEB-4589-97F5-727FA931B81F@yahoo.com> References: <20141030005705.GD26866@ando.pearwood.info> <5451F59E.30608@sotecware.net> <773DC4E0-8DEB-4589-97F5-727FA931B81F@yahoo.com> Message-ID: <5451FFFA.20409@sotecware.net> On 30.10.2014 10:03, Andrew Barnert wrote: > On Oct 30, 2014, at 1:23, Jonas Wielicki wrote: > >> (only referring to the second use case here) >> >> On 30.10.2014 09:07, Antony Lee wrote: >>> [snip] >>> self.any_callable_signal.connect(lambda f: f()) # <- operator.call >>> seems to express this better. >>> def call_in_gui_thread(self, func): >>> self.any_callable_signal.emit(func) >> >> How is that different from directly passing f? Or use functools.partial >> in case you need to pass additional, fixed arguments to f. > > He's passing a function that takes any function f and calls it. That's not the same as passing any particular function. > > Maybe it's more obvious if you compare: > > lambda f: f() > lambda: f() > Right, I missed that, sorry for the noise. regards, jwi From javidcf at gmail.com Thu Oct 30 16:03:41 2014 From: javidcf at gmail.com (Javier Dehesa) Date: Thu, 30 Oct 2014 16:03:41 +0100 Subject: [Python-ideas] "or raise" syntax Message-ID: <5452534D.5060002@gmail.com> This happens to me with some frequency: result = f(args) if not result: # of "if result is None:" raise Exception(...) What if I could just say? result = f(args) or raise Exception(...) I think that's pretty idiomatic. What's more, this would give you a shorter syntax for things like: if x <= y: raise Exception(...) Which you could write simply as: x > y or raise Exception(...) Effectively covering the use case of this proposal http://article.gmane.org/gmane.comp.python.ideas/26005/ Has this ever been considered? Javier From ethan at stoneleaf.us Thu Oct 30 16:13:42 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 30 Oct 2014 08:13:42 -0700 Subject: [Python-ideas] "or raise" syntax In-Reply-To: <5452534D.5060002@gmail.com> References: <5452534D.5060002@gmail.com> Message-ID: <545255A6.6000600@stoneleaf.us> On 10/30/2014 08:03 AM, Javier Dehesa wrote: > > This happens to me with some frequency: > > result = f(args) > if not result: # of "if result is None:" > raise Exception(...) > > What if I could just say? > > result = f(args) or raise Exception(...) Seems like a decent idea, but you can already have most of that: result = f(args) or raise_exc(ValueError, 'args must be ...') and then have 'raise_exc' do the exception raising work. -- ~Ethan~ From tjreedy at udel.edu Thu Oct 30 19:38:16 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 30 Oct 2014 14:38:16 -0400 Subject: [Python-ideas] operator.call / operator.__call__ In-Reply-To: References: <20141030005705.GD26866@ando.pearwood.info> Message-ID: On 10/30/2014 4:07 AM, Antony Lee wrote: > I have two use cases below. Yes, it is true that just defining the > "call" function myself would only take me two lines, but on the other > hand the same is true for most functions in the operator module... The primary intended use of operator.op functions is to be passed as arguments since operators themselves cannot be. By exposing C-coded wrappers, they make this possible more efficiently than would be the case with Python wrappers. There never seemed to be a need to pass 'apply' itself as an argument, and the newer of f(*args, **kwds) made its direct use unneeded. Hence it was deleted. The signature of apply is apply(func, arg[, keywords]). The arguments are not unpacked just to be repacked and unpacked again, as would happen with your renamed version of apply. > - Sometimes, depending on a switch, you want either to call a function > directly, or to pass it to a specialized "caller", e.g. Then either call it directly or pass it to a specialized caller. > def run_in_thread(func, *args, **kwargs): > Thread(target=func, args=args, kwargs=kwargs).start() In its use below, this would be more efficient as def run_in_thread(func, args, kwargs): > def run_here_or_in_thread(in_thread, func, *args, **kwargs): > (run_in_thread if in_thread else operator.call)(func, *args, **kwargs) With the altered sig, this can be written (run_in_thread(func, args, kwargs) if in_thread else func(*args, **kwargs)) -- Terry Jan Reedy From tjreedy at udel.edu Thu Oct 30 19:45:07 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 30 Oct 2014 14:45:07 -0400 Subject: [Python-ideas] "or raise" syntax In-Reply-To: <545255A6.6000600@stoneleaf.us> References: <5452534D.5060002@gmail.com> <545255A6.6000600@stoneleaf.us> Message-ID: On 10/30/2014 11:13 AM, Ethan Furman wrote: > On 10/30/2014 08:03 AM, Javier Dehesa wrote: >> >> This happens to me with some frequency: >> >> result = f(args) >> if not result: # of "if result is None:" >> raise Exception(...) >> >> What if I could just say? >> >> result = f(args) or raise Exception(...) > > Seems like a decent idea, but you can already have most of that: > > result = f(args) or raise_exc(ValueError, 'args must be ...') > > and then have 'raise_exc' do the exception raising work. No need to pass exception class and message separately instead of an exception instance. def raiser(exc): raise exc print(1 or raiser(ValueError('null value'))) print(0 or raiser(ValueError('null value'))) >>> 1 Traceback (most recent call last): File "c:\programs\python34\tem.py", line 5, in print(0 or raiser(ValueError('null value'))) File "c:\programs\python34\tem.py", line 2, in raiser raise exc ValueError: null value It does add another line to the trackback, but this is pretty minor. -- Terry Jan Reedy From bhill at ea.com Thu Oct 30 20:02:27 2014 From: bhill at ea.com (Hill, Bruce) Date: Thu, 30 Oct 2014 19:02:27 +0000 Subject: [Python-ideas] Improving memory usage in shared-key attribute dicts Message-ID: <984f1f8f4ece45fea45cea0d4425d1dd@BY1PR0701MB1126.namprd07.prod.outlook.com> Thanks to PEP 412: "Key-Sharing Dictionary", CPython attribute dictionaries can share keys between multiple instances, so the memory cost of new attribute dicts comes primarily from the values array. In the current implementation, the keys array and the values array are always kept to be the same size. This is done so that once the key's location in the key array has been tracked down, the same array offset can be used on the value array to find the value. Rather than storing values in a sparse array of the same size as the keys array, it would make more sense to store values in a compact array. When a dict uses key sharing, there is an unused field in the PyDictKeyEntry struct ("me_value"), which could be repurposed to hold an index into the value array (perhaps by converting "me_value" into a payload union in PyDictKeyEntry). Since the sparse arrays in dictobject.c never use more than (2n+1)/3 of their entries, this change would reduce the memory footprint of each shared-key dict by roughly 1/3 (or more) and also improve data locality. With the current code, the memory usage looks something like: On the class: keys = {NULL, PyDictKeyEntry(key="keyA", value=), NULL, PyDictKeyEntry(key="keyB", value=)} On every instance: values = {NULL, "value A", NULL, "value B"} In this example, 50% of the values array is holding NULL pointers. With the proposed changes, it would look more like: On the class: keys = {NULL, PyDictKeyEntry(key="keyA", value_index=0), NULL, PyDictKeyEntry(key="keyB", value_index=1)} On every instance: values = {"value A", "value B"} Here, the memory footprint of instance dicts is cut in half because none of the array is wasted. There's a few implementation details to work out (e.g. uncommon cases like deleting attributes in __init__), but does this seem like a good idea to anyone else? From steve at pearwood.info Thu Oct 30 23:42:21 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 31 Oct 2014 09:42:21 +1100 Subject: [Python-ideas] Improving memory usage in shared-key attribute dicts In-Reply-To: <984f1f8f4ece45fea45cea0d4425d1dd@BY1PR0701MB1126.namprd07.prod.outlook.com> References: <984f1f8f4ece45fea45cea0d4425d1dd@BY1PR0701MB1126.namprd07.prod.outlook.com> Message-ID: <20141030224220.GE26866@ando.pearwood.info> On Thu, Oct 30, 2014 at 07:02:27PM +0000, Hill, Bruce wrote: > Thanks to PEP 412: "Key-Sharing Dictionary", CPython attribute > dictionaries can share keys between multiple instances, so the memory > cost of new attribute dicts comes primarily from the values array. In > the current implementation, the keys array and the values array are > always kept to be the same size. This is done so that once the key's > location in the key array has been tracked down, the same array offset > can be used on the value array to find the value. > > Rather than storing values in a sparse array of the same size as the > keys array, it would make more sense to store values in a compact > array. When a dict uses key sharing, there is an unused field in the > PyDictKeyEntry struct ("me_value"), which could be repurposed to hold > an index into the value array (perhaps by converting "me_value" into a > payload union in PyDictKeyEntry). Since the sparse arrays in > dictobject.c never use more than (2n+1)/3 of their entries, this > change would reduce the memory footprint of each shared-key dict by > roughly 1/3 (or more) and also improve data locality. How does this compare to the "Alternate Implementation" described in PEP 412? http://python.org/dev/peps/pep-0412/#id17 -- Steven From ncoghlan at gmail.com Fri Oct 31 10:54:07 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 31 Oct 2014 19:54:07 +1000 Subject: [Python-ideas] Deprecate dunder functions from operator module In-Reply-To: <54519687.6030708@stoneleaf.us> References: <20141030005321.GC26866@ando.pearwood.info> <54519687.6030708@stoneleaf.us> Message-ID: On 30 October 2014 11:38, Ethan Furman wrote: > On 10/29/2014 05:53 PM, Steven D'Aprano wrote: >> >> >> I propose a few things: >> >> * institute a policy that, in the event of a new function being added >> to the operator module, only the dunderless version will be added; >> >> * change the documentation to make it clear that the dunderless >> versions should be used, rather than merely being "convenience" >> functions; >> >> * add a prominent note that the dunder versions exist for backwards >> compatibility only and should not be used in new code. > > > +1 +0 from me (I agree the dunder versions are likely pointless, but I also think they're pretty harmless). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From bhill at ea.com Fri Oct 31 01:00:29 2014 From: bhill at ea.com (Hill, Bruce) Date: Fri, 31 Oct 2014 00:00:29 +0000 Subject: [Python-ideas] Improving memory usage in shared-key attribute dicts Message-ID: On Thu Oct 30 23:42:21 CET 2014, Steven D'Aprano wrote: > How does this compare to the "Alternate Implementation" described in PEP 412? > > http://python.org/dev/peps/pep-0412/#id17 I feel pretty dumb--I didn't read the alternative implementation section. It would be identical. Thanks for catching that. I'm going to try implementing it to see how it pans out.