From ericsnowcurrently at gmail.com Tue Mar 1 17:47:16 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 1 Mar 2016 15:47:16 -0700 Subject: [Python-ideas] Prevent importing yourself? In-Reply-To: <56ABEACA.8000707@nedbatchelder.com> References: <56ABEACA.8000707@nedbatchelder.com> Message-ID: (Wow, I'm late to this thread.) FTR, the underlying cause is that a script's directory is prepended to sys.path. [1] Otherwise importing your script wouldn't be a problem nearly as often. This is something I've hoped we could address at some point. [2] Note that PEP 395 [3] aimed to help with a number of related issues and PEP 432 [4] may still help make it easier to sort out interpreter-startup-related matters like this. Personally I'm still in favor of deprecating "sys.path[0] auto-initialisation", but I'm not gonna hold my breath on that. :) -eric [1] https://hg.python.org/cpython/file/7e48300c7f3b/Modules/main.c#l251 https://hg.python.org/cpython/file/7e48300c7f3b/Lib/runpy.py#l258 [2] http://bugs.python.org/issue13475 [3] http://www.python.org/dev/peps/pep-0395/ [4] http://www.python.org/dev/peps/pep-0432/ On Fri, Jan 29, 2016 at 3:42 PM, Ned Batchelder wrote: > Hi, > > A common question we get in the #python IRC channel is, "I tried importing a > module, but I get an AttributeError trying to use the things it said it > provided." Turns out the beginner named their own file the same as the > module they were trying to use. > > That is, they want to try (for example) the "azure" package. So they make a > file called azure.py, and start with "import azure". The import succeeds, > but it has none of the contents the documentation claims, because they have > imported themselves. It's baffling, because they have used the exact > statements shown in the examples, but it doesn't work. > > Could we make this a more obvious failure? Is there ever a valid reason for > a file to import itself? Is this situation detectable in the import > machinery? > > --Ned. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ericsnowcurrently at gmail.com Tue Mar 1 19:03:58 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 1 Mar 2016 17:03:58 -0700 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: On Fri, Feb 5, 2016 at 2:20 PM, Martin Teichmann wrote: > 3. an ``__attribute_order__`` tuple is left in the class in order to inspect > the order in which attributes were defined. > > [snip] > > The third part of the proposal is to leave a tuple called > ``__attribute_order__`` in the class that contains the order in which > the attributes were defined. This is a very common usecase, many > libraries use an ``OrderedDict`` to store this order. This is a very > simple way to achieve the same goal. > > [snip] > > PEP 422 defined a generic way to add arbitrary namespaces for class > definitions. This approach is much more flexible than just leaving > the definition order in a tuple. The ``__prepare__`` method in a metaclass > supports exactly this behavior. But given that effectively > the only use cases that could be found out in the wild were the > ``OrderedDict`` way of determining the attribute order, it seemed > reasonable to only support this special case. > > The metaclass described in this PEP has been designed to be very simple > such that it could be reasonably made the default metaclass. This was > especially important when designing the attribute order functionality: > This was a highly demanded feature and has been enabled through the > ``__prepare__`` method of metaclasses. This method can be abused in > very weird ways, making it hard to correctly maintain this feature in > CPython. This is why it has been proposed to deprecated this feature, > and instead use ``OrderedDict`` as the standard namespace, supporting > the most important feature while dropping most of the complexity. But > this would have meant that ``OrderedDict`` becomes a language builtin > like dict and set, and not just a standard library class. The choice > of the ``__attribute_order__`` tuple is a much simpler solution to the > problem. Note that I've already been working on making OrderedDict the default class definition namespace type. [1] The patch to do so is actually super simple (if I remember correctly). This was actually one of my main motivations for re-implementing OrderedDict in C (the other was preserving the order of **kwargs). Also note that we're thinking along the same lines regarding __attribute_order__. :) In the tracker issue I referenced you'll see that I proposed roughly the same thing, calling it __definition_order__ (which I prefer as the spelling). I've brought up the idea a couple of times on this list and got a positive response (or so I remember ) and was still planning on pursuing the matter in time for Python 3.7. My point is: this part of your proposal stands on its own and is already being worked on (even if slowly). FYI, one problem with bundling related ideas in a PEP is that the concept of partial acceptance isn't a natural part of the PEP process. There have been a few PEPs where part of the proposal made the cut but didn't go anywhere because the whole PEP was rejected. Anyway, keep the above in mind as it has bearing on your PEP. -eric [1] http://bugs.python.org/issue24254 From lkb.teichmann at gmail.com Wed Mar 2 14:43:25 2016 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Wed, 2 Mar 2016 20:43:25 +0100 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: Hi Eric, Hi List, > Note that I've already been working on making OrderedDict the default > class definition namespace type. [1] The patch to do so is actually > super simple (if I remember correctly). This was actually one of my > main motivations for re-implementing OrderedDict in C (the other was > preserving the order of **kwargs). I vaguely remembered your work when I was writing my PEP, but couldn't find the discussion about it anymore, thanks for pointing me to it, I'll update that in my PEP. > Also note that we're thinking along the same lines regarding > __attribute_order__. :) In the tracker issue I referenced you'll see > that I proposed roughly the same thing, calling it > __definition_order__ (which I prefer as the spelling). I've brought > up the idea a couple of times on this list and got a positive response > (or so I remember ) and was still planning on pursuing the > matter in time for Python 3.7. I agree, __definition_order__ sounds better. I'm not a native speaker, so I need some help on naming things properly... > My point is: this part of your proposal stands on its own and is > already being worked on (even if slowly). FYI, one problem with > bundling related ideas in a PEP is that the concept of partial > acceptance isn't a natural part of the PEP process. There have been a > few PEPs where part of the proposal made the cut but didn't go > anywhere because the whole PEP was rejected. I am not very attached to the idea of __definition_order__, so if the general consensus here is to take it out, I will do that. In an old version of the PEP (i.e. PEP 422) there was a way to open the full __prepare__ scheme through the new metaclass. This was certainly bundling too many ideas in one PEP, so I took that out. I left the __definition_order__ in as it seemed a very simple idea to me, especially because once the new metaclass becomes the standard metaclass, this can be implemented without OrderedDict, and thus we don't have the need to make OrderedDict a builtin type. I'm not sure how the general feeling is here on the list: should I * take out the __definition_order__ and later do the same with an OrderedDict, * or should I leave that in because making OrderedDict a builtin is too much a burden? I'm looking forward to comments on that! Btw, how I want to proceed: I am currently looking how other projects could benefit from PEP 487 (beside my owns...), and got good feedback from IPython. Then I will implement a version for the standard library. As a reminder, there is already a version on PyPI: https://pypi.python.org/pypi/metaclass Greetings Martin From abedillon at gmail.com Wed Mar 2 15:01:29 2016 From: abedillon at gmail.com (Abe Dillon) Date: Wed, 2 Mar 2016 12:01:29 -0800 (PST) Subject: [Python-ideas] Thoughts on lambda expressions Message-ID: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> I'm new here, but I know that lambda expression syntax has probably been discussed to oblivion. I have searched previous posts without finding anything exactly like my idea so here it is: # instead of this hand = sorted(cards, key=lambda card: card.suit) # sort some iterable of cards by suit # do this hand = sorted(cards, key=(card.suit from card)) # sort some iterable of cards by suit # |<= you can stop reading here and still have a good sense of what the code does More generally, I think a superior syntax for lambda would be: ( from ) The reasons I believe that's a superior syntax are: a) In the vast majority of use cases for lambda expressions the call signature can be easily inferred (like in a key function), so moving it after the expression tends to be more readable. b) It doesn't use the esoteric name, 'lambda' which causes its own readability issues. c) It should be compatible with existing code: try: 1/0 except Exception as e: raise (ValueError() from e) # this is already a syntax error so implementing the proposal won't break working code # of this form. I'm not aware of any other uses of the 'from' keyword that would create # ambiguities I'm not sure if this should actually be implemented, but I wanted to share the idea anyway and get some feedback. In my opinion 'lambda' is one of the less elegant pieces of Python syntax not because it's restricted to a single expression, but because the name and syntax are both detrimental to readability. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Wed Mar 2 17:24:39 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Wed, 2 Mar 2016 17:24:39 -0500 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> Message-ID: <56D76827.4020200@sdamon.com> My objections to this are twofold: 1) It looks a lot like a lot of other 'structures' in python. It looks very much like a generator expression or a tuple, so it isn't immediately obvious to me what you are intending (if this was allowed syntax of course.) 2) While it is nice to type that over `lambda :`, typing lambda is no great burden, and that is already just sugar for a regular def statement. I don't think there is much of a great need for even more terse ways of writing an anonymous function, especially ones with the restrictions that python currently places on them (no statements). It just doesn't make a lot of sense to change the syntax at this point. But I think you knew that. On 3/2/2016 15:01, Abe Dillon wrote: > I'm new here, but I know that lambda expression syntax has probably > been discussed to oblivion. I have searched previous posts without > finding anything exactly like my idea so here it is: > > | > # instead of this > hand =sorted(cards,key=lambdacard:card.suit)# sort some iterable of > cards by suit > > # do this > hand =sorted(cards,key=(card.suit fromcard))# sort some iterable of > cards by suit > # |<= you can stop reading here and still have a good sense of > what the code does > | > > > More generally, I think a superior syntax for lambda would be: > > | > (from) > | > > The reasons I believe that's a superior syntax are: > > a) In the vast majority of use cases for lambda expressions the call > signature can be easily inferred (like in a key function), so moving > it after the expression tends to be more readable. > > b) It doesn't use the esoteric name, 'lambda' which causes its own > readability issues. > > c) It should be compatible with existing code: > > | > try: > 1/0 > exceptExceptionase: > raise(ValueError()frome)# this is already a syntax error so > implementing the proposal won't break working code > # of this form. I'm not aware of any other uses of the 'from' keyword > that would create > # ambiguities > | > > I'm not sure if this should actually be implemented, but I wanted to > share the idea anyway and get some feedback. In my opinion 'lambda' is > one of the less elegant pieces of Python syntax not because it's > restricted to a single expression, but because the name and syntax are > both detrimental to readability. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Mar 2 17:31:08 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 2 Mar 2016 15:31:08 -0700 Subject: [Python-ideas] Simpler Customization of Class Creation - PEP 487 In-Reply-To: References: Message-ID: On Wed, Mar 2, 2016 at 12:43 PM, Martin Teichmann wrote: > I agree, __definition_order__ sounds better. I'm not a native speaker, > so I need some help on naming things properly... No worries. Glad we agree. :) > I am not very attached to the idea of __definition_order__, so if the > general consensus here is to take it out, I will do that. I'm not suggesting that you take it out, necessarily. I just wanted to make sure you could make an informed decision. > I'm not sure how the general feeling is here on the list: should I > > * take out the __definition_order__ and later do the same > with an OrderedDict, I don't think there's any harm in leaving it in, regardless of what I'm trying to achieve. It would be sensible to reference issue24254 in the PEP as there's overlap (even if not in the implementation). > * or should I leave that in because making OrderedDict a > builtin is too much a burden? As of 3.6 OrderedDict *is* a builtin type (just not part of the builtins module). So there's zero burden. :) > Btw, how I want to proceed: I am currently looking how other projects > could benefit from PEP 487 (beside my owns...), and got good feedback > from IPython. Then I will implement a version for the standard library. Good plan. > As a reminder, there is already a version on PyPI: > > https://pypi.python.org/pypi/metaclass Cool. I'll take a look. -eric From tritium-list at sdamon.com Wed Mar 2 17:38:57 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Wed, 2 Mar 2016 17:38:57 -0500 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> Message-ID: <56D76B81.1030906@sdamon.com> I hit send before my thought was really complete. I don't know if that is a contrived example or not, but python also already provides a more efficient and readable way of pulling attributes off of an object (a very common misuse of lambda/anonymous functions). |operator.||attrgetter, and it's cousin operator.itemgetter for item access. So for your example code, there is already a cleaner way. I don't know if that is your main use case. If it is not, then perhaps a better example could be provided where this would prove superior? | On 3/2/2016 15:01, Abe Dillon wrote: > I'm new here, but I know that lambda expression syntax has probably > been discussed to oblivion. I have searched previous posts without > finding anything exactly like my idea so here it is: > > | > # instead of this > hand =sorted(cards,key=lambdacard:card.suit)# sort some iterable of > cards by suit > > # do this > hand =sorted(cards,key=(card.suit fromcard))# sort some iterable of > cards by suit > # |<= you can stop reading here and still have a good sense of > what the code does > | > > > More generally, I think a superior syntax for lambda would be: > > | > (from) > | > > The reasons I believe that's a superior syntax are: > > a) In the vast majority of use cases for lambda expressions the call > signature can be easily inferred (like in a key function), so moving > it after the expression tends to be more readable. > > b) It doesn't use the esoteric name, 'lambda' which causes its own > readability issues. > > c) It should be compatible with existing code: > > | > try: > 1/0 > exceptExceptionase: > raise(ValueError()frome)# this is already a syntax error so > implementing the proposal won't break working code > # of this form. I'm not aware of any other uses of the 'from' keyword > that would create > # ambiguities > | > > I'm not sure if this should actually be implemented, but I wanted to > share the idea anyway and get some feedback. In my opinion 'lambda' is > one of the less elegant pieces of Python syntax not because it's > restricted to a single expression, but because the name and syntax are > both detrimental to readability. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Mar 2 17:46:02 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 02 Mar 2016 14:46:02 -0800 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> Message-ID: <56D76D2A.5080107@stoneleaf.us> On 03/02/2016 12:01 PM, Abe Dillon wrote: > More generally, I think a superior syntax for lambda would be: > > ( from ) > > The reasons I believe that's a superior syntax are: > > a) In the vast majority of use cases for lambda expressions the call > signature can be easily inferred (like in a key function), so moving it > after the expression tends to be more readable. And what does it look like when you have more than one paramater in the signature and/or something beside simple attribute lookup? 'open': lambda s, cr, uid, rec, ctx: rec['state'] == 'draft', would instead be: 'open': rec['state'] == 'draft' from (s, cr, uid, rec, ctx), Ouch. That just went from bad to horrid. > b) It doesn't use the esoteric name, 'lambda' which causes its own > readability issues. On the contrary: 'lambda' lets you know immediately what you're dealing with. The syntax you are suggesting looks like: - a (wrong) generator - taking ... items? ... from some kind of container To be fair, it looked like an interesting syntax at first glance, but deeper investigation shows serious drawbacks. -- ~Ethan~ From egregius313 at gmail.com Wed Mar 2 18:02:35 2016 From: egregius313 at gmail.com (Ed Minnix) Date: Wed, 2 Mar 2016 18:02:35 -0500 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <56D76D2A.5080107@stoneleaf.us> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> Message-ID: Hi, I would just like to add a side note on this: The library fn.py implements a class called _Callable, which gives a shorter notation for lambdas, using an underscore ( _ ), instead of declaring parameters (for example, map(_ + 1, range(10)) is the same as map(lambda n: n + 1, range(10)) ). In addition to the regular arithmetic operators, it supports the __getattr__ method. (for instance _.y is the same as lambda a: a.y) Therefore, you do not override the general syntax of Python (other than the fact that you cannot use the code for i, _ in some_iterable: # don?t use, at the end of the loop, fn._ has disappeared from the scope of the module do_something(i) ) Personally, I would propose the adoption of the _ in some standard library module (e.g., functools) rather than overriding the ?from? syntax if the simplification of lambdas in a goal. (Personal I find the _ much more user-friendly) - Ed M > On Mar 2, 2016, at 5:46 PM, Ethan Furman wrote: > > On 03/02/2016 12:01 PM, Abe Dillon wrote: > >> More generally, I think a superior syntax for lambda would be: >> >> ( from ) >> >> The reasons I believe that's a superior syntax are: >> >> a) In the vast majority of use cases for lambda expressions the call >> signature can be easily inferred (like in a key function), so moving it >> after the expression tends to be more readable. > > And what does it look like when you have more than one paramater in the signature and/or something beside simple attribute lookup? > > 'open': lambda s, cr, uid, rec, ctx: rec['state'] == 'draft', > > would instead be: > > 'open': rec['state'] == 'draft' from (s, cr, uid, rec, ctx), > > Ouch. That just went from bad to horrid. > >> b) It doesn't use the esoteric name, 'lambda' which causes its own >> readability issues. > > On the contrary: 'lambda' lets you know immediately what you're dealing with. The syntax you are suggesting looks like: > > - a (wrong) generator > - taking ... items? ... from some kind of container > > To be fair, it looked like an interesting syntax at first glance, but deeper investigation shows serious drawbacks. > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From p.f.moore at gmail.com Wed Mar 2 18:03:45 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 2 Mar 2016 23:03:45 +0000 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <56D76D2A.5080107@stoneleaf.us> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> Message-ID: On 2 March 2016 at 22:46, Ethan Furman wrote: >> a) In the vast majority of use cases for lambda expressions the call >> signature can be easily inferred (like in a key function), so moving it >> after the expression tends to be more readable. > > > And what does it look like when you have more than one paramater in the > signature and/or something beside simple attribute lookup? > > 'open': lambda s, cr, uid, rec, ctx: rec['state'] == 'draft', > > would instead be: > > 'open': rec['state'] == 'draft' from (s, cr, uid, rec, ctx), > > Ouch. That just went from bad to horrid. I actually find the second case more readable, because the expression comes first. Having said that, I don't particularly like the proposal, but it's not really on grounds of readability. Or, for that matter, because it looks like a generator. My problem with the proposal is mainly that trying to cram too much into one expression is nearly always a mistake in Python. The language isn't expression-based and long, over-complex expressions are hard to work with and reason about. Splitting things up into smaller named subexpressions, or using conditional statements and loops rather than conditional expressions and comprehensions, nearly always makes things more readable (unless, like me, you're really bad at naming and end up with bits called e1, e2, e3... :-)) So a "nicer syntax for lambda" isn't really that helpful - we have one already, it's called "def" :-) For background, I've been working recently with a library that focuses strongly on providing a language for building complex expressions. The functionality is really powerful, but oh, boy, it's a struggle to express my intention in a single expression. So the basic problem for me is that the proposal doesn't offer huge benefits, and it's solving a problem that people should probably try to avoid in the first place. Paul PS For the record, I dislike the lambda syntax intensely - the odd keyword, the fact that "lambda" is sometimes almost as long as than the expression I'm using, the use of the mid-line colon which is hard to spot, ... So in principle I'm inclined to support "improvement" proposals. From mal at egenix.com Wed Mar 2 18:07:06 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 3 Mar 2016 00:07:06 +0100 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> Message-ID: <56D7721A.6040705@egenix.com> On 02.03.2016 21:01, Abe Dillon wrote: > I'm new here, but I know that lambda expression syntax has probably been > discussed to oblivion. I have searched previous posts without finding > anything exactly like my idea so here it is: > > # instead of this > hand = sorted(cards, key=lambda card: card.suit) # sort some iterable of > cards by suit > > # do this > hand = sorted(cards, key=(card.suit from card)) # sort some iterable of > cards by suit > # |<= you can stop reading here and still > have a good sense of what the code does Why not this ? import operator hand = sorted(cards, key=operator.attrgetter('suit')) # For the curious: # https://docs.python.org/3.5/library/operator.html#operator.attrgetter -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Mar 03 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ 2016-02-19: Released eGenix PyRun 2.1.2 ... http://egenix.com/go88 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From grant.jenks at gmail.com Wed Mar 2 18:14:02 2016 From: grant.jenks at gmail.com (Grant Jenks) Date: Wed, 2 Mar 2016 15:14:02 -0800 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> Message-ID: On Wed, Mar 2, 2016 at 3:02 PM, Ed Minnix wrote: > Hi, I would just like to add a side note on this: > > The library fn.py implements a class called _Callable, which gives a > shorter notation for lambdas, using an underscore ( _ ), instead of > declaring parameters (for example, map(_ + 1, range(10)) is the same as > map(lambda n: n + 1, range(10)) ). In addition to the regular arithmetic > operators, it supports the __getattr__ method. (for instance _.y is the > same as lambda a: a.y) > > Therefore, you do not override the general syntax of Python (other than > the fact that you cannot use the code > > for i, _ in some_iterable: # don?t use, at the end of the > loop, fn._ has disappeared from the scope of the module > do_something(i) > > ) > > Personally, I would propose the adoption of the _ in some standard library > module (e.g., functools) rather than overriding the ?from? syntax if the > simplification of lambdas in a goal. (Personal I find the _ much more > user-friendly) > > - Ed M +1 for fn.py's underscore functionality. But underscore already has a purpose as a placeholder and in the repl so I think it's a poor choice. In Python 3, you can actually do: from fn import _ as ? print(sorted(cards, key=?.suit)) But that is hard to type on most keyboards. What if we allowed question marks in identifiers and used ``?.suit`` ? Grant -------------- next part -------------- An HTML attachment was scrubbed... URL: From egregius313 at gmail.com Wed Mar 2 18:51:25 2016 From: egregius313 at gmail.com (Ed Minnix) Date: Wed, 2 Mar 2016 18:51:25 -0500 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> Message-ID: <592793ED-969D-4687-8D91-473C3F77D658@gmail.com> Or, for those whose keyboards don?t allow them to efficiently type Unicode, something like a keyword ``it`` would be a good alternative. - Ed > On Mar 2, 2016, at 6:14 PM, Grant Jenks wrote: > > On Wed, Mar 2, 2016 at 3:02 PM, Ed Minnix > wrote: > Hi, I would just like to add a side note on this: > > The library fn.py implements a class called _Callable, which gives a shorter notation for lambdas, using an underscore ( _ ), instead of declaring parameters (for example, map(_ + 1, range(10)) is the same as map(lambda n: n + 1, range(10)) ). In addition to the regular arithmetic operators, it supports the __getattr__ method. (for instance _.y is the same as lambda a: a.y) > > Therefore, you do not override the general syntax of Python (other than the fact that you cannot use the code > > for i, _ in some_iterable: # don?t use, at the end of the loop, fn._ has disappeared from the scope of the module > do_something(i) > > ) > > Personally, I would propose the adoption of the _ in some standard library module (e.g., functools) rather than overriding the ?from? syntax if the simplification of lambdas in a goal. (Personal I find the _ much more user-friendly) > > - Ed M > > +1 for fn.py's underscore functionality. > > But underscore already has a purpose as a placeholder and in the repl so I think it's a poor choice. In Python 3, you can actually do: > > from fn import _ as ? > print(sorted(cards, key=?.suit)) > > But that is hard to type on most keyboards. What if we allowed question marks in identifiers and used ``?.suit`` ? > > Grant > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abedillon at gmail.com Wed Mar 2 19:02:33 2016 From: abedillon at gmail.com (Abe Dillon) Date: Wed, 2 Mar 2016 16:02:33 -0800 (PST) Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <56D76B81.1030906@sdamon.com> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76B81.1030906@sdamon.com> Message-ID: <9562d3ff-313b-4993-98e1-fbef125d6231@googlegroups.com> > I don't know if that is a contrived example or not, but python also > already provides a more efficient and readable way of pulling attributes > off of an object (a very common misuse of lambda/anonymous functions). > operator.attrgetter > My examples are fairly contrived. I know about the operator module and functools.partial -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at mgmiller.net Wed Mar 2 19:00:46 2016 From: python-ideas at mgmiller.net (Mike Miller) Date: Wed, 2 Mar 2016 16:00:46 -0800 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> Message-ID: <56D77EAE.6050805@mgmiller.net> On 2016-03-02 15:03, Paul Moore wrote: > PS For the record, I dislike the lambda syntax intensely - the odd > keyword, the fact that "lambda" is sometimes almost as long as than > the expression I'm using, the use of the mid-line colon which is hard > to spot, ... So in principle I'm inclined to support "improvement" Hmm, can't think of a way to get rid of the colon without it looking like a generator, but perhaps we could tackle the other issues by letting "def" stand in for "lambda": lambda x: x.y def x: x.y It's shorter, not an esoteric word, and is analogous to a named function definition, reminiscent of javascript. The lack of name/parens separates it from the standard form. Doable, but of course its pretty late in the game to be changing lambda, perhaps in Python 4? -Mike From rosuav at gmail.com Wed Mar 2 19:07:08 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 3 Mar 2016 11:07:08 +1100 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <56D77EAE.6050805@mgmiller.net> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> <56D77EAE.6050805@mgmiller.net> Message-ID: On Thu, Mar 3, 2016 at 11:00 AM, Mike Miller wrote: > Hmm, can't think of a way to get rid of the colon without it looking like a > generator, but perhaps we could tackle the other issues by letting "def" > stand in for "lambda": > > lambda x: x.y > > def x: x.y > > It's shorter, not an esoteric word, and is analogous to a named function > definition, reminiscent of javascript. The lack of name/parens separates it > from the standard form. Doable, but of course its pretty late in the game > to be changing lambda, perhaps in Python 4? T From egregius313 at gmail.com Wed Mar 2 19:09:44 2016 From: egregius313 at gmail.com (Ed Minnix) Date: Wed, 2 Mar 2016 19:09:44 -0500 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <56D77EAE.6050805@mgmiller.net> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> <56D77EAE.6050805@mgmiller.net> Message-ID: <34F20643-3C62-4F9F-859A-E69C6CECC7F9@gmail.com> Perhaps a better keyword like ``fun`` (in F#, for instance). Giving ``def`` a new meaning would introduce ambiguities, which would be more trouble than the shorter syntax is worth. - Ed > On Mar 2, 2016, at 7:00 PM, Mike Miller wrote: > > > On 2016-03-02 15:03, Paul Moore wrote: >> PS For the record, I dislike the lambda syntax intensely - the odd >> keyword, the fact that "lambda" is sometimes almost as long as than >> the expression I'm using, the use of the mid-line colon which is hard >> to spot, ... So in principle I'm inclined to support "improvement" > > Hmm, can't think of a way to get rid of the colon without it looking like a generator, but perhaps we could tackle the other issues by letting "def" stand in for "lambda": > > lambda x: x.y > > def x: x.y > > It's shorter, not an esoteric word, and is analogous to a named function definition, reminiscent of javascript. The lack of name/parens separates it from the standard form. Doable, but of course its pretty late in the game to be changing lambda, perhaps in Python 4? > > -Mike > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From abedillon at gmail.com Wed Mar 2 19:01:13 2016 From: abedillon at gmail.com (Abe Dillon) Date: Wed, 2 Mar 2016 16:01:13 -0800 (PST) Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <56D76827.4020200@sdamon.com> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76827.4020200@sdamon.com> Message-ID: > > My objections to this are twofold: > > 1) It looks a lot like a lot of other 'structures' in python. It > looks very much like a generator expression or a tuple, so it isn't > immediately obvious to me what you are intending (if this was allowed > syntax of course.) > I rarely find myself wondering if something is a function call or a tuple or a generator expression or just an expression within '()' to promote order of operations. I don't see why this use case would all of a sudden make visual parsing all that more difficult, especially when there are often obvious context hints leading up to a lambda expression: most_distant_word = max(words, key=(edit_distance(word, "hello") from word)) distant_words = filter((edit_distance(word, most_distant_word) < 5 from word ), words) shortest_distant_word = min(distant_words, key=(len(word.strip()) from word )) lambda statements tend to be most expressive when used in an expected context (like as the 'key' argument to the sorted function or as the first argument to the filter function. 2) While it is nice to type that over `lambda :`, typing lambda is no > great burden, and that is already just sugar for a regular def statement. > I don't think there is much of a great need for even more terse ways of > writing an anonymous function, especially ones with the restrictions that > python currently places on them (no statements). It just doesn't make a > lot of sense to change the syntax at this point. But I think you knew that. > Typing 'lambda' is not at all my concern, in fact `(x+1 from x)` takes the same number of keystrokes as `lambda x:x+1`. My complaint is in readability which is supposed to be Python's strong suit. To most people, 'lambda' is pretty much a nonsense word. It might as well be 'quaple'. Would you be ok with writing: sorted(words, key=quaple word: edit_distance(word, "hello")) or would you rather write: sorted(words, key=(edit_distance(word, "hello") from word)) Most Python constructs are elegantly readable: with lock: mutate_shared_data() if any(thing.is_metal for thing in pockets): alarm.sound() You can write code that doesn't look like a mess of obscure symbols and esoteric words *except* for lambda expressions. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Mar 2 19:17:28 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 3 Mar 2016 11:17:28 +1100 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <56D77EAE.6050805@mgmiller.net> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> <56D77EAE.6050805@mgmiller.net> Message-ID: On Thu, Mar 3, 2016 at 11:00 AM, Mike Miller wrote: > Hmm, can't think of a way to get rid of the colon without it looking like a > generator, but perhaps we could tackle the other issues by letting "def" > stand in for "lambda": > > lambda x: x.y > > def x: x.y > > It's shorter, not an esoteric word, and is analogous to a named function > definition, reminiscent of javascript. The lack of name/parens separates it > from the standard form. Doable, but of course its pretty late in the game > to be changing lambda, perhaps in Python 4? (Wow, I suck with the misclick. Or maybe this is a Zen koan - one letter of response that represents a wealth of wisdom and insight. Or not.) Trouble with this is that there'd be two very similar-looking, but syntactically disparate, constructs: # Define a function called x that calls another function # with itself as a parameter, and discards the return # value, always returning None. def x(): y(x) # Define and then dispose of an anonymous function # which takes a parameter and would return the # result of calling y on that object. def x: y(x) One of them is a statement, the other an expression. I'm not sure if the parser could handle that or not, but a lot of humans will have trouble with it. The similarities between these constructs will mean that a typo could easily flip the interpretation to the other, resulting in a bizarre error message. Consider what happens when you miss off a parenthesis: def function1(name): print("Hello, %s!".format(name) def function2(name): print("Goodbye, %s!".format(name)) Currently, the presence of the keyword 'def' inside an expression (the not-yet-closed print call) is an immediate error. But if "def" can introduce an anonymous function, the reported error might be a complaint about parentheses, or it might be to do with having no operator between the string method call and the function; depending on the exact syntax being used, this could result in extremely confusing errors. Much simpler if statements and expressions have a bit more distinction. And yes, I'm aware of the ternary conditional operator; and I don't think it sets a good precedent here. Comprehensions and genexps at least require bracketing, although they can run into the same oddities: print(x # oops, missed close bracket for x in y: # hey, what's that colon doing there? But I think this example is a smidge contrived. ChrisA From abedillon at gmail.com Wed Mar 2 19:19:42 2016 From: abedillon at gmail.com (Abe Dillon) Date: Wed, 2 Mar 2016 16:19:42 -0800 (PST) Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <56D76D2A.5080107@stoneleaf.us> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> Message-ID: On Wednesday, March 2, 2016 at 4:45:22 PM UTC-6, Ethan Furman wrote: > > what does it look like when you have more than one paramater in the > signature and/or something beside simple attribute lookup? The signature follows the exact same rules as a lambda signature: (func(*args, *kwargs) + offset from func, *args, offset, **kwargs) # vs. lambda func, *args, offset, **kwargs: func(*args, **kwargs) + offset Those both look bad because the expressiveness of lambdas loses its charm when the signature gets long and complicated. 'open': lambda s, cr, uid, rec, ctx: rec['state'] == 'draft', > > would instead be: > > 'open': rec['state'] == 'draft' from (s, cr, uid, rec, ctx), > > Ouch. That just went from bad to horrid. > Not only did you get the syntax wrong, but the lambda version is also horrid which is exacerbated by the fact that it looks like your trying to give the function a name or otherwise store it which defeats whole purpose of a lambda. At that point just use 'def' and stop trying to use lambda where it is ill suited. > > > b) It doesn't use the esoteric name, 'lambda' which causes its own > > readability issues. > > On the contrary: 'lambda' lets you know immediately what you're dealing > with. The syntax you are suggesting looks like: > > - a (wrong) generator > - taking ... items? ... from some kind of container > the idea is from the recipe metaphor for a function: def (): ...instructions to cook ingredients vs. ( from ) It's fair to quibble over the exact implementation (maybe use 'with' instead of 'from') but the main point of the syntax it to put the important bit (i.e.the expression) in front of the (usually) unimportant bit (i.e. the signature) and to swap out an esoteric word (lambda) with something that continues the readability emphasis of Python by using more common words. > > To be fair, it looked like an interesting syntax at first glance, but > deeper investigation shows serious drawbacks. > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python... at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Mar 2 19:27:30 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 3 Mar 2016 11:27:30 +1100 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76827.4020200@sdamon.com> Message-ID: On Thu, Mar 3, 2016 at 11:01 AM, Abe Dillon wrote: > Typing 'lambda' is not at all my concern, in fact `(x+1 from x)` takes the > same number of keystrokes as `lambda x:x+1`. My complaint is in readability > which is supposed to be Python's strong suit. To most people, 'lambda' is > pretty much a nonsense word. It might as well be 'quaple'. Would you be ok > with writing: > > sorted(words, key=quaple word: edit_distance(word, "hello")) There's a difference between nonsense and jargon, though. Words used in programming languages are most often borrowed from a parent language (in Python's case, that's usually either English or algebra), but they don't always retain their exact meanings. Simple words like "if" and "while" are very close to their originals, but "def" (aka "define") has a very broad meaning in English, and doesn't necessarily have to mean "create a function" as it does in Python. No matter what your background is, you'll need to learn at least some of the syntax. Most of us probably learned what + means in grade school; but in algebra, "ab" means "a multiplied by b", whereas Python (like most other programming languages) allows variables to have multiple letters in their names, and spells multiplication with an asterisk. If you have to learn that "quaple" or "lambda" means "anonymous function", is that really worse than learning that " from " means "anonymous function"? ChrisA From abedillon at gmail.com Wed Mar 2 19:37:30 2016 From: abedillon at gmail.com (Abe Dillon) Date: Wed, 2 Mar 2016 16:37:30 -0800 (PST) Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> Message-ID: <44d48cb9-846e-4376-a2af-43c1c0cba1f0@googlegroups.com> On Wednesday, March 2, 2016 at 5:04:10 PM UTC-6, Paul Moore wrote: > > So the basic problem for me is that the proposal doesn't offer huge > benefits, and it's solving a problem that people should probably try > to avoid in the first place. > Yes, that's why I said, "I'm not sure if this should actually be implemented". People talk a lot about the 'expressiveness' of lambda statements, but the situations where they're actually more expressive than obfuscating are very rare and usually involve some idiom that makes the signature definition all but vestigial. filter(lambda num: num > 3, nums) # all that boiler plate before the part we really care about: filter(num > 3, nums) This is a minor improvement suggestion with some insight I've gleaned on code readability (putting the meat before the potatoes often improves readability, as does using more common english words). -------------- next part -------------- An HTML attachment was scrubbed... URL: From sjoerdjob at sjec.nl Wed Mar 2 19:46:27 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Thu, 3 Mar 2016 01:46:27 +0100 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <34F20643-3C62-4F9F-859A-E69C6CECC7F9@gmail.com> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> <56D77EAE.6050805@mgmiller.net> <34F20643-3C62-4F9F-859A-E69C6CECC7F9@gmail.com> Message-ID: <20160303004627.GB17147@sjoerdjob.com> On Wed, Mar 02, 2016 at 07:09:44PM -0500, Ed Minnix wrote: > Perhaps a better keyword like ``fun`` (in F#, for instance). > > Giving ``def`` a new meaning would introduce ambiguities, which would be more trouble than the shorter syntax is worth. > > - Ed > Out of curiosity, what ambiguities do you see being introduced? Right now, only def function_name(foo, bar, baz=None): pass and async def function_name(foo, bar, baz=None): pass are parseable, from what I can see. Of course, type hints as well. Based on that, I can not yet see ambiguities. Am I missing something? One thing I do see as being different, is that `def` takes statements, not an expression. So, one should write an extra `return` as in sorted(datalist, key=def x: return x[5]) (Not requiring `return` here would be very confusing if one were to re-use the keyword `def`.) On the other hand, then one might even do something like sock.send_data(data, on_error=def err: some statements handling err , on_complete=def: draw a kitten ) Which starts looking very javascript-ish to me, so maybe never mind. Not to mention that the placement of the comma is troublesome. And of course that now the indent **within** an expression also has meaning. Now I said javascript, I'm even considering adding extra parenthesis around the arguments, giving (using the simple sort, again): sorted(datalist, key=def (x): return x[5]) And adding some type hints sorted(datalist, key=def (x: List[Int]) -> Int: return x[5]) I myself have learned Haskell before learning Python, so to me the `lambda` is not esoteric, and I have not considered it as such. However, I can understand that it would be considered as such by people coming to Python from other languages such as * Javascript having `function (foo, bar) { statements }` * Java having `(foo, bar) -> expression` or `(foo, bar) -> { statements }`. * C# being basically the same as Java, except for using a `=>` instead of `->`. For Java and C# I only did a quick 2-minute research, but it seems that Java and C# provide anonymous functions, while providing shorthand for expressions. Regarding changing it, I would be wary of it if only the syntax changed, without adding any actual benefits. Using an existing keyword (such as `def`) has the advantage that the meaning of currently working code does not change. From sjoerdjob at sjec.nl Wed Mar 2 20:10:10 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Thu, 3 Mar 2016 02:10:10 +0100 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> Message-ID: <20160303011010.GC17147@sjoerdjob.com> On Wed, Mar 02, 2016 at 04:19:42PM -0800, Abe Dillon wrote: > > ... > the idea is from the recipe metaphor for a function: > > def (): > ...instructions to cook ingredients > > vs. > > ( from ) > > It's fair to quibble over the exact implementation (maybe use 'with' > instead of 'from') but the main point of the syntax it to put the important > bit (i.e.the expression) in front of the (usually) unimportant bit (i.e. > the signature) and to swap out an esoteric word (lambda) with something > that continues the readability emphasis of Python by using more common > words. > So, I'm looking at my recipe book, and it first tells me which ingredients I need, before telling me how to make the thing I want. I also think your perception that the expression is more important than the signature does not in general hold. After all, one writes def foo(bar, baz): instead of def: as foo(bar, baz) Furthermore, regarding simple lambdas---as in lambda x: The extra reading of `lambda x` is not that problematic. However, when you have a lambda needing multiple parameters---as in lambda param1, param2, param3=default: I think the order of the parameters and such is even more important, because it is an easy thing to have in the wrong order. Just out of curiosity, I grepped over the lambdas I could find in my (3.4) standard library. Of the 123 lambdas I found, only 12 of them have more than 1 argument. In most of these 12 cases, the only reason for the many arguments is to capture the value of a variable inside a loop, to make sure it uses the relevant version of that variable (and/or faster lookop). From rosuav at gmail.com Wed Mar 2 20:20:41 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 3 Mar 2016 12:20:41 +1100 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <20160303004627.GB17147@sjoerdjob.com> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> <56D77EAE.6050805@mgmiller.net> <34F20643-3C62-4F9F-859A-E69C6CECC7F9@gmail.com> <20160303004627.GB17147@sjoerdjob.com> Message-ID: On Thu, Mar 3, 2016 at 11:46 AM, Sjoerd Job Postmus wrote: > One thing I do see as being different, is that `def` takes statements, > not an expression. So, one should write an extra `return` as in > > sorted(datalist, key=def x: return x[5]) > > (Not requiring `return` here would be very confusing if one were to > re-use the keyword `def`.) > > On the other hand, then one might even do something like > > sock.send_data(data, > on_error=def err: > some > statements > handling > err > , > on_complete=def: > draw > a > kitten > ) > Someone will correct me if I'm wrong, but I'm pretty sure Python's grammar cannot handle statements inside expressions. Creating a "def" expression that has a suite inside it would mess with everyone's heads AND the parser, and I'm pretty sure that one won't fly. ChrisA From ethan at stoneleaf.us Wed Mar 2 20:26:04 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 02 Mar 2016 17:26:04 -0800 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> Message-ID: <56D792AC.1010000@stoneleaf.us> Note: Whatever fancy stuff you are doing to your messages is totally messing up trying to reply to you. On 03/02/2016 04:19 PM, Abe Dillon wrote: > Not only did you get the syntax wrong, Ah, I see that I did. Oops. > but the lambda version is also > horrid which is exacerbated by the fact that it looks like your trying > to give the function a name or otherwise store it which defeats whole > purpose of a lambda. If I was giving it a name I would use `def`. I am storing it, and that is a very common use of lambdas (to be fair, I stole one line from a multi-line dictionary definition). > At that point just use 'def' and stop trying to use > lambda where it is ill suited. This is exactly where lambda is suited. > the idea is from the recipe metaphor for a function: > > def (): > ...instructions to cook ingredients > > vs. > > ( from ) Huh. Well, it looks good like that, but the actual examples were quite jarring to me. > It's fair to quibble over the exact implementation (maybe use 'with' > instead of 'from') but the main point of the syntax it to put the > important bit (i.e.the expression) in front of the (usually) unimportant > bit (i.e. the signature) and to swap out an esoteric word (lambda) with > something that continues the readability emphasis of Python by using Yeah, it would be better with `with`. But, really, I don't see it happening -- the whole anonymous function thing is not encouraged, so making it easier is not a goal. I wonder if MacroPy[1] would let you try it out? -- ~Ethan~ [1] https://pypi.python.org/pypi/MacroPy From sjoerdjob at sjec.nl Wed Mar 2 20:41:36 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Thu, 3 Mar 2016 02:41:36 +0100 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> <56D77EAE.6050805@mgmiller.net> <34F20643-3C62-4F9F-859A-E69C6CECC7F9@gmail.com> <20160303004627.GB17147@sjoerdjob.com> Message-ID: <20160303014136.GA18052@sjoerdjob.com> On Thu, Mar 03, 2016 at 12:20:41PM +1100, Chris Angelico wrote: > On Thu, Mar 3, 2016 at 11:46 AM, Sjoerd Job Postmus wrote: > > One thing I do see as being different, is that `def` takes statements, > > not an expression. So, one should write an extra `return` as in > > > > sorted(datalist, key=def x: return x[5]) > > > > (Not requiring `return` here would be very confusing if one were to > > re-use the keyword `def`.) > > > > On the other hand, then one might even do something like > > > > sock.send_data(data, > > on_error=def err: > > some > > statements > > handling > > err > > , > > on_complete=def: > > draw > > a > > kitten > > ) > > > > Someone will correct me if I'm wrong, but I'm pretty sure Python's > grammar cannot handle statements inside expressions. Creating a "def" > expression that has a suite inside it would mess with everyone's heads > AND the parser, and I'm pretty sure that one won't fly. I would be very surprised if the current grammar would be able to handle it. After all, it (rightfully) can't handle an assignment inside an expression either---like `if x = foo():`. I'm uncertain as to whether or not the grammar would be (easily) tweakable to allow a `def` suite inside an expression. I'm not familiar enough with the grammar (yet). As for messing with everyone's heads, I'm not worried about that too much yet. I think the example I gave does not look too bad (it somewhat follows Javascript idioms, though, and is thus not Pythonic). Now that Python has gotten async functions, it might very well be used even more for callback based programming. To me, that reminds me of Javascript, where inline definition of callbacks is the norm, not the exception. I'm not claiming that Python idioms in this regard can (or should or will) change in that direction, however I see that Java, Javascript, Ruby, C# (and possibly other languages as well) offer inline definition of multi-statement callbacks. Anyhow, this is completely off-topic, as the topic is about changing the syntax of a simple lambda, not on enabling multi-statement inline function definitions. So, unless somebody gets enthuasiastic enough to fork the thread off, I'll let it rest. From mike at selik.org Wed Mar 2 20:59:19 2016 From: mike at selik.org (Michael Selik) Date: Thu, 03 Mar 2016 01:59:19 +0000 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76827.4020200@sdamon.com> Message-ID: On Thu, Mar 3, 2016 at 12:14 AM Abe Dillon wrote: > lambda statements tend to be most expressive when used in an expected > context (like as the 'key' argument to the sorted function or as the first > argument to the filter function. > I also use lambdas for the various key= functions (sorted, min, max). I tend to use comprehensions instead of lambdas in filter. Still, in the examples you wrote, I might try some alternatives... farthest = max(words, key=partial(edit_distance, target="hello")) distant_words = [w for w in words if edit_distance(w, farthest) < 5] shortest = min(len(word.strip()) for word in distant_words) Typing 'lambda' is not at all my concern, in fact `(x+1 from x)` takes the > same number of keystrokes as `lambda x:x+1`. My complaint is in readability > which is supposed to be Python's strong suit. To most people, 'lambda' is > pretty much a nonsense word. It might as well be 'quaple'. Would you be ok > with writing: > > sorted(words, key=quaple word: edit_distance(word, "hello")) > The phrase ``from thing`` suggests that ``thing`` is a container, not an argument. While I agree that reducing jargon is a good goal, the replacement with ``from`` unfortunately introduces a different jargon rather than making it sound like natural language. As you say, ``quaple`` is nearly as effective as ``lambda`` except for the handful of people who already knew the jargon from other languages. Unfortunately, both ``quaple`` and ``lambda`` are better than your usage of ``from`` because of the cognitive dissonance. Perhaps a keyword such as ``using`` might be better. Besides, I'd rather write that sorted example with functools.partial: sorted(words, key=partial(edit_distance, target='hello')) I'm not saying I think lambdas are great, but so far I haven't seen a convincing example for this new usage of ``from``. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abedillon at gmail.com Wed Mar 2 21:00:08 2016 From: abedillon at gmail.com (Abe Dillon) Date: Wed, 2 Mar 2016 18:00:08 -0800 (PST) Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <20160303011010.GC17147@sjoerdjob.com> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> <20160303011010.GC17147@sjoerdjob.com> Message-ID: <39279233-6232-409d-b5ea-5adaa95a0728@googlegroups.com> On Wednesday, March 2, 2016 at 7:10:37 PM UTC-6, Sjoerd Job Postmus wrote: > > So, I'm looking at my recipe book, and it first tells me which > ingredients I need, before telling me how to make the thing I want. > Recipes are full of steps that could refer to other recipes but offer a more expressive form: cream the butter and sugar together # make cream from butter and sugar caramelize the sugar # make caramel from sugar Trying to apply the whole recipe metaphor to lambda expressions misses the point. Lambda expressions are expressive when their short and surrounded by context (like being the first argument to the 'map' function). They aren't comparable to a stand-alone recipe. It is possible to write arbitrary length programs in a lambda expression but that's not what they're meant for and it usually makes code more obfuscated than expressive. > > I also think your perception that the expression is more important than > the signature does not in general hold. After all, one writes > > def foo(bar, baz): > > > instead of > > def: > > as foo(bar, baz) > Yes, that's because function definitions are different than lambda expressions and have a different use case. I'm not proposing that function definitions should follow the reverse syntax. Defined functions are usually abstracted away from any particular context while lambdas only make sense in very specific contexts. > > Furthermore, regarding simple lambdas---as in > > lambda x: > > The extra reading of `lambda x` is not that problematic. It isn't *that* problematic, but it is noise. It could be better. That's what I'm proposing. > However, when > you have a lambda needing multiple parameters---as in > > lambda param1, param2, param3=default: > > I think the order of the parameters and such is even more important, > because it is an easy thing to have in the wrong order. > Again, the use cases for lambdas with complex signatures are very rare. Your own investigation confirmed this: > Just out of curiosity, I grepped over the lambdas I could find in my > (3.4) standard library. Of the 123 lambdas I found, only 12 of them have > more than 1 argument. > In most of these 12 cases, the only reason for the many arguments is to > capture the value of a variable inside a loop, to make sure it uses the > relevant version of that variable (and/or faster lookop). > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abedillon at gmail.com Wed Mar 2 21:12:05 2016 From: abedillon at gmail.com (Abe Dillon) Date: Wed, 2 Mar 2016 18:12:05 -0800 (PST) Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76827.4020200@sdamon.com> Message-ID: <279ea9b7-32e4-423b-8666-404207cbdef7@googlegroups.com> On Wednesday, March 2, 2016 at 8:00:00 PM UTC-6, Michael Selik wrote: > > I'm not saying I think lambdas are great, but so far I haven't seen a > convincing example for this new usage of ``from`` > I think the case has been made that `with` would be a better keyword. `using` would be great if it didn't introduce a new keyword that could break existing programs. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abedillon at gmail.com Wed Mar 2 21:44:33 2016 From: abedillon at gmail.com (Abe Dillon) Date: Wed, 2 Mar 2016 18:44:33 -0800 (PST) Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <56D792AC.1010000@stoneleaf.us> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> <56D792AC.1010000@stoneleaf.us> Message-ID: <34e80013-fbec-41f7-a886-e1be0cf7c2d3@googlegroups.com> On Wednesday, March 2, 2016 at 7:25:24 PM UTC-6, Ethan Furman wrote: > > > but the lambda version is also > > horrid which is exacerbated by the fact that it looks like your trying > > to give the function a name or otherwise store it which defeats whole > > purpose of a lambda. > > If I was giving it a name I would use `def`. I am storing it, and that > is a very common use of lambdas (to be fair, I stole one line from a > multi-line dictionary definition). > > > At that point just use 'def' and stop trying to use > > lambda where it is ill suited. > > This is exactly where lambda is suited. > > You're right. I'm sorry for getting uppity. I was thinking of the problems that arise when you try to store generators (not lambdas *brain fart*) instead of using them in situ. Whenever I've tried to store generators, it has lead to pain and suffering because they are ephemeral creatures. Lambdas, obviously are not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 2 22:48:39 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 3 Mar 2016 13:48:39 +1000 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> Message-ID: On 3 March 2016 at 06:01, Abe Dillon wrote: > More generally, I think a superior syntax for lambda would be: > > ( from ) > > The reasons I believe that's a superior syntax are: > > a) In the vast majority of use cases for lambda expressions the call > signature can be easily inferred (like in a key function), so moving it > after the expression tends to be more readable. This observation isn't necessarily accurate, as many uses of lambdas regularly rely on lexical scoping to look up some values (hence why the late binding behaviour of such lookups is frequently cited as a problem). > b) It doesn't use the esoteric name, 'lambda' which causes its own > readability issues. Given the readability issues caused by overuse of lambda, there's a school of thought that sees "I like lambda functions, but don't like the lambda keyword, so I avoid using lambda expressions in Python" as a desirable characteristic of the status quo. > c) It should be compatible with existing code: > > try: > 1/0 > except Exception as e: > raise (ValueError() from e) # this is already a syntax error so > implementing the proposal won't break working code > # of this form. I'm not aware of any other > uses of the 'from' keyword that would create > # ambiguities Right, the main downside from an "existing syntax" perspective is that it would be a 3rd use of "from" that's semantically unrelated to the existing uses (imports, exception chaining). > I'm not sure if this should actually be implemented, but I wanted to share > the idea anyway and get some feedback. In my opinion 'lambda' is one of the > less elegant pieces of Python syntax not because it's restricted to a single > expression, but because the name and syntax are both detrimental to > readability. I'm personally not averse to adding new syntactic sugar for lambda expressions, but if we did do something like that, I'd advocate for just stealing Java's spelling (perhaps with the addition of mandatory parentheses, ala generator expressions): https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html#syntax The rationale for that would be: 1. "(params -> expr)" is easier on the eyes than "lambda params: expr" 2. "->" is already used to separate function parameter definitions from the return type annotation 3. It makes moving back and forth between Java & Python easier 4. Swift uses '->' to separate the parameter type and return type declarations in closure expressions [1] 5. It's also similar to the C# lambda expression syntax (which uses "=>" rather than "->") [2] 6. JavaScript ES6 also has an "arrow function" construct similar to the C# lambda expression [3] While that may sound like an "argument from popularity" (and there are certainly aspects of that), my main rationale for advocating for consistency with other popular (or operating system vendor backed) languages with similar capabilities would be to *increase Python's value as a teaching language*: making semantically similar constructs look similar to the way they look elsewhere makes it easier for folks that start their text-based programming education with Python to later make the move to a different platform (if that's where their career and interests takes them), and "helps in preparation for other environments" is a positive characteristic in a world where developers are spoiled for choice when it comes to programming languages and runtimes. However, I'm not interested enough in the idea to propose it myself - I'm interested in the problem from an abstract language design perspective these days, not a "this actually bothers me personally" sense [4]. [1] https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/Expressions.html [2] https://msdn.microsoft.com/en-AU/library/bb397687.aspx [3] https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Functions/Arrow_functions [4] That wasn't always true, as can be seen if you search far enough back in the dev list archives :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Thu Mar 3 01:55:48 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 3 Mar 2016 01:55:48 -0500 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> Message-ID: On 3/2/2016 10:48 PM, Nick Coghlan wrote: > I'm personally not averse to adding new syntactic sugar for lambda > expressions, but if we did do something like that, I'd advocate for > just stealing Java's spelling (perhaps with the addition of mandatory > parentheses, ala generator expressions): > https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html#syntax > > The rationale for that would be: > > 1. "(params -> expr)" is easier on the eyes than "lambda params: expr" > 2. "->" is already used to separate function parameter definitions > from the return type annotation > 3. It makes moving back and forth between Java & Python easier > 4. Swift uses '->' to separate the parameter type and return type > declarations in closure expressions [1] > 5. It's also similar to the C# lambda expression syntax (which uses > "=>" rather than "->") [2] > 6. JavaScript ES6 also has an "arrow function" construct similar to > the C# lambda expression [3] Not bad. 7. It is similar to the pattern matching expressions used in ml and probably elsewhere. fac: 0 -> 1 n -> n * fac(n-1) Of course as a lambda substitute, only one (generic) pattern is allowed and no recursion (absent the use of combinators, but ignore that). > While that may sound like an "argument from popularity" (and there are > certainly aspects of that), my main rationale for advocating for > consistency with other popular (or operating system vendor backed) > languages with similar capabilities would be to *increase Python's > value as a teaching language*: making semantically similar constructs > look similar to the way they look elsewhere makes it easier for folks > that start their text-based programming education with Python to later > make the move to a different platform (if that's where their career > and interests takes them), and "helps in preparation for other > environments" is a positive characteristic in a world where developers > are spoiled for choice when it comes to programming languages and > runtimes. > > However, I'm not interested enough in the idea to propose it myself - > I'm interested in the problem from an abstract language design > perspective these days, not a "this actually bothers me personally" > sense [4]. > > [1] https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/Expressions.html > [2] https://msdn.microsoft.com/en-AU/library/bb397687.aspx > [3] https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Functions/Arrow_functions > [4] That wasn't always true, as can be seen if you search far enough > back in the dev list archives :) -- Terry Jan Reedy From ncoghlan at gmail.com Thu Mar 3 02:54:55 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 3 Mar 2016 17:54:55 +1000 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> Message-ID: > On 3/2/2016 10:48 PM, Nick Coghlan wrote: > >> I'm personally not averse to adding new syntactic sugar for lambda >> expressions, but if we did do something like that, I'd advocate for >> just stealing Java's spelling (perhaps with the addition of mandatory >> parentheses, ala generator expressions): >> >> https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html#syntax >> >> The rationale for that would be: >> >> 1. "(params -> expr)" is easier on the eyes than "lambda params: expr" One qualifier on that: to meet the deliberate LL(1) parsing constraint on the language grammar, the actual syntax would probably need to be "(def params -> expr)" Generator expressions and comprehensions don't need an introductory token as the first child node is an ordinary expression, so the parser doesn't need any advance notice of the larger construct. That isn't the case with a parameter list - those have special parsing rules to allow things like default arguments, *args, and **kwds. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tritium-list at sdamon.com Thu Mar 3 03:25:59 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Thu, 3 Mar 2016 03:25:59 -0500 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> Message-ID: <56D7F517.6060902@sdamon.com> It still comes down to, for me, is ``def foo -> bar(foo)`` really any better than ``lambda foo: bar(foo)``? Are we gaining anything? Does it do anything that the lambda keyword does not? I have been (mostly) following this thread, and I cannot really see a real concrete problem with the lambda keyword and the colon. I don't really want to rain on parades here*, but unless something amazing can come out of new syntax (new functionality, because the readability gain in these proposals is debatable), I really don't think the syntax is going to change. Am I missing some great change in functionality here? If this discussion happened before lambda was added to the language, I think this would be a different story. * ok, I lied, I don't like this suggestion, so maybe I wanted to drizzle on it's parade. A spring shower at best. On 3/3/2016 02:54, Nick Coghlan wrote: >> On 3/2/2016 10:48 PM, Nick Coghlan wrote: >> >>> I'm personally not averse to adding new syntactic sugar for lambda >>> expressions, but if we did do something like that, I'd advocate for >>> just stealing Java's spelling (perhaps with the addition of mandatory >>> parentheses, ala generator expressions): >>> >>> https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html#syntax >>> >>> The rationale for that would be: >>> >>> 1. "(params -> expr)" is easier on the eyes than "lambda params: expr" > One qualifier on that: to meet the deliberate LL(1) parsing constraint > on the language grammar, the actual syntax would probably need to be > "(def params -> expr)" > > Generator expressions and comprehensions don't need an introductory > token as the first child node is an ordinary expression, so the parser > doesn't need any advance notice of the larger construct. That isn't > the case with a parameter list - those have special parsing rules to > allow things like default arguments, *args, and **kwds. > > Cheers, > Nick. > From p.f.moore at gmail.com Thu Mar 3 04:15:52 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 3 Mar 2016 09:15:52 +0000 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <56D77EAE.6050805@mgmiller.net> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> <56D77EAE.6050805@mgmiller.net> Message-ID: On 3 March 2016 at 00:00, Mike Miller wrote: > On 2016-03-02 15:03, Paul Moore wrote: >> >> PS For the record, I dislike the lambda syntax intensely - the odd >> keyword, the fact that "lambda" is sometimes almost as long as than >> the expression I'm using, the use of the mid-line colon which is hard >> to spot, ... So in principle I'm inclined to support "improvement" > > > Hmm, can't think of a way to get rid of the colon without it looking like a > generator, but perhaps we could tackle the other issues by letting "def" > stand in for "lambda": > > lambda x: x.y > > def x: x.y > > It's shorter, not an esoteric word, and is analogous to a named function > definition, reminiscent of javascript. The lack of name/parens separates it > from the standard form. Doable, but of course its pretty late in the game > to be changing lambda, perhaps in Python 4? To reinstate the context: even though I dislike the lambda syntax, in general I *don't* want it "improved" - instead I want a move away from needing it. Things that help: 1. Encouraging the community to appreciate that throwaway named function definitions are an acceptable approach. 2. Specialised syntax or (possibly 3rd party) modules for handling the common cases - the fn.py module sounds like a good example for handling simple expressions. I think we should keep lambda syntax for the relatively few cases where it would remain the best option, but I don't think it needs to be changed - if nothing else the backward compatibility issues would be too great. Paul From p.f.moore at gmail.com Thu Mar 3 04:22:11 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 3 Mar 2016 09:22:11 +0000 Subject: [Python-ideas] Thoughts on lambda expressions In-Reply-To: <44d48cb9-846e-4376-a2af-43c1c0cba1f0@googlegroups.com> References: <054c49e4-901f-4bd7-bc46-bf69afb091c5@googlegroups.com> <56D76D2A.5080107@stoneleaf.us> <44d48cb9-846e-4376-a2af-43c1c0cba1f0@googlegroups.com> Message-ID: On 3 March 2016 at 00:37, Abe Dillon wrote: > On Wednesday, March 2, 2016 at 5:04:10 PM UTC-6, Paul Moore wrote: >> >> So the basic problem for me is that the proposal doesn't offer huge >> benefits, and it's solving a problem that people should probably try >> to avoid in the first place. > > > Yes, that's why I said, "I'm not sure if this should actually be > implemented". People talk a lot about the 'expressiveness' of lambda > statements, but the situations where they're actually more expressive than > obfuscating are very rare and usually involve some idiom that makes the > signature definition all but vestigial. > > filter(lambda num: num > 3, nums) > # all that boiler plate before the part we really care about: filter(num > > 3, nums) > > This is a minor improvement suggestion with some insight I've gleaned on > code readability (putting the meat before the potatoes often improves > readability, as does using more common english words). Understood, and (IMO) you're right about the improved readability. I think the costs mean that it's not worth implementing (as you suspected) but it's still a good principle ro remember. I actually quite liked Perl's "trailing control flow" constructs for much the same reason (put the key thing first). Using Python-like syntax (because I can't remember Perl these days): x = default if x is None x = x + 1 while fn(x) < 100 The problem with these being Perl's usual problem - they add yet more ways of saying the same thing we can already do. Paul From rosuav at gmail.com Mon Mar 7 11:54:41 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 8 Mar 2016 03:54:41 +1100 Subject: [Python-ideas] Pseudo-package for current directory Message-ID: Every once in a while, the issue pops up again of "import X" picking up X.py from the current directory, when a stdlib or pip-installed module named X was wanted. Python almost has a mechanism for protecting against this, in the form of explicit package relative imports; the only problem is that the current directory isn't a package. So the solution is to treat the current directory as a pseudo-package. It'd be a backward-incompatible change, so it would need to be explicitly invoked. Something like: python3 -p somefile.py which would pretend to create an __init__.py in the current directory, change to the parent, and "from dirname import somefile". Is that something that would work with the current import system? It'd offer the same protection that Unix systems have had for decades: the current directory is not in $PATH, and if you want to run a script that's "next door to you", you explicitly request it as ./scriptname. Idle could be protected from accidentally importing someone's "html.py", because "import html" shouldn't ever import from the current directory. ChrisA From random832 at fastmail.com Mon Mar 7 12:07:15 2016 From: random832 at fastmail.com (Random832) Date: Mon, 07 Mar 2016 12:07:15 -0500 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: References: Message-ID: <1457370435.3901655.542029746.6A4C4A69@webmail.messagingengine.com> On Mon, Mar 7, 2016, at 11:54, Chris Angelico wrote: > It'd be a backward-incompatible change, so it would need to be > explicitly invoked. Something like: > > python3 -p somefile.py > > which would pretend to create an __init__.py in the current directory, > change to the parent, and "from dirname import somefile". "#!/usr/bin/env python3 -p" won't work on many systems. So I think it'd be better as a magic statement within the file (possibly simply a future-import, with the possibility of eventually making it the default behavior), rather than a command-line argument. From rosuav at gmail.com Mon Mar 7 12:22:17 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 8 Mar 2016 04:22:17 +1100 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: <1457370435.3901655.542029746.6A4C4A69@webmail.messagingengine.com> References: <1457370435.3901655.542029746.6A4C4A69@webmail.messagingengine.com> Message-ID: On Tue, Mar 8, 2016 at 4:07 AM, Random832 wrote: > On Mon, Mar 7, 2016, at 11:54, Chris Angelico wrote: >> It'd be a backward-incompatible change, so it would need to be >> explicitly invoked. Something like: >> >> python3 -p somefile.py >> >> which would pretend to create an __init__.py in the current directory, >> change to the parent, and "from dirname import somefile". > > "#!/usr/bin/env python3 -p" won't work on many systems. So I think it'd > be better as a magic statement within the file (possibly simply a > future-import, with the possibility of eventually making it the default > behavior), rather than a command-line argument. It can't be a future import, though, as they apply to the file they're in, and not to anywhere else. This directive would change the way that the regular "import" statement behaves, for the whole process. In theory, though, since it doesn't change syntax, it could be something like: import sys sys.package_mode_ACTIVATE() The trouble is that doing this after _any_ imports (including the ones that are done before your script starts) will risk those imports being resolved from local files. Having "import X" after activating package mode is useless if X gets resolved from sys.modules, and it'd be a nightmare to debug. If the -p parameter is a problem, maybe there could be an environment variable that changes the default? Then people could opt-in to pseudo-package mode globally, run all their tests, and see what stops working. FWIW, I do want this to eventually become the default behaviour. But it wouldn't be any time soon; in the meantime, it would be easy enough to recommend that people just "always do this to be safe" (in the same way that Python 2 didn't make new-style classes the default, but everyone's advised to "always subclass object"). By making it a long-term non-default feature, Python gets to provide safety without breaking anyone's code. ChrisA From steve at pearwood.info Mon Mar 7 13:04:34 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 8 Mar 2016 05:04:34 +1100 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: References: Message-ID: <20160307180433.GA12028@ando.pearwood.info> On Tue, Mar 08, 2016 at 03:54:41AM +1100, Chris Angelico wrote: > Every once in a while, the issue pops up again of "import X" picking > up X.py from the current directory, when a stdlib or pip-installed > module named X was wanted. Couldn't we (almost) fix that in an almost backwards-compatible way by moving the current directory to the end of the path, instead of the beginning? If none of the modules in the current directory shadow the standard library, then you won't notice any difference. If you are shadowing the standard library, then you're either doing it deliberately, which I expect will be rare, or it's an accident. If it's deliberate, then you probably know what you're doing and can work around the change of behaviour. If it is an accident, then the standard library will shadow your module, rather than the other way around, which strikes me as better. (E.g. you won't accidently break IDLE by shadowing something IDLE depends on.) > Python almost has a mechanism for > protecting against this, in the form of explicit package relative > imports; the only problem is that the current directory isn't a > package. We shouldn't be talking about "current directory", since that's only relevant when you run Python without specifying a script to run. It is actually the directory containing the script being executed, not the current directory, which gets added to the path. If there is no script, the faux path '' is added to the path, and that gets treated as if it were '.' (the current directory). > So the solution is to treat the current directory as a pseudo-package. > It'd be a backward-incompatible change, so it would need to be > explicitly invoked. Something like: > > python3 -p somefile.py > > which would pretend to create an __init__.py in the current directory, > change to the parent, and "from dirname import somefile". Seems awfully convoluted and hard to understand. How will it interact with running real packages? How about running modules using -m? Even if it works, it relies on the user being knowledgeable enough to run `python3 -p script` instead of `python3 script`, which means the people who most need this new feature are the least likely to use it. I think that's the fatal objection to this: if the user knows enough to run -p, she knows enough to diagnose an accidental shadowing. [...] > Idle could be protected from accidentally importing someone's > "html.py", because "import html" shouldn't ever import from the > current directory. That won't help when the user runs python3 html.py (well perhaps it will help *specifically* with IDLE, but not with the general problem of accidental shadowing). -- Steve From random832 at fastmail.com Mon Mar 7 13:52:01 2016 From: random832 at fastmail.com (Random832) Date: Mon, 07 Mar 2016 13:52:01 -0500 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: <20160307180433.GA12028@ando.pearwood.info> References: <20160307180433.GA12028@ando.pearwood.info> Message-ID: <1457376721.3926402.542147770.3EA10FB5@webmail.messagingengine.com> On Mon, Mar 7, 2016, at 13:04, Steven D'Aprano wrote: > > Idle could be protected from accidentally importing someone's > > "html.py", because "import html" shouldn't ever import from the > > current directory. > > That won't help when the user runs > > python3 html.py It would if doing so *only* loads it as sys.modules['__main__'] and leaves sys.modules['html'] empty. From rosuav at gmail.com Mon Mar 7 14:34:11 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 8 Mar 2016 06:34:11 +1100 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: <20160307180433.GA12028@ando.pearwood.info> References: <20160307180433.GA12028@ando.pearwood.info> Message-ID: On Tue, Mar 8, 2016 at 5:04 AM, Steven D'Aprano wrote: > On Tue, Mar 08, 2016 at 03:54:41AM +1100, Chris Angelico wrote: >> Every once in a while, the issue pops up again of "import X" picking >> up X.py from the current directory, when a stdlib or pip-installed >> module named X was wanted. > > Couldn't we (almost) fix that in an almost backwards-compatible way by > moving the current directory to the end of the path, instead of the > beginning? Good question. Fortunately that one can be tested without major language hacking. I'm curious to know the answer. >> Python almost has a mechanism for >> protecting against this, in the form of explicit package relative >> imports; the only problem is that the current directory isn't a >> package. > > We shouldn't be talking about "current directory", since that's only > relevant when you run Python without specifying a script to run. It is > actually the directory containing the script being executed, not the > current directory, which gets added to the path. If there is no script, > the faux path '' is added to the path, and that gets treated as if it > were '.' (the current directory). Fair criticism. What I really mean is "script directory", which happens to align in the specific (albeit common) case of "python3 script.py" with no path on it. The rest of my proposal is the same, though. >> So the solution is to treat the current directory as a pseudo-package. >> It'd be a backward-incompatible change, so it would need to be >> explicitly invoked. Something like: >> >> python3 -p somefile.py >> >> which would pretend to create an __init__.py in the current directory, >> change to the parent, and "from dirname import somefile". > > Seems awfully convoluted and hard to understand. I know; that's the result of being pedantic. Here's an alternative description: $ python3 -p somefile.py pretends that the current directory is a package, and imports somefile from that package. > How will it interact with running real packages? How about running > modules using -m? I don't think it would be any different, but haven't tested. They'd be exactly the same thing, except that "python3 -m X.Y" looks for Y.py inside package X on sys.path, and "python3 -p somefile.py" takes a file name (which might include a path), and runs it directly. If you're running a real package that isn't on sys.path, you can either put it on sys.path, or use -p mode. Something like this, which currently works (and would still work if the current directory were removed from sys.path): $ mkdir pkg $ echo 'print("init")' >pkg/__init__.py $ echo 'print("module")' >pkg/module.py $ PYTHONPATH=. python3 -m pkg.module init module Or alternatively, this: $ python3 -p pkg/module.py If the semantics of -p are "pretend that __init__.py exists", without precluding one that really does, this should behave correctly. > Even if it works, it relies on the user being knowledgeable enough to > run `python3 -p script` instead of `python3 script`, which means the > people who most need this new feature are the least likely to use it. I > think that's the fatal objection to this: if the user knows enough to > run -p, she knows enough to diagnose an accidental shadowing. Right, this is a concern. But I don't want to break backward compatibility. Otherwise, the common situation of having multiple .py files in the same directory and having them import each other would be broken. Ultimately, I'd like to offer this protection by default. But I'm seeing this as similar to reminding people to type "python3" instead of just "python"; now we just teach people to type "python3 -p", and if they happen to miss off the 3, the system will now tell them (because "python2 -p" is an error). If it's done as an environment variable as well or instead, would that be better? Python distributors and educators could choose to activate it by default, and a venv could apply the protection. >> Idle could be protected from accidentally importing someone's >> "html.py", because "import html" shouldn't ever import from the >> current directory. > > That won't help when the user runs > > python3 html.py > > (well perhaps it will help *specifically* with IDLE, but not with the > general problem of accidental shadowing). True, but if you're asking for help and you're running a thing called "html.py" or "complex.py", the people you talk to have a chance of noticing. But the presence of such a file in the current directory could mess stuff up, and a lot of people don't think to mention every other file they've ever worked on (in their "Python Tinkering" directory) when they run into difficulties. ChrisA From tjreedy at udel.edu Mon Mar 7 17:59:55 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 7 Mar 2016 17:59:55 -0500 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: References: <1457370435.3901655.542029746.6A4C4A69@webmail.messagingengine.com> Message-ID: On 3/7/2016 12:22 PM, Chris Angelico wrote: > In theory, though, since it doesn't change syntax, it could be something like: > > import sys > sys.package_mode_ACTIVATE() > > The trouble is that doing this after _any_ imports (including the ones > that are done before your script starts) will risk those imports being > resolved from local files. Barring a setting in $PYTHONPATH, Python's startup imports will not be resolved from local files. It appears that '' or scriptdir are only added to sys.path afterwards. I base this on an experiment where I tried to shadow one of the Python-coded /lib modules with a local file. C-coded builtin modules cannot be shadowed. I discovered this by experiment and then found "Python includes a number of default finders and importers. The first one knows how to locate built-in modules, and the second knows how to locate frozen modules. A third default finder searches an import path for modules. The import path is a list of locations that may name file system paths or zip files." https://docs.python.org/3/reference/import.html#finders-and-loaders I consider the behavior of local versus stdlib imports depending on the stdlib implementation language to be somewhat problematical. There was discussion of shadowing, I believe here, a few months ago (sometime after 3.5.0 was released). > Having "import X" after activating package > mode is useless if X gets resolved from sys.modules, and it'd be a > nightmare to debug. > > If the -p parameter is a problem, maybe there could be an environment > variable that changes the default? Then people could opt-in to > pseudo-package mode globally, run all their tests, and see what stops > working. > > FWIW, I do want this to eventually become the default behaviour. But > it wouldn't be any time soon; in the meantime, it would be easy enough > to recommend that people just "always do this to be safe" (in the same > way that Python 2 didn't make new-style classes the default, but > everyone's advised to "always subclass object"). By making it a > long-term non-default feature, Python gets to provide safety without > breaking anyone's code. You might want to look at previous discussions of shadowing if you can find them. -- Terry Jan Reedy From rob.cliffe at btinternet.com Mon Mar 7 18:48:30 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 7 Mar 2016 23:48:30 +0000 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: References: Message-ID: <56DE134E.8080608@btinternet.com> On 07/03/2016 16:54, Chris Angelico wrote: > "import html" shouldn't ever import from the > current directory. > > I don't agree. When troubleshooting, you may wish to dig into packages (in the stdlib, or 3rd-party packages) to find out what is going on, and run your own version with (say) diagnostic messages added. I am sure I have done this a few times. Sorry, I can't give chapter and verse. Rob Cliffe. From rosuav at gmail.com Mon Mar 7 19:06:26 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 8 Mar 2016 11:06:26 +1100 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: <56DE134E.8080608@btinternet.com> References: <56DE134E.8080608@btinternet.com> Message-ID: On Tue, Mar 8, 2016 at 10:48 AM, Rob Cliffe wrote: > On 07/03/2016 16:54, Chris Angelico wrote: >> >> "import html" shouldn't ever import from the >> current directory. >> >> > I don't agree. > When troubleshooting, you may wish to dig into packages (in the stdlib, or > 3rd-party packages) to find out what is going on, and run your own version > with (say) diagnostic messages added. > I am sure I have done this a few times. Sorry, I can't give chapter and > verse. In the unusual cases where you actually want that, there are other solutions. You could either change the code to say "from . import html", or spin up a virtual environment and edit the actual code in html.py. Or you could explicitly add a directory to PYTHONPATH for your overrides (~/monkeypatch/html.py or something). The default import mechanism doesn't need to handle overrides IMO. ChrisA From rob.cliffe at btinternet.com Mon Mar 7 19:32:35 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 8 Mar 2016 00:32:35 +0000 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: References: <56DE134E.8080608@btinternet.com> Message-ID: <56DE1DA3.2030307@btinternet.com> Sorry if I'm belabouring the point, I should have included this in my last e-mail. But you seem to be saying that when I write import xyz and I mean "import xyz from the current directory, not from the stdlib [or whatever]" I have to know the names of all the modules in the stdlib [or whatever]. Or else *always* add some boilerplate whenever I mean "import from the current directory". Well, maybe, but I am sure a lot of the time I couldn't be bothered, and then some day I'd run into a not-immediately-obvious bug. I.e.: Changing the current behaviour would surprise me. Rob Cliffe On 08/03/2016 00:06, Chris Angelico wrote: > On Tue, Mar 8, 2016 at 10:48 AM, Rob Cliffe wrote: >> On 07/03/2016 16:54, Chris Angelico wrote: >>> "import html" shouldn't ever import from the >>> current directory. >>> >>> >> I don't agree. >> When troubleshooting, you may wish to dig into packages (in the stdlib, or >> 3rd-party packages) to find out what is going on, and run your own version >> with (say) diagnostic messages added. >> I am sure I have done this a few times. Sorry, I can't give chapter and >> verse. > In the unusual cases where you actually want that, there are other > solutions. You could either change the code to say "from . import > html", or spin up a virtual environment and edit the actual code in > html.py. Or you could explicitly add a directory to PYTHONPATH for > your overrides (~/monkeypatch/html.py or something). The default > import mechanism doesn't need to handle overrides IMO. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From rosuav at gmail.com Mon Mar 7 19:44:26 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 8 Mar 2016 11:44:26 +1100 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: <56DE1DA3.2030307@btinternet.com> References: <56DE134E.8080608@btinternet.com> <56DE1DA3.2030307@btinternet.com> Message-ID: On Tue, Mar 8, 2016 at 11:32 AM, Rob Cliffe wrote: > Sorry if I'm belabouring the point, I should have included this in my last > e-mail. > But you seem to be saying that when I write > import xyz > and I mean "import xyz from the current directory, not from the stdlib [or > whatever]" > I have to know the names of all the modules in the stdlib [or whatever]. > Or else *always* add some boilerplate whenever I mean "import from the > current directory". Well, maybe, but I am sure a lot of the time I couldn't > be bothered, and then some day I'd run into a not-immediately-obvious bug. > I.e.: Changing the current behaviour would surprise me. Which is exactly why I am NOT proposing changing the default behaviour (at least, not any time soon). If you're prepared to accept that "import xyz" will only import from sys.path (which would then not have the script directory in it), then you enable this option, otherwise you don't. When you want that, you say "from . import xyz", and all works as you would expect. ChrisA From steve at pearwood.info Mon Mar 7 19:49:24 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 8 Mar 2016 11:49:24 +1100 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: References: <20160307180433.GA12028@ando.pearwood.info> Message-ID: <20160308004924.GB12028@ando.pearwood.info> On Tue, Mar 08, 2016 at 06:34:11AM +1100, Chris Angelico wrote: [...] > >> So the solution is to treat the current directory as a pseudo-package. > >> It'd be a backward-incompatible change, so it would need to be > >> explicitly invoked. Something like: > >> > >> python3 -p somefile.py > >> > >> which would pretend to create an __init__.py in the current directory, > >> change to the parent, and "from dirname import somefile". [...] > > Even if it works, it relies on the user being knowledgeable enough to > > run `python3 -p script` instead of `python3 script`, which means the > > people who most need this new feature are the least likely to use it. I > > think that's the fatal objection to this: if the user knows enough to > > run -p, she knows enough to diagnose an accidental shadowing. > > Right, this is a concern. But I don't want to break backward > compatibility. Otherwise, the common situation of having multiple .py > files in the same directory and having them import each other would be > broken. Only if they shadow something in the std lib. However we fix this, whatever the mechanism, it is going to break backwards compatibility for those shadowing the stdlib, which is of course the whole point of the exercise. I don't think it's worth trying to gradually introduce this: those affected will either want this change or be able to work around it easily. The work around doesn't involve going through your entire code base changing (potentially) thousands of lines of code. It likely involves changing the name of one or two files, or explicitly adding "." to the front of your path. So even though it is a technical break of backwards compatibility, I think it is (1) minor enough and (2) important enough to warrant just doing it. There are three cases where people would notice a difference: (1) Deliberate shadowing of the std lib. In this case, if you are intentionally shadowing a module, I think you are probably capable of coming up with a work-around for the change. E.g. you might explicitly add '.' to your PYTHONPATH environment variable to force it to the front, so as to get the current behaviour. I think that's a rare case and changing the default behaviour here is acceptible. (2) Accidental shadowing of the std lib where they currently experience mysterious breakage. E.g.: [steve at ando tmp]$ touch time.py [steve at ando tmp]$ idle Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/idlelib/run.py", line 7, in import threading File "/usr/local/lib/python2.7/threading.py", line 13, in from time import time as _time, sleep as _sleep ImportError: cannot import name time This will be fixed, which is the whole purpose of the exercise. (3) Accidental shadowing where they *don't* currently experience breakage, but will after the change. E.g. suppose I have a main script "spam.py" and a file called "code.py" in the same directory, and spam.py imports code. Currently, that shadows the stdlib code.py, but harmlessly since few (if any) things depend on the code module. After the change, the stdlib code.py will now shadow the custom one, causing breakage. I suspect that this will be rare in well-established or professional code, where the devs are more familiar with the std lib, but a bit more common in the case of beginners. This is unfortunate, but the fix is simple: rename your file and change a few imports: `import code` becomes `import mycode as code`. -- Steve From p.f.moore at gmail.com Tue Mar 8 04:24:39 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 8 Mar 2016 09:24:39 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 Message-ID: The "py.exe" wrapper on Windows runs the latest Python version available on the user's PC, but it prefers Python 2 if the user has both Python 2 and 3 installed. With Python 3.5 being released, is it not about time that we switched that default to always prefer the latest version available? For scripts, users can always specify a shebang line, exactly as on Unix. For interactive use, people who prefer Python 2 simply have to create a py.ini file saying [defaults] python=2 At the moment, someone using Python 3 exclusively can use "py" to start the Python interpreter - and with the changes in the Python 3.5 installer, defaulting to a per-user install that's not on PATH, this is often the most convenient approach. But if that person installs Python 2 (say, to test some code on that version) py.exe silently switches to using Python 2. And it does so regardless of whether you specify "make this the default Python" in the installer. It seems to me that this behaviour is much more unexpected than the equivalent of a Python user installing Python 3 (py.exe switches to Python 3, but that's "obviously" the latest version). So I suggest that in Python 3.6, we (finally) switch py.exe to run the latest version of Python on the user's machine *regardless of the major version*. This will be a compatibility break, but arguably no more serious than the fact that if the user installs Python 3.6 and puts it on PATH, then "python.exe" now points to Python 3.6. To cover people just installing Python 3.6 for testing, we could check in the installer and if the user only currently has Python 2 installed, has no py.ini, and doesn't select "make this my default Python" for Python 3.6, then we write a py.ini defaulting to Python 2. I don't know if that's worth the effort, it depends largely on how many people will be affected by the change. Paul From victor.stinner at gmail.com Tue Mar 8 06:22:13 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 8 Mar 2016 12:22:13 +0100 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: Message-ID: 2016-03-08 10:24 GMT+01:00 Paul Moore : > The "py.exe" wrapper on Windows runs the latest Python version > available on the user's PC, but it prefers Python 2 if the user has > both Python 2 and 3 installed. With Python 3.5 being released, is it > not about time that we switched that default to always prefer the > latest version available? On UNIX, it was decided to have python2 and python3 and don't guess user expectations. Why not providing an py3 runner which uses Python 3 by default? Victor From p.f.moore at gmail.com Tue Mar 8 08:17:11 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 8 Mar 2016 13:17:11 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: Message-ID: On 8 March 2016 at 11:22, Victor Stinner wrote: > 2016-03-08 10:24 GMT+01:00 Paul Moore : >> The "py.exe" wrapper on Windows runs the latest Python version >> available on the user's PC, but it prefers Python 2 if the user has >> both Python 2 and 3 installed. With Python 3.5 being released, is it >> not about time that we switched that default to always prefer the >> latest version available? > > On UNIX, it was decided to have python2 and python3 and don't guess > user expectations. > > Why not providing an py3 runner which uses Python 3 by default? I don't know. Personally, I don't like the versioned executables (will we get a py4, py5,...?) but that's just a personal preference. I believe that with an unversioned executable it should use the latest version by default, and the point of my email is that I think we're now at a point where special casing Python 2 is unreasonable, particularly on Windows where there is no tradition of a "system Python" which needs to be Python 2. If someone *else* wants to propose replacing py.exe with py2.exe and py3.exe then they can do that (and I'll be -1 on that proposal). You don't mention the fact that an unadorned "python" command on Unix typically invokes Python 2. As far as I know the only reason that was needed was because system Python scripts were written that way, back before Python 3 was around, and couldn't be updated to be Python 3 compatible. That reason doesn't apply on Windows. And on Windows, "python" has always referred to the currently active version of Python, whether 2 or 3. tl; dr; I'm against making the unversioned name "python" (or "py") forever mean Python 2. There are justifications for it on Unix, but none of those apply on Windows. Paul From allan.clark at gmail.com Tue Mar 8 09:17:23 2016 From: allan.clark at gmail.com (Allan Clark) Date: Tue, 08 Mar 2016 14:17:23 +0000 Subject: [Python-ideas] Map-then-filter in comprehensions Message-ID: tl;dr What is support like for adding an 'as' clause to comprehension syntax? In order to allow map-then-filter, it might look like something this: [y for x in numbers if abs(x) as y > 5] I wish to propose an extension to Python comprehension syntax in an attempt to make it applicable in more areas. I'll first describe the deficiency I perceive in the current comprehension syntax and then propose my extension. I'll then note some drawbacks. For the purposes of illustration I'll use list comprehension syntax but I believe everything I say is equally applicable to set and dictionary comprehensions. I'll also talk about lists but again (more or less) everything applies to the more general concept of iterables. Comprehensions are essentially a filter followed by a map operation. So if you need to perform a filter operation followed by a map operation over a list, this is pretty convenient in Python. Here we are going to take the absolute value of all even numbers within the range -10 to 10. numbers = range(-10, 10) [abs(x) for x in numbers if x % 2 == 0] However, if you wish to perform a map operation and *then* a filter operation this is not so convenient, so suppose we wish to obtain the absolute value of all numbers that have an absolute value larger than 5, we can do this by calling the mapped method twice: abs(x) for x in numbers if abs(x) > 5] This is a bit unsatisfying and even impossible in the case that the mapped method has some side-effect (although arguably if you find yourself in that situation you have taken a mis-step somewhere). An alternative is to apply the mapping first: [y for y in (abs(x) for x in numbers) if y > 5] I have to say I quite like this, as it is pretty explicit, but it is a bit unsatisfying that you require only one comprehension for a filter-then-map but two for a map-then-filter. What would be nice is if we could give a name to the mapped expression and then use that in the filter. [abs(x) as y for x in numbers if y > 5] I don't like this as it means the order of execution is dependent on whether there is an 'as' clause, in particular the 'if' clause itself may do some computation such as in `if f(y) > 5`. An altenative is to allow 'as' expressions in the condition, something like: [y for x in numbers if abs(x) as y > 5] Note that it would be possible to map to an intermediate result, such as: [y**2 for x in numbers if abs(x) as y > 5] Alternatively we could put the 'as' in the pattern: [y**2 for abs(x) as y in numbers if y > 5] I did not like this as it is obscures the fact that 'x' is being set to each element of 'numbers'. Additionally, we might later wish to adopt a functional programming idiom in which we use 'as' for deconstructive assignment whilst giving a name to the entire matched value, for example: [p for (x,y) as p if x > y] Or more generally: (x,y) as a = f(z) But that is getting somewhat off-topic. I promised some drawbacks: * I am confident there are some implementation gotchas nestling in here somewhere. * I could imagine how abuse of such a mechanism to lead to pretty unreadable code. * I'm still not *that* upset by the explicit map first: `[y for y in (abs(x) for x in numbers) if y > 5]` * I could see how it is not immediately obvious what the code does. * It would need to be decided whether you allowed multiple 'as' expression in the condition, particularly using 'and' or 'or' as in 'if f(a) as x > 5 and f(b) as y > 5' To summarise: * It's a touch annoying that comprehensions allow filter-then-map but not map-then-filter * Three proposed syntaxes are: * [abs(x) as y for x in numbers if y > 5] * [y for x in numbers if abs(x) as y > 5] * [y**2 for abs(x) as y in numbers if y > 5] * My favourite is the middle one. Finally these seem to currently be syntax errors so we should not break any existing code. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vgr255 at live.ca Tue Mar 8 09:59:18 2016 From: vgr255 at live.ca (=?utf-8?Q?=C3=89manuel_Barry?=) Date: Tue, 8 Mar 2016 09:59:18 -0500 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: Message-ID: Hello, I?m mostly in favour of extending the comprehension syntax, but not at any cost. The `as` proposal is a big -1 from me. Right now, `as` is used for : - from foo import bar as baz - try: ... except Exception as e: ... And maybe a few others I?ve probably forgotten. The general concept is name binding, so if I see something like [x for x, y in some_iterable as y > 5] I?m going be confused by what sort of name binding it does. A quick glance at the keywords list tells me that no currently-existing keyword is ?the obvious choice? for this purpose. I don?t think such a small niche warrants a new keyword, either. Python is mostly intuitive, and your proposition is anything but intuitive to me. -1 from me for now, but might change if you come up with an intuitive syntax. -Emanuel ~If it doesn?t quack like a duck, add a quack() method~ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Mar 8 10:03:13 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Mar 2016 02:03:13 +1100 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: Message-ID: On Wed, Mar 9, 2016 at 12:17 AM, Paul Moore wrote: > tl; dr; I'm against making the unversioned name "python" (or "py") > forever mean Python 2. There are justifications for it on Unix, but > none of those apply on Windows. https://www.python.org/dev/peps/pep-0394/#exclusion-of-ms-windows On Unix, yes, but not on Windows. +1 for making py.exe now default to Python 3. ChrisA From rosuav at gmail.com Tue Mar 8 10:20:37 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Mar 2016 02:20:37 +1100 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: Message-ID: On Wed, Mar 9, 2016 at 1:59 AM, ?manuel Barry wrote: > I?m going be confused by what sort of name binding it does. A quick glance > at the keywords list tells me that no currently-existing keyword is ?the > obvious choice? for this purpose. I don?t think such a small niche warrants > a new keyword, either. Maybe a pseudo-keyword would be sufficient - comprehension/genexp syntax is pretty solidly specified, so it's less likely to break stuff than in general syntax. I'm really not liking the proposed syntax, but maybe there's an alternative. My first thought on reading this proposal is: "Ah, it's like SQL's 'HAVING' keyword". The trouble is that SQL can identify its columns, but Python doesn't have an easy syntax for "the thing you're about to return". Something like: [abs(x) for x in numbers having _ > 5] The semantics would be equivalent to: def (): result = [] for x in numbers: _ = abs(x) if _ > 5: result.append(_) return result A 'having' clause (and, by the way, I'm not too enthused about the name, but I'm using it as a placeholder because of SQL's use) would have to go after all 'for' and 'if' clauses, and would have access to one special name (maybe _ because of its use in interactive mode, or maybe something else) which holds the value that would be returned. A simple filter-out-the-false-ones could do this: [regex.match(line) for line in lines having _] Of course, people could just switch from 'if' to 'having' across the board, but this has the same consequences that it does in SQL: now *every* row has to be fully processed, only to be discarded at the last step. Using 'having' with no underscore would violate common sense and good style. ChrisA From p.f.moore at gmail.com Tue Mar 8 11:02:45 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 8 Mar 2016 16:02:45 +0000 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: Message-ID: On 8 March 2016 at 14:17, Allan Clark wrote: > tl;dr What is support like for adding an 'as' clause to comprehension > syntax? In order to allow map-then-filter, it might look like something > this: > > [y for x in numbers if abs(x) as y > 5] > > I wish to propose an extension to Python comprehension syntax in an attempt > to make it applicable in more areas. I understand your motivation here, but Python isn't really a functional programming language and isn't intended to be. So comprehensions, while powerful, are only appropriate for relatively simple cases - at the point where they are no longer sufficiently powerful, it's generally better to use an explicit for loop or generator function. (In actual fact, I'm finding more and more these days that I only use comprehensions for very simple cases, and switch to explicit loops quite early). So what seems to me to be missing from your proposal is an explanation of how extending the comprehension syntax is an improvement over not using a comprehension at all. You suggest [y for x in numbers if abs(x) as y > 5] as a simpler alternative to [y for y in (abs(x) for x in numbers) if y > 5] but I'd be much more likely to write results = [] for x in numbers: y = abs(x) if y > 5: results.append(y) or def bounded(it, lower): for val in it: absval = abs(val) if absval > lower: yield absval list(bounded(numbers, 5)) Obviously real world examples would be better than artificial ones, as artificially simple examples make terse notation look better... And in the example using a generator, you'd be able to give it a far more meaningful name with a bit of real-life domain terminology. Paul From allan.clark at gmail.com Tue Mar 8 11:04:34 2016 From: allan.clark at gmail.com (Allan Clark) Date: Tue, 8 Mar 2016 16:04:34 +0000 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: Message-ID: On 8 March 2016 at 15:20, Chris Angelico wrote: > On Wed, Mar 9, 2016 at 1:59 AM, ?manuel Barry wrote: > > I?m going be confused by what sort of name binding it does. A quick > glance > > at the keywords list tells me that no currently-existing keyword is ?the > > obvious choice? for this purpose. I don?t think such a small niche > warrants > > a new keyword, either. > > Yes I certainly agree that this kind of extra syntax should not use a new keyword. > Maybe a pseudo-keyword would be sufficient - comprehension/genexp > syntax is pretty solidly specified, so it's less likely to break stuff > than in general syntax. > > I'm really not liking the proposed syntax, but maybe there's an > alternative. My first thought on reading this proposal is: "Ah, it's > like SQL's 'HAVING' keyword". The trouble is that SQL can identify its > columns, but Python doesn't have an easy syntax for "the thing you're > about to return". Something like: > > [abs(x) for x in numbers having _ > 5] > > The semantics would be equivalent to: > def (): > result = [] > for x in numbers: > _ = abs(x) > if _ > 5: result.append(_) > return result > > I have to agree that I'm not mad-keen on the syntax I proposed either, I just couldn't come up with an elegant alternative, and I guess I hoped someone else might be. > A 'having' clause (and, by the way, I'm not too enthused about the > name, but I'm using it as a placeholder because of SQL's use) would > have to go after all 'for' and 'if' clauses, and would have access to > one special name (maybe _ because of its use in interactive mode, or > maybe something else) which holds the value that would be returned. A > simple filter-out-the-false-ones could do this: > > [regex.match(line) for line in lines having _] > > If I were introducing a new clause, I would probably suggest 'where' from Haskell, which would allow multiple definitions perhaps. I'm not sure, I guess there are two questions, 1) is this a need that should be addressed at all, and 2) if so, what is the right syntax extension. I think it more important to answer 1 and then the syntax-related endless threads can be commenced. ;) Although of course the answer to 1 might be "only if someone proposes an especially elegant syntax". > Of course, people could just switch from 'if' to 'having' across the > board, but this has the same consequences that it does in SQL: now > *every* row has to be fully processed, only to be discarded at the > last step. Using 'having' with no underscore would violate common > sense and good style. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From allan.clark at gmail.com Tue Mar 8 11:20:21 2016 From: allan.clark at gmail.com (Allan Clark) Date: Tue, 8 Mar 2016 16:20:21 +0000 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: Message-ID: On 8 March 2016 at 16:02, Paul Moore wrote: > On 8 March 2016 at 14:17, Allan Clark wrote: > > tl;dr What is support like for adding an 'as' clause to comprehension > > syntax? In order to allow map-then-filter, it might look like something > > this: > > > > [y for x in numbers if abs(x) as y > 5] > > > > I wish to propose an extension to Python comprehension syntax in an > attempt > > to make it applicable in more areas. > > [snip, fair comment] > > So what seems to me to be missing from your proposal is an explanation > of how extending the comprehension syntax is an improvement over not > using a comprehension at all. > Yeah that's a pretty fair observation. > You suggest > > [y for x in numbers if abs(x) as y > 5] > > as a simpler alternative to > > [y for y in (abs(x) for x in numbers) if y > 5] > > but I'd be much more likely to write > > results = [] > for x in numbers: > y = abs(x) > if y > 5: > results.append(y) > > or > > def bounded(it, lower): > for val in it: > absval = abs(val) > if absval > lower: > yield absval > > list(bounded(numbers, 5)) > > Again, fair. Not sure I can quite articulate why I would prefer a comprehension here. A very weak argument would be that such code tends to change, and when it does it may get morphed back into something that *can* be done with a comprehension, but you might end up leaving it as a for-loop with append. > Obviously real world examples would be better than artificial ones, as > artificially simple examples make terse notation look better... And in > the example using a generator, you'd be able to give it a far more > meaningful name with a bit of real-life domain terminology Yeah agreed. I would note that in a real-world example you are giving a name to something that is forced to calculate *and* filter. So your name is going to end-up being something like, say "is_registered_and_voting_for", or "is_highest_tax_bracket_and_is_taxed". Which you might not actually write, and instead opt for something like "tax_amount". Even in your code for my artificial example your generator is named "bounded" but that does not sound like it is doing any filtering at all, it sounds like it is simply bounding all values. Of course to be fair, I didn't give a name for that at all, and probably I want to give a name for the resulting list. Although of course we both need to do that, but at least with a comprehension you only have to come up with a name for the result, not for the result and the associated generator. -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Tue Mar 8 11:21:02 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 8 Mar 2016 16:21:02 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: Message-ID: On 08/03/2016 15:03, Chris Angelico wrote: > On Wed, Mar 9, 2016 at 12:17 AM, Paul Moore wrote: >> tl; dr; I'm against making the unversioned name "python" (or "py") >> forever mean Python 2. There are justifications for it on Unix, but >> none of those apply on Windows. > > https://www.python.org/dev/peps/pep-0394/#exclusion-of-ms-windows > > On Unix, yes, but not on Windows. > > +1 for making py.exe now default to Python 3. > > ChrisA +1 from me as well, it's irritating having to type "py -3" every time. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From p.f.moore at gmail.com Tue Mar 8 12:18:15 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 8 Mar 2016 17:18:15 +0000 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: Message-ID: On 8 March 2016 at 16:20, Allan Clark wrote: > Again, fair. Not sure I can quite articulate why I would prefer a > comprehension here. > A very weak argument would be that such code tends to change, and when it > does it may get morphed back into something that *can* be done with a > comprehension, but you might end up leaving it as a for-loop with append. That's actually a very good point - there's quite a distinct "break point" where you have to rewrite code as a loop rather than a comprehension, and for maintainability purposes that switch tends to be permanent. So delaying the point where you need to switch (assuming the comprehension form is readable and maintainable) is a fair goal. There's also the point that much code is actually one-off scripts, and the break point can be very different in such code - "how I think of it" has a much higher weight in such a situation (often even greater than "is it maintainable" for completely throwaway code). The problem then becomes one of finding a syntax that is natural and not forced. That's hard, particularly with things like the high bar on introducing new keywords meaning you're trying to reuse words that "sort of" suit the situation. >> Obviously real world examples would be better than artificial ones, as >> artificially simple examples make terse notation look better... And in >> the example using a generator, you'd be able to give it a far more >> meaningful name with a bit of real-life domain terminology > > Yeah agreed. > I would note that in a real-world example you are giving a name to something > that is forced to calculate *and* filter. So your name is going to end-up > being something like, say "is_registered_and_voting_for", or > "is_highest_tax_bracket_and_is_taxed". Which you might not actually write, > and instead opt for something like "tax_amount". Even in your code for my > artificial example your generator is named "bounded" but that does not sound > like it is doing any filtering at all, it sounds like it is simply bounding > all values. Of course to be fair, I didn't give a name for that at all, and > probably I want to give a name for the resulting list. Although of course we > both need to do that, but at least with a comprehension you only have to > come up with a name for the result, not for the result and the associated > generator. Again, a good point. Compound clauses make for bad names typically, so if you're thinking in terms of "calculate and filter" it'll be hard to think of a really good name. Maybe a better approach is to look at chaining of individual building blocks. The [y for y in (abs(x) for x in numbers) if y > 5] approach takes that form, but the nesting hides the fact. In some code I wrote today, I used the form data = [abs(x) for x in numbers] data = [x for x in data if x > 5] You can even use generators data = (abs(x) for x in numbers) data = (x for x in data if x > 5) list(data) to make the calculations lazy. To me, that makes the step by step "calculate, then filter" pipeline explicit, and is actually really readable. Of course, once again this is the sort of thing that's very much about personal opinion, and you may hate that style (particularly using the generic variable name "data" over and over). So, overall I'd say I'm not against the idea, but there are some reasonably good alternatives already available, and so coming up with something that's compellingly better than the status quo is going to be a hard job. I'm glad we're having the discussion, though - we need to (re-) explore questions like this to avoid the language stagnating. And who knows when the inspiration will strike? :-) Paul From ethan at stoneleaf.us Tue Mar 8 12:26:26 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Mar 2016 09:26:26 -0800 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: Message-ID: <56DF0B42.5030200@stoneleaf.us> On 03/08/2016 06:59 AM, ?manuel Barry wrote: > The general concept is name binding, so if I see something like > > [x for x, y in some_iterable as y > 5] > > I?m going be confused by what sort of name binding it does. I think everyone would be confused because that code is wrong: it's assigning the `iterable` as `y`, and then comparing that to the value `5`. More realistic (and correct ;) might be: [z for x, y in some_iterable if x+y as z > 10] and the result is a list of numbers whose combined value is greater than 10. A name binding is in fact occuring, so `as` is a fine choice. -- ~Ethan~ From rosuav at gmail.com Tue Mar 8 13:27:13 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Mar 2016 05:27:13 +1100 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: <56DF0B42.5030200@stoneleaf.us> References: <56DF0B42.5030200@stoneleaf.us> Message-ID: On Wed, Mar 9, 2016 at 4:26 AM, Ethan Furman wrote: > On 03/08/2016 06:59 AM, ?manuel Barry wrote: > >> The general concept is name binding, so if I see something like >> >> [x for x, y in some_iterable as y > 5] >> >> I?m going be confused by what sort of name binding it does. > > > I think everyone would be confused because that code is wrong: it's > assigning the `iterable` as `y`, and then comparing that to the value `5`. > > More realistic (and correct ;) might be: > > [z for x, y in some_iterable if x+y as z > 10] > > and the result is a list of numbers whose combined value is greater than 10. > > A name binding is in fact occuring, so `as` is a fine choice. Implement that, and people will ask why they can't then unroll that: def (): result = [] for x, y in some_iterable: if x+y as z > 10: # SyntaxError result.append(z) return z I'm not sure people want name bindings in general expressions (note that this can't be a feature of the 'if' statement, as it's capturing and then continuing on), but as I see it, that's the only consistent way to do what you're attempting there. ChrisA From ethan at stoneleaf.us Tue Mar 8 13:32:24 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Mar 2016 10:32:24 -0800 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: <56DF0B42.5030200@stoneleaf.us> Message-ID: <56DF1AB8.5010101@stoneleaf.us> On 03/08/2016 10:27 AM, Chris Angelico wrote: > Implement that, and people will ask why they can't then unroll that: > > def (): > result = [] > for x, y in some_iterable: > if x+y as z > 10: # SyntaxError > result.append(z) > return z Seriously? def blah(): result = [] for x, y in some_iterable: z = x + y if z > 10: result.append(z) return result # not z That doesn't seem too difficult. ;) -- ~Ethan~ From rosuav at gmail.com Tue Mar 8 13:34:15 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Mar 2016 05:34:15 +1100 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: <56DF1AB8.5010101@stoneleaf.us> References: <56DF0B42.5030200@stoneleaf.us> <56DF1AB8.5010101@stoneleaf.us> Message-ID: On Wed, Mar 9, 2016 at 5:32 AM, Ethan Furman wrote: > On 03/08/2016 10:27 AM, Chris Angelico wrote: > >> Implement that, and people will ask why they can't then unroll that: >> >> def (): >> result = [] >> for x, y in some_iterable: >> if x+y as z > 10: # SyntaxError >> result.append(z) >> return z > > > Seriously? > > def blah(): > result = [] > for x, y in some_iterable: > z = x + y > if z > 10: > result.append(z) > return result # not z > > That doesn't seem too difficult. ;) Ah yes, but that's not how you've written it in the comprehension. You wrote it with 'as'. Believe you me, people WILL expect that outside of comprehensions. Otherwise, you have to explain why a name binding is legal in a condition in a comprehension, but not in any other expression. ChrisA From brett at python.org Tue Mar 8 13:47:26 2016 From: brett at python.org (Brett Cannon) Date: Tue, 08 Mar 2016 18:47:26 +0000 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: References: <1457370435.3901655.542029746.6A4C4A69@webmail.messagingengine.com> Message-ID: On Mon, 7 Mar 2016 at 15:02 Terry Reedy wrote: > On 3/7/2016 12:22 PM, Chris Angelico wrote: > > > In theory, though, since it doesn't change syntax, it could be something > like: > > > > import sys > > sys.package_mode_ACTIVATE() > > > > The trouble is that doing this after _any_ imports (including the ones > > that are done before your script starts) will risk those imports being > > resolved from local files. > > Barring a setting in $PYTHONPATH, Python's startup imports will not be > resolved from local files. It appears that '' or scriptdir are only > added to sys.path afterwards. I base this on an experiment where I > tried to shadow one of the Python-coded /lib modules with a local file. > > C-coded builtin modules cannot be shadowed. I discovered this by > experiment and then found > > "Python includes a number of default finders and importers. The first > one knows how to locate built-in modules, and the second knows how to > locate frozen modules. A third default finder searches an import path > for modules. The import path is a list of locations that may name file > system paths or zip files." > > https://docs.python.org/3/reference/import.html#finders-and-loaders > > I consider the behavior of local versus stdlib imports depending on the > stdlib implementation language to be somewhat problematical. There was > discussion of shadowing, I believe here, a few months ago (sometime > after 3.5.0 was released). > There was a discussion about shadowing modules that wasn't too long ago, but I don't remember when. Basically this idea of dropping '' from sys.path so that the local directory isn't included comes up on occasion. Usually it's someone teaching a beginner who named a module something in the stdlib and then got bit by this that suggests it. This discussion usually comes down to "help the beginners" vs "don't break compatibility!" The discussion is slightly nuanced, though, because '' becomes the location of the file passed on the command-line when it's specified so that things don't have to be in a package to be run (Chris' proposal deals with this by forcing the package concept, although you don't need an imaginary __init__.py since we have the concept of namespace packages). But if you drop the directory that a script is contained in, how do you import any packages that are in that directory? What about __main__.py files in the top directory of code checkouts for easy testing of making executable zipfiles? It's a slippery slope, hence why the semantics have not changed. -Brett > > > Having "import X" after activating package > > mode is useless if X gets resolved from sys.modules, and it'd be a > > nightmare to debug. > > > > If the -p parameter is a problem, maybe there could be an environment > > variable that changes the default? Then people could opt-in to > > pseudo-package mode globally, run all their tests, and see what stops > > working. > > > > FWIW, I do want this to eventually become the default behaviour. But > > it wouldn't be any time soon; in the meantime, it would be easy enough > > to recommend that people just "always do this to be safe" (in the same > > way that Python 2 didn't make new-style classes the default, but > > everyone's advised to "always subclass object"). By making it a > > long-term non-default feature, Python gets to provide safety without > > breaking anyone's code. > > You might want to look at previous discussions of shadowing if you can > find them. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Mar 8 13:53:52 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Mar 2016 05:53:52 +1100 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: References: <1457370435.3901655.542029746.6A4C4A69@webmail.messagingengine.com> Message-ID: On Wed, Mar 9, 2016 at 5:47 AM, Brett Cannon wrote: > Basically this idea of dropping '' from sys.path so that the local directory > isn't included comes up on occasion. Usually it's someone teaching a > beginner who named a module something in the stdlib and then got bit by this > that suggests it. This discussion usually comes down to "help the beginners" > vs "don't break compatibility!" The discussion is slightly nuanced, though, > because '' becomes the location of the file passed on the command-line when > it's specified so that things don't have to be in a package to be run > (Chris' proposal deals with this by forcing the package concept, although > you don't need an imaginary __init__.py since we have the concept of > namespace packages). But if you drop the directory that a script is > contained in, how do you import any packages that are in that directory? Packages in that directory would be subpackages of the pseudo-package. Just as "from . import module" would fetch module.py from the script dir, "from .pkg import module" would fetch module.py from the pkg subdirectory. If the package truly stands alone (such that you want it to be imported as "pkg.module" - you can't say "from . import pkg.module", at least not currently), then possibly you want to add the directory to sys.path. > What about __main__.py files in the top directory of code checkouts for easy > testing of making executable zipfiles? It's a slippery slope, hence why the > semantics have not changed. Not sure what you mean. What about them? Wouldn't they be scripts just like any other, such that they'd be loading local files with "from . import X"? ChrisA From joejev at gmail.com Tue Mar 8 13:57:12 2016 From: joejev at gmail.com (Joseph Jevnik) Date: Tue, 8 Mar 2016 13:57:12 -0500 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: <56DF0B42.5030200@stoneleaf.us> <56DF1AB8.5010101@stoneleaf.us> Message-ID: Haskell has a feature like this in comprehensions where one may write: [r + 1| n <- [1..10], let r = n * 3, r `rem` 4 == 0] Here we are sharing the definition of `r` in both the predicate and the value expresions, we can also use the name in different expressions in both contexts. If we were to translate this to python syntax we could have something like: [r + 1 for n in range(1, 11) for n * 3 as r if r % 4 == 0] There is no reason that the name binding needs to be a part of the predicate expression, they can just be seperate clauses. I think the `for expr as name` is nice because it matches the order that comprehensions over multiple iterators are evaluated like: `[n for n in ns for m in ms]`. I personally just write `map` and `filter` and then have the control to make the correct function get called first but adding something like this might make me more inclined to use comprehensions for the simple cases. On Tue, Mar 8, 2016 at 1:34 PM, Chris Angelico wrote: > On Wed, Mar 9, 2016 at 5:32 AM, Ethan Furman wrote: > > On 03/08/2016 10:27 AM, Chris Angelico wrote: > > > >> Implement that, and people will ask why they can't then unroll that: > >> > >> def (): > >> result = [] > >> for x, y in some_iterable: > >> if x+y as z > 10: # SyntaxError > >> result.append(z) > >> return z > > > > > > Seriously? > > > > def blah(): > > result = [] > > for x, y in some_iterable: > > z = x + y > > if z > 10: > > result.append(z) > > return result # not z > > > > That doesn't seem too difficult. ;) > > Ah yes, but that's not how you've written it in the comprehension. You > wrote it with 'as'. Believe you me, people WILL expect that outside of > comprehensions. Otherwise, you have to explain why a name binding is > legal in a condition in a comprehension, but not in any other > expression. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Mar 8 14:01:42 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Mar 2016 06:01:42 +1100 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: <56DF0B42.5030200@stoneleaf.us> <56DF1AB8.5010101@stoneleaf.us> Message-ID: On Wed, Mar 9, 2016 at 5:57 AM, Joseph Jevnik wrote: > If we were to translate this to python syntax we could have something like: > [r + 1 for n in range(1, 11) for n * 3 as r if r % 4 == 0] > There is no reason that the name binding needs to be a part of the predicate > expression, they can just be seperate clauses. I think the `for expr as > name` is nice because it matches the order that comprehensions over multiple > iterators are evaluated like: `[n for n in ns for m in ms]`. That's somewhat more appealing. Not enthused about "for expr as name"; maybe "with expr as name"? Aside from not calling __enter__ and __exit__, it's the same kind of operation that a with block does. But the semantic difference isn't a good thing. ChrisA From joejev at gmail.com Tue Mar 8 14:07:51 2016 From: joejev at gmail.com (Joseph Jevnik) Date: Tue, 8 Mar 2016 14:07:51 -0500 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: <56DF0B42.5030200@stoneleaf.us> <56DF1AB8.5010101@stoneleaf.us> Message-ID: I was going to write `with expr as name` but I felt that people might find it too conficting with the other use of the `with` keyword. I personally find that it reads very well. We could always go with a new keyword like: `bind expr as name` but that seems pretty heavy. In defense of the `for expr as name` proposal, it does match pretty closely to the nested for statements in a comprehension. On Tue, Mar 8, 2016 at 2:01 PM, Chris Angelico wrote: > On Wed, Mar 9, 2016 at 5:57 AM, Joseph Jevnik wrote: > > If we were to translate this to python syntax we could have something > like: > > [r + 1 for n in range(1, 11) for n * 3 as r if r % 4 == 0] > > There is no reason that the name binding needs to be a part of the > predicate > > expression, they can just be seperate clauses. I think the `for expr as > > name` is nice because it matches the order that comprehensions over > multiple > > iterators are evaluated like: `[n for n in ns for m in ms]`. > > That's somewhat more appealing. Not enthused about "for expr as name"; > maybe "with expr as name"? Aside from not calling __enter__ and > __exit__, it's the same kind of operation that a with block does. But > the semantic difference isn't a good thing. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Mar 8 14:19:39 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 8 Mar 2016 19:19:39 +0000 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: <56DF0B42.5030200@stoneleaf.us> <56DF1AB8.5010101@stoneleaf.us> Message-ID: On 8 March 2016 at 18:57, Joseph Jevnik wrote: > If we were to translate this to python syntax we could have something like: > [r + 1 for n in range(1, 11) for n * 3 as r if r % 4 == 0] Quite seriously, I have no idea what that means. If I saw it in a code review, I'd insist that it were rewritten. Paul From python at mrabarnett.plus.com Tue Mar 8 14:49:16 2016 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 8 Mar 2016 19:49:16 +0000 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: <56DF0B42.5030200@stoneleaf.us> <56DF1AB8.5010101@stoneleaf.us> Message-ID: <56DF2CBC.6000701@mrabarnett.plus.com> On 2016-03-08 19:07, Joseph Jevnik wrote: > I was going to write `with expr as name` but I felt that people might > find it too conficting with the other use of the `with` keyword. I > personally find that it reads very well. We could always go with a new > keyword like: `bind expr as name` but that seems pretty heavy. In > defense of the `for expr as name` proposal, it does match pretty closely > to the nested for statements in a comprehension. > Usually, 'for' is followed by the name(s) that will be bound. There's an example of that just before it! "for x in ..." but "for ... as x". > On Tue, Mar 8, 2016 at 2:01 PM, Chris Angelico > wrote: > > On Wed, Mar 9, 2016 at 5:57 AM, Joseph Jevnik > wrote: > > If we were to translate this to python syntax we could have something like: > > [r + 1 for n in range(1, 11) for n * 3 as r if r % 4 == 0] > > There is no reason that the name binding needs to be a part of the predicate > > expression, they can just be seperate clauses. I think the `for expr as > > name` is nice because it matches the order that comprehensions over multiple > > iterators are evaluated like: `[n for n in ns for m in ms]`. > > That's somewhat more appealing. Not enthused about "for expr as name"; > maybe "with expr as name"? Aside from not calling __enter__ and > __exit__, it's the same kind of operation that a with block does. But > the semantic difference isn't a good thing. > From ethan at stoneleaf.us Tue Mar 8 14:51:33 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Mar 2016 11:51:33 -0800 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: <56DF0B42.5030200@stoneleaf.us> <56DF1AB8.5010101@stoneleaf.us> Message-ID: <56DF2D45.3080501@stoneleaf.us> On 03/08/2016 10:34 AM, Chris Angelico wrote: > On Wed, Mar 9, 2016 at 5:32 AM, Ethan Furman wrote: >> On 03/08/2016 10:27 AM, Chris Angelico wrote: >> >>> Implement that, and people will ask why they can't then unroll that: >>> >>> def (): >>> result = [] >>> for x, y in some_iterable: >>> if x+y as z > 10: # SyntaxError >>> result.append(z) >>> return z >> >> >> Seriously? >> >> def blah(): >> result = [] >> for x, y in some_iterable: >> z = x + y >> if z > 10: >> result.append(z) >> return result # not z >> >> That doesn't seem too difficult. ;) > > Ah yes, but that's not how you've written it in the comprehension. You > wrote it with 'as'. Believe you me, people WILL expect that outside of > comprehensions. Otherwise, you have to explain why a name binding is > legal in a condition in a comprehension, but not in any other > expression. So? Enhance the idea to be allowing (expr) as (name) anywhere: if (file.read() as data): # process data I'm cool with that. :) -- ~Ethan~ From rosuav at gmail.com Tue Mar 8 14:54:37 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Mar 2016 06:54:37 +1100 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: <56DF2D45.3080501@stoneleaf.us> References: <56DF0B42.5030200@stoneleaf.us> <56DF1AB8.5010101@stoneleaf.us> <56DF2D45.3080501@stoneleaf.us> Message-ID: On Wed, Mar 9, 2016 at 6:51 AM, Ethan Furman wrote: > So? Enhance the idea to be allowing > > (expr) as (name) anywhere: > > if (file.read() as data): > # process data > > I'm cool with that. :) Well, okay then. Spin that off as its own idea: Name bindings in expressions. It's a coherent idea, but it's likely to see some fairly stiff opposition :) (For what it's worth, I am _not_ in opposition to it.) ChrisA From pavol.lisy at gmail.com Tue Mar 8 15:06:50 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Tue, 8 Mar 2016 21:06:50 +0100 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: Message-ID: 2016-03-08 15:17 GMT+01:00, Allan Clark : [...] > [y for x in numbers if abs(x) as y > 5] [...] > * It would need to be decided whether you allowed multiple 'as' > expression in the condition, particularly using 'and' or 'or' as in 'if > f(a) as x > 5 and f(b) as y > 5' Just small idea not to forget here: [y+z for x in numbers if abs(x) as y > 5 or x**2 as z>100] "or" is short-circuit operator so z could be not (well) defined. From rosuav at gmail.com Tue Mar 8 15:16:21 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Mar 2016 07:16:21 +1100 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: Message-ID: On Wed, Mar 9, 2016 at 7:06 AM, Pavol Lisy wrote: > 2016-03-08 15:17 GMT+01:00, Allan Clark : > [...] >> [y for x in numbers if abs(x) as y > 5] > [...] >> * It would need to be decided whether you allowed multiple 'as' >> expression in the condition, particularly using 'and' or 'or' as in 'if >> f(a) as x > 5 and f(b) as y > 5' > > Just small idea not to forget here: > > [y+z for x in numbers if abs(x) as y > 5 or x**2 as z>100] > > "or" is short-circuit operator so z could be not (well) defined. Enforcing that the order of operations is as you're implying: [y+z for x in numbers if (abs(x) as y) > 5 or (x**2 as z) > 100] If the first condition is true, the second will not be evaluated. It'll be equivalent to: def (): result = [] for x in numbers: y = abs(x) if y > 5: result.append(y+z) else: z = x**2 if z > 100: result.append(y+z) return result So, yeah, that could surprise a *lot* of people. Either an UnboundLocalError, or retaining the value from the previous iteration. Recommendation: If you use name bindings in the second half, use bitwise operators instead: [y+z for x in numbers if (abs(x) as y) > 5 | (x**2 as z) > 100] That'll force both sides to be evaluated. (Downside: It's reliable only if you actually have True and False.) ChrisA From k7hoven at gmail.com Tue Mar 8 15:30:46 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 8 Mar 2016 22:30:46 +0200 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: Message-ID: On Tue, Mar 8, 2016 at 4:17 PM, Allan Clark wrote: > tl;dr What is support like for adding an 'as' clause to comprehension > syntax? In order to allow map-then-filter, it might look like something > this: > > [y for x in numbers if abs(x) as y > 5] > < [...] How about this (at least it reads quite nicely in English): [abs(x) for x in numbers if > 5] -- Koos From bufordsharkley at gmail.com Tue Mar 8 15:46:03 2016 From: bufordsharkley at gmail.com (Mark Mollineaux) Date: Tue, 8 Mar 2016 12:46:03 -0800 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: Message-ID: This map-then-filter shortcoming is one of the most common warts I encounter with python syntax, and I would absolutely love an improvement here. I think ChrisA is closest with: > I'm really not liking the proposed syntax, but maybe there's an alternative. My first thought on reading this proposal is: "Ah, it's like SQL's 'HAVING' keyword". The trouble is that SQL can identify its columns, but Python doesn't have an easy syntax for "the thing you're about to return". Something like: > > [abs(x) for x in numbers having _ > 5] But I wonder, why not just: [abs(x) for x in numbers if _ > 5] Where _ refers to the return expression within a comprehension, and is interpreted specially in this context? On Tue, Mar 8, 2016 at 7:20 AM, Chris Angelico wrote: > On Wed, Mar 9, 2016 at 1:59 AM, ?manuel Barry wrote: > > I?m going be confused by what sort of name binding it does. A quick > glance > > at the keywords list tells me that no currently-existing keyword is ?the > > obvious choice? for this purpose. I don?t think such a small niche > warrants > > a new keyword, either. > > Maybe a pseudo-keyword would be sufficient - comprehension/genexp > syntax is pretty solidly specified, so it's less likely to break stuff > than in general syntax. > > I'm really not liking the proposed syntax, but maybe there's an > alternative. My first thought on reading this proposal is: "Ah, it's > like SQL's 'HAVING' keyword". The trouble is that SQL can identify its > columns, but Python doesn't have an easy syntax for "the thing you're > about to return". Something like: > > [abs(x) for x in numbers having _ > 5] > > The semantics would be equivalent to: > def (): > result = [] > for x in numbers: > _ = abs(x) > if _ > 5: result.append(_) > return result > > A 'having' clause (and, by the way, I'm not too enthused about the > name, but I'm using it as a placeholder because of SQL's use) would > have to go after all 'for' and 'if' clauses, and would have access to > one special name (maybe _ because of its use in interactive mode, or > maybe something else) which holds the value that would be returned. A > simple filter-out-the-false-ones could do this: > > [regex.match(line) for line in lines having _] > > Of course, people could just switch from 'if' to 'having' across the > board, but this has the same consequences that it does in SQL: now > *every* row has to be fully processed, only to be discarded at the > last step. Using 'having' with no underscore would violate common > sense and good style. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From joejev at gmail.com Tue Mar 8 15:50:27 2016 From: joejev at gmail.com (Joseph Jevnik) Date: Tue, 8 Mar 2016 15:50:27 -0500 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: Message-ID: Currently `_` is not a keyword or special in any way. How do we resolve the ambiguity with: _ = 10 [x for x in xs if _ > 10] On Tue, Mar 8, 2016 at 3:46 PM, Mark Mollineaux wrote: > This map-then-filter shortcoming is one of the most common warts I > encounter with python syntax, and I would absolutely love an improvement > here. > > I think ChrisA is closest with: > > > I'm really not liking the proposed syntax, but maybe there's an alternative. > My first thought on reading this proposal is: "Ah, it's like SQL's > 'HAVING' keyword". The trouble is that SQL can identify its columns, but > Python doesn't have an easy syntax for "the thing you're about to > return". Something like: > > > > [abs(x) for x in numbers having _ > 5] > > But I wonder, why not just: > > [abs(x) for x in numbers if _ > 5] > > Where _ refers to the return expression within a comprehension, and is > interpreted specially in this context? > > > On Tue, Mar 8, 2016 at 7:20 AM, Chris Angelico wrote: > >> On Wed, Mar 9, 2016 at 1:59 AM, ?manuel Barry wrote: >> > I?m going be confused by what sort of name binding it does. A quick >> glance >> > at the keywords list tells me that no currently-existing keyword is ?the >> > obvious choice? for this purpose. I don?t think such a small niche >> warrants >> > a new keyword, either. >> >> Maybe a pseudo-keyword would be sufficient - comprehension/genexp >> syntax is pretty solidly specified, so it's less likely to break stuff >> than in general syntax. >> >> I'm really not liking the proposed syntax, but maybe there's an >> alternative. My first thought on reading this proposal is: "Ah, it's >> like SQL's 'HAVING' keyword". The trouble is that SQL can identify its >> columns, but Python doesn't have an easy syntax for "the thing you're >> about to return". Something like: >> >> [abs(x) for x in numbers having _ > 5] >> >> The semantics would be equivalent to: >> def (): >> result = [] >> for x in numbers: >> _ = abs(x) >> if _ > 5: result.append(_) >> return result >> >> A 'having' clause (and, by the way, I'm not too enthused about the >> name, but I'm using it as a placeholder because of SQL's use) would >> have to go after all 'for' and 'if' clauses, and would have access to >> one special name (maybe _ because of its use in interactive mode, or >> maybe something else) which holds the value that would be returned. A >> simple filter-out-the-false-ones could do this: >> >> [regex.match(line) for line in lines having _] >> >> Of course, people could just switch from 'if' to 'having' across the >> board, but this has the same consequences that it does in SQL: now >> *every* row has to be fully processed, only to be discarded at the >> last step. Using 'having' with no underscore would violate common >> sense and good style. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Mar 8 15:52:25 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Mar 2016 07:52:25 +1100 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: Message-ID: On Wed, Mar 9, 2016 at 7:46 AM, Mark Mollineaux wrote: > This map-then-filter shortcoming is one of the most common warts I encounter > with python syntax, and I would absolutely love an improvement here. > > I think ChrisA is closest with: > >> I'm really not liking the proposed syntax, but maybe there's an >> alternative. My first thought on reading this proposal is: "Ah, it's like >> SQL's 'HAVING' keyword". The trouble is that SQL can identify its columns, >> but Python doesn't have an easy syntax for "the thing you're about to >> return". Something like: >> >> [abs(x) for x in numbers having _ > 5] > > But I wonder, why not just: > > [abs(x) for x in numbers if _ > 5] > > Where _ refers to the return expression within a comprehension, and is > interpreted specially in this context? That makes it somewhat magical; compare these two conditions: [(base//x) % 10 for x in numbers if x if _] The first one MUST be filtered prior to mapping (else you'll get a ZeroDivisionError), but the second has to be filtered afterwards. Putting the two conditions into one comprehension might not be common, but it's a bit odd to have them mean different things. ChrisA From tritium-list at sdamon.com Tue Mar 8 16:11:04 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Tue, 8 Mar 2016 16:11:04 -0500 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: Message-ID: <56DF3FE8.70600@sdamon.com> -1 py.exe should use whatever was first installed by default. If that's 2.7, then it's 2.7. if its python 9.9, then its python 9.9. but it should use whatever was FIRST installed unless the user changes it. On 3/8/2016 04:24, Paul Moore wrote: > The "py.exe" wrapper on Windows runs the latest Python version > available on the user's PC, but it prefers Python 2 if the user has > both Python 2 and 3 installed. With Python 3.5 being released, is it > not about time that we switched that default to always prefer the > latest version available? > > For scripts, users can always specify a shebang line, exactly as on > Unix. For interactive use, people who prefer Python 2 simply have to > create a py.ini file saying > > [defaults] > python=2 > > At the moment, someone using Python 3 exclusively can use "py" to > start the Python interpreter - and with the changes in the Python 3.5 > installer, defaulting to a per-user install that's not on PATH, this > is often the most convenient approach. But if that person installs > Python 2 (say, to test some code on that version) py.exe silently > switches to using Python 2. And it does so regardless of whether you > specify "make this the default Python" in the installer. > > It seems to me that this behaviour is much more unexpected than the > equivalent of a Python user installing Python 3 (py.exe switches to > Python 3, but that's "obviously" the latest version). > > So I suggest that in Python 3.6, we (finally) switch py.exe to run the > latest version of Python on the user's machine *regardless of the > major version*. > > This will be a compatibility break, but arguably no more serious than > the fact that if the user installs Python 3.6 and puts it on PATH, > then "python.exe" now points to Python 3.6. To cover people just > installing Python 3.6 for testing, we could check in the installer and > if the user only currently has Python 2 installed, has no py.ini, and > doesn't select "make this my default Python" for Python 3.6, then we > write a py.ini defaulting to Python 2. I don't know if that's worth > the effort, it depends largely on how many people will be affected by > the change. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ethan at stoneleaf.us Tue Mar 8 16:17:55 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Mar 2016 13:17:55 -0800 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DF3FE8.70600@sdamon.com> References: <56DF3FE8.70600@sdamon.com> Message-ID: <56DF4183.4000308@stoneleaf.us> On 03/08/2016 01:11 PM, Alexander Walters wrote: > -1 py.exe should use whatever was first installed by default. If that's > 2.7, then it's 2.7. if its python 9.9, then its python 9.9. but it > should use whatever was FIRST installed unless the user changes it. Which is still asking for a change from what happens now; i.e. if 3.3 is installed first, then 2.7, py.exe will default to 2.7. I agree whichever is first installed should be default; I also think installer should have an option to change the py.exe default to the version being installed. -- ~Ethan~ From ian.g.kelly at gmail.com Tue Mar 8 16:29:11 2016 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Tue, 8 Mar 2016 14:29:11 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DF4183.4000308@stoneleaf.us> References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> Message-ID: On Tue, Mar 8, 2016 at 2:17 PM, Ethan Furman wrote: > On 03/08/2016 01:11 PM, Alexander Walters wrote: > >> -1 py.exe should use whatever was first installed by default. If that's >> 2.7, then it's 2.7. if its python 9.9, then its python 9.9. but it >> should use whatever was FIRST installed unless the user changes it. > > > Which is still asking for a change from what happens now; i.e. if 3.3 is > installed first, then 2.7, py.exe will default to 2.7. > > I agree whichever is first installed should be default; I also think > installer should have an option to change the py.exe default to the version > being installed. It would also be nice to have a way to change the default without needing to rerun an installer. From tritium-list at sdamon.com Tue Mar 8 16:34:55 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Tue, 8 Mar 2016 16:34:55 -0500 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> Message-ID: <56DF457F.5090401@sdamon.com> On 3/8/2016 16:29, Ian Kelly wrote: > On Tue, Mar 8, 2016 at 2:17 PM, Ethan Furman wrote: >> On 03/08/2016 01:11 PM, Alexander Walters wrote: >> >>> -1 py.exe should use whatever was first installed by default. If that's >>> 2.7, then it's 2.7. if its python 9.9, then its python 9.9. but it >>> should use whatever was FIRST installed unless the user changes it. >> >> Which is still asking for a change from what happens now; i.e. if 3.3 is >> installed first, then 2.7, py.exe will default to 2.7. >> >> I agree whichever is first installed should be default; I also think >> installer should have an option to change the py.exe default to the version >> being installed. > It would also be nice to have a way to change the default without > needing to rerun an installer. I am all for more configuration ex post facto. We are talking about the tyranny of the default though, which is complicated by the fact that it wont see the light of day until 3.6 (unless we backport a modified version of py.exe in future releases of 2.7 and 3.*). From steve.dower at python.org Tue Mar 8 17:00:36 2016 From: steve.dower at python.org (Steve Dower) Date: Tue, 8 Mar 2016 14:00:36 -0800 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DF457F.5090401@sdamon.com> References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> Message-ID: <56DF4B84.6000702@python.org> On 08Mar2016 1334, Alexander Walters wrote: > > On 3/8/2016 16:29, Ian Kelly wrote: >> On Tue, Mar 8, 2016 at 2:17 PM, Ethan Furman wrote: >>> On 03/08/2016 01:11 PM, Alexander Walters wrote: >>> >>>> -1 py.exe should use whatever was first installed by default. If >>>> that's >>>> 2.7, then it's 2.7. if its python 9.9, then its python 9.9. but it >>>> should use whatever was FIRST installed unless the user changes it. >>> >>> Which is still asking for a change from what happens now; i.e. if 3.3 is >>> installed first, then 2.7, py.exe will default to 2.7. >>> >>> I agree whichever is first installed should be default; I also think >>> installer should have an option to change the py.exe default to the >>> version >>> being installed. >> It would also be nice to have a way to change the default without >> needing to rerun an installer. > I am all for more configuration ex post facto. We are talking about the > tyranny of the default though, which is complicated by the fact that it > wont see the light of day until 3.6 (unless we backport a modified > version of py.exe in future releases of 2.7 and 3.*). "First installed" is a very difficult proposition, especially if you want versions of Python pre-3.6 to participate. "Last installed" is slightly better, though not easy to do without triggering systems that detect corrupted installations. Personally I think "latest version" is the best default (compared to the current "latest 2.x version"), so I'm +1 on removing the assumption that "PY_PYTHON" is "2" by default. Users can change their own default with any of the mechanisms described at https://docs.python.org/3/using/windows.html#customization Cheers, Steve From bufordsharkley at gmail.com Tue Mar 8 17:10:05 2016 From: bufordsharkley at gmail.com (Mark Mollineaux) Date: Tue, 8 Mar 2016 14:10:05 -0800 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: Message-ID: > Currently `_` is not a keyword or special in any way. How do we resolve the ambiguity with: > _ = 10 > [x for x in xs if _ > 10] It's special within the REPL (referring to the last returned expression), so it would at least have some precedence in python to be made special in another (somewhat similar) context... Too, there's already special scoping for list comprehensions. > [(base//x) % 10 for x in numbers if x if _] > >The first one MUST be filtered prior to mapping (else you'll get a ZeroDivisionError), but the second has to be filtered afterwards. Putting the two conditions into one comprehension might not be common, but it's a bit odd to have them mean different things. Agreed that two ifs applied at different times is fairly odd, but as long as _ is known to be magical, it's fairly intuitive, in my opinion. Also, Koos's offering above: > [abs(x) for x in numbers if > 5] Is also very nice, but I don't like it when the filtering is: [abs(x) for x in numbers if] On Tue, Mar 8, 2016 at 12:52 PM, Chris Angelico wrote: > On Wed, Mar 9, 2016 at 7:46 AM, Mark Mollineaux > wrote: > > This map-then-filter shortcoming is one of the most common warts I > encounter > > with python syntax, and I would absolutely love an improvement here. > > > > I think ChrisA is closest with: > > > >> I'm really not liking the proposed syntax, but maybe there's an > >> alternative. My first thought on reading this proposal is: "Ah, it's > like > >> SQL's 'HAVING' keyword". The trouble is that SQL can identify its > columns, > >> but Python doesn't have an easy syntax for "the thing you're about to > >> return". Something like: > >> > >> [abs(x) for x in numbers having _ > 5] > > > > But I wonder, why not just: > > > > [abs(x) for x in numbers if _ > 5] > > > > Where _ refers to the return expression within a comprehension, and is > > interpreted specially in this context? > > That makes it somewhat magical; compare these two conditions: > > [(base//x) % 10 for x in numbers if x if _] > > The first one MUST be filtered prior to mapping (else you'll get a > ZeroDivisionError), but the second has to be filtered afterwards. > Putting the two conditions into one comprehension might not be common, > but it's a bit odd to have them mean different things. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Tue Mar 8 17:16:29 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Tue, 08 Mar 2016 14:16:29 -0800 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: <56DF0B42.5030200@stoneleaf.us> <56DF1AB8.5010101@stoneleaf.us> <56DF2D45.3080501@stoneleaf.us> Message-ID: <56DF4F3D.9020109@brenbarn.net> On 2016-03-08 11:54, Chris Angelico wrote: > On Wed, Mar 9, 2016 at 6:51 AM, Ethan Furman wrote: >> >So? Enhance the idea to be allowing >> > >> >(expr) as (name) anywhere: >> > >> > if (file.read() as data): >> > # process data >> > >> >I'm cool with that. :) > Well, okay then. Spin that off as its own idea: Name bindings in > expressions. It's a coherent idea, but it's likely to see some fairly > stiff opposition :) > > (For what it's worth, I am_not_ in opposition to it.) I agree that that option should be considered. I don't see that much value in the current proposals because they only handle the case where you want to filter the returned values based on those same values, but not the case where you want to filter (before or after the map) based on some criterion that re-uses a computed expression, or where you want to re-use an expression that's not in a comprehension at all. Something like: [some_dict[x.lower()] for x in whatever if x.lower() not in exclusion_list and x.lower() in some_dict] Even though this filter is being done before the map, it's still awkward, because you have to keep repeating the .lower(). If we had assignment-as-expression, you could do [some_dict[lx] for x in whatever if lx not in exclusion_list and lx in some_dict where lx=x.lower()] (or whatever the syntax may be). Of course, that example is rather silly since the version with the "where" is barely shorter :-). But hopefully the point is clear. I often find myself doing awkward things like the first example in numerical situations where I want to collect one thing while filtering and mapping based on something else (e.g., work with numbers while filtering and mapping based on their logs or something). Moreover, a general-purpose assignment-as-expression would also be usable in other contexts besides comprehensions. I run into this same repeated-expression thing when doing computations in pandas using .apply(), where you do .apply(lambda x: ...), and maybe the ... involves a repeated expression involving x. With assignment-as-expression you could .apply(lambda x: ... where temp=f(x)) or what have you I think adding assignment-as-expression would be a significant change to Python. I'm not sure whether it would overall be a good change. Almost all the cases where I really feel like I want this are cases where I'm working with data interactively, and it really is easier to use a lambda than define a separate one-off function. For anything larger-scale, I agree that the "problem" can be solved by taking a deep breath and writing a separate function or generator comprehension to do what you want. As someone mentioned earlier on this thread, the original motivating example: foo = [abs(x) for x in numbers if abs(x) > 5] already has a crystal clear solution: abses = (abs(x) for x in numbers) foo = [x for x in abses if x > 5] There is really nothing wrong with this existing solution unless you're working interactively and want to do things more tersely. But if you're doing that, you're likely to want to do other kinds of terse expressions besides list comprehensions too. So basically, to me the issue is much more general than "I want to filter after mapping in comprehensions". What I want is to write expressions that re-use an intermediate computation. Some of those are in comprehensions, some aren't. Most of the cases where I want to do this are in interactive situations where I'm using a lot of one-off lambdas and temp expressions. If that's something we want to support more, assignment-as-expression could be quite useful, but if we don't, it's a different story. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From p.f.moore at gmail.com Tue Mar 8 17:18:21 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 8 Mar 2016 22:18:21 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DF4B84.6000702@python.org> References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> Message-ID: On 8 March 2016 at 22:00, Steve Dower wrote: > "First installed" is a very difficult proposition, especially if you want > versions of Python pre-3.6 to participate. "Last installed" is slightly > better, though not easy to do without triggering systems that detect > corrupted installations. Also, any dependency on installation order gets *very* messy if someone installs and uninstalls various versions. What does the default become if I uninstall the version that was previously the first (or last) installed? Would we need to keep details of precisely when everything was installed? > Personally I think "latest version" is the best default (compared to the > current "latest 2.x version"), so I'm +1 on removing the assumption that > "PY_PYTHON" is "2" by default. Latest version has the additional advantage of being essentially the current behaviour, just with the special case of "but prefer Python 2 over Python 3 if both are present" removed. (Note that technically the current rule is not simply "latest 2.x version", as if the user only has Python 3 installed, the latest 3.x version is used). The only other option I can think of is "use the one the user selected as the default Python (i.e., introspect the association for ".py" files) but even then we need a default default (!) for people who don't select any default Python. It's also messy to implement. And it's a bigger change from current behaviour. > Users can change their own default with any of the mechanisms described at > https://docs.python.org/3/using/windows.html#customization Absolutely. This should *only* apply when there is no user customisation - and it needs no further configurability, because the current methods are perfectly sufficient. Paul From sjoerdjob at sjec.nl Tue Mar 8 17:21:31 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Tue, 8 Mar 2016 23:21:31 +0100 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: Message-ID: <20160308222131.GA19481@sjoerdjob.com> On Tue, Mar 08, 2016 at 02:17:23PM +0000, Allan Clark wrote: > tl;dr What is support like for adding an 'as' clause to comprehension > syntax? In order to allow map-then-filter, it might look like something > this: > > [y for x in numbers if abs(x) as y > 5] > > I wish to propose an extension to Python comprehension syntax in an attempt > to make it applicable in more areas. I'll first describe the deficiency I > perceive in the current comprehension syntax and then propose my extension. > I'll then note some drawbacks. For the purposes of illustration I'll use > list comprehension syntax but I believe everything I say is equally > applicable to set and dictionary comprehensions. I'll also talk about lists > but again (more or less) everything applies to the more general concept of > iterables. > > Comprehensions are essentially a filter followed by a map operation. So if > you need to perform a filter operation followed by a map operation over a > list, this is pretty convenient in Python. Here we are going to take the > absolute value of all even numbers within the range -10 to 10. > > numbers = range(-10, 10) > [abs(x) for x in numbers if x % 2 == 0] > > However, if you wish to perform a map operation and *then* a filter > operation this is not so convenient, so suppose we wish to obtain the > absolute value of all numbers that have an absolute value larger than 5, we > can do this by calling the mapped method twice: > > abs(x) for x in numbers if abs(x) > 5] > > This is a bit unsatisfying and even impossible in the case that the mapped > method has some side-effect (although arguably if you find yourself in that > situation you have taken a mis-step somewhere). An alternative is to apply > the mapping first: > > [y for y in (abs(x) for x in numbers) if y > 5] > > I have to say I quite like this, as it is pretty explicit, but it is a bit > unsatisfying that you require only one comprehension for a filter-then-map > but two for a map-then-filter. What would be nice is if we could give a > name to the mapped expression and then use that in the filter. > > [abs(x) as y for x in numbers if y > 5] > > I don't like this as it means the order of execution is dependent on > whether there is an 'as' clause, in particular the 'if' clause itself may > do some computation such as in `if f(y) > 5`. > > An altenative is to allow 'as' expressions in the condition, something > like: > > [y for x in numbers if abs(x) as y > 5] > > Note that it would be possible to map to an intermediate result, such as: > > [y**2 for x in numbers if abs(x) as y > 5] > > Alternatively we could put the 'as' in the pattern: > > [y**2 for abs(x) as y in numbers if y > 5] > > I did not like this as it is obscures the fact that 'x' is being set to > each element of 'numbers'. Additionally, we might later wish to adopt a > functional programming idiom in which we use 'as' for deconstructive > assignment whilst giving a name to the entire matched value, for example: > > [p for (x,y) as p if x > y] > > Or more generally: > > (x,y) as a = f(z) > > But that is getting somewhat off-topic. I promised some drawbacks: > * I am confident there are some implementation gotchas nestling in here > somewhere. > * I could imagine how abuse of such a mechanism to lead to pretty > unreadable code. > * I'm still not *that* upset by the explicit map first: `[y for y in > (abs(x) for x in numbers) if y > 5]` > * I could see how it is not immediately obvious what the code does. > * It would need to be decided whether you allowed multiple 'as' > expression in the condition, particularly using 'and' or 'or' as in 'if > f(a) as x > 5 and f(b) as y > 5' > > To summarise: > * It's a touch annoying that comprehensions allow filter-then-map but > not map-then-filter > * Three proposed syntaxes are: > * [abs(x) as y for x in numbers if y > 5] > * [y for x in numbers if abs(x) as y > 5] > * [y**2 for abs(x) as y in numbers if y > 5] > * My favourite is the middle one. > > Finally these seem to currently be syntax errors so we should not break any > existing code. Seeing the many replies, I'm a bit lost as where to best comment. After thinking about it for a while, I think that currently it's not impossible to do what you want with comprehensions, just a bit convoluted. [y for x in numbers for y in [abs(x)] if y > 5] The tricky part being the `for y in [abs(x)]` basically doing what you want: bind the value `abs(x)` to a name (`y`). Benefits: - It requires no new syntax, no new semantics. - You can easily define multiple items at once. [y + z for x in numbers for y, z in [(abs(x), sgn(x)] if y > 5] - You can still use the original value, in contrast to [y for y in (abs(x) for x in numbers) if y > 5] Downside: - Very unreadable, and probably a long way off from being idiomatic Python. As for yet another syntax suggestion (if we want to introduce something). [y for x in numbers with abs(x) as y if y > 5] The benefit is that it reads quite natural: "with expr as name". Another benefit is that keywords get reused. However, that already has semantics outside a comprehension for context-managers. TL;DR: It's possible with [y for x in numbers for y in [abs(x)] if y > 5] But the syntax is ugly enough that it does warrant some extra syntactic sugar (or something with the same semantics but better performance characteristics). From p.f.moore at gmail.com Tue Mar 8 17:58:16 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 8 Mar 2016 22:58:16 +0000 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: <20160308222131.GA19481@sjoerdjob.com> References: <20160308222131.GA19481@sjoerdjob.com> Message-ID: On 8 March 2016 at 22:21, Sjoerd Job Postmus wrote: > As for yet another syntax suggestion (if we want to introduce > something). > > [y for x in numbers with abs(x) as y if y > 5] > > The benefit is that it reads quite natural: "with expr as name". > Another benefit is that keywords get reused. Agreed, this is a plausible suggestion. But rather than just providing an example usage, it would be helpful to see the full syntax of comprehension with the proposed addition. See https://docs.python.org/3/reference/expressions.html#grammar-token-comprehension - note that a comprehension can have arbitrarily many for and if clauses, interspersed in any order as long as the first one is a "for". I'm guessing you'd add "with as " as simply a third option. But what would the semantics be? I'm guessing that "with xxx as yyy" translates basically as a statement "yyy = xxx" in the notional expansion described in that section ("considering each of the for or if clauses a block...") Assuming that is the proposed definition, a few questions arise: 1. Is the behaviour this would assign to "with as x with as x" (i.e., repeated bindings of the same name) what we'd want? Is it likely to cause confusion in practice? 2. Do we need to restrict the ? Consider that "with something as global_var[0]" is allowed by the definition - that would leak values out of the comprehension. This isn't new - the "for" variable works like this already - but is it something that's more likely to be abused than currently? Should we care? (Python doesn't tend to prohibit people from writing bad code). > However, that already has semantics outside a comprehension for > context-managers. This is indeed a concern, albeit not a huge one - the definition has to point out that in the expansion "with" should be read as an assignment (written backwards) not as a with statement. It's not an impossible burden, but it is mildly inconsistent. This is probably the syntax I prefer of the ones suggested so far. But I still haven't seen any *really* convincing arguments that we need anything new in the first place. Paul From greg.ewing at canterbury.ac.nz Tue Mar 8 18:34:26 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 09 Mar 2016 12:34:26 +1300 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: <56DF2CBC.6000701@mrabarnett.plus.com> References: <56DF0B42.5030200@stoneleaf.us> <56DF1AB8.5010101@stoneleaf.us> <56DF2CBC.6000701@mrabarnett.plus.com> Message-ID: <56DF6182.3090906@canterbury.ac.nz> MRAB wrote: > "for x in ..." but "for ... as x". for x = expr -- Greg From tjreedy at udel.edu Tue Mar 8 20:47:49 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 8 Mar 2016 20:47:49 -0500 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: Message-ID: On 3/8/2016 4:24 AM, Paul Moore wrote: > The "py.exe" wrapper on Windows runs the latest Python version > available on the user's PC, but it prefers Python 2 if the user has > both Python 2 and 3 installed. With Python 3.5 being released, is it > not about time that we switched that default to always prefer the > latest version available? I agree. -- Terry Jan Reedy From tjreedy at udel.edu Tue Mar 8 20:55:01 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 8 Mar 2016 20:55:01 -0500 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: Message-ID: On 3/8/2016 9:17 AM, Allan Clark wrote: > tl;dr What is support like for adding an 'as' clause to comprehension > syntax? In order to allow map-then-filter, it might look like something > this: > > [y for x in numbers if abs(x) as y > 5] -1 Comprehensions abbreviate a particular pattern of collection initialization, nested for and if statements, and collection augmentation innermost, with the only binding being the loop names. They are easily translated back to the original pattern, although move than one level of nesting can challenge comprehension in the normal meaning of the work. I strongly feel that the current correspondence between conprehensions and statements should be maintained. -- Terry Jan Reedy From ncoghlan at gmail.com Tue Mar 8 21:34:12 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 9 Mar 2016 12:34:12 +1000 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: References: <1457370435.3901655.542029746.6A4C4A69@webmail.messagingengine.com> Message-ID: On 9 March 2016 at 04:47, Brett Cannon wrote: > But if you drop the directory that a script is > contained in, how do you import any packages that are in that directory? > What about __main__.py files in the top directory of code checkouts for easy > testing of making executable zipfiles? It's a slippery slope, hence why the > semantics have not changed. It's also not uncommon for setup.py files to expect to be able to import the package they relate to. Accordingly, the proposal I'm most amenable to is the one to simply change the default precedence of the current directory on sys.path by moving it to the end of the filesystem search (after the standard library, site-packages, user site-packages, etc). This wouldn't affect the -m switch (or the runpy module in general), since that manipulates sys.path directly, and would mean creating a "socket.py" script to experiment with the socket modules would "just work". Such a change poses its own compatibility problems (relating not only to deliberate shadowing of installed modules, but also to software that assumes the first entry in sys.path is the current directory or the script directory and removes it), but such software is already broken when run in isolated mode (which entirely skips adding the current/script directory to sys.path), so pushing folks to find more robust approaches to handling such cases wouldn't be a bad thing. >From a practical implementation perspective, one possible approach would be to handle this as a second PathImporter on sys.meta_path (after the regular sys.path one), and lose the current/script directory entry from sys.path. That way the impact on other sys.path manipulation code would be minimal - the rest of the startup sequence could continue to just append to sys.path as it does today. The compatibility break (even if only in limited circumstances) means any such change *would* require a PEP (and probably a way to opt back in to the old behaviour), but the fact this is also an opportunity to *simplify* the implementation (by giving the current/script directory a dedicated meta_path entry rather than using a special token in sys.path) makes me at least willing to consider the idea. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tritium-list at sdamon.com Tue Mar 8 22:59:39 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Tue, 8 Mar 2016 22:59:39 -0500 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> Message-ID: <56DF9FAB.1080204@sdamon.com> How about the better option: No default preset default.. Make the user pick every time it's installed. Or the best option of them all: call Python 3... python3.exe and the shell launcher py3.exe. Or stop it with the launcher. I cannot think of many situations where a non-technical user would install the base python, and run modules. Most user applications ship their own interpreter. In enterprise situations, you would want WAY more control of what python you are using than py.exe offers. So who is py.exe for? So i am changing my -1 from changing the default behavior to -1 for its continued existence. On 3/8/2016 17:18, Paul Moore wrote: > On 8 March 2016 at 22:00, Steve Dower wrote: >> "First installed" is a very difficult proposition, especially if you want >> versions of Python pre-3.6 to participate. "Last installed" is slightly >> better, though not easy to do without triggering systems that detect >> corrupted installations. > Also, any dependency on installation order gets *very* messy if > someone installs and uninstalls various versions. What does the > default become if I uninstall the version that was previously the > first (or last) installed? Would we need to keep details of precisely > when everything was installed? > >> Personally I think "latest version" is the best default (compared to the >> current "latest 2.x version"), so I'm +1 on removing the assumption that >> "PY_PYTHON" is "2" by default. > Latest version has the additional advantage of being essentially the > current behaviour, just with the special case of "but prefer Python 2 > over Python 3 if both are present" removed. (Note that technically the > current rule is not simply "latest 2.x version", as if the user only > has Python 3 installed, the latest 3.x version is used). > > The only other option I can think of is "use the one the user selected > as the default Python (i.e., introspect the association for ".py" > files) but even then we need a default default (!) for people who > don't select any default Python. It's also messy to implement. And > it's a bigger change from current behaviour. > >> Users can change their own default with any of the mechanisms described at >> https://docs.python.org/3/using/windows.html#customization > Absolutely. This should *only* apply when there is no user > customisation - and it needs no further configurability, because the > current methods are perfectly sufficient. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Tue Mar 8 23:05:01 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Mar 2016 15:05:01 +1100 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DF9FAB.1080204@sdamon.com> References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> Message-ID: On Wed, Mar 9, 2016 at 2:59 PM, Alexander Walters wrote: > How about the better option: No default preset default.. Make the user pick > every time it's installed. > > Or the best option of them all: call Python 3... python3.exe and the shell > launcher py3.exe. > > Or stop it with the launcher. > Have a read of PEP 397 if you want to know why the launcher exists. https://www.python.org/dev/peps/pep-0397/ ChrisA From tritium-list at sdamon.com Tue Mar 8 23:06:55 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Tue, 8 Mar 2016 23:06:55 -0500 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> Message-ID: <56DFA15F.7090000@sdamon.com> On 3/8/2016 23:05, Chris Angelico wrote: > > Have a read of PEP 397 if you want to know why the launcher exists. > > https://www.python.org/dev/peps/pep-0397/ > > ChrisA But who does that actually help? I contest this attempts to solve a problem (in the wrong way) that does not actually exist. From ethan at stoneleaf.us Tue Mar 8 23:14:33 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Mar 2016 20:14:33 -0800 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DFA15F.7090000@sdamon.com> References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: <56DFA329.3090403@stoneleaf.us> On 03/08/2016 08:06 PM, Alexander Walters wrote: > But who does that actually help? I contest this attempts to solve a > problem (in the wrong way) that does not actually exist. If you don't like it, ignore it. It helps me. -- ~Ethan~ From tritium-list at sdamon.com Tue Mar 8 23:17:36 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Tue, 8 Mar 2016 23:17:36 -0500 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DFA329.3090403@stoneleaf.us> References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <56DFA329.3090403@stoneleaf.us> Message-ID: <56DFA3E0.5080703@sdamon.com> I cant ignore it because it modifies my system when I install python, and is a massive pain to scrub off the system if it accidentally gets installed. On 3/8/2016 23:14, Ethan Furman wrote: > On 03/08/2016 08:06 PM, Alexander Walters wrote: > >> But who does that actually help? I contest this attempts to solve a >> problem (in the wrong way) that does not actually exist. > > If you don't like it, ignore it. > > It helps me. > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ethan at stoneleaf.us Tue Mar 8 23:19:04 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Mar 2016 20:19:04 -0800 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DFA3E0.5080703@sdamon.com> References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <56DFA329.3090403@stoneleaf.us> <56DFA3E0.5080703@sdamon.com> Message-ID: <56DFA438.8090706@stoneleaf.us> On 03/08/2016 08:17 PM, Alexander Walters wrote: > I cant ignore it because it modifies my system when I install python, > and is a massive pain to scrub off the system if it accidentally gets > installed. Really? I thought it only installed two files into the windows directory? -- ~Ethan~ From tritium-list at sdamon.com Tue Mar 8 23:22:19 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Tue, 8 Mar 2016 23:22:19 -0500 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DFA438.8090706@stoneleaf.us> References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <56DFA329.3090403@stoneleaf.us> <56DFA3E0.5080703@sdamon.com> <56DFA438.8090706@stoneleaf.us> Message-ID: <56DFA4FB.6070903@sdamon.com> It modifies the windows registry to set a file association, which is a pain when you want double clicking a python file to open a text editor. So you have to change file associations, which was made really painful in recent versions of windows. On 3/8/2016 23:19, Ethan Furman wrote: > On 03/08/2016 08:17 PM, Alexander Walters wrote: > >> I cant ignore it because it modifies my system when I install python, >> and is a massive pain to scrub off the system if it accidentally gets >> installed. > > Really? I thought it only installed two files into the windows > directory? > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From jcgoble3 at gmail.com Tue Mar 8 23:27:36 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Tue, 8 Mar 2016 23:27:36 -0500 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DFA4FB.6070903@sdamon.com> References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <56DFA329.3090403@stoneleaf.us> <56DFA3E0.5080703@sdamon.com> <56DFA438.8090706@stoneleaf.us> <56DFA4FB.6070903@sdamon.com> Message-ID: On Tue, Mar 8, 2016 at 11:06 PM, Alexander Walters wrote: > On 3/8/2016 23:05, Chris Angelico wrote: >> >> >> Have a read of PEP 397 if you want to know why the launcher exists. >> >> https://www.python.org/dev/peps/pep-0397/ >> >> ChrisA > > But who does that actually help? I contest this attempts to solve a problem > (in the wrong way) that does not actually exist. Linux users are used to being able to type "./script.py" on the command line and have the shebang tell the OS which version of Python to use. Windows doesn't support that because it instead works off of associating the file extension globally with an individual program, so ".py" can only be associated with a single executable, which is unworkable if you have both python2 and python3 scripts. py.exe solves that problem. So unless you have an alternative solution to enable Unix-like script launching with multiple Python versions from the command line, proposing to eliminate py.exe is a non-starter, at least with me. On Tue, Mar 8, 2016 at 11:22 PM, Alexander Walters wrote: > It modifies the windows registry to set a file association, which is a pain > when you want double clicking a python file to open a text editor. So you > have to change file associations, which was made really painful in recent > versions of windows. So add an option to the installer to not set that file association. It's been a while since I installed a Python on my Windows machine, but IIRC that option may already be there (I could be mistaken, though). From tritium-list at sdamon.com Tue Mar 8 23:32:34 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Tue, 8 Mar 2016 23:32:34 -0500 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <56DFA329.3090403@stoneleaf.us> <56DFA3E0.5080703@sdamon.com> <56DFA438.8090706@stoneleaf.us> <56DFA4FB.6070903@sdamon.com> Message-ID: <56DFA762.40201@sdamon.com> On 3/8/2016 23:27, Jonathan Goble wrote: > On Tue, Mar 8, 2016 at 11:06 PM, Alexander Walters > wrote: >> On 3/8/2016 23:05, Chris Angelico wrote: >>> >>> Have a read of PEP 397 if you want to know why the launcher exists. >>> >>> https://www.python.org/dev/peps/pep-0397/ >>> >>> ChrisA >> But who does that actually help? I contest this attempts to solve a problem >> (in the wrong way) that does not actually exist. > Linux users are used to being able to type "./script.py" on the > command line and have the shebang tell the OS which version of Python > to use. Windows doesn't support that because it instead works off of > associating the file extension globally with an individual program, so > ".py" can only be associated with a single executable, which is > unworkable if you have both python2 and python3 scripts. py.exe solves > that problem. > > So unless you have an alternative solution to enable Unix-like script > launching with multiple Python versions from the command line, > proposing to eliminate py.exe is a non-starter, at least with me. Windows is not *nix. Supporting *nix idioms should not be a priority. If you want a python application that launches by double clicking it, which is the intent of py.exe by the pep, create a shortcut with the fully qualified path to python and the path to the module, or use py2exe. If you want to launch something on the command line, put the python you want on %PATH%. Or install a separately obtainable py.exe installer. > On Tue, Mar 8, 2016 at 11:22 PM, Alexander Walters > wrote: >> It modifies the windows registry to set a file association, which is a pain >> when you want double clicking a python file to open a text editor. So you >> have to change file associations, which was made really painful in recent >> versions of windows. > So add an option to the installer to not set that file association. > It's been a while since I installed a Python on my Windows machine, > but IIRC that option may already be there (I could be mistaken, > though). It's there, and VERY easy to miss. Once installed, it is a pain to get rid of, since the uninstaller does NOT take it out. From jsbueno at python.org.br Wed Mar 9 00:02:32 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 9 Mar 2016 02:02:32 -0300 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: Message-ID: So, as for the "post filter" - one can currently just express that in Python as: [y for y in (abs(x) for x in numbers) if y > 5] It may have one "internal generator" - but save for the little performance saving, I don't see this as less readable than the proposals above. As for the reuse of a value that might come in an expensive computation, that is something I always missed in Python - so, a couple years ago I put together a small package that can do just that: duplicate a computed value in a seamlesway, without requiring an explicit variable assignment. To do that, I use the mechanics of stack-based languages such as Postscript and Forth - as of now, one can do: ` from stackfull import push, popclear, clear [popclear() for x in numbers if push(expensive_calculation(x)) > 5] ` The functions pergorm stack operation in a list that is created in a transparent way on the current Python execution frame local variables dictionary. Works fine in Python, Python2 and pypy. https://pypi.python.org/pypi/stackfull/0.9 (Yes, I've just updated it a little bit, due to this on-going discussion) --- And no, I am not suggesting this should go into the stdlib, I am just pointing it as a helper to people who would like that right now. js -><- On 8 March 2016 at 22:55, Terry Reedy wrote: > On 3/8/2016 9:17 AM, Allan Clark wrote: >> >> tl;dr What is support like for adding an 'as' clause to comprehension >> syntax? In order to allow map-then-filter, it might look like something >> this: >> >> [y for x in numbers if abs(x) as y > 5] > > > -1 > > Comprehensions abbreviate a particular pattern of collection initialization, > nested for and if statements, and collection augmentation innermost, with > the only binding being the loop names. They are easily translated back to > the original pattern, although move than one level of nesting can challenge > comprehension in the normal meaning of the work. I strongly feel that the > current correspondence between conprehensions and statements should be > maintained. > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From sjoerdjob at sjec.nl Wed Mar 9 00:16:21 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Wed, 9 Mar 2016 06:16:21 +0100 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: <20160308222131.GA19481@sjoerdjob.com> Message-ID: <20160309051621.GA30980@sjoerdjob.com> On Tue, Mar 08, 2016 at 10:58:16PM +0000, Paul Moore wrote: > On 8 March 2016 at 22:21, Sjoerd Job Postmus wrote: > > As for yet another syntax suggestion (if we want to introduce > > something). > > > > [y for x in numbers with abs(x) as y if y > 5] > > > > The benefit is that it reads quite natural: "with expr as name". > > Another benefit is that keywords get reused. > > Agreed, this is a plausible suggestion. But rather than just providing > an example usage, it would be helpful to see the full syntax of > comprehension with the proposed addition. See > https://docs.python.org/3/reference/expressions.html#grammar-token-comprehension It took me a while to figure this out (battling with the compiler over this), but I settled on the following grammar. comprehension ::= expression comp_for comp_for ::= "for" target_list "in" or_test [comp_iter] comp_iter ::= comp_for | comp_if | comp_with comp_if ::= "if" expression_nocond [comp_iter] comp_with ::= "with" or_test "as" target (For comp_with you probably want target_list instead of a single target. However, I felt like double-checking that I got my actual assumptions right by first implementing it (I've got it working now), and seeing as I'm not that experienced with modifying the Python parse/ast/symtable/compile phases, I decided to cop out and take the easy route. It should be fairly trivial to extend it to a target_list instead of a target.) > - note that a comprehension can have arbitrarily many for and if > clauses, interspersed in any order as long as the first one is a > "for". I'm guessing you'd add "with as > " as simply a third option. But what would the semantics > be? I'm guessing that "with xxx as yyy" translates basically as a > statement "yyy = xxx" in the notional expansion described in that > section ("considering each of the for or if clauses a block...") Trying to word it in such a way: "... considering each of the `for` or `if` clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. The `with expr as target` should be considered equivalent to `target = expr`. (And here we already see the downside of this idea). Normally a comprehension of the form (expr1 for target1 in expr2 for target2 in expr3 if expr4) boils down to for target1 in expr2: for target2 in expr3: if expr4: yield expr1 The natural extension would be for (expr1 for target1 in expr2 with expr3 as target2 if expr4) to reduce as follows. for target1 in expr2: with expr3 as target2: if expr4: yield expr1 Instead, it becomes for target1 in expr2: target2 = expr3: if expr4: yield expr1 But of course we're not going to have context managers in comprehensions, are we? So this inconsistency is somewhat forgiveable. > Assuming that is the proposed definition, a few questions arise: > > 1. Is the behaviour this would assign to "with as x with > as x" (i.e., repeated bindings of the same name) what we'd want? Is it > likely to cause confusion in practice? You mean similar to >>> [x for x in range(5) for x in range(x)] [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] I've never been confused by that in practice, as most people use good names for stuff. So I would see no ultra-important reason to prevent blocking it for this case and not in others. > 2. Do we need to restrict the ? Consider that "with > something as global_var[0]" is allowed by the definition - that would > leak values out of the comprehension. This isn't new - the "for" > variable works like this already - but is it something that's more > likely to be abused than currently? Should we care? (Python doesn't > tend to prohibit people from writing bad code). I'd suggest not to bother with resolving naming conflicts or assignments to things other than names. [__ for foo.bar in range(10)] is already valid Python. No need to limit new features in ways which their 'friends' are not. > > However, that already has semantics outside a comprehension for > > context-managers. > > This is indeed a concern, albeit not a huge one - the definition has > to point out that in the expansion "with" should be read as an > assignment (written backwards) not as a with statement. It's not an > impossible burden, but it is mildly inconsistent. > > This is probably the syntax I prefer of the ones suggested so far. But > I still haven't seen any *really* convincing arguments that we need > anything new in the first place. I agree on not having seen a really convincing argument yet. Especially since `for target in [expr]` works as well as `with expr as target`. The one thing I can think of is common subexpression elimination. [cheap_process(expensive_thingy(x)) for x in items if expensive_thingy(x) > 0] or [cheap_process(y) for x in items with expensive_thingy(x) as y if y > 0] But in Python3, you could just as well write [cheap_process(x) for x in map(expensive_thingy, items) if x > 0] > Paul From ncoghlan at gmail.com Wed Mar 9 00:32:10 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 9 Mar 2016 15:32:10 +1000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DFA15F.7090000@sdamon.com> References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: On 9 March 2016 at 14:06, Alexander Walters wrote: > On 3/8/2016 23:05, Chris Angelico wrote: >> >> Have a read of PEP 397 if you want to know why the launcher exists. >> >> https://www.python.org/dev/peps/pep-0397/ > > But who does that actually help? I contest this attempts to solve a problem > (in the wrong way) that does not actually exist. Volunteers don't generally donate their time to solve problems they don't have (and employers don't typically donate their employees' time to solve problems they or their customers don't have), which means "I don't personally have that problem, so I deny its very existence" is almost never a compelling line of argument in a community-driven open source project. In this case, we needed a cross-platform way for Python 3 scripts to indicate they were Python 3 only, and Python 2 scripts to indicate they were Python 2 only. *nix already had a way to indicate that via shebang lines, so it was natural to make that existing solution also work on Windows, rather than inventing something Windows specific (which would rather defeat the "cross-platform" objective). The fact that the chosen implementation strategy also made the "py" launcher usable at the command line was more of a side effect than the primary purpose of the exercise. In that regard, I think the meaning of "Make py.exe default to Python 3" as proposed in this thread needs to be spelled out more explicitly. If the proposal is to change the default behaviour for: * launching the interactive interpreter * running scripts without a shebang line * running scripts with an unrecognised shebang line then I think it's reasonable to switch all those to default to the newest CPython runtime on the machine, regardless of whether that's Python 2 or Python 3. However, if the proposal is also to change the handling of the following shebang lines: * #!/usr/bin/python * #!/usr/local/bin/python * #!/usr/bin/env python * #!python then I think those should continue to be handled in a manner consistent with PEP 394 (and as PEP 397 currently specifies): as referring to Python 2 (unless PY_PYTHON explicitly says otherwise or only Python 3 is available). Independently of the main topic of the thread, it does sound to me like there may be a valid enhancement request for the Windows installer to add a "Repair" mode that allows some of these settings (like adding or removing the file association) to be changed post-installation. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Wed Mar 9 00:37:46 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 08 Mar 2016 21:37:46 -0800 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: <56DFB6AA.7030807@stoneleaf.us> Excellent summary, and agreed. -- ~Ethan~ From rosuav at gmail.com Wed Mar 9 00:39:00 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 9 Mar 2016 16:39:00 +1100 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: On Wed, Mar 9, 2016 at 4:32 PM, Nick Coghlan wrote: > If the proposal is to change the default behaviour for: > > * launching the interactive interpreter > * running scripts without a shebang line > * running scripts with an unrecognised shebang line > > then I think it's reasonable to switch all those to default to the > newest CPython runtime on the machine, regardless of whether that's > Python 2 or Python 3. > > However, if the proposal is also to change the handling of the > following shebang lines: > > * #!/usr/bin/python > * #!/usr/local/bin/python > * #!/usr/bin/env python > * #!python > > then I think those should continue to be handled in a manner > consistent with PEP 394 (and as PEP 397 currently specifies): as > referring to Python 2 (unless PY_PYTHON explicitly says otherwise or > only Python 3 is available). Agree with both of these points. AIUI the original proposal was solely about the first half, with no recommendation of change to the second. What does py.exe do with a "#!/usr/bin/python" if the only Python installed is 3.5? Does it run it under 3.5, or error out? ChrisA From p.f.moore at gmail.com Wed Mar 9 04:47:28 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Mar 2016 09:47:28 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: On 9 March 2016 at 05:39, Chris Angelico wrote: > On Wed, Mar 9, 2016 at 4:32 PM, Nick Coghlan wrote: >> If the proposal is to change the default behaviour for: >> >> * launching the interactive interpreter >> * running scripts without a shebang line >> * running scripts with an unrecognised shebang line >> >> then I think it's reasonable to switch all those to default to the >> newest CPython runtime on the machine, regardless of whether that's >> Python 2 or Python 3. >> >> However, if the proposal is also to change the handling of the >> following shebang lines: >> >> * #!/usr/bin/python >> * #!/usr/local/bin/python >> * #!/usr/bin/env python >> * #!python >> >> then I think those should continue to be handled in a manner >> consistent with PEP 394 (and as PEP 397 currently specifies): as >> referring to Python 2 (unless PY_PYTHON explicitly says otherwise or >> only Python 3 is available). > > Agree with both of these points. AIUI the original proposal was solely > about the first half, with no recommendation of change to the second. My original post was prompted by interactive use, so I hadn't thought of the #!/usr/bin/python case. I can see Nick's point here. But... > What does py.exe do with a "#!/usr/bin/python" if the only Python > installed is 3.5? Does it run it under 3.5, or error out? I believe it runs Python 3. I no longer have a system with just Python 3 on, so I can't quickly test this, but I'd be surprised if it failed, based on my experience. And Nick noted that using Python 3 if it's the only one available is what the PEP says. So with regard to changing the behaviour of shebang lines I see the following points: 1. Compatibility with Unix, where an unadorned "python" in a shebang means "python 2". I personally hate this behaviour, and view it as a nasty wart of Unix. So I'm not in favour of promoting it on Windows (IMO, if you're writing scripts for publication, be explicit about whether you want Python 2 or 3 - an unqualified "python" in a shebang line should be only for cases of "I don't care, my code works for both" or purely personal code). But compatibility. 2. From what I recall of the py.exe code, implementing the exception only for shebang lines would be more complex than the current code, so I'd prefer there to be a demonstrated benefit. Unfortunately, it's really hard to know how much the feature is used. I only use it myself for personal scripts (and not much even then, tbh, because Powershell is weird with file associations). 3. On Python 3 only systems, the current behaviour isn't consistent with Unix anyway. I don't disagree with limiting the change to not affect unversioned shebang lines, but I'm not sure how much effort I'd personally be willing to put into special-casing that. We can see on that one when I start to look at the actual code, I guess... Paul From contrebasse at gmail.com Wed Mar 9 05:30:51 2016 From: contrebasse at gmail.com (Joseph Martinot-Lagarde) Date: Wed, 9 Mar 2016 10:30:51 +0000 (UTC) Subject: [Python-ideas] Pseudo-package for current directory References: <1457370435.3901655.542029746.6A4C4A69@webmail.messagingengine.com> Message-ID: Nick Coghlan writes: > Accordingly, the proposal I'm most amenable to is the one to simply > change the default precedence of the current directory on sys.path by > moving it to the end of the filesystem search (after the standard > library, site-packages, user site-packages, etc). This wouldn't affect > the -m switch (or the runpy module in general), since that manipulates > sys.path directly, and would mean creating a "socket.py" script to > experiment with the socket modules would "just work". > That would mean that any addition of a module in the standard libary is a compatibility break, because the new module in the standard library would shadow any module in the local directory. To me the cleanest (from a user point of view) way would be to enable "from . import xxx" in scripts outside packages. When I learned about relative imports I was really surprised by this limitation. If it was possible I'd use it everywhere. Joseph From niki.spahiev at gmail.com Wed Mar 9 07:16:18 2016 From: niki.spahiev at gmail.com (Niki Spahiev) Date: Wed, 9 Mar 2016 14:16:18 +0200 Subject: [Python-ideas] Pseudo-package for current directory In-Reply-To: References: <1457370435.3901655.542029746.6A4C4A69@webmail.messagingengine.com> Message-ID: On 9.03.2016 12:30, Joseph Martinot-Lagarde wrote: > > To me the cleanest (from a user point of view) way would be to enable "from > . import xxx" in scripts outside packages. When I learned about relative > imports I was really surprised by this limitation. If it was possible I'd > use it everywhere. with python 3.5 import machinery its 3 line hack to fill sys.modules[''].__path__ Niki From eryksun at gmail.com Wed Mar 9 07:49:19 2016 From: eryksun at gmail.com (eryk sun) Date: Wed, 9 Mar 2016 06:49:19 -0600 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: On Wed, Mar 9, 2016 at 3:47 AM, Paul Moore wrote: > I don't disagree with limiting the change to not affect unversioned > shebang lines, but I'm not sure how much effort I'd personally be > willing to put into special-casing that. We can see on that one when I > start to look at the actual code, I guess... I agree with Nick that Python 2 compatibility should be maintained for virtual paths in shebangs. I think a straight-forward solution is for locate_python() to gain a "virtual_mode" BOOL parameter, which is TRUE only when processing a virtual shebang, i.e. when called from maybe_handle_shebang(). For example, when wanted_ver is empty, locate_python would search the following sequence: configured_value = get_configured_value(config_key); if (configured_value) result = find_python_by_version(configured_value); if (result == NULL) result = find_python_by_version(virtual_mode ? L"2" : L"3"); if (result == NULL) result = find_python_by_version(virtual_mode ? L"3" : L"2"); This change works for me with various combinations of enabled PythonCore registry keys, PY_PYTHON values, and shebang lines. Note that the behavior of the plain "#!python" virtual shebang gets lumped in with virtual Unix paths, but I don't know why anyone would add this shebang without a target version. From p.f.moore at gmail.com Wed Mar 9 08:21:30 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Mar 2016 13:21:30 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: On 9 March 2016 at 12:49, eryk sun wrote: > On Wed, Mar 9, 2016 at 3:47 AM, Paul Moore wrote: >> I don't disagree with limiting the change to not affect unversioned >> shebang lines, but I'm not sure how much effort I'd personally be >> willing to put into special-casing that. We can see on that one when I >> start to look at the actual code, I guess... > > I agree with Nick that Python 2 compatibility should be maintained for > virtual paths in shebangs. I think a straight-forward solution is for > locate_python() to gain a "virtual_mode" BOOL parameter, which is TRUE > only when processing a virtual shebang, i.e. when called from > maybe_handle_shebang(). For example, when wanted_ver is empty, > locate_python would search the following sequence: > > configured_value = get_configured_value(config_key); > if (configured_value) > result = find_python_by_version(configured_value); > if (result == NULL) > result = find_python_by_version(virtual_mode ? L"2" : L"3"); > if (result == NULL) > result = find_python_by_version(virtual_mode ? L"3" : L"2"); > > This change works for me with various combinations of enabled > PythonCore registry keys, PY_PYTHON values, and shebang lines. Note > that the behavior of the plain "#!python" virtual shebang gets lumped > in with virtual Unix paths, but I don't know why anyone would add this > shebang without a target version. Thanks for reviewing the code for me - that does indeed look perfectly straightforward so I'm OK with following Nick's suggestion here. As there's not been much in the way of resistance to the proposal, do people think it requires a PEP? I plan on letting this thread run its course either way, it's just a matter of whether the next step should be a tracker item or a PEP. Thanks, Paul From contrebasse at gmail.com Wed Mar 9 08:34:09 2016 From: contrebasse at gmail.com (Joseph Martinot-Lagarde) Date: Wed, 9 Mar 2016 13:34:09 +0000 (UTC) Subject: [Python-ideas] Pseudo-package for current directory References: <1457370435.3901655.542029746.6A4C4A69@webmail.messagingengine.com> Message-ID: Niki Spahiev writes: > > On 9.03.2016 12:30, Joseph Martinot-Lagarde wrote: > > > > To me the cleanest (from a user point of view) way would be to enable "from > > . import xxx" in scripts outside packages. When I learned about relative > > imports I was really surprised by this limitation. If it was possible I'd > > use it everywhere. > > with python 3.5 import machinery its 3 line hack to fill > sys.modules[''].__path__ > > Niki I wasn't aware of this, thanks for the tip. Still, a 3 line hack is not beginner-friendly, it's still a hack. If it's so simple and useful why not having it by default ? I see it's been discussed in PEP 338 and 366, but it would solve the issue discussed here. From steve at pearwood.info Wed Mar 9 12:03:52 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 10 Mar 2016 04:03:52 +1100 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DFA4FB.6070903@sdamon.com> References: <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <56DFA329.3090403@stoneleaf.us> <56DFA3E0.5080703@sdamon.com> <56DFA438.8090706@stoneleaf.us> <56DFA4FB.6070903@sdamon.com> Message-ID: <20160309170352.GC12028@ando.pearwood.info> On Tue, Mar 08, 2016 at 11:22:19PM -0500, Alexander Walters complained about py.exe: > It modifies the windows registry to set a file association, which is a > pain when you want double clicking a python file to open a text editor. > So you have to change file associations, which was made really painful > in recent versions of windows. Sounds like your real beef is with Windows, not py.exe. As a programmer, couldn't you script something to change the file association and make it easier? When the pain of doing something gets greater than the pain of writing a short script, that's what I do. If you're having this problem, perhaps others are too, and the installer could make the file association change optional? -- Steve From steve at pearwood.info Wed Mar 9 12:09:24 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 10 Mar 2016 04:09:24 +1100 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56DFA762.40201@sdamon.com> References: <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <56DFA329.3090403@stoneleaf.us> <56DFA3E0.5080703@sdamon.com> <56DFA438.8090706@stoneleaf.us> <56DFA4FB.6070903@sdamon.com> <56DFA762.40201@sdamon.com> Message-ID: <20160309170924.GD12028@ando.pearwood.info> On Tue, Mar 08, 2016 at 11:32:34PM -0500, Alexander Walters wrote: > Windows is not *nix. Supporting *nix idioms should not be a priority. Python is over two decades old and we're up to version 3.5. You can hardly call this "a priority". It's been a long time coming. Besides, it surely depends on whether or not the idiom in question is valuable. Windows, and DOS before it, has copied many idioms from Unix over the years, as Unix/Linux have copied from Windows. For better or worse. > >So add an option to the installer to not set that file association. > >It's been a while since I installed a Python on my Windows machine, > >but IIRC that option may already be there (I could be mistaken, > >though). > > It's there, and VERY easy to miss. Once installed, it is a pain to get > rid of, since the uninstaller does NOT take it out. Then you should submit a bug or enhancement request to have that fixed. -- Steve From breamoreboy at yahoo.co.uk Wed Mar 9 16:20:21 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 9 Mar 2016 21:20:21 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: On 09/03/2016 13:21, Paul Moore wrote: > On 9 March 2016 at 12:49, eryk sun wrote: >> On Wed, Mar 9, 2016 at 3:47 AM, Paul Moore wrote: >>> I don't disagree with limiting the change to not affect unversioned >>> shebang lines, but I'm not sure how much effort I'd personally be >>> willing to put into special-casing that. We can see on that one when I >>> start to look at the actual code, I guess... >> >> I agree with Nick that Python 2 compatibility should be maintained for >> virtual paths in shebangs. I think a straight-forward solution is for >> locate_python() to gain a "virtual_mode" BOOL parameter, which is TRUE >> only when processing a virtual shebang, i.e. when called from >> maybe_handle_shebang(). For example, when wanted_ver is empty, >> locate_python would search the following sequence: >> >> configured_value = get_configured_value(config_key); >> if (configured_value) >> result = find_python_by_version(configured_value); >> if (result == NULL) >> result = find_python_by_version(virtual_mode ? L"2" : L"3"); >> if (result == NULL) >> result = find_python_by_version(virtual_mode ? L"3" : L"2"); >> >> This change works for me with various combinations of enabled >> PythonCore registry keys, PY_PYTHON values, and shebang lines. Note >> that the behavior of the plain "#!python" virtual shebang gets lumped >> in with virtual Unix paths, but I don't know why anyone would add this >> shebang without a target version. > > Thanks for reviewing the code for me - that does indeed look perfectly > straightforward so I'm OK with following Nick's suggestion here. > > As there's not been much in the way of resistance to the proposal, do > people think it requires a PEP? I plan on letting this thread run its > course either way, it's just a matter of whether the next step should > be a tracker item or a PEP. > > Thanks, > Paul I do not believe that this needs a PEP, a tracker item should be fine IMHO. We should just ensure that the original authors of the PEP are aware, as missing out their opinions would be a disservice to the Python Windows community, as small as that may be when compared to *nix. Just my ?0.02p worth. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From rosuav at gmail.com Wed Mar 9 16:25:25 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 10 Mar 2016 08:25:25 +1100 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: On Thu, Mar 10, 2016 at 8:20 AM, Mark Lawrence wrote: > I do not believe that this needs a PEP, a tracker item should be fine IMHO. > We should just ensure that the original authors of the PEP are aware, as > missing out their opinions would be a disservice to the Python Windows > community, as small as that may be when compared to *nix. > Given that this is basically a Windows-only change, it doesn't matter how large or small the Windows community is - they're the people who primarily matter :) If someone proposed a change to the way things are done on OS/2, it'd be worth getting the opinions of both of the Python OS/2 users... :P ChrisA From k7hoven at gmail.com Thu Mar 10 08:11:25 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 10 Mar 2016 15:11:25 +0200 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: Message-ID: On Wed, Mar 9, 2016 at 12:10 AM, Mark Mollineaux wrote: [...] > Also, Koos's offering above: > >> [abs(x) for x in numbers if > 5] > > Is also very nice, but I don't like it when the filtering is: > > [abs(x) for x in numbers if] > I assume you mean filter out zeros? Then it could be [abs(x) for x in numbers if != 0] Or someone might want to do [foo(x) for x in things if not None] Another possibility might be [abs(x) if > 5 for x in numbers] [foo(x) if not None for x in things] -- Koos From rob.cliffe at btinternet.com Thu Mar 10 08:40:00 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 10 Mar 2016 13:40:00 +0000 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: References: Message-ID: <56E17930.6040109@btinternet.com> On 10/03/2016 13:11, Koos Zevenhoven wrote: > On Wed, Mar 9, 2016 at 12:10 AM, Mark Mollineaux > wrote: > [...] > >> Also, Koos's offering above: >> >>> [abs(x) for x in numbers if > 5] >> Is also very nice, but I don't like it when the filtering is: >> >> [abs(x) for x in numbers if] >> > [snip] > > > [foo(x) for x in things if not None] > > > [foo(x) for x in things if *is* not None] # What you wrote is already valid syntax, equivalent to [foo(x) for x in things] -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Mar 10 09:45:52 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 10 Mar 2016 14:45:52 +0000 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: <56E17930.6040109@btinternet.com> References: <56E17930.6040109@btinternet.com> Message-ID: On 10 March 2016 at 13:40, Rob Cliffe wrote: > [foo(x) for x in things if is not None] # What you wrote is already valid > syntax, equivalent to [foo(x) for x in things] ... which pretty much explains why this syntax isn't a good idea. Paul From k7hoven at gmail.com Thu Mar 10 09:48:01 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 10 Mar 2016 16:48:01 +0200 Subject: [Python-ideas] FW: Map-then-filter in comprehensions In-Reply-To: <56E17930.6040109@btinternet.com> References: <56E17930.6040109@btinternet.com> Message-ID: On Thu, Mar 10, 2016 at 3:40 PM, Rob Cliffe wrote: [...] > > [foo(x) for x in things if is not None] # What you wrote is already valid > syntax, equivalent to [foo(x) for x in things] Oh, of course! Thanks. I suppose my confusion may be an argument against the syntax ;) -- Koos From pavol.lisy at gmail.com Thu Mar 10 11:53:36 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Thu, 10 Mar 2016 17:53:36 +0100 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: <20160309051621.GA30980@sjoerdjob.com> References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> Message-ID: 2016-03-09 6:16 GMT+01:00, Sjoerd Job Postmus : [...] > Trying to word it in such a way: > > "... considering each of the `for` or `if` clauses a block, nesting from > left to right, and evaluating the expression to produce an element each > time the innermost block is reached. The `with expr as target` should be > considered equivalent to `target = expr`. > > (And here we already see the downside of this idea). Normally a > comprehension of the form > > (expr1 for target1 in expr2 for target2 in expr3 if expr4) > > boils down to > > for target1 in expr2: > for target2 in expr3: > if expr4: > yield expr1 > > The natural extension would be for > > (expr1 for target1 in expr2 with expr3 as target2 if expr4) > > to reduce as follows. > > for target1 in expr2: > with expr3 as target2: > if expr4: > yield expr1 > > Instead, it becomes > > for target1 in expr2: > target2 = expr3: > if expr4: > yield expr1 > > But of course we're not going to have context managers in > comprehensions, are we? So this inconsistency is somewhat forgiveable. If we want variable assignment we have already "for var in [expr]" syntax. We could discuss if "with expr as var" syntax is more beautiful. (or if it not against There should be one-- and preferably only one --obvious way to do it.) But why omit context manager semantics in "with expr as var" assignment syntax? I personally don't like idea that semantics could be context sensitive in this way. (and we could already do pretty complex things in comprehension) From thektulu.pp at gmail.com Thu Mar 10 17:20:21 2016 From: thektulu.pp at gmail.com (=?UTF-8?B?TWljaGHFgiDFu3Vrb3dza2k=?=) Date: Thu, 10 Mar 2016 23:20:21 +0100 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> Message-ID: Some time ago I was trying to solve the same issue but using a new keyword "where", and I thought that new keyword is too much for just list comprehension filtering, so I've made it something like assignments in expresion, eg. (x+y)**2 + (x-y)**2 where x=1, y=2 So for list comprehension I can write: [stripped for line in lines if stripped where stripped=line.strip()] or: result = map(f, objs) where f=lambda x: x.return_something() or: it = iter(lines) while len(line) > 4 where line=next(it, '').strip(): print(line) or: lambda x, y: ( 0 if z == 0 else 1 if z > 0 else -1) where z = x + y or even: lambda something: d where (d, _)=something, d['a']=1 I even implemented it: https://github.com/thektulu/cpython/commit/9e669d63d292a639eb6ba2ecea3ed2c0c23f2636 and it works nicely. I was thinking to reuse "with [expr] as [var]" but I also don't like idea of context sensitive semantics, and I even thought that maybe someone, someday would want to write "content = fp.read() with open('foo.txt') as fp"... The "where" keyword is from guards pattern in Haskell :) 2016-03-10 17:53 GMT+01:00 Pavol Lisy : > 2016-03-09 6:16 GMT+01:00, Sjoerd Job Postmus : > [...] > > Trying to word it in such a way: > > > > "... considering each of the `for` or `if` clauses a block, nesting from > > left to right, and evaluating the expression to produce an element each > > time the innermost block is reached. The `with expr as target` should be > > considered equivalent to `target = expr`. > > > > (And here we already see the downside of this idea). Normally a > > comprehension of the form > > > > (expr1 for target1 in expr2 for target2 in expr3 if expr4) > > > > boils down to > > > > for target1 in expr2: > > for target2 in expr3: > > if expr4: > > yield expr1 > > > > The natural extension would be for > > > > (expr1 for target1 in expr2 with expr3 as target2 if expr4) > > > > to reduce as follows. > > > > for target1 in expr2: > > with expr3 as target2: > > if expr4: > > yield expr1 > > > > Instead, it becomes > > > > for target1 in expr2: > > target2 = expr3: > > if expr4: > > yield expr1 > > > > But of course we're not going to have context managers in > > comprehensions, are we? So this inconsistency is somewhat forgiveable. > > If we want variable assignment we have already "for var in [expr]" syntax. > > We could discuss if "with expr as var" syntax is more beautiful. (or > if it not against There should be one-- and preferably only one > --obvious way to do it.) > > But why omit context manager semantics in "with expr as var" assignment > syntax? > > I personally don't like idea that semantics could be context sensitive > in this way. (and we could already do pretty complex things in > comprehension) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leewangzhong+python at gmail.com Thu Mar 10 21:58:27 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Thu, 10 Mar 2016 21:58:27 -0500 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> Message-ID: On Thu, Mar 10, 2016 at 5:20 PM, Micha? ?ukowski wrote: > 2016-03-10 17:53 GMT+01:00 Pavol Lisy : >> We could discuss if "with expr as var" syntax is more beautiful. (or >> if it not against There should be one-- and preferably only one >> --obvious way to do it.) >> >> But why omit context manager semantics in "with expr as var" assignment >> syntax? >> >> I personally don't like idea that semantics could be context sensitive >> in this way. (and we could already do pretty complex things in >> comprehension) > I was thinking to reuse "with [expr] as [var]" but I > also don't like idea of context sensitive semantics, and I even thought that > maybe someone, someday would want to write "content = fp.read() with > open('foo.txt') as fp"... This is my understanding of `with Expr() as Name: Block()`, in pseudocode (ignoring exception handling). _context = Expr() Name = _context.__enter__(...) exec(Block, {"Name": Name}) _context.__exit__(...) Strawman proposal: Extend `with` to allow non-context managers. _context = Expr() if hasattr(_context, '__enter__'): # Context manager. Name = _context.__enter__(...) exec(Block, {"Name": Name}) _context.__exit__(...) else: # Normal object: treat as name bind Name = _context exec(Block, {"Name": Name}) Then you're allowed to write with 5 as x: print(x*2) And if your comprehension `with` receives context managers, they will be treated as such. # Closes each file after reading the second byte. second_bytes = [f.read(1) for fname in fnames with open(fname) as f if f.read(1) != '\0'] # Expanded to a loop. second_bytes = [] for fname in fnames: with open(fname) as f: if f.read(1) != '\0': second_bytes.append(f.read(1)) This breaks TOOWTDI by giving another way to assign. But it might not violate the spirit: I think very few people would trade a simple assignment for a backward assignment and an extra indentation level (!), especially newbies from other languages. TOOWTDI doesn't want competing ways of doing things, and this way isn't competitive. But that means it's a feature not intended for use, and only added for consistency with a new feature. The comprehension `with` will be perfectly aligned and translates naturally to the current `with`. It introduces an inconsistency between context managers and other objects, though. Any code that used to work will still work, but now any code that accidentally uses a non-context manager will not fail fast. It might also make the real use of normal `with` harder to learn. You have a gotcha for people trying to process and store a list of context managers. (Are locks used like this?) I believe that use will be both rare and advanced, so the gotcha is worse than one that is common and beginner-level. From ncoghlan at gmail.com Thu Mar 10 22:51:22 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 11 Mar 2016 13:51:22 +1000 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> Message-ID: On 11 March 2016 at 12:58, Franklin? Lee wrote: > On Thu, Mar 10, 2016 at 5:20 PM, Micha? ?ukowski wrote: >> 2016-03-10 17:53 GMT+01:00 Pavol Lisy : >>> We could discuss if "with expr as var" syntax is more beautiful. (or >>> if it not against There should be one-- and preferably only one >>> --obvious way to do it.) >>> >>> But why omit context manager semantics in "with expr as var" assignment >>> syntax? >>> >>> I personally don't like idea that semantics could be context sensitive >>> in this way. (and we could already do pretty complex things in >>> comprehension) >> I was thinking to reuse "with [expr] as [var]" but I >> also don't like idea of context sensitive semantics, and I even thought that >> maybe someone, someday would want to write "content = fp.read() with >> open('foo.txt') as fp"... > > This is my understanding of `with Expr() as Name: Block()`, in > pseudocode (ignoring exception handling). > > _context = Expr() > Name = _context.__enter__(...) > exec(Block, {"Name": Name}) > _context.__exit__(...) with doesn't create a new scope - it's more akin to a try/finally statement with a particular form (PEP 343 defines the full expansion, and that expansion is still broadly accurate, although some the specific details are different these days) The only compound statements in Python that create new scopes are "def" and "class". It's also worth reading PEP 343 for Guido's explanation for why the context management assignment syntax isn't "with VAR = EXPR": it's because EXPR *isn't* the thing being assigned, but rather a stepping stone towards obtaining that thing. This characteristic is true of all of the places where "as" is currently used instead of "=": with CM_EXPR as VAR: # VAR = CM_EXPR.__enter__(), not CM_EXPR except EXC_TYPE_EXPR as VAR: # VAR = the caught exception, not the exception type constraint # There's an implied "del VAR" at the end of the suite, but still not a full implicit scope import MODULE_REFERENCE as VAR from MODULE_REFERENCE import NAME as VAR # Here, the LHS isn't even a normal expression - it's a module name, or a module name plus an attribute name Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From leewangzhong+python at gmail.com Fri Mar 11 00:38:05 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Fri, 11 Mar 2016 00:38:05 -0500 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> Message-ID: On Mar 10, 2016 10:51 PM, "Nick Coghlan" wrote: > > On 11 March 2016 at 12:58, Franklin? Lee wrote: > > On Thu, Mar 10, 2016 at 5:20 PM, Micha? ?ukowski wrote: > >> 2016-03-10 17:53 GMT+01:00 Pavol Lisy : > >>> We could discuss if "with expr as var" syntax is more beautiful. (or > >>> if it not against There should be one-- and preferably only one > >>> --obvious way to do it.) > >>> > >>> But why omit context manager semantics in "with expr as var" assignment > >>> syntax? > >>> > >>> I personally don't like idea that semantics could be context sensitive > >>> in this way. (and we could already do pretty complex things in > >>> comprehension) > >> I was thinking to reuse "with [expr] as [var]" but I > >> also don't like idea of context sensitive semantics, and I even thought that > >> maybe someone, someday would want to write "content = fp.read() with > >> open('foo.txt') as fp"... > > > > This is my understanding of `with Expr() as Name: Block()`, in > > pseudocode (ignoring exception handling). > > > > _context = Expr() > > Name = _context.__enter__(...) > > exec(Block, {"Name": Name}) > > _context.__exit__(...) > > with doesn't create a new scope - it's more akin to a try/finally > statement with a particular form (PEP 343 defines the full expansion, > and that expansion is still broadly accurate, although some the > specific details are different these days) > > The only compound statements in Python that create new scopes are > "def" and "class". Sorry, that was just sloppiness with `exec`. I wanted to convey the block using the new `Name`, and "function call" popped up first. I erased a comment about how, after running the block, `Name` would still hold the value it received. I forgot to mention that people might choose the proposed `with` for assignment over `=` if they wrongly thought that the `with` form deleted `Name` afterward. > It's also worth reading PEP 343 for Guido's explanation for why the > context management assignment syntax isn't "with VAR = EXPR": it's > because EXPR *isn't* the thing being assigned, but rather a stepping > stone towards obtaining that thing. This characteristic is true of all > of the places where "as" is currently used instead of "=": > > with CM_EXPR as VAR: > # VAR = CM_EXPR.__enter__(), not CM_EXPR > > except EXC_TYPE_EXPR as VAR: > # VAR = the caught exception, not the exception type constraint > # There's an implied "del VAR" at the end of the suite, but > still not a full implicit scope > > import MODULE_REFERENCE as VAR > from MODULE_REFERENCE import NAME as VAR > # Here, the LHS isn't even a normal expression - it's a module > name, or a module name plus an attribute name Then would it be a problem if it defaulted to the value of EXPR directly when that value isn't of a specially-handled type (such as a context manager, for `with`)? Other than that learning `with` in comprehensions might make it harder to learn context managers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavol.lisy at gmail.com Fri Mar 11 07:08:46 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Fri, 11 Mar 2016 13:08:46 +0100 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> Message-ID: 2016-03-11 6:38 GMT+01:00, Franklin? Lee : > On Mar 10, 2016 10:51 PM, "Nick Coghlan" wrote: >> [...] >> It's also worth reading PEP 343 for Guido's explanation for why the >> context management assignment syntax isn't "with VAR = EXPR": it's >> because EXPR *isn't* the thing being assigned, but rather a stepping >> stone towards obtaining that thing. This characteristic is true of all >> of the places where "as" is currently used instead of "=": [...] > Then would it be a problem if it defaulted to the value of EXPR directly > when that value isn't of a specially-handled type (such as a context > manager, for `with`)? Other than that learning `with` in comprehensions > might make it harder to learn context managers. Or we could think about syntax "with VAR = EXPR" where it is plain assignment. If we wanted in PEP343 (and still we want) to have different syntax for different semantics then it is better, I think. PS. from my point of view I still don't see why is "with" so much better than existing syntax "for VAR in [EXPR]" for example why is this not enough? -> import unicodedata [(c, i, cat, name, norm) for i in range(1, 0x110000) for c in [chr(i)] for cat in [unicodedata.category(chr(i))] for name in [unicodedata.name(chr(i), 'noname')] for uname in [name.upper()] # name is defined in previous row for norm in [unicodedata.normalize('NFKC', chr(i))] if 'OPERATOR' in uname if norm != c ] PPS. if we want to change syntax the "for VAR = EXPR" is also possibility From jbvsmo at gmail.com Fri Mar 11 07:42:13 2016 From: jbvsmo at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Bernardo?=) Date: Fri, 11 Mar 2016 09:42:13 -0300 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> Message-ID: On Thu, Mar 10, 2016 at 7:20 PM, Micha? ?ukowski wrote: > Some time ago I was trying to solve the same issue but using a new keyword > "where", and I thought that new keyword is too much for just list > comprehension filtering, so I've made it something like assignments in > expresion, eg. > > (x+y)**2 + (x-y)**2 where x=1, y=2 > > So for list comprehension I can write: > > [stripped for line in lines if stripped where stripped=line.strip()] > Why are we not funding this? But seriously, why? This is great. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at 2sn.net Fri Mar 11 08:13:02 2016 From: python at 2sn.net (Alexander Heger) Date: Sat, 12 Mar 2016 00:13:02 +1100 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: <20160308222131.GA19481@sjoerdjob.com> References: <20160308222131.GA19481@sjoerdjob.com> Message-ID: I agree, the second suggestion was also my first though when I read the initial post. Seems very natural to me. If a syntax extension is not desired, the first suggestion looks like a cookbook example on how to do this to me. Could use tuple instead of list. +1 On 9 March 2016 at 09:21, Sjoerd Job Postmus wrote: > On Tue, Mar 08, 2016 at 02:17:23PM +0000, Allan Clark wrote: >> tl;dr What is support like for adding an 'as' clause to comprehension >> syntax? In order to allow map-then-filter, it might look like something >> this: >> >> [y for x in numbers if abs(x) as y > 5] >> >> I wish to propose an extension to Python comprehension syntax in an attempt >> to make it applicable in more areas. I'll first describe the deficiency I >> perceive in the current comprehension syntax and then propose my extension. >> I'll then note some drawbacks. For the purposes of illustration I'll use >> list comprehension syntax but I believe everything I say is equally >> applicable to set and dictionary comprehensions. I'll also talk about lists >> but again (more or less) everything applies to the more general concept of >> iterables. >> >> Comprehensions are essentially a filter followed by a map operation. So if >> you need to perform a filter operation followed by a map operation over a >> list, this is pretty convenient in Python. Here we are going to take the >> absolute value of all even numbers within the range -10 to 10. >> >> numbers = range(-10, 10) >> [abs(x) for x in numbers if x % 2 == 0] >> >> However, if you wish to perform a map operation and *then* a filter >> operation this is not so convenient, so suppose we wish to obtain the >> absolute value of all numbers that have an absolute value larger than 5, we >> can do this by calling the mapped method twice: >> >> abs(x) for x in numbers if abs(x) > 5] >> >> This is a bit unsatisfying and even impossible in the case that the mapped >> method has some side-effect (although arguably if you find yourself in that >> situation you have taken a mis-step somewhere). An alternative is to apply >> the mapping first: >> >> [y for y in (abs(x) for x in numbers) if y > 5] >> >> I have to say I quite like this, as it is pretty explicit, but it is a bit >> unsatisfying that you require only one comprehension for a filter-then-map >> but two for a map-then-filter. What would be nice is if we could give a >> name to the mapped expression and then use that in the filter. >> >> [abs(x) as y for x in numbers if y > 5] >> >> I don't like this as it means the order of execution is dependent on >> whether there is an 'as' clause, in particular the 'if' clause itself may >> do some computation such as in `if f(y) > 5`. >> >> An altenative is to allow 'as' expressions in the condition, something >> like: >> >> [y for x in numbers if abs(x) as y > 5] >> >> Note that it would be possible to map to an intermediate result, such as: >> >> [y**2 for x in numbers if abs(x) as y > 5] >> >> Alternatively we could put the 'as' in the pattern: >> >> [y**2 for abs(x) as y in numbers if y > 5] >> >> I did not like this as it is obscures the fact that 'x' is being set to >> each element of 'numbers'. Additionally, we might later wish to adopt a >> functional programming idiom in which we use 'as' for deconstructive >> assignment whilst giving a name to the entire matched value, for example: >> >> [p for (x,y) as p if x > y] >> >> Or more generally: >> >> (x,y) as a = f(z) >> >> But that is getting somewhat off-topic. I promised some drawbacks: >> * I am confident there are some implementation gotchas nestling in here >> somewhere. >> * I could imagine how abuse of such a mechanism to lead to pretty >> unreadable code. >> * I'm still not *that* upset by the explicit map first: `[y for y in >> (abs(x) for x in numbers) if y > 5]` >> * I could see how it is not immediately obvious what the code does. >> * It would need to be decided whether you allowed multiple 'as' >> expression in the condition, particularly using 'and' or 'or' as in 'if >> f(a) as x > 5 and f(b) as y > 5' >> >> To summarise: >> * It's a touch annoying that comprehensions allow filter-then-map but >> not map-then-filter >> * Three proposed syntaxes are: >> * [abs(x) as y for x in numbers if y > 5] >> * [y for x in numbers if abs(x) as y > 5] >> * [y**2 for abs(x) as y in numbers if y > 5] >> * My favourite is the middle one. >> >> Finally these seem to currently be syntax errors so we should not break any >> existing code. > > Seeing the many replies, I'm a bit lost as where to best comment. After > thinking about it for a while, I think that currently it's not > impossible to do what you want with comprehensions, just a bit > convoluted. > > [y for x in numbers for y in [abs(x)] if y > 5] > > The tricky part being the `for y in [abs(x)]` basically doing what you > want: bind the value `abs(x)` to a name (`y`). > > Benefits: > - It requires no new syntax, no new semantics. > - You can easily define multiple items at once. > [y + z for x in numbers for y, z in [(abs(x), sgn(x)] if y > 5] > - You can still use the original value, in contrast to > [y for y in (abs(x) for x in numbers) if y > 5] > > Downside: > - Very unreadable, and probably a long way off from being idiomatic > Python. > > As for yet another syntax suggestion (if we want to introduce > something). > > [y for x in numbers with abs(x) as y if y > 5] > > The benefit is that it reads quite natural: "with expr as name". > Another benefit is that keywords get reused. > However, that already has semantics outside a comprehension for > context-managers. > > > > TL;DR: It's possible with > [y for x in numbers for y in [abs(x)] if y > 5] > But the syntax is ugly enough that it does warrant some extra syntactic > sugar (or something with the same semantics but better performance > characteristics). > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From sjoerdjob at sjec.nl Fri Mar 11 08:23:51 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Fri, 11 Mar 2016 14:23:51 +0100 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> Message-ID: <20160311132351.GA32080@sjoerdjob.com> On Thu, Mar 10, 2016 at 11:20:21PM +0100, Micha? ?ukowski wrote: > Some time ago I was trying to solve the same issue but using a new keyword > "where", and I thought that new keyword is too much for just list > comprehension filtering, so I've made it something like assignments in > expresion, eg. > > (x+y)**2 + (x-y)**2 where x=1, y=2 > > So for list comprehension I can write: > > [stripped for line in lines if stripped where stripped=line.strip()] > > or: > > result = map(f, objs) where f=lambda x: x.return_something() > > or: > > it = iter(lines) > while len(line) > 4 where line=next(it, '').strip(): > print(line) > > or: > > lambda x, y: ( > 0 if z == 0 else > 1 if z > 0 else > -1) where z = x + y > > or even: > > lambda something: d where (d, _)=something, d['a']=1 > > I even implemented it: > https://github.com/thektulu/cpython/commit/9e669d63d292a639eb6ba2ecea3ed2c0c23f2636 > > and it works nicely. I was thinking to reuse "with [expr] as [var]" but I > also don't like idea of context sensitive semantics, and I even thought > that maybe someone, someday would want to write "content = fp.read() with > open('foo.txt') as fp"... > > The "where" keyword is from guards pattern in Haskell :) But in Haskell, the `where` keyword also considers scoping. That is, outside the statement/expression with the `where`, you can't access the variables introduced by the where. Even though the `where` looks kind-of-nice, it (at least to me) is also a bit confusing with respect to evaluation order. Consider [ stripped for idx, line in enumerate(lines) if idx >= 5 or stripped where stripped=line.strip() ] (intended semantics: give me all lines (stripped), but ignore any lines that are whitespace-only in the first 5 lines) retval = [] for idx, line in enumerate(lines): stripped = line.strip() if idx >= 5 or stripped: retval.append(stripped) now I'm not very sure, but I expect what actually happens is: retval = [] for idx, line in enumerate(lines): if idx < 5: stripped = line.strip() if idx >= 5 or stripped: retval.append(stripped) that is, should I read it as (if idx >= 5 or stripped) where stripped=line.strip() or if idx >= 5 or (stripped where stripped=line.strip()) For comprehensions, I'd think the 'let' statement might make more sense. Abusing Haskell's notation: [ stripped | (idx, line) <- zip [0..] lines, let stripped = strip line, idx >= 5 || length stripped > 0 ] Porting this to something Python-ish, it'd be [ stripped for idx, line in enumerate(lines) let stripped = line.strip() if idx >= 5 or stripped ] where `let` is a keyword (possibly only applicable in a compexpr). In Haskell it's a keyword everywhere, but it has somewhat different semantics. From thektulu.pp at gmail.com Fri Mar 11 10:29:14 2016 From: thektulu.pp at gmail.com (=?UTF-8?B?TWljaGHFgiDFu3Vrb3dza2k=?=) Date: Fri, 11 Mar 2016 16:29:14 +0100 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: <20160311132351.GA32080@sjoerdjob.com> References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> <20160311132351.GA32080@sjoerdjob.com> Message-ID: > > > But in Haskell, the `where` keyword also considers scoping. That is, > outside the statement/expression with the `where`, you can't access the > variables introduced by the where. > > Yes, but I wanted it to be simple and powerful, not necessarily right in every context. Even though the `where` looks kind-of-nice, it (at least to me) is also > a bit confusing with respect to evaluation order. Consider > > [ stripped for idx, line in enumerate(lines) if idx >= 5 or stripped > where stripped=line.strip() ] > > (intended semantics: give me all lines (stripped), but ignore > any lines that are whitespace-only in the first 5 lines) > > retval = [] > for idx, line in enumerate(lines): > stripped = line.strip() > if idx >= 5 or stripped: > retval.append(stripped) > > now I'm not very sure, but I expect what actually happens is: > > retval = [] > for idx, line in enumerate(lines): > if idx < 5: > stripped = line.strip() > if idx >= 5 or stripped: > retval.append(stripped) > > that is, should I read it as > (if idx >= 5 or stripped) where stripped=line.strip() > or > if idx >= 5 or (stripped where stripped=line.strip()) > I've implemented it as "or_test [where_expr]" so the default order is the same as in: (if idx >= 5 or stripped) where stripped=line.strip() I wanted it to always "scope" left as much as possible - in precedence order between "lambda" and "... if ... else ..." - but left recursion forced me to place it after "... if ... else ..." which can't be used without brackets in list comprehension "if" filtering. But one can always control order with brackets, and make it work like in second example. For comprehensions, I'd think the 'let' statement might make more sense. > Abusing Haskell's notation: > > [ stripped | (idx, line) <- zip [0..] lines, let stripped = strip > line, idx >= 5 || length stripped > 0 ] > > Porting this to something Python-ish, it'd be > > [ stripped for idx, line in enumerate(lines) let stripped = > line.strip() if idx >= 5 or stripped ] > > where `let` is a keyword (possibly only applicable in a compexpr). In > Haskell it's a keyword everywhere, but it has somewhat different > semantics. > I was thinking about "let", but as I said, I think that new keyword should be more powerful than just filtering in list comprehensions, and in regular expresions it would look like this: if let x=foo() in x: print(x) Which does not look great, and there is problem with "in" keyword that make this expresion ambiguous. -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri Mar 11 14:59:16 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 11 Mar 2016 21:59:16 +0200 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> Message-ID: On Fri, Mar 11, 2016 at 2:42 PM, Jo?o Bernardo wrote: > > On Thu, Mar 10, 2016 at 7:20 PM, Micha? ?ukowski > wrote: >> [...] >> (x+y)**2 + (x-y)**2 where x=1, y=2 >> >> So for list comprehension I can write: >> >> [stripped for line in lines if stripped where stripped=line.strip()] > > Why are we not funding this? > But seriously, why? This is great. > One cool thing about this 'where' thing is that, if it made a new scope on every iteration, you could do [lambda x: x * j for i in range(10) where j = i] and it would actually create ten *different* functions (as opposed to without the 'where j = i'). - Koos From abarnert at yahoo.com Fri Mar 11 15:25:16 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 11 Mar 2016 12:25:16 -0800 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> <20160311132351.GA32080@sjoerdjob.com> Message-ID: <90745071-3E89-471C-91E0-18052ACCBE99@yahoo.com> On Mar 11, 2016, at 07:29, Micha? ?ukowski wrote: >> >> But in Haskell, the `where` keyword also considers scoping. That is, >> outside the statement/expression with the `where`, you can't access the >> variables introduced by the where. > > Yes, but I wanted it to be simple and powerful, not necessarily right in every context. > >> Even though the `where` looks kind-of-nice, it (at least to me) is also >> a bit confusing with respect to evaluation order. Consider >> >> [ stripped for idx, line in enumerate(lines) if idx >= 5 or stripped where stripped=line.strip() ] >> >> (intended semantics: give me all lines (stripped), but ignore >> any lines that are whitespace-only in the first 5 lines) >> >> retval = [] >> for idx, line in enumerate(lines): >> stripped = line.strip() >> if idx >= 5 or stripped: >> retval.append(stripped) >> >> now I'm not very sure, but I expect what actually happens is: >> >> retval = [] >> for idx, line in enumerate(lines): >> if idx < 5: >> stripped = line.strip() >> if idx >= 5 or stripped: >> retval.append(stripped) >> >> that is, should I read it as >> (if idx >= 5 or stripped) where stripped=line.strip() >> or >> if idx >= 5 or (stripped where stripped=line.strip()) > > I've implemented it as "or_test [where_expr]" so the default order is the same as in: > (if idx >= 5 or stripped) where stripped=line.strip() This makes where look like another top-level comprehension clause like for and if, but it doesn't follow the same nesting rules as the other clauses. Which means that whenever something isn't trivial enough to just read holistically, trying to apply the dead-simple rule of "write each clause as a nested statement, in the same order" will just lead to confusion, instead of immediately telling you the interpretation. Where that line is depends on how experienced you are with Python, how much time you've spent recently in a language with similar but not identical comprehension syntax, how tired you are, etc., but that doesn't matter--any comprehension with a where clause has at least three clauses, and is likely to be complicated enough to be over that line for some people. So, if you're going to recommend that this is only used when it's simple enough that it doesn't matter that the translation is confusing, that would mean recommending never using it, which implies that we shouldn't have it. A new clause that nests in order, like some of your other variations, doesn't have this problem. > > I wanted it to always "scope" left as much as possible - in precedence order between "lambda" and "... if ... else ..." - but left recursion forced me to place it after "... if ... else ..." which can't be used without brackets in list comprehension "if" filtering. > But one can always control order with brackets, and make it work like in second example. > >> For comprehensions, I'd think the 'let' statement might make more sense. >> Abusing Haskell's notation: >> >> [ stripped | (idx, line) <- zip [0..] lines, let stripped = strip line, idx >= 5 || length stripped > 0 ] >> >> Porting this to something Python-ish, it'd be >> >> [ stripped for idx, line in enumerate(lines) let stripped = line.strip() if idx >= 5 or stripped ] >> >> where `let` is a keyword (possibly only applicable in a compexpr). In >> Haskell it's a keyword everywhere, but it has somewhat different >> semantics. > > I was thinking about "let", but as I said, I think that new keyword should be more powerful than just filtering in list comprehensions, and in regular expresions it would look like this: > if let x=foo() in x: > print(x) > > Which does not look great, and there is problem with "in" keyword that make this expresion ambiguous. I think you're overcomplicating this. You want let to be a statement, but also have a value--but you only need one or the other. As a statement, there's no need for a value; you're just creating or rebinding a variable for use in the controlled suite: let x = foo(): if x: print(x) And in a comprehension, it's just another clause that converts to the identical compound statement and nests the same as other clauses: [stripped for line in file let stripped=line.strip() if stripped] Hopefully you don't want scoping, but if you do, it's pretty obvious that the scope is the controlled suite; still no need for an in clause. If you really want a let expression instead of a statement, you still don't need an in clause unless you want scoping. Just make it an assignment expression whose value is the assigned value, just like in C and friends: if let x = foo(): print(x) Which extends easily to: if (let x = foo()) > 5: print(x) [stripped for line in file if let stripped=line.strip()] The only reason you'd need an in clause is if you wanted scoped expressions. But those would be almost useless in Python because, by design, you can't do too much inside an expression. But if you really want that, the design is again obvious: just as in functional languages, it's equivalent to a lambda call: let x=foo() in (print(x) if x > 5 else None) (lambda x: print(x) if x > 5 else None)(foo()) ... which is obviously unpythonic enough and useless enough that I think scoped let expressions are immediately dismissible as an idea. -------------- next part -------------- An HTML attachment was scrubbed... URL: From thektulu.pp at gmail.com Fri Mar 11 18:59:11 2016 From: thektulu.pp at gmail.com (=?UTF-8?B?TWljaGHFgiDFu3Vrb3dza2k=?=) Date: Sat, 12 Mar 2016 00:59:11 +0100 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: <90745071-3E89-471C-91E0-18052ACCBE99@yahoo.com> References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> <20160311132351.GA32080@sjoerdjob.com> <90745071-3E89-471C-91E0-18052ACCBE99@yahoo.com> Message-ID: Let statement is pointless - all things you can achievie by simply assign variables. Let expresion in form "let x=1" cannot "return" value when you want to define more than one variable - with "in" keyword you can define (like in Haskell) more than one variable, and then "return" a value, or you would need to assume that the last defined variable is returned, which is not intuitive :) 11-03-2016 21:25, "Andrew Barnert" napisa?(a): > > > On Mar 11, 2016, at 07:29, Micha? ?ukowski wrote: > >>> >>> But in Haskell, the `where` keyword also considers scoping. That is, >>> outside the statement/expression with the `where`, you can't access the >>> variables introduced by the where. >>> >> >> Yes, but I wanted it to be simple and powerful, not necessarily right in every context. >> >>> Even though the `where` looks kind-of-nice, it (at least to me) is also >>> a bit confusing with respect to evaluation order. Consider >>> >>> [ stripped for idx, line in enumerate(lines) if idx >= 5 or stripped where stripped=line.strip() ] >>> >>> (intended semantics: give me all lines (stripped), but ignore >>> any lines that are whitespace-only in the first 5 lines) >>> >>> retval = [] >>> for idx, line in enumerate(lines): >>> stripped = line.strip() >>> if idx >= 5 or stripped: >>> retval.append(stripped) >>> >>> now I'm not very sure, but I expect what actually happens is: >>> >>> retval = [] >>> for idx, line in enumerate(lines): >>> if idx < 5: >>> stripped = line.strip() >>> if idx >= 5 or stripped: >>> retval.append(stripped) >>> >>> that is, should I read it as >>> (if idx >= 5 or stripped) where stripped=line.strip() >>> or >>> if idx >= 5 or (stripped where stripped=line.strip()) >> >> >> I've implemented it as "or_test [where_expr]" so the default order is the same as in: >> (if idx >= 5 or stripped) where stripped=line.strip() > > > This makes where look like another top-level comprehension clause like for and if, but it doesn't follow the same nesting rules as the other clauses. Which means that whenever something isn't trivial enough to just read holistically, trying to apply the dead-simple rule of "write each clause as a nested statement, in the same order" will just lead to confusion, instead of immediately telling you the interpretation. > > Where that line is depends on how experienced you are with Python, how much time you've spent recently in a language with similar but not identical comprehension syntax, how tired you are, etc., but that doesn't matter--any comprehension with a where clause has at least three clauses, and is likely to be complicated enough to be over that line for some people. So, if you're going to recommend that this is only used when it's simple enough that it doesn't matter that the translation is confusing, that would mean recommending never using it, which implies that we shouldn't have it. > > A new clause that nests in order, like some of your other variations, doesn't have this problem. > >> >> I wanted it to always "scope" left as much as possible - in precedence order between "lambda" and "... if ... else ..." - but left recursion forced me to place it after "... if ... else ..." which can't be used without brackets in list comprehension "if" filtering. >> But one can always control order with brackets, and make it work like in second example. >> >>> For comprehensions, I'd think the 'let' statement might make more sense. >>> Abusing Haskell's notation: >>> >>> [ stripped | (idx, line) <- zip [0..] lines, let stripped = strip line, idx >= 5 || length stripped > 0 ] >>> >>> Porting this to something Python-ish, it'd be >>> >>> [ stripped for idx, line in enumerate(lines) let stripped = line.strip() if idx >= 5 or stripped ] >>> >>> where `let` is a keyword (possibly only applicable in a compexpr). In >>> Haskell it's a keyword everywhere, but it has somewhat different >>> semantics. >> >> >> I was thinking about "let", but as I said, I think that new keyword should be more powerful than just filtering in list comprehensions, and in regular expresions it would look like this: >> if let x=foo() in x: >> print(x) >> >> >> Which does not look great, and there is problem with "in" keyword that make this expresion ambiguous. > > > I think you're overcomplicating this. You want let to be a statement, but also have a value--but you only need one or the other. > > As a statement, there's no need for a value; you're just creating or rebinding a variable for use in the controlled suite: > > let x = foo(): > if x: > print(x) > > And in a comprehension, it's just another clause that converts to the identical compound statement and nests the same as other clauses: > > [stripped for line in file let stripped=line.strip() if stripped] > > Hopefully you don't want scoping, but if you do, it's pretty obvious that the scope is the controlled suite; still no need for an in clause. > > If you really want a let expression instead of a statement, you still don't need an in clause unless you want scoping. Just make it an assignment expression whose value is the assigned value, just like in C and friends: > > > if let x = foo(): > print(x) > > Which extends easily to: > > if (let x = foo()) > 5: > print(x) > > [stripped for line in file if let stripped=line.strip()] > > The only reason you'd need an in clause is if you wanted scoped expressions. But those would be almost useless in Python because, by design, you can't do too much inside an expression. But if you really want that, the design is again obvious: just as in functional languages, it's equivalent to a lambda call: > > let x=foo() in (print(x) if x > 5 else None) > > (lambda x: print(x) if x > 5 else None)(foo()) > > ... which is obviously unpythonic enough and useless enough that I think scoped let expressions are immediately dismissible as an idea. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tymoteusz.jankowski at gmail.com Sat Mar 12 05:12:08 2016 From: tymoteusz.jankowski at gmail.com (Tymoteusz Jankowski) Date: Sat, 12 Mar 2016 10:12:08 +0000 Subject: [Python-ideas] Version control system link in package meta data Message-ID: Hi, tldr: install project and its requirements as cloned repositories (if possible) Let's say I'm developing requests library which relies on these packages. My workflow is this: $ git clone https://github.com/kennethreitz/requests.git $ cd requests $ virtualenv requests $ . requests/bin/activate.fish $ pip install -r requirements.txt Now I can change Requests library easily and commit changes, but when i have to change a library from Requests' requirements I have to clone and reinstall library (It's boring + I'm lazy). I need tool that works like this: $ magic-command install requests 1 the tool checks in Requests package meta where sources are stored 2 clone the source 3 do the same for each package from requirements (or fallback to current method) I could write the tool and be the only one in the world using it ;) but there should be an option for storing repository link. Could you advice anything? I found this: https://www.python.org/dev/peps/pep-0345/#project-url-multiple-use but 1 I'm not sure it's right option 2 I can't see handler for this option in distutils -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Sat Mar 12 05:44:35 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Sat, 12 Mar 2016 05:44:35 -0500 Subject: [Python-ideas] Version control system link in package meta data In-Reply-To: References: Message-ID: <56E3F313.7060908@sdamon.com> (cross-replying to distutils-sig at python.org) For a tool like this to work reliably, the semantics of the project urls would have to change (as of right now, there is no set requirement that a VCS link in project URL's point to ), nor is there a requirement that the field even be present. For this to be useful for your tool, projects would have to be required, or at least heavily encouraged, to put VCS links in their pypi listings. Moreover, such a requirement would additionally need to limit what type of VCS links you post as to make the tool reasonable to maintain. (Not everyone uses git, let alone github. Do you want to write a tool that supports Team Foundation Server?) Another option would be to pull in source packages with your tool, but that too would require that people actually post source packages for their projects on PyPI. There is no such requirement as it stands, and the number of pure-python projects posting sdists may actually decrease with the uptake of universal wheels. Automating this process is not impossible. As you said, you could write your own tool to do it, but as it stands, even with that tool, you would have to do quite a bit of manual legwork still. I agree, it would be nice if everyone used git (or any of a small set of VCS), and all the packages on pypi listed their repositories in the metadata. If that were the case, this tool might already exist. In the current state of things, though, i don't think it makes much sense to produce a general purpose tool for this. On 3/12/2016 05:12, Tymoteusz Jankowski wrote: > Hi, > tldr: install project and its requirements as cloned repositories (if > possible) > > Let's say I'm developing requests > library which relies on > these > packages. > My workflow is this: > > $ git clone https://github.com/kennethreitz/requests.git > $ cd requests > $ virtualenv requests > $ . requests/bin/activate.fish > $ pip install -r requirements.txt > > Now I can change Requests library easily and commit changes, but when > i have to change a library from Requests' requirements I have to clone > and reinstall library (It's boring + I'm lazy). > I need tool that works like this: > > $ magic-command install requests > 1 the tool checks in Requests package meta where sources are stored > 2 clone the source > 3 do the same for each package from requirements (or fallback to > current method) > > I could write the tool and be the only one in the world using it ;) > but there should be an option for storing repository link. > Could you advice anything? > I found this: > https://www.python.org/dev/peps/pep-0345/#project-url-multiple-use > but > 1 I'm not sure it's right option > 2 I can't see handler for this option in distutils > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Mar 12 15:14:46 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 12 Mar 2016 12:14:46 -0800 Subject: [Python-ideas] Map-then-filter in comprehensions In-Reply-To: References: <20160308222131.GA19481@sjoerdjob.com> <20160309051621.GA30980@sjoerdjob.com> <20160311132351.GA32080@sjoerdjob.com> <90745071-3E89-471C-91E0-18052ACCBE99@yahoo.com> Message-ID: <7DE31A8D-8A0D-4FDC-BE90-5E1E88EE09D2@yahoo.com> On Mar 11, 2016, at 15:59, Micha? ?ukowski wrote: > > Let statement is pointless - all things you can achievie by simply assign variables. > Except that a let statement starts with a keyword, meaning it can be turned into a comprehension clause, which is the entire point of this thread, and of the (unattributed) message you were replying to. Look at the example you replied to: > [ stripped for idx, line in enumerate(lines) let stripped = line.strip() if idx >= 5 or stripped ] > That's using let as a clause that maps to a statement with the exact same semantics as an assignment statement, and it nests the same way as other comprehension clauses. That's exactly what people are looking for here. Anything you add on top of that, unless it has some independent rationale, is just complication for no benefit. > Let expresion in form "let x=1" cannot "return" value when you want to define more than one variable- with "in" keyword you can define (like in Haskell) more than one variable, and then "return" a value, or you would need to assume that the last defined variable is returned, which is not intuitive :) > Assignments in Python always have a single value. In "x = y = 3", the value is 3. In "x, y = 3, 4" the value is (3, 4). So the value of "let x = y = 3" is 3, and the value of "let x, y = 3, 4" is (3, 4). There's no need for any extra assumptions. The reason Haskell needs a more structured let is scoping (and the fact that Haskell doesn't do sequencing). > 11-03-2016 21:25, "Andrew Barnert" napisa?(a): > > > > > > On Mar 11, 2016, at 07:29, Micha? ?ukowski wrote: > > > >>> > >>> But in Haskell, the `where` keyword also considers scoping. That is, > >>> outside the statement/expression with the `where`, you can't access the > >>> variables introduced by the where. > >>> > >> > >> Yes, but I wanted it to be simple and powerful, not necessarily right in every context. > >> > >>> Even though the `where` looks kind-of-nice, it (at least to me) is also > >>> a bit confusing with respect to evaluation order. Consider > >>> > >>> [ stripped for idx, line in enumerate(lines) if idx >= 5 or stripped where stripped=line.strip() ] > >>> > >>> (intended semantics: give me all lines (stripped), but ignore > >>> any lines that are whitespace-only in the first 5 lines) > >>> > >>> retval = [] > >>> for idx, line in enumerate(lines): > >>> stripped = line.strip() > >>> if idx >= 5 or stripped: > >>> retval.append(stripped) > >>> > >>> now I'm not very sure, but I expect what actually happens is: > >>> > >>> retval = [] > >>> for idx, line in enumerate(lines): > >>> if idx < 5: > >>> stripped = line.strip() > >>> if idx >= 5 or stripped: > >>> retval.append(stripped) > >>> > >>> that is, should I read it as > >>> (if idx >= 5 or stripped) where stripped=line.strip() > >>> or > >>> if idx >= 5 or (stripped where stripped=line.strip()) > >> > >> > >> I've implemented it as "or_test [where_expr]" so the default order is the same as in: > >> (if idx >= 5 or stripped) where stripped=line.strip() > > > > > > This makes where look like another top-level comprehension clause like for and if, but it doesn't follow the same nesting rules as the other clauses. Which means that whenever something isn't trivial enough to just read holistically, trying to apply the dead-simple rule of "write each clause as a nested statement, in the same order" will just lead to confusion, instead of immediately telling you the interpretation. > > > > Where that line is depends on how experienced you are with Python, how much time you've spent recently in a language with similar but not identical comprehension syntax, how tired you are, etc., but that doesn't matter--any comprehension with a where clause has at least three clauses, and is likely to be complicated enough to be over that line for some people. So, if you're going to recommend that this is only used when it's simple enough that it doesn't matter that the translation is confusing, that would mean recommending never using it, which implies that we shouldn't have it. > > > > A new clause that nests in order, like some of your other variations, doesn't have this problem. > > > >> > >> I wanted it to always "scope" left as much as possible - in precedence order between "lambda" and "... if ... else ..." - but left recursion forced me to place it after "... if ... else ..." which can't be used without brackets in list comprehension "if" filtering. > >> But one can always control order with brackets, and make it work like in second example. > >> > >>> For comprehensions, I'd think the 'let' statement might make more sense. > >>> Abusing Haskell's notation: > >>> > >>> [ stripped | (idx, line) <- zip [0..] lines, let stripped = strip line, idx >= 5 || length stripped > 0 ] > >>> > >>> Porting this to something Python-ish, it'd be > >>> > >>> [ stripped for idx, line in enumerate(lines) let stripped = line.strip() if idx >= 5 or stripped ] > >>> > >>> where `let` is a keyword (possibly only applicable in a compexpr). In > >>> Haskell it's a keyword everywhere, but it has somewhat different > >>> semantics. > >> > >> > >> I was thinking about "let", but as I said, I think that new keyword should be more powerful than just filtering in list comprehensions, and in regular expresions it would look like this: > >> if let x=foo() in x: > >> print(x) > >> > >> > >> Which does not look great, and there is problem with "in" keyword that make this expresion ambiguous. > > > > > > I think you're overcomplicating this. You want let to be a statement, but also have a value--but you only need one or the other. > > > > As a statement, there's no need for a value; you're just creating or rebinding a variable for use in the controlled suite: > > > > let x = foo(): > > if x: > > print(x) > > > > And in a comprehension, it's just another clause that converts to the identical compound statement and nests the same as other clauses: > > > > [stripped for line in file let stripped=line.strip() if stripped] > > > > Hopefully you don't want scoping, but if you do, it's pretty obvious that the scope is the controlled suite; still no need for an in clause. > > > > If you really want a let expression instead of a statement, you still don't need an in clause unless you want scoping. Just make it an assignment expression whose value is the assigned value, just like in C and friends: > > > > > > if let x = foo(): > > print(x) > > > > Which extends easily to: > > > > if (let x = foo()) > 5: > > print(x) > > > > [stripped for line in file if let stripped=line.strip()] > > > > The only reason you'd need an in clause is if you wanted scoped expressions. But those would be almost useless in Python because, by design, you can't do too much inside an expression. But if you really want that, the design is again obvious: just as in functional languages, it's equivalent to a lambda call: > > > > let x=foo() in (print(x) if x > 5 else None) > > > > (lambda x: print(x) if x > 5 else None)(foo()) > > > > ... which is obviously unpythonic enough and useless enough that I think scoped let expressions are immediately dismissible as an idea. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Sat Mar 12 18:38:43 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sat, 12 Mar 2016 23:38:43 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: On 9 March 2016 at 21:25, Chris Angelico wrote: > On Thu, Mar 10, 2016 at 8:20 AM, Mark Lawrence wrote: >> I do not believe that this needs a PEP, a tracker item should be fine IMHO. >> We should just ensure that the original authors of the PEP are aware, as >> missing out their opinions would be a disservice to the Python Windows >> community, as small as that may be when compared to *nix. > > Given that this is basically a Windows-only change, it doesn't matter > how large or small the Windows community is - they're the people who > primarily matter :) If someone proposed a change to the way things are > done on OS/2, it'd be worth getting the opinions of both of the Python > OS/2 users... :P Although it is a change to Windows-only I think it is important to think about the consistency between the way that this works on Windows compared with other operating systems. Inconsistencies here can be awkward for novices. Allow me to spell out my scenario... I teach a couple of University programming units in which students learn to program in Python, C and Java. The cohort for these units this year was 160 students. I often find myself giving the students instructions such as you can run this script I've given you with $ python myscript.py Now imagine that you've given that instruction to 160 students mostly using Windows and OSX (and a few Linux users). I find it very annoying that this could invoke Python 2 or 3 depending on the OS etc that the students are using. py.exe should solve this since I can write py -3 and then (I assume) it would lead to an error if no Python 3 version is available. However that doesn't work on OSX or Linux. Okay so maybe we should just use the shebangs. I can explain to all my students what shebangs are and instruct the OSX students to set the executable bit. Then I can say so just run $ ./myscript.py however on Windows this uses file associations which is flaky. In the past I've seen bugs where e.g. input/output redirection didn't work for scripts run in this way. It also may not work in myriad other contexts where a command line may be used. One example is from Makefiles. So when my students are working in C I can give them a Python script that will test the output of their C program and I can tell them to add this to their Makefile e.g.: test: myprog.exe ./myscript.py At this point (on Windows) various failures are possible. If make simply calls CreateProcess it fails since myscript.py isn't an exe file (I believe it also accepts .bat files but does not observe file associations). If make inspects the shebang it may come to a very different conclusion from the one that py.exe would have (e.g. that python3 does not exist since it is not on PATH). Over time I've seen this go wrong in a number of ways and have ultimately concluded that anything involving using file associations is fragile. Personally I would really like it if we could have improved consistency across OSes when it comes to invoking Python. So I would really like it if we can either have py (and maybe py3) *everywhere* or if that's not possible find some solution that does work everywhere. Can we have a single command line that guarantees to run Python 3 (or error out)? -- Oscar From breamoreboy at yahoo.co.uk Sat Mar 12 20:25:50 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sun, 13 Mar 2016 01:25:50 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: On 12/03/2016 23:38, Oscar Benjamin wrote: > On 9 March 2016 at 21:25, Chris Angelico wrote: >> On Thu, Mar 10, 2016 at 8:20 AM, Mark Lawrence wrote: >>> I do not believe that this needs a PEP, a tracker item should be fine IMHO. >>> We should just ensure that the original authors of the PEP are aware, as >>> missing out their opinions would be a disservice to the Python Windows >>> community, as small as that may be when compared to *nix. >> >> Given that this is basically a Windows-only change, it doesn't matter >> how large or small the Windows community is - they're the people who >> primarily matter :) If someone proposed a change to the way things are >> done on OS/2, it'd be worth getting the opinions of both of the Python >> OS/2 users... :P > > Although it is a change to Windows-only I think it is important to > think about the consistency between the way that this works on Windows > compared with other operating systems. Inconsistencies here can be > awkward for novices. Allow me to spell out my scenario... > > I teach a couple of University programming units in which students > learn to program in Python, C and Java. The cohort for these units > this year was 160 students. I often find myself giving the students > instructions such as you can run this script I've given you with > > $ python myscript.py > > Now imagine that you've given that instruction to 160 students mostly > using Windows and OSX (and a few Linux users). I find it very annoying > that this could invoke Python 2 or 3 depending on the OS etc that the > students are using. py.exe should solve this since I can write py -3 > and then (I assume) it would lead to an error if no Python 3 version > is available. However that doesn't work on OSX or Linux. > > Okay so maybe we should just use the shebangs. I can explain to all my > students what shebangs are and instruct the OSX students to set the > executable bit. Then I can say so just run > > $ ./myscript.py > > however on Windows this uses file associations which is flaky. In the > past I've seen bugs where e.g. input/output redirection didn't work > for scripts run in this way. It also may not work in myriad other > contexts where a command line may be used. One example is from > Makefiles. So when my students are working in C I can give them a > Python script that will test the output of their C program and I can > tell them to add this to their Makefile e.g.: > > test: myprog.exe > ./myscript.py > > At this point (on Windows) various failures are possible. If make > simply calls CreateProcess it fails since myscript.py isn't an exe > file (I believe it also accepts .bat files but does not observe file > associations). If make inspects the shebang it may come to a very > different conclusion from the one that py.exe would have (e.g. that > python3 does not exist since it is not on PATH). Over time I've seen > this go wrong in a number of ways and have ultimately concluded that > anything involving using file associations is fragile. > > Personally I would really like it if we could have improved > consistency across OSes when it comes to invoking Python. So I would > really like it if we can either have py (and maybe py3) *everywhere* > or if that's not possible find some solution that does work > everywhere. Can we have a single command line that guarantees to run > Python 3 (or error out)? > > -- > Oscar The thing that drives me nuts is that a shebang line screws up running a pyw file on windows. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From abarnert at yahoo.com Sat Mar 12 23:09:37 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 12 Mar 2016 20:09:37 -0800 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> Message-ID: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> On Mar 12, 2016, at 15:38, Oscar Benjamin wrote: > >> On 9 March 2016 at 21:25, Chris Angelico wrote: >>> On Thu, Mar 10, 2016 at 8:20 AM, Mark Lawrence wrote: >>> I do not believe that this needs a PEP, a tracker item should be fine IMHO. >>> We should just ensure that the original authors of the PEP are aware, as >>> missing out their opinions would be a disservice to the Python Windows >>> community, as small as that may be when compared to *nix. >> >> Given that this is basically a Windows-only change, it doesn't matter >> how large or small the Windows community is - they're the people who >> primarily matter :) If someone proposed a change to the way things are >> done on OS/2, it'd be worth getting the opinions of both of the Python >> OS/2 users... :P > > Although it is a change to Windows-only I think it is important to > think about the consistency between the way that this works on Windows > compared with other operating systems. Inconsistencies here can be > awkward for novices. Allow me to spell out my scenario... > > I teach a couple of University programming units in which students > learn to program in Python, C and Java. The cohort for these units > this year was 160 students. I often find myself giving the students > instructions such as you can run this script I've given you with > > $ python myscript.py > > Now imagine that you've given that instruction to 160 students mostly > using Windows and OSX (and a few Linux users). I find it very annoying > that this could invoke Python 2 or 3 depending on the OS etc that the > students are using. py.exe should solve this since I can write py -3 > and then (I assume) it would lead to an error if no Python 3 version > is available. However that doesn't work on OSX or Linux. What if Python 3 created a symlink (or, for non-NTFS drives, a copy) of Python.exe named Python3.exe? Would that be sufficient for you to just tell your students to run: $ python3 myscript.py At any rate, this seems to be a separate issue from the one this thread is about. It's already true that py.exe is Windows-specific; no change to what it does is going to make the same command lines work on Windows and *nix. Only some new proposal (add a py command for Unix, add python2 and python3 links for Windows, make Python.exe itself a dispatcher as Apple's /usr/bin/python is on Mac, ...) is going to help you. So you might be better off creating a new thread instead of adding on to one that's mostly been resolved and a lot of people have stopped reading. From p.f.moore at gmail.com Sun Mar 13 07:39:19 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 13 Mar 2016 11:39:19 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: On 13 March 2016 at 04:09, Andrew Barnert via Python-ideas wrote: > At any rate, this seems to be a separate issue from the one this thread is about. It's already true that py.exe is Windows-specific; no change to what it does is going to make the same command lines work on Windows and *nix. Only some new proposal (add a py command for Unix, add python2 and python3 links for Windows, make Python.exe itself a dispatcher as Apple's /usr/bin/python is on Mac, ...) is going to help you. So you might be better off creating a new thread instead of adding on to one that's mostly been resolved and a lot of people have stopped reading. Agreed. Just to summarise the final state of this thread, the consensus seems to be: 1. For interactive use, make py.exe launch the latest version installed on the machine, without special-casing a preference for Python 2. 2. For shebang lines where an unversioned name is used, retain the current behaviour (for compatibility with Unix). 3. When the user explicitly chooses a version, or has configured the launcher via the ini file or environment variables, no change to current behaviour. 4. The change is small enough that it doesn't need a PEP. On that basis, I'll make some time to brush off my C skills and raise a tracker item, targeted at Python 3.6. As far as I know, there's no special exemption for new features in the launcher being backported to older versions of Python, and there's no "standalone" launcher installer, so this change will only affect people who install the launcher via the Python 3.6 or later installer. Paul From k7hoven at gmail.com Sun Mar 13 15:13:48 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 13 Mar 2016 21:13:48 +0200 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: The past and future of Python (a little off topic, sorry :): Python 2: $ python myapp.py Python 3: $ py myapp.py Python 4: $ p myapp.py Python 5: $ myapp Python 6: Executing Python applications will be natively built into every OS, and the problem of a cross-platform way to conveniently distribute and run a Python *application* will be magically solved. - Koos From oscar.j.benjamin at gmail.com Mon Mar 14 13:18:26 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 14 Mar 2016 17:18:26 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: On 13 March 2016 at 11:39, Paul Moore wrote: > On 13 March 2016 at 04:09, Andrew Barnert via Python-ideas > wrote: >> At any rate, this seems to be a separate issue from the one this thread is about. It's already true that py.exe is Windows-specific; no change to what it does is going to make the same command lines work on Windows and *nix. Only some new proposal (add a py command for Unix, add python2 and python3 links for Windows, make Python.exe itself a dispatcher as Apple's /usr/bin/python is on Mac, ...) is going to help you. So you might be better off creating a new thread instead of adding on to one that's mostly been resolved and a lot of people have stopped reading. > > Agreed. I guess it's maybe just because this thread has made me realise that py.exe has changed purpose with respect to what it seemed (to me) to be before. When I was using Windows I used to always arrange for python and python3 to be available on PATH and I only used the "py" command as a way of explicitly selecting a particular non-default Python version. The idea that the default for py.exe needs to be python 3 suggests that people are now using "py" as the primary command line invocation of Python on Windows. I think is unfortunate as it takes Windows off in a different direction from every other OS. That intention wasn't the impression I had of PEP 397 at the time. -- Oscar From tjreedy at udel.edu Mon Mar 14 15:52:25 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 14 Mar 2016 15:52:25 -0400 Subject: [Python-ideas] How the heck does async/await work in Python 3.5 In-Reply-To: References: <56c7d145$0$1597$c3e8da3$5496439d@news.astraweb.com> <56CCC98C.5060504@mail.de> <56CEA414.7070609@udel.edu> Message-ID: <56E71679.30305@udel.edu> On 2/25/2016 11:47 AM, Guido van Rossum wrote: > On Wed, Feb 24, 2016 at 10:49 PM, Terry Reedy wrote: >> My memory is that any replacement event loop is expected to be a full >> implmentation of the network io loop, so one cannot just make a gui loop >> that works with coroutines. > > That's not correct. If an I/O loop does not implement the network I/O > APIs it just cannot be used for network I/O, but it can still > implement all the other functionality (like call_later). The tk loop already has the run and call functionality (and gui event stuff). The reason I want to integrate tkinter and asyncio is to be able to use the I/O APIs. In particular, I want to experiment with subprocess_exec. Assuming that my reading of the PEP is correct, I want to try using it run code checkers on code (such as the contents of an IDLE editor file). A GSOC student worked out everything but how to do so without blocking and freezing the ui. Beyond that, we need to replace the unreliable (but non-blocking) socket connection between IDLE and the user execution process). > There are > other loops that don't implement everything (e.g. the subprocess and > datagram APIs are not supported by Proactor, According to the PEP and doc https://docs.python.org/3/library/asyncio-eventloops.html#windows, the Proactor is the loop that supports subprocesses on Windows, while Windows Selector does not. I will find out for sure soon enough. > and signal support is > spotty, too). If you find some implementation barrier that prevents > that from working it should be fixed. After lots of reading and thinking, I decided to start with simplest this that might work, and came up with the following surprisingly simple subclass. It works with GUI versions of the first two examples in the doc. It will satisfy many simple cases of using tkinter and asyncio together. I will continue with other examples The docstring describes possible ways to improve the integration. I am not sure whether root.update handles the events handled by update_idletasks (or where 'idletasks' come from). If not, update_idletask() calls may be needed. Serhiy should know. --- '''Proof of concept for integrating asyncio and tk loops. Terry Jan Reedy Run with 'python -i' or from IDLE editor to keep tk window alive. This version simply calls tk root.update in run_forever. Without this, the tk window only appears after loop.stop, unless root.update() is put in callbacks instead. If there are no pending asyncio callbacks, self._run_once will block indefinitely in a .select(None) call, waiting for an I/O event. To ensure that all tk events and callbacks get timely attention, either _run_once should be revised or __init__ should start an asyncio callback loop so that there is always a pending callback. The details will depend on whether the GUI or network IO is intended to get priority attention. ''' import asyncio as ai import threading import tkinter as tk import datetime # for example # On Windows, need proactor for subprocess functions EventLoop = ai.ProactorEventLoop # Following gets ai.SelectorEventLoop # EventLoop = type(ai.get_event_loop()) class TkEventLoop(EventLoop): def __init__(self, root): self.root = root super().__init__() def run_forever(self): # copy with 1 new line for proof of concept """Run until stop() is called.""" self._check_closed() if self.is_running(): raise RuntimeError('Event loop is running.') self._set_coroutine_wrapper(self._debug) self._thread_id = threading.get_ident() try: while True: self._run_once() # can block indefinitely self.root.update() # new line if self._stopping: break finally: self._stopping = False self._thread_id = None self._set_coroutine_wrapper(False) root = tk.Tk() loop = TkEventLoop(root) ai.set_event_loop(loop) # Combine 2 event loop examples from BaseEventLoop doc. hello = tk.Label(root) timel = tk.Label(root) hello.pack() timel.pack() def hello_world(loop): hello['text'] = 'Hello World' loop.call_soon(hello_world, loop) def display_date(end_time, loop): timel['text'] = datetime.datetime.now() if (loop.time() + 1.0) < end_time: loop.call_later(1, display_date, end_time, loop) else: loop.stop() end_time = loop.time() + 5.1 loop.call_soon(display_date, end_time, loop) # Blocking call interrupted by loop.stop() loop.run_forever() --- > The await expression (or yield from) does not require an event loop. > Heck, it doesn't require futures. Without either, are async/await actually useful? The binary/abinary examples just shows that there is not much penalty for using async/await when not needed. On my machine, abinary is pretty consistently 2% slower. > It uses coroutines and it is totally > up to the manager of those how they are scheduled (read Greg Ewing's > very early explanations of yield from -- some have no I/O at all). What do you mean by 'manager of [coroutines]'? For loop, event loop, task, the nebulous 'scheduler', or something else? > But there's something else that you might be after. After getting non-blocking gui and io events (with callbacks) working together, I an after asynchronous iteration. The asyncio modules used cooroutines for suspension, but as far as I can tell, not for iteration. Normal synchronous iteration is fine for compute-bound iterators and a single iterator whose delays are tolerable because the thread or process has nothing better to do. (And the system will switch to something else when the thread blocks, so the cpu is not actually blocked.) But it does not work for concurrent intermitant streams. For example, consider the datetime display in the example above. The code is scattered around in three places. I would like to be able to replace the pieces with synchronous-like code, something like the following. async def display_date(interval, end_time, loop=ai.get_event_loop()): label = tk.Label(root) label.pack() async for tick in timer_soon(interval, end_time, loop): label['text'] = datetime.datetime.now() The asynchronous iterator timer_soon would incorporate call_soon, call_later, and stop calls and 'yield' True at the appointed times. Doing so allows the widget creation and repeated widget event handling to be combined in one function. How do I write it? The async iterator examples in PEP 492 (the second is Example 1) shows all the boiler plate, which I could write from the spec given, but omit the essential detail of how to write the inner cooroutine that gets awaited on. async def fetch_data(self): ... async def _prefetch(self): ... ??? > When you're implementing this API on top of tkinter, you'll probably > find that you'll have to use tkinter's way of sleeping anyways, so the > implementation of waiting in BaseEventLoop using a selector is not > useful for this scenario. Are you referring to the possibly indefinite blocking call to select in ._run_once? A possible rewrite is to block on a tk root.mainloop call and make periodic non-blocking select(0) calls in callbacks. If this were done, it might make sense to re-write call_later and call_at to use tk's root.after and callback scheduler. I don't know if the same can be done with call_soon while maintaining the detailed run_forever spec in the 3.5.1 doc. > There are probably some possible refactorings in the asyncio package > to help you reuse a little more code, I decided to start by reusing all code and just add a tk call. TkEventLoop will pass the same test suite as its superclass. > but all in all I still think it would be very useful to have an > asyncio loop integrated with Tkinter. Most definitely. 'An' example is trivial, as posted above. I will see how far it goes in running other examples. > (Of course Tkinter does support network I/O, Tcl's createfilehandler function is only available on unix. I read somewhere that is uses selector polling. > or somehow figure out how to wait using a Selector > *or* tkinter events in the same loop. I suspect that waiting on one and polling the other is the only cross-platform solution. -- Terry Jan Reedy From p.f.moore at gmail.com Mon Mar 14 19:01:21 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 14 Mar 2016 23:01:21 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: On 14 March 2016 at 17:18, Oscar Benjamin wrote: > I guess it's maybe just because this thread has made me realise that > py.exe has changed purpose with respect to what it seemed (to me) to > be before. When I was using Windows I used to always arrange for > python and python3 to be available on PATH and I only used the "py" > command as a way of explicitly selecting a particular non-default > Python version. The idea that the default for py.exe needs to be > python 3 suggests that people are now using "py" as the primary > command line invocation of Python on Windows. I think is unfortunate > as it takes Windows off in a different direction from every other OS. > That intention wasn't the impression I had of PEP 397 at the time. Indeed, it wasn't, and I would have always said that people should put the version of Python that they want on PATH. Indeed, I still do that. I don't think there's any "official" change in policy, nor is there a groundswell of people using py.exe to invoke Python on the command line. But PATH management remains messy and is probably even more so now that per-user installs are the default (per-user PATH entries go after system PATH entries on Windows, which makes the default behaviour complex to describe, there's no way around that). So using py.exe (which is always on PATH, and which gives access to any Python) is possibly a more attractive option nowadays. I've personally been experimenting with using it, and I've found that the "Python 2 takes priority" behaviour is suboptimal for my interactive use - this thread implies that I'm not alone in that. So I'll be following up with a change to improve that default. That's all, as far as I'm concerned. But as for "the canonical way to run Python" - much less "the canonical way to ensure you're running Python 3 (or 2)" - I don't think there's any such thing, and honestly there never has been since Python 3 came into existence and a proportion of the community needed "python" to continue referring to Python 2, even though a newer version had been released. If you (or anyone else) wanted to put together a proposal for such a cross-platform standard way of launching specific Python versions, then by all means go ahead. I can imagine it would be immensely useful to have such a thing in a teaching context. Personally it would be of no value to me, though, so I'm not going to be interested in such a proposal. I apologise if I gave the impression in this thread that there's any sort of "official" change in advice on how to launch Python. As far as I know, there hasn't been. Paul From ncoghlan at gmail.com Tue Mar 15 00:47:34 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Mar 2016 14:47:34 +1000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: On 15 March 2016 at 09:01, Paul Moore wrote: > I apologise if I gave the impression in this thread that there's any > sort of "official" change in advice on how to launch Python. As far as > I know, there hasn't been. It's still "python" - that works regardless of version on Windows (with a suitably configured PATH), in virtual environments and in pyenv and conda environments (and probably in Enthought Canopy as well, but I haven't checked that). The only places where it doesn't work are: * Windows without a properly configured PATH (and without a tool like vex or conda) * accessing Python 3 on *nix systems (without a tool like vex, pyenv or conda) On the *nix side of things, the long term objective is to get "python" to a point of being usable for the latter case, but there's still a lot of other things to be done before that's a realistic option (e.g. migrating system packages over to using their own dedicated binary or symlink, so "python" can be switched to point to something else without affecting system services). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Mar 15 04:56:12 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 15 Mar 2016 08:56:12 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: On 15 March 2016 at 04:47, Nick Coghlan wrote: > * accessing Python 3 on *nix systems (without a tool like vex, pyenv or conda) > > On the *nix side of things, the long term objective is to get "python" > to a point of being usable for the latter case, but there's still a > lot of other things to be done before that's a realistic option (e.g. > migrating system packages over to using their own dedicated binary or > symlink, so "python" can be switched to point to something else > without affecting system services). Ah, OK. I hadn't realised that was a goal for Unix. That's good to know. Paul From oscar.j.benjamin at gmail.com Tue Mar 15 07:25:47 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 15 Mar 2016 11:25:47 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: On 15 March 2016 at 04:47, Nick Coghlan wrote: > On 15 March 2016 at 09:01, Paul Moore wrote: >> I apologise if I gave the impression in this thread that there's any >> sort of "official" change in advice on how to launch Python. As far as >> I know, there hasn't been. You don't need to apologise in any case but that's not what I meant. Also I'm not disagreeing with your proposal to change the default behaviour of py.exe. It's not so much "official policy" that I was lamenting but just the way things have evolved. I don't think that anyone really intended to reach the current situation where the different OSes have diverged and there's no longer a clear uniform command line to run a Python script. > It's still "python" - that works regardless of version on Windows > (with a suitably configured PATH), in virtual environments and in > pyenv and conda environments (and probably in Enthought Canopy as > well, but I haven't checked that). The only places where it doesn't > work are: > > * Windows without a properly configured PATH (and without a tool like > vex or conda) AFAICT the default behaviour of the Windows installer does not add Python to PATH. Correct me if I'm wrong as I haven't used it myself for some time but when I tell my students to "install Python 3.5" and then to type "python" in the terminal a lot of the Windows users come back to me with PATH problems (specifically that python was not added to PATH). > * accessing Python 3 on *nix systems (without a tool like vex, pyenv or conda) > > On the *nix side of things, the long term objective is to get "python" > to a point of being usable for the latter case, but there's still a > lot of other things to be done before that's a realistic option (e.g. > migrating system packages over to using their own dedicated binary or > symlink, so "python" can be switched to point to something else > without affecting system services). Well that would initially improve things but what happens when python 4 comes around? Really I think that for something like this having major-versioned executables is a good idea. It's just a shame that it wasn't done consistently. -- Oscar From abarnert at yahoo.com Tue Mar 15 07:50:26 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 15 Mar 2016 04:50:26 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: <28EC2634-6908-4017-BBD8-7198B618963F@yahoo.com> On Mar 15, 2016, at 04:25, Oscar Benjamin wrote: > >> On 15 March 2016 at 04:47, Nick Coghlan wrote: >> >> On the *nix side of things, the long term objective is to get "python" >> to a point of being usable for the latter case, but there's still a >> lot of other things to be done before that's a realistic option (e.g. >> migrating system packages over to using their own dedicated binary or >> symlink, so "python" can be switched to point to something else >> without affecting system services). > > Well that would initially improve things but what happens when python > 4 comes around? Nothing. It'll be called just "python" on both platforms. Just like when Python 2.0 came out, it was just called "python", not "python2", and was no more incompatible with 1.5 than 1.5 had been with 1.4. Python 3 was a once-in-a-lifetime, or at least once-in-a-generation, thing, not a plan to overhaul everything once/decade. (At least that's the impression I get from the PEPs, Guido's blogs, other dev's blogs, and the mailing lists.) The only way I can imagine Python 4.0 being incompatible is 3.9 is followed by 3.10 and somewhere in the distant future the memory of the 2.5->3.0 transition has faded (or the *nix environment has changed) to the point where a similar 3.14->4.0 seems like a good idea. From p.f.moore at gmail.com Tue Mar 15 08:01:56 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 15 Mar 2016 12:01:56 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: On 15 March 2016 at 11:25, Oscar Benjamin wrote: > AFAICT the default behaviour of the Windows installer does not add > Python to PATH. Correct me if I'm wrong as I haven't used it myself > for some time but when I tell my students to "install Python 3.5" and > then to type "python" in the terminal a lot of the Windows users come > back to me with PATH problems (specifically that python was not added > to PATH). I would have thought that is relatively easy to resolve. A pain to have to do in a classroom environment, certainly, but manageable. The reasons Python is not added to PATH are something that has been debated over and over again on various lists. As far as I know, neither option is ideal, but "not on PATH by default" is typically seen as the least likely to fail in confusing ways. You could (relatively easily) write a small Windows Scripting Host script to locate the version of Python you wanted people to use via the registry, and add it to the front of PATH. That would need elevation to work, of course, but would give a known-good starting point. On Unix, tell people just "run python3". Having versioned executables available on Windows might be helpful, but only a little bit as they still wouldn't be on PATH. Paul From oscar.j.benjamin at gmail.com Tue Mar 15 08:38:48 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 15 Mar 2016 12:38:48 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: On 15 March 2016 at 12:01, Paul Moore wrote: > On 15 March 2016 at 11:25, Oscar Benjamin wrote: >> AFAICT the default behaviour of the Windows installer does not add >> Python to PATH. Correct me if I'm wrong as I haven't used it myself >> for some time but when I tell my students to "install Python 3.5" and >> then to type "python" in the terminal a lot of the Windows users come >> back to me with PATH problems (specifically that python was not added >> to PATH). > > I would have thought that is relatively easy to resolve. A pain to > have to do in a classroom environment, certainly, but manageable. It's easy enough to resolve if they bring their computer directly to me. It's less easy to resolve by email. It just becomes something many students waste time trying to fix when they're not yet ready to meaningfully learn from the experience. > The > reasons Python is not added to PATH are something that has been > debated over and over again on various lists. As far as I know, > neither option is ideal, but "not on PATH by default" is typically > seen as the least likely to fail in confusing ways. Why would we not want python on PATH? If it's because of having multiple installed Pythons then maybe it's possible to improve the installer so that it gives the user a selection of which is on PATH and adds it by default if it isn't already there. A simple rule: "if Python is not on PATH then the installer assumes that the user wants to add it" would suffice. > You could (relatively easily) write a small Windows Scripting Host > script to locate the version of Python you wanted people to use via > the registry, and add it to the front of PATH. That would need > elevation to work, of course, but would give a known-good starting > point. I would probably have more success telling them to tick the "add to PATH" box on the installer :) (A proportion would get it wrong whatever I say though) > On Unix, tell people just "run python3". It's not always a question of telling people what command to run. The particular scenario that just came up is that of a C program with a Makefile and a Python script that tests the output of the C program. So I give the students a skeleton project that looks like proj-1.0.zip/ proj-1.0/ projmain.c Makefile tests/ runtests.py The idea is that they should fill in the C code to complete a program. The runtests.py script runs their program testing input/output combinations and scores how much they have completed. Now in Makefile I have test: proj.exe python tests/runtests.py So the students can run "make test" to rebuild their project and run the tests. The question is how to set this up when "python" is not guaranteed to be on PATH and could be many different versions of Python. ATM I have to exhaustively test all combinations of Python 2.7 and 3.5 and Windows, OSX and Linux and then I still hit up against problems because the way in which they've installed Python is not the same as the machines I tested on. Testing the output of a subprocess in 2/3 compatible code is painful because it comes out as bytes in 3 and str in 2. Really the only necessary incompatibility that I should need to deal with in this scenario is Windows line endings. All the others like which version of python will run, whether or not python is on PATH and the 2/3 incompatibilities are all down to Python rather than the inherent variabilities of the OSes. -- Oscar From p.f.moore at gmail.com Tue Mar 15 09:12:53 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 15 Mar 2016 13:12:53 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: On 15 March 2016 at 12:38, Oscar Benjamin wrote: > Why would we not want python on PATH? You should probably go and hunt out previous discussions (your best bet is likely the python-dev archives). They were pretty extensive, and I'd likely get details wrong if I tried to summarise. > If it's because of having > multiple installed Pythons then maybe it's possible to improve the > installer so that it gives the user a selection of which is on PATH > and adds it by default if it isn't already there. Certainly it's only an issue if the user has multiple Pythons. Of course, this can include things like conda or ActiveState Python, or programs like Lilypond that bundle Python. Some issues I recall: 1. You can improve the Python 3.6 installer, but if the user installs Python 2.7 after 3.6, they'll get the old installer with that. And conda may do something else altogether. So fixing just one installer doesn't remove the issue. 2. User installs can only add to the user part of PATH, which the OS adds *after* the system PATH. So if you do a system install and a user install, the system install will *always* take precendence, regardless of what you ask for. This is particularly problematic as Python 3.5+ defaults to a user install. 3. When you ask for a new Python to be added to PATH, should previous ones be removed? If you then uninstall the new Python, will you be left with nothing on PATH? I've seen the view expressed that PATH is for the user to manage and installers have no business changing it at all. While I'm not sure that many people who write installers would agree with that, having dealt with my own share of messed up PATHs caused by installers, I can certainly sympathise with that position. Paul From ncoghlan at gmail.com Tue Mar 15 09:19:56 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Mar 2016 23:19:56 +1000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56DF3FE8.70600@sdamon.com> <56DF4183.4000308@stoneleaf.us> <56DF457F.5090401@sdamon.com> <56DF4B84.6000702@python.org> <56DF9FAB.1080204@sdamon.com> <56DFA15F.7090000@sdamon.com> <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: On 15 March 2016 at 21:25, Oscar Benjamin wrote: > On 15 March 2016 at 04:47, Nick Coghlan wrote: >> * Windows without a properly configured PATH (and without a tool like >> vex or conda) > > AFAICT the default behaviour of the Windows installer does not add > Python to PATH. Correct me if I'm wrong as I haven't used it myself > for some time but when I tell my students to "install Python 3.5" and > then to type "python" in the terminal a lot of the Windows users come > back to me with PATH problems (specifically that python was not added > to PATH). Yeah, it's unfortunately currently still off by default. Since it's sysadmins that don't like it being on by default, and they're better equipped to change it back than Windows users being introduced to the command line for the first time, I'd prefer to see that particular default flipped to be opt-out rather than opt-in. Educators are the folks best positioned to argue for that change, since you're the ones that have to deal with the fallout of beginners installing Python by clicking through the installer prompts without realising they need to request the automatic PATH modification. >> * accessing Python 3 on *nix systems (without a tool like vex, pyenv or conda) >> >> On the *nix side of things, the long term objective is to get "python" >> to a point of being usable for the latter case, but there's still a >> lot of other things to be done before that's a realistic option (e.g. >> migrating system packages over to using their own dedicated binary or >> symlink, so "python" can be switched to point to something else >> without affecting system services). > > Well that would initially improve things but what happens when python > 4 comes around? Really I think that for something like this having > major-versioned executables is a good idea. It's just a shame that it > wasn't done consistently. The major & minor versioned executables exist on *nix if people want to use them, but tend to make scripts and instructions more platform specific than they need to be. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From desmoulinmichel at gmail.com Tue Mar 15 12:50:33 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 15 Mar 2016 17:50:33 +0100 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> Message-ID: <56E83D59.5080707@gmail.com> Le 15/03/2016 14:12, Paul Moore a ?crit : > On 15 March 2016 at 12:38, Oscar Benjamin wrote: >> Why would we not want python on PATH? > > You should probably go and hunt out previous discussions (your best > bet is likely the python-dev archives). They were pretty extensive, > and I'd likely get details wrong if I tried to summarise. > >> If it's because of having >> multiple installed Pythons then maybe it's possible to improve the >> installer so that it gives the user a selection of which is on PATH >> and adds it by default if it isn't already there. > > Certainly it's only an issue if the user has multiple Pythons. Of > course, this can include things like conda or ActiveState Python, or > programs like Lilypond that bundle Python. > > Some issues I recall: > > 1. You can improve the Python 3.6 installer, but if the user installs > Python 2.7 after 3.6, they'll get the old installer with that. And > conda may do something else altogether. So fixing just one installer > doesn't remove the issue. > 2. User installs can only add to the user part of PATH, which the OS > adds *after* the system PATH. So if you do a system install and a user > install, the system install will *always* take precendence, regardless > of what you ask for. This is particularly problematic as Python 3.5+ > defaults to a user install. > 3. When you ask for a new Python to be added to PATH, should previous > ones be removed? If you then uninstall the new Python, will you be > left with nothing on PATH? > > I've seen the view expressed that PATH is for the user to manage and > installers have no business changing it at all. While I'm not sure > that many people who write installers would agree with that, having > dealt with my own share of messed up PATHs caused by installers, I can > certainly sympathise with that position. > > Paul Yes, but you are able to deal with those issues. And they happen rarely. Failing to run Python in the terminal occurs every single introduction course I give. Every single one. And without help, a beginner will have trouble solving it because it's hard to search in Google for something you know nothing about. So some of them, the ones that can understand English and see the value of looking it up for hours, may solve it. The other ones will give up. Another problem is the gigantic time and energy wasted for this : I have to tell about this box in every course, write it down (with screenshot) in every tutorials, mention it in every videos I do. And yet, yet! People still come to me, write comments, emails, not understanding why typing "python" in this big black windows doesn't work. And then, you have the worst case scenario. The one where the person, a bit tech saavy, tried to edit the PATH manually in this tiny, unresizeable window, and messed it up. Now, try to diagnose that over skype, or on a forum, or on twitter... I already made the choice to spends hours of my life helping people with Python, but I wish I could spend it on explaining how they can make their computer do nice stuff, not on making sure they tick boxes. From p.f.moore at gmail.com Tue Mar 15 13:47:24 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 15 Mar 2016 17:47:24 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56E83D59.5080707@gmail.com> References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> Message-ID: On 15 March 2016 at 16:50, Michel Desmoulin wrote: > Yes, but you are able to deal with those issues. And they happen rarely. Agreed. One of the issues with the debates about whether to add Python to PATH by default was that the people involved - the Python core devs - were not at all typical users (they probably have multiple Python versions installed, and they are comfortable with technical details like PATH management, etc). As Nick said, we need people like educators, with real experience of what the stumbling blocks for new users are, to provide a more realistic justification for the change. Maybe someone in this discussion could raise a feature request on bugs.python.org suggesting this gets changed? Paul From random832 at fastmail.com Tue Mar 15 15:31:55 2016 From: random832 at fastmail.com (Random832) Date: Tue, 15 Mar 2016 15:31:55 -0400 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> Message-ID: <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> On Tue, Mar 15, 2016, at 13:47, Paul Moore wrote: > Agreed. One of the issues with the debates about whether to add Python > to PATH by default was that the people involved - the Python core devs > - were not at all typical users (they probably have multiple Python > versions installed, and they are comfortable with technical details > like PATH management, etc). I think what might be most appropriate would be to have a "Python Console" in the start menu (alongside the IDLE entry) which sources a bat file to add the correct python version (and the scripts directory for pip and easy_install) to that window's PATH, rather than installing it in the global PATH, and users can learn to start the console from that icon when they want to run python commands. Inspired by the fact that e.g. Visual Studio has something that does the same. From steve.dower at python.org Tue Mar 15 16:58:11 2016 From: steve.dower at python.org (Steve Dower) Date: Tue, 15 Mar 2016 13:58:11 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> Message-ID: <56E87763.80000@python.org> On 15Mar2016 1231, Random832 wrote: > I think what might be most appropriate would be to have a "Python > Console" in the start menu (alongside the IDLE entry) which sources a > bat file to add the correct python version (and the scripts directory > for pip and easy_install) to that window's PATH, rather than installing > it in the global PATH, and users can learn to start the console from > that icon when they want to run python commands. Inspired by the fact > that e.g. Visual Studio has something that does the same. This is one of two solutions I'd be comfortable with. The other is to rename "py[w].exe" as "python[w].exe". PATH is used to resolve system DLLs and common commands. Over time, better security features are being added for core system components, but it is still possible to hijack programs this way. It also has a serious conflict problem once the second version of Python is installed with this option enabled, in that *anything* currently referring to just "python" will change. In my experience, it's mostly Python 3.x turning into 2.7 because a keen Python developer meets a jaded one and gets told they should only be using old things... Windows Installer does not have a nice mechanism to handle ordering of PATH entries or detecting existing ones, unless all the installers are cooperating. It certainly doesn't handle any case here where users may add/remove Python installs whenever they like. This is why the launcher was created. I have always been -1 on modifying PATH by default, and trading an early problem for a later, more subtle, problem does not convince me. And I say this as someone who has taught classes of undergraduate students how to use Python on Windows, and had to deal with the same problems. I'm not trying to lord it over the installer just because I'm the one who wrote and maintains it - if someone else files a patch and one of the other committers puts it in, so be it. But I'd much rather (a) make users choose up front to have an environment where "python.exe" is easily accessible, or (b) make "python.exe" consistently launch the right version. Cheers, Steve From p.f.moore at gmail.com Tue Mar 15 17:10:26 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 15 Mar 2016 21:10:26 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56E87763.80000@python.org> References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> <56E87763.80000@python.org> Message-ID: On 15 March 2016 at 20:58, Steve Dower wrote: > On 15Mar2016 1231, Random832 wrote: >> >> I think what might be most appropriate would be to have a "Python >> Console" in the start menu (alongside the IDLE entry) which sources a >> bat file to add the correct python version (and the scripts directory >> for pip and easy_install) to that window's PATH, rather than installing >> it in the global PATH, and users can learn to start the console from >> that icon when they want to run python commands. Inspired by the fact >> that e.g. Visual Studio has something that does the same. > > > This is one of two solutions I'd be comfortable with. > > The other is to rename "py[w].exe" as "python[w].exe". I'm very conscious that as an experienced user, I don't actually care what the installer does, as I'll reconfigure things the way I want anyway. However, in my opinion, the worst part of having a "Python command prompt" start menu item is that it assumes everyone wants to use cmd.exe. But there's also powershell users to consider. Furthermore, having a command prompt that does what you want doesn't help people who want to run Python from something like a scheduled task. Of course, for Python it's no issue, as you can use the full path to the interpreter and that's fine - I learned my dislike of this technique from Visual Studio, where it's not that simple... I'm not sure renaming py.exe as python.exe is such a good idea either. I don't have a strong reason to give, just a suspicion that the subtly different arguments, and the fact that it (by design) doesn't respect PATH, make me concerned that it'll cause even more subtle issues than the current approach. Also, it's not possible for users with their own approaches to "opt out", where a PATH setting can be altered by the user. Anyway, as I say, I'll probably stick to doing things my own way whatever happens. Paul From zachary.ware+pyideas at gmail.com Tue Mar 15 17:14:40 2016 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Tue, 15 Mar 2016 16:14:40 -0500 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56E87763.80000@python.org> References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> <56E87763.80000@python.org> Message-ID: On Tue, Mar 15, 2016 at 3:58 PM, Steve Dower wrote: > On 15Mar2016 1231, Random832 wrote: >> >> I think what might be most appropriate would be to have a "Python >> Console" in the start menu (alongside the IDLE entry) which sources a >> bat file to add the correct python version (and the scripts directory >> for pip and easy_install) to that window's PATH, rather than installing >> it in the global PATH, and users can learn to start the console from >> that icon when they want to run python commands. Inspired by the fact >> that e.g. Visual Studio has something that does the same. > > > This is one of two solutions I'd be comfortable with. > > The other is to rename "py[w].exe" as "python[w].exe". I've wondered about allowing the launcher to determine the Python it should invoke by its own name. That is, if the launcher is named 'python3.exe' it should behave as 'py -3'; 'python2.7.exe' should behave as 'py -2.7'. I wrote a patch to that effect a year or two ago but never got it polished up enough to submit. What about doing this, and at launcher install time (or at 'repair installation' time) create links for each found Python (and python2/python3 if they exist)? -- Zach From steve.dower at python.org Tue Mar 15 17:36:26 2016 From: steve.dower at python.org (Steve Dower) Date: Tue, 15 Mar 2016 14:36:26 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> <56E87763.80000@python.org> Message-ID: <56E8805A.7020208@python.org> On 15Mar2016 1414, Zachary Ware wrote: > On Tue, Mar 15, 2016 at 3:58 PM, Steve Dower wrote: >> On 15Mar2016 1231, Random832 wrote: >>> >>> I think what might be most appropriate would be to have a "Python >>> Console" in the start menu (alongside the IDLE entry) which sources a >>> bat file to add the correct python version (and the scripts directory >>> for pip and easy_install) to that window's PATH, rather than installing >>> it in the global PATH, and users can learn to start the console from >>> that icon when they want to run python commands. Inspired by the fact >>> that e.g. Visual Studio has something that does the same. >> >> >> This is one of two solutions I'd be comfortable with. >> >> The other is to rename "py[w].exe" as "python[w].exe". > > I've wondered about allowing the launcher to determine the Python it > should invoke by its own name. That is, if the launcher is named > 'python3.exe' it should behave as 'py -3'; 'python2.7.exe' should > behave as 'py -2.7'. I wrote a patch to that effect a year or two ago > but never got it polished up enough to submit. What about doing this, > and at launcher install time (or at 'repair installation' time) create > links for each found Python (and python2/python3 if they exist)? > Very tough to automatically install for all found Pythons like this through an MSI, but if the not-found exit code is 9009 then there's no reason we can't just install copies for as many versions as we like. What would also be nice is "activatepyX.Y" scripts (presumably in both .cmd and .ps1 formats) that would modify the current PATH. I started a PEP a while back proposing these but never finished. If we start adding this many files I'd also want to move the launchers into their own directory (probably "C:\Program Files\Common Files\Python\Launcher") and add that to the global PATH. As long as we strictly control what files are in there, I have no problem with adding it - we just don't want any random DLL or EXE getting in there. Cheers, Steve From chris.barker at noaa.gov Tue Mar 15 21:59:51 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 15 Mar 2016 18:59:51 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56E83D59.5080707@gmail.com> References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> Message-ID: <3593462996689522105@unknownmsgid> > Failing to run Python in the terminal occurs every single introduction > course I give. Every single one. And without help, a beginner will have > trouble solving Absolutely! I have the same experience in my classes. The reality is that MS has not updated the GUI for setting PATH since Windows 3.1 as far as I can tell. It sucks. So we really, really, want this to Just Work for newbies. Yes, things get complicated with multiple versions of Python, etc, but that is going to require understanding s fair bit about PATH et al. No matter how you slice it. Let's be sure the simple case is easy and obvious. -CHB > it because it's hard to search in Google for something > you know nothing about. So some of them, the ones that can understand > English and see the value of looking it up for hours, may solve it. The > other ones will give up. > > Another problem is the gigantic time and energy wasted for this : I have > to tell about this box in every course, write it down (with screenshot) > in every tutorials, mention it in every videos I do. And yet, yet! > People still come to me, write comments, emails, not understanding why > typing "python" in this big black windows doesn't work. > > And then, you have the worst case scenario. The one where the person, a > bit tech saavy, tried to edit the PATH manually in this tiny, > unresizeable window, and messed it up. Now, try to diagnose that over > skype, or on a forum, or on twitter... > > I already made the choice to spends hours of my life helping people with > Python, but I wish I could spend it on explaining how they can make > their computer do nice stuff, not on making sure they tick boxes. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From jcgoble3 at gmail.com Tue Mar 15 22:10:10 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Tue, 15 Mar 2016 22:10:10 -0400 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <3593462996689522105@unknownmsgid> References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> Message-ID: On Tue, Mar 15, 2016 at 9:59 PM, Chris Barker - NOAA Federal wrote: > The reality is that MS has not updated the GUI for setting PATH since > Windows 3.1 as far as I can tell. It sucks. Actually, Windows 10 (or at least Windows 10 with the latest updates) introduced a much, MUCH nicer interface for PATH editing. Here's a screenshot: http://snag.gy/WH4Oj.jpg With this interface, one can edit individual entries (an edit to the Python 2.7 \Scripts folder in the system PATH is in progress in this screenshot), use the browse button to select new folders via GUI, tweak the order of the list, easily remove entries, and so forth. The details of the plaintext version of PATH that one would use at the command prompt are hidden behind the "edit text" button. I recently performed a massive cleanup to my PATHs after discovered this new interface, and it was really easy. From desmoulinmichel at gmail.com Wed Mar 16 09:59:56 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 16 Mar 2016 14:59:56 +0100 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> Message-ID: <56E966DC.1050808@gmail.com> Le 16/03/2016 03:10, Jonathan Goble a ?crit : > On Tue, Mar 15, 2016 at 9:59 PM, Chris Barker - NOAA Federal > wrote: >> The reality is that MS has not updated the GUI for setting PATH since >> Windows 3.1 as far as I can tell. It sucks. > > Actually, Windows 10 (or at least Windows 10 with the latest updates) > introduced a much, MUCH nicer interface for PATH editing. > > Here's a screenshot: http://snag.gy/WH4Oj.jpg > > With this interface, one can edit individual entries (an edit to the > Python 2.7 \Scripts folder in the system PATH is in progress in this > screenshot), use the browse button to select new folders via GUI, > tweak the order of the list, easily remove entries, and so forth. The > details of the plaintext version of PATH that one would use at the > command prompt are hidden behind the "edit text" button. I recently > performed a massive cleanup to my PATHs after discovered this new > interface, and it was really easy. > I noticed that last month, and it's a good thing, but universities and companies are not going to upgrade for years (if they do it at all). Most of my students, when using windows, are using Windows 7. A few is running Vista or 8. Sometime, I meet some XP (yeah, I know). I only met one using Windows 10, and it was his own laptop. Plus fixing a way to fix a problem is, while improving the situation, probably not the best way to look at the situation. From abarnert at yahoo.com Wed Mar 16 11:29:39 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 16 Mar 2016 08:29:39 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56E87763.80000@python.org> References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> <56E87763.80000@python.org> Message-ID: On Mar 15, 2016, at 13:58, Steve Dower wrote: > >> On 15Mar2016 1231, Random832 wrote: >> I think what might be most appropriate would be to have a "Python >> Console" in the start menu (alongside the IDLE entry) which sources a >> bat file to add the correct python version (and the scripts directory >> for pip and easy_install) to that window's PATH, rather than installing >> it in the global PATH, and users can learn to start the console from >> that icon when they want to run python commands. Inspired by the fact >> that e.g. Visual Studio has something that does the same. > > This is one of two solutions I'd be comfortable with. > > The other is to rename "py[w].exe" as "python[w].exe". > > PATH is used to resolve system DLLs and common commands. Over time, better security features are being added for core system components, but it is still possible to hijack programs this way. But isn't the whole point here that we want novices to be able to treat python and/or python3 as a "common command", just like py? That implies either putting the executable in an existing PATH directory, or adding the executable's directory to the PATH (or, of course, renaming py, as you suggest, since that already gets put into an existing directory). > It also has a serious conflict problem once the second version of Python is installed with this option enabled, in that *anything* currently referring to just "python" will change. If this is only intended for rank beginners, what about just disabling the checkbox, with a warning, if there's already a program named "Python" anywhere on the PATH? Then, when someone gets into trouble because they're installing two Pythons, they'll get a warning and be able to ask their teacher or search StackOverflow or whatever and get a quick answer. And that means we don't have to solve the conflict problems, the uninstall leave-behind problems, etc. for multiple pythons. From chris.barker at noaa.gov Wed Mar 16 13:17:59 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 16 Mar 2016 10:17:59 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56E966DC.1050808@gmail.com> References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> <56E966DC.1050808@gmail.com> Message-ID: > > > Actually, Windows 10 (or at least Windows 10 with the latest updates) > > introduced a much, MUCH nicer interface for PATH editing. > > > > Here's a screenshot: http://snag.gy/WH4Oj.jpg > Finally! and after only 20 years. Great to see that. But we will be dealing with pre Windows10 for a a good long time :-) Plus fixing a way to fix a problem is, while improving the situation, > probably not the best way to look at the situation. > well, it does change teh burden teh choice is between auto-setting PATH, and requiring people to set/adjust it themselves -- each has it's advantages. And how easy it is to adjsut it yourself does effect the trade-off. But as we will be supporting Windows pre-10 for a long time, we still need it to "just work" for less sophisticated users without them having to mess with PATH. As in: If someone installs a version of python with defaults, they should then get that version when they type "python" (or maybe "py") at the command line. If they want something other than the last one they installed, then they'll have to mess with PATH or other configuration... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Mar 16 13:22:42 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 16 Mar 2016 10:22:42 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> <56E87763.80000@python.org> Message-ID: On Wed, Mar 16, 2016 at 8:29 AM, Andrew Barnert via Python-ideas < python-ideas at python.org> wrote: > If this is only intended for rank beginners, what about just disabling the > checkbox, with a warning, if there's already a program named "Python" > anywhere on the PATH? Then, when someone gets into trouble because they're > installing two Pythons, they'll get a warning and be able to ask their > teacher or search StackOverflow > please no -- it should be easy and obvious to "upgrade" python -- i.e. the last one you installed is what "just works". If the default does that, then a non-defautl option to not override the existing one would be nice: """ Warning: there is an existing python installed on this system: check here if you want to continue to have the old version be used as the default. Leve the box unchecked if you want the version to be the new default. """ or something like that. After all, what do people expect when they have MSWord version X, and then install MSWord version X+1 ? They expect that when they run Word after that, they'll get the new version. Is this going to surprise anyone??? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Mar 16 13:32:06 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 16 Mar 2016 17:32:06 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> <56E966DC.1050808@gmail.com> Message-ID: On 16 March 2016 at 17:17, Chris Barker wrote: > As in: > > If someone installs a version of python with defaults, they should then get > that version when they type "python" (or maybe "py") at the command line. > > If they want something other than the last one they installed, then they'll > have to mess with PATH or other configuration... In theory, this makes sense to me. What it needs is someone to champion it. And address the inevitable questions/objections. The ones I immediately thought of were: 1. What if I want to install a version *not* for day to day use but just for testing (i.e., there needs to be an easily accessible "leave PATH alone" option). 2. How does this interact with uninstalls and upgrades? If I install 3.7, then 3.8 then uninstall 3.7, then install 3.6, then upgrade 3.8 to 3.8.1, what's left on PATH? (The answer I *want* is 3.8.1. That may well be a nightmare to implement). How does the above change if somewhere in the middle of that process I manually change PATH (in which case I may well *not* want 3.8.1 as the end result!) It's easy enough to dismiss point (2) as "well, you know what you're doing", but I have experience of people doing a series of installs like this, getting confused with PATH and doing a google search to find out how to "fix" it. So it is a genuine situation a non-expert user could find themselves in. > After all, what do people expect when they have MSWord version X, and then install MSWord version X+1 But nobody *ever* runs multiple versions of MS Word. Whereas there are genuine reasons for people (and not just experts, although admittedly not complete beginners) to run multiple versions of Python. The MS Word argument implies that the Python installer should automatically and silently uninstall any older versions of Python present on the user's machine. Paul From breamoreboy at yahoo.co.uk Wed Mar 16 13:41:01 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 16 Mar 2016 17:41:01 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> <56E966DC.1050808@gmail.com> Message-ID: On 16/03/2016 17:17, Chris Barker wrote: > > Actually, Windows 10 (or at least Windows 10 with the latest updates) > > introduced a much, MUCH nicer interface for PATH editing. > > > > Here's a screenshot: http://snag.gy/WH4Oj.jpg > > Finally! and after only 20 years. Great to see that. But we will be > dealing with pre Windows10 for a a good long time :-) > Then grab a third party tool that does the same job. I use the Rapid Environment Editor http://www.rapidee.com/en/about but I understand that there are others. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From steve.dower at python.org Wed Mar 16 13:46:16 2016 From: steve.dower at python.org (Steve Dower) Date: Wed, 16 Mar 2016 10:46:16 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> <56E87763.80 000@python.org> Message-ID: The uninstall problem is actually that "python" would stop working completely in that case, and it's not necessarily obvious why or how to fix it. The leave-behind problem doesn't really matter as the directory is not searched if it doesn't exist, but we also don't leave behind entries added by the installer. A user who customizes their own PATH will obviously need to manually remove the entry, but that's a manual action both times so I don't see any concern there. At one point I was trying to convince the Windows team to add a global "python" command to all Windows installs (probably 10 and later - we don't backport stuff like this, but it could be a manual install) that would act like the py.exe launcher but also automatically download Python if you didn't have it. What would people think if I got that to happen? Top-posted from my Windows Phone -----Original Message----- From: "Andrew Barnert" Sent: ?3/?16/?2016 8:30 To: "Steve Dower" Cc: "python-ideas at python.org" Subject: Re: [Python-ideas] Make py.exe default to Python 3 On Mar 15, 2016, at 13:58, Steve Dower wrote: > >> On 15Mar2016 1231, Random832 wrote: >> I think what might be most appropriate would be to have a "Python >> Console" in the start menu (alongside the IDLE entry) which sources a >> bat file to add the correct python version (and the scripts directory >> for pip and easy_install) to that window's PATH, rather than installing >> it in the global PATH, and users can learn to start the console from >> that icon when they want to run python commands. Inspired by the fact >> that e.g. Visual Studio has something that does the same. > > This is one of two solutions I'd be comfortable with. > > The other is to rename "py[w].exe" as "python[w].exe". > > PATH is used to resolve system DLLs and common commands. Over time, better security features are being added for core system components, but it is still possible to hijack programs this way. But isn't the whole point here that we want novices to be able to treat python and/or python3 as a "common command", just like py? That implies either putting the executable in an existing PATH directory, or adding the executable's directory to the PATH (or, of course, renaming py, as you suggest, since that already gets put into an existing directory). > It also has a serious conflict problem once the second version of Python is installed with this option enabled, in that *anything* currently referring to just "python" will change. If this is only intended for rank beginners, what about just disabling the checkbox, with a warning, if there's already a program named "Python" anywhere on the PATH? Then, when someone gets into trouble because they're installing two Pythons, they'll get a warning and be able to ask their teacher or search StackOverflow or whatever and get a quick answer. And that means we don't have to solve the conflict problems, the uninstall leave-behind problems, etc. for multiple pythons. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Mar 16 14:06:50 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 16 Mar 2016 11:06:50 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> <56E87763.80 000@python.org> Message-ID: <56E9A0BA.5080707@stoneleaf.us> On 03/16/2016 10:46 AM, Steve Dower wrote: > At one point I was trying to convince the Windows team to add a global > "python" command to all Windows installs (probably 10 and later - we > don't backport stuff like this, but it could be a manual install) that > would act like the py.exe launcher but also automatically download > Python if you didn't have it. What would people think if I got that to > happen? That would be awesome. Download the latest 2.7 and the latest 3.x (installed in that order), and install py.exe. -- ~Ethan~ From oscar.j.benjamin at gmail.com Wed Mar 16 14:15:42 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 16 Mar 2016 18:15:42 +0000 Subject: [Python-ideas] Adding python to Windows. Was :Re: Make py.exe default to Python 3 Message-ID: On 16 Mar 2016 17:47, "Steve Dower" wrote: > > The uninstall problem is actually that "python" would stop working completely in that case, and it's not necessarily obvious why or how to fix it. > > The leave-behind problem doesn't really matter as the directory is not searched if it doesn't exist, but we also don't leave behind entries added by the installer. A user who customizes their own PATH will obviously need to manually remove the entry, but that's a manual action both times so I don't see any concern there. > > At one point I was trying to convince the Windows team to add a global "python" command to all Windows installs (probably 10 and later - we don't backport stuff like this, but it could be a manual install) that would act like the py.exe launcher but also automatically download Python if you didn't have it. What would people think if I got that to happen? That would be amazing! Then I guess all the major OSes would ship with Python making simple Python scripts portable and solving a lot of the problems that pyinstaller/py2exe try to solve. Can you expand on how that might work? The detail affects which problems it would solve. Would it be like the system Python on OSX? -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Wed Mar 16 14:44:07 2016 From: steve.dower at python.org (Steve Dower) Date: Wed, 16 Mar 2016 11:44:07 -0700 Subject: [Python-ideas] Adding python to Windows. Was :Re: Make py.exe default to Python 3 In-Reply-To: References: Message-ID: <56E9A977.4010303@python.org> On 16Mar2016 1115, Oscar Benjamin wrote: > That would be amazing! Then I guess all the major OSes would ship with > Python making simple Python scripts portable and solving a lot of the > problems that pyinstaller/py2exe try to solve. > > Can you expand on how that might work? The detail affects which problems > it would solve. Would it be like the system Python on OSX? So my main concern about getting Python shipping with Windows has always been long-term compatibility. Basically, we're in the same category as LTS RHEL: whatever ships now *will still be 100% compatible with what we are shipping in ten years time* (yes, that's an aspiration more than a guarantee, but it certainly rules out 3.n->3.(n+1) upgrades). As a result, if Python did ship in Windows, I'd be trying really hard to make it an internal implementation detail to discourage people from ever relying on it. As Nick Coghlan said when I last chatted with him about this, "please, please, learn from our mistakes" :) So the idea that I was working on was effectively the py.exe launcher, but taking over the global "python" name (and probably "python3"/"python2"), and with the addition of useful error messages (i.e. with URLs or extra tools) for when you try and run "python" without having installed it. And also add proper support for configuring defaults and active virtual environments (which are both in the current py.exe launcher, but seem to not be well known). Shebang support would stick around and be encouraged, along with specifying Python language versions such that scripts will run with the correct version on new machines. The install step may be painful the first time, but since Python 3.5+ does not require admin privileges, it should be quick and easy for most people. I don't see us going out of our way to update legacy versions to do this - they'll continue to need admin rights. This would obviously be a big and unexpected change for users, but it seems to be the most viable way of making Python universally available without locking into a specific version for 10+ years or essentially forking a "Microsoft Python" that ends up being very different from CPython releases. And the value only comes from this feature being available by default on all machines, so it becomes a question of "does this add enough value to be worth the pain?" For education purposes, it certainly does. My gut feel is that when this sort of change comes with an operating system upgrade (not just an update), it'll be okay, and if it's opt-in for earlier versions then people won't be too surprised. But, alas, I've been finding myself with not enough support around the company to get it through, and not enough time to do all the politicking and engineering work myself. I started a launcher rewrite at https://github.com/zooba/PyLauncher if anyone is interested in looking (it's very C++, as opposed to C). Can't promise that it'd remain open-source or accept contributions, but I'd certainly try. Cheers, Steve From p.f.moore at gmail.com Wed Mar 16 15:54:17 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 16 Mar 2016 19:54:17 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <56E9A0BA.5080707@stoneleaf.us> References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> <56E9A0BA.5080707@stoneleaf.us> Message-ID: On 16 March 2016 at 18:06, Ethan Furman wrote: > On 03/16/2016 10:46 AM, Steve Dower wrote: > >> At one point I was trying to convince the Windows team to add a global >> "python" command to all Windows installs (probably 10 and later - we >> don't backport stuff like this, but it could be a manual install) that >> would act like the py.exe launcher but also automatically download >> Python if you didn't have it. What would people think if I got that to >> happen? > > > That would be awesome. > > Download the latest 2.7 and the latest 3.x (installed in that order), and > install py.exe. Agreed it would be awesome. I see no reason to bother with 2.x, though. If the user knows enough to know they want 2.x they can download and install it. Otherwise just give people the latest and greatest Python. Paul From ethan at stoneleaf.us Wed Mar 16 15:57:50 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 16 Mar 2016 12:57:50 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> <56E9A0BA.5080707@stoneleaf.us> Message-ID: <56E9BABE.3020303@stoneleaf.us> On 03/16/2016 12:54 PM, Paul Moore wrote: > On 16 March 2016 at 18:06, Ethan Furman wrote: >> Download the latest 2.7 and the latest 3.x (installed in that order), and >> install py.exe. > > Agreed it would be awesome. I see no reason to bother with 2.x, > though. If the user knows enough to know they want 2.x they can > download and install it. Otherwise just give people the latest and > greatest Python. The point is they may not know they need/want 2.x, but need it anyway. 2.7 is not dead yet. -- ~Ethan~ From abarnert at yahoo.com Wed Mar 16 18:07:28 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 16 Mar 2016 15:07:28 -0700 Subject: [Python-ideas] Adding python to Windows. Was :Re: Make py.exe default to Python 3 In-Reply-To: References: Message-ID: On Mar 16, 2016, at 11:15, Oscar Benjamin wrote: > On 16 Mar 2016 17:47, "Steve Dower" wrote: > > > > The uninstall problem is actually that "python" would stop working completely in that case, and it's not necessarily obvious why or how to fix it. > > > > The leave-behind problem doesn't really matter as the directory is not searched if it doesn't exist, but we also don't leave behind entries added by the installer. A user who customizes their own PATH will obviously need to manually remove the entry, but that's a manual action both times so I don't see any concern there. > > > > At one point I was trying to convince the Windows team to add a global "python" command to all Windows installs (probably 10 and later - we don't backport stuff like this, but it could be a manual install) that would act like the py.exe launcher but also automatically download Python if you didn't have it. What would people think if I got that to happen? > > That would be amazing! > I love Steve's idea, but I think you're overselling it. > Then I guess all the major OSes would ship with Python making simple Python scripts portable > Only if they're dual-version (2.7/3.x), and don't depend on anything from PyPI, and can ignore Windows 8 and earlier. That rules out novice education, many utility scripts that would be twice as complicated if they have to be dual-version or can't use requests or whatever, etc. > and solving a lot of the problems that pyinstaller/py2exe try to solve. > I don't think it will. Even ignoring 3.x, py2app isn't any simpler since Apple started shipping 2.7 preinstalled. In fact, if anything, it's gotten more complicated. The hard part of py2exe isn't wrapping Python.exe and python36.zip up together with your site-packages.zip, it's working out what has to go in that site-packages.zip, and what extra DLLs are needed because they're linked by a .pyd or explicitly opened via ctypes/cffi, and so on. So, even if you could rely on python and its stdlib already being present, py2exe would still have to do all the hard stuff. But you really can't even rely on that. For example, you may need a specific version (obviously 3.6+ is good enough for 2016, but will that still be true by the time "requires Windows 10" is a reasonable thing for apps to specify?); you'll almost certainly explicitly require either 32-bit or 64-bit; etc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Wed Mar 16 18:13:01 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 16 Mar 2016 22:13:01 +0000 Subject: [Python-ideas] Adding python to Windows. Was :Re: Make py.exe default to Python 3 In-Reply-To: <56E9A977.4010303@python.org> References: <56E9A977.4010303@python.org> Message-ID: On 16 Mar 2016 18:44, "Steve Dower" wrote: > > On 16Mar2016 1115, Oscar Benjamin wrote: >> >> That would be amazing! Then I guess all the major OSes would ship with >> Python making simple Python scripts portable and solving a lot of the >> problems that pyinstaller/py2exe try to solve. >> >> Can you expand on how that might work? The detail affects which problems >> it would solve. Would it be like the system Python on OSX? > > > So my main concern about getting Python shipping with Windows has always been long-term compatibility. Basically, we're in the same category as LTS RHEL: whatever ships now *will still be 100% compatible with what we are shipping in ten years time* (yes, that's an aspiration more than a guarantee, but it certainly rules out 3.n->3.(n+1) upgrades). > > As a result, if Python did ship in Windows, I'd be trying really hard to make it an internal implementation detail to discourage people from ever relying on it. As Nick Coghlan said when I last chatted with him about this, "please, please, learn from our mistakes" :) I can understand that. The OSX system Python is like that. You're not supposed to mess with it and when you want to do some programming every guide suggests to install a newer non-system Python. I often see reports of people getting into trouble after removing the system Python because they wanted a newer version for some reason. However I do really like the fact that there is *some* Python installed and Windows is the only OS where this is not the case. As long as some Python is installed I can bundle a bunch of Python code in a zip file and I've got a portable program. On Windows it's only portable if you ask someone to install Python first which is often a deal-breaker (hence py2exe). > So the idea that I was working on was effectively the py.exe launcher, but taking over the global "python" name (and probably "python3"/"python2"), and with the addition of useful error messages (i.e. with URLs or extra tools) for when you try and run "python" without having installed it. And also add proper support for configuring defaults and active virtual environments (which are both in the current py.exe launcher, but seem to not be well known). Shebang support would stick around and be encouraged, along with specifying Python language versions such that scripts will run with the correct version on new machines. > > The install step may be painful the first time, but since Python 3.5+ does not require admin privileges, it should be quick and easy for most people. I don't see us going out of our way to update legacy versions to do this - they'll continue to need admin privileges So this Python.exe is a launcher that is bundled with Windows which solves your concerns about PATH manipulation and then when Python is actually installed it hooks into this launcher? > This would obviously be a big and unexpected change for users, but it seems to be the most viable way of making Python universally available without locking into a specific version for 10+ years or essentially forking a "Microsoft Python" that ends up being very different from CPython releases. > > And the value only comes from this feature being available by default on all machines, so it becomes a question of "does this add enough value to be worth the pain?" For education purposes, it certainly does. My gut feel is that when this sort of change comes with an operating system upgrade (not just an update), it'll be okay, and if it's opt-in for earlier versions then people won't be too surprised. > > But, alas, I've been finding myself with not enough support around the company to get it through, and not enough time to do all the politicking and engineering work myself. I started a launcher rewrite at https://github.com/zooba/PyLauncher if anyone is interested in looking (it's very C++, as opposed to C). Can't promise that it'd remain open-source or accept contributions, but I'd certainly try. To what extent can the same benefits be achieved by bundling the same launcher in the Python installer? Is it just that it would require elevated privileges? I'm not sure that that would be a significant problem for e.g. my students. They'd just have to type in their passwords (on their own laptops) right? -- Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 16 18:17:06 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 16 Mar 2016 15:17:06 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> <56E87763.80000@python.org> Message-ID: <01EE281B-2245-4DEF-AF57-C45324D88909@yahoo.com> On Mar 16, 2016, at 10:22, Chris Barker wrote: > >> On Wed, Mar 16, 2016 at 8:29 AM, Andrew Barnert via Python-ideas wrote: >> If this is only intended for rank beginners, what about just disabling the checkbox, with a warning, if there's already a program named "Python" anywhere on the PATH? Then, when someone gets into trouble because they're installing two Pythons, they'll get a warning and be able to ask their teacher or search StackOverflow > > please no -- it should be easy and obvious to "upgrade" python -- i.e. the last one you installed is what "just works". > > If the default does that, then a non-defautl option to not override the existing one would be nice: > > """ > Warning: there is an existing python installed on this system: check here if you want to continue to have the old version be used as the default. Leve the box unchecked if you want the version to be the new default. > """ > > or something like that. > > After all, what do people expect when they have MSWord version X, and then install MSWord version X+1 ? They expect that when they run Word after that, they'll get the new version. Is this going to surprise anyone??? Last time I did that, I ended up with Office 2007 in a completely separate directory from Office 2003, with their own Start Menu folders. The one under "Office" was still 2003, but there was a new one under "Microsoft Office" that I had to find. And Win-F "winword" still ran 2003. And .doc files were still owned by 2003 (but docx by 2007). And all of my system templates and most of my addons from 2003 weren't available in 2007 until I manually reinstalled them for it. And so on. I don't think anyone expects that with Office. There are other kinds of apps where people _do_ expect it. For example, if you have TuneUp 2.4.3 and then install TuneUp 2.5.7, when you run a TuneUp (whether via the Start menu, or Win-F, or because you had the "auto-start with iTunes checkbox on, etc.), you'll definitely get 2.5.7. But that's because 2.4.3 no longer exists on your system. I don't think we want a Python 3.6 install to automatically uninstall or overwrite 2.7, or vice-versa. Which means it's like Office, not like TuneUp. Installing two Python versions on the same machine is always going to require some kind of management by the user. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rantingrickjohnson at gmail.com Wed Mar 16 18:50:53 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Wed, 16 Mar 2016 15:50:53 -0700 (PDT) Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy Message-ID: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> Has anyone else found this to be too syntactically noisy? from module import Foo as _Foo, bar as _bar That is horrifically noisy IMO. The problem is, how do we remove the noise without sacrificing intuitiveness? My first idea was to do this: from module import_private Foo, bar And while it's self explanatory, it's also too long. So i thought... from module _import Foo, bar I'm leaning more towards the latter, but i'm not loving it either. Any ideas? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Wed Mar 16 19:38:22 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 17 Mar 2016 10:38:22 +1100 Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> Message-ID: <85bn6ej8nl.fsf@benfinney.id.au> Rick Johnson writes: > Has anyone else found this to be too syntactically noisy? > > from module import Foo as _Foo, bar as _bar > > That is horrifically noisy IMO. Agreed. What is wrong with:: import lorem as _lorem do_something_with(_lorem.Foo) -- \ ?Nothing is so common as to imitate one's enemies, and to use | `\ their weapons.? ?Voltaire, _Dictionnaire Philosophique_ | _o__) | Ben Finney From steve at pearwood.info Wed Mar 16 20:35:45 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 Mar 2016 11:35:45 +1100 Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> Message-ID: <20160317003545.GD8022@ando.pearwood.info> On Wed, Mar 16, 2016 at 03:50:53PM -0700, Rick Johnson wrote: > from module _import Foo, bar That's anything but self-explanatory, since it gives no hint that Foo and bar will be imported under some name other than Foo and bar. Besides, why is "import x as _x" so special to require special syntax? Out of the infinite number of names that x could be imported as (import x as y etc) what's so special about _x that it deserves a short-cut? -- Steve From rantingrickjohnson at gmail.com Wed Mar 16 21:13:09 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Wed, 16 Mar 2016 18:13:09 -0700 (PDT) Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: <85bn6ej8nl.fsf@benfinney.id.au> References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <85bn6ej8nl.fsf@benfinney.id.au> Message-ID: <3b15d825-9499-4524-9208-d41d53923592@googlegroups.com> On Wednesday, March 16, 2016 at 6:39:01 PM UTC-5, Ben Finney wrote: > Agreed. What is wrong with: > > import lorem as _lorem > do_something_with(_lorem.Foo) Yes, in the case when multiple symbols are imported+privatized, that's an improvement, and hiding multiple symbols behind a single symbol *CAN* help to avoid name clashes as well (less names, less chances of clashing) but, in essence, all you've done is to move "syntactic noise" from a single location, to a minimum of one other location, and, potentially, many diverse locations. So yes, useful technique, but not one that fulfills my current "selfish desires" -- I need something sweet baby! O:-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 16 21:31:26 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 16 Mar 2016 18:31:26 -0700 Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: <3b15d825-9499-4524-9208-d41d53923592@googlegroups.com> References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <85bn6ej8nl.fsf@benfinney.id.au> <3b15d825-9499-4524-9208-d41d53923592@googlegroups.com> Message-ID: On Mar 16, 2016, at 18:13, Rick Johnson wrote: > > On Wednesday, March 16, 2016 at 6:39:01 PM UTC-5, Ben Finney wrote: > > Agreed. What is wrong with: > > > > import lorem as _lorem > > do_something_with(_lorem.Foo) > > Yes, in the case when multiple symbols are > imported+privatized, that's an improvement, and hiding > multiple symbols behind a single symbol *CAN* help to avoid > name clashes as well (less names, less chances of clashing) > but, in essence, all you've done is to move "syntactic > noise" from a single location, to a minimum of one other > location, and, potentially, many diverse locations. So yes, > useful technique, but not one that fulfills my current > "selfish desires" -- I need something sweet baby! O:-) This seems like something that shouldn't be done in general, so doesn't need a language fix--but if you for some specific reason need to do it all the time, just write a function for it: def _postimport(mod, names): g = sys._getframe(1).f_globals for name in names.split(): g['_'+name] = mod[name] So: import thingy _postimport(thingy, 'spam eggs cheese') use(_spam, _cheese) Of course you can wrap up the import and _postimport in a single function: _magic_from_import('thingy', 'spam eggs cheese') Or use MacroPy to make the syntax nicer. Or, if even that's not good enough, write a simple token-processing import hook that finds "_import" tokens and converts them to calls to your wrapper function. Of course the farther you go down this path, the less readable your code becomes to someone who doesn't know about your function/macro/hook, but if there's really so much boilerplate that it's getting in the way, the tradeoff might be worth it. From rantingrickjohnson at gmail.com Wed Mar 16 21:52:13 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Wed, 16 Mar 2016 18:52:13 -0700 (PDT) Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: <20160317003545.GD8022@ando.pearwood.info> References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <20160317003545.GD8022@ando.pearwood.info> Message-ID: <007b7330-9b57-42a6-b1df-5af2029db1fb@googlegroups.com> On Wednesday, March 16, 2016 at 7:43:05 PM UTC-5, Steven D'Aprano wrote: > On Wed, Mar 16, 2016 at 03:50:53PM -0700, Rick Johnson wrote: > > > from module _import Foo, bar > > That's anything but self-explanatory, since it gives no > hint that Foo and bar will be imported under some name > other than Foo and bar. Yes, I'll agree it would be far too esoteric for a language that promotes readability. If i end up writing my own hack though, i'll probably go this way, simply because it's the most compact solution i think of at the moment. > Besides, why is "import x as _x" so special to require special syntax? Well, for a single symbol, it's not bad. But consider something like: from module import Foo as _Foo, bar as _bar, BAZ as _BAZ, spam as _spam, eggs as _eggs Now, that may seem like a contrived example, but i've witnessed much longer "run-on import lines" than that. 79 chars will be eaten-up really quickly. But what eats up most of the space, is the redundant " as _...". When importing "piecemeal", it's impossible to avoid writing the actual symbol names , but, if we can avoid the redundant repetition of " as _...", it would be an improvement. > Out of the infinite number of names that x could be > imported as (import x as y etc) what's so special about _x > that it deserves a short-cut? Not exactly sure what your saying here, but, i'll fancy a guess... If you're suggesting that my syntax would be limited to merely "adding a single leading underscore to the public symbol (or symbols) as they are imported", then yes, you are correct! The intended purpose is to: "automate the privatization of public symbols during the import process". Of course, if they already have a leading underscore, they will remain unchanged. But i wonder what should be done when they have more than one underscore? Hmm... >From my POV, I don't see anything wrong with "single purpose utilities". In fact, i find them to be more intuitive. The whole: "Do one thing, and do it well" philosophy... -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Mar 16 21:53:35 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 17 Mar 2016 12:53:35 +1100 Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <85bn6ej8nl.fsf@benfinney.id.au> <3b15d825-9499-4524-9208-d41d53923592@googlegroups.com> Message-ID: On Thu, Mar 17, 2016 at 12:31 PM, Andrew Barnert via Python-ideas wrote: > This seems like something that shouldn't be done in general, so doesn't need a language fix--but if you for some specific reason need to do it all the time, just write a function for it: > > def _postimport(mod, names): > g = sys._getframe(1).f_globals > for name in names.split(): > g['_'+name] = mod[name] > > So: > > import thingy > _postimport(thingy, 'spam eggs cheese') > use(_spam, _cheese) > > Of course you can wrap up the import and _postimport in a single function: > > _magic_from_import('thingy', 'spam eggs cheese') > > Or use MacroPy to make the syntax nicer. > > Or, if even that's not good enough, write a simple token-processing import hook that finds "_import" tokens and converts them to calls to your wrapper function. > > Of course the farther you go down this path, the less readable your code becomes to someone who doesn't know about your function/macro/hook, but if there's really so much boilerplate that it's getting in the way, the tradeoff might be worth it. > ... and then you decide, hey, let's just define __all__ and not worry about adorning all those names with underscores, because the definition of public vs private can be independent of the symbols themselves. ChrisA From rantingrickjohnson at gmail.com Wed Mar 16 21:58:55 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Wed, 16 Mar 2016 18:58:55 -0700 (PDT) Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <85bn6ej8nl.fsf@benfinney.id.au> <3b15d825-9499-4524-9208-d41d53923592@googlegroups.com> Message-ID: <91345ed8-13ca-4f0e-96b2-703d76475c03@googlegroups.com> On Wednesday, March 16, 2016 at 8:54:06 PM UTC-5, Chris Angelico wrote: > ... and then you decide, hey, let's just define __all__ and not worry > about adorning all those names with underscores, because the > definition of public vs private can be independent of the symbols > themselves. > I avoid __all__ like the plague. Too easy for it to get out of sync with the API when i forget to add a new symbol. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Wed Mar 16 22:05:21 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 17 Mar 2016 13:05:21 +1100 Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <85bn6ej8nl.fsf@benfinney.id.au> <3b15d825-9499-4524-9208-d41d53923592@googlegroups.com> Message-ID: <857fh1kgf2.fsf@benfinney.id.au> Rick Johnson writes: > On Wednesday, March 16, 2016 at 6:39:01 PM UTC-5, Ben Finney wrote: > > Agreed. What is wrong with: > > > > import lorem as _lorem > > do_something_with(_lorem.Foo) > > Yes, in the case when multiple symbols are imported+privatized, that's > an improvement It's got the strong advantage of already being an established idiom. > but not one that fulfills my current "selfish desires" -- I need > something sweet baby! O:-) You're in the wrong forum, then. This is for discussing improvements to Python. -- \ ?We suffer primarily not from our vices or our weaknesses, but | `\ from our illusions.? ?Daniel J. Boorstin, historian, 1914?2004 | _o__) | Ben Finney From steve at pearwood.info Wed Mar 16 22:10:13 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 17 Mar 2016 13:10:13 +1100 Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: <007b7330-9b57-42a6-b1df-5af2029db1fb@googlegroups.com> References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <20160317003545.GD8022@ando.pearwood.info> <007b7330-9b57-42a6-b1df-5af2029db1fb@googlegroups.com> Message-ID: <20160317021012.GE8022@ando.pearwood.info> On Wed, Mar 16, 2016 at 06:52:13PM -0700, Rick Johnson wrote: > On Wednesday, March 16, 2016 at 7:43:05 PM UTC-5, Steven D'Aprano wrote: > > Out of the infinite number of names that x could be > > imported as (import x as y etc) what's so special about _x > > that it deserves a short-cut? > > Not exactly sure what your saying here, but, i'll fancy a > guess... "import as" is a general mechanism for renaming variables on import. When can use any new name we like: from module import x as a from module import x as b from module import x as c ... from module import x as z and all get treated the same way: whatever name you want to rename x as, you use the same, standard syntax: "import oldname as newname". But you are suggesting a new keyword, or other built-in functionality, to handle *one special case only*: import oldname as _oldname The Zen of Python tells us about special cases: Special cases aren't special enough to break the rules. Of course, the Zen of Python isn't absolute. If there really is something special enough to justify a special case, then the "Special cases" koan doesn't apply. Hence my question: why is _oldname so special that it deserves special syntax? I'm not questioning that it's special. I'm questioning what makes it sufficiently special that it overrides the design principles espoused by the Zen of Python. -- Steven From dw+python-ideas at hmmz.org Wed Mar 16 22:12:38 2016 From: dw+python-ideas at hmmz.org (David Wilson) Date: Thu, 17 Mar 2016 02:12:38 +0000 Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: <91345ed8-13ca-4f0e-96b2-703d76475c03@googlegroups.com> References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <85bn6ej8nl.fsf@benfinney.id.au> <3b15d825-9499-4524-9208-d41d53923592@googlegroups.com> <91345ed8-13ca-4f0e-96b2-703d76475c03@googlegroups.com> Message-ID: <20160317021238.GB13670@k3> On Wed, Mar 16, 2016 at 06:58:55PM -0700, Rick Johnson wrote: > I avoid? __all__ like the plague. Too easy for it to get out of sync with the > API when i forget to add a new symbol. In that case why not simply also avoid 'from .. import *' like the plague? It seems far simpler than renaming every single import, which places a permanent burden on future maintainers, code completing editors, grep, etc. FWIW I'd probably _break _down _and cry _if _faced _with _maintaining _code _like _that. David From rantingrickjohnson at gmail.com Wed Mar 16 23:02:05 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Wed, 16 Mar 2016 20:02:05 -0700 (PDT) Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: <20160317021238.GB13670@k3> References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <85bn6ej8nl.fsf@benfinney.id.au> <3b15d825-9499-4524-9208-d41d53923592@googlegroups.com> <91345ed8-13ca-4f0e-96b2-703d76475c03@googlegroups.com> <20160317021238.GB13670@k3> Message-ID: On Wednesday, March 16, 2016 at 9:19:52 PM UTC-5, David Wilson wrote: > > On Wed, Mar 16, 2016 at 06:58:55PM -0700, Rick Johnson wrote: > > > I avoid __all__ like the plague. Too easy for it to get out of sync > with the > > API when i forget to add a new symbol. > > In that case why not simply also avoid 'from .. import *' like the > plague? I do. > It seems far simpler than renaming every single import, which > places a permanent burden on future maintainers, code completing > editors, grep, etc. > Oh, i understand your confusion now. You think i'm importing *EVERY* symbol this way. No, i'm only importing the symbols i need. And, just FYI, not all modules exist as a "top level component". Many times, you need to split parts of a "component hierarchy" across multiple modules, with each module in the "chain", building upon the last, to create a complex system, that is greater than the sum of all it's parts. For instance: I might start with a "base module" that exposes a few public symbols. Granted, the module may have many private variables, but it only exposes a handful to the public. If i then "inherit" from that module, and "extend it", the symbols that i import from the first module may have made sense to be public in the first module, but now, in the new module, they are merely sub-components of a larger interface, and should therefore be private. (i use the terms "extend", and "inherit" very loosely here) If i simply wanted *ALL* the pubic symbols, and the module i was importing *FROM* followed the Python style guide by using a single underscore for private symbols, then "from module import *" is the way to go -- because i would *ONLY* receive the public symbols. > FWIW I'd probably _break _down _and cry _if _faced _with _maintaining > _code _like _that. > This reminds me of a funny story. I once told my doctor "When i do this, it hurts". And you what he told me? He said: "Well, don't do that" -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Wed Mar 16 23:03:38 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 16 Mar 2016 21:03:38 -0600 Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: <85bn6ej8nl.fsf@benfinney.id.au> References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <85bn6ej8nl.fsf@benfinney.id.au> Message-ID: On Wed, Mar 16, 2016 at 5:38 PM, Ben Finney wrote: > What is wrong with:: > > import lorem as _lorem > > do_something_with(_lorem.Foo) +1 -eric From rantingrickjohnson at gmail.com Wed Mar 16 23:38:21 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Wed, 16 Mar 2016 20:38:21 -0700 (PDT) Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <85bn6ej8nl.fsf@benfinney.id.au> <3b15d825-9499-4524-9208-d41d53923592@googlegroups.com> Message-ID: On Wednesday, March 16, 2016 at 8:32:12 PM UTC-5, Andrew Barnert via Python-ideas wrote: > This seems like something that shouldn't be done in general, so doesn't > need a language fix--but if you for some specific reason need to do it all > the time, just write a function for it: > > def _postimport(mod, names): > g = sys._getframe(1).f_globals > for name in names.split(): > g['_'+name] = mod[name] > > So: > > import thingy > _postimport(thingy, 'spam eggs cheese') > use(_spam, _cheese) > > Of course you can wrap up the import and _postimport in a single function: > > _magic_from_import('thingy', 'spam eggs cheese') > > Or use MacroPy to make the syntax nicer. > > Or, if even that's not good enough, write a simple token-processing import > hook that finds "_import" tokens and converts them to calls to your wrapper > function. > > Of course the farther you go down this path, the less readable your code > becomes to someone who doesn't know about your function/macro/hook, but if > there's really so much boilerplate that it's getting in the way, the > tradeoff might be worth it. > I like your ideas, and thanks for taking the time to write up a few lines of code. Looks like this will have to be a feature that i implement for myself. Heck, I had already planned on re-implementing the entire import mechanism anyway, because I like to spread my module source code across multiple files, and i want to explicitly define *HOW* those source files are combined to create namespaces at run-time, so that i can keep them separate, whist easily sharing state between them.. Python does not allow me to do this without resorting to "import contortions" and monkey patching -- but i know how to do it! *evil grin* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Mar 17 02:17:28 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 17 Mar 2016 16:17:28 +1000 Subject: [Python-ideas] Adding python to Windows. Was :Re: Make py.exe default to Python 3 In-Reply-To: References: <56E9A977.4010303@python.org> Message-ID: On 17 March 2016 at 08:13, Oscar Benjamin wrote: > On 16 Mar 2016 18:44, "Steve Dower" wrote: >> As a result, if Python did ship in Windows, I'd be trying really hard to >> make it an internal implementation detail to discourage people from ever >> relying on it. As Nick Coghlan said when I last chatted with him about this, >> "please, please, learn from our mistakes" :) > > I can understand that. The OSX system Python is like that. You're not > supposed to mess with it and when you want to do some programming every > guide suggests to install a newer non-system Python. I often see reports of > people getting into trouble after removing the system Python because they > wanted a newer version for some reason. "This is for operating system use, do not touch unless you're developing an operating system component" is the direction we're wanting to take the system Python installation on Fedora by moving it out to a dedicated "system-python" executable: https://fedoraproject.org/wiki/Changes/System_Python Once we can get the OS itself to a point of not caring what gets run when you invoke "/usr/bin/python" or "/usr/bin/python3", then we gain a lot more flexibility in how we handle the default symlinks (e.g. replacing them with something like Geoffrey Thomas's Python-launcher-for-POSIX-systems [1], or just letting machine admins configure them via existing alternatives management systems) Cheers, Nick. [1] https://ldpreload.com/blog/usr-bin-python-23 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Thu Mar 17 04:10:04 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 17 Mar 2016 17:10:04 +0900 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> <56E966DC.1050808@gmail.com> Message-ID: <22250.26204.510546.41119@turnbull.sk.tsukuba.ac.jp> Paul Moore writes: > On 16 March 2016 at 17:17, Chris Barker wrote: > > If someone installs a version of python with defaults, they > > should then get that version when they type "python" (or maybe > > "py") at the command line. > > > > If they want something other than the last one they installed, then they'll > > have to mess with PATH or other configuration... > > In theory, this makes sense to me. Really? To me, it never made a lot of sense even in theory. It's most likely to totally screw exactly the users it's addressed to (the ones who don't know how, don't wanna know how, and probably shouldn't be allowed[1], to manipulate PATH). It seems to me that the situation is 1. Power users and developers are often using a new Python in a "trial" situation or otherwise not wanting an *installed* Python on PATH. For historical reasons, they are used to solving that by not putting it on PATH by default. This weighed large in earlier discussions for the reasons you (Paul) give. As you also acknowledge, that large weight (vs. educators' needs) is hardly considered desirable by anybody, but the educator faction (often well-represented by folks like Raymond Hettinger and David Beazley) either didn't weigh in or didn't object or in the end conceded their objections were weak/had no good solution. 2. Python tries to provide both strong guarantees of backward compatibility without inhibiting development. This resulted in a kind of "punctuated equilibrium" evolution where x.y was theoretically backward-compatible with x.y-1, but that's never been quite true (bugs, *very* occasional changes of syntax as compared to the size of the language, errors that are caught and handled differently from the new version extension's definition, etc). Surprising users with *intermittent* and *hard to diagnose* bugs in other applications has never been considered acceptable, and the *nix distros I'm most familiar with (MacPorts, Debian, Gentoo) all provide a "python-select" utility for explicit choice of binding of "python", "python2", and "python3" to executables, with a user decision required to change the current bindings. This suggests to me that there are excellent reasons based in experience against defaulting to "upgrading" Python. "Sufficiently naive users" are just as vulnerable to such surprises, and as an educator myself, I would be unlikely to want to take responsibility for some other teacher's software being broken by my suggestion to upgrade Python so that *my* software can use newer syntax. I'd rather deal with my own problems by teaching students to deal with PATH (but see fn [2]). 3. In a previous geological epoch, naive beginner users used to be used to fixing up PATH, but that gave way in the early days of the current epoch to "computers for the rest of us (and big margins for Apple and Microsoft)" and "DLL hell", which by and large has been remedied by app-ization (ie, including all the DLLs you need, providing resources in a fixed hierarchy relative to the executable, and hard-coding those relative paths into the executable). This is how Microsoft and Apple arrange to ensure that only one instance of an app need to be installed, and resulted in such inelegant devices as "fat binaries" and the Microsoft Word format menu for saving. Unfortunately, this flat-out contradicts the Unix philosophy of modularized tools that do only one thing, and in the case of hosts involved in applications that face the Doomsday Machine usually called "the Internet", it's essential that certain services (crypto, authn, authz, ...) and important that others (anything with an overflowable buffer, which is everything) be upgraded in one place asap, rather than discarding the whole system in order to upgrade all apps that might be exposed. CPython chose to *not* distribute itself as app-ized by default, and in fact, doesn't do that at all, leaving it up to 3rd parties. I don't really see how to do that, anyway, given the huge amount of useful software that expects a previously-installed Python. Anyway, AFAIK that corner of the ecosystem is actually well-populated (Active State, Anaconda, etc). Why isn't one of those the appropriate solution to this problem for educators?[2] > > After all, what do people expect when they have MSWord version X, > > and then install MSWord version X+1 > But nobody *ever* runs multiple versions of MS Word. Slightly off the mark, I think. The reason analogies to Word or Excel implode is that they suck as general-purpose programming environments (IMHO) and so rarely depend on third-party modules.[3] People write programs "strictly from need" (they're actually manipulating Word files), or because it's the only hammer they've got (accountants writing financial models in Excel), or backward-compatibility with programs written by one-hammer developers (financial models in Excel). And the language-level backward-compatibility burden is entirely borne by Microsoft -- which they are quite good at AFAICT, and that (expensive!) backward-compatibility is why "nobody [aka "Andrew Barnert"] *ever*[sic] runs multiple versions of Word." Footnotes: [1] http://linux-mafia.com/~rick/faq/crybaby.html#allowed [2] Personally, the great majority of my students use Macs and at least on Macs the PSF-installed IDLE provides a nice icon in Applications, and that's how they get started, and even those who fall into the "shouldn't be allowed" group have little trouble handling Python 2 and Python 3 on the same box. So I don't even bother with those, although I'm currently reviewing Anaconda as a better solution for statistical applications. [3] Counter-examples welcome, I recognize my limited experience here. From p.f.moore at gmail.com Thu Mar 17 05:14:25 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 17 Mar 2016 09:14:25 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <22250.26204.510546.41119@turnbull.sk.tsukuba.ac.jp> References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> <56E966DC.1050808@gmail.com> <22250.26204.510546.41119@turnbull.sk.tsukuba.ac.jp> Message-ID: On 17 March 2016 at 08:10, Stephen J. Turnbull wrote: >> > If someone installs a version of python with defaults, they > > > should then get that version when they type "python" (or maybe > > > "py") at the command line. > > > > > > If they want something other than the last one they installed, then they'll > > > have to mess with PATH or other configuration... > > > > In theory, this makes sense to me. > > Really? To me, it never made a lot of sense even in theory. It's > most likely to totally screw exactly the users it's addressed to (the > ones who don't know how, don't wanna know how, and probably shouldn't > be allowed[1], to manipulate PATH). Hmm, I think I follow what you're getting at. It does seem reasonable to me that if a user installs (a version of) Python, then they should find that the "python" command works and executes that version. Particularly so now that a default install of Python 3.5 on Windows puts the executables in a relatively hard to locate directory (so manually adding to PATH is tedious). Replace the "python" command with "py" in this statement and we have the current behaviour of the Python installer. I'm OK with conceding that anyone installing a second version of Python counts as an "advanced" user and should manage the results themselves :-) (Exception: if you uninstall your current version first, you're back to the "new install" case). So for new users (without Python install): "Install your preferred copy of Python, and use the command py to run Python (either to run a script, or for interactive use)". Everyone else should be prepared to accept a (brief) explanation on how to manage multiple versions of software on their computer. That seems reasonable to me, but I'm not an educator (at least, I only teach Python to IT professionals). I'm also inclined to assume that for the *really* "new to IT" users, Idle (which is available as a start menu item) is a better starting point anyway. But as I've already said, I'd rather leave the actual proposals to people with experience teaching new users. I offer the above simply as a perspective on how people with too much knowledge to empathise with complete newcomers might see the arguments :-) Paul From oscar.j.benjamin at gmail.com Thu Mar 17 06:48:00 2016 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 17 Mar 2016 10:48:00 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> <56E966DC.1050808@gmail.com> <22250.26204.510546.41119@turnbull.sk.tsukuba.ac.jp> Message-ID: On 17 March 2016 at 09:14, Paul Moore wrote: > On 17 March 2016 at 08:10, Stephen J. Turnbull wrote: >>> > If someone installs a version of python with defaults, they >> > > should then get that version when they type "python" (or maybe >> > > "py") at the command line. >> > > >> > > If they want something other than the last one they installed, then they'll >> > > have to mess with PATH or other configuration... >> > >> > In theory, this makes sense to me. >> >> Really? To me, it never made a lot of sense even in theory. It's >> most likely to totally screw exactly the users it's addressed to (the >> ones who don't know how, don't wanna know how, and probably shouldn't >> be allowed[1], to manipulate PATH). > > Hmm, I think I follow what you're getting at. > > It does seem reasonable to me that if a user installs (a version of) > Python, then they should find that the "python" command works and > executes that version. Great! I'm glad that we're agreed on the basic point that: Python is installed in standard way implies -> "python" command works in all the usual places > Particularly so now that a default install of > Python 3.5 on Windows puts the executables in a relatively hard to > locate directory (so manually adding to PATH is tedious). Replace the > "python" command with "py" in this statement and we have the current > behaviour of the Python installer. > > I'm OK with conceding that anyone installing a second version of > Python counts as an "advanced" user and should manage the results > themselves :-) (Exception: if you uninstall your current version > first, you're back to the "new install" case). > > So for new users (without Python install): "Install your preferred > copy of Python, and use the command py to run Python (either to run a > script, or for interactive use)". What? I thought we just agreed it should be "python"! :) This brings us full circle back to the reason I interjected in this thread: why are we making the standard invocation of Python different on Windows? I'll drag up a bit of conversation from earlier: On 15 March 2016 at 04:47, Nick Coghlan wrote: > On 15 March 2016 at 09:01, Paul Moore wrote: >> >> I apologise if I gave the impression in this thread that there's any >> sort of "official" change in advice on how to launch Python. As far as >> I know, there hasn't been. > > It's still "python" - that works regardless of version on Windows > (with a suitably configured PATH), in virtual environments and in > pyenv and conda environments (and probably in Enthought Canopy as > well, but I haven't checked that). (and it's also "python" on every other non-Windows OS I've used, leaving aside "python3") So are we agreed or not that "python" is the way to run... Python? > I'm also inclined to assume that for the *really* "new to IT" users, Idle > (which is available as a start menu item) is a better starting point > anyway. My own students are introduced via Idle and have no problem (on Windows or OSX) locating Idle graphically. It's later on when we expect them to use the command line they discover that they have PATH problems. But this is happening at the first point that they use the terminal (it seems natural since they've learned Python already that it should be one of the first terminal commands they would use). The installer already has the capability and a tickbox for this. See here: https://docs.python.org/3.5/using/windows.html The tickbox that says: "Install launcher for all users (recommended)" is ticked already and describes itself as being "recommended". The one that says "Add Python 3.5 to PATH" is unticked and apparently not "recommended". If we're agreed that the "python" command should work once Python is installed then why are we not "recommending" that people tick the box that does this? Advanced users should understand what this means and can decide for themselves whether or not they want to tick/untick this box so I don't see why they're relevant on the issue of what is default and what is recommended. Novices won't understand what the text means so we really need to think about them when choosing a default behaviour here. How about a UI that shows the user what Pythons are installed and allows them to select which one is on PATH? Another option would be to add a path configuration menu to Idle. This way a user can open Idle, go to something like settings->PATH configuration and then be presented with a list of installed Pythons and the ability to select which should be on PATH. That would be better than asking novices to go edit PATH but it still begs the question: if this is part of the standard setup then why is not standard to do it when installing? -- Oscar From p.f.moore at gmail.com Thu Mar 17 06:58:24 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 17 Mar 2016 10:58:24 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> <56E966DC.1050808@gmail.com> <22250.26204.510546.41119@turnbull.sk.tsukuba.ac.jp> Message-ID: On 17 March 2016 at 10:48, Oscar Benjamin wrote: >> So for new users (without Python install): "Install your preferred >> copy of Python, and use the command py to run Python (either to run a >> script, or for interactive use)". > > What? I thought we just agreed it should be "python"! :) > > This brings us full circle back to the reason I interjected in this > thread: why are we making the standard invocation of Python different > on Windows? Because for me, personally, I don't view cross-platform compatibility of "the command you use to invoke Python" as a goal. At least not until it's possible on Unix for "python" to invoke Python 3. I will happily concede that your goals are different. Paul From desmoulinmichel at gmail.com Thu Mar 17 09:46:28 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Thu, 17 Mar 2016 14:46:28 +0100 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <1458070315.4009226.550113906.2627828D@webmail.messagingengine.com> <56E87763.80 000@python.org> Message-ID: <56EAB534.4040802@gmail.com> Le 16/03/2016 18:46, Steve Dower a ?crit : > The uninstall problem is actually that "python" would stop working > completely in that case, and it's not necessarily obvious why or how to > fix it. > > The leave-behind problem doesn't really matter as the directory is not > searched if it doesn't exist, but we also don't leave behind entries > added by the installer. A user who customizes their own PATH will > obviously need to manually remove the entry, but that's a manual action > both times so I don't see any concern there. > > At one point I was trying to convince the Windows team to add a global > "python" command to all Windows installs (probably 10 and later - we > don't backport stuff like this, but it could be a manual install) that > would act like the py.exe launcher but also automatically download > Python if you didn't have it. What would people think if I got that to > happen? > I would pay for that to happen :) > Top-posted from my Windows Phone > ------------------------------------------------------------------------ > From: Andrew Barnert > Sent: ?3/?16/?2016 8:30 > To: Steve Dower > Cc: python-ideas at python.org > Subject: Re: [Python-ideas] Make py.exe default to Python 3 > > On Mar 15, 2016, at 13:58, Steve Dower wrote: >> >>> On 15Mar2016 1231, Random832 wrote: >>> I think what might be most appropriate would be to have a "Python >>> Console" in the start menu (alongside the IDLE entry) which sources a >>> bat file to add the correct python version (and the scripts directory >>> for pip and easy_install) to that window's PATH, rather than installing >>> it in the global PATH, and users can learn to start the console from >>> that icon when they want to run python commands. Inspired by the fact >>> that e.g. Visual Studio has something that does the same. >> >> This is one of two solutions I'd be comfortable with. >> >> The other is to rename "py[w].exe" as "python[w].exe". >> >> PATH is used to resolve system DLLs and common commands. Over time, > better security features are being added for core system components, but > it is still possible to hijack programs this way. > > But isn't the whole point here that we want novices to be able to treat > python and/or python3 as a "common command", just like py? > > That implies either putting the executable in an existing PATH > directory, or adding the executable's directory to the PATH (or, of > course, renaming py, as you suggest, since that already gets put into an > existing directory). > >> It also has a serious conflict problem once the second version of > Python is installed with this option enabled, in that *anything* > currently referring to just "python" will change. > > If this is only intended for rank beginners, what about just disabling > the checkbox, with a warning, if there's already a program named > "Python" anywhere on the PATH? Then, when someone gets into trouble > because they're installing two Pythons, they'll get a warning and be > able to ask their teacher or search StackOverflow or whatever and get a > quick answer. And that means we don't have to solve the conflict > problems, the uninstall leave-behind problems, etc. for multiple pythons. > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From steve at pearwood.info Thu Mar 17 11:45:47 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 Mar 2016 02:45:47 +1100 Subject: [Python-ideas] Add citation() to site.py Message-ID: <20160317154546.GF8022@ando.pearwood.info> The standard site.py module adds pseudo-builtins: copyright credits exit help license quit I suggest one more: citation(). Python is being used heavily in scientific and academic fields, where it is often the convention to provide citations and references to the software used. The question of how to cite Python comes up from time to time, e.g.: https://mail.python.org/pipermail/tutor/2016-March/108460.html http://academia.stackexchange.com/questions/5482/how-do-i-reference-the-python-programming-language-in-a-thesis-or-a-paper http://www.gossamer-threads.com/lists/python/python/105846 http://grokbase.com/t/python/python-list/04684fggwd/citing-python I think that having a standard answer available for this question is good practice, and will help strength Python's position as a good scientific language. SciPy, SymPy and iPython all document how they prefer to be cited: https://scipy.org/citing.html https://github.com/sympy/sympy#citation http://ipython.org/citing.html As do other languages such as Mathematica: http://support.wolfram.com/kb/472 But I think it would be a good idea to emulate the R language, which provides a function that gives the prefered citation. At the R prompt: > citation() To cite R in publications use: R Core Team (2014). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL http://www.R-project.org/. A BibTeX entry for LaTeX users is @Manual{, title = {R: A Language and Environment for Statistical Computing}, author = {{R Core Team}}, organization = {R Foundation for Statistical Computing}, address = {Vienna, Austria}, year = {2014}, url = {http://www.R-project.org/}, } We have invested a lot of time and effort in creating R, please cite it when using it for data analysis. See also ?citation("pkgname")? for citing R packages. * * * If you provide a package name such as "splines", the R's output is the same except that the first line starts with: "The ?splines? package is part of R." (I don't have any third-party packages installed to test, but presumably they will offer customised output.) My suggestion is that we follow R's lead and add a citation() function to site.py which gives a suggested, standard, citation string that people can copy. Obviously we cannot expect to match all standard formats, but we can provide one and folks can convert to the format their university of journal requires. (Converting between academic reference styles is out of scope of this proposal.) The initial implementation might look something like this: year = platform.python_build()[1].split()[2] vers = platform.python_version() _CITE = """\ To cite Python in publications, please use: Python Core Team ({}). Python {}: A dynamic, open source programming language. Python Software Foundation. URL https://www.python.org/. """.format(year, vers) def citation(): print(_CITE) Or perhaps it should use the same implementation as copyright, credits and license. Thoughts? -- Steve From ethan at stoneleaf.us Thu Mar 17 11:56:04 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 17 Mar 2016 08:56:04 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <20160317154546.GF8022@ando.pearwood.info> References: <20160317154546.GF8022@ando.pearwood.info> Message-ID: <56EAD394.9080501@stoneleaf.us> On 03/17/2016 08:45 AM, Steven D'Aprano wrote: > The standard site.py module adds pseudo-builtins: > > copyright > credits > exit > help > license > quit > > > I suggest one more: citation(). No opinion on exact implementation, but for the idea in general: +1 -- ~Ethan~ From srkunze at mail.de Thu Mar 17 12:17:02 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 17 Mar 2016 17:17:02 +0100 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <56EAD394.9080501@stoneleaf.us> References: <20160317154546.GF8022@ando.pearwood.info> <56EAD394.9080501@stoneleaf.us> Message-ID: <56EAD87E.9060808@mail.de> On 17.03.2016 16:56, Ethan Furman wrote: > On 03/17/2016 08:45 AM, Steven D'Aprano wrote: >> The standard site.py module adds pseudo-builtins: >> >> copyright >> credits >> exit >> help >> license >> quit >> >> >> I suggest one more: citation(). > > No opinion on exact implementation, but for the idea in general: > > +1 > Same for me. +1 Best, Sven From abarnert at yahoo.com Thu Mar 17 12:32:47 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 17 Mar 2016 09:32:47 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: <22250.26204.510546.41119@turnbull.sk.tsukuba.ac.jp> References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> <56E966DC.1050808@gmail.com> <22250.26204.510546.41119@turnbull.sk.tsukuba.ac.jp> Message-ID: <9F085825-29F3-4AA4-B135-B4096ED6986C@yahoo.com> On Mar 17, 2016, at 01:10, Stephen J. Turnbull wrote: > > And the language-level backward-compatibility burden is entirely borne > by Microsoft -- which they are quite good at AFAICT, and that > (expensive!) backward-compatibility is why "nobody [aka "Andrew > Barnert"] *ever*[sic] runs multiple versions of Word." I wasn't running Word 2003 and Word 2007 as VBA development platforms, or using them side by side because I wanted both.[1] That's just what the installer decided to do.[2] Most people don't notice this, because how many users launch Word from the command line or via Win-R?[3] Not to mention that most people don't keep upgrading the same Windows box continuously over Office's 4-year dev cycle;[4] they're either still running 2003, or they threw that laptop away long ago. At any rate, the point is that Word is a terrible example of "everybody expects side-by-side installs but the latest-installed version wins"--most people don't expect side-by-side installs, and when you do get them, the latest version installed doesn't win (and, even if it did, they wouldn't notice). That doesn't necessarily mean it would be the wrong decision for Python, just that the argument "everyone else does it" doesn't help, because it's not true. If we want to talk about language-level compatibility, Visual Studio might be a better example than Office. And, as mentioned earlier in the thread, Visual Studio solves this problem by putting a "switching script" on the PATH, which adds the selected copy's executables to the PATH when run (and by adding special start menu shortcuts to open a new command prompt with the PATH set up). So, CL.EXE doesn't come from the latest MSVS installed either. [1]: In fact, the only reason I needed Word 2007 in the first place was that i contracted for a company that insisted I have it, even though every document they sent me worked just fine in OpenOffice... [2]: This is just a guess, but I'd bet it's because the $100 Office Home Professional 2007 didn't include the same subset of apps as the $100 Office Personal Business 2003 it replaced, and they didn't want people complaining about the upgrade taking away their PowerPoint or Access or Publisher or whatever the way people complained during the 2000-2003 upgrade. You'd think they could solve that by just offering the same set of packages of apps under consistent names and price points so this never came up, and I'll bet every 4 years, the suite-integration dev team for Office thinks they've convinced management that's a good idea, and then 3 months before release they discover that marketing convinced PM otherwise, and they now have to deal with it. [3]: People _do_ launch it via Win search for "word" nowadays, but that uses Windows' clever search guessing and learning, not anything configurable by the installer or the user, unlike Win-R "winword". [4]: Or, if they do, either it's an IT-managed upgrade, or they're installing a pirated copy of the full Office suite (which probably has a custom installer?). From Nikolaus at rath.org Thu Mar 17 12:32:55 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Thu, 17 Mar 2016 09:32:55 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <20160317154546.GF8022@ando.pearwood.info> (Steven D'Aprano's message of "Fri, 18 Mar 2016 02:45:47 +1100") References: <20160317154546.GF8022@ando.pearwood.info> Message-ID: <87bn6doyiw.fsf@thinkpad.rath.org> On Mar 18 2016, Steven D'Aprano wrote: > The standard site.py module adds pseudo-builtins: > > copyright > credits > exit > help > license > quit > > > I suggest one more: citation(). I think the core question is: are the (core?) Python contributors or the PSF interested in being referenced? If so, adding this would definitely help (and would certainly generate some citations from me). If not, there seems little point. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From ericsnowcurrently at gmail.com Thu Mar 17 13:25:09 2016 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 17 Mar 2016 11:25:09 -0600 Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <85bn6ej8nl.fsf@benfinney.id.au> <3b15d825-9499-4524-9208-d41d53923592@googlegroups.com> Message-ID: On Mar 16, 2016 21:38, "Rick Johnson" wrote: > Heck, I had already planned on re-implementing the entire import mechanism anyway, because I like to spread my module source code across multiple files, and i want to explicitly define *HOW* those source files are combined to create namespaces at run-time, so that i can keep them separate, whist easily sharing state between them.. Python does not allow me to do this without resorting to "import contortions" and monkey patching -- but i know how to do it! *evil grin* So like Go does packages? You should be able to do that using a custom loader and a path-entry finder to provide that loader. See the importlib docs for more info. Just be careful to only handle your special directories or else normal packages will break. FWIW, there are more idiomatic ways to combine multiple files into a single module. For instance, make a package containing your separate files but make the files private (leading underscore). Then put "from XXX import *" for each of them. If you want to limit what's exported from the package, either define __all__ in the sub-files or use "from XXX import YYY" in the package __init__.py. As far as sharing a namespace between the sub-files, I'd recommend against an implicit mechanism (like Go does). I've found that it makes it harder to discover code. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Mar 17 16:03:19 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 17 Mar 2016 16:03:19 -0400 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <87bn6doyiw.fsf@thinkpad.rath.org> References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> Message-ID: On 3/17/2016 12:32 PM, Nikolaus Rath wrote: > On Mar 18 2016, Steven D'Aprano wrote: >> The standard site.py module adds pseudo-builtins: >> >> copyright >> credits >> exit >> help >> license >> quit >> >> >> I suggest one more: citation(). Excellent idea, though I suggest 'citation' instead. The current splash line is Type "copyright", "credits" or "license()" for more information. The difference between no () and () required is a matter of length, and a 3 or 4 line citation fits the no () case. >>> copyright Copyright (c) 2001-2015 Python Software Foundation. All Rights Reserved. Copyright (c) 2000 BeOpen.com. All Rights Reserved. Copyright (c) 1995-2001 Corporation for National Research Initiatives. All Rights Reserved. Copyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam. All Rights Reserved. >>> credits Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands for supporting Python development. See www.python.org for more information. >>> license Type license() to see the full license text The full license text is several pages, run through a pager. (quit and exit also have custom representations.) In addition, the front page of the docs should have a new item Citation under 'Meta Information'. The linked page could have multiple formats. > I think the core question is: are the (core?) Python contributors or the > PSF interested in being referenced? This one says YES!. Citing software used in a project is standard, but often a nuisance to get right. > If so, adding this would definitely > help (and would certainly generate some citations from me). -- Terry Jan Reedy From steve.dower at python.org Thu Mar 17 16:20:06 2016 From: steve.dower at python.org (Steve Dower) Date: Thu, 17 Mar 2016 13:20:06 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <87bn6doyiw.fsf@thinkpad.rath.org> References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> Message-ID: I'd be most interested in Guido's opinion, but personally I think Python is already well known enough to be a "household name" in any publication that might include a citation. Nobody cites C++ or Java, to my knowledge, and I'd rather be in that category than the other. Top-posted from my Windows Phone -----Original Message----- From: "Nikolaus Rath" Sent: ?3/?17/?2016 9:34 To: "python-ideas at python.org" Subject: Re: [Python-ideas] Add citation() to site.py On Mar 18 2016, Steven D'Aprano wrote: > The standard site.py module adds pseudo-builtins: > > copyright > credits > exit > help > license > quit > > > I suggest one more: citation(). I think the core question is: are the (core?) Python contributors or the PSF interested in being referenced? If so, adding this would definitely help (and would certainly generate some citations from me). If not, there seems little point. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Mar 17 16:32:17 2016 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 17 Mar 2016 13:32:17 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> Message-ID: On Mar 17, 2016 1:21 PM, "Steve Dower" wrote: > > I'd be most interested in Guido's opinion, but personally I think Python is already well known enough to be a "household name" in any publication that might include a citation. Nobody cites C++ or Java, to my knowledge, and I'd rather be in that category than the other. Norms are somewhat in flux here -- there's a lot of discussion going on in academic circles right now about how to appropriately credit software. Traditionally the answer has been that software mostly doesn't get credited at all, which has predictably pernicious effects... One way to think about it is that providing a preferred citation is useful for folks who want to cite, and folks who don't want to cite will ignore it. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 17 16:35:14 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 17 Mar 2016 13:35:14 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> Message-ID: <56EB1502.5010102@stoneleaf.us> On 03/17/2016 01:32 PM, Nathaniel Smith wrote: > One way to think about it is that providing a preferred citation is > useful for folks who want to cite, and folks who don't want to cite > will ignore it. I would also think that citing the software increases the ease of duplicating the results. -- ~Ethan~ From rosuav at gmail.com Thu Mar 17 16:41:42 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 18 Mar 2016 07:41:42 +1100 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <56EB1502.5010102@stoneleaf.us> References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> Message-ID: On Fri, Mar 18, 2016 at 7:35 AM, Ethan Furman wrote: > I would also think that citing the software increases the ease of > duplicating the results. > If that's the intention, citation() should include enough details to ensure this. There's been talk of changing the default PRNG, so that kind of thing would have to be identifiable (eg if the citation says "CPython 3.6.1" and it's known that the PRNG changed in 3.6, people will be able to understand why the results differ on 3.5). Probably sys.version will have enough info for that. ChrisA From ethan at stoneleaf.us Thu Mar 17 16:49:47 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 17 Mar 2016 13:49:47 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> Message-ID: <56EB186B.1020906@stoneleaf.us> On 03/17/2016 01:41 PM, Chris Angelico wrote: > On Fri, Mar 18, 2016 at 7:35 AM, Ethan Furman wrote: >> I would also think that citing the software increases the ease of >> duplicating the results. > > If that's the intention, citation() should include enough details to > ensure this. There's been talk of changing the default PRNG, so that > kind of thing would have to be identifiable (eg if the citation says > "CPython 3.6.1" and it's known that the PRNG changed in 3.6, people > will be able to understand why the results differ on 3.5). Probably > sys.version will have enough info for that. If the citation doesn't have enough information to enable somebody to go and get the same software package it seems rather pointless. As one example other goodies bundled with a distribution will vary betwixt them. -- ~Ethan~ From Nikolaus at rath.org Thu Mar 17 17:24:36 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Thu, 17 Mar 2016 14:24:36 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <56EB1502.5010102@stoneleaf.us> (Ethan Furman's message of "Thu, 17 Mar 2016 13:35:14 -0700") References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> Message-ID: <871t78pzl7.fsf@thinkpad.rath.org> On Mar 17 2016, Ethan Furman wrote: > On 03/17/2016 01:32 PM, Nathaniel Smith wrote: > >> One way to think about it is that providing a preferred citation is >> useful for folks who want to cite, and folks who don't want to cite >> will ignore it. > > I would also think that citing the software increases the ease of > duplicating the results. Really? What kind of citation that you have in mind that would help more than "...we used a Python program to do [bla]..." somewhere in the paper? Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From steve.dower at python.org Thu Mar 17 17:38:37 2016 From: steve.dower at python.org (Steve Dower) Date: Thu, 17 Mar 2016 14:38:37 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <56EB1502.5010102@stoneleaf.us> References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> Message-ID: <56EB23DD.5050908@python.org> On 17Mar2016 1335, Ethan Furman wrote: > On 03/17/2016 01:32 PM, Nathaniel Smith wrote: > > > One way to think about it is that providing a preferred citation is > > useful for folks who want to cite, and folks who don't want to cite > > will ignore it. > > I would also think that citing the software increases the ease of > duplicating the results. Afraid not. Posting a requirements.txt or a conda spec file would be far more valuable for this, or these days sharing a Jupyter Notebook with the code and dependency-spec embedded. It's also arguable as to how valuable reproducing the results in an identical environment actually is. Yes, your code runs on a different machine, but if your "research" is code then you are a developer, not a scientist. Someone actually needs to implement the idea in a different environment (and ideally they know just how it differs) to demonstrate that it is an actual result and not just a fluke. Cheers, Steve From guido at python.org Thu Mar 17 17:51:37 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Mar 2016 14:51:37 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <56EB23DD.5050908@python.org> References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> Message-ID: I replied from my cell phone but it bounced (again :-( ). I wrote: """ I think this should be in the docs, not in the code. (The reason to have copyright() in the code is purely legalistic.) """ But I would be much more interested in hearing what the citation text should be before discussing how to add it to the code. (I still think adding it to the code isn't very helpful -- we can't add this to the older versions of code that people are likely to be using.) On Thu, Mar 17, 2016 at 2:38 PM, Steve Dower wrote: > On 17Mar2016 1335, Ethan Furman wrote: >> >> On 03/17/2016 01:32 PM, Nathaniel Smith wrote: >> >> > One way to think about it is that providing a preferred citation is >> > useful for folks who want to cite, and folks who don't want to cite >> > will ignore it. >> >> I would also think that citing the software increases the ease of >> duplicating the results. > > > Afraid not. Posting a requirements.txt or a conda spec file would be far > more valuable for this, or these days sharing a Jupyter Notebook with the > code and dependency-spec embedded. > > It's also arguable as to how valuable reproducing the results in an > identical environment actually is. Yes, your code runs on a different > machine, but if your "research" is code then you are a developer, not a > scientist. Someone actually needs to implement the idea in a different > environment (and ideally they know just how it differs) to demonstrate that > it is an actual result and not just a fluke. > > Cheers, > Steve > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From njs at pobox.com Thu Mar 17 19:08:35 2016 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 17 Mar 2016 16:08:35 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <56EB23DD.5050908@python.org> References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> Message-ID: On Thu, Mar 17, 2016 at 2:38 PM, Steve Dower wrote: > On 17Mar2016 1335, Ethan Furman wrote: >> >> On 03/17/2016 01:32 PM, Nathaniel Smith wrote: >> >> > One way to think about it is that providing a preferred citation is >> > useful for folks who want to cite, and folks who don't want to cite >> > will ignore it. >> >> I would also think that citing the software increases the ease of >> duplicating the results. > > > Afraid not. Posting a requirements.txt or a conda spec file would be far > more valuable for this, or these days sharing a Jupyter Notebook with the > code and dependency-spec embedded. Docker containers are also getting intense interest for this use case. It is true that citations can provide some help with reproduction (e.g. the cite might give you a hint whether the code attached to the paper was tested on py2 or py3), but yeah, cites are more about distributing credit and gathering metrics ("how many papers published last year used python?") than about reproducibility per se. > It's also arguable as to how valuable reproducing the results in an > identical environment actually is. Yes, your code runs on a different > machine, but if your "research" is code then you are a developer, not a > scientist. The complication here is that it turns out that if you follow this definition, then there are fewer and fewer scientists every year. Soon there will be none left :-). These questions around reproducibility/replicability are also extremely hot topics in science right now... not clear how it will all play out, but I think it's safe to say that there are a lot of scientists who think recording and communicating exact environments is very valuable. -n -- Nathaniel J. Smith -- https://vorpus.org From alexander.belopolsky at gmail.com Thu Mar 17 19:31:40 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 17 Mar 2016 19:31:40 -0400 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> Message-ID: On Thu, Mar 17, 2016 at 7:08 PM, Nathaniel Smith wrote: > > These questions around reproducibility/replicability are also > extremely hot topics in science right now... not clear how it will all > play out, but I think it's safe to say that there are a lot of > scientists who think recording and communicating exact environments is > very valuable. "In the good old days physicists repeated each other's experiments, just to be sure. Today they stick to FORTRAN, so that they can share each other's programs, bugs included." -- Edsger W.Dijkstra, 18 June 1975 -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Mar 17 19:33:02 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 Mar 2016 10:33:02 +1100 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> Message-ID: <20160317233254.GG8022@ando.pearwood.info> On Thu, Mar 17, 2016 at 02:51:37PM -0700, Guido van Rossum wrote: > I replied from my cell phone but it bounced (again :-( ). I wrote: > > """ > I think this should be in the docs, not in the code. (The reason to > have copyright() in the code is purely legalistic.) > """ If this is in the code, a possible future enhancement would be for it to take an optional module or module name and query that module for its preferred citation. That's what R does. > But I would be much more interested in hearing what the citation text > should be before discussing how to add it to the code. (I still think > adding it to the code isn't very helpful -- we can't add this to the > older versions of code that people are likely to be using.) True, but some day -- hopefully not that far away -- 3.6 will be the "older version" that people are likely to be using. My suggested citation format is based heavily on that used by R. It could be implemented something like this: year = platform.python_build()[1].split()[2] vers = platform.python_version() _CITE = """\ To cite Python in publications, please use: Python Core Team ({}). Python {}: A dynamic, open source programming language. Python Software Foundation. URL https://www.python.org/. """.format(year, vers) which would end up looking like this: To cite Python in publications, please use: Python Core Team (2015). Python 3.6.0a0: A dynamic, open source programming language. Python Software Foundation. URL https://www.python.org/. I'm not married to any particular wording or preferred format. -- Steve From guido at python.org Thu Mar 17 19:45:00 2016 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Mar 2016 16:45:00 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <20160317233254.GG8022@ando.pearwood.info> References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> Message-ID: Well, I entirely fail to understand why that belongs in the code rather than in the docs. "Because R does it" seems a pretty poor reason. But then again I've never cared about citations. On Thu, Mar 17, 2016 at 4:33 PM, Steven D'Aprano wrote: > On Thu, Mar 17, 2016 at 02:51:37PM -0700, Guido van Rossum wrote: >> I replied from my cell phone but it bounced (again :-( ). I wrote: >> >> """ >> I think this should be in the docs, not in the code. (The reason to >> have copyright() in the code is purely legalistic.) >> """ > > If this is in the code, a possible future enhancement would be for it to > take an optional module or module name and query that module for its > preferred citation. That's what R does. > > >> But I would be much more interested in hearing what the citation text >> should be before discussing how to add it to the code. (I still think >> adding it to the code isn't very helpful -- we can't add this to the >> older versions of code that people are likely to be using.) > > True, but some day -- hopefully not that far away -- 3.6 will be the > "older version" that people are likely to be using. > > > My suggested citation format is based heavily on that used by R. It > could be implemented something like this: > > year = platform.python_build()[1].split()[2] > vers = platform.python_version() > _CITE = """\ > To cite Python in publications, please use: > > Python Core Team ({}). Python {}: A dynamic, open > source programming language. Python Software Foundation. > URL https://www.python.org/. > > """.format(year, vers) > > > which would end up looking like this: > > To cite Python in publications, please use: > > Python Core Team (2015). Python 3.6.0a0: A dynamic, open > source programming language. Python Software Foundation. > URL https://www.python.org/. > > > I'm not married to any particular wording or preferred format. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From steve at pearwood.info Thu Mar 17 19:39:36 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 18 Mar 2016 10:39:36 +1100 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> Message-ID: <20160317233936.GH8022@ando.pearwood.info> On Thu, Mar 17, 2016 at 07:31:40PM -0400, Alexander Belopolsky wrote: > On Thu, Mar 17, 2016 at 7:08 PM, Nathaniel Smith wrote: > > > > These questions around reproducibility/replicability are also > > extremely hot topics in science right now... not clear how it will all > > play out, but I think it's safe to say that there are a lot of > > scientists who think recording and communicating exact environments is > > very valuable. > > > "In the good old days physicists repeated each other's experiments, just to > be sure. Today they stick to FORTRAN, so that they can share each other's > programs, bugs included." -- Edsger W.Dijkstra, 18 June 1975 I'm reminded of this quote from William H. Press, et al, "Numerical Recipes in Pascal": "If all scientific papers whose results are in doubt because of bad Randoms were to disappear from library shelves, there would be a gap on each shelf about as big as your fist." -- Steve From steve.dower at python.org Thu Mar 17 19:46:06 2016 From: steve.dower at python.org (Steve Dower) Date: Thu, 17 Mar 2016 16:46:06 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> Message-ID: <56EB41BE.3050103@python.org> On 17Mar2016 1608, Nathaniel Smith wrote: > On Thu, Mar 17, 2016 at 2:38 PM, Steve Dower wrote: >> It's also arguable as to how valuable reproducing the results in an >> identical environment actually is. Yes, your code runs on a different >> machine, but if your "research" is code then you are a developer, not a >> scientist. > > The complication here is that it turns out that if you follow this > definition, then there are fewer and fewer scientists every year. Soon > there will be none left :-). Nah, someone can just write a seminal paper that redefines science in terms of Python and we'll be good for another few decades :) > These questions around reproducibility/replicability are also > extremely hot topics in science right now... not clear how it will all > play out, but I think it's safe to say that there are a lot of > scientists who think recording and communicating exact environments is > very valuable. I certainly agree with this, but the point of communicating the exact environment is not always so that other can reproduce the environment (which includes the problem, parametisation, benchmarks, preprocessing and more as well as the software itself), but so they can compare environments. In some cases it could be used so the environment can be identical and a change to the work be compared instead, but I'd consider that "advancement" rather than "reproduction". But this is a tangent I'm not desperately keen to follow through on, so I'll leave it here. --- On the actual text of the citation, have the big "citation formats" (IEEE, APA, etc.) defined formats/metadata for versioned software products? I don't think they had last time I used them, but that was four years ago now. Cheers, Steve From njs at pobox.com Thu Mar 17 19:54:41 2016 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 17 Mar 2016 16:54:41 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <20160317233254.GG8022@ando.pearwood.info> References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> Message-ID: On Thu, Mar 17, 2016 at 4:33 PM, Steven D'Aprano wrote: > > On Thu, Mar 17, 2016 at 02:51:37PM -0700, Guido van Rossum wrote: > > I replied from my cell phone but it bounced (again :-( ). I wrote: > > > > """ > > I think this should be in the docs, not in the code. (The reason to > > have copyright() in the code is purely legalistic.) > > """ > > If this is in the code, a possible future enhancement would be for it to > take an optional module or module name and query that module for its > preferred citation. That's what R does. I think it would be premature for the stdlib to try to standardize machine readable citation metadata for third-party packages, or even a general API for accessing them. There are a lot of complex issues in this space that are still being explored by third-party packages like duecredit: https://github.com/duecredit/duecredit (Notice that the citation() function in R actually does some rather complicated things and returns a rather complicated object: https://stat.ethz.ch/R-manual/R-devel/library/utils/html/citation.html https://stat.ethz.ch/R-manual/R-devel/library/utils/html/bibentry.html ) -n -- Nathaniel J. Smith -- https://vorpus.org From abarnert at yahoo.com Thu Mar 17 20:01:10 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 17 Mar 2016 17:01:10 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> Message-ID: On Mar 17, 2016, at 16:08, Nathaniel Smith wrote: > >> On Thu, Mar 17, 2016 at 2:38 PM, Steve Dower wrote: >>> On 17Mar2016 1335, Ethan Furman wrote: >>> >>>> On 03/17/2016 01:32 PM, Nathaniel Smith wrote: >>>> >>>> One way to think about it is that providing a preferred citation is >>>> useful for folks who want to cite, and folks who don't want to cite >>>> will ignore it. >>> >>> I would also think that citing the software increases the ease of >>> duplicating the results. >> >> >> Afraid not. Posting a requirements.txt or a conda spec file would be far >> more valuable for this, or these days sharing a Jupyter Notebook with the >> code and dependency-spec embedded. > > Docker containers are also getting intense interest for this use case. It's going to be so much fun trying to look back at early 21st century science from the future. "Up to about 2015, they were still publishing 'papers' written in the relatively simple 'postscript' language that we easily reverse engineered an interpreter for. Over the next few decades, they started publishing in more complicated formats, like entire virtual machines that were run by some software we no longer have access to that's only portable to operating systems that we have the source code for but don't know how to build. Successfully extracting some memory proteins from a not-entirely-decomposed body in what used to be Australia caused a bit of excitement last megasecond, but so far all we've been able to learn is how to construct a hat out of red felt. So for this segment of the class, we'll be looking almost entirely at whatever pop-science articles were recoverable from a backup of the Internet archive." From tritium-list at sdamon.com Thu Mar 17 23:10:15 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Thu, 17 Mar 2016 23:10:15 -0400 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> Message-ID: <56EB7197.1090807@sdamon.com> On 3/17/2016 19:45, Guido van Rossum wrote: > Well, I entirely fail to understand why that belongs in the code > rather than in the docs. "Because R does it" seems a pretty poor > reason. But then again I've never cared about citations. > > I think that last sentence there speaks volumes on this subject. From rantingrickjohnson at gmail.com Fri Mar 18 00:04:22 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Thu, 17 Mar 2016 21:04:22 -0700 (PDT) Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <85bn6ej8nl.fsf@benfinney.id.au> <3b15d825-9499-4524-9208-d41d53923592@googlegroups.com> Message-ID: On Thursday, March 17, 2016 at 12:25:36 PM UTC-5, Eric Snow wrote: > So like Go does packages? You should be able to do that > using a custom loader and a path-entry finder to provide > that loader. See the importlib docs for more info. Just > be careful to only handle your special directories or else > normal packages will break. Yes, i had planned to do it in a manner that will exist side by with the existing "official import mechanism". It's not something i plan to do for "every single module". Only those that have become "too large to easily edit". > FWIW, there are more idiomatic ways to combine multiple > files into a single module. For instance, make a package > containing your separate files but make the files private > (leading underscore). Then put "from XXX import *" for > each of them. If you want to limit what's exported from > the package, either define __all__ in the sub-files or use > "from XXX import YYY" in the package __init__.py. Sure. That will allow me to "spread-out a large module's source code into N smaller files", and then let Python "combine the objects, defined in those multiple files, under one monolithic namespace at run-time"... but i can't define shared state between them *UNTIL* run-time -- at least, not without resorting to "import spider webs" and monkey patching. I'm not trying to "define a module from N objects contained in N source files", no, i'm trying to "define a *NAMESPACE* from N source files". "N source files" that would transparently share state just as though all the source existed in the exact same file. > As far as sharing a namespace between the sub-files, I'd > recommend against an implicit mechanism (like Go does). > I've found that it makes it harder to discover code. Hmm, you're not the only person that has raised this issue. I'm unaware of how Go handles namespaces, but i've used other languages that *DO* offer a programmer the option of "explicitly defining module namespace", and after writing code for many years, I have not once encountered the issues that you and others raise. I'm speculating here, but it may be because i'm very OCD about creating strong interfaces, and i avoid writing code that rebinds symbols "in the shadows". Of course, i'm not suggesting that you, or anyone else, is incompetent, I'm merely pondering, because it does sound like a legitimate possibility... Hey, i can tell you from personal experience: there is nothing worst than implementing a "grand idea", only to find out later, that it was "fools gold". O:-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri Mar 18 03:33:17 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 18 Mar 2016 16:33:17 +0900 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> Message-ID: <22251.44861.126294.891753@turnbull.sk.tsukuba.ac.jp> Nathaniel Smith writes: > Steve Dower wrote: > > It's also arguable as to how valuable reproducing the results in an > > identical environment actually is. Yes, your code runs on a different > > machine, but if your "research" is code then you are a developer, not a > > scientist. > > The complication here is that it turns out that if you follow this > definition, then there are fewer and fewer scientists every year. Soon > there will be none left :-). Actually, there is an ever-increasing number of wannabes who just go through the motions. At least among business academics, they are the ones most likely to cite (rather than publish in an "appendix available from the author") software (including programming languages like Python and application environments like R, as well as pure applications like SPSS). I suppose you might consider that an argument against citation(). ;-) Which is a partial answer to Guido: it turns out that in academic business statistics, there is substantial disagreement about appropriate models (especially in what is called "factor analysis" and "structural equation modeling"), which lead to differing calculations. As it happens, SPSS by default provides a particular factor analysis function that uses a now-deprecated model. You wouldn't know that without reading the SPSS manual. Other examples relevant to citing Python (with version info) include PRNGs (already mentioned), and validation depending on order of iteration of dicts, which are implementation dependent. this doesn't necessarily motivate for implementation in the interpreter, though. > These questions around reproducibility/replicability are also > extremely hot topics in science right now... not clear how it will all > play out, but I think it's safe to say that there are a lot of > scientists who think recording and communicating exact environments is > very valuable. I'm sure Dr. Obokata wishes she had been able to do so! From tjreedy at udel.edu Fri Mar 18 03:34:25 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 18 Mar 2016 03:34:25 -0400 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> Message-ID: On 3/17/2016 5:51 PM, Guido van Rossum wrote: > I replied from my cell phone but it bounced (again :-( ). I wrote: > > """ > I think this should be in the docs, not in the code. (The reason to > have copyright() in the code is purely legalistic.) > """ I think something on or accessible from the front doc page would be enough in most cases. The current copyright page is just 7 lines. A citation format could be added below and the link changed to Copyright and Citation Format. The people helped would include students who are supposed to add 'professional' citations in the bibliography of papers they write. This change could, of course, be backported. -- Terry Jan Reedy From tjreedy at udel.edu Fri Mar 18 03:41:28 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 18 Mar 2016 03:41:28 -0400 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> Message-ID: On 3/17/2016 7:45 PM, Guido van Rossum wrote: > But then again I've never cared about citations. Citations are not just in Bibliographies. PEPs have them in footnotes, which was a traditional place for citations, though in the modern form of hyperlinks. The tracker is full of in-line citations, also in the form of hyperlinks. The formalized citation proposed by Stephen would include the URL. -- Terry Jan Reedy From gyromagnetic at gmail.com Fri Mar 18 10:09:11 2016 From: gyromagnetic at gmail.com (Gyro Funch) Date: Fri, 18 Mar 2016 08:09:11 -0600 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> Message-ID: On 3/18/2016 1:41 AM, Terry Reedy wrote: > On 3/17/2016 7:45 PM, Guido van Rossum wrote: > >> But then again I've never cared about citations. > > Citations are not just in Bibliographies. PEPs have them in > footnotes, which was a traditional place for citations, though > in the modern form of hyperlinks. The tracker is full of > in-line citations, also in the form of hyperlinks. The > formalized citation proposed by Stephen would include the URL. > Could 'citation' be an optional entry in setup.py instead? I cite software in papers I write, so having easy access to this information for each package I use in a given analysis would make this task easier. From gvanrossum at gmail.com Fri Mar 18 13:32:37 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Fri, 18 Mar 2016 10:32:37 -0700 Subject: [Python-ideas] PEP 484 evolution Message-ID: I have a few different things I'd like to add to PEP 484. Should I start this here or on pyrhon-dev? The hope is to get these into 3.5.2. When is that going out? --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From lkb.teichmann at gmail.com Fri Mar 18 14:11:29 2016 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Fri, 18 Mar 2016 19:11:29 +0100 Subject: [Python-ideas] Is multiple inheritance Pythonic? Message-ID: Hi List, Summary: Multiple inheritance is not used in new code of the standard library anymore, even where it could be of good use. Should multiple inheritance thus be abandoned, or supported more? I am currently developing a little library for the communication in our project. It is supposed to support different protocols, encryptions and addressing standards. At first I thought I'd just use multiple inheritance. A user of the library would then just write a little stub-class, as in: class MyProtocol(URLAdressMixin, SSLEncryptionMixin, HTTPProtocol): pass and voil?! everything is ready. But because I try to be one of the cool guys, I tried to do all of that using asyncio. It has a pluggable event loop, where you can plug in your own event loop with the specifics of your project. Again, I thought, let's to multiple inheritance! I needed some priority scheduling, so just write a mixin that does that, and mix it into the base class of the event loop. To my astonishment, the designers of asyncio explicitly decided against that option. They prefer not to let users subclass at all. Instead, asyncio has a factory plugin system, as an example, one can call set_task_factory to modify what the BaseEventLoop.create_task method is doing. Things like that used to be clearly the domain of mixin classes. The question arised, why would one do that? Is multiple inheritance bad, or not Pythonic? Guessing what the decisions of the asyncio developers were, several problems come to my mind. Firstly, metaclasses a notoriously problematic in a multiple inheritance context. Some of you might have seen my work on PEP 487 to mitigate that problem. Another issue are the stub classes mentioned above. One ends up writing lots of little stub classes combining mixins together. (An example from https://github.com/jupyter/qtconsole/blob/master/qtconsole/inprocess.py: class QtInProcessKernelManager(QtKernelManagerMixin, InProcessKernelManager):) Many users don't like that, it's too much programming, too little using. Another problem is that mixins cannot be mixed-in at runtime anymore. Once the event loop is running, I cannot sneak in a new task factory anymore. It is questionable whether that's desired. Another argument is that using the factory way, a library might "sneak in" its factories without the user noticing. But in my opinion, that's against the Pythonic concept of being explicit. A library should rather document: "Hey, you need to include SuperTaskMixin into your Eventloop", rather than silently setting a new factory. This also touches mainainability a lot. Imagine you are using two libraries, both at some time deciding they need to set a task factory. That won't work out. All of this is nicely solved in a multiple inheritance scheme: the method resolution order using super() is flexible enough to allow two independent extensions of create_task, as long as it is properly documented how to use it. So both concepts, multiple inheritance as well as pluggable factories, have their pros and cons. According to the Zen, there should ideally only be one way of doing things, but given that I'm not Dutch I just don't see what that way is. Please help me out. In my opinion, either a) multiple inheritance should be declared not Pythonic and frowned upon, or b) made easy to use and well supported In case of a), it would be nice to have proper facilities set up to have a good factory plugin mechanism (for example one that also allows to put in several plugins) Or in the b) case, it should be easier to use multiple inheritance and mixins. I was thinking about a class algebra, so following the above example one could create a mixed class simply as MyProtocol = SSLMixin + HTTPProtocol. This would lead do options (going back to asyncio) like asyncio.set_event_loop((PriorityMixin + QtEventLoop)()) What do you all think about that? Greetings Martin From mahmoud at hatnote.com Fri Mar 18 14:47:22 2016 From: mahmoud at hatnote.com (Mahmoud Hashemi) Date: Fri, 18 Mar 2016 11:47:22 -0700 Subject: [Python-ideas] Is multiple inheritance Pythonic? In-Reply-To: References: Message-ID: Discerning use of multiple inheritance is very Pythonic. Look no further than Exception use cases: class PathAccessError(KeyError, IndexError, TypeError): It might look a little wonky at first, but when you understand the use case, you'll see the utility. There are loads of these sorts of use cases sprinkled throughout the community, in some of the most Pythonic libraries around. Mahmoud On Fri, Mar 18, 2016 at 11:11 AM, Martin Teichmann wrote: > Hi List, > > > Summary: Multiple inheritance is not used in new code of the standard > library anymore, even where it could be of good use. Should multiple > inheritance thus be abandoned, or supported more? > > > I am currently developing a little library for the communication in our > project. It is supposed to support different protocols, encryptions and > addressing standards. > > At first I thought I'd just use multiple inheritance. A user of the library > would then just write a little stub-class, as in: > > class MyProtocol(URLAdressMixin, SSLEncryptionMixin, HTTPProtocol): > pass > > and voil?! everything is ready. > > But because I try to be one of the cool guys, I tried to do all of that > using > asyncio. It has a pluggable event loop, where you can plug in your own > event loop with the specifics of your project. Again, I thought, let's > to multiple > inheritance! I needed some priority scheduling, so just write a mixin that > does > that, and mix it into the base class of the event loop. > > To my astonishment, the designers of asyncio explicitly decided against > that > option. They prefer not to let users subclass at all. Instead, asyncio has > a factory plugin system, as an example, one can call set_task_factory > to modify what the BaseEventLoop.create_task method is doing. Things > like that used to be clearly the domain of mixin classes. > > The question arised, why would one do that? Is multiple inheritance bad, > or not Pythonic? > > Guessing what the decisions of the asyncio developers were, several > problems > come to my mind. Firstly, metaclasses a notoriously problematic in a > multiple > inheritance context. Some of you might have seen my work on PEP 487 to > mitigate that problem. > > Another issue are the stub classes mentioned above. > One ends up writing lots of little stub classes combining mixins > together. (An example from > https://github.com/jupyter/qtconsole/blob/master/qtconsole/inprocess.py: > class QtInProcessKernelManager(QtKernelManagerMixin, > InProcessKernelManager):) > Many users don't like that, it's too much programming, too little using. > > Another problem is that mixins cannot be mixed-in at runtime anymore. > Once the event loop is running, I cannot sneak in a new task factory > anymore. > It is questionable whether that's desired. Another argument is that > using the factory way, a library might "sneak in" its factories without the > user noticing. But in my opinion, that's against the Pythonic concept of > being explicit. A library should rather document: "Hey, you need to include > SuperTaskMixin into your Eventloop", rather than silently setting a new > factory. > > This also touches mainainability a lot. Imagine you are using two > libraries, > both at some time deciding they need to set a task factory. That won't work > out. > > All of this is nicely solved in a multiple inheritance scheme: the > method resolution > order using super() is flexible enough to allow two independent extensions > of > create_task, as long as it is properly documented how to use it. > > So both concepts, multiple inheritance as well as pluggable factories, have > their pros and cons. According to the Zen, there should ideally only be > one way of doing things, but given that I'm not Dutch I just don't see what > that way is. Please help me out. > > In my opinion, either > > a) multiple inheritance should be declared not Pythonic and frowned upon, > or > b) made easy to use and well supported > > In case of a), it would be nice to have proper facilities set up to have a > good > factory plugin mechanism (for example one that also allows to put in > several plugins) > > Or in the b) case, it should be easier to use multiple inheritance and > mixins. I was thinking about a class algebra, so following the above > example one could create a mixed class simply as > MyProtocol = SSLMixin + HTTPProtocol. This would lead do options > (going back to asyncio) like > > asyncio.set_event_loop((PriorityMixin + QtEventLoop)()) > > What do you all think about that? > > Greetings > > Martin > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Mar 18 14:57:13 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 18 Mar 2016 14:57:13 -0400 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: Message-ID: On 3/18/2016 1:32 PM, Guido van Rossum wrote: > I have a few different things I'd like to add to PEP 484. Should I start > this here or on pyrhon-dev? I would say that it depends on whether the ideas need to be kicked around a bit or whether you think they are in final form. > The hope is to get these into 3.5.2. When is > that going out? Last I knew, it was scheduled for next May, but https://www.python.org/dev/peps/pep-0478/ does not say anything. -- Terry Jan Reedy From guido at python.org Fri Mar 18 15:09:52 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 18 Mar 2016 12:09:52 -0700 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: Message-ID: It's a mix. The list of things I definitely want to get to is now here: https://github.com/python/typing/milestones/3.5.2 On Fri, Mar 18, 2016 at 11:57 AM, Terry Reedy wrote: > On 3/18/2016 1:32 PM, Guido van Rossum wrote: >> >> I have a few different things I'd like to add to PEP 484. Should I start >> this here or on pyrhon-dev? > > > I would say that it depends on whether the ideas need to be kicked around a > bit or whether you think they are in final form. > >> The hope is to get these into 3.5.2. When is >> >> that going out? > > > Last I knew, it was scheduled for next May, but > https://www.python.org/dev/peps/pep-0478/ > does not say anything. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From rantingrickjohnson at gmail.com Fri Mar 18 15:46:29 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Fri, 18 Mar 2016 12:46:29 -0700 (PDT) Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: Message-ID: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> On Friday, March 18, 2016 at 2:11:00 PM UTC-5, Guido van Rossum wrote: > > It's a mix. > IMHO, I'd venture to say that any proposal that is *NOT* already explicitly scheduled for implementation, or any parts thereof that "you consider debatable", should be discussed here. Python-dev seems more appropriate for things that are either "already in the language", or "will be in the language no matter what anyone says or does otherwise". -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Mar 18 15:51:06 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 18 Mar 2016 12:51:06 -0700 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> Message-ID: So please read my list of issues and see where you stand yourself on them. https://github.com/python/typing/milestones/3.5.2 On Fri, Mar 18, 2016 at 12:46 PM, Rick Johnson wrote: > > > On Friday, March 18, 2016 at 2:11:00 PM UTC-5, Guido van Rossum wrote: >> >> It's a mix. > > > IMHO, I'd venture to say that any proposal that is *NOT* already explicitly > scheduled for implementation, or any parts thereof that "you consider > debatable", should be discussed here. Python-dev seems more appropriate for > things that are either "already in the language", or "will be in the > language no matter what anyone says or does otherwise". -- --Guido van Rossum (python.org/~guido) From sjoerdjob at sjec.nl Fri Mar 18 15:47:05 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Fri, 18 Mar 2016 20:47:05 +0100 Subject: [Python-ideas] Unified database exception structure Message-ID: <20160318194705.GA27028@sjoerdjob.com> Dear all, I'm not sure if this has been discussed before, but I have a bit of a concern with respect to the exceptions defined in the DB-API 2.0 (PEP 0249), especially with respect to code written to support multiple database backends (for instance, through an ORM, or manually). >From what I understand, the PEP requires the module adhering to the database API to specify its own exceptions with pre-determined names, for instance: >>> psycopg2.DatabaseError.__mro__ (, , , , ) >>> sqlite3.DatabaseError.__mro__ (, , , , ) Now, there is not a single base class (except for `Exception`) which I should feel comfortable about catching. I think it would have been better to see something like >>> psycopg2.DatabaseError.__mro__ (, , , , ) >>> sqlite3.DatabaseError.__mro__ (, , , , ) Of course, this is too late to change now, and maybe I'm alone in thinking this would make more sense, and think the best we can have is to have `sqlite3.DatabaseError` (and friends) be formed by extending both `dbapi.DatabaseError` and `sqlite3.Error` (which in turn would extend `dbapi.Error`). To make that happen, I think it would be good to see one of the following situations happen: - A module cq package gets created holding the generic exception classes, and slowly (but surely) all the independent database libraries start to use those classes in their __mro__ as well. The sqlite3 library could be the first or the last, because it is in the stdlib, depending on whether the new module/package also gets to be in the stdlib. - We wait for Python 3.6 or higher to allow using virtual base classes in `except ...:` statements (Python 2.7 already allows it). This way we could create a package which just registers the relevant database libraries exception tree with its own. The downside to going with the first route is that I envision it a slow route, which means it will also be a long time before I can use it in code which uses libraries which have not been updated yet. The upside with the second route is that there is already work being done on supporting `except VirtualBaseClass:` (see https://bugs.python.org/issue12029). Also, it means that it should be possible to just write 1 package to rule them all. Another advantage is that that solution would be backwards compatible with Python 2.7. The downside is that it's not compatible with Python 3.5. Do you have any thoughts about this? Kind regards, Sjoerd Job From chris.barker at noaa.gov Fri Mar 18 16:35:06 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 18 Mar 2016 13:35:06 -0700 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> <56E966DC.1050808@gmail.com> Message-ID: On Wed, Mar 16, 2016 at 10:32 AM, Paul Moore wrote: > On 16 March 2016 at 17:17, Chris Barker wrote: > > As in: > > > > If someone installs a version of python with defaults, they should then > get > > that version when they type "python" (or maybe "py") at the command line. > > > > If they want something other than the last one they installed, then > they'll > > have to mess with PATH or other configuration... > > In theory, this makes sense to me. What it needs is someone to > champion it. Isn't that what this conversation is doing? :-) > 1. What if I want to install a version *not* for day to day use but > just for testing (i.e., there needs to be an easily accessible "leave > PATH alone" option). > yes, there should be a "leave PATH alone" checkbox, that is unchecked by default. If you want to get really fancy, you could require a question be asked of the user if there is already a python there. I have no idea if it's had to do this in a regular installer. > 2. How does this interact with uninstalls and upgrades? If I install > 3.7, then 3.8 then uninstall 3.7, then install 3.6, then upgrade 3.8 > to 3.8.1, what's left on PATH? (The answer I *want* is 3.8.1. That may > well be a nightmare to implement). wait, I'm confused -- if the last thing you do is upgrade to 3.8.1, then that will be in the PATH by default, but what if you: install 3.7, then install 3.8, then uninstall 3.8 -- would 3.7 be left on the PATH? That could be pretty darn tricky -- though, again, if you undestand PATH manipulations, you can fix it, and if you don't you can go back and re-install 3.7, just to get the PATH set. It's easy enough to dismiss point (2) as "well, you know what you're > doing", but I have experience of people doing a series of installs > like this, getting confused with PATH and doing a google search to > find out how to "fix" it. So it is a genuine situation a non-expert > user could find themselves in. > yup but if their google search turns up: "either hand-manipulate the PATH to fix this, or simply re-install the one you want" then we're in fine shape. It's not rare for "re-install the software" to be the easiest solution to messed up configuration. > After all, what do people expect when they have MSWord version X, and > then install MSWord version X+1 > > But nobody *ever* runs multiple versions of MS Word. Whereas there are > genuine reasons for people (and not just experts, although admittedly > not complete beginners) to run multiple versions of Python. The MS > Word argument implies that the Python installer should automatically > and silently uninstall any older versions of Python present on the > user's machine. no -- but it wouldn't be so bad if it did *appear* to have done that. The naive user will not be surprised -- it's common behavior -- and just reinstall the old one, even if all it does is re-set the PATH. The more sophisticated user will know to go in and mess with PATH. In short, if we overly simplistically break the world down inot people that "get" PATH, and those that don't: The default jsut click through witout reading it intstall should do the most expected thing for don't-get-PATH folks -- i.e. they get access to the mosst recent version installed. It should be possible for do-get-PATH folks to customize the system (ideally without having to get the registry, too!) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rantingrickjohnson at gmail.com Fri Mar 18 16:40:17 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Fri, 18 Mar 2016 13:40:17 -0700 (PDT) Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> Message-ID: <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> On Friday, March 18, 2016 at 2:51:26 PM UTC-5, Guido van Rossum wrote: > So please read my list of issues and see where you stand > yourself on them. Are you kidding? I'd be delighted to offer my opinion! It's what i, *DO*! I'm not sure what kind of "time frame" you're expecting here, but you'll have to allow me a "reasonable" amount of time to (1) read over your list, (2) catch up on the history of the "type hints proposal", and (3) compose a thoughtful response -- I've never been the type that likes to rush to judgement, or shoot from hip, especially, regarding something as important as introducing a new feature that could negativity affect readability. Also, i must admit that, outside of a few discussions on Pyhton-list, I have not followed the "type hints evolution" as closely as, well, perhaps i should have. In fact, to tell you the truth, (psst: this is just between you and me!) i kinda thought it would be punted around for awhile, and then return to obscurity... But since this proposal seems to be moving "full steam ahead", and i consider myself to be an integral part of this fine community, then, as a member, it is my *DUTY* to be wise of these things. Please stand by... -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Mar 18 16:44:13 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 18 Mar 2016 21:44:13 +0100 Subject: [Python-ideas] Is multiple inheritance Pythonic? In-Reply-To: References: Message-ID: <56EC689D.7080504@egenix.com> On 18.03.2016 19:11, Martin Teichmann wrote: > Is multiple inheritance bad, or not Pythonic? Definitely not. But at the same time, it's not always the answer to everything either. Preventing subclassing of code is usually a conscious design decision in Python and has it's purpose as well, e.g. to reduce performance overhead by not having to worry about corner cases which the base code does not address. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Mar 18 2016) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ 2016-03-07: Released eGenix pyOpenSSL 0.13.14 ... http://egenix.com/go89 2016-02-19: Released eGenix PyRun 2.1.2 ... http://egenix.com/go88 ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From guido at python.org Fri Mar 18 16:49:42 2016 From: guido at python.org (Guido van Rossum) Date: Fri, 18 Mar 2016 13:49:42 -0700 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> Message-ID: "OK" On Fri, Mar 18, 2016 at 1:40 PM, Rick Johnson wrote: > On Friday, March 18, 2016 at 2:51:26 PM UTC-5, Guido van Rossum wrote: >> So please read my list of issues and see where you stand >> yourself on them. > > Are you kidding? > > I'd be delighted to offer my opinion! > > It's what i, *DO*! > > I'm not sure what kind of "time frame" you're expecting > here, but you'll have to allow me a "reasonable" amount of > time to (1) read over your list, (2) catch up on the history > of the "type hints proposal", and (3) compose a thoughtful > response -- I've never been the type that likes to rush to > judgement, or shoot from hip, especially, regarding > something as important as introducing a new feature that > could negativity affect readability. > > Also, i must admit that, outside of a few discussions on > Pyhton-list, I have not followed the "type hints evolution" > as closely as, well, perhaps i should have. In fact, to tell > you the truth, (psst: this is just between you and me!) i > kinda thought it would be punted around for awhile, and then > return to obscurity... > > But since this proposal seems to be moving "full steam > ahead", and i consider myself to be an integral part of this > fine community, then, as a member, it is my *DUTY* to be > wise of these things. > > Please stand by... > -- --Guido van Rossum (python.org/~guido) From abarnert at yahoo.com Fri Mar 18 16:59:53 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 18 Mar 2016 13:59:53 -0700 Subject: [Python-ideas] Is multiple inheritance Pythonic? In-Reply-To: References: Message-ID: <4ECFE5A1-71C0-4FEC-93AC-58498AD6D883@yahoo.com> On Mar 18, 2016, at 11:11, Martin Teichmann wrote: > > Or in the b) case, it should be easier to use multiple inheritance and > mixins. I was thinking about a class algebra, so following the above > example one could create a mixed class simply as > MyProtocol = SSLMixin + HTTPProtocol. That means you now have an anonymous class[1]--no name to show up in tracebacks or inspect, no way to jump to the class definition in IDEs, probably even no way to pickle instances unless you invent some new mechanism to make that work. Also, this conflicts with the notion of type algebra from other languages: the sum of two or more types is like a tagged union.[2] Also, it's not much of an algebra if the only operation is (non-commutative) addition. What does MyMixin * YourMixin, or 2 * MyMixin mean?[3] Also, + implies commutativity. That implication only goes so far (it's not true from string concatenation, obviously), but it's still going to flavor people's expectations. And finally, what's the benefit over this: class MyProtocol(SSLMixin, HTTPProtocol): pass Of course if you really want to do this, you can already just call the type function directly (although notice that even there, and even in the C API, you still have to pass some string in for the name...). Or, if you really want, create a metaclass that subclasses type and adds an __add__ method. (You could even write it as a "metaclass mixin"). But I think very little Pythonic code would want such a thing, so it's a bad idea to make it any easier than it is today. From a larger point of view, I don't think there's any problem to be solved here. Your question is like asking whether we should get rid of (single) inheritance or delegation. They're both useful, in mostly different contexts--there's some overlap, and there are also cases where you should be using one but could instead abuse the other, but that doesn't mean either one is bad and needs to be declared unpythonic. It means that both are pythonic within their range and unpythonic outside it (and when your use falls into the overlap, you have to make a judgment call). --- [1]: If you're thinking "but anonymous classes are good in Java", they're needed in Java to work around a problem Python has never had (how do you do closures in a language with no free functions?), and they're acceptable because nobody looks at the classes at runtime. [2]: That is, an object that can act as either a MyMixin or a YourMixin (after you check which) but never both at the same time. [3]: In other languages, these operations create product types--tuples or records. From chris.barker at noaa.gov Fri Mar 18 17:29:37 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 18 Mar 2016 14:29:37 -0700 Subject: [Python-ideas] Importing public symbols and simultainiously privatizing them, is too noisy In-Reply-To: <007b7330-9b57-42a6-b1df-5af2029db1fb@googlegroups.com> References: <73a65a22-6440-4fde-ba99-fcd864a652d0@googlegroups.com> <20160317003545.GD8022@ando.pearwood.info> <007b7330-9b57-42a6-b1df-5af2029db1fb@googlegroups.com> Message-ID: On Wed, Mar 16, 2016 at 6:52 PM, Rick Johnson wrote: > > Besides, why is "import x as _x" so special to require special syntax? > It's not :-) I know I do, for instance, from matplotlib import pylot as plt But have NEVER done the leading underscore thing... > from module import Foo as _Foo, bar as _bar, BAZ as _BAZ, spam as _spam, > eggs as _eggs > if you are mirroring an entire namespace, or a god fraction of one then use a module name! import module as _mod then use _mod.Foo, etc..... Now, that may seem like a contrived example, but i've > witnessed much longer "run-on import lines" than that. > I have too, but I think it's bad style -- if you are importing a LOT of names from one module, just import the darn module -- giving it a shorter name if you like. This has become a really standard practice, like: import numpy as np for instance. The intended purpose is to: "automate the privatization of > public symbols during the import process". > I'm really confused about the use case for "privatization of public symbols" at all, but again, if you need a lot of them, use the module name to prefix them. Heck give it a one character name, and then it's hardly more typing than the underscore... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Mar 18 17:38:35 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 18 Mar 2016 21:38:35 +0000 Subject: [Python-ideas] Make py.exe default to Python 3 In-Reply-To: References: <3A4D4663-070B-4632-B697-F0F8FCEED776@yahoo.com> <56E83D59.5080707@gmail.com> <3593462996689522105@unknownmsgid> <56E966DC.1050808@gmail.com> Message-ID: On 18 March 2016 at 20:35, Chris Barker wrote: >> In theory, this makes sense to me. What it needs is someone to >> champion it. > > Isn't that what this conversation is doing? :-) Not really. By "champion" I mean "take the lead" - i.e., pull together all the arguments, summarise them, produce a proposal, write a patch (or get someone else to write it) create the tracker item, get a core dev to support it and shepherd it through to being committed. At the moment, all we're doing is discussing possibilities, nobody's stepped up to take the lead. I get the feeling that some participants in the discussion might think that discussion is enough, but I've seen too many of these threads fizzle out because no-one is interested enough, or has enough time, to do the hard bits. I just wanted to be clear that I started this thread for a *different* proposal, which came to a conclusion that I plan (other commitments permitting) to get implemented. But I'm *not* sufficiently interested in this new proposal to take it forward, so someone else will need to step up at some point if the idea is to go anywhere. Paul From tjreedy at udel.edu Fri Mar 18 17:53:11 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 18 Mar 2016 17:53:11 -0400 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: Message-ID: On 3/18/2016 3:09 PM, Guido van Rossum wrote: > It's a mix. The list of things I definitely want to get to is now > here: https://github.com/python/typing/milestones/3.5.2 #2: backport aab2c59 to python2/typing.py I believe you are referring to a back-compatibility module on PyPI. If so, this is out-of-scope for python-ideas. The only connection with 3.5.2 is that 2.7.12 should come out about the same time. #3, #1: many parameters require long comment signature? These are a question and possible solution for the type comments intended for 2.7 (though allowed, I believe, in 3.5 and earlier with the back-compatibility module). The issue: some functions have so many parameters that a type comment will not fit on one line. A proposal discussed here before is multiline type comments. The alternate solution is to allow '...' as an abbreviation. My very slight objection is that this will mean that the type comment no longer matches the function signature above. The counter-argument is that duplicating a long parameter type list, especially when one only cares about typing the return type, is obnoxious. I presume this proposal is based on problems with real code. + something. I do not understand, in #3, "Would it be valid to annotate just a few argument [and use ... also, I presume]? and "Yes". What would it look like? Is that part of #1? #4 Ambiguity of "Z = Dict[T, int]". I don't understand enough to understand the 2nd interpretation, so 'pass'. Ambiguity is bad, though. #5 A proposal to allow 'Type[T]'? Where "T = TypeVar['T']"? Just unpostponed from last May. #6 "What does it mean to inherit from a generic type without specifying type parameters?" I am not clear on what patch is being proposed. This seems to still be in ideas stage. #7 Allow @overload in implementation as well as stub files. You originally rejected this, then reopened in January with discussion here and pydev. Is more discussion needed? Final votes on pydev? -- Terry Jan Reedy From lkb.teichmann at gmail.com Fri Mar 18 18:34:43 2016 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Fri, 18 Mar 2016 23:34:43 +0100 Subject: [Python-ideas] Is multiple inheritance Pythonic? In-Reply-To: <56EC689D.7080504@egenix.com> References: <56EC689D.7080504@egenix.com> Message-ID: Hi, >> Is multiple inheritance bad, or not Pythonic? > > Definitely not. > > But at the same time, it's not always the answer to everything > either. Actually, I agree with you. But then I found the discussion about asyncio.BaseEventLoop.create_task (especially: https://github.com/python/asyncio/issues/208 and https://codereview.appspot.com/110820049/ ), which is now in the standard library, probably the most Pythonic code out there. During this discussion, all participants argue as if class inheritance was something EVIL. And they were not just random people, but in my opinion some of the finest Python programmers out there. They desperately try to avoid letting users inherit classes, even in cases where it seemed obvious to me that inheritance is the right choice - but well, who am I to decide that. So I asked myself, and hereby the Python community, is there something so bad about inheritance that I missed? The problem they wanted to solve is that the behavior of BaseEventLoop.create_task should be changeable. Obvious, I thought (and did in my code) inherit from the class, overwrite the method! But what if I want to overwrite that method in something that inherits from BaseEventLoop, say, some QtEventLoop? No problem, I thought, just supply a Mixin, and a simple class EventLoop(AMixin, QtEventLoop) does the job. This is even chainable, multiple mixins can be used as long as they properly call super(). Instead, the developers decided to write a set_task_factory method, which overwrites create_task. It has none of the above mentioned advantages. That's where I started thinking, and I am still puzzled what obstacles might be out there, and decided that inheritance must be something magically non Pythonic, and I am still wondering why that might be. Greetings Martin From rantingrickjohnson at gmail.com Fri Mar 18 18:51:36 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Fri, 18 Mar 2016 15:51:36 -0700 (PDT) Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> Message-ID: <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> @Everyone: Okay i'm completely confused as to were we are in the "type hints" evolution. It seems to have taken all sorts of random twists and turns -- will someone please explain? (see examples below) @Guido: i've skimmed PEP484, and i've read your handful of "issues", but i need to know *EXACTLY* what a type hint declaration will look like *BEFORE* i can offer my opinion on these matters. At this point, i'm not overly concerned about the dirty details of how type hints will be implemented "under the hood", my first concern is, to understand what *onerous* this feature will be placing on the programmer, and, what *damage* it will could cause to the "general readability" of Python source code. In other words -- i need to decide if this is a good idea or not. In pep484, the declaration seems to be firmly coupled/interleaved with the source code: def greeting(name: str) -> str: return 'Hello ' + name Oh my. But if so, i would hardly call this "a hint". Are we going to introduce `->` as a method for defining the return type of a function, and special-case `:` for binding types to symbols in sigs? But the rabbit hole goes deeper... After reading PEP484, i'm looking over your list of issues, and i see you using "magic comments" like this: def whatever( arg1, # type: int arg2, # type: int arg3, # type: int arg4, # type: str arg5, # type: str arg6, # type: str ): # type: (...) -> float # this is not a type comment (note that "..." above is literal) return (arg1 + arg2 + arg3 + float(arg4) + float(arg5) + float(arg6)) So which is it? The former? The latter? Or both? PS: As i await your response, I'm praying to the FSM that type hints will be implemented as the latter, and *NOT* the former. And i'm *SO* fearful of the thought of the former, that i'm *literally* shaking as i write this response-- *PLEASE* tell me it's the latter. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Fri Mar 18 18:59:14 2016 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 18 Mar 2016 17:59:14 -0500 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> Message-ID: -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ On Mar 18, 2016 5:51 PM, "Rick Johnson" wrote: > > > @Everyone: Okay i'm completely confused as to were we are in > the "type hints" evolution. It seems to have taken all sorts > of random twists and turns -- will someone please explain? > > (see examples below) > > @Guido: i've skimmed PEP484, and i've read your handful of > "issues", but i need to know *EXACTLY* what a type hint > declaration will look like *BEFORE* i can offer my opinion > on these matters. At this point, i'm not overly concerned > about the dirty details of how type hints will be > implemented "under the hood", my first concern is, to > understand what *onerous* this feature will be placing on > the programmer, and, what *damage* it will could cause to > the "general readability" of Python source code. In other > words -- i need to decide if this is a good idea or not. > > In pep484, the declaration seems to be firmly > coupled/interleaved with the source code: > > def greeting(name: str) -> str: > return 'Hello ' + name > > Oh my. But if so, i would hardly call this "a hint". Are we > going to introduce `->` as a method for defining the return > type of a function, and special-case `:` for binding types > to symbols in sigs? > Neither! Python 3 type hints are just function annotations. It's the job of a type checker/linter (like mypy) to actually decipher the annotations as types. Adding/removing type hints won't have a runtime effect on your program. > But the rabbit hole goes deeper... > > After reading PEP484, i'm looking over your list of issues, > and i see you using "magic comments" like this: > > def whatever( > arg1, # type: int > arg2, # type: int > arg3, # type: int > arg4, # type: str > arg5, # type: str > arg6, # type: str > ): > # type: (...) -> float > # this is not a type comment (note that "..." above is literal) > return (arg1 + arg2 + arg3 + > float(arg4) + float(arg5) + float(arg6)) > > So which is it? The former? The latter? Or both? > I think the latter is for Python 2. > PS: As i await your response, I'm praying to the FSM that > type hints will be implemented as the latter, and *NOT* the > former. And i'm *SO* fearful of the thought of the former, > that i'm *literally* shaking as i write this response-- > *PLEASE* tell me it's the latter. > PEP 484 already was passed using the former syntax...so you might be a little late saying that... > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Mar 18 19:06:29 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 18 Mar 2016 16:06:29 -0700 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> Message-ID: <56EC89F5.5000904@stoneleaf.us> On 03/18/2016 03:51 PM, Rick Johnson wrote: [snippety snip snip] For those who have not yet encountered "Ranting Rick Johnson": he is a troll from Python List; please do not waste your time with him. -- ~Ethan~ From p.f.moore at gmail.com Fri Mar 18 19:07:58 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 18 Mar 2016 23:07:58 +0000 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> Message-ID: On 18 March 2016 at 22:51, Rick Johnson wrote: > In pep484, the declaration seems to be firmly > coupled/interleaved with the source code: > > def greeting(name: str) -> str: > return 'Hello ' + name > > Oh my. But if so, i would hardly call this "a hint". Are we > going to introduce `->` as a method for defining the return > type of a function, and special-case `:` for binding types > to symbols in sigs? You do realise that PEP 484 was implemented in Python 3.5, don't you? See https://docs.python.org/3/library/typing.html The question here is around some additions (which I don't understand too well, not being a user of type hints) that Guido is looking to add in 3.5.2. Paul From abarnert at yahoo.com Fri Mar 18 19:50:49 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 18 Mar 2016 16:50:49 -0700 Subject: [Python-ideas] Is multiple inheritance Pythonic? In-Reply-To: References: <56EC689D.7080504@egenix.com> Message-ID: <4267AB45-713C-4654-B8CC-51DD4A144A00@yahoo.com> On Mar 18, 2016, at 15:34, Martin Teichmann wrote: > > Hi, > >>> Is multiple inheritance bad, or not Pythonic? >> >> Definitely not. >> >> But at the same time, it's not always the answer to everything >> either. > > Actually, I agree with you. But then I found the discussion about > asyncio.BaseEventLoop.create_task (especially: > https://github.com/python/asyncio/issues/208 and > https://codereview.appspot.com/110820049/ ), > which is now in the standard library, probably the most Pythonic > code out there. What makes asyncio the most Pythonic code out there, as opposed to other stdlib modules like enum, selectors, etc.?[1] > During this discussion, all participants argue as if class inheritance > was something EVIL. You're misinterpreting them. They believe that making people inherit from API classes and override internal methods is not the best way to allow customizing those APIs. That one particular use of inheritance is almost always a bad one. But that doesn't at all mean that all possible uses of inheritance are bad. For example, nobody suggested that it was evil for IntEnum to subclass both int and Enum--in fact, Enum was specifically designed (and debugged) to make that as easy as possible, and to allow IntEnum to serve as an example for user enum types. And a lot of the same people were involved in that discussion. So, what _are_ the reasons inheritance isn't the best way to handle this kind of customization? The two most famous are: 1. With subclassing-for-specialization, it's rarely clear--even in a language like C++ where the required protected and virtual act as markers, much less in Python--which methods you are expected to override.[2] With a set_spam_cheeser interface, it's obvious what they expected you to customize: just the spam-cheeser. 2. Who constructs the object? Often it's the framework itself, which means if the only way for you to customize framework.MeatCooker.spam_cheeser is to subclass, then you also need some way to tell framework.Kitchen to create an app.MyMeatCooker instead of a framework.MeatCooker. Which means you need a framework.KitchenFactory that you can tell to create an app.MyKitchen... Other times, the app creates the object, synthesizing customizations from various different third-party libs, but inheritance can cause its own problems there.[4] The general trend here is easier to see (and therefore so are the motivations behind it) in the brace-family languages. Unlike Python, which has always[5] treated classes and methods and everything else as first-class objects, they have to invent a whole new language feature (first interfaces, then delegates/method-pointers, then event-handlers) at each step, which usually means a whole new language and stdlib. Fortunately, when people figure out a better way of doing things in Python, we can just gradually start using the better way in place of the old way. --- [1]: In fact, I suspect Guido would say that there are many ways in which asyncio is a special case, which is part of the reason he had to write a relatively complex stdlib module in the first place (rather than the usual pattern of letting different alternatives compete and then maybe considering adding a simplified version of the winner to the stdlib years later). [2]: And often, if you override methods the class designer didn't expect, it works in some cases but not others, so you design around it and then realize it doesn't work and isn't supposed to work, and then you end up having to override three other methods and copy and paste all but one line and then subclass an associated class and override two of its methods to fix the bug. [3]: ... which may be an interface with multiple methods, but most simply it's just one function. [4]: Admittedly, those problems aren't as bad in Python as in brace-family language--it just means you have to define and document a way for all those libraries and apps to cooperate properly, and write your code carefully so that bad cooperation can be debugged, which isn't all that hard. But it's still a lot harder than not doing it, and you don't have to do any of that with separate customization hooks. [5]: OK, there were some limitations before 2.2, but I think we can ignore those. From lkb.teichmann at gmail.com Fri Mar 18 20:47:53 2016 From: lkb.teichmann at gmail.com (Martin Teichmann) Date: Sat, 19 Mar 2016 01:47:53 +0100 Subject: [Python-ideas] Is multiple inheritance Pythonic? In-Reply-To: <4267AB45-713C-4654-B8CC-51DD4A144A00@yahoo.com> References: <56EC689D.7080504@egenix.com> <4267AB45-713C-4654-B8CC-51DD4A144A00@yahoo.com> Message-ID: Hi Andrew, Hi List, > So, what _are_ the reasons inheritance isn't the best way to handle this kind of customization? The two most famous are: thanks, that was a interesting read. Hopefully not just for me but also for others. I still have to let that sink in: inheritance, even multiple is fine, just not across a framework boundary, because you never know how the user is overriding the methods. For me, coming from a GUI background (Qt, that is) that's still astonishing, as there inherit-and-override is a concept used very often, especially across the framework boundary. > 1. With subclassing-for-specialization, it's rarely clear--even in a language like C++ where the required protected and virtual act as markers, much less in Python--which methods you are expected to override. In the GUI world, that's simply solved by documentation. Qt, and also PyQt, document very precisely which methods are supposed to be overridden, and how. And I still like that concept, as it has a lot of flexibility. You may call super(), or not, depending on what you want to do, even overriding a method several times in the inhertance chain is not uncommon, even across framework boundaries. With set_task_factory, I'm still trying to figure out how to call the base implementation, as super() won't do it. (Thats another topic, but if someone could tell me I would be really happy. I simply want to keep track of all created tasks, so I would simply love to call super().create_task(), log the created task and I'm done. No I cannot use Task(), as I don't know whether the EventLoop actually uses that.) > The general trend here is easier to see (and therefore so are the motivations behind it) in the brace-family languages. Unlike Python, which has always[5] treated classes and methods and everything else as first-class objects, they have to invent a whole new language feature (first interfaces, then delegates/method-pointers, then event-handlers) at each step, which usually means a whole new language and stdlib. Fortunately, when people figure out a better way of doing things in Python, we can just gradually start using the better way in place of the old way. What do you mean with the last sentence, what is that better way? Having a set_whatever_factory instead of inheritance, is that what you mean? Thanks for the enlightening post Martin From andrey.vlasovskikh at gmail.com Fri Mar 18 21:45:25 2016 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Sat, 19 Mar 2016 04:45:25 +0300 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code Message-ID: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> With the addition of the comment-based syntax [3] for Python 2.7 + 3 in PEP 0484 having a Python 2/3 compatible way of adding type hints for text and binary values becomes important. Following the issue #1141 at the Mypy GitHub site [1], I've came up with a draft proposal based on those ideas that I'd like to discuss here. # Abstract This proposal contains recommendations on how to annotate text/binary data in newly added PEP 0484 comment-based type hints in order to make them Python 2/3 compatible when the single-source approach to porting from 2 to 3 is used. It introduces a new type `typing.Text` that represents text data in both Python 2 and 3, deprecates `str -> unicode` promotion used in type checkers, suggests an approach for type checkers to find implicit conversion errors by tracking ASCII text/binary values, recommends that type checkers should warn about `unicode` in the 2+3 mode. # Rationale With the addition of the comment-based syntax for Python 2.7 + 3 having a Python 2/3 compatible way of annotating types of text and binary values becomes important. Currently having a single-source code base is the main approach to 2/3 compatibility, so it is highly desirable to have 2/3 compatible comment-based type hints that would help porting code from 2 to 2+3 to 3. While migrating their code from Python 2 to 3 users are most likely to discover the following types of text/binary errors (presumably, in the descending order of their frequency in typical code): 1. Implicit text/binary conversions removed in Python 3 2. Calling changed APIs that accept or return text/binary data 3. Calling removed/changed methods of text/binary types 4. Overriding special text/binary methods and using the related built-ins (`str()`, `repr()`, `unicode()`) Only the first two types of errors -- implicit conversions and calling changed text/binary APIs -- depend on being able to express the semantics of Python 2+3 compatible text/binary interfaces using type hints. PEP 0484 doesn't contain any recommendations on how to document various typical cases in text/binary APIs in order to make type hints 2+3 compatible. # Proposal This document is based on some text/binary handling options and the problems associated with them propsed at python/mypy#1141 by Jukka Lehtosalo, Guido van Rossum, and others [1]. It also takes into account the experience of the PyCharm team with their pre-PEP484 notation for type hints [2] and handling Python 2/3 issues reported by users in PyCharm code inspections. ## Handling removed implicit conversions In addition to the existing types (`bytes`, `str`, `unicode`, `typing.AnyStr`) let's introduce a new type for *2+3 compatible text data* -- `typing.Text` (should we add a fake built-in `unicode` type for Python 3 to type checkers instead of introducing a new name?): * `typing.Text`: text data * Python 2: `unicode` * Python 3: `str` Just to remind the semantics of the existing types: * `bytes`: binary data * Python 2: `bytes` (== `str`) * Python 3: `bytes` * `str`: "native" string, `type('foo')` * Python 2: `str` * Python 3: `str` * `unicode`: Python 2-only text data * Python 2: `unicode` * Python 3: error * `typing.AnyStr`: type variable constrained to both text and binary data With the addition of `typing.Text` it is possible to express the type analogous to `typing.AnyStr` that doesn't impose any type constraints (should we call it `typing.BaseString`?): * `typing.Union[typing.Text, bytes]`: both text and binary data when a type varibale isn't needed Using only `typing.Text`, `bytes`, `str`, and `typing.AnyStr` in the type hints for an API would mean that this API is Python 2 and 3 compatible in respect to implicit text/binary conversions. For Python 2 we should *not* have the implicit `str` -> `unicode` promotion since it hides errors related to implicit conversions. For 7-bit ASCII string literals in Python 2 type checkers should infer special internal types `typing._AsciiStr` and `typing._AsciiUnicode` that are compatible with both `str` and `unicode` (a *special type-checking rule* is needed): class _AsciiStr(str): pass class _AsciiUnicode(unicode): pass The details of inferring ASCII types are up to specific type checkers. In the 2+3 mode type checkers should show errors when comment- or stub- based type hints contain `unicode`. ## Examples of typical 2+3 functions A function that accepts "native" strings. It uses implicit ASCII unicode-to-str conversion at runtime in Python 2 and accepts only text data in Python 3: def getattr(o: Any, name: str, default: Any = None) -> Any: ... A function that does implicit str-to-unicode conversion at runtime in Python 2 and accepts only text data in Python 3: def hello_rus(name: Text) -> Text: return u'??????, ' + name A function that transforms text-to-text or binary-to-binary or handles both text and binary data in some other way in both Python 2 and 3: def listdir(path: AnyStr) -> AnyStr: ... A function that works with both text and binary data in Python 2 and 3, where the author of the function some reason doens't want to have a type variable associated with `AnyStr`: def upper_len(s: Union[bytes, Text]) -> int: return len(s.upper()) A PEP-3333 compatible WSGI app function that uses "native" strings for environ and headers data while returning an iterable over binary data in both Python 2 and 3: def app(environ: Dict[str, Any], start_response: Callable[[str, List[Tuple[str, str]]], None]) \ -> Iterable[bytes]: ... A type inference example that features a type checker being able to infer `typing._AsciiStr` or `typing._AsciiUnicode` types for Python 2 using the functions defined above: method_name = u'update' # _AsciiUnicode getattr({}, method_name) # OK, implicit ASCII-only unicode-to-bytes in Py2 nonascii_data = b'\xff' # _AsciiStr hello_rus(nonascii_data) # Type checker warning # Non-ASCII bytes are not compatible with Text # _AsciiUnicode + bytes u'foo' + b'\xff' # Type checker warning # Non-ASCII bytes are not compatible with Text def f(x: AnyStr, y: AnyStr) -> AnyStr: return os.path.join('base', x, y) # _AsciiStr compatible with AnyStr # since it's compatible with # both str and unicode There are cases mentioned in [1] where more advanced type inference rules are required in order to be able to handle ASCII types. It remains unclear if these rules would be easy enough to implement in type checkers. ## Handling other types of text/binary errors No new types besides `typing.Text` are needed in order to find errors of the other types of errors listed in the Rationale section. Based on the type hints that use the above text / binary types, type checkers in the 2+3 mode should show errors when the user accesses the attributes of these types not available in both Python 2 and Python 3. [1]: https://github.com/python/mypy/issues/1141 [2]: https://github.com/JetBrains/python-skeletons#types [3]: https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code -- Andrey Vlasovskikh Web: http://pirx.ru/ From wes.turner at gmail.com Fri Mar 18 22:12:30 2016 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 18 Mar 2016 21:12:30 -0500 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: This sounds like a more correct approach, thanks. Looking at MarkupSafe (and, now, f-strings), would/will it be possible to use Typing.Text as a base class for even-more abstract string types ("strypes") e.g. XML, XHTML, HTML4, HTML5, HTML5.1, SQL? There are implicit casts and contextual adaptations/transformations (which MarkupSafe specs a bit). (I've no real code here, just a general idea that we're not tracking enough string metadata to be safe here) On Mar 18, 2016 8:45 PM, "Andrey Vlasovskikh" wrote: > With the addition of the comment-based syntax [3] for Python 2.7 + 3 in > PEP 0484 having a Python 2/3 compatible way of adding type hints for text > and binary values becomes important. > > Following the issue #1141 at the Mypy GitHub site [1], I've came up with a > draft proposal based on those ideas that I'd like to discuss here. > > > # Abstract > > This proposal contains recommendations on how to annotate text/binary data > in > newly added PEP 0484 comment-based type hints in order to make them Python > 2/3 > compatible when the single-source approach to porting from 2 to 3 is used. > > It introduces a new type `typing.Text` that represents text data in both > Python > 2 and 3, deprecates `str -> unicode` promotion used in type checkers, > suggests > an approach for type checkers to find implicit conversion errors by > tracking ASCII > text/binary values, recommends that type checkers should warn about > `unicode` in > the 2+3 mode. > > > # Rationale > > With the addition of the comment-based syntax for Python 2.7 + 3 having a > Python > 2/3 compatible way of annotating types of text and binary values becomes > important. Currently having a single-source code base is the main approach > to > 2/3 compatibility, so it is highly desirable to have 2/3 compatible > comment-based type hints that would help porting code from 2 to 2+3 to 3. > > While migrating their code from Python 2 to 3 users are most likely to > discover > the following types of text/binary errors (presumably, in the descending > order > of their frequency in typical code): > > 1. Implicit text/binary conversions removed in Python 3 > 2. Calling changed APIs that accept or return text/binary data > 3. Calling removed/changed methods of text/binary types > 4. Overriding special text/binary methods and using the related built-ins > (`str()`, `repr()`, `unicode()`) > > Only the first two types of errors -- implicit conversions and calling > changed > text/binary APIs -- depend on being able to express the semantics of > Python 2+3 > compatible text/binary interfaces using type hints. > > PEP 0484 doesn't contain any recommendations on how to document various > typical > cases in text/binary APIs in order to make type hints 2+3 compatible. > > > # Proposal > > This document is based on some text/binary handling options and the > problems > associated with them propsed at python/mypy#1141 by Jukka Lehtosalo, Guido > van > Rossum, and others [1]. It also takes into account the experience of the > PyCharm > team with their pre-PEP484 notation for type hints [2] and handling Python > 2/3 > issues reported by users in PyCharm code inspections. > > > ## Handling removed implicit conversions > > In addition to the existing types (`bytes`, `str`, `unicode`, > `typing.AnyStr`) > let's introduce a new type for *2+3 compatible text data* -- `typing.Text` > (should > we add a fake built-in `unicode` type for Python 3 to type checkers > instead of > introducing a new name?): > > * `typing.Text`: text data > * Python 2: `unicode` > * Python 3: `str` > > Just to remind the semantics of the existing types: > > * `bytes`: binary data > * Python 2: `bytes` (== `str`) > * Python 3: `bytes` > * `str`: "native" string, `type('foo')` > * Python 2: `str` > * Python 3: `str` > * `unicode`: Python 2-only text data > * Python 2: `unicode` > * Python 3: error > * `typing.AnyStr`: type variable constrained to both text and binary data > > With the addition of `typing.Text` it is possible to express the type > analogous > to `typing.AnyStr` that doesn't impose any type constraints (should we > call it > `typing.BaseString`?): > > * `typing.Union[typing.Text, bytes]`: both text and binary data when a type > varibale isn't needed > > Using only `typing.Text`, `bytes`, `str`, and `typing.AnyStr` in the type > hints > for an API would mean that this API is Python 2 and 3 compatible in > respect to > implicit text/binary conversions. > > For Python 2 we should *not* have the implicit `str` -> `unicode` promotion > since it hides errors related to implicit conversions. > > For 7-bit ASCII string literals in Python 2 type checkers should infer > special > internal types `typing._AsciiStr` and `typing._AsciiUnicode` that are > compatible > with both `str` and `unicode` (a *special type-checking rule* is needed): > > class _AsciiStr(str): > pass > > class _AsciiUnicode(unicode): > pass > > The details of inferring ASCII types are up to specific type checkers. > > In the 2+3 mode type checkers should show errors when comment- or stub- > based > type hints contain `unicode`. > > > ## Examples of typical 2+3 functions > > A function that accepts "native" strings. It uses implicit ASCII > unicode-to-str conversion at runtime in Python 2 and accepts only text > data in > Python 3: > > def getattr(o: Any, name: str, default: Any = None) -> Any: ... > > A function that does implicit str-to-unicode conversion at runtime in > Python 2 > and accepts only text data in Python 3: > > def hello_rus(name: Text) -> Text: > return u'??????, ' + name > > A function that transforms text-to-text or binary-to-binary or handles > both text > and binary data in some other way in both Python 2 and 3: > > def listdir(path: AnyStr) -> AnyStr: ... > > A function that works with both text and binary data in Python 2 and 3, > where > the author of the function some reason doens't want to have a type variable > associated with `AnyStr`: > > def upper_len(s: Union[bytes, Text]) -> int: > return len(s.upper()) > > A PEP-3333 compatible WSGI app function that uses "native" strings for > environ > and headers data while returning an iterable over binary data in both > Python 2 > and 3: > > def app(environ: Dict[str, Any], > start_response: Callable[[str, List[Tuple[str, str]]], None]) \ > -> Iterable[bytes]: ... > > A type inference example that features a type checker being able to infer > `typing._AsciiStr` or `typing._AsciiUnicode` types for Python 2 using the > functions defined above: > > method_name = u'update' # _AsciiUnicode > getattr({}, method_name) # OK, implicit ASCII-only unicode-to-bytes > in Py2 > > nonascii_data = b'\xff' # _AsciiStr > hello_rus(nonascii_data) # Type checker warning > # Non-ASCII bytes are not compatible with > Text > > # _AsciiUnicode + bytes > u'foo' + b'\xff' # Type checker warning > # Non-ASCII bytes are not compatible with > Text > > def f(x: AnyStr, y: AnyStr) -> AnyStr: > return os.path.join('base', x, y) # _AsciiStr compatible with > AnyStr > # since it's compatible with > # both str and unicode > > There are cases mentioned in [1] where more advanced type inference rules > are > required in order to be able to handle ASCII types. It remains unclear if > these rules would be easy enough to implement in type checkers. > > > ## Handling other types of text/binary errors > > No new types besides `typing.Text` are needed in order to find errors of > the > other types of errors listed in the Rationale section. > > Based on the type hints that use the above text / binary types, type > checkers > in the 2+3 mode should show errors when the user accesses the attributes of > these types not available in both Python 2 and Python 3. > > > [1]: https://github.com/python/mypy/issues/1141 > [2]: https://github.com/JetBrains/python-skeletons#types > [3]: > https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code > > > -- > Andrey Vlasovskikh > > Web: http://pirx.ru/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Mar 18 23:16:28 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 18 Mar 2016 23:16:28 -0400 Subject: [Python-ideas] Is multiple inheritance Pythonic? In-Reply-To: References: <56EC689D.7080504@egenix.com> Message-ID: On 3/18/2016 6:34 PM, Martin Teichmann wrote: > The problem they wanted to solve is that the behavior of > BaseEventLoop.create_task should be changeable. Obvious, > I thought (and did in my code) inherit from the class, overwrite > the method! But what if I want to overwrite that method in > something that inherits from BaseEventLoop, say, some > QtEventLoop? No problem, I thought, just supply a > Mixin, and a simple class EventLoop(AMixin, QtEventLoop) > does the job. This is even chainable, multiple mixins can > be used as long as they properly call super(). The derived subclasses, SelectorEventLoop and ProactorEventLoop, can be subclassed. Four days ago I posted a trivial subclass that re-defined one method, .run_forever. This allows the use of tkinter and asyncio together. I hope it is not an accident that I was able to do so so easily ;-). -- Terry Jan Reedy From rantingrickjohnson at gmail.com Sat Mar 19 09:05:04 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sat, 19 Mar 2016 06:05:04 -0700 (PDT) Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> Message-ID: On Friday, March 18, 2016 at 5:59:42 PM UTC-5, Ryan Gonzalez wrote: > Neither! Python 3 type hints are just function > annotations. It's the job of a type checker/linter (like > mypy) to actually decipher the annotations as types. > Adding/removing type hints won't have a runtime effect on > your program. Thanks for explaining -- i now understand what's going on here... So even though type-hints are "interleaved within the code", at run-time, they have no meaning, and are simply ignored as if they were comments -- it's my worst nightmare come true -- starting with Python 3.5, the readability of Python source will never be the same again. And while i understand that type-hints are optional, once they placed in the code, it is impossible not to be forced to read them. > I think the latter [code example you provided] is for Python 2. Hmm, being that Guido said his "list of issues" was relevant only to 3.5.2, that would not make sense to me. But perhaps i'm missing something again??? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rantingrickjohnson at gmail.com Sat Mar 19 09:24:09 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sat, 19 Mar 2016 06:24:09 -0700 (PDT) Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> Message-ID: <43e966fa-731f-4afa-9e25-19263fe05020@googlegroups.com> On Friday, March 18, 2016 at 6:08:23 PM UTC-5, Paul Moore wrote: > You do realize that PEP 484 was implemented in Python 3.5, > don't you? I do now, and to say that i am shocked would be an understatement -- no, this is clinical depression, plain and simple... > The question here is around some additions (which I don't > understand too well, not being a user of type hints) that > Guido is looking to add in 3.5.2. Paul Yes, i'm fully aware of the scope of what he was requesting opinions *FOR*.... but for me to comment on his "list of issues" without first bringing myself up to speed with the "evolution", would be both naive and disingenuous. @Guido. At this point ,i don't know if i can offer any opinions on your list, because i'm fundamentally opposed to the way type-hints have been implemented. Granted, i'm not opposed to Python having type-hints, or even at some time in the future, supporting "optional typing", i'm just opposed to the way in which they were implemented -- magic comments would have been more Pythonic, at least IMHO. This type- hints thing has fundamentally transformed what Python *IS*. No longer can it market itself as a language who promotes readability, or is "executable pseudo code". Why do we always feel that we have to make the same mistakes that everyone else has made? Can't we do better? Can't we evolve without violating our fundamental philosophy? This is a very sad for me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Mar 19 09:27:04 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 20 Mar 2016 00:27:04 +1100 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> Message-ID: <20160319132704.GL8022@ando.pearwood.info> On Fri, Mar 18, 2016 at 03:51:36PM -0700, Rick Johnson wrote: > > @Everyone: Okay i'm completely confused as to were we are in > the "type hints" evolution. It seems to have taken all sorts > of random twists and turns -- will someone please explain? Rick, PEP 484 has been discussed and discussed and discussed here, and on Python-Dev. To avoid boring everyone else to death, I suggest you raise your initial questions on the python-list mailing list (comp.lang.python newsgroup). If you raise them there, I promise I'll answer. (I think other c.l.p regulars ChrisA and Terry are also familiar enough with PEP 484 to answer your questions.) Then, once you have a better grasp of the background, you can come back here with any questions about Guido's newer proposals. Consider that just a suggestion for the sake of keeping discussion here focused on the newest parts of the PEP, rather than going over old ground. -- Steve From steve at pearwood.info Sat Mar 19 09:39:41 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 20 Mar 2016 00:39:41 +1100 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> Message-ID: <20160319133940.GM8022@ando.pearwood.info> On Fri, Mar 18, 2016 at 08:09:11AM -0600, Gyro Funch wrote: > Could 'citation' be an optional entry in setup.py instead? That matches my proposal. I'm not suggesting it should be a built-in built-in, it can be added by site.py like copyright, license, help, quit etc. If Guido has really strong objections to this, I'd be satisfied with it being a FAQ entry. But I think an advantage of code over a static FAQ is that the code can automatically fill in the version and year. -- Steve From ncoghlan at gmail.com Sat Mar 19 09:45:13 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Mar 2016 23:45:13 +1000 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> Message-ID: On 19 March 2016 at 00:09, Gyro Funch wrote: > On 3/18/2016 1:41 AM, Terry Reedy wrote: >> On 3/17/2016 7:45 PM, Guido van Rossum wrote: >> >>> But then again I've never cared about citations. >> >> Citations are not just in Bibliographies. PEPs have them in >> footnotes, which was a traditional place for citations, though >> in the modern form of hyperlinks. The tracker is full of >> in-line citations, also in the form of hyperlinks. The >> formalized citation proposed by Stephen would include the URL. >> > > Could 'citation' be an optional entry in setup.py instead? > I cite software in papers I write, so having easy access to this > information for each package I use in a given analysis would make > this task easier. I'd actually suggest that programmatic generation of citations for packages distributed through PyPI would be better handled as a third party project. Software developers that aren't themselves also research scientists aren't going to care to provide their own citation details, so it's best to establish a convention that doesn't rely on software publishers doing anything they aren't already doing. In the specific case of CPython, rather than asking Guido and the rest of the core development team "How do you want to be cited?", it probably makes more sense to ask "We're proposing to standardise on citing CPython this way, are you OK with that?" (e.g. by proposing a patch for a new "Citation" link in the meta-information on the docs home page at https://docs.python.org/3/ ). The reason I suggest that approach is that most (all?) of us aren't research scientists, so we have no idea what typical conventions are for citations, nor how those conventions are changing. However, the docs are much easier to amend than the standard library, while still being versioned along with the rest of the software, so adding citation information there is a low risk way of answering the question if folks do want to cite us. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rantingrickjohnson at gmail.com Sat Mar 19 10:28:54 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sat, 19 Mar 2016 07:28:54 -0700 (PDT) Subject: [Python-ideas] PEP 484 evolution In-Reply-To: <20160319132704.GL8022@ando.pearwood.info> References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> Message-ID: On Saturday, March 19, 2016 at 8:27:33 AM UTC-5, Steven D'Aprano wrote: > Rick, PEP 484 has been discussed and discussed and > discussed here, and on Python-Dev. To avoid boring > everyone else to death, I suggest you raise your initial > questions on the python-list mailing list My last two responses should have cleared that up... > (comp.lang.python newsgroup). If you raise them there, I > promise I'll answer. (I think other c.l.p regulars ChrisA > and Terry are also familiar enough with PEP 484 to answer > your questions.) Well thanks for offering to help, and at some point i may start a thread over there, or on my blog, but if i do it now, in the state of shock that i'm in, it will be an expletive laced tirade of epic proportions. I'm far too emotional at this time to maintain even a *MODICUM* of civility in an unmoderated forum, and as time goes on, i'm only going to become more angry if a proper audience is not offered for this grievance. > Then, once you have a better grasp of the background, you > can come back here with any questions about Guido's newer > proposals. Consider that just a suggestion for the sake > of keeping discussion here focused on the newest parts of > the PEP, rather than going over old ground. I don't understand why this group is not the appropriate place to discuss matters concerning Python. Even a discussion that revolves around the subjects of "is type- hints a good idea", or "could we implement type-hints without violating our fundamental philosophy" seems highly appropriate here, and if we move over to Python-list (as you request), we will loose the valuable input of Guido, because he does not participate on that list in any form. @Guido I beseech you to at *MINIMUM* reconsider this implementation. Put it on hold for a short time. It is not too late to prevent catastrophe. I assure you, we can add type-hints, and even optional typing, without violating Python's fundamental philosophy. But once this feature propagates, in its currently form, you can't just press the "undo button". We cannot allow Python to became another "cookie cutter language". We must ask ourselves, "What is it that makes Python *DIFFERENT*"? "What is it that makes Python *SPECIAL*? If you composed a list, starting with the most important features that make Python special, ask yourself -- were would "type-hints" be on that list? Would it even be *ON* the list? I believe type hints is a welcomed addition to "Python's evolution", but not in any form that will undermine the attributes that make Python so special. At *LEAST* entertain a new discussion. A new discussion that invites *ALL* members to participate, regardless of "status" That's all i'm asking... And if the implementation goes ahead "as is", at least those who object, will get the audience they deserve, and will be content knowing that they did everything in their power to prevent catastrophe. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Mar 19 10:36:50 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 20 Mar 2016 01:36:50 +1100 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> Message-ID: On Sun, Mar 20, 2016 at 1:28 AM, Rick Johnson wrote: > @Guido I beseech you to at *MINIMUM* reconsider this > implementation. Put it on hold for a short time. It is not > too late to prevent catastrophe. I assure you, we can add > type-hints, and even optional typing, without violating > Python's fundamental philosophy. But once this feature > propagates, in its currently form, you can't just press the > "undo button". What, you mean the form that was released in Python 3.5 on Dec 6th 2015 after *extensive* discussion on every Python-related mailing list, newsgroup, subreddit, blog, and blue-faced baboon? Yeah. Can't just press the undo button now. Or are you talking about the way that annotations were added to the language in Python 3.0, released in 2008? If this enhancement to an already-existing feature is what finally makes you go and take your ranting to some other language, fine. Goodbye. Otherwise, take it to python-list and, in concurrence with Steven's pledge, I will answer any questions/concerns raised in good faith. ChrisA From rantingrickjohnson at gmail.com Sat Mar 19 11:05:59 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sat, 19 Mar 2016 08:05:59 -0700 (PDT) Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> Message-ID: <7d6da7e1-d589-41af-aebf-db6e6c844e16@googlegroups.com> On Saturday, March 19, 2016 at 9:37:22 AM UTC-5, Chris Angelico wrote: On Sun, Mar 20, 2016 at 1:28 AM, Rick Johnson > What, you mean the form that was released in Python 3.5 on > Dec 6th 2015 after *extensive* discussion on every Python- > related mailing list, newsgroup, subreddit, blog, and > blue-faced baboon? Yeah. Can't just press the undo button > now. Or are you talking about the way that annotations > were added to the language in Python 3.0, released in > 2008? I'm talking specifically about the propagation of "type-hint annotations" within source code in the wild. Conceivably, Python could add a million unique annotations to the language, but no harm is done to readability *UNTIL* the annotations are utilized by Python source code in the wild. My argument is that these "type-hint annotations" have not *YET* propagated -- at least not to any critical mass -- and therefore, we still have time to prevent this ship from crashing on the rocks. > If this enhancement to an already-existing feature is what > finally makes you go and take your ranting to some other > language, fine. Goodbye. Chris, i know you'd be happier if i was not a member of this community. And you've made that point clear on multiple occasions. > Otherwise, take it to python-list and, in concurrence with > Steven's pledge, I will answer any questions/concerns > raised in good faith. What purpose, other than emotional release, will that serve? I could achieve the same by yelling at a wall. -------------- next part -------------- An HTML attachment was scrubbed... URL: From julien at palard.fr Sat Mar 19 12:30:53 2016 From: julien at palard.fr (Julien Palard) Date: Sat, 19 Mar 2016 17:30:53 +0100 Subject: [Python-ideas] https://docs.python.org/fr/ ? Message-ID: <56ED7EBD.5050804@palard.fr> o/ The french translation of the Python Documentation [1][2] has translated 20% of the pageviews of docs.python.org. I think it's the right moment to push it do docs.python.org. So there's some questions ! And I'd like feedback. TL;DR (with my personal choices): - URL may be "http://docs.python.org/fr/" - For localized variations of languages we should use dash and lowercase like "docs.python.org/pt-br/" - po files may be hosted on the python's github - existing script to build doc may be patched to build translations - each translations may crosslink to others - untranslated strings may be visually marked as so I also opened: http://bugs.python.org/issue26546. # Chronology, dependencies The only blocking decision here is the URL, (also reviewing my patch ...), with those two, translated docs can be pushed to production, and the other steps can be discussed and applied one by one. # The URL ## CCTLD vs path vs subdomain I think we should use a variation of "docs.python.org/fr/" for simplicity and clarity. I think we should avoid using CCTLDs as they're sometime hard or near impossible to obtain (may cost a lot of time), also some are expensive, so it's time and money we clearly don't need to loose. Last possibility I see is to use a subdomain, like fr.docs.python.org or docs.fr.python.org but I don't think it's the role / responsibility of the sub-domain to do it. So I'm for docs.python.org/LANGUAGE_TAG/ (without moving current documentation inside a /en/). ## Language tag in path ### Dropping the default locale of a language I personally think we should not show the region in case it's redundant: so to use "fr" instead of "fr-FR", "de" instead of "de-DE", but keeping the possibility to use a locale code when it's not redundant like for "pt-br" or "de-AT" (German ('de') as used in Austria ('AT')). I think so because I don't think we'll have a lot of locale variations (like de-AT, fr-CH, fr-CA, ...) so it will be most of the time redundant (visually heavy, longer to type, longer to read) but we'll still need some locale (pt-BR typically). ### gettext VS IETF language tag format gettext goes by using an underscore between language and locale [3] and IETF goes by using a dash [4][5]. As sphinx is using gettext, and gettext uses underscore we may choose underscore too. But URLs are not here to leak the underlying implementation, and the IETF looks like to be the standard way to represent language tags. Also I visually prefer the dash over the underscore, so I'm for the dash here. ### Lower case vs upper case local tag RFC 5646 section-2.1 tells us language tags are not case sensitive, yet ISO3166-1 recommends that country codes (part of the language tag) be capitalized. I personally prefer the all-lowercase one as paths in URLs typically are lowercase. I searched for `inurl:"pt-br"` to see if I'm not too far away from the usage here and usage seems to agree with me, although there's some "pt-BR" in urls. # Where to host the translated files Currently we're hosting the *po* files in the afpy's (Francophone association for python) [6] github [1] but it may make sense to use (in the generation scripts) a more controlled / restricted clone in the python github, at least to have a better view of who can push on the documentation. We may want to choose between aggregating all translations under the same git repository but I don't feel it's useful. # How to Currently, a python script [7] is used to generate `docs.python.org`, I proposed a patch in [8] to make this script clone and build the french translation too, it's a simple and effective way, I don't think we need more ? Any idea welcome. In our side, we have a Makefile [12] to build the translated doc which is only a thin layer on top of the Sphinx Makefile. So my proposed patch to build scripts "just" delegate the build to our Makefile which itself delegate the hard work to the Sphinx Makefile. # Next ? ## Document how to translate Python I think I can (should) write a documentation on "how to start a Python doc translation project" and "how to migrate existing [9][10][11] python doc translation projects to docs.python.org" if french does goes docs.python.org because it may hopefully motivate people to do the same, and I think our structure is a nice way to do it (A Makefile to generate the doc, all versions translated, people mainly working on latest version, scripts to propagating translations to older version, etc...). ## Crosslinking between existing translations Once the translations are on `docs.python.org`, crosslinks may be established so people on a version can be aware of other version, and easily switch to them. I'm not a UI/UX man but I think we may have a select box right before the existing select box about version, on the top-left corner. Right before because it'll reflect the path: /fr/3.5/ -> [select box fr][select box 3.5]. ## Marking as "untranslated, you can help" the untranslated paragraphs The translations will always need work to follow upstream modifications: marking untranslated paragraphs as so may transform the "Oh they suck, this paragraph is not even translated :-(" to "Hey, nice I can help translating that !". There's an opened sphinx-doc ticket to do so [13] but I have not worked on it yet. As previously said I'm real bad at designing user interfaces, so I don't even visualize how I'd like it to be. [1] http://www.afpy.org/doc/python/3.5/ [2] https://github.com/afpy/python_doc_fr [3] https://www.gnu.org/software/gettext/manual/html_node/Locale-Names.html [4] http://tools.ietf.org/html/rfc5646 [5] https://en.wikipedia.org/wiki/IETF_language_tag [6] http://www.afpy.org/ [7] https://github.com/python/docsbuild-scripts/ [8] http://bugs.python.org/issue26546 [9] http://docs.python.jp/3/ [10] https://github.com/python-doc-ja/python-doc-ja [11] http://docs.python.org.ar/tutorial/3/index.html [12] https://github.com/AFPy/python_doc_fr/blob/master/Makefile [13] https://github.com/sphinx-doc/sphinx/issues/1246 -- Julien Palard From steve at pearwood.info Sat Mar 19 12:46:09 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 20 Mar 2016 03:46:09 +1100 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> Message-ID: <20160319164609.GN8022@ando.pearwood.info> On Sat, Mar 19, 2016 at 07:28:54AM -0700, Rick Johnson wrote: > I don't understand why this group is not the appropriate > place to discuss matters concerning Python. It is appropriate to discuss matters concerning Python. It is not appropriate to dig up matters that were settled months or years ago. Type annotations have been available in Python 3 for seven or eight years, MyPy has been around for long enough to prove that the concept of type-hinting is workable, and the typing module has been approved and released. This is not something which is a spur of the moment decision, even if it is new to you. > And if the implementation goes ahead "as is", at least those > who object, will get the audience they deserve, and will be > content knowing that they did everything in their power to > prevent catastrophe. Take a deep breath and calm down. The type-hints are completely optional and nobody will be forced to use them. -- Steve From guido at python.org Sat Mar 19 14:51:40 2016 From: guido at python.org (Guido van Rossum) Date: Sat, 19 Mar 2016 11:51:40 -0700 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: I like the way this is going. I think it needs to be a separate PEP; PEP 484 is already too long and this topic deserves being written up carefully (like you have done here). I have a few remarks. * Do we really need _AsciiUnicode? I see the point of _AsciiStr, because Python 2 accepts 'x' + u'' but fails '\xff' + u'', so 'x' needs to be of type _AsciiStr while '\xff' should not (it should be just str). However there's no difference in how u'x' is treated from how u'\u1234' or u'\xff' are treated -- none of them can be concatenated to '\xff' and all of them can be concatenated to _'x'. * It would be helpful to spell out exactly what is and isn't allowed when different core types (bytes, str, unicode, Text) meet in Python 2 and in Python 3. Something like a table with a row and a column for each and the type of x+y (or "error") in each of the cells. * I propose that Python 2+3 mode is just the intersection of what Python 2 and Python 3 mode allow. (In mypy, I don't think we'll implement this -- users will just have to run mypy twice with and without --py2. But for PyCharm it makes sense to be able to declare this. Yet I think it would be good not to have to spell out separately which rules it uses, defining it as the intersection of 2 and 3 is all we need. On Fri, Mar 18, 2016 at 6:45 PM, Andrey Vlasovskikh wrote: > With the addition of the comment-based syntax [3] for Python 2.7 + 3 in PEP 0484 having a Python 2/3 compatible way of adding type hints for text and binary values becomes important. > > Following the issue #1141 at the Mypy GitHub site [1], I've came up with a draft proposal based on those ideas that I'd like to discuss here. > > > # Abstract > > This proposal contains recommendations on how to annotate text/binary data in > newly added PEP 0484 comment-based type hints in order to make them Python 2/3 > compatible when the single-source approach to porting from 2 to 3 is used. > > It introduces a new type `typing.Text` that represents text data in both Python > 2 and 3, deprecates `str -> unicode` promotion used in type checkers, suggests > an approach for type checkers to find implicit conversion errors by tracking ASCII > text/binary values, recommends that type checkers should warn about `unicode` in > the 2+3 mode. > > > # Rationale > > With the addition of the comment-based syntax for Python 2.7 + 3 having a Python > 2/3 compatible way of annotating types of text and binary values becomes > important. Currently having a single-source code base is the main approach to > 2/3 compatibility, so it is highly desirable to have 2/3 compatible > comment-based type hints that would help porting code from 2 to 2+3 to 3. > > While migrating their code from Python 2 to 3 users are most likely to discover > the following types of text/binary errors (presumably, in the descending order > of their frequency in typical code): > > 1. Implicit text/binary conversions removed in Python 3 > 2. Calling changed APIs that accept or return text/binary data > 3. Calling removed/changed methods of text/binary types > 4. Overriding special text/binary methods and using the related built-ins > (`str()`, `repr()`, `unicode()`) > > Only the first two types of errors -- implicit conversions and calling changed > text/binary APIs -- depend on being able to express the semantics of Python 2+3 > compatible text/binary interfaces using type hints. > > PEP 0484 doesn't contain any recommendations on how to document various typical > cases in text/binary APIs in order to make type hints 2+3 compatible. > > > # Proposal > > This document is based on some text/binary handling options and the problems > associated with them propsed at python/mypy#1141 by Jukka Lehtosalo, Guido van > Rossum, and others [1]. It also takes into account the experience of the PyCharm > team with their pre-PEP484 notation for type hints [2] and handling Python 2/3 > issues reported by users in PyCharm code inspections. > > > ## Handling removed implicit conversions > > In addition to the existing types (`bytes`, `str`, `unicode`, `typing.AnyStr`) > let's introduce a new type for *2+3 compatible text data* -- `typing.Text` (should > we add a fake built-in `unicode` type for Python 3 to type checkers instead of > introducing a new name?): > > * `typing.Text`: text data > * Python 2: `unicode` > * Python 3: `str` > > Just to remind the semantics of the existing types: > > * `bytes`: binary data > * Python 2: `bytes` (== `str`) > * Python 3: `bytes` > * `str`: "native" string, `type('foo')` > * Python 2: `str` > * Python 3: `str` > * `unicode`: Python 2-only text data > * Python 2: `unicode` > * Python 3: error > * `typing.AnyStr`: type variable constrained to both text and binary data > > With the addition of `typing.Text` it is possible to express the type analogous > to `typing.AnyStr` that doesn't impose any type constraints (should we call it > `typing.BaseString`?): > > * `typing.Union[typing.Text, bytes]`: both text and binary data when a type > varibale isn't needed > > Using only `typing.Text`, `bytes`, `str`, and `typing.AnyStr` in the type hints > for an API would mean that this API is Python 2 and 3 compatible in respect to > implicit text/binary conversions. > > For Python 2 we should *not* have the implicit `str` -> `unicode` promotion > since it hides errors related to implicit conversions. > > For 7-bit ASCII string literals in Python 2 type checkers should infer special > internal types `typing._AsciiStr` and `typing._AsciiUnicode` that are compatible > with both `str` and `unicode` (a *special type-checking rule* is needed): > > class _AsciiStr(str): > pass > > class _AsciiUnicode(unicode): > pass > > The details of inferring ASCII types are up to specific type checkers. > > In the 2+3 mode type checkers should show errors when comment- or stub- based > type hints contain `unicode`. > > > ## Examples of typical 2+3 functions > > A function that accepts "native" strings. It uses implicit ASCII > unicode-to-str conversion at runtime in Python 2 and accepts only text data in > Python 3: > > def getattr(o: Any, name: str, default: Any = None) -> Any: ... > > A function that does implicit str-to-unicode conversion at runtime in Python 2 > and accepts only text data in Python 3: > > def hello_rus(name: Text) -> Text: > return u'??????, ' + name > > A function that transforms text-to-text or binary-to-binary or handles both text > and binary data in some other way in both Python 2 and 3: > > def listdir(path: AnyStr) -> AnyStr: ... > > A function that works with both text and binary data in Python 2 and 3, where > the author of the function some reason doens't want to have a type variable > associated with `AnyStr`: > > def upper_len(s: Union[bytes, Text]) -> int: > return len(s.upper()) > > A PEP-3333 compatible WSGI app function that uses "native" strings for environ > and headers data while returning an iterable over binary data in both Python 2 > and 3: > > def app(environ: Dict[str, Any], > start_response: Callable[[str, List[Tuple[str, str]]], None]) \ > -> Iterable[bytes]: ... > > A type inference example that features a type checker being able to infer > `typing._AsciiStr` or `typing._AsciiUnicode` types for Python 2 using the > functions defined above: > > method_name = u'update' # _AsciiUnicode > getattr({}, method_name) # OK, implicit ASCII-only unicode-to-bytes in Py2 > > nonascii_data = b'\xff' # _AsciiStr > hello_rus(nonascii_data) # Type checker warning > # Non-ASCII bytes are not compatible with Text > > # _AsciiUnicode + bytes > u'foo' + b'\xff' # Type checker warning > # Non-ASCII bytes are not compatible with Text > > def f(x: AnyStr, y: AnyStr) -> AnyStr: > return os.path.join('base', x, y) # _AsciiStr compatible with AnyStr > # since it's compatible with > # both str and unicode > > There are cases mentioned in [1] where more advanced type inference rules are > required in order to be able to handle ASCII types. It remains unclear if > these rules would be easy enough to implement in type checkers. > > > ## Handling other types of text/binary errors > > No new types besides `typing.Text` are needed in order to find errors of the > other types of errors listed in the Rationale section. > > Based on the type hints that use the above text / binary types, type checkers > in the 2+3 mode should show errors when the user accesses the attributes of > these types not available in both Python 2 and Python 3. > > > [1]: https://github.com/python/mypy/issues/1141 > [2]: https://github.com/JetBrains/python-skeletons#types > [3]: https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code > > > -- > Andrey Vlasovskikh > > Web: http://pirx.ru/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guido) From brett at python.org Sat Mar 19 15:42:00 2016 From: brett at python.org (Brett Cannon) Date: Sat, 19 Mar 2016 19:42:00 +0000 Subject: [Python-ideas] keeping things civil (was: Re: PEP 484 evolution) In-Reply-To: <7d6da7e1-d589-41af-aebf-db6e6c844e16@googlegroups.com> References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <7d6da7e1-d589-41af-aebf-db6e6c844e16@googlegroups.com> Message-ID: On Sat, 19 Mar 2016 at 08:09 Rick Johnson wrote: > On Saturday, March 19, 2016 at 9:37:22 AM UTC-5, Chris Angelico wrote: > > On Sun, Mar 20, 2016 at 1:28 AM, Rick Johnson > > What, you mean the form that was released in Python 3.5 on > > Dec 6th 2015 after *extensive* discussion on every Python- > > related mailing list, newsgroup, subreddit, blog, and > > blue-faced baboon? Yeah. Can't just press the undo button > > now. Or are you talking about the way that annotations > > were added to the language in Python 3.0, released in > > 2008? > > I'm talking specifically about the propagation of "type-hint > annotations" within source code in the wild. Conceivably, > Python could add a million unique annotations to the > language, but no harm is done to readability *UNTIL* the > annotations are utilized by Python source code in the wild. > My argument is that these "type-hint annotations" have not > *YET* propagated -- at least not to any critical mass -- and > therefore, we still have time to prevent this ship from > crashing on the rocks. > I think that's a bit dramatic. As others have pointed out, this has already been discussed quite extensively and decided upon. The community has not had enough time to make an informed opinion on this topic to ask that it be removed. > > > > If this enhancement to an already-existing feature is what > > finally makes you go and take your ranting to some other > > language, fine. Goodbye. > > > Chris, i know you'd be happier if i was not a member of this > community. And you've made that point clear on multiple > occasions. > > > > Otherwise, take it to python-list and, in concurrence with > > Steven's pledge, I will answer any questions/concerns > > raised in good faith. > > What purpose, other than emotional release, will that serve? > I could achieve the same by yelling at a wall. > The offers seemed reasonable, so throwing them back in their faces doesn't really seem to help anything. Based on Chris' reaction there seems to be some history from somewhere else that is spilling over here. I'm going to remind everyone -- not just Rick because he's new here but also those who have reacted harshly to Rick being here -- that this list falls under the PSF Code of Conduct and it will be enforced. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Mar 19 15:50:30 2016 From: brett at python.org (Brett Cannon) Date: Sat, 19 Mar 2016 19:50:30 +0000 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: Message-ID: On Fri, 18 Mar 2016 at 11:57 Terry Reedy wrote: > On 3/18/2016 1:32 PM, Guido van Rossum wrote: > > I have a few different things I'd like to add to PEP 484. Should I start > > this here or on pyrhon-dev? > > I would say that it depends on whether the ideas need to be kicked > around a bit or whether you think they are in final form. > That's basically the rule of them I take. Basically I view python-ideas as the place to start the question of "is this a good idea or is it bonkers?" Once you know that answer you either take to python-dev or drop it. And as a core dev you can use your own judgment as to whether you need the extra step of discussing it here first and instead bypass this list. > > > The hope is to get these into 3.5.2. When is > > that going out? > > Last I knew, it was scheduled for next May, but > https://www.python.org/dev/peps/pep-0478/ > does not say anything. > Since 3.5.1 came out in December I would expect either at or after PyCon US. Larry is RM so he's the only one that really knows. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rantingrickjohnson at gmail.com Sat Mar 19 16:14:30 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sat, 19 Mar 2016 13:14:30 -0700 (PDT) Subject: [Python-ideas] PEP 484 evolution In-Reply-To: <20160319164609.GN8022@ando.pearwood.info> References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> Message-ID: On Sat, Mar 19, 2016 at 11:46 AM, Steven D'Aprano wrote: > Take a deep breath and calm down. The type-hints are > completely optional and nobody will be forced to use them. I don't think it's fair for you to tell me to: "calm down because this won't effect you". What evidence are you basing that theory on? Type-hits will effect everyone. Telling me to calm down is like: blowing up and damn, and then telling the peasants to "clam down" because their houses, and more troubling, their entire village was destroyed as a result of actions that were "beyond their control". You, and a few others supporters, keep claiming that "type- hints" are optional -- but while they may be optional to *write*, they are *not* optional to *read*. Furthermore, once source code is written that includes this "interleaved syntax", the readability of the code will become *permanently* damaged. Observe the following: # SIMPLE EXAMPLE WITHOUT TYPE HINTS: def greeting(name): return 'Hello, {}'.format(name) # SIMPLE EXAMPLE WITH TYPE HINTS: def greeting(name: str) -> str: return 'Hello, {}'.format(name) I couldn't just ignore the ": str" and the "-> str" even if i wanted to, no more than i could ignore braces -- if python gave us the option of defining blocks with braces. No, it's there *forever*, mucking up the readability of the code. And if that example seems too simplistic, then feast your eyes on this "abortion of readability": def greeting(names: Union[List[str], Dict[int, List[str]]]) -> Union[ List[str], Dict[int, List[str]]]: fmt = 'Hello, {}' if isinstance(names, dict): return [(k, fmt.format(', '.join(v))) for k, v in names.items()] else: return fmt.format(', '.join(names)) When faced with code like that, my first reaction is to go running back into the loving arms of Java -- why would i use Python? When Java offers *REAL* static typing; is compiled; runs *EVERYWHERE*; executes far more quickly; and is not suffering from an identity crisis. ============================================================ BUT THERE IS STILL HOPE... ============================================================ I assure you, we can introduce a "type-hint-spec" that won't destroy the readability of Python source code -- by moving the spec *OUTSIDE* of sigs... Consider this: def greeting(name): # typehint: greeting(name:str) -> str return 'Hello, {}'.format(name) ...or even this: @typehint: greeting(name:str) -> str def greeting(name): return 'Hello, {}'.format(name) ...or by using stub files (but i understand why people hate them, because of the "syncing issues" involved.) ...any of which, are a more Pythonic way to evolve. and for sequence members, or other "member types", creating new objects that enforce "explicit typing", along the lines of array.array(type), is far more pythonic that introducing *line noise* to force *all* objects to submit to this type- hint nightmare. >>> import array >>> a1 = array.array >>> a1 = array.array('i') >>> a1.append(10) >>> a1.append('10') Traceback (most recent call last): File "", line 1, in a1.append('10') TypeError: an integer is required You don't need type-hints where type can be *enforced*. ============================================================ WHAT DO WE CONSIDER IMPORTANT? ============================================================ *READABILITY* is the most cited reason for choosing Python, followed closely behind by: rapid development and the freedom of dynamic typing -- take away any of those, and what's the point anymore? What's next, braces? How deep does this "philosophical perversion" go? I can sum up this "supposed feature" in one sentence... type-hints (as implemented) shackles a programmer with *ALL* of the syntactical onerous that any statically typed language would demand, except, without offering any benefits of *REAL* type-checking -- rr Should a "feature", that is marketed as a "hint", shackle the programmer with the same onerous that the *REAL* feature would? Consider the following observation of type-hints: You have to write the hint *YOURSELF*, then, you have to process the hint *YOURSELF* -- python does nothing but ignore your "type-hint syntax". So it's like: "Hey Python, i'll pay you two dollars if you'll allow me to write this nasty type-hint syntax"... So, python takes the two dollars, silently ignores the nasty syntax, and then charges every reader a "readability fee"... SOMETHING IS SOOOOO WRONG HERE! Contrary to popular belief, "type-hints" is not "feature", no, it's a *FUNDAMENTAL TRANSFORMATION* of Python into a completely different language! One that makes the same old mistakes that every other language has made. "Welcome to mediocrity, leave your expectations at the door" If you think the people who love Python, because of it's readability, are going to take this "type-hints absurdity" lying down, well, then, you and "your supporters" are in for a rude awakening! Whether intentional or not, type-hints will effect *EVERY* Python programmer at some point in the future -- even those that wish to avoid it. And since Python has already suffered a terrible blow from the release of Python3000, this "type-hints feature", if implemented "as is", looks to be the final "nail in it's coffin". But we can prevent this if we do something *NOW*. I implore each and every member to ask themselves the following questions: (1) What is the "essence of Python", that makes it such an attractive language to use? (2) If you had to compose a short list of the best attributes of Python -- would type-hints be on that list? If so, explain... (3) How do you justify the irreversible damage to Python's readability, that will be caused by this current implementation of type-hints -- surely you're not taking the position that: "no damage will be caused". (4) What is *SO* important about type-hints that it over- rides our fundamental philosophy? -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Sat Mar 19 17:14:01 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 19 Mar 2016 22:14:01 +0100 Subject: [Python-ideas] Unified database exception structure In-Reply-To: <20160318194705.GA27028@sjoerdjob.com> References: <20160318194705.GA27028@sjoerdjob.com> Message-ID: <56EDC119.6010004@egenix.com> Please take this idea to the DB-SIG, where the DB-API is discussed. Thanks. On 18.03.2016 20:47, Sjoerd Job Postmus wrote: > Dear all, > > I'm not sure if this has been discussed before, but I have a bit of a > concern with respect to the exceptions defined in the DB-API 2.0 (PEP > 0249), especially with respect to code written to support multiple > database backends (for instance, through an ORM, or manually). > ... -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From breamoreboy at yahoo.co.uk Sat Mar 19 18:28:52 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sat, 19 Mar 2016 22:28:52 +0000 Subject: [Python-ideas] keeping things civil In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <7d6da7e1-d589-41af-aebf-db6e6c844e16@googlegroups.com> Message-ID: On 19/03/2016 19:42, Brett Cannon wrote: > > On Sat, 19 Mar 2016 at 08:09 Rick Johnson > Based on Chris' reaction there seems to be some history from somewhere > else that is spilling over here. I'm going to remind everyone -- not > just Rick because he's new here but also those who have reacted harshly > to Rick being here -- that this list falls under the PSF Code of Conduct > and it will be enforced. > As all ready pointed out rr is a well known troll from the main Python mailing list. Unless he talks about tkinter/IDLE you can usually treat his posts with several tons of salt. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From tjreedy at udel.edu Sat Mar 19 18:42:36 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 19 Mar 2016 18:42:36 -0400 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> Message-ID: On 3/19/2016 4:14 PM, Rick Johnson wrote: > You, and a few others supporters, keep claiming that "type- > hints" are optional -- but while they may be optional to > *write*, they are *not* optional to *read*. Just to be sure you know, one of the options is to put type-hints in a separate stub file, so only those concerned will ever see them. In particular, type hints for the stdlib will have to be kept separate. > def greeting(names: Union[List[str], Dict[int, List[str]]]) -> Union[ > List[str], Dict[int, List[str]]]: I agree that the extra overloading makes this hard to read by obscuring the essential parts. Anything like the above should be in private code or stub files. -- Terry Jan Reedy From rantingrickjohnson at gmail.com Sat Mar 19 20:57:28 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sat, 19 Mar 2016 17:57:28 -0700 (PDT) Subject: [Python-ideas] keeping things civil (was: Re: PEP 484 evolution) In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <7d6da7e1-d589-41af-aebf-db6e6c844e16@googlegroups.com> Message-ID: On Saturday, March 19, 2016 at 2:43:01 PM UTC-5, Brett Cannon wrote: > I think that's a bit dramatic. As others have pointed out, > this has already been discussed quite extensively and > decided upon. The community has not had enough time to > make an informed opinion on this topic to ask that it be > removed. Hi Brett, I'm wondering what your last sentence is implying. Could you elaborate please? Here are my two interpretations: (1) The community is actively discussing the issues i have raised in this thread, but will need more time to decide if my fears are indeed valid... or (2) The community is only testing type-hints in the wild, and may decide, at a later date, to remove them if the roll out does not go as expected... > The offers seemed reasonable, so throwing them back in > their faces doesn't really seem to help anything. Yes, i will admit that the *offers* they provided where "reasonable". But i have dealt with Steven and Chris and Mark for many years, and i can tell you from experience, that once we move this discussion over to Python-list (which is unmoderated), the hospitality and decency will quickly evaporate into the ether. I prefer for this not to descend into a flame war, and would like to engage in a civil discourse. I have strong reservations against this "type- hints feature", at least in its current form, and i know i'm not the only person who does. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rantingrickjohnson at gmail.com Sat Mar 19 21:34:07 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sat, 19 Mar 2016 18:34:07 -0700 (PDT) Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> Message-ID: On Saturday, March 19, 2016 at 5:43:12 PM UTC-5, Terry Reedy wrote: > Just to be sure you know, one of the options is to put > type-hints in a separate stub file, so only those > concerned will ever see them. Yes, I was aware of the "stubfile option", and had stubfiles been the *ONLY* option, i would have thrown my support behind this type-hints thing 110%! Just think of it... Stub files would offload *ALL* the onerous on the *SINGLE* programmer who wanted to employ them, and would not force *ANY* reader to read them -- unless of course, the reader *WANTED* to. This would have been wonderful! It would have been great! It would have been [Insert your happy word here]! But alas, this is not what we have implemented. > In particular, type hints for the stdlib will have to be > kept separate. That's a wonderful start. Now, if we can *ONLY* convince the powers that be, to make it mandatory everywhere! > > def greeting(names: Union[List[str], Dict[int, List[str]]]) -> Union[ > List[str], Dict[int, List[str]]]: > > I agree that the extra overloading makes this hard to read > by obscuring the essential parts. Anything like the above > should be in private code or stub files. Agreed. But unless the convention is forced, we will see code like this everywhere. Terry, you're closer to these folks than i am. You've *GOT* to try and convince them that this "interleaved typehints" thing is going to be a disaster -- see if you can gather some "off the record" support. If stubfiles were mandatory: (1) Almost all of the work, that has been invested so far by Python-dev, will *NOT* be for nothing. (2) Typehints will *NOT* go away. (3) We can save readability *AND* keep type-hints. (win-win) If we can convince them to make stubfiles mandatory, then we will prevent the propagation of "interleaved type-hint" code in the wild. Then, we can redesign the type-hint-spec as a magic comment -- instead of the current "interleaved dead code" implementation. This is really the only way out of this mess. We cannot dump type-hints, because too many folks are counting on them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Mar 19 21:58:28 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 20 Mar 2016 12:58:28 +1100 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> Message-ID: On Sun, Mar 20, 2016 at 12:34 PM, Rick Johnson wrote: > Yes, I was aware of the "stubfile option", and had stubfiles > been the *ONLY* option, i would have thrown my support > behind this type-hints thing 110%! > Stub files have *never* been the only option, for the same reason that external documentation is not a replacement for docstrings. Keeping information near the code it's connected to gives it a FAR better chance of staying current than having it out-of-line does. Stub files are supported for those times when it's impossible or inadvisable to use inline annotations (eg for Py2 compatibility - the comments are ugly compared to the proper annotations), but the preferred way to provide type hints is to use the PEP 3107 annotation syntax. ChrisA From tritium-list at sdamon.com Sat Mar 19 22:21:26 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Sat, 19 Mar 2016 22:21:26 -0400 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> Message-ID: <56EE0926.5070408@sdamon.com> On 3/19/2016 21:58, Chris Angelico wrote: > Stub files have *never* been the only option, for the same reason that > external documentation is not a replacement for docstrings. Keeping > information near the code it's connected to gives it a FAR better > chance of staying current than having it out-of-line does. Stub files > are supported for those times when it's impossible or inadvisable to > use inline annotations (eg for Py2 compatibility - the comments are > ugly compared to the proper annotations), but the preferred way to > provide type hints is to use the PEP 3107 annotation syntax. > > ChrisA > I don't exactly see how stub files would be any different than tests in this regard. Unless I am totally misunderstanding something, the type hints only exist to make static analysis by tools (presumably in a test suite) easier. Why would a failing static analysis check go unnoticed and not the a test? Why is there no argument to integrate the test suite into the module under test, by the same logic? When documentation does not get updated to reflect the reality of the code, a test run does not fail, a task does not turn red in CI, tools don't return non-zero. If you care about type-hints, keeping hints in stub files, and not updating those stubs, breaks things. Why muck up the code for people who don't care about type hints? From rosuav at gmail.com Sat Mar 19 22:29:36 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 20 Mar 2016 13:29:36 +1100 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: <56EE0926.5070408@sdamon.com> References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> <56EE0926.5070408@sdamon.com> Message-ID: On Sun, Mar 20, 2016 at 1:21 PM, Alexander Walters wrote: > I don't exactly see how stub files would be any different than tests in this > regard. Unless I am totally misunderstanding something, the type hints only > exist to make static analysis by tools (presumably in a test suite) easier. > Why would a failing static analysis check go unnoticed and not the a test? > Why is there no argument to integrate the test suite into the module under > test, by the same logic? There are - look at things like doctest. However, tests tend to be much longer-winded than type hints - even the most verbose of type hints - so the cost of keeping them inline is far higher. The benefits are similar, though. ChrisA From brett at python.org Sat Mar 19 22:35:24 2016 From: brett at python.org (Brett Cannon) Date: Sun, 20 Mar 2016 02:35:24 +0000 Subject: [Python-ideas] keeping things civil (was: Re: PEP 484 evolution) In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <7d6da7e1-d589-41af-aebf-db6e6c844e16@googlegroups.com> Message-ID: On Sat, 19 Mar 2016 at 17:57 Rick Johnson wrote: > > > On Saturday, March 19, 2016 at 2:43:01 PM UTC-5, Brett Cannon wrote: > > I think that's a bit dramatic. As others have pointed out, > > this has already been discussed quite extensively and > > decided upon. The community has not had enough time to > > make an informed opinion on this topic to ask that it be > > removed. > > Hi Brett, > > I'm wondering what your last sentence is implying. Could you > elaborate please? Here are my two interpretations: > > (1) The community is actively discussing the issues i have > raised in this thread, but will need more time to decide > if my fears are indeed valid... > > or > > (2) The community is only testing type-hints in the wild, > and may decide, at a later date, to remove them if the roll > out does not go as expected... > The latter. Python 3.5.0 only came out this past September so it has not even been a year yet. Plus things like this take multiple years to work their way into people's development processes so any talk of removing something that would break backwards-compatibility is very premature. > > > > The offers seemed reasonable, so throwing them back in > > their faces doesn't really seem to help anything. > > Yes, i will admit that the *offers* they provided where > "reasonable". But i have dealt with Steven and Chris and > Mark for many years, and i can tell you from experience, > that once we move this discussion over to Python-list (which > is unmoderated), the hospitality and decency will quickly > evaporate into the ether. > That's fine, but I have dealt with them for years as well and find them rather reasonable individuals. > I prefer for this not to descend > into a flame war, and would like to engage in a civil > discourse. I have strong reservations against this "type- > hints feature", at least in its current form, and i know i'm > not the only person who does. > You're allowed to have reservations, but the point is the debate on type hints in regards as to whether they should exist or not is over and done with as the decision to have them has already been made as shown by them existing in a release of Python. Drudging the topic up here serves no purpose as they will not be removed and so debating it is just a waste of time. It would be better to discuss how to improve them as Guido is doing as that will be productive. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Mar 19 23:12:08 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 19 Mar 2016 20:12:08 -0700 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> Message-ID: On Mar 19, 2016, at 18:34, Rick Johnson wrote: > > On Saturday, March 19, 2016 at 5:43:12 PM UTC-5, Terry Reedy wrote: > > Just to be sure you know, one of the options is to put > > type-hints in a separate stub file, so only those > > concerned will ever see them. > > Yes, I was aware of the "stubfile option", and had stubfiles > been the *ONLY* option, i would have thrown my support > behind this type-hints thing 110%! That would definitely turn me off the feature. In practice, I've found that, at least for me, the largest benefit of PEP 484 has come from documenting interfaces. There are a reasonable fraction of cases where a name and type for each parameter is enough to understand the use of a function, but a name alone isn't. Of course there are other cases where the types get in the way--so I just don't specify them there. This shouldn't be too surprising--ML, Haskell, etc. users have long known that sometimes it aids readability to explicitly declare a type even when the inference engine makes it unnecessary. Why should it be any different when dynamic typing always makes it unnecessary? Anyway, if stub files were the only way to specify types, PEP 484 would probably still be useful for some people, as in Dropbox's project, but it would be a lot less interesting to me. From ncoghlan at gmail.com Sun Mar 20 04:01:49 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Mar 2016 18:01:49 +1000 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> Message-ID: On 20 March 2016 at 06:14, Rick Johnson wrote: > On Sat, Mar 19, 2016 at 11:46 AM, Steven D'Aprano > wrote: > >> Take a deep breath and calm down. The type-hints are >> completely optional and nobody will be forced to use them. > > I don't think it's fair for you to tell me to: "calm down > because this won't effect you". What evidence are you basing > that theory on? The fact we've been down this road before, especially with the introduction of ABCs (much of the fearful rhetoric around PEP 484 bears a striking similarity to that around PEP 3119 nigh on a decade ago). If the Python community were going to go overboard with explicit type hints, we'd already see a large proportion of projects running pylint with very strict configuration settings, or using the existing typechecking tools like Google's pytypedecl. However, that's precisely what we *don't* see - instead we see folks using those capabilities responsibly, adopting them when the complexity of their project or organisational structure calls for it, and omitting them otherwise. (And many of the folks that don't have the relevant problems remain puzzled as to why these tools that help improve maintainability for large projects and large teams exist in the first place) The fear response of "other people are going to overuse these, which means *I'm* going to have to read a lot of code containing them, which means programming in Python is going to become less fun" is perfectly understandable, but isn't really backed up by the community's collective behavioural history. (Monkeypatching, multiple inheritance and metaclasses are other capabilities that fall into a similar category - really powerful, and when you need them, you're grateful they're there, but best avoided when you don't have a genuine need for them) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Sun Mar 20 13:36:49 2016 From: brett at python.org (Brett Cannon) Date: Sun, 20 Mar 2016 17:36:49 +0000 Subject: [Python-ideas] CoC violation warning (was: Re: PEP 484 evolution) In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> Message-ID: I tried to be calm about asking the tone to be turned down on this topic, but I feel like I wasn't listened to, so consider this your *official warning, Rick*: *stop the bombastic, insulting tone or you will be banned*. It is not considerate or respectful to call type hints an "abortion of readability" and claim we are suffering from "an identity crisis" thanks to this "philosophical perversion" brought on by this "type-hint nightmare". Plenty of people on this list had a thorough, cordial discussion on this topic long before you joined it and you do not have the right to come here and suddenly start disparaging their hard work (not to mention those people who aren't on this list who also participated in the discussion and eventual implementation). If type hints upset you that much you are welcome to not use Python anymore and "go running back into the loving arms of Java" if you prefer. I'm also asking everyone to consider this discussion of removing type hints dead, so please be respectful of other people's time and don't reply to this thread. On Sat, 19 Mar 2016 at 13:15 Rick Johnson wrote: > On Sat, Mar 19, 2016 at 11:46 AM, Steven D'Aprano > wrote: > > > > Take a deep breath and calm down. The type-hints are > > completely optional and nobody will be forced to use them. > > I don't think it's fair for you to tell me to: "calm down > because this won't effect you". What evidence are you basing > that theory on? Type-hits will effect everyone. > > Telling me to calm down is like: blowing up and damn, and > then telling the peasants to "clam down" because their > houses, and more troubling, their entire village was > destroyed as a result of actions that were "beyond their > control". > > You, and a few others supporters, keep claiming that "type- > hints" are optional -- but while they may be optional to > *write*, they are *not* optional to *read*. > > Furthermore, once source code is written that includes this > "interleaved syntax", the readability of the code will become > *permanently* damaged. > > Observe the following: > > # SIMPLE EXAMPLE WITHOUT TYPE HINTS: > > def greeting(name): > return 'Hello, {}'.format(name) > > # SIMPLE EXAMPLE WITH TYPE HINTS: > > > def greeting(name: str) -> str: > return 'Hello, {}'.format(name) > > I couldn't just ignore the ": str" and the "-> str" even if > i wanted to, no more than i could ignore braces -- if python > gave us the option of defining blocks with braces. No, it's > there *forever*, mucking up the readability of the code. And > if that example seems too simplistic, then feast your eyes > on this "abortion of readability": > > def greeting(names: Union[List[str], Dict[int, List[str]]]) -> Union[ > List[str], Dict[int, List[str]]]: > fmt = 'Hello, {}' > if isinstance(names, dict): > return [(k, fmt.format(', '.join(v))) for k, v in > names.items()] > else: > return fmt.format(', '.join(names)) > > When faced with code like that, my first reaction is to go > running back into the loving arms of Java -- why would i use > Python? When Java offers *REAL* static typing; is compiled; > runs *EVERYWHERE*; executes far more quickly; and is not > suffering from an identity crisis. > > ============================================================ > BUT THERE IS STILL HOPE... > ============================================================ > > I assure you, we can introduce a "type-hint-spec" that won't > destroy the readability of Python source code -- by moving > the spec *OUTSIDE* of sigs... > > Consider this: > > def greeting(name): > # typehint: greeting(name:str) -> str > return 'Hello, {}'.format(name) > > ...or even this: > > @typehint: greeting(name:str) -> str > def greeting(name): > return 'Hello, {}'.format(name) > > ...or by using stub files (but i understand why people hate > them, because of the "syncing issues" involved.) > > ...any of which, are a more Pythonic way to evolve. > > and for sequence members, or other "member types", creating > new objects that enforce "explicit typing", along the lines > of array.array(type), is far more pythonic that introducing > *line noise* to force *all* objects to submit to this type- > hint nightmare. > > >>> import array > >>> a1 = array.array > >>> a1 = array.array('i') > >>> a1.append(10) > >>> a1.append('10') > > Traceback (most recent call last): > File "", line 1, in > a1.append('10') > TypeError: an integer is required > > You don't need type-hints where type can be *enforced*. > > ============================================================ > WHAT DO WE CONSIDER IMPORTANT? > ============================================================ > > *READABILITY* is the most cited reason for choosing Python, > followed closely behind by: rapid development and the > freedom of dynamic typing -- take away any of those, and > what's the point anymore? What's next, braces? How deep does > this "philosophical perversion" go? I can sum up this > "supposed feature" in one sentence... > > type-hints (as implemented) shackles a programmer with > *ALL* of the syntactical onerous that any statically typed > language would demand, except, without offering any benefits > of *REAL* type-checking -- rr > > Should a "feature", that is marketed as a "hint", shackle the > programmer with the same onerous that the *REAL* feature > would? > > Consider the following observation of type-hints: > > You have to write the hint *YOURSELF*, then, you have to > process the hint *YOURSELF* -- python does nothing but > ignore your "type-hint syntax". > > So it's like: "Hey Python, i'll pay you two dollars if > you'll allow me to write this nasty type-hint syntax"... > So, python takes the two dollars, silently ignores the > nasty syntax, and then charges every reader a "readability > fee"... > > SOMETHING IS SOOOOO WRONG HERE! > > Contrary to popular belief, "type-hints" is not "feature", > no, it's a *FUNDAMENTAL TRANSFORMATION* of Python into a > completely different language! One that makes the same old > mistakes that every other language has made. > > "Welcome to mediocrity, leave your expectations at the door" > > If you think the people who love Python, because of it's > readability, are going to take this "type-hints absurdity" > lying down, well, then, you and "your supporters" are in for > a rude awakening! Whether intentional or not, type-hints > will effect *EVERY* Python programmer at some point in the > future -- even those that wish to avoid it. > > And since Python has already suffered a terrible blow from > the release of Python3000, this "type-hints feature", if > implemented "as is", looks to be the final "nail in it's > coffin". > > But we can prevent this if we do something *NOW*. > > I implore each and every member to ask themselves the > following questions: > > (1) What is the "essence of Python", that makes it such an > attractive language to use? > > (2) If you had to compose a short list of the best > attributes of Python -- would type-hints be on that list? > If so, explain... > > (3) How do you justify the irreversible damage to Python's > readability, that will be caused by this current > implementation of type-hints -- surely you're not taking the > position that: "no damage will be caused". > > (4) What is *SO* important about type-hints that it over- > rides our fundamental philosophy? > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Sun Mar 20 14:12:58 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Sun, 20 Mar 2016 19:12:58 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body Message-ID: <56EEE82A.6050505@mail.de> Hi Python-ideas, this is a follow up from an thread of Python-List: https://mail.python.org/pipermail/python-list/2016-March/705205.html What do you think about the following proposal? *Abstract* Control structures such as "for", "while" and "try" feature some various additional clauses being executed under certain conditions. This proposal adds another extra clause to "for" and "while" in order to provide an easy way to discover whether the body of a loop as never been executed at all. *Motivation** *Coming from the Web application development, we periodically can see a benefit of a hypothetical "empty" clause of Python's for loop like this: for item in my_iterator: # do per item empty: # do something else The same goes for "while" although there the choice of the name "empty" might be questionable. "empty" refers to the involved iterable which is not really existing in case of "while". However, there is not need of introducing inconsistencies among the loop constructs as both feature the "else" clause as well. Better suggestions are welcome; also see the keyword section. *Keyword* Keywords under consideration have been in (an attempt of dispassionate) order of preference: 1) empty -> most obvious, most people responded with a solution to solve the intended problem 2) else -> already taken but to some would be the preferred one 3) or -> as alternative if a new keyword is too much 4) except -> as alternative if a new keyword is too much 5) instead -> less obvious and new keyword *Use-Cases* 1) As humans mostly have a hard time to discover the difference between *nothing**as error* and *nothing**as really empty*, GUI/Web interfaces display an explicit notice in case of empty lists to signify the correct execution of a process/action/search or the other case. 2) Single way and no boilerplate of handling empty loops. As the discussion on Python-List showed are there quite some number of different ways of achieving this kind of functionality. 3) Less errors in development and maintenance. Some of the proposed solutions would not work in some circumstances (either during development or depending on the type of the items) which can hide errors. *Implementation* To be done. *Maintainer* Me. *Issues** *People I talked to suggested "else" as an alternative to "empty" because "empty is not quite clear". Unfortunately, "else" is already taken and is itself not quite clear. "then" has been proposed as an alternative for "else" in an attempt for proper understanding of "else". If that would be an accepted proposal, it would make room for "else" to be used for the usage of the "empty keyword proposed here. *Related Work** *This proposal is partly inspired by template engines like Django or jinja2 which both already provide such facility. Django: {% for item in my_iterator %} ... {% empty %} ... {% endfor %} jinja2: {% for item in my_iterator %} ... {% else %} ... {% endfor %} Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sun Mar 20 16:16:50 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 20 Mar 2016 13:16:50 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56EEE82A.6050505@mail.de> References: <56EEE82A.6050505@mail.de> Message-ID: On Mar 20, 2016, at 11:12, Sven R. Kunze wrote: > > Issues > People I talked to suggested "else" as an alternative to "empty" because "empty is not quite clear". Unfortunately, "else" is already taken and is itself not quite clear. "then" has been proposed as an alternative for "else" in an attempt for proper understanding of "else". If that would be an accepted proposal, it would make room for "else" to be used for the usage of the "empty keyword proposed here. Besides the backward compatibility issue, changing "else" to "then" would be horribly confusing. I suspect anyone who thinks that would be an improvement doesn't actually understand or like for...else, and they'd be happier just eliminating it, not renaming it. An else clause is testing that no break was hit inside the loop. Look at a typical example: for elem in seq: if good(elem): break else: raise ValueError The word "then" would make no sense there. "Find the first good element, else raise an exception" makes sense; "find the first good element, then raise an exception" means the opposite of what we want. Anyway, none of this speaks for or against your main proposal, it just means (at least in my opinion) that this alternative option is off the table. More generally, I think if this feature were to be added, "empty" is a reasonable name. The idiomatic way to write it today is something like: elem = empty = object() for elem in seq: do_stuff(elem) if elem is empty: do_empty_seq_stuff() So you're basically looking for syntactic sugar that abbreviated "if ... is empty:", right? -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Sun Mar 20 16:26:03 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Sun, 20 Mar 2016 13:26:03 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: (Nick Coghlan's message of "Sat, 19 Mar 2016 23:45:13 +1000") References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> Message-ID: <87io0goq04.fsf@vostro.rath.org> On Mar 19 2016, Nick Coghlan wrote: > The reason I suggest that approach is that most (all?) of us aren't > research scientists, so we have no idea what typical conventions are > for citations, nor how those conventions are changing. Which I believe makes it completely pointless to cite Python at all. As far as I can see, nowadays citations are given for two reasons: 1. To give the reader a starting point to get more information on a topic. 2. To formally acknowledge the work done by someone else (who ends up with an increased number of citations for the cited publication, which is unfortunately a crucial metric in most academic hiring and evaluation processes). In case of Python, an explicit citation thus adds nothing. Relevant information is easily found by any search engine, and if Nick is right than no one stands to gain anything from the citations either. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From rantingrickjohnson at gmail.com Sun Mar 20 16:35:09 2016 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sun, 20 Mar 2016 13:35:09 -0700 (PDT) Subject: [Python-ideas] CoC violation warning (was: Re: PEP 484 evolution) In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> Message-ID: On Sunday, March 20, 2016 at 12:37:48 PM UTC-5, Brett Cannon wrote: > I tried to be calm about asking the tone to be turned down > on this topic, but I feel like I wasn't listened to, so > consider this your official warning, Rick: stop the > bombastic, insulting tone or you will be banned. Hello Brett (and all Group Members), Since it had become obvious to me, that the emotions surrounding this subject-matter present a high probability of turning toxic, I had made the decision to drop this thread last night -- there is no need for me to continually beat this dead horse, the community has made its decision. At this time, i want to thank everyone who participated, especially those who offered constructive replies. Also, I want everyone to know that my objections to type-hints are not exclusively based on emotion, no, I am still convinced that type-hints (as implemented) has the potential to undermine the very essence of what makes Python unique among all other languages -- but that's just my opinion, feel free to ignore if you don't agree. Out of respect for those who have so graciously volunteered their time to implement this feature, i pledge that i will not express my reservations regarding type-hints in this group *EVER* again. However, i will be composing a thoughtful and exhaustive dissent (regarding type-hints) and posting it publicly very soon. If anyone shares my reservations of type-hints (as implemented), please do not reply here, instead, contact me via email, so that we may discuss these matters privately. And as always, any discussions with me are 100% confidential. Thanks all And Happy Pythoning! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Mar 20 18:35:27 2016 From: mertz at gnosis.cx (David Mertz) Date: Sun, 20 Mar 2016 15:35:27 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <56EAD394.9080501@stoneleaf.us> References: <20160317154546.GF8022@ando.pearwood.info> <56EAD394.9080501@stoneleaf.us> Message-ID: +1 On Mar 17, 2016 8:56 AM, "Ethan Furman" wrote: > On 03/17/2016 08:45 AM, Steven D'Aprano wrote: > >> The standard site.py module adds pseudo-builtins: >> >> copyright >> credits >> exit >> help >> license >> quit >> >> >> I suggest one more: citation(). >> > > No opinion on exact implementation, but for the idea in general: > > +1 > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vito.detullio at gmail.com Sun Mar 20 19:23:37 2016 From: vito.detullio at gmail.com (Vito De Tullio) Date: Mon, 21 Mar 2016 00:23:37 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body References: <56EEE82A.6050505@mail.de> Message-ID: Andrew Barnert via Python-ideas wrote: > More generally, I think if this feature were to be added, "empty" is a > reasonable name. The idiomatic way to write it today is something like: > > elem = empty = object() > for elem in seq: > do_stuff(elem) > if elem is empty: > do_empty_seq_stuff() > > So you're basically looking for syntactic sugar that abbreviated "if ... > is empty:", right? Maybe I'm missing something, but shoulnd't be like if seq: for elem in seq: do_stuff(elem) else: do_empty_seq_stuff() a trivial code handling an empty list? -- By ZeD From robertc at robertcollins.net Sun Mar 20 19:27:36 2016 From: robertc at robertcollins.net (Robert Collins) Date: Mon, 21 Mar 2016 12:27:36 +1300 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> Message-ID: On 21 March 2016 at 12:23, Vito De Tullio wrote: > Andrew Barnert via Python-ideas wrote: > >> More generally, I think if this feature were to be added, "empty" is a >> reasonable name. The idiomatic way to write it today is something like: >> >> elem = empty = object() >> for elem in seq: >> do_stuff(elem) >> if elem is empty: >> do_empty_seq_stuff() >> >> So you're basically looking for syntactic sugar that abbreviated "if ... >> is empty:", right? > > Maybe I'm missing something, but shoulnd't be like > > if seq: > for elem in seq: > do_stuff(elem) > else: > do_empty_seq_stuff() > > a trivial code handling an empty list? seq = iter([]) will cause your example to not run do_empty_seq_stuff(). -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From vito.detullio at gmail.com Sun Mar 20 19:39:24 2016 From: vito.detullio at gmail.com (Vito De Tullio) Date: Mon, 21 Mar 2016 00:39:24 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body References: <56EEE82A.6050505@mail.de> Message-ID: Robert Collins wrote: >> Maybe I'm missing something, but shoulnd't be like >> >> if seq: >> for elem in seq: >> do_stuff(elem) >> else: >> do_empty_seq_stuff() >> >> a trivial code handling an empty list? > > seq = iter([]) > > will cause your example to not run do_empty_seq_stuff(). oh, ok. sorry, didn't think about the iterator object. ...thinking about it, is there a reason why the boolean interpretation of an empty iterator is true? I know it's not trivial to know if an iterator is empty (still, there are a pair of recipes online, like http://code.activestate.com/recipes/413614-testing-for-an-empty-iterator/ or the first part of the response at http://stackoverflow.com/a/3114423/273593 ) but shoulnd't be more "pythonic" to just have a "falsy" value for an "empty" iterator? -- By ZeD From vgr255 at live.ca Sun Mar 20 19:48:33 2016 From: vgr255 at live.ca (=?iso-8859-1?Q?=C9manuel_Barry?=) Date: Sun, 20 Mar 2016 19:48:33 -0400 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> Message-ID: > From Vito De Tullio > Sent: Sunday, March 20, 2016 7:39 PM > To: python-ideas at python.org > Subject: Re: [Python-ideas] Control Flow - Never Executed Loop Body > > ...thinking about it, is there a reason why the boolean interpretation of an > empty iterator is true? > > I know it's not trivial to know if an iterator is empty (still, there are a > pair of recipes online, like http://code.activestate.com/recipes/413614- > testing-for-an-empty-iterator/ or the first part of the response at > http://stackoverflow.com/a/3114423/273593 ) but shoulnd't be more > "pythonic" > to just have a "falsy" value for an "empty" iterator? Not trivial, indeed. Consider the following: def gen(): while random.choice([0, 1]): yield "spam" Is it empty? Is it not? You can't tell when it will be, and while that is a bad example (I hope something like that doesn't get used!), arbitrary generators can have side effects, or be empty under certain circumstances and not under others. That being said, we can maybe make the built-in iterators return a falsey value in such cases, and let third-party iterators handle that on their own, if they feel like doing that. Generators would always be true, but it's an opt-in solution, and custom iterators can already define their own __bool__ if they want. -Emanuel ~ Anything can be a duck if you try hard enough ~ From abarnert at yahoo.com Sun Mar 20 19:54:20 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 20 Mar 2016 16:54:20 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> Message-ID: <23C77B6A-960B-4B62-BA0B-955BB863D171@yahoo.com> On Mar 20, 2016, at 16:39, Vito De Tullio wrote: > > Robert Collins wrote: > >>> Maybe I'm missing something, but shoulnd't be like >>> >>> if seq: >>> for elem in seq: >>> do_stuff(elem) >>> else: >>> do_empty_seq_stuff() >>> >>> a trivial code handling an empty list? >> >> seq = iter([]) >> >> will cause your example to not run do_empty_seq_stuff(). > > oh, ok. sorry, didn't think about the iterator object. > > > ...thinking about it, is there a reason why the boolean interpretation of an > empty iterator is true? > > I know it's not trivial to know if an iterator is empty (still, there are a > pair of recipes online, like http://code.activestate.com/recipes/413614-testing-for-an-empty-iterator/ or the first part of the response at > http://stackoverflow.com/a/3114423/273593 ) but shoulnd't be more "pythonic" > to just have a "falsy" value for an "empty" iterator? It's not just "not trivial". The only way to do it with full generality (working for, e.g., generators) is to wrap the Iterator in a "peekable iterator" type (or something similar but more powerful, like tee). But that changes the semantics of the iterator: every time you consume element #n, it produces #n+1, not #n. And it's not hard to think of generators where that would be horribly inappropriate. For example, imagine a generator that's reading responses off a network socket. If you try to read response #n+1 when you haven't sent request #n+1 yet (because you're about to look at response #n), it'll fail, or block, or read incorrect information, or something else terrible. And there are other problems. For example, what if the generator needs to be able to handle g.throw; if you've replaced iterable with chain([peeked], iterable) or similar, that won't work. Of course the iterator protocol _could_ have been designed differently (consider C++, where iterators have separate operators to get the current value and to advance to the next one), which would have changed the design of generator functions and generator expressions and various other things that came later, so these particular problems would never have arisen. There are obvious advantages and disadvantages to each design. But Python chose one design a decade and a half ago, and a lot has been built on top of that design, and it's not going to change now. And, given that design, there's no way to make iterators peekable. Which means that it wouldn't be pythonic to make empty iterators falsey, because that's impossible to do without peeking, and I'm pretty sure the only reason the Zen doesn't say possible is better than impossible is that it's so obvious it doesn't need to be said. :) It's also worth noting that what you're asking for is really just a special case of LBYL vs. EAFP. Usually, it's better to just use the object and then see what happened, rather than checking in advance what will happen if you use it. Sometimes you can't avoid LBYL, or it's better because of the specifics of what you're doing, but in general, EAFP is the "default" way of thinking in pythonic code. From abarnert at yahoo.com Sun Mar 20 20:13:57 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 20 Mar 2016 17:13:57 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <23C77B6A-960B-4B62-BA0B-955BB863D171@yahoo.com> References: <56EEE82A.6050505@mail.de> <23C77B6A-960B-4B62-BA0B-955BB863D171@yahoo.com> Message-ID: On Mar 20, 2016, at 16:54, Andrew Barnert via Python-ideas wrote: > On Mar 20, 2016, at 16:39, Vito De Tullio wrote: >> >> Robert Collins wrote: >> >>>> Maybe I'm missing something, but shoulnd't be like >>>> >>>> if seq: >>>> for elem in seq: >>>> do_stuff(elem) >>>> else: >>>> do_empty_seq_stuff() >>>> >>>> a trivial code handling an empty list? >>> >>> seq = iter([]) >>> >>> will cause your example to not run do_empty_seq_stuff(). >> >> oh, ok. sorry, didn't think about the iterator object. >> >> >> ...thinking about it, is there a reason why the boolean interpretation of an >> empty iterator is true? Possibly a better way to put this than my other answer: When you really do need to deal with any arbitrary iterable, it's more Pythonic to use the "if element is empty" post-check. When you expect your input to be a collection* (as was clearly your intuition here), go ahead and make use of that, using exactly the "if seq:" code that you wrote. If you need to deal with an iterable that might be an iterator rather than a collection, but if so it's guaranteed to be safely peekable... Well, I don't think that will ever actually come up, but if it does, you can do the peek-and-stuff-back thing instead of the "if element is empty". I think it would probably be simpler and more readable to do the post-test, and it would also avoid having to get across to your reader the idea that you can handle both collections and safely-peekable iterators but not non-safely-peekable iterators, but if you have some reason for that precondition that you wanted to express anyway, maybe using the peek-and-stuff code helps get it across? * Or "reiterable" or "repeatable iterable" or "non-iterator iterable" or any of the other identical or closely parallel concepts. I don't want to restart the argument about which one is more useful here; they all make the right distinction in this case. From steve at pearwood.info Sun Mar 20 20:29:45 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 21 Mar 2016 11:29:45 +1100 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <87io0goq04.fsf@vostro.rath.org> References: <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> <87io0goq04.fsf@vostro.rath.org> Message-ID: <20160321002945.GT8022@ando.pearwood.info> On Sun, Mar 20, 2016 at 01:26:03PM -0700, Nikolaus Rath wrote: > On Mar 19 2016, Nick Coghlan wrote: > > The reason I suggest that approach is that most (all?) of us aren't > > research scientists, so we have no idea what typical conventions are > > for citations, nor how those conventions are changing. > > Which I believe makes it completely pointless to cite Python at all. As > far as I can see, nowadays citations are given for two reasons: > > 1. To give the reader a starting point to get more information on a > topic. > > 2. To formally acknowledge the work done by someone else (who ends up > with an increased number of citations for the cited publication, > which is unfortunately a crucial metric in most academic hiring and > evaluation processes). > > In case of Python, an explicit citation thus adds nothing. I'm afraid I don't understand your reasoning here. Both of your two reasons apply: a citation to Python gives the reader a starting point to get more information, and it formally acknowledges the work done by others. So a citation adds exactly the two things that you say citations are used for. You might feel that everybody knows how to use google, and that a formal acknowledgement is pointless because nobody cares, but that's a value judgement about the usefulness of what the citation adds, not whether it adds them or not. Useful or not, scientific papers do cite the software they use. This proposal is about making it easier for people to do so, not about changing their behaviour. For the record, I think citations of software are useful, but other arguments have convinced me that for the time being at least, this is best handled as documentation rather than code. More on that shortly. -- Steve From steve at pearwood.info Sun Mar 20 20:47:08 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 21 Mar 2016 11:47:08 +1100 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> Message-ID: <20160321004708.GA12526@ando.pearwood.info> On Thu, Mar 17, 2016 at 04:54:41PM -0700, Nathaniel Smith wrote: > I think it would be premature for the stdlib to try to standardize > machine readable citation metadata for third-party packages, or even a > general API for accessing them. There are a lot of complex issues in > this space that are still being explored by third-party packages like > duecredit: > https://github.com/duecredit/duecredit Thanks for the link, although that is far and beyond anything that I'm suggesting. > (Notice that the citation() function in R actually does some rather > complicated things and returns a rather complicated object: > https://stat.ethz.ch/R-manual/R-devel/library/utils/html/citation.html > https://stat.ethz.ch/R-manual/R-devel/library/utils/html/bibentry.html > ) Given this, and comments from others (esp. Guido and Nick) I think it makes sense to start with documentation. One advantage of changing the docs is that it can apply to all versions, not just 3.6. I've raised a tracker item here: http://bugs.python.org/issue26597 Do we have consensus that this should be a separate page in the documentation, under "Meta Information"? The only other relevant place I think would be a FAQ, except I'm not sure that it is *quite* frequent enough :-) https://docs.python.org/3/index.html (scroll right to the bottom) Thanks for the feedback from everyone! -- Steve From steve at pearwood.info Sun Mar 20 21:32:37 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 21 Mar 2016 12:32:37 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> Message-ID: <20160321013234.GB12526@ando.pearwood.info> On Sun, Mar 20, 2016 at 01:16:50PM -0700, Andrew Barnert via Python-ideas wrote: > On Mar 20, 2016, at 11:12, Sven R. Kunze wrote: > > > > Issues > > People I talked to suggested "else" as an alternative to "empty" because "empty is not quite clear". Unfortunately, "else" is already taken and is itself not quite clear. "then" has been proposed as an alternative for "else" in an attempt for proper understanding of "else". If that would be an accepted proposal, it would make room for "else" to be used for the usage of the "empty keyword proposed here. > > Besides the backward compatibility issue, changing "else" to "then" > would be horribly confusing. I suspect anyone who thinks that would be > an improvement doesn't actually understand or like for...else, and > they'd be happier just eliminating it, not renaming it. "then" is my idea, and not only do I understand "for...else", but I like it and use it, and certainly don't want to eliminate it. I just hate the keyword used. Let me just start by saying that I realise that actually changing "for...else" to "for...then" is at least 8 years too late, so I know this is a non-starter. It would be particularly confusing to change "else" to "then" AND add a new "else" with different semantics at the same time. > An else clause is testing that no break was hit inside the loop. Look > at a typical example: That's one way of thinking about it. But I don't think it is helpful to think of it as setting an invisible flag "a break was hit inside the loop", and then testing it. I think that a more natural way to think about it is that "break" jumps out of the entire for...else compound statement. This has the big advantage that it actually matches what the byte code does in all the versions I've looked at. The "else" block is *unconditionally* executed after the "for" block. There's no "if not flag". Hence "then" is a better name for the construct: "for ... then ...". "break" doesn't set a flag, it jumps right out of the "for...else" statement altogether, not just out of the loop part, but the "else" part as well. (As I said, this matches what the byte code actually does.) As a beginner, I spent a very long time completely perplexed and confused by the behaviour of "for...else" because I understood it to mean "run the for block, *or else* run the else block". In other words, I understood from the keyword that "else" ran *if the for block didn't*, i.e. when the loop iterator is empty. A perfectly natural mistake to make, and I'm not the only person to have made it. This (wrong, incorrect) interpretation matches the most common and familiar use of "else", namely in if...else statements: if ...: a else: b You can get a, or b, but not both. In English, "else" represents an alternative. This does not come even close to matching the behaviour of for...else, which (in the absense of a "break" executes a *and* b, rather than a *or* b: for ...: a else: b I'm not Dutch, but I think that "else" is not a good name for a block of code which unconditionally executes after the for/while/try block. I think it's also unfortunately that it often looks like an indentation error: for x in seq: do_stuff() if condition: break else: do_more() I've seen people "fix" the indentation on code like this and wonder why the code then does the wrong thing. But, like I said, we're stuck with it, and I'm not seriously proposing a change. I think we'd be better off now if the keyword had been "then" from the beginning, but the pain of changing it *now* outweighs the benefit. -- Steve From steve at pearwood.info Sun Mar 20 21:44:33 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 21 Mar 2016 12:44:33 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56EEE82A.6050505@mail.de> References: <56EEE82A.6050505@mail.de> Message-ID: <20160321014433.GC12526@ando.pearwood.info> On Sun, Mar 20, 2016 at 07:12:58PM +0100, Sven R. Kunze wrote: > for item in my_iterator: > # do per item > empty: > # do something else "empty" is a poor choice, as I expect that it will break a lot of code that already uses it as a variable. if empty: ... > *Keyword* > Keywords under consideration have been in (an attempt of dispassionate) > order of preference: > 1) empty -> most obvious, most people responded with a solution to solve > the intended problem > 2) else -> already taken but to some would be the preferred one > 3) or -> as alternative if a new keyword is too much I'm not too keen on the look of "or" for this: for x in iterable: block or: alternative but I think it's the least worst of the proposed keywords. > 4) except -> as alternative if a new keyword is too much I think that "except" completely fails to describe what the keyword does, and it will give some people the misunderstanding that it catches errors in the body of the for-loop. -- Steve From rosuav at gmail.com Sun Mar 20 21:52:20 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 21 Mar 2016 12:52:20 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <20160321013234.GB12526@ando.pearwood.info> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> Message-ID: On Mon, Mar 21, 2016 at 12:32 PM, Steven D'Aprano wrote: > That's one way of thinking about it. But I don't think it is helpful to > think of it as setting an invisible flag "a break was hit inside the > loop", and then testing it. I think that a more natural way to think > about it is that "break" jumps out of the entire for...else compound > statement. This has the big advantage that it actually matches what the > byte code does in all the versions I've looked at. Exactly. I came across this situation in a C++ program, and ended up writing the 'break' as a 'goto' - something like this, but with better names: for (int i=0;i References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> Message-ID: On Mar 20, 2016, at 18:32, Steven D'Aprano wrote: > >> On Sun, Mar 20, 2016 at 01:16:50PM -0700, Andrew Barnert via Python-ideas wrote: >>> On Mar 20, 2016, at 11:12, Sven R. Kunze wrote: >>> >>> Issues >>> People I talked to suggested "else" as an alternative to "empty" because "empty is not quite clear". Unfortunately, "else" is already taken and is itself not quite clear. "then" has been proposed as an alternative for "else" in an attempt for proper understanding of "else". If that would be an accepted proposal, it would make room for "else" to be used for the usage of the "empty keyword proposed here. >> >> Besides the backward compatibility issue, changing "else" to "then" >> would be horribly confusing. I suspect anyone who thinks that would be >> an improvement doesn't actually understand or like for...else, and >> they'd be happier just eliminating it, not renaming it. > > "then" is my idea, and not only do I understand "for...else", but I like > it and use it, and certainly don't want to eliminate it. I just hate the > keyword used. > > Let me just start by saying that I realise that actually changing > "for...else" to "for...then" is at least 8 years too late, so I know > this is a non-starter. It would be particularly confusing to change > "else" to "then" AND add a new "else" with different semantics at the > same time. > >> An else clause is testing that no break was hit inside the loop. Look >> at a typical example: > > That's one way of thinking about it. But I don't think it is helpful to > think of it as setting an invisible flag "a break was hit inside the > loop", and then testing it. I think that a more natural way to think > about it is that "break" jumps out of the entire for...else compound > statement. This has the big advantage that it actually matches what the > byte code does in all the versions I've looked at. The way I've always thought about it is that it's more like a try/else than an if/else: it runs unless you've jumped out of the whole for statement (via break instead of raise). I think Nick Coghlan has a blog post that explains this very nicely, showing both the novice-understandable intuition and how a theoretical Python implementation could implement it (which is close to how CPython actually does, but maybe simpler), so I won't go over the details here. From what you write later, you also don't like the naming of try/else--in which case it's not surprising that you don't like the naming of for/else. > The "else" block is *unconditionally* executed after the "for" block. The key difference is whether you treat "unless jumped out of" as meaning "unconditionally". I can see the sense of your way of looking at it (there's certainly a sense in which "while True:" loops "unconditionally", even if you have a break or return inside the loop, right?), so I understand where you're coming from. However, I still think _most_ people who don't like for/else don't actually understand it. But of course I can see that as another argument against the naming--if it's confused this many people over the last 20 years, maybe it's inherently confusing? At any rate, as you say at the end, there's no point arguing about this. The "else" clause isn't going to be renamed, and, even if it were, it wouldn't be reused for this new purpose, so that alternative to "empty" isn't worth discussing. (If I were designing a new Python-like language, I might rethink using "else"--but I don't think I'd use "then", or just leave it out the way Boo did.) From leewangzhong+python at gmail.com Sun Mar 20 22:41:02 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Sun, 20 Mar 2016 22:41:02 -0400 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> Message-ID: On Mar 20, 2016 10:10 PM, "Andrew Barnert via Python-ideas" < python-ideas at python.org> wrote: > > On Mar 20, 2016, at 18:32, Steven D'Aprano wrote: > > > >> On Sun, Mar 20, 2016 at 01:16:50PM -0700, Andrew Barnert via Python-ideas wrote: > >>> On Mar 20, 2016, at 11:12, Sven R. Kunze wrote: > >>> > >>> Issues > >>> People I talked to suggested "else" as an alternative to "empty" because "empty is not quite clear". Unfortunately, "else" is already taken and is itself not quite clear. "then" has been proposed as an alternative for "else" in an attempt for proper understanding of "else". If that would be an accepted proposal, it would make room for "else" to be used for the usage of the "empty keyword proposed here. > >> > >> Besides the backward compatibility issue, changing "else" to "then" > >> would be horribly confusing. I suspect anyone who thinks that would be > >> an improvement doesn't actually understand or like for...else, and > >> they'd be happier just eliminating it, not renaming it. > > > > "then" is my idea, and not only do I understand "for...else", but I like > > it and use it, and certainly don't want to eliminate it. I just hate the > > keyword used. > > > > Let me just start by saying that I realise that actually changing > > "for...else" to "for...then" is at least 8 years too late, so I know > > this is a non-starter. It would be particularly confusing to change > > "else" to "then" AND add a new "else" with different semantics at the > > same time. > > > >> An else clause is testing that no break was hit inside the loop. Look > >> at a typical example: > > > > That's one way of thinking about it. But I don't think it is helpful to > > think of it as setting an invisible flag "a break was hit inside the > > loop", and then testing it. I think that a more natural way to think > > about it is that "break" jumps out of the entire for...else compound > > statement. This has the big advantage that it actually matches what the > > byte code does in all the versions I've looked at. > > The way I've always thought about it is that it's more like a try/else than an if/else: it runs unless you've jumped out of the whole for statement (via break instead of raise). I think Nick Coghlan has a blog post that explains this very nicely, showing both the novice-understandable intuition and how a theoretical Python implementation could implement it (which is close to how CPython actually does, but maybe simpler), so I won't go over the details here. > > From what you write later, you also don't like the naming of try/else--in which case it's not surprising that you don't like the naming of for/else. I read the "else:", when attached to "try:", as "alternative to except:" (which makes sense even if no explicit "except" block exists). In "if-else", it means "alternative to if:" (or "if: and elif:"). If "switch-case" existed, "else:" would mean "alternative to case:". But in "for:"/"while:", there is no block keyword that the "else:" is the alternative to. It definitely isn't the alternative to the loop block itself, but that's probably the most common wrong intuition. (Disclaimer: I use "for-else" and "try-else" when I can. I don't dislike them at all. I'm just trying to explain the newbie confusion.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Mar 20 23:19:54 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 20 Mar 2016 23:19:54 -0400 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <20160321013234.GB12526@ando.pearwood.info> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> Message-ID: On 3/20/2016 9:32 PM, Steven D'Aprano wrote: > I understood from the keyword that "else" ran *if the for block > didn't*, i.e. when the loop iterator is empty. > A perfectly natural mistake to make, Why is the truth a mistake? > and I'm not the only person to have made it. > This (wrong, incorrect) interpretation To me, this is disrespectful of other people. > matches the most common > and familiar use of "else", namely in if...else statements: > if ...: > a > else: > b > > > You can get a, or b, but not both. In English, "else" represents an > alternative. This does not come even close to matching the behaviour of > for...else, which (in the absense of a "break" executes a *and* b, > rather than a *or* b: Block a may never be executed. In any case, once it is, the loop starts over, the test is repeated, and either a or b is executed. > > for ...: > a > else: > b I see it differently. For both while and for, block b is the alternative to executing a, when the condition (explicit in 'while', implicit in 'for) is false, just as in an 'if' statement. I know I am not going to convince you, but please don't call me 'wrong' for seeing why 'else' makes sense and for understanding how 'while' is constructed from 'if' and 'jump'. The essential difference between an if statement and a while statement is the jump back that causes a repeat of the test. -- Terry Jan Reedy From stephen at xemacs.org Mon Mar 21 01:20:39 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 21 Mar 2016 14:20:39 +0900 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <87io0goq04.fsf@vostro.rath.org> References: <20160317154546.GF8022@ando.pearwood.info> <87bn6doyiw.fsf@thinkpad.rath.org> <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> <87io0goq04.fsf@vostro.rath.org> Message-ID: <22255.33959.446972.167589@turnbull.sk.tsukuba.ac.jp> Nikolaus Rath writes: > In case of Python, an explicit citation thus adds nothing. Relevant > information is easily found by any search engine, and if Nick is > right than no one stands to gain anything from the citations > either. While Steven gave a good answer, I'd like to provide a social science researcher's slant. Python gets a mention. As the politicians say, "Use all the epithets you like, but spell my name right!" Believe it or not, Python is *not* a "household word" in academic business and economics fields. A lot of us are teaching Python by preference, but there's still a large overhang of (mostly, but not invariably, older) researchers who only know Java, C/C++, or FORTRAN[sic] as "scientific" programming languages. To researchers, a citation is a pointer to an authoritative source, and specifically authoritative in specifying the *cited reference*, not current versions or random "hits". To those who don't know anything about Python, the fact that there is an authoritative citation gives them some confidence that Python itself is an ongoing entity. OTOH, web searchs are not going to give you authoritative sources. The reasons for changing citation practice are not changes in these social relationships, but rather (1) changing publication channels requires changes of "pointers", and (in the case of many web pages which are generated dynamically) a more precise date of publication (ie, the date viewed) (2) some pointers are more efficient and accurate than others, and they are being introduced as alternatives (often preferred) to the traditional ones. From stephen at xemacs.org Mon Mar 21 04:01:20 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 21 Mar 2016 17:01:20 +0900 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> Message-ID: <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> ?manuel Barry writes: > > From Vito De Tullio > > shouldn't it be more "pythonic" to just have a "falsy" value for > > an "empty" iterator? > > Not trivial, indeed. Consider the following: > > def gen(): > while random.choice([0, 1]): > yield "spam" AFAICS the OP's idea has a trivial and efficient expansion: for item in iterable: # implements __next__ # do for each item empty: # do exactly when iterable raises StopIteration on the first pass becomes item = _sentinel = object() for item in iterable: # do for each item if item is _sentinel: # do exactly when iterable raises StopIteration on the first pass (works in light testing but maybe I've missed something). But to me that means saving one line and a few characters in the trailer clause is not worth syntax. > Is it empty? Is it not? You can't tell when it will be, and while that is a > bad example (I hope something like that doesn't get used!), Replace random with polling an input source or accessing a database, and you get the same nondeterminism. Steve From rosuav at gmail.com Mon Mar 21 04:06:29 2016 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 21 Mar 2016 19:06:29 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> Message-ID: On Mon, Mar 21, 2016 at 7:01 PM, Stephen J. Turnbull wrote: > item = _sentinel = object() > for item in iterable: > # do for each item > if item is _sentinel: > # do exactly when iterable raises StopIteration on the first pass What if 'iterable' is locals().values()? Can you, with perfect reliability, recognize that case? AIUI this is exactly why next() and __next__() are defined to "return a value or raise", rather than "return a value or return a magic no-more-values value", because there's always the possibility that the no-more-values value is a legitimately-yielded value. Maybe this situation isn't important enough or common enough to justify dedicated syntax, but it's definitely a possibility. ChrisA From victor.stinner at gmail.com Mon Mar 21 06:38:13 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 21 Mar 2016 11:38:13 +0100 Subject: [Python-ideas] https://docs.python.org/fr/ ? In-Reply-To: <56ED7EBD.5050804@palard.fr> References: <56ED7EBD.5050804@palard.fr> Message-ID: Hi, 2016-03-19 17:30 GMT+01:00 Julien Palard : > The french translation of the Python Documentation [1][2] has translated 20% > of the pageviews of docs.python.org. I think it's the right moment to push > it do docs.python.org. So there's some questions ! And I'd like feedback. Nice. > TL;DR (with my personal choices): > - URL may be "http://docs.python.org/fr/" > - ... > - existing script to build doc may be patched to build translations If we choose to officially have translated documentation, I suggest to ensure that it has the same content thant the english. I don't want to read an outdated translated doc. I mean that all doc (en+fr) must be recompiled when a patch is merged. > - ... > - untranslated strings may be visually marked as so I don't think that it's needed. It's quite easy to notice untranslated parts... > ### Dropping the default locale of a language > > I personally think we should not show the region in case it's redundant: so > to use "fr" instead of "fr-FR", "de" instead of "de-DE", but keeping the > possibility to use a locale code when it's not redundant like for "pt-br" or > "de-AT" (German ('de') as used in Austria ('AT')). IMHO we should see what PHP did, since PHP has a long history with documentation translation. http://php.net/manual/en/function.strlen.php http://php.net/manual/fr/function.strlen.php http://php.net/manual/pt_BR/function.strlen.php So /fr/ for french, but /pt_BR/ for Brazilian Portuguese. We can use something similar for Python. They chose to have /en/ in the URL of the original (english) documentation, *but* http://php.net/strlen is the translated doc. I would really prefer to not move on more time the Python doc (add /en/ to the URL), since *a lot* of articles have references to the Python doc. Or if you really want to do that, please add redirection (forever?). I'm lazy and I really like writing http://php.net/ to get the documentation of a function. It's much more difficult to get the doc of len in the Python doc... > ### gettext VS IETF language tag format > > gettext goes by using an underscore between language and locale [3] and IETF > goes by using a dash [4][5]. PHP chose an underscore, it's more common in URLs. I prefer pt_BR. Anyway, users will use a dropdown list rather than explicitly writing the URL, no? > # Where to host the translated files > > Currently we're hosting the *po* files in the afpy's (Francophone > association for python) [6] github [1] but it may make sense to use (in the > generation scripts) a more controlled / restricted clone in the python > github, at least to have a better view of who can push on the documentation. > > We may want to choose between aggregating all translations under the same > git repository but I don't feel it's useful. Do you plan to use a web UI to allow anyone to translate easily the doc? It may have an impact on the authentication system. Should such contribution be pushed directly and become a draft which requires a review? > ## Crosslinking between existing translations > > Once the translations are on `docs.python.org`, crosslinks may be > established so people on a version can be aware of other version, and easily > switch to them. I'm not a UI/UX man but I think we may have a select box > right before the existing select box about version, on the top-left corner. > Right before because it'll reflect the path: /fr/3.5/ -> [select box > fr][select box 3.5]. Usually (on web sites, the language is chosen at the top right. Victor From steve at pearwood.info Mon Mar 21 09:29:52 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 22 Mar 2016 00:29:52 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> Message-ID: <20160321132952.GD12526@ando.pearwood.info> On Sun, Mar 20, 2016 at 11:19:54PM -0400, Terry Reedy wrote: > On 3/20/2016 9:32 PM, Steven D'Aprano wrote: > > > I understood from the keyword that "else" ran *if the for block > > didn't*, i.e. when the loop iterator is empty. > > A perfectly natural mistake to make, > > Why is the truth a mistake? It's not the truth. The else block does *not* only run if the for block doesn't run. It runs regardless of whether the loop iterable is empty or not. > > and I'm not the only person to have made it. > > This (wrong, incorrect) interpretation > > To me, this is disrespectful of other people. It's a statement of fact. I have seen other people make the same mistake I made: interpreting the "else" clause as only running if the loop iterable is empty. People have been mislead by the keyword. What's disrespectful about pointing this out? > > matches the most common > > and familiar use of "else", namely in if...else statements: > > >if ...: > > a > >else: > > b > > > > > >You can get a, or b, but not both. In English, "else" represents an > >alternative. This does not come even close to matching the behaviour of > >for...else, which (in the absense of a "break" executes a *and* b, > >rather than a *or* b: > > Block a may never be executed. In any case, once it is, the loop starts > over, the test is repeated, and either a or b is executed. It's an IF...else statement. There's no loop. > >for ...: > > a > >else: > > b > > I see it differently. For both while and for, block b is the > alternative to executing a, when the condition (explicit in 'while', > implicit in 'for) is false, just as in an 'if' statement. That's not what for...else does. Try it with an empty sequence and a non-empty sequence: for x in [1,2]: print("inside for block") else: print("inside else block") for x in []: print("inside for block") else: print("inside else block") BOTH the empty and non-empty cases print "inside else block". It is certainly not "the alternative". -- Steve From chris.barker at noaa.gov Mon Mar 21 12:34:21 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 21 Mar 2016 09:34:21 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <20160321132952.GD12526@ando.pearwood.info> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> Message-ID: I've taught enough and I'm sure everyone else here has too, to know that the "else" in a for loop in non-intuitive to a lot of folks. And maybe a different keyword would have been clearer. But it is what it is, could we keep this discussion to the proposed addition? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Mar 21 12:31:23 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 21 Mar 2016 09:31:23 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> Message-ID: On Mon, Mar 21, 2016 at 1:01 AM, Stephen J. Turnbull wrote: > AFAICS the OP's idea has a trivial and efficient expansion: > I'll take your word for it that this is efficient, and it is trivial in amount of code to write, but it is not the least bit trivial to read or write. It took me a good while to understand what it did, and how it works, and I never would have thought of it myself. > item = _sentinel = object() > for item in iterable: > # do for each item > if item is _sentinel: > # do exactly when iterable raises StopIteration on the first pass > I would have done the klunky thing: _sentinal = False for item in iterable: _sentinel = True # do something if not _sentinel # do something else if the loop never ran almost as compact, less efficient and enough to make me see why the OP want's something new. Howver, the motivating example was "the difference between *nothing**as error* and *nothing** as really empty*, GUI/Web interfaces display an explicit notice in case of empty lists to signify the correct execution of a process/action/search or the other case." Sure -- but this seem to be mixing validation with processing code -- you really want to validate your inputs first. AND: this would mostly be containers, rather than iterables, so the idiom of: if input: for item in input: # do something else: raise EmptyInputError works fine. All that being said: how about "elempty"?, to go with elif? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Mar 21 14:29:02 2016 From: brett at python.org (Brett Cannon) Date: Mon, 21 Mar 2016 18:29:02 +0000 Subject: [Python-ideas] CoC violation warning (was: Re: PEP 484 evolution) In-Reply-To: References: <5f8595e0-dd90-4377-8de5-9b2f2a98fd45@googlegroups.com> <812a2397-dd00-4e59-81b9-7be44dff6df7@googlegroups.com> <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> Message-ID: On Sun, 20 Mar 2016 at 13:35 Rick Johnson wrote: > On Sunday, March 20, 2016 at 12:37:48 PM UTC-5, Brett Cannon wrote: > > I tried to be calm about asking the tone to be turned down > > on this topic, but I feel like I wasn't listened to, so > > consider this your official warning, Rick: stop the > > bombastic, insulting tone or you will be banned. > > Hello Brett (and all Group Members), > > Since it had become obvious to me, that the emotions > surrounding this subject-matter present a high probability > of turning toxic, I had made the decision to drop this > thread last night -- there is no need for me to continually > beat this dead horse, the community has made its decision. > Thank you for recognizing that your emotions were getting the better of you. But please do realize, Rick, that the warning does stand and it is not relegated to just this discussion thread, so please take the time in the future to think through your emails to this list before you send them as I would prefer to not have to ban you (or anyone else for that matter; banning people is a bit stressful for me as I don't exactly enjoy that responsibility, but I will deal with the stress if I feel anyone causes people to dread reading emails from this list due to their behaviour). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Mon Mar 21 14:37:56 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 21 Mar 2016 19:37:56 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> Message-ID: <56F03F84.7050907@gmail.com> Le 21/03/2016 17:34, Chris Barker a ?crit : > I've taught enough and I'm sure everyone else here has too, to know that > the "else" in a for loop in non-intuitive to a lot of folks. And maybe a > different keyword would have been clearer. But it is what it is, could > we keep this discussion to the proposed addition? +1 I would love "else" semantics to be changed as well, but we can't. What about: for x in stuff: foo(x) or: bar() It's a bit weird when you read it the first time, but: - It conveys pretty well the intent. - While it still closes to the original "or" semantics, you can't confuse this syntax with the other syntax for "or". - I can't think of a use case where the parser can find that ambigious. - It's short. > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From mike at selik.org Mon Mar 21 15:27:52 2016 From: mike at selik.org (Michael Selik) Date: Mon, 21 Mar 2016 19:27:52 +0000 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> Message-ID: On Mon, Mar 21, 2016 at 12:35 PM Chris Barker wrote: > I've taught enough and I'm sure everyone else here has too, to know that > the "else" in a for loop in non-intuitive to a lot of folks. And maybe a > different keyword would have been clearer. But it is what it is, could we > keep this discussion to the proposed addition? > I'm not sure the keyword being "else" is the real problem. It's certainly correlated, but I've noticed a similar confusion for "finally" and "else" in "try" blocks, where the keywords are quite natural. The confusion might in fact simply be a difficulty with the concept "do this if the loop completed without breaking" rather than a difficulty linking that concept to the word "else". Either way, that's a tangent, and I agree that discussion on "else" should be (mostly) separate from a discussion on adding a new keyword. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vito.detullio at gmail.com Mon Mar 21 16:57:03 2016 From: vito.detullio at gmail.com (Vito De Tullio) Date: Mon, 21 Mar 2016 21:57:03 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> Message-ID: Chris Angelico wrote: >> item = _sentinel = object() >> for item in iterable: >> # do for each item >> if item is _sentinel: >> # do exactly when iterable raises StopIteration on the first pass > > What if 'iterable' is locals().values()? Can you, with perfect > reliability, recognize that case? AIUI this is exactly why next() and > __next__() are defined to "return a value or raise", rather than > "return a value or return a magic no-more-values value", because > there's always the possibility that the no-more-values value is a > legitimately-yielded value. so, the "correct" way to handle this is something like try: e = next(iterable) except StopIteration: empty_suite() # there is no element else: main_suite(e) # handle the first element for e in iterable: main_suite(e) # handle the rest of the elements ? apart for the readability you need to "copy" the same code two times (or refactor in a function) -- By ZeD From desmoulinmichel at gmail.com Mon Mar 21 19:06:32 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 00:06:32 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> Message-ID: <56F07E78.9020309@gmail.com> Itertools is great, and some functions in it are more used than others: - islice; - chain; - dropwhile, takewhile; Unfortunatly many people don't use them because they don't know it exists, but also are not aware of the importance of generators in Python and all in all, the central place iteration has in the language. But I must confess that after 12 years of Python, I always delay the use of it as well: - I have to type the import in every single module and shell session (ok I got PYTHONSTARTUP setup to autoimport, but most people don't). - All functions fell verbose for such a common use case. - If I start to use it, I can say good by to simpler syntaxes such as [] and +. - It always take me a minutes to get dropwhile/takewhile right. They works the opposite way of my brain. The changes I'm going to propose do not add new syntax to Python, but yet would streamline the use of this nice tool and blend it into the language core. Make slicing accept callables ============================= One day someone asked me something similar to: "I got a list of numbers, how do I filter this list so that I stop when numbers are bigger than 4." So I said: print([x for x in numbers if x > 4]) But then he said: "No, I want to stop reading any number I encounter after the first x > 4." "Oh". Then: import itertools def stop(element): return not element > 4 print(list(itertools.takewhile(stop, numbers)) I actually got it wrong 2 times, first I forgot the "not", then I mixed up the parameters in takewhile. I was going to then introduce lambda but my colleagues looked at me in a sad way after glancing at the code and I backed up. So my first proposal is to be able to do: def stop(element): return element > 4 print(numbers[:stop]) It's quite pythonic, easy to understand : the end of the slice is when this condition is met. Any not the strange way takewhile work, which is "carry on as long as this condition is met". We could also extend itertools.islice to accept such parameter. Slicing any iterable ====================== Now, while I do like islice, I miss the straigthforwardness of [:]: from itertools import islice def func_accepting_any_iterable(foo): return bar(islice(foo, 3, 7)) It's verbose, and renders the [3:7] syntaxe almost useless if you don't have control over the code creating the iterable you are going to process since you don't know what it's going to be. So the second proposal is to allow: def func_accepting_any_iterable(foo): return bar(foo[3:7]) The slicing would then return a list if it's a list, a typle if it's a tuple, and a islice(generator) if it's a generator. If somebody uses a negative index, it would then raises a ValueError like islice. This would make duck typing and iteration even easier in Python. Chaining iterable ================== Iterating on heterogenous iterable is not clear. You can add lists with lists and tuples with tuples, but if you need more, then you need itertools.chain. Few people know about it, so I usually see duplicate loops and conversion to lists/tuples. So My first proposal is to overload the "&" operator so that anything defining __iter__ can be used with it. Then you can just do: chaining = "abc" & [True, False] & (x * x for x in range(10)) for element in chaining: print(element) Instead of: from itertools import chain chaining = chain("abc", [True, False], (x * x for x in range(10))) for element in chaining: print(element) From rosuav at gmail.com Mon Mar 21 19:36:31 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 22 Mar 2016 10:36:31 +1100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F07E78.9020309@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: On Tue, Mar 22, 2016 at 10:06 AM, Michel Desmoulin wrote: > Itertools is great, and some functions in it are more used than others: > > - islice; > - chain; > - dropwhile, takewhile; >... > > The changes I'm going to propose do not add new syntax to Python, but > yet would streamline the use of this nice tool and blend it into the > language core. You're not the first to ask for something like this :) Let's get *really* specific about semantics, though - and particularly about the difference between iterables, iterators, and generators. > Make slicing accept callables > ============================= > > So my first proposal is to be able to do: > > def stop(element): > return element > 4 > print(numbers[:stop]) > > It's quite pythonic, easy to understand : the end of the slice is when > this condition is met. Any not the strange way takewhile work, which is > "carry on as long as this condition is met". > > We could also extend itertools.islice to accept such parameter. This cannot be defined for arbitrary iterables, unless you're proposing to mandate it in some way. (It conflicts with the way a list handles slicing, for instance.) Even for arbitrary iterators, it may be quite tricky (since iterators are based on a protocol, not a type); but maybe it would be worth proposing an "iterator mixin" that handles this for you, eg: class IteratorOperations: def __getitem__(self, thing): if isinstance(thing, slice): if has_function_in_criteria(slice): return self.takeuntil(s.start, s.stop) return itertools.islice(...) def takeuntil(self, start, stop): val = next(self) while start is not None and not start(val): val = next(self) while stop is None or not stop(val): yield val val = next(self) As long as you inherit from that, you get these operations made available to you. Now, if you're asking this about generators specifically, then it might be possible to add this (since all generators are of the same type). It wouldn't be as broad as the itertools functions (which can operate on any iterable), but could be handy if you do a lot with gens, plus it's hard to subclass them. > Slicing any iterable > ====================== > > So the second proposal is to allow: > > def func_accepting_any_iterable(foo): > return bar(foo[3:7]) > > The slicing would then return a list if it's a list, a typle if it's a > tuple, and a islice(generator) if it's a generator. If somebody uses a > negative index, it would then raises a ValueError like islice. > > This would make duck typing and iteration even easier in Python. Again, while I am sympathetic to the problem, it's actually very hard; islice always returns the same kind of thing, but slicing syntax can return all manner of different things, because it's up to the object on the left: >>> range(10)[3:7] range(3, 7) >>> "Hello, world!"[3:7] 'lo, ' >>> [1, 4, 2, 8, 5, 7, 1, 4, 2, 8, 5, 7][3:7] [8, 5, 7, 1] >>> memoryview(b"Hello, world!")[3:7] You don't want these to start returning islice objects. You mentioned lists, but other types will also return themselves when sliced. Possibly the solution here is actually to redefine object.__getitem__? Currently, it simply raises TypeError - not subscriptable. Instead, it could call iter() on itself, and then attempt to islice it. That would mean that the TypeError would change to "is not iterable" (insignificant difference), anything that already defines __getitem__ will be unaffected (good), and anything that's iterable but not subscriptable would automatically islice itself (potentially a trap, if people don't know what they're doing). > Chaining iterable > ================== > > Iterating on heterogenous iterable is not clear. > > You can add lists with lists and tuples with tuples, but if you need > more, then you need itertools.chain. Few people know about it, so I > usually see duplicate loops and conversion to lists/tuples. > > So My first proposal is to overload the "&" operator so that anything > defining __iter__ can be used with it. > > Then you can just do: > > chaining = "abc" & [True, False] & (x * x for x in range(10)) > for element in chaining: > print(element) > > Instead of: > > from itertools import chain > chaining = chain("abc", [True, False], (x * x for x in range(10))) > for element in chaining: > print(element) Again, anything involving operators is tricky, since anything can override its handling. But if you require that the first one be a specific iterator class, you can simply add __and__ to it to do what you want: class iter: iter = iter # snapshot the default 'iter' def __init__(self, *args): self.iter = self.iter(*args) # break people's minds def __iter__(self): return self def __next__(self): return next(self.iter) def __and__(self, other): yield from self.iter yield from other Okay, so you'd probably do it without the naughty bits, but still :) As long as you call iter() on the first thing in the chain, everything else will work. ChrisA From rob.cliffe at btinternet.com Mon Mar 21 19:53:03 2016 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 21 Mar 2016 23:53:03 +0000 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F03F84.7050907@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F03F84.7050907@gmail.com> Message-ID: <56F0895F.2090009@btinternet.com> On 21/03/2016 18:37, Michel Desmoulin wrote: > > Le 21/03/2016 17:34, Chris Barker a ?crit : >> I've taught enough and I'm sure everyone else here has too, to know that >> the "else" in a for loop in non-intuitive to a lot of folks. And maybe a >> different keyword would have been clearer. But it is what it is, could >> we keep this discussion to the proposed addition? > +1 I would love "else" semantics to be changed as well, but we can't. > > What about: > > for x in stuff: > foo(x) > or: > bar() The trouble with "or" or "else" after a for-loop is that it suggests (following English usage) alternative actions: Either execute this for-loop, or do something else. If I were designing Python 4, I might opt for for x in stuff: foo(x) ifnobreak: bar() Pro: "ifnobreak" is at least explicit, and not particularly likely to clash with an already-used variable name. Con: "ifnobreak" is too long and wordy. But I can't think of anything better. "aftercomplete", "whencomplete" are just as bad. Hell, I'm coming round to "then". What about allowing "then" to have an indent between that of the for-statement and the for-body: for x in stuff(): foo(x) then: bar() Of course, you still have to learn it, like all idioms, but having learned it, isn't it a bit more readable? The idea could be extended to other suites, conveying that everything indented is associated with the initial statement of the suite: try: foo() except SomeException: bar() Rob Cliffe From desmoulinmichel at gmail.com Mon Mar 21 19:59:58 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 00:59:58 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: <56F08AFE.2040501@gmail.com> The solution I'm currently using involves having a class called g, and everytime I want to manipulate an iterable, I just wrap it in g(). Then I got an object with all those semantics (and actually a lot more). Maybe we can make only those things apply to the objects returned by iter() ? (Also, about slicing accepting callable, actually g() goes a bit overboard and accept any object. If the object is not an int or callable, then it's used as a sentinel value. Not sure if I should speak about that here.) Le 22/03/2016 00:36, Chris Angelico a ?crit : > On Tue, Mar 22, 2016 at 10:06 AM, Michel Desmoulin > wrote: >> Itertools is great, and some functions in it are more used than others: >> >> - islice; >> - chain; >> - dropwhile, takewhile; >> ... >> >> The changes I'm going to propose do not add new syntax to Python, but >> yet would streamline the use of this nice tool and blend it into the >> language core. > > You're not the first to ask for something like this :) Let's get > *really* specific about semantics, though - and particularly about the > difference between iterables, iterators, and generators. > >> Make slicing accept callables >> ============================= >> >> So my first proposal is to be able to do: >> >> def stop(element): >> return element > 4 >> print(numbers[:stop]) >> >> It's quite pythonic, easy to understand : the end of the slice is when >> this condition is met. Any not the strange way takewhile work, which is >> "carry on as long as this condition is met". >> >> We could also extend itertools.islice to accept such parameter. > > This cannot be defined for arbitrary iterables, unless you're > proposing to mandate it in some way. (It conflicts with the way a list > handles slicing, for instance.) Even for arbitrary iterators, it may > be quite tricky (since iterators are based on a protocol, not a type); > but maybe it would be worth proposing an "iterator mixin" that handles > this for you, eg: > > class IteratorOperations: > def __getitem__(self, thing): > if isinstance(thing, slice): > if has_function_in_criteria(slice): return > self.takeuntil(s.start, s.stop) > return itertools.islice(...) > def takeuntil(self, start, stop): > val = next(self) > while start is not None and not start(val): > val = next(self) > while stop is None or not stop(val): > yield val > val = next(self) > > As long as you inherit from that, you get these operations made > available to you. > > Now, if you're asking this about generators specifically, then it > might be possible to add this (since all generators are of the same > type). It wouldn't be as broad as the itertools functions (which can > operate on any iterable), but could be handy if you do a lot with > gens, plus it's hard to subclass them. > >> Slicing any iterable >> ====================== >> >> So the second proposal is to allow: >> >> def func_accepting_any_iterable(foo): >> return bar(foo[3:7]) >> >> The slicing would then return a list if it's a list, a typle if it's a >> tuple, and a islice(generator) if it's a generator. If somebody uses a >> negative index, it would then raises a ValueError like islice. >> >> This would make duck typing and iteration even easier in Python. > > Again, while I am sympathetic to the problem, it's actually very hard; > islice always returns the same kind of thing, but slicing syntax can > return all manner of different things, because it's up to the object > on the left: > >>>> range(10)[3:7] > range(3, 7) >>>> "Hello, world!"[3:7] > 'lo, ' >>>> [1, 4, 2, 8, 5, 7, 1, 4, 2, 8, 5, 7][3:7] > [8, 5, 7, 1] >>>> memoryview(b"Hello, world!")[3:7] > > > You don't want these to start returning islice objects. You mentioned > lists, but other types will also return themselves when sliced. > > Possibly the solution here is actually to redefine object.__getitem__? > Currently, it simply raises TypeError - not subscriptable. Instead, it > could call iter() on itself, and then attempt to islice it. That would > mean that the TypeError would change to "is not iterable" > (insignificant difference), anything that already defines __getitem__ > will be unaffected (good), and anything that's iterable but not > subscriptable would automatically islice itself (potentially a trap, > if people don't know what they're doing). > >> Chaining iterable >> ================== >> >> Iterating on heterogenous iterable is not clear. >> >> You can add lists with lists and tuples with tuples, but if you need >> more, then you need itertools.chain. Few people know about it, so I >> usually see duplicate loops and conversion to lists/tuples. >> >> So My first proposal is to overload the "&" operator so that anything >> defining __iter__ can be used with it. >> >> Then you can just do: >> >> chaining = "abc" & [True, False] & (x * x for x in range(10)) >> for element in chaining: >> print(element) >> >> Instead of: >> >> from itertools import chain >> chaining = chain("abc", [True, False], (x * x for x in range(10))) >> for element in chaining: >> print(element) > > Again, anything involving operators is tricky, since anything can > override its handling. But if you require that the first one be a > specific iterator class, you can simply add __and__ to it to do what > you want: > > class iter: > iter = iter # snapshot the default 'iter' > def __init__(self, *args): > self.iter = self.iter(*args) # break people's minds > def __iter__(self): return self > def __next__(self): return next(self.iter) > def __and__(self, other): > yield from self.iter > yield from other > > Okay, so you'd probably do it without the naughty bits, but still :) > As long as you call iter() on the first thing in the chain, everything > else will work. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From rosuav at gmail.com Mon Mar 21 20:02:20 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 22 Mar 2016 11:02:20 +1100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F08AFE.2040501@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F08AFE.2040501@gmail.com> Message-ID: On Tue, Mar 22, 2016 at 10:59 AM, Michel Desmoulin wrote: > The solution I'm currently using involves having a class called g, and > everytime I want to manipulate an iterable, I just wrap it in g(). > > Then I got an object with all those semantics (and actually a lot more). > > Maybe we can make only those things apply to the objects returned by > iter() ? Oh, that's easy then. iter = g :) > (Also, about slicing accepting callable, actually g() goes a bit > overboard and accept any object. If the object is not an int or > callable, then it's used as a sentinel value. Not sure if I should speak > about that here.) Nothing wrong with that. The semantics are still the same; you simply have several ways of defining "the end has been reached". ChrisA From desmoulinmichel at gmail.com Mon Mar 21 20:03:45 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 01:03:45 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F0895F.2090009@btinternet.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F03F84.7050907@gmail.com> <56F0895F.2090009@btinternet.com> Message-ID: <56F08BE1.1000000@gmail.com> Le 22/03/2016 00:53, Rob Cliffe a ?crit : > > > On 21/03/2016 18:37, Michel Desmoulin wrote: >> >> Le 21/03/2016 17:34, Chris Barker a ?crit : >>> I've taught enough and I'm sure everyone else here has too, to know that >>> the "else" in a for loop in non-intuitive to a lot of folks. And maybe a >>> different keyword would have been clearer. But it is what it is, could >>> we keep this discussion to the proposed addition? >> +1 I would love "else" semantics to be changed as well, but we can't. >> >> What about: >> >> for x in stuff: >> foo(x) >> or: >> bar() > The trouble with "or" or "else" after a for-loop is that it suggests > (following English usage) alternative actions: Either execute this > for-loop, or do something else. But that's exactly the goal of "or" here. Run only if the loop is never executed since stuff is empty. > If I were designing Python 4, I might opt for > > for x in stuff: > foo(x) > ifnobreak: > bar() Python 4 won't break compat the way Python 3 did (Guido's words), so Python 4 is not a better opportunity for introducing new stuff than any other release. But I do like the "nobreak" better than else (you can drop the if, it's just noise). However, again, we can't remove else, and "there should be one way to do it" prevents us to add duplicates keywords. > > Pro: "ifnobreak" is at least explicit, and not particularly likely to > clash with an already-used variable name. > Con: "ifnobreak" is too long and wordy. But I can't think of anything > better. "aftercomplete", "whencomplete" are just as bad. > Hell, I'm coming round to "then". What about allowing "then" to have an > indent between that of the for-statement and the for-body: > > for x in stuff(): > foo(x) > then: > bar() > > Of course, you still have to learn it, like all idioms, but having > learned it, isn't it a bit more readable? > The idea could be extended to other suites, conveying that everything > indented is associated with the initial statement of the suite: > > try: > foo() > except SomeException: > bar() > > Rob Cliffe > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mike at selik.org Mon Mar 21 20:04:38 2016 From: mike at selik.org (Michael Selik) Date: Tue, 22 Mar 2016 00:04:38 +0000 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F0895F.2090009@btinternet.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F03F84.7050907@gmail.com> <56F0895F.2090009@btinternet.com> Message-ID: On Mon, Mar 21, 2016 at 7:59 PM Rob Cliffe wrote: > What about allowing "then" to have an > indent between that of the for-statement and the for-body: > > for x in stuff(): > foo(x) > then: > bar() > > Of course, you still have to learn it, like all idioms, but having > learned it, isn't it a bit more readable? > Either I haven't learned it or more indentation levels makes things harder to read. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Mon Mar 21 20:45:08 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 21 Mar 2016 17:45:08 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F07E78.9020309@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: <9D3EE7B6-2C69-40F9-93FF-8A5F25F4CA7D@yahoo.com> On Mar 21, 2016, at 16:06, Michel Desmoulin wrote: In addition to all the issues Chris raised... > > So my first proposal is to be able to do: > > def stop(element): > return element > 4 > print(numbers[:stop]) The first issue is pretty minor compared to the later ones, but already shows the problems of thinking about only lists and iterators, so I won't skip it: Applying this to a sequence that copies when slicing makes sense. Applying it to an iterator (together with your #2) makes sense. Applying it to a type that returns views on slicing, like a NumPy array, doesn't necessarily make sense, especially f that type is mutable, like a NumPy array. (Should the view change if the first element > 4 changes?) > Slicing any iterable > ====================== > > Now, while I do like islice, I miss the straigthforwardness of [:]: > > > from itertools import islice > > def func_accepting_any_iterable(foo): > return bar(islice(foo, 3, 7)) > > It's verbose, and renders the [3:7] syntaxe almost useless if you don't > have control over the code creating the iterable you are going to > process since you don't know what it's going to be. > > So the second proposal is to allow: > > def func_accepting_any_iterable(foo): > return bar(foo[3:7]) > > The slicing would then return a list if it's a list, a typle if it's a > tuple, and a islice(generator) if it's a generator. If somebody uses a > negative index, it would then raises a ValueError like islice. And what if it's a dict? Returning an islice of an iterator over the dict _works_, but it's almost certainly not what you want, because an iterator over the dict gives you the dict's keys, not its values. If d[3:7] means anything, I'd expect it to mean something like {k: v for (k, v) in d.items() if 3<=k<7}, or {v for ...same...}, not 4 arbitrary keys out of the dict. (Imagine that d's keys are all integers. Surely you'd want it to include d[3], d[4], d[5], and d[6], right?) And what if it's one of the collections on PyPI that already provides a non-islice-like meaning for slice syntax? For example, many of the sorted-dict types do key slicing, which returns you something like {k: v for (k, v) in d.items() if 3<=k<7} but still sorted, in log rather than linear time, and sometimes a view rather than a copy. And what if it's a collection for which indexing makes no sense, not even the wrong kind of sense, like a set? It'll slice out 4 values in arbitrary order iff there are at least 7 values. What's the good in that? And what if it's an iterator that isn't a generator? Does it just return some arbitrary new iterator type, even if the input type provided some additional interface on top of Iterator (as generators do)? Also, even with iterators a lot of things you do with slicing no longer make sense, but would run and silently do the wrong thing. For example: if matches_header(seq[:6]): handle_body(seq[6:]) If seq is an iterator, that first line is going to consume the first 6 elements, which means the second is now going to start on the 12th element rather than the 6th. It will be a lot less fun to debug "why do some of my messages lose their first word or so, but others work fine?" than the current "why do I sometimes get a TypeError telling me that type list_iterator isn't indexable?" > Chaining iterable > ================== > > Iterating on heterogenous iterable is not clear. > > You can add lists with lists and tuples with tuples, but if you need > more, then you need itertools.chain. Few people know about it, so I > usually see duplicate loops and conversion to lists/tuples. > > So My first proposal is to overload the "&" operator so that anything > defining __iter__ can be used with it. So, what does {1} & {1, 2, 3} do? This is a trick question: sets already define the & operator, as do other set-like collections. So this proposal either has to treat sets as not iterable, or break the set interface. All of these are part of the same problem: you're assuming that all iterables are either sequences or generators, but many of them--including two very important built-in types, not to mention all of the builtin types' iterators--are not. From mike at selik.org Tue Mar 22 01:49:22 2016 From: mike at selik.org (Michael Selik) Date: Tue, 22 Mar 2016 01:49:22 -0400 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F07E78.9020309@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> > On Mar 21, 2016, at 7:06 PM, Michel Desmoulin wrote: > if you need more, then you need itertools. Few people know about it, so I > usually see duplicate loops and conversion to lists/tuples. Why do you think the new syntax you're suggesting will be more discoverable and therefore more well known than the itertools module? From graham.gower at gmail.com Tue Mar 22 03:34:10 2016 From: graham.gower at gmail.com (Graham Gower) Date: Tue, 22 Mar 2016 18:04:10 +1030 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> Message-ID: On 22 March 2016 at 16:19, Michael Selik wrote: > > >> On Mar 21, 2016, at 7:06 PM, Michel Desmoulin wrote: >> if you need more, then you need itertools. Few people know about it, so I >> usually see duplicate loops and conversion to lists/tuples. > > Why do you think the new syntax you're suggesting will be more discoverable and therefore more well known than the itertools module? And consider the other side of the coin. If I read code using a function of the itertools module, its easy to look up the definition; if I read code featuring a rarely used syntax, how easy is it to come up with search engine terms to describe the syntax, and thus find documentation for the feature? From tritium-list at sdamon.com Tue Mar 22 04:39:37 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Tue, 22 Mar 2016 04:39:37 -0400 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> Message-ID: <56F104C9.7050805@sdamon.com> On 3/22/2016 03:34, Graham Gower wrote: > > And consider the other side of the coin. If I read code using a > function of the itertools module, its easy to look up the definition; > if I read code featuring a rarely used syntax, how easy is it to come > up with search engine terms to describe the syntax, and thus find > documentation for the feature? > This is a pure devil's advocate response, as I don't feel there is a need to add itertools functionality to the language itself. Googling for 'Python with keyword' or 'Python lambda keyword', for example, give very good results (documentation or examples in the first 5 results on both). Any proposal involving adding a new keyword would probably be as discoverable as the standard library. ... Which is the big gotcha with arguments involving new keywords. If it makes it exactly as discoverable, then nothing is gained by new syntax, and we are back where we started. Your point is accurate for any syntax that does not involve new keywords, or for new uses of existing keywords. From srkunze at mail.de Tue Mar 22 08:07:25 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 22 Mar 2016 13:07:25 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> Message-ID: <56F1357D.9040507@mail.de> On 21.03.2016 21:57, Vito De Tullio wrote: > Chris Angelico wrote: >> >> What if 'iterable' is locals().values()? Can you, with perfect >> reliability, recognize that case? AIUI this is exactly why next() and >> __next__() are defined to "return a value or raise", rather than >> "return a value or return a magic no-more-values value", because >> there's always the possibility that the no-more-values value is a >> legitimately-yielded value. > so, the "correct" way to handle this is something like > > try: > e = next(iterable) > except StopIteration: > empty_suite() # there is no element > else: > main_suite(e) # handle the first element > for e in iterable: > main_suite(e) # handle the rest of the elements > > ? > > apart for the readability you need to "copy" the same code two times (or > refactor in a function) Interesting. I didn't think of this solution. However, I assume that when I do "object()" I got something unique among all other objects. So, item = sentinel = object() for item in collection: main_suite(item) if item is sentinel: empty_suite() Is still quite correct, right? Best, Sven From srkunze at mail.de Tue Mar 22 08:16:30 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 22 Mar 2016 13:16:30 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> Message-ID: <56F1379E.3010608@mail.de> On 21.03.2016 17:31, Chris Barker wrote: > All that being said: > > how about "elempty"?, to go with elif? > I am mostly dispassionate about the wording. So, why not elempty? [Btw. most people consider existing keywords for re-using as they cannot be used for variables in the first place] Best, Sven From rosuav at gmail.com Tue Mar 22 08:16:55 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 22 Mar 2016 23:16:55 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F1357D.9040507@mail.de> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <56F1357D.9040507@mail.de> Message-ID: On Tue, Mar 22, 2016 at 11:07 PM, Sven R. Kunze wrote: > However, I assume that when I do "object()" I got something unique among all > other objects. So, > > item = sentinel = object() > for item in collection: > main_suite(item) > if item is sentinel: > empty_suite() > > Is still quite correct, right? Sure it is. Perfectly correct. >>> def nope(): ... item = sentinel = object() ... for item in locals().values(): ... print("We have:", item) ... if item is sentinel: ... print("We have no items.") ... >>> nope() We have: We have: We have no items. >>> An iterator can return *any* *object*. That's why Python uses an exception (StopIteration) to signal the absence of an object. More reliable is to use that absence, either by exception or by probing a dictionary's keys: def better(): # Make sure item is unbound try: del item except NameError: pass for item in locals().values(): print("We have:", item) if 'item' not in locals(): print("We have no items.") But now we're getting into the realm of ugly code to deal with edge cases. Like with "yield from", language support can be justified when there's a simple and obvious *but imperfect* way to do something ("yield from x" is not the same as "for item in x: yield item"). ChrisA From srkunze at mail.de Tue Mar 22 08:20:05 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 22 Mar 2016 13:20:05 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <20160321014433.GC12526@ando.pearwood.info> References: <56EEE82A.6050505@mail.de> <20160321014433.GC12526@ando.pearwood.info> Message-ID: <56F13875.3030804@mail.de> On 21.03.2016 02:44, Steven D'Aprano wrote: > On Sun, Mar 20, 2016 at 07:12:58PM +0100, Sven R. Kunze wrote: > >> for item in my_iterator: >> # do per item >> empty: >> # do something else > "empty" is a poor choice, as I expect that it will break a lot of code > that already uses it as a variable. > > if empty: > ... So, "empty" is off the table I suppose. > > >> *Keyword* >> Keywords under consideration have been in (an attempt of dispassionate) >> order of preference: >> 1) empty -> most obvious, most people responded with a solution to solve >> the intended problem >> 2) else -> already taken but to some would be the preferred one >> 3) or -> as alternative if a new keyword is too much > I'm not too keen on the look of "or" for this: > > for x in iterable: > block > or: > alternative > > > but I think it's the least worst of the proposed keywords. Got it but I am open for suggestions. :-) > >> 4) except -> as alternative if a new keyword is too much > I think that "except" completely fails to describe what the keyword > does, and it will give some people the misunderstanding that it catches > errors in the body of the for-loop. Now, that you mention it. This might give birth to a completely different idea. What about? for item in collection: # do for item except EmptyCollection: # do if collection is empty except StopIteration: # do after the loop Which basically merges try and for? Best, Sven From srkunze at mail.de Tue Mar 22 08:27:05 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 22 Mar 2016 13:27:05 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <56F1357D.9040507@mail.de> Message-ID: <56F13A19.7080909@mail.de> On 22.03.2016 13:16, Chris Angelico wrote: > On Tue, Mar 22, 2016 at 11:07 PM, Sven R. Kunze wrote: >> However, I assume that when I do "object()" I got something unique among all >> other objects. So, >> >> item = sentinel = object() >> for item in collection: >> main_suite(item) >> if item is sentinel: >> empty_suite() >> >> Is still quite correct, right? > Sure it is. Perfectly correct. > >>>> def nope(): > ... item = sentinel = object() > ... for item in locals().values(): > ... print("We have:", item) > ... if item is sentinel: > ... print("We have no items.") > ... >>>> nope() > We have: > We have: > We have no items. > An iterator can return *any* *object*. That's why Python uses an > exception (StopIteration) to signal the absence of an object. Oh, you are right. That's definitely an ugly edge case. Any proposed solution should just work given the simplicity of the problem. > More reliable is to use that absence, either by exception or by > probing a dictionary's keys: > > def better(): > # Make sure item is unbound > try: del item > except NameError: pass > for item in locals().values(): > print("We have:", item) > if 'item' not in locals(): > print("We have no items.") Interesting. > But now we're getting into the realm of ugly code to deal with edge > cases. Like with "yield from", language support can be justified when > there's a simple and obvious *but imperfect* way to do something > ("yield from x" is not the same as "for item in x: yield item"). Exactly. Despite the fact that "yield from" is shorter it can handle edge cases more beautiful. Best, Sven From rosuav at gmail.com Tue Mar 22 08:29:42 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 22 Mar 2016 23:29:42 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F13A19.7080909@mail.de> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <56F1357D.9040507@mail.de> <56F13A19.7080909@mail.de> Message-ID: On Tue, Mar 22, 2016 at 11:27 PM, Sven R. Kunze wrote: >> But now we're getting into the realm of ugly code to deal with edge >> cases. Like with "yield from", language support can be justified when >> there's a simple and obvious *but imperfect* way to do something >> ("yield from x" is not the same as "for item in x: yield item"). > > > Exactly. Despite the fact that "yield from" is shorter it can handle edge > cases more beautiful. It's not enough shorter than the simple 'for' loop to justify the dedicated syntax. But check out its *actual* equivalent code and you'll see that while we're getting toward similar territory, the for-ifempty loop isn't quite here yet :) https://www.python.org/dev/peps/pep-0380/#formal-semantics ChrisA From srkunze at mail.de Tue Mar 22 08:30:50 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 22 Mar 2016 13:30:50 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> Message-ID: <56F13AFA.8050907@mail.de> On 20.03.2016 21:16, Andrew Barnert wrote: > More generally, I think if this feature were to be added, "empty" is a > reasonable name. The idiomatic way to write it today is something like: > > elem = empty = object() > for elem in seq: > do_stuff(elem) > if elem is empty: > do_empty_seq_stuff() > > So you're basically looking for syntactic sugar that abbreviated "if > ... is empty:", right? > Basically. The future discussion showed that the idiomatic way is not 100% correct but I think you got the idea. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Mar 22 08:33:36 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 22 Mar 2016 13:33:36 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <56F1357D.9040507@mail.de> <56F13A19.7080909@mail.de> Message-ID: <56F13BA0.6030209@mail.de> On 22.03.2016 13:29, Chris Angelico wrote: > It's not enough shorter than the simple 'for' loop to justify the > dedicated syntax. But check out its *actual* equivalent code and > you'll see that while we're getting toward similar territory, the > for-ifempty loop isn't quite here yet :) > > https://www.python.org/dev/peps/pep-0380/#formal-semantics I remember that. I was quite surprise to find such a huge amount of equivalent python code to handle all corner and edge cases correctly. If I were to become the one maintaining this, I hope we don't discover even more edge cases. :D Best, Sven From desmoulinmichel at gmail.com Tue Mar 22 08:45:21 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 13:45:21 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <9D3EE7B6-2C69-40F9-93FF-8A5F25F4CA7D@yahoo.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <9D3EE7B6-2C69-40F9-93FF-8A5F25F4CA7D@yahoo.com> Message-ID: <56F13E61.80501@gmail.com> Le 22/03/2016 01:45, Andrew Barnert a ?crit : > On Mar 21, 2016, at 16:06, Michel Desmoulin wrote: > > In addition to all the issues Chris raised... >> >> So my first proposal is to be able to do: >> >> def stop(element): >> return element > 4 >> print(numbers[:stop]) > > The first issue is pretty minor compared to the later ones, but already shows the problems of thinking about only lists and iterators, so I won't skip it: > > Applying this to a sequence that copies when slicing makes sense. Applying it to an iterator (together with your #2) makes sense. Applying it to a type that returns views on slicing, like a NumPy array, doesn't necessarily make sense, especially f that type is mutable, like a NumPy array. (Should the view change if the first element > 4 changes?) > Numpy is not part of the stdlib. We should not prevent adding a feature in Python because it will not immediately benefit an 3rd party lib, even a famous one. The proposal doesn't hurt Numpy : they override __getitem__ and choose not to accept the default behavior anyway, and are not affected by the default behavior. Besides, you generally try to not mix Numpy and non Numpy manipulation code as it has it's own semantics (no for loop, special slicing, etc.). >> Slicing any iterable >> ====================== >> >> Now, while I do like islice, I miss the straigthforwardness of [:]: >> >> >> from itertools import islice >> >> def func_accepting_any_iterable(foo): >> return bar(islice(foo, 3, 7)) >> >> It's verbose, and renders the [3:7] syntaxe almost useless if you don't >> have control over the code creating the iterable you are going to >> process since you don't know what it's going to be. >> >> So the second proposal is to allow: >> >> def func_accepting_any_iterable(foo): >> return bar(foo[3:7]) >> >> The slicing would then return a list if it's a list, a typle if it's a >> tuple, and a islice(generator) if it's a generator. If somebody uses a >> negative index, it would then raises a ValueError like islice. > > And what if it's a dict? Returning an islice of an iterator over the dict _works_, but it's almost certainly not what you want, because an iterator over the dict gives you the dict's keys, not its values. If d[3:7] means anything, I'd expect it to mean something like {k: v for (k, v) in d.items() if 3<=k<7}, or {v for ...same...}, not 4 arbitrary keys out of the dict. (Imagine that d's keys are all integers. Surely you'd want it to include d[3], d[4], d[5], and d[6], right?) > This is a point to discuss. I would raise a ValueError, trying to slice a dict is almost always a mistake. E.G: if you design a function that needs an argument to be sliceable (event with islice), you usually don't want people to pass in dicts, and when you strangely do (to sample maybe ?), then you would cast it manually. It would be a rare use case, compared to the multiple occasions you need a more generic slicing. > And what if it's one of the collections on PyPI that already provides a non-islice-like meaning for slice syntax? For example, many of the sorted-dict types do key slicing, which returns you something like {k: v for (k, v) in d.items() if 3<=k<7} but still sorted, in log rather than linear time, and sometimes a view rather than a copy. > See the point about Numpy. > And what if it's a collection for which indexing makes no sense, not even the wrong kind of sense, like a set? It'll slice out 4 values in arbitrary order iff there are at least 7 values. What's the good in that? > See the point about dict. > And what if it's an iterator that isn't a generator? Does it just return some arbitrary new iterator type, even if the input type provided some additional interface on top of Iterator (as generators do)? > This can be discussed and is more about the proper implementation, but does not discard the validity of the idea. > Also, even with iterators a lot of things you do with slicing no longer make sense, but would run and silently do the wrong thing. For example: > > if matches_header(seq[:6]): > handle_body(seq[6:]) > > If seq is an iterator, that first line is going to consume the first 6 elements, which means the second is now going to start on the 12th element rather than the 6th. It will be a lot less fun to debug "why do some of my messages lose their first word or so, but others work fine?" than the current "why do I sometimes get a TypeError telling me that type list_iterator isn't indexable?" Either you use generators or you don't. If you use generators, you know they will be consumed when you pass them around. This has nothing to do with the slicing syntax. The one problem I can see is when: - seq is a generator you didn't produce; - you don't know it's a generator. - you get a surprising behavior because slicing cause no errors. It's an edge case and is worth considering if it's going to be a blocker or not. Also, one alternative is to only add slicing to all objects returned by iter() in the stdlib. This would force people to explicitly mark that they know what they are doing, and while less convenient, remains very handy. >> Chaining iterable >> ================== >> >> Iterating on heterogenous iterable is not clear. >> >> You can add lists with lists and tuples with tuples, but if you need >> more, then you need itertools.chain. Few people know about it, so I >> usually see duplicate loops and conversion to lists/tuples. >> >> So My first proposal is to overload the "&" operator so that anything >> defining __iter__ can be used with it. > > So, what does {1} & {1, 2, 3} do? This is a trick question: sets already define the & operator, as do other set-like collections. So this proposal either has to treat sets as not iterable, or break the set interface. Indeed I forgot about sets. Maybe there is another operator that would do the trick, such as "<<". We can't use "+" as it would be confusing, and sets() overide a LOT of operators. Anyway, let's not throw the baby with the water. This is the least important part of the proposal, and any part can be changed, improved or ditched. That's what Python-ideas is for. > > All of these are part of the same problem: you're assuming that all iterables are either sequences or generators, but many of them--including two very important built-in types, not to mention all of the builtin types' iterators--are not. "Slicing any iterable" was a bad title. It should have been "extend the slicing application". I'm not assuming any iterable is a sequence or a generator, I think we can come up with a reasonable behavior for the case slicing doesn't make sense, adding more power to this handy tool. Please also consider the iter() proposal as a more verbose alternative, yet still powerful alternative. Maybe even easier to implement ? From desmoulinmichel at gmail.com Tue Mar 22 08:52:21 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 13:52:21 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> Message-ID: <56F14005.1020005@gmail.com> Le 22/03/2016 06:49, Michael Selik a ?crit : > > >> On Mar 21, 2016, at 7:06 PM, Michel Desmoulin wrote: >> if you need more, then you need itertools. Few people know about it, so I >> usually see duplicate loops and conversion to lists/tuples. > > Why do you think the new syntax you're suggesting will be more discoverable and therefore more well known than the itertools module? > It's a gut feeling really, coming from the fact I train people in Python for a living: - most of my colleagues and myself always show the builtins way before anything else. I rarely have the time to even show itertools. - same goes for tutorials online. Itertools is rarerly demonstrated. - a whole module is more impressive than a few behavior and my student will usually delay reading about any doc related to one module. They don't even know where to start or how to express what they are looking for - it's something they will try (or discover by mistake): random playing with operators and data is something I witness a lot. Random playing with imports, not so much. I may be wrong, I can't finance a study with a big representative sample to prove it. But again, nobody can do so for anything on this list. From desmoulinmichel at gmail.com Tue Mar 22 08:58:41 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 13:58:41 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> Message-ID: <56F14181.2000701@gmail.com> Le 22/03/2016 08:34, Graham Gower a ?crit : > On 22 March 2016 at 16:19, Michael Selik wrote: >> >> >>> On Mar 21, 2016, at 7:06 PM, Michel Desmoulin wrote: >>> if you need more, then you need itertools. Few people know about it, so I >>> usually see duplicate loops and conversion to lists/tuples. >> >> Why do you think the new syntax you're suggesting will be more discoverable and therefore more well known than the itertools module? > > And consider the other side of the coin. If I read code using a > function of the itertools module, its easy to look up the definition; > if I read code featuring a rarely used syntax, how easy is it to come > up with search engine terms to describe the syntax, and thus find > documentation for the feature? > I see your point. Again, this is the weakest point of my proposal, we should not discard all of it just because of that. And maybe we can come up with something better. E.G: I suggested before that iter() returns objects with advanced semantics for [:] as an alternative to add slicing to generators. Maybe the same objects can come with a chain() method, which is explicit. Or provide an additional builtin function, which does help with those issues if changing iter() is not possible. From rosuav at gmail.com Tue Mar 22 09:05:56 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 23 Mar 2016 00:05:56 +1100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F14181.2000701@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> Message-ID: On Tue, Mar 22, 2016 at 11:58 PM, Michel Desmoulin wrote: > E.G: I suggested before that iter() returns objects with advanced > semantics for [:] as an alternative to add slicing to generators. The builtin iter() function simply returns whatever __iter__() returns. I've offered a way for you to easily shadow the builtin with your own class (which can then provide whatever it likes), but it's going to have to work by wrapping the original iterator, which will break generators (unless you explicitly pass everything through, and even then you'll break things that check if isinstance(x, generator), because you can't subclass generator). ChrisA From steve at pearwood.info Tue Mar 22 09:10:18 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 23 Mar 2016 00:10:18 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <56F1357D.9040507@mail.de> Message-ID: <20160322131018.GF12526@ando.pearwood.info> On Tue, Mar 22, 2016 at 11:16:55PM +1100, Chris Angelico wrote: > On Tue, Mar 22, 2016 at 11:07 PM, Sven R. Kunze wrote: > > However, I assume that when I do "object()" I got something unique among all > > other objects. So, > > > > item = sentinel = object() > > for item in collection: > > main_suite(item) > > if item is sentinel: > > empty_suite() > > > > Is still quite correct, right? > > Sure it is. Perfectly correct. Are you being sarcastic? > >>> def nope(): > ... item = sentinel = object() > ... for item in locals().values(): > ... print("We have:", item) > ... if item is sentinel: > ... print("We have no items.") > ... > >>> nope() > We have: > We have: > We have no items. I think that this is a completely artificial edge case that's never going to come up in practice, and is easy to work around: Just Don't Do That. It's perfectly safe if somebody passes you locals() from *another* scope, because it cannot have access to *your* local sentinel. So you only need worry about one case: when *you* pass *your own* locals as the iterable. Hence, *just don't do it*. "But what if I want to iterate over locals() and handle the case where it is empty?" if not locals(): handle_empty else: for value in locals().values(): ... Don't get me wrong, I think it was very clever that you thought of this case. But I don't think this is ever going to come up in practice. -- Steve From rosuav at gmail.com Tue Mar 22 09:21:34 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 23 Mar 2016 00:21:34 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <20160322131018.GF12526@ando.pearwood.info> References: <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <56F1357D.9040507@mail.de> <20160322131018.GF12526@ando.pearwood.info> Message-ID: On Wed, Mar 23, 2016 at 12:10 AM, Steven D'Aprano wrote: > On Tue, Mar 22, 2016 at 11:16:55PM +1100, Chris Angelico wrote: >> On Tue, Mar 22, 2016 at 11:07 PM, Sven R. Kunze wrote: >> > However, I assume that when I do "object()" I got something unique among all >> > other objects. So, >> > >> > item = sentinel = object() >> > for item in collection: >> > main_suite(item) >> > if item is sentinel: >> > empty_suite() >> > >> > Is still quite correct, right? >> >> Sure it is. Perfectly correct. > > Are you being sarcastic? > >> >>> def nope(): I was, hence the function name. > It's perfectly safe if somebody passes you locals() from *another* > scope, because it cannot have access to *your* local sentinel. So you > only need worry about one case: when *you* pass *your own* locals as the > iterable. Hence, *just don't do it*. And make sure the function you call doesn't use sys._getframe to get them for you. > Don't get me wrong, I think it was very clever that you thought of this > case. But I don't think this is ever going to come up in practice. Maybe. But it's an edge case of a form that could come up in other ways - I don't know of a way to enumerate all objects in CPython, but it's a perfectly reasonable way to debug a garbage collector. Point is that iteration can return literally *any* object. The object() sentinel is better than the None sentinel is better than the "if not item" falsiness check, but the only perfect solution is an extremely ugly one (and I'm not guaranteeing that even that is perfect - only that *I* haven't found a flaw in it yet). At some point, the solution is good enough for your code. ChrisA From steve at pearwood.info Tue Mar 22 09:28:28 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 23 Mar 2016 00:28:28 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F13A19.7080909@mail.de> References: <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <56F1357D.9040507@mail.de> <56F13A19.7080909@mail.de> Message-ID: <20160322132828.GG12526@ando.pearwood.info> On Tue, Mar 22, 2016 at 01:27:05PM +0100, Sven R. Kunze wrote: [...] > >>>>def nope(): > >... item = sentinel = object() > >... for item in locals().values(): > >... print("We have:", item) > >... if item is sentinel: > >... print("We have no items.") > >... > >>>>nope() > >We have: > >We have: > >We have no items. > >An iterator can return *any* *object*. That's why Python uses an > >exception (StopIteration) to signal the absence of an object. > > Oh, you are right. That's definitely an ugly edge case. Any proposed > solution should just work given the simplicity of the problem. I think we're starting to give far more weight to that case than it deserves. It doesn't deserve a note in the docs, let alone new syntax to "solve it". Nobody is going to do it, except to win a bet. Certainly nobody is going to do it *accidently*. > >But now we're getting into the realm of ugly code to deal with edge > >cases. Like with "yield from", language support can be justified when > >there's a simple and obvious *but imperfect* way to do something > >("yield from x" is not the same as "for item in x: yield item"). > > Exactly. Despite the fact that "yield from" is shorter it can handle > edge cases more beautiful. For plain old iteration, "yield from" does nothing more than iterating over the iterable and yielding. There's no "edge cases" here, there are some pretty major differences. As the PEP says: If yielding of values is the only concern, this can be performed without much difficulty using a loop such as for v in g: yield v However, if the subgenerator is to interact properly with the caller in the case of calls to send() , throw() and close() , things become considerably more difficult. https://www.python.org/dev/peps/pep-0380/ "yield from" doesn't fix some weird edge case with iteration. It provides some significant new functionality which is *very* tricky to get right. That's not the case here with detecting an empty iterator. There are two perfectly adequate and simple ways to detect an empty iterator: # 1 empty = True for x in iterable: empty = False do_stuff() if empty: handle_empty() #2 x = sentinel = object() assert iterable is not locals().values() # *wink* for x in iterable: do_stuff() if x is sentinel: handle_empty() -- Steve From steve at pearwood.info Tue Mar 22 09:48:26 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 23 Mar 2016 00:48:26 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <56F1357D.9040507@mail.de> <20160322131018.GF12526@ando.pearwood.info> Message-ID: <20160322134826.GH12526@ando.pearwood.info> On Wed, Mar 23, 2016 at 12:21:34AM +1100, Chris Angelico wrote: > > It's perfectly safe if somebody passes you locals() from *another* > > scope, because it cannot have access to *your* local sentinel. So you > > only need worry about one case: when *you* pass *your own* locals as the > > iterable. Hence, *just don't do it*. > > And make sure the function you call doesn't use sys._getframe to get > them for you. Are you saying that I might have code like this: def test(): x = sentinel = object() iterable = some_function() for x in iterable: # you know the rest and some_function() might use _getframe to steal sentinel and return it back to me as iterable = [sentinel]? I'll take my chances with that, thanks very much. I am as concerned by that as I am that some_function might use ctypes to hack the value of integers so that 1 + 1 returns 4. Yes, it could happen. No, I don't care to take any special precautions to avoid it happening. I think a far more likely possibility is that somebody or something has monkey-patched object() to always return the same instance, regardless of who calls it from where, and now my sentinel is the same as everyone else's sentinel. Am I bothered by this? Not in the least. I mean, seriously, we write code all time relying on the behaviour of builtin functions, knowing full well that any piece of code anywhere might monkey-patch them to do something weird. Should we insist on special syntax that returns the len() of sequences because of an edge-case "what if len has been monkey-patched?". No. We just say "well don't monkey-patch len." In case it's not obvious by now, I don't think there's any need for syntax to handle the empty iterable case. -1 on the proposal. -- Steve From stephen at xemacs.org Tue Mar 22 11:09:42 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 23 Mar 2016 00:09:42 +0900 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> Message-ID: <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> Chris Barker writes: > On Mon, Mar 21, 2016 at 1:01 AM, Stephen J. Turnbull > wrote: > > > AFAICS the OP's idea has a trivial and efficient expansion: > > I'll take your word for it that this is efficient, and it is > trivial in amount of code to write, but it is not the least bit > trivial to read or write. By "efficient" I mean only that the loop is unchanged. Bonus is that only one assignment and one test are needed outside of the loop. The only reason I could think of at the time that it might not be trivial to read is that a reader is unfamiliar with the "sentinel = object()" idiom. Otherwise, "for" and "if" are the most basic of Python syntax, so I felt safe in using that word. Writing is another matter, but that's covered under the "sometimes ya gotta be Dutch" clause. :-) > Howver, the motivating example was "the difference between *nothing**as > error* and *nothing** as really empty*, GUI/Web interfaces display an > explicit notice in case of empty lists to signify the correct execution of > a process/action/search or the other case." > > Sure -- but this seem to be mixing validation with processing code -- you > really want to validate your inputs first. If it's input, yes, but in my experience this would mostly be dealing with internal state such as a database. Then EAFP applies. For example, in paging, an empty page is a natural signal for end of content. > AND: this would mostly be containers, "Mostly", maybe, but that's not to say that iterators are an unimportant case. And if your application involves a mix of containers and iterators, you may still prefer to use the same idioms for both cases. > All that being said: > > how about "elempty"?, to go with elif? -1 Not to my taste, to say the least. From mike at selik.org Tue Mar 22 11:23:31 2016 From: mike at selik.org (Michael Selik) Date: Tue, 22 Mar 2016 15:23:31 +0000 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F14005.1020005@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14005.1020005@gmail.com> Message-ID: On Tue, Mar 22, 2016 at 8:52 AM Michel Desmoulin wrote: > Le 22/03/2016 06:49, Michael Selik a ?crit: > >> On Mar 21, 2016, at 7:06 PM, Michel Desmoulin < > desmoulinmichel at gmail.com> wrote: > >> if you need more, then you need itertools. Few people know about it, so > I > >> usually see duplicate loops and conversion to lists/tuples. > > > > Why do you think the new syntax you're suggesting will be more > discoverable and therefore more well known than the itertools module? > > > > It's a gut feeling really, coming from the fact I train people in Python > for a living. > > I may be wrong, I can't finance a study with a big representative sample > to prove it. But again, nobody can do so for anything on this list. > Luckily we have the case of set.union and set.intersection versus the pipe and ampersand operators. One way you could provide some evidence is comparing the frequency of usage of the method vs operator in public projects. We might need to then discard code examples written by experts from our analysis, but it'd be a start. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Tue Mar 22 11:29:34 2016 From: steve.dower at python.org (Steve Dower) Date: Tue, 22 Mar 2016 08:29:34 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> Message-ID: What if itertools.iter was added with the expanded semantics? Possibly even a new name, but similar use to the built-in func. Top-posted from my Windows Phone -----Original Message----- From: "Chris Angelico" Sent: ?3/?22/?2016 6:06 Cc: "python-ideas" Subject: Re: [Python-ideas] Integrate some itertools into the Python syntax On Tue, Mar 22, 2016 at 11:58 PM, Michel Desmoulin wrote: > E.G: I suggested before that iter() returns objects with advanced > semantics for [:] as an alternative to add slicing to generators. The builtin iter() function simply returns whatever __iter__() returns. I've offered a way for you to easily shadow the builtin with your own class (which can then provide whatever it likes), but it's going to have to work by wrapping the original iterator, which will break generators (unless you explicitly pass everything through, and even then you'll break things that check if isinstance(x, generator), because you can't subclass generator). ChrisA _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Tue Mar 22 12:01:40 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 17:01:40 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> Message-ID: <56F16C64.6000606@gmail.com> It still an import in every file you want to use it, and in the shell. We can already have 3rd party lib to do this, so no need to change Python for that. Le 22/03/2016 16:29, Steve Dower a ?crit : > What if itertools.iter was added with the expanded semantics? Possibly > even a new name, but similar use to the built-in func. > > Top-posted from my Windows Phone > ------------------------------------------------------------------------ > From: Chris Angelico > Sent: ?3/?22/?2016 6:06 > Cc: python-ideas > Subject: Re: [Python-ideas] Integrate some itertools into the Python syntax > > On Tue, Mar 22, 2016 at 11:58 PM, Michel Desmoulin > wrote: >> E.G: I suggested before that iter() returns objects with advanced >> semantics for [:] as an alternative to add slicing to generators. > > The builtin iter() function simply returns whatever __iter__() > returns. I've offered a way for you to easily shadow the builtin with > your own class (which can then provide whatever it likes), but it's > going to have to work by wrapping the original iterator, which will > break generators (unless you explicitly pass everything through, and > even then you'll break things that check if isinstance(x, generator), > because you can't subclass generator). > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From p.f.moore at gmail.com Tue Mar 22 12:18:44 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 22 Mar 2016 16:18:44 +0000 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F16C64.6000606@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> Message-ID: On 22 March 2016 at 16:01, Michel Desmoulin wrote: > It still an import in every file you want to use it, and in the shell. > We can already have 3rd party lib to do this, so no need to change > Python for that. Note that sys, os, re, math, datetime are all "an import in every file you want to use it". The bar for changing Python is higher than just avoiding an import. Paul From srkunze at mail.de Tue Mar 22 13:21:24 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 22 Mar 2016 18:21:24 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> Message-ID: <56F17F14.9090306@mail.de> On 22.03.2016 16:09, Stephen J. Turnbull wrote: > Chris Barker writes: > > All that being said: > > > > how about "elempty"?, to go with elif? > > -1 Not to my taste, to say the least. Hmm, it seems there is no easy solution for this. What do you think about an alternative that can handle more than empty and else? for item in collection: # do for item except NeverExecuted: # do if collection is empty It basically merges "try" and "for" and make "for" emit EmptyCollection. So, independent of the initial "never executed loop body" use-case, one could also emulate the "else" clause by: for item in collection: # do for item except StopIteration: # do after the loop Thinking this further, it might enable much more use-cases and stays extensible without the need to invent all kinds of special keywords. So, this "for" might roughly be equivalent to (using the "old for"): class NeverExecuted(StopIteration): pass i = iter(collection) try: e = next(i) except StopIteration: if [NeverExecuted/StopIteration is catched]: raise NeverExecuted else: # do for item e for e in i: # do for item e else: if [StopIteration is catched]: raise StopIteration I hope that's not too convoluted. Best, Sven From desmoulinmichel at gmail.com Tue Mar 22 13:51:50 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 18:51:50 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> Message-ID: <56F18636.40100@gmail.com> Le 22/03/2016 17:18, Paul Moore a ?crit : > On 22 March 2016 at 16:01, Michel Desmoulin wrote: >> It still an import in every file you want to use it, and in the shell. >> We can already have 3rd party lib to do this, so no need to change >> Python for that. > > Note that sys, os, re, math, datetime are all "an import in every file > you want to use it". The bar for changing Python is higher than just > avoiding an import. > > Paul > This is an appeal to consider islice & Co as important as normal slicing. Indeed, you will have most certainly iterables in any code that use datetime, re or math. You have no certainty of having datetime, re or math imported in any code dealing with iterables. Think about how annoying it would be to do: >>> from builtins import slice >>> [1, 2, 3, 4, 5, 6][slice(2, 4)] [3, 4] For every slice. We don't, because slicing is part of our standard data processing toolkit. And I think we can make it even better. We already do in some places. E.G: range(10)[3:5] works while range() generate values on the fly. Why ? Because it's convenient, expressive and Pythonic. Well, if you process a file and you want to limit it to all lines after the first "BEGIN SECTION" (there can be other) and before the first "STOP" (there can be others), but only 10000 lines max, and not load the whole file in memory, you could do: def foo(p): with open(p) as f: def begin: return x == "BEGIN SECTION" def end: return x == "STOP" return f[begin, end][:10000] It's very clean, very convenient, very natural, and memory efficient. Now compare it with itertools: from itertools import takewhile, dropwhile, islice def foo(p): with open(p) as f: def begin: return x != "BEGIN SECTION" def end: return x != "STOP" return islice(takewhile(end, dropwhile(begin, f)), 0, 10000) It's ugly, hard to read, hard to write. In Python, you are always iterating on something, it makes sense to make sure we have the best tooling to do at our fingertips. From pavol.lisy at gmail.com Tue Mar 22 13:54:16 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Tue, 22 Mar 2016 18:54:16 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <20160321013234.GB12526@ando.pearwood.info> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> Message-ID: 2016-03-21 2:32 GMT+01:00, Steven D'Aprano : [...] > But, like I said, we're stuck with it, and I'm not seriously proposing a > change. I think we'd be better off now if the keyword had been "then" > from the beginning, but the pain of changing it *now* outweighs the > benefit. A. Just a little analyze about this theoretical possibility for x in seq: if good(x):break # (1) then: all_bad() # (2) else: empty() # (3) (3) seems to be alternative to (2) like "if then else" , but is not , it is alternative to (1) "union" (2) Also empty is not contradictory to all_bad! 'else' seems to be not ideal but yes we have what we have B. if we don't want to define new keywords and want to map all (?) possibilities (empty vs non empty, break vs no break) and don't want to break backward compatibility then we could analyze this: for i in seq: if cond(i): print('non empty & break') # (1) break and: print('non empty & all') # (2) or: print('no elements') # (3) else: print('no break') # (4) (1) is exclusive but (4) could be with (2) or (3) (2) and (3) are contradictory I have not opinion about order of and, or, else in this moment. From desmoulinmichel at gmail.com Tue Mar 22 13:57:43 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 18:57:43 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F17F14.9090306@mail.de> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> Message-ID: <56F18797.30209@gmail.com> Le 22/03/2016 18:21, Sven R. Kunze a ?crit : > On 22.03.2016 16:09, Stephen J. Turnbull wrote: >> Chris Barker writes: >> > All that being said: >> > >> > how about "elempty"?, to go with elif? >> >> -1 Not to my taste, to say the least. > > Hmm, it seems there is no easy solution for this. What do you think > about an alternative that can handle more than empty and else? > > for item in collection: > # do for item > except NeverExecuted: > # do if collection is empty > > It basically merges "try" and "for" and make "for" emit EmptyCollection. > So, independent of the initial "never executed loop body" use-case, one > could also emulate the "else" clause by: > > for item in collection: > # do for item > except StopIteration: > # do after the loop > It's an interesting idea. Generalizing except for more block I mean. Imagine: if user: print(user.permissions[0]) except IndexError: print('anonymous') with open('foo') as f: print(f.read()) except IOError: print('meh') for item in collection: # do for item except NeverExecuted: # do if collection is empty Not only does it allow interesting new usages such as the for loop empty pattern, but it does save a level of indentation for many classic use cases where you would nest try/except. > Thinking this further, it might enable much more use-cases and stays > extensible without the need to invent all kinds of special keywords. > > > So, this "for" might roughly be equivalent to (using the "old for"): > > class NeverExecuted(StopIteration): > pass > > i = iter(collection) > try: > e = next(i) > except StopIteration: > if [NeverExecuted/StopIteration is catched]: > raise NeverExecuted > else: > # do for item e > for e in i: > # do for item e > else: > if [StopIteration is catched]: > raise StopIteration > > > I hope that's not too convoluted. > > Best, > Sven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From guido at python.org Tue Mar 22 13:58:15 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 22 Mar 2016 10:58:15 -0700 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: One thing I can do without committing to too much is to add a Text type to typing.py. It would have no (defined) behavior but it could be imported and used in annotations. Then mypy and other checkers could start using it and we could even experiment with different proposals without having to make more changes to typing.py (which we've found are hard to push out, because it's in the 3.5 stdlib -- it's provisional so we can change it, but we can't easily change what's already in 3.5.0 or 3.5.1). Should it be possible to subclass Text, and what should it mean? Or perhaps at runtime (i.e. in typing.py) Text would just be an alias for str in Python 3 and an alias for unicode in Python 2? That's easiest. From srkunze at mail.de Tue Mar 22 14:02:26 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 22 Mar 2016 19:02:26 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F18797.30209@gmail.com> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <56F18797.30209@gmail.com> Message-ID: <56F188B2.3060102@mail.de> On 22.03.2016 18:57, Michel Desmoulin wrote: > > Le 22/03/2016 18:21, Sven R. Kunze a ?crit : >> On 22.03.2016 16:09, Stephen J. Turnbull wrote: >>> Chris Barker writes: >>> > All that being said: >>> > >>> > how about "elempty"?, to go with elif? >>> >>> -1 Not to my taste, to say the least. >> Hmm, it seems there is no easy solution for this. What do you think >> about an alternative that can handle more than empty and else? >> >> for item in collection: >> # do for item >> except NeverExecuted: >> # do if collection is empty >> >> It basically merges "try" and "for" and make "for" emit EmptyCollection. >> So, independent of the initial "never executed loop body" use-case, one >> could also emulate the "else" clause by: >> >> for item in collection: >> # do for item >> except StopIteration: >> # do after the loop >> > It's an interesting idea. Generalizing except for more block I mean. > > Imagine: > > if user: > print(user.permissions[0]) > except IndexError: > print('anonymous') > > with open('foo') as f: > print(f.read()) > except IOError: > print('meh') > > for item in collection: > # do for item > except NeverExecuted: > # do if collection is empty > > Not only does it allow interesting new usages such as the for loop empty > pattern, but it does save a level of indentation for many classic use > cases where you would nest try/except. That is indeed an important detail. I know from our internal guidelines that it is recommended to reduce the indentation as much as possible. So, this would come in handy. Best, Sven From ethan at stoneleaf.us Tue Mar 22 14:41:15 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 22 Mar 2016 11:41:15 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F18636.40100@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> Message-ID: <56F191CB.2090106@stoneleaf.us> On 03/22/2016 10:51 AM, Michel Desmoulin wrote: > def foo(p): > with open(p) as f: > def begin: > return x == "BEGIN SECTION" > def end: > return x == "STOP" > return f[begin, end][:10000] > > It's very clean, very convenient, very natural, and memory efficient. Except the 10,000 limit doesn't happen until /after/ the end block is reached -- which could be a million lines later. > Now compare it with itertools: > > from itertools import takewhile, dropwhile, islice > > def foo(p): > with open(p) as f: > def begin: > return x != "BEGIN SECTION" > def end: > return x != "STOP" > return islice(takewhile(end, dropwhile(begin, f)), 0, 10000) > > It's ugly, hard to read, hard to write. Yes, it is. I would make a function: def take_between(begin=0, end=None, limit=None): """ return a list with all items between `begin` and `end`, but no more than `limit` begin -> an `int`, or function that returns `True` on first line to keep end -> None (for no end), an `int` relative to start of iterable, or a function that returns True on last item to keep limit -> max number of items to keep, or None for all items """ > In Python, you are always iterating on something, it makes sense to make > sure we have the best tooling to do at our fingertips. In Python, you are always using data containers -- yet we don't have a plethora of different tree types built in. A simple import is "at our fingertips". On a more supportive note: Are you aware of `more_itertools`[1] ? If it doesn't already have `takeuntil` and `dropuntil` and your (pretty cool) `iter`, perhaps you could get your ideas included there. -- ~Ethan~ [1] https://pypi.python.org/pypi/more-itertools/ From tjreedy at udel.edu Tue Mar 22 14:49:44 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 22 Mar 2016 14:49:44 -0400 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <20160321132952.GD12526@ando.pearwood.info> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> Message-ID: On 3/21/2016 9:29 AM, Steven D'Aprano wrote: > On Sun, Mar 20, 2016 at 11:19:54PM -0400, Terry Reedy wrote: >> On 3/20/2016 9:32 PM, Steven D'Aprano wrote: >> >>> I understood from the keyword that "else" ran *if the for block >>> didn't*, i.e. when the loop iterator is empty. >>> A perfectly natural mistake to make, >> Why is the truth a mistake? > > It's not the truth. Yes it is. When the iterator is empty, possibly after many iterations, and the iterator is tested, the else clause runs. > The else block does *not* only run if the for block doesn't run. I have never, ever, claimed that. Initially empty and empty after all items have been processed are different. >>> matches the most common >>> and familiar use of "else", namely in if...else statements: >> >>> if ...: >>> a >>> else: >>> b >>> >>> >>> You can get a, or b, but not both. In English, "else" represents an >>> alternative. This does not come even close to matching the behaviour of >>> for...else, which (in the absense of a "break" executes a *and* b, >>> rather than a *or* b: >> >> Block a may never be executed. In any case, once it is, the loop starts >> over, the test is repeated, and either a or b is executed. > > It's an IF...else statement. There's no loop. It becomes a while loop when 'go to if-test' is added after 'a'. >>> for ...: >>> a >>> else: >>> b >> >> I see it differently. For both while and for, block b is the >> alternative to executing a, when the condition (explicit in 'while', >> implicit in 'for) is false, just as in an 'if' statement. > > That's not what for...else does. Try it with an empty sequence and a > non-empty sequence: > > for x in [1,2]: print("inside for block") > else: print("inside else block") > > for x in []: print("inside for block") > else: print("inside else block") > > BOTH the empty and non-empty cases print "inside else block". It is > certainly not "the alternative". Each time next(iterable) is called by the for loop machinery, executing b is the alternative to executing a. You and I view the overall looping process differently. The way I see it, as repeating if-else whenever if is true, makes 'else' perfectly sensible. You see it differently, in a way that makes 'else' less sensible. Here is an alternative way to construct 'while condition' from if-else. Imagine that we only have a loop-forever primative (implemented internally as an unconditional jump). Python currently spells this as 'while True'. Then 'while condition' could written as the following (which works today). while True: if condition: true_part() else: false_part() break The false_part is the alternative to the true_part. -- Terry Jan Reedy From brenbarn at brenbarn.net Tue Mar 22 14:53:46 2016 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Tue, 22 Mar 2016 11:53:46 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F08AFE.2040501@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F08AFE.2040501@gmail.com> Message-ID: <56F194BA.9080205@brenbarn.net> On 2016-03-21 16:59, Michel Desmoulin wrote: > The solution I'm currently using involves having a class called g, and > everytime I want to manipulate an iterable, I just wrap it in g(). > > Then I got an object with all those semantics (and actually a lot more). I'm not sure I see the value in new syntax, but I think what you describe here would be a useful addition to itertools. Yes, you would still have to import itertools, but you could just import this one "convenient iterator" class and use that, which could allow for concise but readable code that otherwise would have to use many different itertools functions. It would require some thought about what sorts of overloaded operators (if any) would be appropriate for various itertools functions, but I think it could still be a gain even if the main advantage was just being able to write stuff like "niceIterator(x)[10:20] + niceIterator(y)[20:30]". Since it would just be a convenience wrapper, there need be no worries about it not working for particular kinds of iterables (e.g., ones that already define their own behavior for certain operators); you just wouldn't use it for ones where it masked behavior you cared about on the underlying iterable. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From k7hoven at gmail.com Tue Mar 22 15:10:06 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 22 Mar 2016 21:10:06 +0200 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F191CB.2090106@stoneleaf.us> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> Message-ID: On Tue, Mar 22, 2016 at 8:41 PM, Ethan Furman wrote: > On 03/22/2016 10:51 AM, Michel Desmoulin wrote: > >> def foo(p): >> with open(p) as f: >> def begin: >> return x == "BEGIN SECTION" >> def end: >> return x == "STOP" >> return f[begin, end][:10000] >> >> It's very clean, very convenient, very natural, and memory efficient. > > > Except the 10,000 limit doesn't happen until /after/ the end block is > reached -- which could be a million lines later. if f[begin, end] is a generator, the 10000 limit may happen before the end block is reached, which I think was the point. Python 3 does give generators and iterators a more central role than Python 2. One thought that comes to mind is to add some stuff from itertools as attributes of iter. For example: iter.slice(...). Kind of like a factory method, although iter is not really a type. - Koos From ethan at stoneleaf.us Tue Mar 22 15:23:49 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 22 Mar 2016 12:23:49 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> Message-ID: <56F19BC5.30407@stoneleaf.us> On 03/22/2016 12:10 PM, Koos Zevenhoven wrote: > On Tue, Mar 22, 2016 at 8:41 PM, Ethan Furman wrote: >> On 03/22/2016 10:51 AM, Michel Desmoulin wrote: >> >>> def foo(p): >>> with open(p) as f: >>> def begin: >>> return x == "BEGIN SECTION" >>> def end: >>> return x == "STOP" >>> return f[begin, end][:10000] >>> >>> It's very clean, very convenient, very natural, and memory efficient. >> >> >> Except the 10,000 limit doesn't happen until /after/ the end block is >> reached -- which could be a million lines later. > > > if f[begin, end] is a generator, the 10000 limit may happen before the > end block is reached, which I think was the point. That is wrong, which was my point: the `[:10000]` doesn't take effect until after `f[begin:end]` (whatever it is) is evaluated. -- ~Ethan~ From desmoulinmichel at gmail.com Tue Mar 22 15:26:29 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 20:26:29 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F19BC5.30407@stoneleaf.us> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> <56F19BC5.30407@stoneleaf.us> Message-ID: <56F19C65.4030801@gmail.com> Le 22/03/2016 20:23, Ethan Furman a ?crit : > On 03/22/2016 12:10 PM, Koos Zevenhoven wrote: >> On Tue, Mar 22, 2016 at 8:41 PM, Ethan Furman wrote: >>> On 03/22/2016 10:51 AM, Michel Desmoulin wrote: >>> >>>> def foo(p): >>>> with open(p) as f: >>>> def begin: >>>> return x == "BEGIN SECTION" >>>> def end: >>>> return x == "STOP" >>>> return f[begin, end][:10000] >>>> >>>> It's very clean, very convenient, very natural, and memory efficient. >>> >>> >>> Except the 10,000 limit doesn't happen until /after/ the end block is >>> reached -- which could be a million lines later. >> >> >> if f[begin, end] is a generator, the 10000 limit may happen before the >> end block is reached, which I think was the point. > > That is wrong, which was my point: the `[:10000]` doesn't take effect > until after `f[begin:end]` (whatever it is) is evaluated. > > -- > ~Ethan~ [begin, end] and [:10000] are applied one next() at a time. begin, then end, then :10000 for the first next(), then again in that order for the following next() call, etc. That's the whole point. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From k7hoven at gmail.com Tue Mar 22 15:26:45 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 22 Mar 2016 21:26:45 +0200 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> Message-ID: On Tue, Mar 22, 2016 at 9:10 PM, Koos Zevenhoven wrote: > On Tue, Mar 22, 2016 at 8:41 PM, Ethan Furman wrote: >> On 03/22/2016 10:51 AM, Michel Desmoulin wrote: >> >>> def foo(p): >>> with open(p) as f: >>> def begin: >>> return x == "BEGIN SECTION" >>> def end: >>> return x == "STOP" >>> return f[begin, end][:10000] >>> >>> It's very clean, very convenient, very natural, and memory efficient. >> >> >> Except the 10,000 limit doesn't happen until /after/ the end block is >> reached -- which could be a million lines later. > > > if f[begin, end] is a generator, the 10000 limit may happen before the > end block is reached, which I think was the point. > I should add, though, that the the function should probably return list(f[begin, end][:10000]) or something, before the file f is closed. - Koos From ethan at stoneleaf.us Tue Mar 22 15:34:01 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 22 Mar 2016 12:34:01 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> Message-ID: <56F19E29.7090700@stoneleaf.us> On 03/22/2016 11:49 AM, Terry Reedy wrote: > On 3/21/2016 9:29 AM, Steven D'Aprano wrote: >> On Sun, Mar 20, 2016 at 11:19:54PM -0400, Terry Reedy wrote: >>> On 3/20/2016 9:32 PM, Steven D'Aprano wrote: >>> >>>> I understood from the keyword that "else" ran *if the for block >>>> didn't*, i.e. when the loop iterator is empty. >>>> A perfectly natural mistake to make, > >>> Why is the truth a mistake? >> >> It's not the truth. > > Yes it is. When the iterator is empty, possibly after many iterations, > and the iterator is tested, the else clause runs. > > > The else block does *not* only run if the for block doesn't run. > > I have never, ever, claimed that. Initially empty and empty after all > items have been processed are different. This is the heart of the issue: *initially empty* and *empty after all items are exhausted* are different, but for/else and while/else treat them the same. -- ~Ethan~ From ethan at stoneleaf.us Tue Mar 22 15:39:24 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 22 Mar 2016 12:39:24 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F19C65.4030801@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> <56F19BC5.30407@stoneleaf.us> <56F19C65.4030801@gmail.com> Message-ID: <56F19F6C.2070605@stoneleaf.us> On 03/22/2016 12:26 PM, Michel Desmoulin wrote: > Le 22/03/2016 20:23, Ethan Furman a ?crit : >> On 03/22/2016 12:10 PM, Koos Zevenhoven wrote: >>> On Tue, Mar 22, 2016 at 8:41 PM, Ethan Furman wrote: >>>> On 03/22/2016 10:51 AM, Michel Desmoulin wrote: >>>> >>>>> def foo(p): >>>>> with open(p) as f: >>>>> def begin: >>>>> return x == "BEGIN SECTION" >>>>> def end: >>>>> return x == "STOP" >>>>> return f[begin, end][:10000] >>>>> >>>>> It's very clean, very convenient, very natural, and memory efficient. >>>> >>>> >>>> Except the 10,000 limit doesn't happen until /after/ the end block is >>>> reached -- which could be a million lines later. >>> >>> >>> if f[begin, end] is a generator, the 10000 limit may happen before the >>> end block is reached, which I think was the point. >> >> That is wrong, which was my point: the `[:10000]` doesn't take effect >> until after `f[begin:end]` (whatever it is) is evaluated. > > [begin, end] and [:10000] are applied one next() at a time. > > begin, then end, then :10000 for the first next(), > > then again in that order for the following next() call, etc. > > That's the whole point. That may be the point, but that is not what the above code does. Since you don't believe me, let's break it down: f[begin:end] -> grabs a section of p. This could be 5 lines or 50000000 [:10000] -> take the first 10000 of the previous result return -> send those (up to) 10000 lines back -- ~Ethan~ From ethan at stoneleaf.us Tue Mar 22 15:41:46 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 22 Mar 2016 12:41:46 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F19F6C.2070605@stoneleaf.us> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> <56F19BC5.30407@stoneleaf.us> <56F19C65.4030801@gmail.com> <56F19F6C.2070605@stoneleaf.us> Message-ID: <56F19FFA.3060303@stoneleaf.us> On 03/22/2016 12:39 PM, Ethan Furman wrote: > On 03/22/2016 12:26 PM, Michel Desmoulin wrote: >> Le 22/03/2016 20:23, Ethan Furman a ?crit : >>> On 03/22/2016 12:10 PM, Koos Zevenhoven wrote: >>>> On Tue, Mar 22, 2016 at 8:41 PM, Ethan Furman wrote: >>>>> On 03/22/2016 10:51 AM, Michel Desmoulin wrote: >>>>> >>>>>> def foo(p): >>>>>> with open(p) as f: >>>>>> def begin: >>>>>> return x == "BEGIN SECTION" >>>>>> def end: >>>>>> return x == "STOP" >>>>>> return f[begin, end][:10000] >>>>>> >>>>>> It's very clean, very convenient, very natural, and memory efficient. >>>>> >>>>> >>>>> Except the 10,000 limit doesn't happen until /after/ the end block is >>>>> reached -- which could be a million lines later. >>>> >>>> >>>> if f[begin, end] is a generator, the 10000 limit may happen before the >>>> end block is reached, which I think was the point. >>> >>> That is wrong, which was my point: the `[:10000]` doesn't take effect >>> until after `f[begin:end]` (whatever it is) is evaluated. > > >> [begin, end] and [:10000] are applied one next() at a time. >> >> begin, then end, then :10000 for the first next(), >> >> then again in that order for the following next() call, etc. >> >> That's the whole point. > > That may be the point, but that is not what the above code does. Since > you don't believe me, let's break it down: > > f[begin:end] -> grabs a section of p. This could be 5 lines or 50000000 > > [:10000] -> take the first 10000 of the previous result > > return -> send those (up to) 10000 lines back In case my point still isn't clear: if you just stored 49,990,000 lines just to throw them away, you are not being memory efficient. -- ~Ethan~ From desmoulinmichel at gmail.com Tue Mar 22 15:43:55 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 20:43:55 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F19FFA.3060303@stoneleaf.us> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> <56F19BC5.30407@stoneleaf.us> <56F19C65.4030801@gmail.com> <56F19F6C.2070605@stoneleaf.us> <56F19FFA.3060303@stoneleaf.us> Message-ID: <56F1A07B.1030000@gmail.com> No you didn't store the lines, the whole points is that they are returning generators. In that context, f[begin, end] does itertools.takewhile and dropwhile and [:10000] is doing islice(0, 10000). None of those functions store anything. Please read the begining of thread. Le 22/03/2016 20:41, Ethan Furman a ?crit : > On 03/22/2016 12:39 PM, Ethan Furman wrote: >> On 03/22/2016 12:26 PM, Michel Desmoulin wrote: >>> Le 22/03/2016 20:23, Ethan Furman a ?crit : >>>> On 03/22/2016 12:10 PM, Koos Zevenhoven wrote: >>>>> On Tue, Mar 22, 2016 at 8:41 PM, Ethan Furman wrote: >>>>>> On 03/22/2016 10:51 AM, Michel Desmoulin wrote: >>>>>> >>>>>>> def foo(p): >>>>>>> with open(p) as f: >>>>>>> def begin: >>>>>>> return x == "BEGIN SECTION" >>>>>>> def end: >>>>>>> return x == "STOP" >>>>>>> return f[begin, end][:10000] >>>>>>> >>>>>>> It's very clean, very convenient, very natural, and memory >>>>>>> efficient. >>>>>> >>>>>> >>>>>> Except the 10,000 limit doesn't happen until /after/ the end block is >>>>>> reached -- which could be a million lines later. >>>>> >>>>> >>>>> if f[begin, end] is a generator, the 10000 limit may happen before the >>>>> end block is reached, which I think was the point. >>>> >>>> That is wrong, which was my point: the `[:10000]` doesn't take effect >>>> until after `f[begin:end]` (whatever it is) is evaluated. >> > >>> [begin, end] and [:10000] are applied one next() at a time. >>> >>> begin, then end, then :10000 for the first next(), >>> >>> then again in that order for the following next() call, etc. >>> >>> That's the whole point. >> >> That may be the point, but that is not what the above code does. Since >> you don't believe me, let's break it down: >> >> f[begin:end] -> grabs a section of p. This could be 5 lines or 50000000 >> >> [:10000] -> take the first 10000 of the previous result >> >> return -> send those (up to) 10000 lines back > > In case my point still isn't clear: if you just stored 49,990,000 lines > just to throw them away, you are not being memory efficient. > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From desmoulinmichel at gmail.com Tue Mar 22 15:45:49 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 20:45:49 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> Message-ID: <56F1A0ED.9080801@gmail.com> Le 22/03/2016 20:26, Koos Zevenhoven a ?crit : > On Tue, Mar 22, 2016 at 9:10 PM, Koos Zevenhoven wrote: >> On Tue, Mar 22, 2016 at 8:41 PM, Ethan Furman wrote: >>> On 03/22/2016 10:51 AM, Michel Desmoulin wrote: >>> >>>> def foo(p): >>>> with open(p) as f: >>>> def begin: >>>> return x == "BEGIN SECTION" >>>> def end: >>>> return x == "STOP" >>>> return f[begin, end][:10000] >>>> >>>> It's very clean, very convenient, very natural, and memory efficient. >>> >>> >>> Except the 10,000 limit doesn't happen until /after/ the end block is >>> reached -- which could be a million lines later. >> >> >> if f[begin, end] is a generator, the 10000 limit may happen before the >> end block is reached, which I think was the point. >> > > I should add, though, that the the function should probably return > list(f[begin, end][:10000]) or something, before the file f is closed. > > - Koos Yeah the semantic of my function is wrong. It should rather expect a file like object as a parameter to avoid this bug. Plus, the begin() and end() functions are actually syntax errors ^^ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From k7hoven at gmail.com Tue Mar 22 15:52:51 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 22 Mar 2016 21:52:51 +0200 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F19FFA.3060303@stoneleaf.us> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> <56F19BC5.30407@stoneleaf.us> <56F19C65.4030801@gmail.com> <56F19F6C.2070605@stoneleaf.us> <56F19FFA.3060303@stoneleaf.us> Message-ID: If your point is that generators can sometimes be hard, then your point is clear :-). Maybe using them should not be made to look more straightforward than it actually is. Which brings me back to... One thought that comes to mind is to add some stuff from itertools as attributes of iter. For example: iter.slice(...). Kind of like a factory method, although iter is not really a type. - Koos From desmoulinmichel at gmail.com Tue Mar 22 15:58:16 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 20:58:16 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> <56F19BC5.30407@stoneleaf.us> <56F19C65.4030801@gmail.com> <56F19F6C.2070605@stoneleaf.us> <56F19FFA.3060303@stoneleaf.us> Message-ID: <56F1A3D8.3070800@gmail.com> Le 22/03/2016 20:52, Koos Zevenhoven a ?crit : > If your point is that generators can sometimes be hard, then your > point is clear :-). Maybe using them should not be made to look more > straightforward than it actually is. I would have made the same mistake with itertools, let be frank. Which brings me back to... > > One thought that comes to mind is to add some stuff from itertools as > attributes of iter. For example: iter.slice(...). Kind of like a > factory method, although iter is not really a type. It's an interesting though. Actually we could also make something like iter.wraps(iterable) to return and object with the properties discussed in that thread. It would be fully backward compatible, explicit, yet convenient enough. > > - Koos > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From abarnert at yahoo.com Tue Mar 22 16:03:35 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 22 Mar 2016 13:03:35 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F18636.40100@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> Message-ID: On Mar 22, 2016, at 10:51, Michel Desmoulin wrote: > > For every slice. We don't, because slicing is part of our standard data > processing toolkit. And I think we can make it even better. > > We already do in some places. E.G: range(10)[3:5] works while range() > generate values on the fly. Why ? Because it's convenient, expressive > and Pythonic. Range is a sequence, just like list and tuple, not an iterator, like generator and list_iterator. So it supports slicing because sequences generally support slicing. Not because it's so convenient and expressive that it's worth making a special case, just because it follows the normal rules. It's amazing to me that so many people still describe xrange/3.x range as an iterator or generator, and then draw further incorrect lessons from that. Yes, the docs were a bit confusing up to about 2.4, but how long ago is that now? Lazy sequences are still sequences. Strict non-sequences (like dict) are still not sequences. Lazy vs. strict has almost[1] nothing to do with sequence vs. iterator (or sequence vs. non-sequence collection, or collection vs. iterator, or anything else.) --- [1]: Just "almost nothing" because you can't really write a strict generator--or, rather, you sort of can in a way, but it still has to look lazy to the user, so there's no point. From desmoulinmichel at gmail.com Tue Mar 22 16:11:11 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 21:11:11 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> Message-ID: <56F1A6DF.2010808@gmail.com> Le 22/03/2016 21:03, Andrew Barnert a ?crit : > On Mar 22, 2016, at 10:51, Michel Desmoulin wrote: >> >> For every slice. We don't, because slicing is part of our standard data >> processing toolkit. And I think we can make it even better. >> >> We already do in some places. E.G: range(10)[3:5] works while range() >> generate values on the fly. Why ? Because it's convenient, expressive >> and Pythonic. > > Range is a sequence, just like list and tuple, not an iterator, like generator and list_iterator. So it supports slicing because sequences generally support slicing. Not because it's so convenient and expressive that it's worth making a special case, just because it follows the normal rules. > > It's amazing to me that so many people still describe xrange/3.x range as an iterator or generator, and then draw further incorrect lessons from that. Yes, the docs were a bit confusing up to about 2.4, but how long ago is that now? Lazy sequences are still sequences. Strict non-sequences (like dict) are still not sequences. Lazy vs. strict has almost[1] nothing to do with sequence vs. iterator (or sequence vs. non-sequence collection, or collection vs. iterator, or anything else.) > > --- Yes, I worded my title with "all iterables", it was a mistake, I didn't think people would raise the debate on dict, set, etc because it was obvious to me that nobody would imagine I would try to make slicing work on those. I was wrong, but please don't make drown the debate into a battle of semantics. Ok, let's recenter to lazy sequences, and how convenient it would be to be able to limit them to a size or a condition without needing itertools. And in that context, the fact is that range(1000000) doesn't create 1000000 items in memory, and it's very convenient to be able to slice it. And it would would be as convenient to be able to do so on generators (which are sequences) instead of using islice. From abarnert at yahoo.com Tue Mar 22 16:11:52 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 22 Mar 2016 13:11:52 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> <56F19BC5.30407@stoneleaf.us> <56F19C65.4030801@gmail.com> <56F19F6C.2070605@stoneleaf.us> <56F19FFA.3060303@stoneleaf.us> Message-ID: On Mar 22, 2016, at 12:52, Koos Zevenhoven wrote: > > One thought that comes to mind is to add some stuff from itertools as > attributes of iter. For example: iter.slice(...). Kind of like a > factory method, although iter is not really a type. The one thing from itertools that I think people most miss out on by not knowing itertools is chain.from_iterable, with chain itself a close second. Would calling them iter.chain.from_iterable and iter.chain help? I'm not sure. Other people have proposed promoting one or both to builtins (maybe named flatten and chain, respectively), but that hasn't gone over very well. Making them "class methods" of iter seems a lot less disruptive. The other thing that I think might help is a way of writing "slice literals" more easily. Just having islice easier to find doesn't help much, because the main hurdle in writing and reading the code seems to be translating between the natural slice syntax and islice's (range-ish) unpythonic signature. Maybe islice(it, slice[:10000])? Or even islice(it)[:10000]? From k7hoven at gmail.com Tue Mar 22 16:13:03 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 22 Mar 2016 22:13:03 +0200 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F1A0ED.9080801@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> <56F1A0ED.9080801@gmail.com> Message-ID: On Tue, Mar 22, 2016 at 9:45 PM, Michel Desmoulin wrote: > > > Le 22/03/2016 20:26, Koos Zevenhoven a ?crit : >> On Tue, Mar 22, 2016 at 9:10 PM, Koos Zevenhoven wrote: >>> On Tue, Mar 22, 2016 at 8:41 PM, Ethan Furman wrote: >>>> On 03/22/2016 10:51 AM, Michel Desmoulin wrote: >>>> >>>>> def foo(p): >>>>> with open(p) as f: >>>>> def begin: >>>>> return x == "BEGIN SECTION" >>>>> def end: >>>>> return x == "STOP" >>>>> return f[begin, end][:10000] >>>>> >>>>> It's very clean, very convenient, very natural, and memory efficient. >>>> >>>> >>>> Except the 10,000 limit doesn't happen until /after/ the end block is >>>> reached -- which could be a million lines later. >>> >>> >>> if f[begin, end] is a generator, the 10000 limit may happen before the >>> end block is reached, which I think was the point. >>> >> >> I should add, though, that the the function should probably return >> list(f[begin, end][:10000]) or something, before the file f is closed. >> >> - Koos > > Yeah the semantic of my function is wrong. It should rather expect a > file like object as a parameter to avoid this bug. Yeah, and perhaps be a generator function and yield from f[begin:end][:10000] instead of returning it. But beginners may end up trying this. It may be better to have them as functions (either in itertools or perhaps as attributes of iter), which may help them see what's actually going on. Or help them see that they actually don't know how it works and should just make a for loop for this to be more clear to them. - Koos From ethan at stoneleaf.us Tue Mar 22 16:17:26 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 22 Mar 2016 13:17:26 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F1A07B.1030000@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> <56F19BC5.30407@stoneleaf.us> <56F19C65.4030801@gmail.com> <56F19F6C.2070605@stoneleaf.us> <56F19FFA.3060303@stoneleaf.us> <56F1A07B.1030000@gmail.com> Message-ID: <56F1A856.5090505@stoneleaf.us> On 03/22/2016 12:43 PM, Michel Desmoulin wrote: > No you didn't store the lines, the whole points is that they are > returning generators. > > In that context, f[begin, end] does itertools.takewhile and > dropwhile and [:10000] is doing islice(0, 10000). None of those > functions store anything. > > Please read the begining of thread. I presume you are referring to this line: > The slicing would then return a list if it's a list, a tuple if it's > a tuple, and a islice(generator) if it's a generator. Of which the big problem is that the object returned from an open call is not a list, not a tuple, and not a generator. For the sake of argument let's pretend you meant "and an islice(iter) for everything else". So the actual proposal is to add `__getitem__` to `object` (that way all classes that define their own `__getitem__` such as list and tuple are not affected), and the substance of this new `__getitem__` is to: - check for an `__iter__` method (if none, raise TypeError) - check the start, stop, step arguments to determine whether dropwhile, takewhile, or islice is needed - return the appropriate object Okay, it's an interesting proposal. I would suggest you make the changes, run the test suite, see if anything blows up. As a short-cut maybe you could add the code to the appropriate ABC and test that way. -- ~Ethan~ From abarnert at yahoo.com Tue Mar 22 16:20:21 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 22 Mar 2016 13:20:21 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F1A6DF.2010808@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F1A6DF.2010808@gmail.com> Message-ID: <55478086-95F0-4AFD-BEA1-914552A0D4C0@yahoo.com> On Mar 22, 2016, at 13:11, Michel Desmoulin wrote: > > Le 22/03/2016 21:03, Andrew Barnert a ?crit : >> On Mar 22, 2016, at 10:51, Michel Desmoulin wrote: >>> >>> For every slice. We don't, because slicing is part of our standard data >>> processing toolkit. And I think we can make it even better. >>> >>> We already do in some places. E.G: range(10)[3:5] works while range() >>> generate values on the fly. Why ? Because it's convenient, expressive >>> and Pythonic. >> >> Range is a sequence, just like list and tuple, not an iterator, like generator and list_iterator. So it supports slicing because sequences generally support slicing. Not because it's so convenient and expressive that it's worth making a special case, just because it follows the normal rules. >> >> It's amazing to me that so many people still describe xrange/3.x range as an iterator or generator, and then draw further incorrect lessons from that. Yes, the docs were a bit confusing up to about 2.4, but how long ago is that now? Lazy sequences are still sequences. Strict non-sequences (like dict) are still not sequences. Lazy vs. strict has almost[1] nothing to do with sequence vs. iterator (or sequence vs. non-sequence collection, or collection vs. iterator, or anything else.) >> >> --- > > Yes, I worded my title with "all iterables", it was a mistake, I didn't > think people would raise the debate on dict, set, etc because it was > obvious to me that nobody would imagine I would try to make slicing work > on those. > > I was wrong, but please don't make drown the debate into a battle of > semantics. Ok, let's recenter to lazy sequences, and how convenient it > would be to be able to limit them to a size or a condition without > needing itertools. This is not just a battle of (meaningless) semantics. You seem to be completely misinterpreting the actual language semantics of Python, and therefore making proposals that don't make any sense. Case in point: you can _already_ limit lazy sequences to a size without needing itertools. That's exactly what you're doing in your range example. So we don't need to fix the language to make something possible when that something is already possible. > And in that context, the fact is that range(1000000) doesn't create > 1000000 items in memory, and it's very convenient to be able to slice > it. And it would would be as convenient to be able to do so on > generators (which are sequences) instead of using islice. No, generators are _not_ sequences. And that's exactly the point you're missing. You have to understand the difference between iterators and other iterables to use Python effectively. That's just fundamental to so much of Python that there's no way around that. Maybe that was a language design mistake, but trying to muddy the waters further and confuse even more people is hardly a solution. Meanwhile, the range type points to a ready solution: when you want slicing (or other sequence behavior) with laziness, you can always write a lazy sequence type, usually in only a few lines of code. (I've got a couple blog posts on it, if you can't figure it out yourself.) And that solves the problem without introducing any confusion about what it should mean to slice an iterator, or any problems of the "if header(spam[:6]): process_body(spam[6:])" type. From abarnert at yahoo.com Tue Mar 22 16:27:06 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 22 Mar 2016 13:27:06 -0700 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: On Mar 22, 2016, at 10:58, Guido van Rossum wrote: > > One thing I can do without committing to too much is to add a Text > type to typing.py. It would have no (defined) behavior but it could be > imported and used in annotations. Then mypy and other checkers could > start using it and we could even experiment with different proposals > without having to make more changes to typing.py (which we've found > are hard to push out, because it's in the 3.5 stdlib -- it's > provisional so we can change it, but we can't easily change what's > already in 3.5.0 or 3.5.1). > > Should it be possible to subclass Text, and what should it mean? > > Or perhaps at runtime (i.e. in typing.py) Text would just be an alias > for str in Python 3 and an alias for unicode in Python 2? That's > easiest. It seems like the worry is that you may need to change it again (e.g., to some "virtual type" that's like unicode in 2.7 except that it doesn't have a constructor from str or methods that accept str), and changing something that's in the stdlib (even in a provisional module) is hard? If so, could you define it as an alias for str in Python 3.5+, and leave it up to backports to define appropriately? Of course you're writing the 2.7 backport, and you'll define it as an alias for unicode--but if you later decide that was too inflexible, you can change the backport without having any effect on the PEP, 3.5 docs, or 3.5 stdlib. From vito.detullio at gmail.com Tue Mar 22 16:36:23 2016 From: vito.detullio at gmail.com (Vito De Tullio) Date: Tue, 22 Mar 2016 21:36:23 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body References: <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <56F1357D.9040507@mail.de> <20160322131018.GF12526@ando.pearwood.info> <20160322134826.GH12526@ando.pearwood.info> Message-ID: Steven D'Aprano wrote: > def test(): > x = sentinel = object() > iterable = some_function() > for x in iterable: > # you know the rest onestly, I just don't like the sentinel approach... while not perfect I prefer the explicit check on the throws of the StopIteration of the next function iterable = iter(some_function()) try: e = iter(iterable) except StopIteration: # empty else: # stuff with e for e in iterable: # stuff with e -- By ZeD From guido at python.org Tue Mar 22 17:07:40 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 22 Mar 2016 14:07:40 -0700 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: Good point -- changing the 2.7 backport is simpler than changing the 3.5 stdlib. And it's beyond a shadow of a doubt that in 3.x it'll just be an alias for str. On Tue, Mar 22, 2016 at 1:27 PM, Andrew Barnert wrote: > On Mar 22, 2016, at 10:58, Guido van Rossum wrote: >> >> One thing I can do without committing to too much is to add a Text >> type to typing.py. It would have no (defined) behavior but it could be >> imported and used in annotations. Then mypy and other checkers could >> start using it and we could even experiment with different proposals >> without having to make more changes to typing.py (which we've found >> are hard to push out, because it's in the 3.5 stdlib -- it's >> provisional so we can change it, but we can't easily change what's >> already in 3.5.0 or 3.5.1). >> >> Should it be possible to subclass Text, and what should it mean? >> >> Or perhaps at runtime (i.e. in typing.py) Text would just be an alias >> for str in Python 3 and an alias for unicode in Python 2? That's >> easiest. > > It seems like the worry is that you may need to change it again (e.g., to some "virtual type" that's like unicode in 2.7 except that it doesn't have a constructor from str or methods that accept str), and changing something that's in the stdlib (even in a provisional module) is hard? > > If so, could you define it as an alias for str in Python 3.5+, and leave it up to backports to define appropriately? Of course you're writing the 2.7 backport, and you'll define it as an alias for unicode--but if you later decide that was too inflexible, you can change the backport without having any effect on the PEP, 3.5 docs, or 3.5 stdlib. > -- --Guido van Rossum (python.org/~guido) From andrey.vlasovskikh at gmail.com Tue Mar 22 17:36:40 2016 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Wed, 23 Mar 2016 00:36:40 +0300 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: <9BE878B7-B2A5-4E16-B798-5381FC6DBF44@gmail.com> > 2016-03-19, ? 5:12, Wes Turner ???????(?): > > This sounds like a more correct approach, thanks. > > Looking at MarkupSafe (and, now, f-strings), would/will it be possible to use Typing.Text as a base class for even-more abstract string types ("strypes") e.g. XML, XHTML, HTML4, HTML5, HTML5.1, SQL? There are implicit casts and contextual adaptations/transformations (which MarkupSafe specs a bit). (I've no real code here, just a general idea that we're not tracking enough string metadata to be safe here) I believe having separate string types for XML or SQL content is out of the scope of this proposal. In PyCharm we already treat the contents of string literals with SQL as a separate SQL syntax tree and we understand basic string operations like concatenation or formatting. Going beyond that with the help of XML/SQL/etc. string types is possible, but I doubt we need a standard for that. -- Andrey Vlasovskikh Web: http://pirx.ru/ From wes.turner at gmail.com Tue Mar 22 17:40:39 2016 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 22 Mar 2016 16:40:39 -0500 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: <9BE878B7-B2A5-4E16-B798-5381FC6DBF44@gmail.com> References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> <9BE878B7-B2A5-4E16-B798-5381FC6DBF44@gmail.com> Message-ID: On Mar 22, 2016 4:36 PM, "Andrey Vlasovskikh" wrote: > > > > 2016-03-19, ? 5:12, Wes Turner ???????(?): > > > > This sounds like a more correct approach, thanks. > > > > Looking at MarkupSafe (and, now, f-strings), would/will it be possible to use Typing.Text as a base class for even-more abstract string types ("strypes") e.g. XML, XHTML, HTML4, HTML5, HTML5.1, SQL? There are implicit casts and contextual adaptations/transformations (which MarkupSafe specs a bit). (I've no real code here, just a general idea that we're not tracking enough string metadata to be safe here) > > I believe having separate string types for XML or SQL content is out of the scope of this proposal. > > In PyCharm we already treat the contents of string literals with SQL as a separate SQL syntax tree and we understand basic string operations like concatenation or formatting. Going beyond that with the help of XML/SQL/etc. string types is possible, but I doubt we need a standard for that. At the least, it would be helpful to either have: a) a slot / attribute for additional string type metadata (is this an object subclass that I can just add attrs to) b) a minimal Text base class SQL is harder because dialects. > > -- > Andrey Vlasovskikh > > Web: http://pirx.ru/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Mar 22 17:45:34 2016 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 22 Mar 2016 16:45:34 -0500 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> <9BE878B7-B2A5-4E16-B798-5381FC6DBF44@gmail.com> Message-ID: ... OT (I'm finished): https://github.com/cloudera/ibis/blob/master/ibis/sql/alchemy.py On Mar 22, 2016 4:40 PM, "Wes Turner" wrote: > > On Mar 22, 2016 4:36 PM, "Andrey Vlasovskikh" < > andrey.vlasovskikh at gmail.com> wrote: > > > > > > > 2016-03-19, ? 5:12, Wes Turner ???????(?): > > > > > > This sounds like a more correct approach, thanks. > > > > > > Looking at MarkupSafe (and, now, f-strings), would/will it be possible > to use Typing.Text as a base class for even-more abstract string types > ("strypes") e.g. XML, XHTML, HTML4, HTML5, HTML5.1, SQL? There are implicit > casts and contextual adaptations/transformations (which MarkupSafe specs a > bit). (I've no real code here, just a general idea that we're not tracking > enough string metadata to be safe here) > > > > I believe having separate string types for XML or SQL content is out of > the scope of this proposal. > > > > In PyCharm we already treat the contents of string literals with SQL as > a separate SQL syntax tree and we understand basic string operations like > concatenation or formatting. Going beyond that with the help of > XML/SQL/etc. string types is possible, but I doubt we need a standard for > that. > > At the least, it would be helpful to either have: > > a) a slot / attribute for additional string type metadata (is this an > object subclass that I can just add attrs to) > b) a minimal Text base class > > SQL is harder because dialects. > > > > > -- > > Andrey Vlasovskikh > > > > Web: http://pirx.ru/ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Tue Mar 22 17:47:19 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 22 Mar 2016 22:47:19 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F1A856.5090505@stoneleaf.us> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> <56F19BC5.30407@stoneleaf.us> <56F19C65.4030801@gmail.com> <56F19F6C.2070605@stoneleaf.us> <56F19FFA.3060303@stoneleaf.us> <56F1A07B.1030000@gmail.com> <56F1A856.5090505@stoneleaf.us> Message-ID: <56F1BD67.9040801@gmail.com> Le 22/03/2016 21:17, Ethan Furman a ?crit : > On 03/22/2016 12:43 PM, Michel Desmoulin wrote: >> No you didn't store the lines, the whole points is that they are >> returning generators. >> >> In that context, f[begin, end] does itertools.takewhile and >> dropwhile and [:10000] is doing islice(0, 10000). None of those >> functions store anything. >> >> Please read the begining of thread. > > I presume you are referring to this line: > >> The slicing would then return a list if it's a list, a tuple if it's >> a tuple, and a islice(generator) if it's a generator. > > Of which the big problem is that the object returned from an open call > is not a list, not a tuple, and not a generator. > > For the sake of argument let's pretend you meant "and an islice(iter) > for everything else". > > So the actual proposal is to add `__getitem__` to `object` (that way all > classes that define their own `__getitem__` such as list and tuple are > not affected), and the substance of this new `__getitem__` is to: > > - check for an `__iter__` method (if none, raise TypeError) > - check the start, stop, step arguments to determine whether > dropwhile, takewhile, or islice is needed > - return the appropriate object > > Okay, it's an interesting proposal. > > I would suggest you make the changes, run the test suite, see if > anything blows up. I have literally no idea where to start. I hope this doesn't require editing the C source code because I don't know C. > > As a short-cut maybe you could add the code to the appropriate ABC and > test that way. > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From andrey.vlasovskikh at gmail.com Tue Mar 22 17:51:49 2016 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Wed, 23 Mar 2016 00:51:49 +0300 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: <003E38E6-3529-45DB-A3F0-C5424C2BC6C4@gmail.com> > 2016-03-22, ? 20:58, Guido van Rossum ???????(?): > > One thing I can do without committing to too much is to add a Text > type to typing.py. It would have no (defined) behavior but it could be > imported and used in annotations. Then mypy and other checkers could > start using it and we could even experiment with different proposals > without having to make more changes to typing.py (which we've found > are hard to push out, because it's in the 3.5 stdlib -- it's > provisional so we can change it, but we can't easily change what's > already in 3.5.0 or 3.5.1). > > Should it be possible to subclass Text, and what should it mean? > > Or perhaps at runtime (i.e. in typing.py) Text would just be an alias > for str in Python 3 and an alias for unicode in Python 2? That's > easiest. Defining typing.Text as an alias to str in Python 3 and unicode for Python 2 (the way six.text_type is defined) looks like a good idea. I would recommend to prohibit subclassing typing.Text at the moment in the module docs and in PEP 484. We can always allow subclassing it later, but right now it's not clear wether it's safe or not given the fact that it's defined conditionally for 2/3. -- Andrey Vlasovskikh Web: http://pirx.ru/ From srkunze at mail.de Tue Mar 22 17:52:50 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 22 Mar 2016 22:52:50 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <56F1357D.9040507@mail.de> <20160322131018.GF12526@ando.pearwood.info> <20160322134826.GH12526@ando.pearwood.info> Message-ID: <56F1BEB2.2030407@mail.de> On 22.03.2016 21:36, Vito De Tullio wrote: > Steven D'Aprano wrote: > >> def test(): >> x = sentinel = object() >> iterable = some_function() >> for x in iterable: >> # you know the rest > onestly, I just don't like the sentinel approach... while not perfect I > prefer the explicit check on the throws of the StopIteration of the next > function > > > iterable = iter(some_function()) > try: > e = iter(iterable) > except StopIteration: > # empty > else: > # stuff with e > for e in iterable: > # stuff with e I tend to agree. Best, Sven From andrey.vlasovskikh at gmail.com Tue Mar 22 18:18:30 2016 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Wed, 23 Mar 2016 01:18:30 +0300 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: > 2016-03-19, ? 21:51, Guido van Rossum ???????(?): > > I like the way this is going. I think it needs to be a separate PEP; > PEP 484 is already too long and this topic deserves being written up > carefully (like you have done here). I would like to experiment with various text/binary types for Python 2 and 3 for some time before coming up with a PEP about it. And I would like everybody interested in 2/3 compatible type hints join the discussion. My perspective (mostly PyCharm-specific) might be a bit narrow here. > * Do we really need _AsciiUnicode? I see the point of _AsciiStr, > because Python 2 accepts 'x' + u'' but fails '\xff' + u'', so 'x' > needs to be of type _AsciiStr while '\xff' should not (it should be > just str). However there's no difference in how u'x' is treated from > how u'\u1234' or u'\xff' are treated -- none of them can be > concatenated to '\xff' and all of them can be concatenated to _'x'. I was concerned with UnicodeEncodeErrors in Python 2 during implicit conversions from unicode to bytes: getattr(obj, u'Non-ASCII-name') There are several places in the Python 2 API where these ASCII-based unicode->bytes conversions take place, so the _AsciiUnicode type comes to mind. > * It would be helpful to spell out exactly what is and isn't allowed > when different core types (bytes, str, unicode, Text) meet in Python 2 > and in Python 3. Something like a table with a row and a column for > each and the type of x+y (or "error") in each of the cells. Agreed. I'll try to come up with specific rules for handling text/binary types (bytes, str, unicode, Text, _Ascii*) in Python 2 and 3. For me the rules for dealing with _Ascii* look the most problematic at the moment as it's unclear how these types should propagate via text-handling functions. > * I propose that Python 2+3 mode is just the intersection of what > Python 2 and Python 3 mode allow. (In mypy, I don't think we'll > implement this -- users will just have to run mypy twice with and > without --py2. But for PyCharm it makes sense to be able to declare > this. Yet I think it would be good not to have to spell out separately > which rules it uses, defining it as the intersection of 2 and 3 is all > we need. Yes, there is no need in having a specific 2+3 mode, I was really referring to the intersection of the Python 2 and 3 APIs when the user accesses a text / binary method not available in both. -- Andrey Vlasovskikh Web: http://pirx.ru/ From steve at pearwood.info Tue Mar 22 18:34:22 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 23 Mar 2016 09:34:22 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <56F1357D.9040507@mail.de> <20160322131018.GF12526@ando.pearwood.info> <20160322134826.GH12526@ando.pearwood.info> Message-ID: <20160322223422.GI12526@ando.pearwood.info> On Tue, Mar 22, 2016 at 09:36:23PM +0100, Vito De Tullio wrote: > Steven D'Aprano wrote: > > > def test(): > > x = sentinel = object() > > iterable = some_function() > > for x in iterable: > > # you know the rest > > onestly, I just don't like the sentinel approach... while not perfect I > prefer the explicit check on the throws of the StopIteration of the next > function [...] I have no objection to the try...except approach either. Another approach is to call next() with a default: first = next(iterator, sentinel) if first is sentinel: empty else: for x in itertools.chain([first], iterator): process(x) We're spoiled for choice here: there are many ways to process an empty iterable separately from a non-empty one. -- Steve From steve at pearwood.info Tue Mar 22 19:21:37 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 23 Mar 2016 10:21:37 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F17F14.9090306@mail.de> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> Message-ID: <20160322232136.GJ12526@ando.pearwood.info> On Tue, Mar 22, 2016 at 06:21:24PM +0100, Sven R. Kunze wrote: > On 22.03.2016 16:09, Stephen J. Turnbull wrote: > >Chris Barker writes: > > > All that being said: > > > > > > how about "elempty"?, to go with elif? > > > >-1 Not to my taste, to say the least. > > Hmm, it seems there is no easy solution for this. Possibly with the exception of the three or four previously existing easy solutions :-) > What do you think > about an alternative that can handle more than empty and else? > > for item in collection: > # do for item > except NeverExecuted: > # do if collection is empty > > It basically merges "try" and "for" and make "for" emit EmptyCollection. Does this mean that every single for-loop that doesn't catch NeverExecuted (or EmptyCollection) will raise an exception? If not, then how will this work? Is this a special kind of exception-like process that *only* operates inside for loops? What will an explicit "raise NeverExecuted" do? > So, independent of the initial "never executed loop body" use-case, one > could also emulate the "else" clause by: > > for item in collection: > # do for item > except StopIteration: > # do after the loop That doesn't work, for two reasons: (1) Not all for-loops use iterators. The venerable old "sequence protocol" is still supported for sequences that don't support __iter__. So there may not be any StopIteration raised at all. (2) Even if StopIteration is raised, the for-loop catches it (in a manner of speaking) and consumes it. So to have this work, we would need to have the for-loop re-raise StopIteration... but what happens if you don't include an except StopIteration clause? Does every bare for-loop with no "except" now print a traceback and halt processing? If not, why not? -- Steve From abarnert at yahoo.com Tue Mar 22 19:56:31 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 22 Mar 2016 16:56:31 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <20160322232136.GJ12526@ando.pearwood.info> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> Message-ID: <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> On Mar 22, 2016, at 16:21, Steven D'Aprano wrote: > >> On Tue, Mar 22, 2016 at 06:21:24PM +0100, Sven R. Kunze wrote: >>> On 22.03.2016 16:09, Stephen J. Turnbull wrote: >>> Chris Barker writes: >>>> All that being said: >>>> >>>> how about "elempty"?, to go with elif? >>> >>> -1 Not to my taste, to say the least. >> >> Hmm, it seems there is no easy solution for this. > > Possibly with the exception of the three or four previously existing > easy solutions :-) The only question is whether any of them are obvious enough (even for novices). If not, we could argue whether any proposed change would be significantly _more_ obvious. And, if so, then the question is whether the change is worth the cost. But I think what we have is already obvious enough. (Especially since 90% of the time, when you need to do something special on empty, you explicitly have a sequence, not any iterable, so it's just "if not seq:".) So really, this proposal is really just asking for syntactic sugar that complicates the language in exchange for making some already-understandable code a little more concise, which doesn't seem worth it. >> What do you think >> about an alternative that can handle more than empty and else? >> >> for item in collection: >> # do for item >> except NeverExecuted: >> # do if collection is empty >> >> It basically merges "try" and "for" and make "for" emit EmptyCollection. > > Does this mean that every single for-loop that doesn't catch > NeverExecuted (or EmptyCollection) will raise an exception? Elsewhere he mentioned that EmptyCollection would be a subclass of StopIteration. Presumably, every iterator type (or just the builtin ones, and hopefully "many" others?) would have special code to raise EmptyCollection if empty. Like this pseudocode for list_iterator: def __next__(self): if not self.lst: raise EmptyCollection elif self.i >= len(self.lst): raise StopIteration else: i = self.i self.i += 1 return self.lst[self.i] Or, alternatively, for itself would do this. The for loop bytecode would have to change to stash an "any values seen" flag somewhere such that if it sees StopIteration and hasn't seen any values, it converts that to an EmptyCollection. Or any of the other equivalents (e.g., the compiler could unroll the first PyIter_Next from loop from the rest of them to handle it specially). But this seems like it would add a lot of overhead and complexity to every loop whether desired or not. > If not, then how will this work? Is this a special kind of > exception-like process that *only* operates inside for loops? > > What will an explicit "raise NeverExecuted" do? Presumably that's the same question as what an explicit raise StopIteration does. Just as there's nothing stopping you from writing a __next__ method that raises StopIteration but then yields more values of called again, there's nothing stopping you from raising NeverExecuted pathologically, but you shouldn't do so. M >> So, independent of the initial "never executed loop body" use-case, one >> could also emulate the "else" clause by: >> >> for item in collection: >> # do for item >> except StopIteration: >> # do after the loop > > That doesn't work, for two reasons: > > (1) Not all for-loops use iterators. The venerable old "sequence > protocol" is still supported for sequences that don't support __iter__. > So there may not be any StopIteration raised at all. I think there always is. IIRC, PyObject_Iter (the C API function used by iter and by for loops) actually constructs a sequence iterator object if the object doesn't have tp_iter (__iter__ for Python types) but does have tp_sequence (__getitem__ for Python types, but, e.g., dict has __getitem__ without having tp_sequence). And the for loop doesn't treat that sequence iterator any different from "real" iterators returned by __iter__; it just calls tp_next (__next__) until StopIteration. (And the "other half" of the old-style sequence protocol, that lets old-style sequences be reversed if they have a length, is similarly implemented by the C API underneath the reversed function.) I'm on my phone right now, so I can't double-check any of the details, but I'm 80% sure they're all at least pretty close... > (2) Even if StopIteration is raised, the for-loop catches it (in a > manner of speaking) and consumes it. > > So to have this work, we would need to have the for-loop re-raise > StopIteration... but what happens if you don't include an except > StopIteration clause? Does every bare for-loop with no "except" now > print a traceback and halt processing? If not, why not? I think this could be made to work: a for loop without an except clause handles StopIteration the same as today (by jumping to the else clause), but one that does have one or more except clauses just treats it like a normal exception. Of course this would mean for/except/else is now legal but useless, which could be confusing ("why does my else clause no longer run when I add an 'except ValueError' clause?"). More generally, I think the fact that for/except StopIteration is almost but not quite identical to plain for would be confusing more often than helpful. But I think it is a coherent proposal, even if it's not one I like. :) From nickeubank at gmail.com Tue Mar 22 23:06:48 2016 From: nickeubank at gmail.com (Nick Eubank) Date: Wed, 23 Mar 2016 03:06:48 +0000 Subject: [Python-ideas] `to_file()` method for strings Message-ID: As a social scientists trying to help other social scientists move from language like R, Stata, and Matlab into Python, one of the behaviors I've found unnecessarily difficult to explain is the "file.open()/file.close()" idiom (or, alternatively, context managers). In normal operating systems, and many high level languages, saving is a one-step operation. I understand there are situations where an open file handle is useful, but it seems a simple `to_file` method on strings (essentially wrapping a context-manager) would be really nice, as it would save users from learning this idiom. Apparently there's something like this in the Path library ( https://docs.python.org/dev/library/pathlib.html#pathlib.Path.write_text) , but I suspect most people have no idea about that method (I've been doing python for years and this has always been a personal frustration, and I've asked several others for better options and no one had one to offer), and it seems like it would make much more sense as a string method. If someone has a string they want to save to disk, I can't imagine anyone looking in the Path library. I respect the desire to avoid bloat -- the context manager or open/close idiom has just felt unnecessarily complicated (dare I say unpythonic?) for a common task. Thanks! Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Tue Mar 22 23:29:24 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 22 Mar 2016 23:29:24 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: Message-ID: On Tue, Mar 22, 2016 at 11:06 PM, Nick Eubank wrote: > it seems a simple `to_file` method on strings (essentially wrapping a > context-manager) would be really nice -1 It is a rare situation when you would want to write just a single string to a file. In most cases you write several strings and or do other file operations between opening and closing a file stream. The proposed to_file() method may become an attractive nuisance leading to highly inefficient code. Remember: opening or closing a file is still in most setups a mechanical operation that involves moving macroscopic physical objects, not just electrons. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nickeubank at gmail.com Tue Mar 22 23:32:28 2016 From: nickeubank at gmail.com (Nick Eubank) Date: Wed, 23 Mar 2016 03:32:28 +0000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: Message-ID: My sense is that writing a single string is rare for software developers, but it's pretty common among data scientists -- I output "single strings" to LaTeX all the time. On Tue, Mar 22, 2016 at 8:29 PM Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Tue, Mar 22, 2016 at 11:06 PM, Nick Eubank > wrote: > >> it seems a simple `to_file` method on strings (essentially wrapping a >> context-manager) would be really nice > > > -1 > > It is a rare situation when you would want to write just a single string > to a file. In most cases you write several strings and or do other file > operations between opening and closing a file stream. The proposed > to_file() method may become an attractive nuisance leading to highly > inefficient code. Remember: opening or closing a file is still in most > setups a mechanical operation that involves moving macroscopic physical > objects, not just electrons. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Tue Mar 22 23:33:56 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 22 Mar 2016 20:33:56 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: Message-ID: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> On Mar 22, 2016, at 20:06, Nick Eubank wrote: > > As a social scientists trying to help other social scientists move from language like R, Stata, and Matlab into Python, one of the behaviors I've found unnecessarily difficult to explain is the "file.open()/file.close()" idiom (or, alternatively, context managers). In normal operating systems, and many high level languages, saving is a one-step operation. > > I understand there are situations where an open file handle is useful, but it seems a simple `to_file` method on strings (essentially wrapping a context-manager) would be really nice, as it would save users from learning this idiom. Funny, I've seen people looking for a one-step file-load more often than file-save. But really, the two go together; any environment that provided one but not the other would seem strange to me. The question is, if you stick the save as a string method s.to_file(path), where do you stick the load? Is it a string class method str.from_file(path)? That means you have to teach novices about class methods very early on... Alternatively, they could both be builtins, but adding two new builtins is a pretty heavy-duty change. Anything else seems like it would be too non-parallel to be discoverable by novices or remembered by occasional Python users. I'd assume you'd also want this on bytes (and probably bytearray) for dealing with binary files. Anyway, another huge advantage of this proposal is that it can handle atomic writes properly. After all, even novices can quickly learn this: with open(path, 'w') as f: f.write(s) ... but how many of them can even understand this: with tempfile.NamedTemporaryFile('w', dir=os.path.dirname(path), delete=False) as f: f.write(s) f.flush() os.replace(f.path, path) (That's assuming I even got the latter right. I'm on my phone right now, so I can't check, but I wouldn't be surprised if there's a mistake there...) One last thing: I think many of Cocoa, .NET, Qt, Java, etc. have a similar facility. And some of them come from languages that force everything to be a method rather than a function, which means they'd have good answers for the "where does from_file go?" question. Any serious proposal should probably survey those frameworks (along with the scientific systems--and probably how NumPy compared to the latter). Actually, one _more_ last thing. Having this as a Path method makes sense; your problem is that novices don't discover pathlib, and experts don't use pathlib because it's not usable in most non-trivial cases (i.e., you can't use it even with stdlib modules like zipfile, much less third-party libs, without explicitly "casting" Path and str back and forth all over the place). Assuming that the long-term goal is to get everyone using pathlib, rather than just abandoning it as a stdlib fossil of a nice idea that didn't work out, copying its useful features like read_file and write_file out to builtins could be taken as working against that long-term goal. (Also compare Cocoa, which quasi-deprecated hundreds of methods, and then actually deprecated and removed some, to get everyone to switch to using NSURL for all paths.) From guido at python.org Tue Mar 22 23:39:10 2016 From: guido at python.org (Guido van Rossum) Date: Tue, 22 Mar 2016 20:39:10 -0700 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: On Tue, Mar 22, 2016 at 3:18 PM, Andrey Vlasovskikh wrote: > >> 2016-03-19, ? 21:51, Guido van Rossum ???????(?): >> >> I like the way this is going. I think it needs to be a separate PEP; >> PEP 484 is already too long and this topic deserves being written up >> carefully (like you have done here). > > I would like to experiment with various text/binary types for Python 2 and 3 for some time before coming up with a PEP about it. And I would like everybody interested in 2/3 compatible type hints join the discussion. My perspective (mostly PyCharm-specific) might be a bit narrow here. As you wish! >> * Do we really need _AsciiUnicode? I see the point of _AsciiStr, >> because Python 2 accepts 'x' + u'' but fails '\xff' + u'', so 'x' >> needs to be of type _AsciiStr while '\xff' should not (it should be >> just str). However there's no difference in how u'x' is treated from >> how u'\u1234' or u'\xff' are treated -- none of them can be >> concatenated to '\xff' and all of them can be concatenated to _'x'. > > I was concerned with UnicodeEncodeErrors in Python 2 during implicit conversions from unicode to bytes: > > getattr(obj, u'Non-ASCII-name') > > There are several places in the Python 2 API where these ASCII-based unicode->bytes conversions take place, so the _AsciiUnicode type comes to mind. OK, so you want the type of u'hello' to be _AsciiUnicode but the type of u'????????????' to be just unicode, right? And getattr()'s second argument would be typed as... What? >> * It would be helpful to spell out exactly what is and isn't allowed >> when different core types (bytes, str, unicode, Text) meet in Python 2 >> and in Python 3. Something like a table with a row and a column for >> each and the type of x+y (or "error") in each of the cells. > > Agreed. I'll try to come up with specific rules for handling text/binary types (bytes, str, unicode, Text, _Ascii*) in Python 2 and 3. For me the rules for dealing with _Ascii* look the most problematic at the moment as it's unclear how these types should propagate via text-handling functions. You can try that out at runtime though. >> * I propose that Python 2+3 mode is just the intersection of what >> Python 2 and Python 3 mode allow. (In mypy, I don't think we'll >> implement this -- users will just have to run mypy twice with and >> without --py2. But for PyCharm it makes sense to be able to declare >> this. Yet I think it would be good not to have to spell out separately >> which rules it uses, defining it as the intersection of 2 and 3 is all >> we need. > > Yes, there is no need in having a specific 2+3 mode, I was really referring to the intersection of the Python 2 and 3 APIs when the user accesses a text / binary method not available in both. Cool. -- --Guido van Rossum (python.org/~guido) From abarnert at yahoo.com Tue Mar 22 23:51:53 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Tue, 22 Mar 2016 20:51:53 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: Message-ID: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> On Mar 22, 2016, at 20:29, Alexander Belopolsky wrote: > >> On Tue, Mar 22, 2016 at 11:06 PM, Nick Eubank wrote: >> it seems a simple `to_file` method on strings (essentially wrapping a context-manager) would be really nice > > -1 > > It is a rare situation when you would want to write just a single string to a file. I do it all the time in other languages when dealing with smallish files. Python's very nice file-object concept, slant toward iterator-based processing, and amazingly consistent ecosystem means that the same issues don't apply, so I'd rarely do the same thing. But for users migrating to Python from another language, or using Python occasionally while primarily using another language, I can see it being a lot more attractive. Also, you're neglecting the atomic-write issue. Coming up with a good API for an iterative atomic write is hard; for single-string write-all, it's just an extra flag on the function. > In most cases you write several strings and or do other file operations between opening and closing a file stream. The proposed to_file() method may become an attractive nuisance leading to highly inefficient code. Remember: opening or closing a file is still in most setups a mechanical operation that involves moving macroscopic physical objects, not just electrons. All the more reason to assemble the whole thing in memory and write it all at once, rather than streaming it out. Then you don't have to worry about how good the buffering is, what happens if there's a power failure (or just an exception, if you don't use with statements) halfway through, etc. It's definitely going to be as fast and as safe as possible if all of the details are done by the stdlib instead of user code. (I trust Python's buffering in 3.6, but not in 2.x or 3.1--and I've seen people even in modern 3.x try to "optimize" by opening files in raw mode and writing 7 bytes here and 18 bytes there, which is going to be much slower than concatenating onto a buffer and writing blocks at a time...) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nickeubank at gmail.com Tue Mar 22 23:58:28 2016 From: nickeubank at gmail.com (Nick Eubank) Date: Wed, 23 Mar 2016 03:58:28 +0000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: Thanks for the thoughts both! I'm not opposed to `a = str.read_file()` -- it does require knowing classes to grok it, but it's super readable and intuitive to look at (i.e. pythonic?). Regarding bytes, I was thinking `to_file()` would include a handful of arguments to support unusual encodings or bytes, but leaving the default to utf-8 text. On Tue, Mar 22, 2016 at 8:52 PM Andrew Barnert wrote: > On Mar 22, 2016, at 20:29, Alexander Belopolsky < > alexander.belopolsky at gmail.com> wrote: > > > On Tue, Mar 22, 2016 at 11:06 PM, Nick Eubank > wrote: > >> it seems a simple `to_file` method on strings (essentially wrapping a >> context-manager) would be really nice > > > -1 > > It is a rare situation when you would want to write just a single string > to a file. > > > I do it all the time in other languages when dealing with smallish files. > Python's very nice file-object concept, slant toward iterator-based > processing, and amazingly consistent ecosystem means that the same issues > don't apply, so I'd rarely do the same thing. But for users migrating to > Python from another language, or using Python occasionally while primarily > using another language, I can see it being a lot more attractive. > > Also, you're neglecting the atomic-write issue. Coming up with a good API > for an iterative atomic write is hard; for single-string write-all, it's > just an extra flag on the function. > > In most cases you write several strings and or do other file operations > between opening and closing a file stream. The proposed to_file() method > may become an attractive nuisance leading to highly inefficient code. > Remember: opening or closing a file is still in most setups a mechanical > operation that involves moving macroscopic physical objects, not just > electrons. > > > All the more reason to assemble the whole thing in memory and write it all > at once, rather than streaming it out. Then you don't have to worry about > how good the buffering is, what happens if there's a power failure (or just > an exception, if you don't use with statements) halfway through, etc. It's > definitely going to be as fast and as safe as possible if all of the > details are done by the stdlib instead of user code. (I trust Python's > buffering in 3.6, but not in 2.x or 3.1--and I've seen people even in > modern 3.x try to "optimize" by opening files in raw mode and writing 7 > bytes here and 18 bytes there, which is going to be much slower than > concatenating onto a buffer and writing blocks at a time...) > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Wed Mar 23 00:13:07 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 23 Mar 2016 00:13:07 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: Message-ID: On Tue, Mar 22, 2016 at 11:32 PM, Nick Eubank wrote: > I output "single strings" to LaTeX all the time. > If you do this in an interactive session - take a look at ipython. Its %writefile [1] magic command is not quite what you want but close. You may also have a better chance getting a new feature in ipython than here. If you do that from a script - stop doing that - and your script will run faster. [1] https://ipython.org/ipython-doc/3/interactive/magics.html#cellmagic-writefile -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Wed Mar 23 00:22:49 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 23 Mar 2016 00:22:49 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: On Tue, Mar 22, 2016 at 11:33 PM, Andrew Barnert via Python-ideas < python-ideas at python.org> wrote: > Anyway, another huge advantage of this proposal is that it can handle > atomic writes properly. After all, even novices can quickly learn this: > > with open(path, 'w') as f: > f.write(s) > > ... but how many of them can even understand this: > > with tempfile.NamedTemporaryFile('w', dir=os.path.dirname(path), > delete=False) as f: > f.write(s) > f.flush() > os.replace(f.path, path) > This would be a reasonable addition as a pathlib.Path method. > > (That's assuming I even got the latter right. I'm on my phone right now, > so I can't check, but I wouldn't be surprised if there's a mistake there...) > You've got it wrong, but I understand what you tried to achieve. Note that the "write to temp and move" trick may not work if your /tmp and your path are mounted on different filesystems. And with some filesystems it may not work at all, but I agree that it would be nice to have a state of the art atomic write method somewhere in stdlib. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Wed Mar 23 00:32:53 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 23 Mar 2016 00:32:53 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On Tue, Mar 22, 2016 at 11:51 PM, Andrew Barnert wrote: > Then you don't have to worry about how good the buffering is, what happens > if there's a power failure ... You are right: it is not uncommon for data scientists to prefer loosing all the computed data rather than the last few blocks in this case. :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Mar 23 00:33:15 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 23 Mar 2016 15:33:15 +1100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: On Wed, Mar 23, 2016 at 3:22 PM, Alexander Belopolsky wrote: > > On Tue, Mar 22, 2016 at 11:33 PM, Andrew Barnert via Python-ideas > wrote: >> with tempfile.NamedTemporaryFile('w', dir=os.path.dirname(path), >> delete=False) as f: >> f.write(s) >> f.flush() >> os.replace(f.path, path) > You've got it wrong, but I understand what you tried to achieve. Note that > the "write to temp and move" trick may not work if your /tmp and your path > are mounted on different filesystems. And with some filesystems it may not > work at all, but I agree that it would be nice to have a state of the art > atomic write method somewhere in stdlib. It's specifically selecting a directory for the temp file, so it ought to work. However, I'm not certain in my own head of the interaction between NamedTemporaryFile with delete=False and os.replace (nor of exactly how the latter differs from os.rename); what exactly happens when the context manager exits here? And what happens if there's an exception in the middle of this and stuff doesn't complete properly? Are there points at which this could (a) destroy data by deleting without saving, or (b) leave cruft around? This would be very nice to have as either stdlib or a well-documented recipe. ChrisA From tomuxiong at gmail.com Wed Mar 23 00:35:05 2016 From: tomuxiong at gmail.com (Thomas Nyberg) Date: Tue, 22 Mar 2016 21:35:05 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: Message-ID: <56F21CF9.7000005@gmail.com> Not to knock too much on R, but this kind of thing was one of the things I always really hated about that language. Separating the operations of converting a string and then writing/printing/sending/etc. that string is just so much more flexible. It reminds me of the many objects I worked with in R that had natural print methods, but did not have natural conversions to strings (at least nowhere near as easy as it should have been). There were so many times that having the operations as separate steps (e.g. having print just call call __str__() or whatever) is so much better. I understand the argument that it feels more natural to certain people, but it really incentivizes bad design imo. Definitely a big -1 from me. On 03/22/2016 08:06 PM, Nick Eubank wrote: > > As a social scientists trying to help other social scientists move from > language like R, Stata, and Matlab into Python, one of the behaviors > I've found unnecessarily difficult to explain is the > "file.open()/file.close()" idiom (or, alternatively, context managers). > In normal operating systems, and many high level languages, saving is a > one-step operation. > > I understand there are situations where an open file handle is useful, > but it seems a simple `to_file` method on strings (essentially wrapping > a context-manager) would be really nice, as it would save users from > learning this idiom. > > Apparently there's something like this in the Path library > (https://docs.python.org/dev/library/pathlib.html#pathlib.Path.write_text) > , but I suspect most people have no idea about that method (I've been > doing python for years and this has always been a personal frustration, > and I've asked several others for better options and no one had one to > offer), and it seems like it would make much more sense as a string > method. If someone has a string they want to save to disk, I can't > imagine anyone looking in the Path library. > > I respect the desire to avoid bloat -- the context manager or open/close > idiom has just felt unnecessarily complicated (dare I say unpythonic?) > for a common task. > > > Thanks! > > Nick > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From rosuav at gmail.com Wed Mar 23 00:37:55 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 23 Mar 2016 15:37:55 +1100 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: On Wed, Mar 23, 2016 at 2:39 PM, Guido van Rossum wrote: >> I was concerned with UnicodeEncodeErrors in Python 2 during implicit conversions from unicode to bytes: >> >> getattr(obj, u'Non-ASCII-name') >> >> There are several places in the Python 2 API where these ASCII-based unicode->bytes conversions take place, so the _AsciiUnicode type comes to mind. > > OK, so you want the type of u'hello' to be _AsciiUnicode but the type > of u'????????????' to be just unicode, right? And getattr()'s second > argument would be typed as... What? AIUI, getattr's second argument is simply 'str'; but in Python 2, _AsciiUnicode (presumably itself a subclass of unicode) can be implicitly promoted to str. A non-ASCII attribute name works fine, but getattr converts unicode to str using the 'ascii' codec. ChrisA From alexander.belopolsky at gmail.com Wed Mar 23 00:40:56 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 23 Mar 2016 00:40:56 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: On Wed, Mar 23, 2016 at 12:33 AM, Chris Angelico wrote: > >> with tempfile.NamedTemporaryFile('w', dir=os.path.dirname(path), > .. > It's specifically selecting a directory for the temp file, so it ought > to work. I did not notice this. Thanks, Chris. Still not all filesystems do rename atomically. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Mar 23 00:49:26 2016 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 22 Mar 2016 21:49:26 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: On Tue, Mar 22, 2016 at 9:33 PM, Chris Angelico wrote: > On Wed, Mar 23, 2016 at 3:22 PM, Alexander Belopolsky > wrote: >> >> On Tue, Mar 22, 2016 at 11:33 PM, Andrew Barnert via Python-ideas >> wrote: >>> with tempfile.NamedTemporaryFile('w', dir=os.path.dirname(path), >>> delete=False) as f: >>> f.write(s) >>> f.flush() >>> os.replace(f.path, path) >> You've got it wrong, but I understand what you tried to achieve. Note that >> the "write to temp and move" trick may not work if your /tmp and your path >> are mounted on different filesystems. And with some filesystems it may not >> work at all, but I agree that it would be nice to have a state of the art >> atomic write method somewhere in stdlib. > > It's specifically selecting a directory for the temp file, so it ought > to work. However, I'm not certain in my own head of the interaction > between NamedTemporaryFile with delete=False and os.replace (nor of > exactly how the latter differs from os.rename); what exactly happens > when the context manager exits here? And what happens if there's an > exception in the middle of this and stuff doesn't complete properly? > Are there points at which this could (a) destroy data by deleting > without saving, or (b) leave cruft around? > > This would be very nice to have as either stdlib or a well-documented recipe. Also: cross-platform support (Windows Is Different), handling of permissions, do you care about xattrs?, when you say "atomic" then do you mean atomic WRT power loss?, ... it's not actually 100% clear to me that a one-size-fits-all atomic write implementation is possible. -n -- Nathaniel J. Smith -- https://vorpus.org From pavol.lisy at gmail.com Wed Mar 23 01:06:58 2016 From: pavol.lisy at gmail.com (Pavol Lisy) Date: Wed, 23 Mar 2016 06:06:58 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F1A07B.1030000@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> <56F191CB.2090106@stoneleaf.us> <56F19BC5.30407@stoneleaf.us> <56F19C65.4030801@gmail.com> <56F19F6C.2070605@stoneleaf.us> <56F19FFA.3060303@stoneleaf.us> <56F1A07B.1030000@gmail.com> Message-ID: 2016-03-22 20:43 GMT+01:00, Michel Desmoulin : > No you didn't store the lines, the whole points is that they are > returning generators. > > In that context, f[begin, end] does itertools.takewhile and dropwhile > and [:10000] is doing islice(0, 10000). None of those functions store > anything. > > Please read the begining of thread. It would be true if f is generator. If f is list then we could probably want to have another operator to avoid memory consumption return list(f{begin,end}{:10000}) #return list(f{begin,end}[:10000]) # this could be same From ncoghlan at gmail.com Wed Mar 23 01:13:14 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 23 Mar 2016 15:13:14 +1000 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F07E78.9020309@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: On 22 March 2016 at 09:06, Michel Desmoulin wrote: > So my first proposal is to be able to do: > > def stop(element): > return element > 4 > print(numbers[:stop]) > > It's quite pythonic, easy to understand : the end of the slice is when > this condition is met. Any not the strange way takewhile work, which is > "carry on as long as this condition is met". > > We could also extend itertools.islice to accept such parameter. iter() already has a two-argument form to accept a callable+sentinel value, so it may not be unreasonable to offer an "until" callback to say when to stop iterating. That is: def stop(element): return element >4 print(list(iter(numbers, until=stop))) Slicing arbitrary iterables may be amenable to a str.join style solution, by putting the functionality on slice objects, rather than on the iterables: slice(3, 7).iter(iterable) (Whether or not to make slice notation usable outside subscript operations could then be tackled as an independent question) For itertools.chain, it may make sense to simply promote it to the builtins. I'm not the least bit sure about the wisdom of the first two ideas, but the last one seems straightforward enough, and I'd be comfortable with us putting "chain" on the same tier as existing builtin iteration related tools like enumerate and zip. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Mar 23 01:43:09 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 23 Mar 2016 15:43:09 +1000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: On 23 March 2016 at 13:33, Andrew Barnert via Python-ideas wrote: > On Mar 22, 2016, at 20:06, Nick Eubank wrote: >> >> As a social scientists trying to help other social scientists move from language like R, Stata, and Matlab into Python, one of the behaviors I've found unnecessarily difficult to explain is the "file.open()/file.close()" idiom (or, alternatively, context managers). In normal operating systems, and many high level languages, saving is a one-step operation. >> >> I understand there are situations where an open file handle is useful, but it seems a simple `to_file` method on strings (essentially wrapping a context-manager) would be really nice, as it would save users from learning this idiom. > > Funny, I've seen people looking for a one-step file-load more often than file-save. But really, the two go together; any environment that provided one but not the other would seem strange to me. > > The question is, if you stick the save as a string method s.to_file(path), where do you stick the load? Is it a string class method str.from_file(path)? That means you have to teach novices about class methods very early on... Alternatively, they could both be builtins, but adding two new builtins is a pretty heavy-duty change. Anything else seems like it would be too non-parallel to be discoverable by novices or remembered by occasional Python users. > > I'd assume you'd also want this on bytes (and probably bytearray) for dealing with binary files. A key part of the problem here is that from a structured design perspective, in-memory representation, serialisation and persistence are all separate concerns. If you're creating a domain specific language (and yes, I count R, Stata and MATLAB as domain specific), then you can make some reasonable assumptions for all of those, and hide more of the complexities from your users. In Python, for example, the Pandas IO methods are closer to what other data analysis and modelling environments are able to offer: http://pandas.pydata.org/pandas-docs/stable/io.html NumPy similarly has some dedicated IO routines: http://docs.scipy.org/doc/numpy/reference/routines.io.html However, for a general purpose language, providing convenient "I don't care about the details, just do something sensible" defaults gets trickier, as not only are suitable defaults often domain specific, you have a greater responsibility to help folks figure out when they have ventured into territory where those defaults are no longer appropriate. If you can't figure out a reasonable set of default behaviours, or can't figure out how to nudge people towards alternatives when they start hitting the limits of the default behaviour, then you're often better off ducking the question entirely and getting people to figure it out for themselves. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wes.turner at gmail.com Wed Mar 23 02:12:35 2016 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 23 Mar 2016 01:12:35 -0500 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> <9BE878B7-B2A5-4E16-B798-5381FC6DBF44@gmail.com> Message-ID: * Text.encoding * Text.lang (urn:ietf:rfc:3066) * ... IRIs and RDF literals may be useful test cases here: * https://en.m.wikipedia.org/wiki/Comparison_of_Unicode_encodings * https://en.m.wikipedia.org/wiki/Control_character * https://en.wikipedia.org/wiki/Internationalized_resource_identifier * is this already punycoded? * http://rdflib.readthedocs.org/en/stable/rdf_terms.html * http://rdflib.readthedocs.org/en/stable/apidocs/rdflib.html#rdflib.term.Literal (value, datatype, lang (RFC 3066)) On Mar 22, 2016 4:40 PM, "Wes Turner" wrote: > > On Mar 22, 2016 4:36 PM, "Andrey Vlasovskikh" < > andrey.vlasovskikh at gmail.com> wrote: > > > > > > > 2016-03-19, ? 5:12, Wes Turner ???????(?): > > > > > > This sounds like a more correct approach, thanks. > > > > > > Looking at MarkupSafe (and, now, f-strings), would/will it be possible > to use Typing.Text as a base class for even-more abstract string types > ("strypes") e.g. XML, XHTML, HTML4, HTML5, HTML5.1, SQL? There are implicit > casts and contextual adaptations/transformations (which MarkupSafe specs a > bit). (I've no real code here, just a general idea that we're not tracking > enough string metadata to be safe here) > > > > I believe having separate string types for XML or SQL content is out of > the scope of this proposal. > > > > In PyCharm we already treat the contents of string literals with SQL as > a separate SQL syntax tree and we understand basic string operations like > concatenation or formatting. Going beyond that with the help of > XML/SQL/etc. string types is possible, but I doubt we need a standard for > that. > > At the least, it would be helpful to either have: > > a) a slot / attribute for additional string type metadata (is this an > object subclass that I can just add attrs to) > b) a minimal Text base class > > SQL is harder because dialects. > > > > > -- > > Andrey Vlasovskikh > > > > Web: http://pirx.ru/ > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 23 02:45:52 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 23 Mar 2016 16:45:52 +1000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: On 23 March 2016 at 15:44, Nick Eubank wrote: > True -- my feeling is that if we're ok with defaults in a pathlib method, > why not put them in a string method? pathlib is explicitly about filesytem management and manipulation, while strings don't inherently have anything to do with filesystems, and only a little bit to do with serialisation (via str.encode). Having some lower level convenience functions in io may make sense (specifically io.write_bytes(path, data) and io.write_text(path, data, encoding=None, errors=None), but the str builtin definitely isn't the right place for the capability. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From andrey.vlasovskikh at gmail.com Wed Mar 23 03:37:57 2016 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Wed, 23 Mar 2016 10:37:57 +0300 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: > 2016-03-23, ? 6:39, Guido van Rossum ???????(?): > >>> * Do we really need _AsciiUnicode? I see the point of _AsciiStr, >>> because Python 2 accepts 'x' + u'' but fails '\xff' + u'', so 'x' >>> needs to be of type _AsciiStr while '\xff' should not (it should be >>> just str). However there's no difference in how u'x' is treated from >>> how u'\u1234' or u'\xff' are treated -- none of them can be >>> concatenated to '\xff' and all of them can be concatenated to _'x'. >> >> I was concerned with UnicodeEncodeErrors in Python 2 during implicit conversions from unicode to bytes: >> >> getattr(obj, u'Non-ASCII-name') >> >> There are several places in the Python 2 API where these ASCII-based unicode->bytes conversions take place, so the _AsciiUnicode type comes to mind. > > OK, so you want the type of u'hello' to be _AsciiUnicode but the type > of u'????????????' to be just unicode, right? And getattr()'s second > argument would be typed as... What? The type of the second argument would be str, the "native string" type. If people use from __future__ import unicode_literals then there are many places in Python 2 where str is expected but an ASCII-unicode literal is given. Having the internal _AsciiUnicode type that inherits from unicode while being compatible with str (and bytes) would solve this issue. -- Andrey Vlasovskikh Web: http://pirx.ru/ From andrey.vlasovskikh at gmail.com Wed Mar 23 03:45:51 2016 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Wed, 23 Mar 2016 10:45:51 +0300 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: <31A90ED1-5115-4AF4-935A-89DE3922BD1D@gmail.com> > 2016-03-23, ? 7:37, Chris Angelico ???????(?): > > On Wed, Mar 23, 2016 at 2:39 PM, Guido van Rossum wrote: >>> I was concerned with UnicodeEncodeErrors in Python 2 during implicit conversions from unicode to bytes: >>> >>> getattr(obj, u'Non-ASCII-name') >>> >>> There are several places in the Python 2 API where these ASCII-based unicode->bytes conversions take place, so the _AsciiUnicode type comes to mind. >> >> OK, so you want the type of u'hello' to be _AsciiUnicode but the type >> of u'????????????' to be just unicode, right? And getattr()'s second >> argument would be typed as... What? > > AIUI, getattr's second argument is simply 'str'; but in Python 2, > _AsciiUnicode (presumably itself a subclass of unicode) can be > implicitly promoted to str. A non-ASCII attribute name works fine, but > getattr converts unicode to str using the 'ascii' codec. Right. I'm not sure that a non-ASCII attribute name is fine in Python 2 though. -- Andrey Vlasovskikh Web: http://pirx.ru/ From rosuav at gmail.com Wed Mar 23 03:48:29 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 23 Mar 2016 18:48:29 +1100 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: <31A90ED1-5115-4AF4-935A-89DE3922BD1D@gmail.com> References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> <31A90ED1-5115-4AF4-935A-89DE3922BD1D@gmail.com> Message-ID: On Wed, Mar 23, 2016 at 6:45 PM, Andrey Vlasovskikh wrote: > >> 2016-03-23, ? 7:37, Chris Angelico ???????(?): >> >> On Wed, Mar 23, 2016 at 2:39 PM, Guido van Rossum wrote: >>>> I was concerned with UnicodeEncodeErrors in Python 2 during implicit conversions from unicode to bytes: >>>> >>>> getattr(obj, u'Non-ASCII-name') >>>> >>>> There are several places in the Python 2 API where these ASCII-based unicode->bytes conversions take place, so the _AsciiUnicode type comes to mind. >>> >>> OK, so you want the type of u'hello' to be _AsciiUnicode but the type >>> of u'????????????' to be just unicode, right? And getattr()'s second >>> argument would be typed as... What? >> >> AIUI, getattr's second argument is simply 'str'; but in Python 2, >> _AsciiUnicode (presumably itself a subclass of unicode) can be >> implicitly promoted to str. A non-ASCII attribute name works fine, but >> getattr converts unicode to str using the 'ascii' codec. > > Right. I'm not sure that a non-ASCII attribute name is fine in Python 2 though. It's legal. I don't know that it's a good idea, but it is legal. rosuav at sikorsky:~$ python Python 2.7.11+ (default, Feb 22 2016, 16:38:42) [GCC 5.3.1 20160220] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class NS(object): pass ... >>> ns = NS() >>> setattr(ns, "\x00", "null") >>> getattr(ns, "\x00") 'null' >>> setattr(ns, "\xA9", "copyright") >>> getattr(ns, "\xA9") 'copyright' >>> dir(ns) ['\x00', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '\xa9'] ChrisA From abarnert at yahoo.com Wed Mar 23 03:51:56 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 00:51:56 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: On Mar 22, 2016, at 21:40, Alexander Belopolsky wrote: > > >> On Wed, Mar 23, 2016 at 12:33 AM, Chris Angelico wrote: >> >> with tempfile.NamedTemporaryFile('w', dir=os.path.dirname(path), >> .. >> It's specifically selecting a directory for the temp file, so it ought >> to work. > > > I did not notice this. Thanks, Chris. Still not all filesystems do rename atomically. That's why I used os.replace, not os.rename. Which _is_ guaranteed to either replace atomically or do nothing and fail (and works with the most important filesystems on Windows 2003+ and every major POSIX OS). So that's not the problem either. The "delete=False" is also not a problem. Doing the replace inside the NamedTemporaryFile may be a problem on Windows. Or maybe it was doing it right after a flush without something else in between? I forget. Whatever it is, I have a solution in the library that I use (which also emulates os.replace on Python 2.6-7 and 3.1-2), so I don't have to remember how to solve it every time. One argument against adding it to write_file (besides the argument that novices don't discover pathlib and experts don't use it for other reasons) is that I think the one obvious way to do it should be atomic by default for novices, but that would be a breaking API change to the pathlib function. But if that were the way to get atomic writes in the stdlib, I'd start using pathlib for them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 23 03:56:04 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 00:56:04 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On Mar 22, 2016, at 21:32, Alexander Belopolsky wrote: > >> On Tue, Mar 22, 2016 at 11:51 PM, Andrew Barnert wrote: >> Then you don't have to worry about how good the buffering is, what happens if there's a power failure ... > > You are right: it is not uncommon for data scientists to prefer loosing all the computed data rather than the last few blocks in this case. :-) The question is whether it's better to have the complete previous version of the data, or 21% of the new version of the data.[1] And the answer is different in different cases, which is why atomic pretty much has to be an option (or a separate function), not always-on. But I think on by default makes sense. --- [1]: In fact, with delete=False and a sensible pattern, atomic actually gives you the best of both worlds: the previous version of the data is still there, and 21% of the new version is in a tempfile that you can recover if you know what you're doing... -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 23 04:13:16 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 01:13:16 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: <3F5C43C1-83BB-42B9-AF6F-4E52BFA79B04@yahoo.com> On Mar 22, 2016, at 21:49, Nathaniel Smith wrote: > >> On Tue, Mar 22, 2016 at 9:33 PM, Chris Angelico wrote: >> On Wed, Mar 23, 2016 at 3:22 PM, Alexander Belopolsky >> wrote: >>> >>> On Tue, Mar 22, 2016 at 11:33 PM, Andrew Barnert via Python-ideas >>> wrote: >>>> with tempfile.NamedTemporaryFile('w', dir=os.path.dirname(path), >>>> delete=False) as f: >>>> f.write(s) >>>> f.flush() >>>> os.replace(f.path, path) >>> You've got it wrong, but I understand what you tried to achieve. Note that >>> the "write to temp and move" trick may not work if your /tmp and your path >>> are mounted on different filesystems. And with some filesystems it may not >>> work at all, but I agree that it would be nice to have a state of the art >>> atomic write method somewhere in stdlib. >> >> It's specifically selecting a directory for the temp file, so it ought >> to work. However, I'm not certain in my own head of the interaction >> between NamedTemporaryFile with delete=False and os.replace (nor of >> exactly how the latter differs from os.rename); what exactly happens >> when the context manager exits here? And what happens if there's an >> exception in the middle of this and stuff doesn't complete properly? >> Are there points at which this could (a) destroy data by deleting >> without saving, or (b) leave cruft around? >> >> This would be very nice to have as either stdlib or a well-documented recipe. > > Also: cross-platform support (Windows Is Different), I know a lot of people who never touch Windows with a 10-foot pole think this problem is still unsolved, but that's not true. Microsoft added sufficient atomic-replace APIs in 2003 (in the Win32 API only, not in crt/libc), and as of 3.3, Python's os.replace really is guaranteed to either atomically replace the file or leave it untouched and raise an exception on all platforms (as long as the files are on the same filesystem, and as long as there's not an EIO or equivalent because of an already-corrupted filesystem or a physical error on the media). (For platforms besides Windows and POSIX, it does this just by not existing and raising a NameError...) Likewise for safe temporary files--as of 3.3, tempfile.NamedTemporaryFile is safe on every platform where it exists, and that includes Windows. > handling of > permissions, do you care about xattrs? That can be handled effectively the same way as copy vs. copy2 if desired. I don't know if it's important enough, but if it is, it's easy. (My library actually does have options for preserving different levels of stuff, but I never use them.) > when you say "atomic" then do > you mean atomic WRT power loss? Write-and-replace is atomic WRT both exceptions and power loss. Until the replace succeeds, the old version of the file is still there. This is guaranteed by POSIX and by Windows. If the OS can't offer that on some filesystem, it won't let you call os.replace. > , ... it's not actually 100% clear to > me that a one-size-fits-all atomic write implementation is possible. At least a one-size-fits-all-POSIX-and-post-XP-Windows solution is possible, and that's good enough to many things in the stdlib today. If atomic writes had "availability: Unix, Windows", like most of the os module, I think that would be fine. From desmoulinmichel at gmail.com Wed Mar 23 06:04:03 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 23 Mar 2016 11:04:03 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: <56F26A13.2030006@gmail.com> Le 23/03/2016 06:13, Nick Coghlan a ?crit : > On 22 March 2016 at 09:06, Michel Desmoulin wrote: >> So my first proposal is to be able to do: >> >> def stop(element): >> return element > 4 >> print(numbers[:stop]) >> >> It's quite pythonic, easy to understand : the end of the slice is when >> this condition is met. Any not the strange way takewhile work, which is >> "carry on as long as this condition is met". >> >> We could also extend itertools.islice to accept such parameter. > > iter() already has a two-argument form to accept a callable+sentinel > value, so it may not be unreasonable to offer an "until" callback to > say when to stop iterating. > > That is: > > def stop(element): > return element >4 > print(list(iter(numbers, until=stop))) Do you propose to make a "since" param too ? > > Slicing arbitrary iterables may be amenable to a str.join style > solution, by putting the functionality on slice objects, rather than > on the iterables: > > slice(3, 7).iter(iterable) Those solutions are already much more convenient than using itertools, but it's going to look very strange: def foo(p): with open(p) as f: def begin(): return x.startswith('#') def end(): return x != "STOP" yield from slice(0, 10).iter(iter(f, since=begin, until=en)) The last line is really not Pythonic. Compared to: def foo(p): with open(p) as f: def begin(): return x.startswith('#') def end(): return x != "STOP" yield from f[begin:end][:10] But I beleive there is an idea here. Attaching things to slice is cleaner than to iter(): def foo(p): with open(p) as f: def begin(): return x.startswith('#') def end(): return x != "STOP" yield from slice.wraps(f)[begin:end][:10] > > (Whether or not to make slice notation usable outside subscript > operations could then be tackled as an independent question) > > For itertools.chain, it may make sense to simply promote it to the builtins. Same problem as with new keywords : it can be a problem with people using chain as a var name. > > I'm not the least bit sure about the wisdom of the first two ideas, > but the last one seems straightforward enough, and I'd be comfortable > with us putting "chain" on the same tier as existing builtin iteration > related tools like enumerate and zip. To be fair, I don't nearly use chain as much as zip or enumerate, so I'm not going to push for something as drastic. > > Regards, > Nick. > From desmoulinmichel at gmail.com Wed Mar 23 06:08:08 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 23 Mar 2016 11:08:08 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> Message-ID: <56F26B08.4060802@gmail.com> Le 23/03/2016 00:56, Andrew Barnert via Python-ideas a ?crit : > On Mar 22, 2016, at 16:21, Steven D'Aprano wrote: >> >>> On Tue, Mar 22, 2016 at 06:21:24PM +0100, Sven R. Kunze wrote: >>>> On 22.03.2016 16:09, Stephen J. Turnbull wrote: >>>> Chris Barker writes: >>>>> All that being said: >>>>> >>>>> how about "elempty"?, to go with elif? >>>> >>>> -1 Not to my taste, to say the least. >>> >>> Hmm, it seems there is no easy solution for this. >> >> Possibly with the exception of the three or four previously existing >> easy solutions :-) > > The only question is whether any of them are obvious enough (even for novices). If not, we could argue whether any proposed change would be significantly _more_ obvious. And, if so, then the question is whether the change is worth the cost. But I think what we have is already obvious enough. (Especially since 90% of the time, when you need to do something special on empty, you explicitly have a sequence, not any iterable, so it's just "if not seq:".) So really, this proposal is really just asking for syntactic sugar that complicates the language in exchange for making some already-understandable code a little more concise, which doesn't seem worth it. > >>> What do you think >>> about an alternative that can handle more than empty and else? >>> >>> for item in collection: >>> # do for item >>> except NeverExecuted: >>> # do if collection is empty >>> >>> It basically merges "try" and "for" and make "for" emit EmptyCollection. >> >> Does this mean that every single for-loop that doesn't catch >> NeverExecuted (or EmptyCollection) will raise an exception? > > Elsewhere he mentioned that EmptyCollection would be a subclass of StopIteration. > > Presumably, every iterator type (or just the builtin ones, and hopefully "many" others?) would have special code to raise EmptyCollection if empty. Like this pseudocode for list_iterator: > > def __next__(self): > if not self.lst: > raise EmptyCollection > elif self.i >= len(self.lst): > raise StopIteration > else: > i = self.i > self.i += 1 > return self.lst[self.i] > > Or, alternatively, for itself would do this. The for loop bytecode would have to change to stash an "any values seen" flag somewhere such that if it sees StopIteration and hasn't seen any values, it converts that to an EmptyCollection. Or any of the other equivalents (e.g., the compiler could unroll the first PyIter_Next from loop from the rest of them to handle it specially). But this seems like it would add a lot of overhead and complexity to every loop whether desired or not. > >> If not, then how will this work? Is this a special kind of >> exception-like process that *only* operates inside for loops? >> >> What will an explicit "raise NeverExecuted" do? > > Presumably that's the same question as what an explicit raise StopIteration does. Just as there's nothing stopping you from writing a __next__ method that raises StopIteration but then yields more values of called again, there's nothing stopping you from raising NeverExecuted pathologically, but you shouldn't do so. M > >>> So, independent of the initial "never executed loop body" use-case, one >>> could also emulate the "else" clause by: >>> >>> for item in collection: >>> # do for item >>> except StopIteration: >>> # do after the loop >> >> That doesn't work, for two reasons: >> >> (1) Not all for-loops use iterators. The venerable old "sequence >> protocol" is still supported for sequences that don't support __iter__. >> So there may not be any StopIteration raised at all. > > I think there always is. > > IIRC, PyObject_Iter (the C API function used by iter and by for loops) actually constructs a sequence iterator object if the object doesn't have tp_iter (__iter__ for Python types) but does have tp_sequence (__getitem__ for Python types, but, e.g., dict has __getitem__ without having tp_sequence). And the for loop doesn't treat that sequence iterator any different from "real" iterators returned by __iter__; it just calls tp_next (__next__) until StopIteration. (And the "other half" of the old-style sequence protocol, that lets old-style sequences be reversed if they have a length, is similarly implemented by the C API underneath the reversed function.) > > I'm on my phone right now, so I can't double-check any of the details, but I'm 80% sure they're all at least pretty close... > >> (2) Even if StopIteration is raised, the for-loop catches it (in a >> manner of speaking) and consumes it. >> >> So to have this work, we would need to have the for-loop re-raise >> StopIteration... but what happens if you don't include an except >> StopIteration clause? Does every bare for-loop with no "except" now >> print a traceback and halt processing? If not, why not? > > I think this could be made to work: a for loop without an except clause handles StopIteration the same as today (by jumping to the else clause), but one that does have one or more except clauses just treats it like a normal exception. > > Of course this would mean for/except/else is now legal but useless, which could be confusing ("why does my else clause no longer run when I add an 'except ValueError' clause?"). > > More generally, I think the fact that for/except StopIteration is almost but not quite identical to plain for would be confusing more often than helpful. > > But I think it is a coherent proposal, even if it's not one I like. :) And what do you think about adding except clauses to if ? with ? while ? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From victor.stinner at gmail.com Wed Mar 23 06:28:54 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 23 Mar 2016 11:28:54 +0100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <3F5C43C1-83BB-42B9-AF6F-4E52BFA79B04@yahoo.com> References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <3F5C43C1-83BB-42B9-AF6F-4E52BFA79B04@yahoo.com> Message-ID: If you are interested by "atomic write into a file", see the practical issues to implement a portable function: http://bugs.python.org/issue8604 Victor From rosuav at gmail.com Wed Mar 23 06:37:57 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 23 Mar 2016 21:37:57 +1100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F26A13.2030006@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> Message-ID: On Wed, Mar 23, 2016 at 9:04 PM, Michel Desmoulin wrote: >> (Whether or not to make slice notation usable outside subscript >> operations could then be tackled as an independent question) >> >> For itertools.chain, it may make sense to simply promote it to the builtins. > > Same problem as with new keywords : it can be a problem with people > using chain as a var name. > Less of a problem though - it'd only be an issue for people who (a) use chain as a var name, and (b) want to use the new shorthand. Their code will continue to work identically with itertools.chain (or not using it at all). With a new keyword, their code would instantly fail. ChrisA From steve at pearwood.info Wed Mar 23 07:48:40 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 23 Mar 2016 22:48:40 +1100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: <20160323114840.GK12526@ando.pearwood.info> On Wed, Mar 23, 2016 at 12:40:56AM -0400, Alexander Belopolsky wrote: > On Wed, Mar 23, 2016 at 12:33 AM, Chris Angelico wrote: > > > >> with tempfile.NamedTemporaryFile('w', dir=os.path.dirname(path), > > .. > > It's specifically selecting a directory for the temp file, so it ought > > to work. > > > I did not notice this. Thanks, Chris. Still not all filesystems do rename > atomically. I don't think that Python can guarantee to do more than the file system. The best Python can do is promise to do an atomic write if the OS and file system support it, otherwise to be no worse than what the OS and file system support. -- Steve From leewangzhong+python at gmail.com Wed Mar 23 09:01:08 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Wed, 23 Mar 2016 09:01:08 -0400 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: The following syntax is not a real proposal, because there are way too many places it intrudes on, and you wouldn't be able to define a generator which is also a callable. That said... Slicing a list is fundamentally different from slicing an iterable. One changes state, necessarily. Every time you say "foo[:2]", it will give a different result, until you get no result at all. Instead, I'll write it "foo(:2)", and ignore the fact that that's what functions look like, and whether we CAN extend function syntax to make this legal. Using parens reflects the syntax difference between list comprehensions and generator expressions. On Mar 21, 2016 7:36 PM, "Chris Angelico" wrote: > > On Tue, Mar 22, 2016 at 10:06 AM, Michel Desmoulin > wrote: > > > > Make slicing accept callables > > ============================= > > > > So my first proposal is to be able to do: > > > > def stop(element): > > return element > 4 > > print(numbers[:stop]) > > > > It's quite pythonic, easy to understand : the end of the slice is when > > this condition is met. Any not the strange way takewhile work, which is > > "carry on as long as this condition is met". > > > > We could also extend itertools.islice to accept such parameter. I like this. I don't think it should happen, due to existing parts of the language it might step on, but I like it. foo(f:g:h) will mean filter(h, takewhile(g, dropwhile(negate(f), foo))) (possibly clearer if we imagine being able to write: Itertooler(foo).dropwhile(negate(f)).takewhile(g).filter(h) ) where def negate(f): def negate_f(*args, **kwargs): return not f(*args, **kwargs) return negate_f > > Slicing any iterable > > ====================== > > > > So the second proposal is to allow: > > > > def func_accepting_any_iterable(foo): > > return bar(foo[3:7]) > > > > The slicing would then return a list if it's a list, a typle if it's a > > tuple, and a islice(generator) if it's a generator. If somebody uses a > > negative index, it would then raises a ValueError like islice. > > > > This would make duck typing and iteration even easier in Python. > > Again, while I am sympathetic to the problem, it's actually very hard; > islice always returns the same kind of thing, but slicing syntax can > return all manner of different things, because it's up to the object > on the left: What if `foo(3:5)` always returned a lazy iterator? (Then `iter(o)` == `o(:)`?) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Mar 23 09:43:11 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 23 Mar 2016 13:43:11 +0000 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: On 23 March 2016 at 13:01, Franklin? Lee wrote: > I like this. I don't think it should happen, due to existing parts of the > language it might step on, but I like it. > > foo(f:g:h) will mean > > filter(h, takewhile(g, dropwhile(negate(f), foo))) > > (possibly clearer if we imagine being able to write: > Itertooler(foo).dropwhile(negate(f)).takewhile(g).filter(h) > ) > > where > > def negate(f): > def negate_f(*args, **kwargs): > return not f(*args, **kwargs) > return negate_f As an alternative, you could use a "pipeline" style of approach, as shown in Steven D'Aprano's recipe at http://code.activestate.com/recipes/580625-collection-pipeline-in-python/ foo | SkipUntil(f) | While(g) | Filter(h) (the recipe doesn't define SkipUntil or While, but it's not hard to see how you would write them). As a side note, your code does takewhile(g), where I think the original proposal would have done takeuntil(g) (takewhile(negate(g)) - which is another argument for having explicit named filtering operations rather than implicit behaviour based on syntax. Honestly, something like this looks much more readable to me than playing games with the slice syntax. Paul From xksteven at uchicago.edu Wed Mar 23 10:51:03 2016 From: xksteven at uchicago.edu (Steven Basart) Date: Wed, 23 Mar 2016 09:51:03 -0500 Subject: [Python-ideas] [Python Ideas] Random weighted choice idea to add to random lib Message-ID: Hello, I had an idea that has in some form been proposed before. I think a random weighted choice function would be a very good addition to the random module in the python standard library. Here's a basic implementation of what's required albeit slightly inefficient as the search is linear as opposed to creating a binary tree and taking log n time to find the correct choice. pmf is short for probability mass function. It doesn't handle all error cases but it is more of an example to show the idea. def randweighted(self,pmf=list): """Return an int in the range [0,len(list)] weighted by list of values. Raises error if list is empty. """ if(pmf == []): raise ValueError("pmf is equal to empty list") cummassf = [x[:i] for i in pmf] randVal = self._randbelow(len(pmf)) for x in cummassf: if randVal < x: return x There was a ticket that got started before. It has since stalled because I believe the original author intended the library to get called over an over so they came up with generators which would be more efficient if passing in the same list. Otherwise there isn't really a point to using generators in my opinion. http://bugs.python.org/issue18844 ~Steven -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 23 11:53:57 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 08:53:57 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F26B08.4060802@gmail.com> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F26B08.4060802@gmail.com> Message-ID: <2336A5F4-7A2E-4A1A-A16A-736F513BCE32@yahoo.com> On Mar 23, 2016, at 03:08, Michel Desmoulin wrote: >> But I think it is a coherent proposal, even if it's not one I like. :) > > And what do you think about adding except clauses to if ? with ? while ? Well, they don't have the problem that for has (for is already implicitly handling a specific exception in a specific way; none of them are), so they're coherent even without a solution to that problem. (With has the additional problem that it may not be immediately obvious on first glance whether the exit gets calls before or after the except clause, but if so, people can read the docs the first time they come across it and remember it after that.) But I also think they're even less necessary. They'd all be pure syntactic sugar for nesting the statement and a try/except, so we'd be making the language more complicated to learn and remember, and encouraging more code that isn't backward compatible. That's not a huge cost, but the benefit isn't very big either. For _really_ short cases, I think we want except expressions (PEP 463); for longish code, there's nothing wrong with an explicit try; the range for which hiding an implicit try inside if and while would really improve things is small enough that I don't think the benefit outweighs the cost. But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind. From abarnert at yahoo.com Wed Mar 23 12:13:17 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 09:13:17 -0700 Subject: [Python-ideas] [Python Ideas] Random weighted choice idea to add to random lib In-Reply-To: References: Message-ID: <831C9C40-4CF8-427E-9AED-877BE8C64446@yahoo.com> On Mar 23, 2016, at 07:51, Steven Basart wrote: > > Hello, > > I had an idea that has in some form been proposed before. I think a random weighted choice function would be a very good addition to the random module in the python standard library. > > Here's a basic implementation of what's required albeit slightly inefficient as the search is linear as opposed to creating a binary tree and taking log n time to find the correct choice. pmf is short for probability mass function. It doesn't handle all error cases but it is more of an example to show the idea. > > > def randweighted(self,pmf=list): > """Return an int in the range [0,len(list)] weighted by list of values. > Raises error if list is empty. > """ > if(pmf == []): > raise ValueError("pmf is equal to empty list") > cummassf = [x[:i] for i in pmf] > randVal = self._randbelow(len(pmf)) > for x in cummassf: > if randVal < x: > return x Why a default value of list, especially since that's just going to raise a TypeError later on? Why the (unpythonic) test for "pmf == []" instead of just testing "if not pmf:"? Then it'll work on, say, tuples or third-party sorted lists or any other sequence. (With a tiny change, you can make it work on any iterable, but I don't know whether that's desirable.) The x[:i] is accessing an unbound local named x. What's it supposed to be? Whatever x is, each element of cummassf is going to be some kind of sequence, so randVal < x on each element is going to raise a TypeError. Maybe you wanted a sum somewhere? If you fix that, doesn't this accumulate floating point errors, so there's a significant chance the result won't add up to 1.0 even if all of the weights are correct to 1 ULP? If the weights _aren't_ correct, surely you'd want a ValueError or something. This code will just return None if the sum is too low, and silently choose from the first 1.0 if it's too high. (I think np.random.choice raises ValueError if not isclose(sum(p), 1.0), but I'd have to check.) Finally, this is clearly a weighted randrange (with a fixed min of 0), but it claims to be a weighted choice. Obviously you can build the latter out of the former (e.g., take a val:weight mapping, call randweighted on the values, then do next(islice(keys, result, None)), but you do need to actually build it (and show what the API for it will be) if that's the motivating reason. (Also, I think an _unnormalized_ val:weight mapping might be a better API--after all, that means you can immediately call it on a Counter--but you probably want to look at what numpy, R, Mathematica, etc. do...) > There was a ticket that got started before. It has since stalled because I believe the original author intended the library to get called over an over so they came up with generators which would be more efficient if passing in the same list. Otherwise there isn't really a point to using generators in my opinion. > > http://bugs.python.org/issue18844 Why not just pick up from their original implementation, fix any bugs, and propose that, instead of starting over from scratch? -------------- next part -------------- An HTML attachment was scrubbed... URL: From julien at palard.fr Wed Mar 23 12:19:19 2016 From: julien at palard.fr (Julien Palard) Date: Wed, 23 Mar 2016 17:19:19 +0100 Subject: [Python-ideas] https://docs.python.org/fr/ ? In-Reply-To: References: <56ED7EBD.5050804@palard.fr> Message-ID: <56F2C207.8030902@palard.fr> Hi, On 03/21/2016 11:38 AM, Victor Stinner wrote: > Hi, > > If we choose to officially have translated documentation, I suggest to > ensure that it has the same content thant the english. I don't want to > read an outdated translated doc. I mean that all doc (en+fr) must be > recompiled when a patch is merged. I like the idea of running "msgmerge" at each patch so outdated translations are rapidly removed (marked as "fuzzy"), it's probably possible and easy. Our makefile provide a "msgmerge_all" to do that on all versions. I'll keep it in mind. >> - ... >> - untranslated strings may be visually marked as so > > I don't think that it's needed. It's quite easy to notice untranslated parts... It's strongly linked to your "Do you plan to use a web UI to allow anyone to translate easily": I think we should provide a simple way to propose little corrections or even whole paragraph translations, but: - Not soon, the project does not depend on it and it's more a Sphinx-doc feature than a Python documentations feature, it should be treated separately. - From my point of view, proposed translations goes to a moderation queue, they can't be accepted directly. - Some services like crowdin (on which we have an unlimited free plan, thanks Crowdin !) provide those kind of features (and crowdin is already interfaced with poedit). However I have not used crowdin that much and still prefer poedit+git for that. We may also write a git bot auto-generating pull requests... Many ideas here, that we should probably discuss in a sphinx-related issue or mailing list. >> ### Dropping the default locale of a language > > IMHO we should see what PHP did, since PHP has a long history with > documentation translation. > > So /fr/ for french, but /pt_BR/ for Brazilian Portuguese. We can use > something similar for Python. Dropping a redundant local (fr_FR -> fr) seems logic, but I still prefer "pt-br" over "pt_BR", counterbalancing the PHP doc using "pt_BR" we have some big player using pt-br like: - wordpress: https://codex.wordpress.org/pt-br:Codex_Multilingua - facebook: https://pt-br.facebook.com/login/ - msn: www.msn.com/pt-br/ - skype: www.skype.com/pt-br > I would really prefer to not move on more time the Python doc (add /en/ > to the URL), I agree, we don't have to move the actual doc under `/en/`. If setting redirections we can set them from `/en/` to `/` just in case someone modify /fr/ to /en/ manually. > I'm lazy and I really like writing http://php.net/ to > get the documentation of a function. > > It's much more difficult to get the doc of len in the Python doc... That's a whole other subject, but as we're on it, instead of giving a 404 for https://docs.python.org/3.5/len or even https://docs.python.org/len why not displaying a search result page with hopefully len as a first result ? I was sur I read some RFC telling that a 404 should return "relevant links" but can't find it in the 2616 :( >> ### gettext VS IETF language tag format >> >> gettext goes by using an underscore between language and locale [3] and IETF >> goes by using a dash [4][5]. > > PHP chose an underscore, it's more common in URLs. I prefer pt_BR. I can only disagree, dashes seems more common than underscores in URLs from my point of view, see typically slugs (https://en.wikipedia.org/wiki/Semantic_URL#Slug) but my link contradicts itself with a "Semantic_URL" instead of a "Semantic-URL" ... slugs (with dashes) are commonly used like at stackoverflow, yahoo, amazon, the Python doc itself (https://docs.python.org/3.5/library/stdtypes.html#numeric-types-int-float-complex), Also the Google recommendation "We recommend that you use hyphens (-) instead of underscores (_) in your URLs." https://support.google.com/webmasters/answer/76329?hl=en -- Julien Palard +08 99 49 05 40 http://mdk.fr From brett at python.org Wed Mar 23 12:22:05 2016 From: brett at python.org (Brett Cannon) Date: Wed, 23 Mar 2016 16:22:05 +0000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: On Tue, 22 Mar 2016 at 23:46 Nick Coghlan wrote: > On 23 March 2016 at 15:44, Nick Eubank wrote: > > True -- my feeling is that if we're ok with defaults in a pathlib method, > > why not put them in a string method? > > pathlib is explicitly about filesytem management and manipulation, > while strings don't inherently have anything to do with filesystems, > and only a little bit to do with serialisation (via str.encode). > > Having some lower level convenience functions in io may make sense > (specifically io.write_bytes(path, data) and io.write_text(path, data, > encoding=None, errors=None), but the str builtin definitely isn't the > right place for the capability. > I agree this has nothing to do with `str` or `bytes` directly and should not be exposed on those types. And if you look at the issue that Victor linked to, getting a function that behaves consistently for atomic file operations across OSs is very tough. I think it would be best is follow the advice Victor gave in the issue he linked to and for someone to make a library with defaults they think are reasonable for atomic file operations, put it on PyPI, and get community feedback before trying to add it to the stdlib. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Wed Mar 23 12:31:56 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 23 Mar 2016 09:31:56 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <20160321002945.GT8022@ando.pearwood.info> (Steven D'Aprano's message of "Mon, 21 Mar 2016 11:29:45 +1100") References: <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> <87io0goq04.fsf@vostro.rath.org> <20160321002945.GT8022@ando.pearwood.info> Message-ID: <87poul40lf.fsf@thinkpad.rath.org> On Mar 21 2016, Steven D'Aprano wrote: > On Sun, Mar 20, 2016 at 01:26:03PM -0700, Nikolaus Rath wrote: >> On Mar 19 2016, Nick Coghlan wrote: >> > The reason I suggest that approach is that most (all?) of us aren't >> > research scientists, so we have no idea what typical conventions are >> > for citations, nor how those conventions are changing. >> >> Which I believe makes it completely pointless to cite Python at all. As >> far as I can see, nowadays citations are given for two reasons: >> >> 1. To give the reader a starting point to get more information on a >> topic. >> >> 2. To formally acknowledge the work done by someone else (who ends up >> with an increased number of citations for the cited publication, >> which is unfortunately a crucial metric in most academic hiring and >> evaluation processes). >> >> In case of Python, an explicit citation thus adds nothing. > > I'm afraid I don't understand your reasoning here. Both of your two > reasons apply: a citation to Python gives the reader a starting point to > get more information As I said, I don't think a formal reference to something like [1] Python Core Team (2015). Python 3.6.0a0: A dynamic, open source programming language. Python Software Foundation. URL https://www.python.org/. gives the reader a better starting point than just writing "...a Python script was used to...". > and it formally acknowledges the work done by > others. Yeah, but the "others" don't benefit from it and don't care about it. > You might feel that everybody knows how to use google, and that a formal > acknowledgement is pointless because nobody cares, but that's a value > judgement about the usefulness of what the citation adds. Indeed. I judge the value of the extra information to be less than the value of the space consumed by it, so I consider it pointless. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From desmoulinmichel at gmail.com Wed Mar 23 12:48:02 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 23 Mar 2016 17:48:02 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <2336A5F4-7A2E-4A1A-A16A-736F513BCE32@yahoo.com> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F26B08.4060802@gmail.com> <2336A5F4-7A2E-4A1A-A16A-736F513BCE32@yahoo.com> Message-ID: <56F2C8C2.7070802@gmail.com> Le 23/03/2016 16:53, Andrew Barnert a ?crit : > On Mar 23, 2016, at 03:08, Michel Desmoulin wrote: > >>> But I think it is a coherent proposal, even if it's not one I like. :) >> >> And what do you think about adding except clauses to if ? with ? while ? > > Well, they don't have the problem that for has (for is already implicitly handling a specific exception in a specific way; none of them are), so they're coherent even without a solution to that problem. > > (With has the additional problem that it may not be immediately obvious on first glance whether the exit gets calls before or after the except clause, but if so, people can read the docs the first time they come across it and remember it after that.) > > But I also think they're even less necessary. They'd all be pure syntactic sugar for nesting the statement and a try/except, so we'd be making the language more complicated to learn and remember, and encouraging more code that isn't backward compatible. That's not a huge cost, but the benefit isn't very big either. For _really_ short cases, I think we want except expressions (PEP 463); for longish code, there's nothing wrong with an explicit try; the range for which hiding an implicit try inside if and while would really improve things is small enough that I don't think the benefit outweighs the cost. > Would love to see except expression win, but hasn't been rejected by the BDFL ? Plus it's not really related to the current proposal. > But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind. > The more I think about it, the more I think it's bad idea. We are mixing semantics of loops and exceptions or conditions and exceptions. I withdraw this part of the "general exceptions" proposal. I'm going back to the simpler and less dangerous proposal: for stuff in foo: bar(stuff) or: default() From mike at selik.org Wed Mar 23 13:13:58 2016 From: mike at selik.org (Michael Selik) Date: Wed, 23 Mar 2016 13:13:58 -0400 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> Message-ID: > On Mar 23, 2016, at 6:37 AM, Chris Angelico wrote: > > On Wed, Mar 23, 2016 at 9:04 PM, Michel Desmoulin > wrote: >>> (Whether or not to make slice notation usable outside subscript >>> operations could then be tackled as an independent question) >>> >>> For itertools.chain, it may make sense to simply promote it to the builtins. >> >> Same problem as with new keywords : it can be a problem with people >> using chain as a var name. >> > > Less of a problem though - it'd only be an issue for people who (a) > use chain as a var name, and (b) want to use the new shorthand. Their > code will continue to work identically with itertools.chain (or not > using it at all). With a new keyword, their code would instantly fail. I enjoy using ``chain`` as a variable name. It has many meanings outside of iteration tools. Three cheers for namespaces. Many of Python 3?s changes versus Python 2 were to take the pure stance -- change special statement syntax to function calls, move a few builtins into appropriate modules. If chain were a builtin, we?d have someone suggesting it move into itertools. From njs at pobox.com Wed Mar 23 13:14:07 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 23 Mar 2016 10:14:07 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <3F5C43C1-83BB-42B9-AF6F-4E52BFA79B04@yahoo.com> References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <3F5C43C1-83BB-42B9-AF6F-4E52BFA79B04@yahoo.com> Message-ID: On Mar 23, 2016 1:13 AM, "Andrew Barnert" wrote: > > On Mar 22, 2016, at 21:49, Nathaniel Smith wrote: > > > >> On Tue, Mar 22, 2016 at 9:33 PM, Chris Angelico wrote: > >> On Wed, Mar 23, 2016 at 3:22 PM, Alexander Belopolsky > >> wrote: > >>> > >>> On Tue, Mar 22, 2016 at 11:33 PM, Andrew Barnert via Python-ideas > >>> wrote: > >>>> with tempfile.NamedTemporaryFile('w', dir=os.path.dirname(path), > >>>> delete=False) as f: > >>>> f.write(s) > >>>> f.flush() > >>>> os.replace(f.path, path) > >>> You've got it wrong, but I understand what you tried to achieve. Note that > >>> the "write to temp and move" trick may not work if your /tmp and your path > >>> are mounted on different filesystems. And with some filesystems it may not > >>> work at all, but I agree that it would be nice to have a state of the art > >>> atomic write method somewhere in stdlib. > >> > >> It's specifically selecting a directory for the temp file, so it ought > >> to work. However, I'm not certain in my own head of the interaction > >> between NamedTemporaryFile with delete=False and os.replace (nor of > >> exactly how the latter differs from os.rename); what exactly happens > >> when the context manager exits here? And what happens if there's an > >> exception in the middle of this and stuff doesn't complete properly? > >> Are there points at which this could (a) destroy data by deleting > >> without saving, or (b) leave cruft around? > >> > >> This would be very nice to have as either stdlib or a well-documented recipe. > > > > Also: cross-platform support (Windows Is Different), > > I know a lot of people who never touch Windows with a 10-foot pole think this problem is still unsolved, but that's not true. Microsoft added sufficient atomic-replace APIs in 2003 (in the Win32 API only, not in crt/libc), and as of 3.3, Python's os.replace really is guaranteed to either atomically replace the file or leave it untouched and raise an exception on all platforms (as long as the files are on the same filesystem, and as long as there's not an EIO or equivalent because of an already-corrupted filesystem or a physical error on the media). (For platforms besides Windows and POSIX, it does this just by not existing and raising a NameError...) Likewise for safe temporary files--as of 3.3, tempfile.NamedTemporaryFile is safe on every platform where it exists, and that includes Windows. Ah, thanks! I indeed didn't know about this, and missed that the code was calling os.replace rather than os.rename. > > handling of > > permissions, do you care about xattrs? > > That can be handled effectively the same way as copy vs. copy2 if desired. I don't know if it's important enough, but if it is, it's easy. (My library actually does have options for preserving different levels of stuff, but I never use them.) Right, but this is the kind of thing that makes me worry about a one-size-fits-all solution :-). > > when you say "atomic" then do > > you mean atomic WRT power loss? > > Write-and-replace is atomic WRT both exceptions and power loss. Until the replace succeeds, the old version of the file is still there. This is guaranteed by POSIX and by Windows. If the OS can't offer that on some filesystem, it won't let you call os.replace. POSIX doesn't guarantee anything whatsoever over power loss. Individual filesystem implementations make somewhat stronger guarantees, but it's a mess: http://danluu.com/file-consistency/ At the very least atomicity requires fsync'ing the new file before calling rename, or you might end up with: Original code: new = open("new") new.write(...) new.close() os.replace("old", "new") Gets reordered on the way to the hard drive to become: new = open("new") os.replace("new", "old") new.write(...) new.close() POSIX does of course guarantee that if the OS reorders things like this then it has to hide that from you -- processes will always see the write happen before the rename. Except if there's a power loss, now we can have: Gets executed as: new = open("new") os.replace("new", "old") --- whoops, power lost here, and so is the file contents --- But fsync is a very expensive operation; there are plenty of applications for atomic writes where this is unnecessary (e.g. if the file is being used as an IPC mechanism, so power loss -> the processes die, no one cares about their IPC channel anymore). And there are plenty of applications where this is insufficient (e.g. if you expect/need atomic_write(path1, data1); atomic_write(path2, data2) to guarantee that the two atomic writes can't be reordered relative to each other). I don't want to get sucked into a long debate about this; it's entirely likely that adding something like that original recipe to the stdlib would be an improvement, so long as it had *very* detailed docs explaining the exact tradeoffs made. All I want to do is raise a cautionary flag that such an effort would need to tread carefully :-) -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Mar 23 13:18:22 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 23 Mar 2016 19:18:22 +0200 Subject: [Python-ideas] [Python Ideas] Random weighted choice idea to add to random lib In-Reply-To: <831C9C40-4CF8-427E-9AED-877BE8C64446@yahoo.com> References: <831C9C40-4CF8-427E-9AED-877BE8C64446@yahoo.com> Message-ID: On 23.03.16 18:13, Andrew Barnert via Python-ideas wrote: >> http://bugs.python.org/issue18844 > > Why not just pick up from their original implementation, fix any bugs, > and propose that, instead of starting over from scratch? Are there any bugs? From abarnert at yahoo.com Wed Mar 23 13:22:07 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 10:22:07 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F2C8C2.7070802@gmail.com> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F26B08.4060802@gmail.com> <2336A5F4-7A2E-4A1A-A16A-736F513BCE32@yahoo.com> <56F2C8C2.7070802@gmail.com> Message-ID: <8CC650C5-4482-437D-A5F4-DD3185957EBB@yahoo.com> On Mar 23, 2016, at 09:48, Michel Desmoulin wrote: > > Le 23/03/2016 16:53, Andrew Barnert a ?crit : >> On Mar 23, 2016, at 03:08, Michel Desmoulin wrote: >> >>>> But I think it is a coherent proposal, even if it's not one I like. :) >>> >>> And what do you think about adding except clauses to if ? with ? while ? >> >> Well, they don't have the problem that for has (for is already implicitly handling a specific exception in a specific way; none of them are), so they're coherent even without a solution to that problem. >> >> (With has the additional problem that it may not be immediately obvious on first glance whether the exit gets calls before or after the except clause, but if so, people can read the docs the first time they come across it and remember it after that.) >> >> But I also think they're even less necessary. They'd all be pure syntactic sugar for nesting the statement and a try/except, so we'd be making the language more complicated to learn and remember, and encouraging more code that isn't backward compatible. That's not a huge cost, but the benefit isn't very big either. For _really_ short cases, I think we want except expressions (PEP 463); for longish code, there's nothing wrong with an explicit try; the range for which hiding an implicit try inside if and while would really improve things is small enough that I don't think the benefit outweighs the cost. > > Would love to see except expression win, but hasn't been rejected by the > BDFL ? Plus it's not really related to the current proposal. I thought PEP 463 was just stalled waiting on a reference implementation before he'd declare either way? At any rate, the reason it's related is that every case I could think of where I'd want to use if/except is so trivial that I'd much rather use an if expression in an except expression. There may be cases that are not quite trivial enough to demand that, but still small enough that saving one line of code makes a difference; just because I can't think of any certainly doesn't mean they don't exist. :) But, given the rest of your reply, I guess it doesn't matter. >> But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind. > > The more I think about it, the more I think it's bad idea. We are mixing > semantics of loops and exceptions or conditions and exceptions. > > I withdraw this part of the "general exceptions" proposal. I'm going > back to the simpler and less dangerous proposal: > > for stuff in foo: > bar(stuff) > or: > default() You may want to get everyone back on track by writing a complete proposal (does or come before or after else? what exactly are the semantics? how does it translate to bytecode, if you understand how for already translates?) with just this suggestion some motivating examples. I still think this isn't really necessary. In every case I can think of where I really want to do something special on empty, I definitely have a sequence, and can just do "if not foo:", or I need a count rather than just an empty flag, or I'm already doing some EAFP handling, or... But again, just because I can't think of examples doesn't mean they don't exist. It just implies that maybe the other people you're trying to convince can't think of them either, so it would be better for you to show us. From Andy.Henshaw at gtri.gatech.edu Wed Mar 23 13:13:03 2016 From: Andy.Henshaw at gtri.gatech.edu (Henshaw, Andy) Date: Wed, 23 Mar 2016 17:13:03 +0000 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> Message-ID: > -----Original Message----- > > On Mon, Mar 21, 2016 at 7:01 PM, Stephen J. Turnbull > wrote: > > item = _sentinel = object() > > for item in iterable: > > # do for each item > > if item is _sentinel: > > # do exactly when iterable raises StopIteration on the first > > pass > > What if 'iterable' is locals().values()? Can you, with perfect reliability, > recognize that case? AIUI this is exactly why next() and > __next__() are defined to "return a value or raise", rather than "return a > value or return a magic no-more-values value", because there's always the > possibility that the no-more-values value is a legitimately-yielded value. > > Maybe this situation isn't important enough or common enough to justify > dedicated syntax, but it's definitely a possibility. > > ChrisA is_empty = True for item in iterable: is_empty = False # do for each item if is_empty: # do exactly when iterable raises StopIteration on the first Maybe I'm missing something, but this seems to be an unnecessary addition. Andy From abarnert at yahoo.com Wed Mar 23 13:39:28 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 10:39:28 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> Message-ID: <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> On Mar 23, 2016, at 10:13, Michael Selik wrote: > >> On Mar 23, 2016, at 6:37 AM, Chris Angelico wrote: >> >> On Wed, Mar 23, 2016 at 9:04 PM, Michel Desmoulin >> wrote: >>>> (Whether or not to make slice notation usable outside subscript >>>> operations could then be tackled as an independent question) >>>> >>>> For itertools.chain, it may make sense to simply promote it to the builtins. >>> >>> Same problem as with new keywords : it can be a problem with people >>> using chain as a var name. >> >> Less of a problem though - it'd only be an issue for people who (a) >> use chain as a var name, and (b) want to use the new shorthand. Their >> code will continue to work identically with itertools.chain (or not >> using it at all). With a new keyword, their code would instantly fail. > > I enjoy using ``chain`` as a variable name. It has many meanings outside of iteration tools. Three cheers for namespaces. As Chris just explained in the message you're replying to, this wouldn't affect you. I've used "vars" and "compile" and "dir" as local variables many times; as long as I don't need to call the builtin in the same scope, it's not a problem. The same would be true if you keep using "chain". Unless you want to chain iterables of your chains together, it'll never arise. Also, putting things in builtins isn't _always_ bad. It's just a high standard that has to be met. I don't think anyone believes any, all, and enumerate fail to meet that standard. So the question is just whether chain meets it. My problem is that I'm not sure chain really does meet it. It's chain.from_iterable that I often see people reaching for and not finding, and moving chain to builtins won't help those people find it. (This is compounded by the fact that even a novice can figure out how to do chain given chain.from_iterable, but not the other way around.) Also, for something we expect novices to start using as soon as they discover iterators, it seems worrisome that we'd be expecting them to understand the idea of a static method on something that looks like a function but is actually a type before they can have a clue of what it means. In another thread a year or two ago, someone suggested making chain.from_iterable into a builtin with a different name, maybe "flatten". But that now means we're adding _two_ builtins rather than one, and they're very closely related but don't appear so in the docs, which obviously increases the negatives... Still, I like adding chain (and/or flatten) to builtins a lot more than I like adding sequence behavior to some iterators, or adding a whole new kind of function-like slicing syntax to all iterables, or any of the other proposals here. From alexander.belopolsky at gmail.com Wed Mar 23 13:48:25 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 23 Mar 2016 13:48:25 -0400 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <87poul40lf.fsf@thinkpad.rath.org> References: <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> <87io0goq04.fsf@vostro.rath.org> <20160321002945.GT8022@ando.pearwood.info> <87poul40lf.fsf@thinkpad.rath.org> Message-ID: On Wed, Mar 23, 2016 at 12:31 PM, Nikolaus Rath wrote: > > and it formally acknowledges the work done by > > others. > > Yeah, but the "others" don't benefit from it and don't care about it. What's the basis for this assertion? Even if you do a survey of all present and past contributors and they all agree with you, it does not mean that increased visibility in academic publications will not attract future contributors who would value proper citations. Personally, I am +0 on the idea. -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Wed Mar 23 14:01:37 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 23 Mar 2016 19:01:37 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <8CC650C5-4482-437D-A5F4-DD3185957EBB@yahoo.com> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F26B08.4060802@gmail.com> <2336A5F4-7A2E-4A1A-A16A-736F513BCE32@yahoo.com> <56F2C8C2.7070802@gmail.com> <8CC650C5-4482-437D-A5F4-DD3185957EBB@yahoo.com> Message-ID: <56F2DA01.9080608@gmail.com> Le 23/03/2016 18:22, Andrew Barnert a ?crit : > On Mar 23, 2016, at 09:48, Michel Desmoulin wrote: >> >> Le 23/03/2016 16:53, Andrew Barnert a ?crit : >>> On Mar 23, 2016, at 03:08, Michel Desmoulin wrote: >>> >>>>> But I think it is a coherent proposal, even if it's not one I like. :) >>>> >>>> And what do you think about adding except clauses to if ? with ? while ? >>> >>> Well, they don't have the problem that for has (for is already implicitly handling a specific exception in a specific way; none of them are), so they're coherent even without a solution to that problem. >>> >>> (With has the additional problem that it may not be immediately obvious on first glance whether the exit gets calls before or after the except clause, but if so, people can read the docs the first time they come across it and remember it after that.) >>> >>> But I also think they're even less necessary. They'd all be pure syntactic sugar for nesting the statement and a try/except, so we'd be making the language more complicated to learn and remember, and encouraging more code that isn't backward compatible. That's not a huge cost, but the benefit isn't very big either. For _really_ short cases, I think we want except expressions (PEP 463); for longish code, there's nothing wrong with an explicit try; the range for which hiding an implicit try inside if and while would really improve things is small enough that I don't think the benefit outweighs the cost. >> >> Would love to see except expression win, but hasn't been rejected by the >> BDFL ? Plus it's not really related to the current proposal. > > I thought PEP 463 was just stalled waiting on a reference implementation before he'd declare either way? > > At any rate, the reason it's related is that every case I could think of where I'd want to use if/except is so trivial that I'd much rather use an if expression in an except expression. There may be cases that are not quite trivial enough to demand that, but still small enough that saving one line of code makes a difference; just because I can't think of any certainly doesn't mean they don't exist. :) > > But, given the rest of your reply, I guess it doesn't matter. > >>> But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind. >> >> The more I think about it, the more I think it's bad idea. We are mixing >> semantics of loops and exceptions or conditions and exceptions. >> >> I withdraw this part of the "general exceptions" proposal. I'm going >> back to the simpler and less dangerous proposal: >> >> for stuff in foo: >> bar(stuff) >> or: >> default() > > You may want to get everyone back on track by writing a complete proposal (does or come before or after else? what exactly are the semantics? how does it translate to bytecode, if you understand how for already translates?) with just this suggestion some motivating examples. > > I still think this isn't really necessary. In every case I can think of where I really want to do something special on empty, I definitely have a sequence, and can just do "if not foo:", or I need a count rather than just an empty flag, or I'm already doing some EAFP handling, or... But again, just because I can't think of examples doesn't mean they don't exist. It just implies that maybe the other people you're trying to convince can't think of them either, so it would be better for you to show us. > Ok, but before that, is anyone still attached to another solution ? From mike at selik.org Wed Mar 23 14:19:12 2016 From: mike at selik.org (Michael Selik) Date: Wed, 23 Mar 2016 14:19:12 -0400 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> Message-ID: <303B3A8E-5791-4E43-9CE6-5B679D8D02C6@selik.org> > On Mar 23, 2016, at 1:13 PM, Henshaw, Andy wrote: > >> -----Original Message----- >> >> On Mon, Mar 21, 2016 at 7:01 PM, Stephen J. Turnbull >> wrote: >>> item = _sentinel = object() >>> for item in iterable: >>> # do for each item >>> if item is _sentinel: >>> # do exactly when iterable raises StopIteration on the first >>> pass >> >> What if 'iterable' is locals().values()? Can you, with perfect reliability, >> recognize that case? AIUI this is exactly why next() and >> __next__() are defined to "return a value or raise", rather than "return a >> value or return a magic no-more-values value", because there's always the >> possibility that the no-more-values value is a legitimately-yielded value. >> >> Maybe this situation isn't important enough or common enough to justify >> dedicated syntax, but it's definitely a possibility. >> >> ChrisA > > is_empty = True > for item in iterable: > is_empty = False > # do for each item > if is_empty: > # do exactly when iterable raises StopIteration on the first > > Maybe I'm missing something, but this seems to be an unnecessary addition. It?s comparable in purpose to the ``else`` clause on a ``for`` or ``while``. It eliminates the need for a flag variable, like ``was_completed`` or ``is_empty``. It saves two lines of code, but more importantly avoids the accidental flip of True/False when setting the flag variable. Whether that warrants adding new syntax... I?m not convinced by the current proposals, but I can imagine finding a good keyword for the purpose. A keyword as good as ``else`` at least. From rosuav at gmail.com Wed Mar 23 14:58:38 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 24 Mar 2016 05:58:38 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <8CC650C5-4482-437D-A5F4-DD3185957EBB@yahoo.com> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F26B08.4060802@gmail.com> <2336A5F4-7A2E-4A1A-A16A-736F513BCE32@yahoo.com> <56F2C8C2.7070802@gmail.com> <8CC650C5-4482-437D-A5F4-DD3185957EBB@yahoo.com> Message-ID: On Thu, Mar 24, 2016 at 4:22 AM, Andrew Barnert via Python-ideas wrote: >> Would love to see except expression win, but hasn't been rejected by the >> BDFL ? Plus it's not really related to the current proposal. > > I thought PEP 463 was just stalled waiting on a reference implementation before he'd declare either way? No; AIUI it was informally rejected. ChrisA From jsbueno at python.org.br Wed Mar 23 15:06:25 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 23 Mar 2016 16:06:25 -0300 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> Message-ID: On 23 March 2016 at 14:13, Henshaw, Andy wrote: >> -----Original Message----- >> >> On Mon, Mar 21, 2016 at 7:01 PM, Stephen J. Turnbull >> wrote: >> > item = _sentinel = object() >> > for item in iterable: >> > # do for each item >> > if item is _sentinel: >> > # do exactly when iterable raises StopIteration on the first >> > pass >> >> What if 'iterable' is locals().values()? Can you, with perfect reliability, >> recognize that case? AIUI this is exactly why next() and >> __next__() are defined to "return a value or raise", rather than "return a >> value or return a magic no-more-values value", because there's always the >> possibility that the no-more-values value is a legitimately-yielded value. >> >> Maybe this situation isn't important enough or common enough to justify >> dedicated syntax, but it's definitely a possibility. >> >> ChrisA > > is_empty = True > for item in iterable: > is_empty = False > # do for each item > if is_empty: > # do exactly when iterable raises StopIteration on the first For me, after checking all other e-mails, this is the "one and obvious way" of doing it, does not matter if it "seems" boring - it had not to do with being boring, or having one extra variable - it has to do with being readable. Event the variant that would do item = sentinel = object() for ... is less readable, if more elegant. As for adding different syntax for this, I am +0 For the idea of adding "except" clauses to 'with', 'for' and 'while' (along with a "EmptyIteration" exception) I am + 0.5 > > Maybe I'm missing something, but this seems to be an unnecessary addition. > > Andy > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From mike at selik.org Wed Mar 23 15:24:29 2016 From: mike at selik.org (Michael Selik) Date: Wed, 23 Mar 2016 19:24:29 +0000 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> Message-ID: On Wed, Mar 23, 2016 at 1:39 PM Andrew Barnert wrote: > On Mar 23, 2016, at 10:13, Michael Selik wrote: > > > >> On Mar 23, 2016, at 6:37 AM, Chris Angelico wrote: > >> > >> On Wed, Mar 23, 2016 at 9:04 PM, Michel Desmoulin > >> wrote: > >>>> (Whether or not to make slice notation usable outside subscript > >>>> operations could then be tackled as an independent question) > >>>> > >>>> For itertools.chain, it may make sense to simply promote it to the > builtins. > >>> > >>> Same problem as with new keywords : it can be a problem with people > >>> using chain as a var name. > >> > >> Less of a problem though - it'd only be an issue for people who (a) > >> use chain as a var name, and (b) want to use the new shorthand. Their > >> code will continue to work identically with itertools.chain (or not > >> using it at all). With a new keyword, their code would instantly fail. > > > > I enjoy using ``chain`` as a variable name. It has many meanings outside > of iteration tools. Three cheers for namespaces. > > As Chris just explained in the message you're replying to, this wouldn't > affect you. I've used "vars" and "compile" and "dir" as local variables > many times; as long as I don't need to call the builtin in the same scope, > it's not a problem. The same would be true if you keep using "chain". > Unless you want to chain iterables of your chains together, it'll never > arise. > Depends what you mean by "affect". It'll affect how I read my colleagues' code. I want to see ``from itertools import chain`` at the top of their modules if they're using chain in the sense of chaining iterables. > Also, putting things in builtins isn't _always_ bad. It's just a high > standard that has to be met. I don't think anyone believes any, all, and > enumerate fail to meet that standard. So the question is just whether chain > meets it. > Part of why ``any`` and ``all`` succeed as builtins is their natural language meanings are quite close to their Python meaning. ``enumerate`` succeeds because it is unusual in natural language. For many people, Python might be the most frequent context for using that word. Some evidence that ``chain`` should be promoted to builtin would be that ``chain`` is used more often than the rest of the itertools library and that the word "chain" is rarely used except as the itertools chain. Luckily one could search some public projects on GitHub to provide that evidence. If no one bothers to do so, I'm guessing the desire isn't so great. > My problem is that I'm not sure chain really does meet it. It's > chain.from_iterable that I often see people reaching for and not finding, > and moving chain to builtins won't help those people find it. (This is > compounded by the fact that even a novice can figure out how to do chain > given chain.from_iterable, but not the other way around.) Also, for > something we expect novices to start using as soon as they discover > iterators, it seems worrisome that we'd be expecting them to understand the > idea of a static method on something that looks like a function but is > actually a type before they can have a clue of what it means. > > > In another thread a year or two ago, someone suggested making > chain.from_iterable into a builtin with a different name, maybe "flatten". > But that now means we're adding _two_ builtins rather than one, and they're > very closely related but don't appear so in the docs, which obviously > increases the negatives... > > Still, I like adding chain (and/or flatten) to builtins a lot more than I > like adding sequence behavior to some iterators, or adding a whole new kind > of function-like slicing syntax to all iterables, or any of the other > proposals here. > I like the LazyList you wrote. If the need for operator-style slicing on iterators were great, I think we'd find a decent amount of usage of such a wrapper. Its absence is evidence that ``from itertools import islice`` is more pleasant. As has been discussed many times, lazy sequences like range cause some confusion by providing both generator behavior and getitem/slicing. Perhaps it's better to keep the two varieties of iterables, lazy and non-lazy, more firmly separated by having ``lst[a:b]`` used exclusively for true sequences and ``islice(g, a, b)`` for generators. Just yesterday I got frustrated by the difference of return types from subscripting bytes versus str while trying to keep my code 2/3 compatible: ``bytestring[0]`` gives an integer while ``textstring[0]`` gives a 1-length str. I had to resort to the awkwardness of ``s[0:1]`` not knowing whether I'll have a Python 3 bytes or Python 2 str. I prefer imports to inconsistency. A couple questions to help clarify the situation: 1. Do you prefer ``a.union(b)`` or ``a | b``? 2. Do you prefer ``datetime.now() + timedelta(days=5)`` or ``5.days.from_now``? I think the answer to #2 is clear, we prefer Python to Ruby (more specifically the Rails sub-community). The answer to #1 is more difficult. I'm often tempted to say ``a | b`` for its elegance, but I keep coming back to ``a.union(b)`` as clunky but readable, easy to explain, etc. -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Mar 23 16:17:32 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 23 Mar 2016 21:17:32 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> Message-ID: <56F2F9DC.5000203@mail.de> On 23.03.2016 00:56, Andrew Barnert via Python-ideas wrote: > The only question is whether any of them are obvious enough (even for > novices). If not, we could argue whether any proposed change would be > significantly _more_ obvious. And, if so, then the question is whether > the change is worth the cost. But I think what we have is already > obvious enough. It was reasonable for both Django and for jinja2 to add this. People actually asked for it https://github.com/mitsuhiko/jinja2/issues/77 > (Especially since 90% of the time, when you need to do something > special on empty, you explicitly have a sequence, not any iterable, so > it's just "if not seq:".) Repeating that you think it will be a sequence will make it one. We cannot keep track of all possible implementations in our code base. So, if somebody changes from sequence to iterator for whatever reason, it should work without weird errors. > So really, this proposal is really just asking for syntactic sugar > that complicates the language in exchange for making some > already-understandable code a little more concise, which doesn't seem > worth it. Did you even think what you just said? Almost everything in Python is "just syntactic sugar" compared to most other Turing-complete languages. To put it differently: every construct that abstracts away "goto" is "just syntactic sugar". >>> What do you think >>> about an alternative that can handle more than empty and else? >>> >>> for item in collection: >>> # do for item >>> except NeverExecuted: >>> # do if collection is empty >>> >>> It basically merges "try" and "for" and make "for" emit EmptyCollection. >> Does this mean that every single for-loop that doesn't catch >> NeverExecuted (or EmptyCollection) will raise an exception? > Elsewhere he mentioned that EmptyCollection would be a subclass of StopIteration. > > Presumably, every iterator type (or just the builtin ones, and hopefully "many" others?) would have special code to raise EmptyCollection if empty. Like this pseudocode for list_iterator: > > def __next__(self): > if not self.lst: > raise EmptyCollection > elif self.i >= len(self.lst): > raise StopIteration > else: > i = self.i > self.i += 1 > return self.lst[self.i] Interesting. That would be an alternative approach. > Or, alternatively, for itself would do this. I think the most important question is: do we want to support the "ifempty" feature in the first place? I can see a lot of discussion around it; that makes it controversial and that means there is half/half support/rejection (support by more people + rejection by less people but those have many good questions we would need to answer before we get this proposal to work). The only real reason against it so far: "it makes the language more complicated because I don't need it". Not entirely compelling but understandable. I further see that people would like this feature GIVEN a very GOOD syntax/approach and I entirely agree with that. > The for loop bytecode would have to change to stash an "any values seen" flag somewhere such that if it sees StopIteration and hasn't seen any values, it converts that to an EmptyCollection. Or any of the other equivalents (e.g., the compiler could unroll the first PyIter_Next from loop from the rest of them to handle it specially). Something like that. > But this seems like it would add a lot of overhead and complexity to every loop whether desired or not. If the "for" does not have any empty-equivalent clauses, there is no need to introduce that overhead in the first place. So we can conclude: 1) none overhead for regular "for"s 2) less overhead for "for-ifempty" because it would be done in C and not in Python >> If not, then how will this work? Is this a special kind of >> exception-like process that *only* operates inside for loops? >> >> What will an explicit "raise NeverExecuted" do? > Presumably that's the same question as what an explicit raise StopIteration does. Just as there's nothing stopping you from writing a __next__ method that raises StopIteration but then yields more values of called again, there's nothing stopping you from raising NeverExecuted pathologically, but you shouldn't do so. M > >>> So, independent of the initial "never executed loop body" use-case, one >>> could also emulate the "else" clause by: >>> >>> for item in collection: >>> # do for item >>> except StopIteration: >>> # do after the loop >> That doesn't work, for two reasons: >> >> (1) Not all for-loops use iterators. The venerable old "sequence >> protocol" is still supported for sequences that don't support __iter__. >> So there may not be any StopIteration raised at all. > I think there always is. > > IIRC, PyObject_Iter (the C API function used by iter and by for loops) actually constructs a sequence iterator object if the object doesn't have tp_iter (__iter__ for Python types) but does have tp_sequence (__getitem__ for Python types, but, e.g., dict has __getitem__ without having tp_sequence). And the for loop doesn't treat that sequence iterator any different from "real" iterators returned by __iter__; it just calls tp_next (__next__) until StopIteration. (And the "other half" of the old-style sequence protocol, that lets old-style sequences be reversed if they have a length, is similarly implemented by the C API underneath the reversed function.) > > I'm on my phone right now, so I can't double-check any of the details, but I'm 80% sure they're all at least pretty close... I think I would have to deal with the old protocol given the proposal is accepted. >> (2) Even if StopIteration is raised, the for-loop catches it (in a >> manner of speaking) and consumes it. >> >> So to have this work, we would need to have the for-loop re-raise >> StopIteration... but what happens if you don't include an except >> StopIteration clause? Does every bare for-loop with no "except" now >> print a traceback and halt processing? If not, why not? > I think this could be made to work: a for loop without an except clause handles StopIteration the same as today (by jumping to the else clause), but one that does have one or more except clauses just treats it like a normal exception. > > Of course this would mean for/except/else is now legal but useless, which could be confusing ("why does my else clause no longer run when I add an 'except ValueError' clause?"). One could disallow "else" in case any "except" is defined. > More generally, I think the fact that for/except StopIteration is almost but not quite identical to plain for would be confusing more often than helpful. You bet how people think about "else". "So, 'else' is always executed after the for?" "Yes, but only when there is no 'break' executed in the 'for'" "... *thinking* ... okay ..." > But I think it is a coherent proposal, even if it's not one I like. :) Best, Sven From srkunze at mail.de Wed Mar 23 16:26:03 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 23 Mar 2016 21:26:03 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <2336A5F4-7A2E-4A1A-A16A-736F513BCE32@yahoo.com> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F26B08.4060802@gmail.com> <2336A5F4-7A2E-4A1A-A16A-736F513BCE32@yahoo.com> Message-ID: <56F2FBDB.9040507@mail.de> On 23.03.2016 16:53, Andrew Barnert via Python-ideas wrote: > On Mar 23, 2016, at 03:08, Michel Desmoulin wrote: > >>> But I think it is a coherent proposal, even if it's not one I like. :) >> And what do you think about adding except clauses to if ? with ? while ? > Well, they don't have the problem that for has (for is already implicitly handling a specific exception in a specific way; none of them are), so they're coherent even without a solution to that problem. Could you describe what you mean by coherent? > (With has the additional problem that it may not be immediately obvious on first glance whether the exit gets calls before or after the except clause, but if so, people can read the docs the first time they come across it and remember it after that.) > > But I also think they're even less necessary. They'd all be pure syntactic sugar for nesting the statement and a try/except, so we'd be making the language more complicated to learn and remember, I completely disagree here. > and encouraging more code that isn't backward compatible. What exactly is the problem with that? Everything discussed on python-ideas would be 3.6+ anyway. > That's not a huge cost, but the benefit isn't very big either. For _really_ short cases, I think we want except expressions (PEP 463); for longish code, there's nothing wrong with an explicit try; the range for which hiding an implicit try inside if and while would really improve things is small enough that I don't think the benefit outweighs the cost. That's your opinion here. And I find it interesting that others already have thought of the merging of try and if, for and while. It would definitely come in handy. But I understand from your previous responses that you'd rather use Assembler because it is has no unneeded syntactic sugar. ;-) Just kidding. > But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind. From what I can see regarding the if-except proposal, the improvement would be constant. So, either you like it (as I do) or you don't see them as necessary (as you do). There is nothing somebody can do to convince you that saving 1 line of code and 1 indentation level across 20 lines of code are good or bad. Best, Sven From xksteven at uchicago.edu Wed Mar 23 16:43:13 2016 From: xksteven at uchicago.edu (Steven Basart) Date: Wed, 23 Mar 2016 15:43:13 -0500 Subject: [Python-ideas] Fwd: [Python Ideas] Random weighted choice idea to add to random lib In-Reply-To: References: <831C9C40-4CF8-427E-9AED-877BE8C64446@yahoo.com> <8862D69D-0BFD-4559-8282-796951738C97@yahoo.com> Message-ID: Hello I'm going a change from my example code and talk about the suggestions I would make to the issue I mentioned because as you said it does have accumulation of floating point errors and is a bit simplistic. Maybe it is better to be simple but not at the expense of gross inefficiency of (case of O(log(n)) vs O(n)). So here's how I'd change his code from the issue here: http://bugs.python.org/issue18844 the code on here: http://bugs.python.org/review/18844/patch/12674/46425 I've marked what I would insert with a plus and a remove with a minus at the beginning of the line First only expose the weighted_choice function and leave the generator as the backend. Keep it simple. so from here on I'd change weighted_choice_generator to _weighted_choice_generator - "SystemRandom", "weighted_choice_generator", "weighted_choice"] + "SystemRandom", "weighted_choice"] Second we could store the previous data to prevent having to recreate the binary tree def weighted_choice(self, data): + # we could store a copy of the data to prevent remaking the tree however + # this will take up order(n) space to go to sublinear time in search + # or store a hash of the value + if self.hashval == hash(tuple(data)): + return next(self._weighted_choice_generator(data)) + else: + return self._weighted_choice_generator(data) + # Other approach is to do away with the function above and simply rename the function + # below as weighted_choice because even this would remake the tree but efficiently find the value each time def _weighted_choice_generator(self, data): I dislike of having to create two new functions into random lib API when I think only really one is required. Ultimately I think we could do away with the top function and simply keep the generator function but call it weighted_choice. On Wed, Mar 23, 2016 at 12:11 PM, Andrew Barnert wrote: > On Mar 23, 2016, at 09:43, Steven Basart wrote: >> >> Hello Andrew, >> >>> On Wed, Mar 23, 2016 at 11:13 AM, Andrew Barnert wrote: >>> >>> On Mar 23, 2016, at 07:51, Steven Basart wrote: >>> >>> Hello, >>> >>> I had an idea that has in some form been proposed before. I think a random weighted choice function would be a very good addition to the random module in the python standard library. >>> >>> Here's a basic implementation of what's required albeit slightly inefficient as the search is linear as opposed to creating a binary tree and taking log n time to find the correct choice. pmf is short for probability mass function. It doesn't handle all error cases but it is more of an example to show the idea. >>> >>> >>> def randweighted(self,pmf=list): >>> """Return an int in the range [0,len(list)] weighted by list of values. >>> Raises error if list is empty. >>> """ >>> if(not pmf): >>> raise ValueError("pmf is equal to empty list") >>> cummassf = [sum(pmf[:i]) for i in pmf] >>> randVal = self._randbelow(len(pmf)) >>> for x in cummassf: >>> if randVal < x: >>> return x >>> >>> >>> Why a default value of list, especially since that's just going to raise a TypeError later on? >>> >>> Why the (unpythonic) test for "pmf == []" instead of just testing "if not pmf:"? Then it'll work on, say, tuples or third-party sorted lists or any other sequence. (With a tiny change, you can make it work on any iterable, but I don't know whether that's desirable.) >>> >>> The x[:i] is accessing an unbound local named x. What's it supposed to be? >> It was supposed to be >> >> cummassf = [sum(pmf[:i]) for i in pmf] >> >> Sorry about that I updated the above example accordingly. >> >>> Whatever x is, each element of cummassf is going to be some kind of sequence, so randVal < x on each element is going to raise a TypeError. Maybe you wanted a sum somewhere? >> Yes this is correct and I updated how it was supposed to be above. >>> If you fix that, doesn't this accumulate floating point errors, so there's a significant chance the result won't add up to 1.0 even if all of the weights are correct to 1 ULP? >>> >>> If the weights _aren't_ correct, surely you'd want a ValueError or something. This code will just return None if the sum is too low, and silently choose from the first 1.0 if it's too high. (I think np.random.choice raises ValueError if not isclose(sum(p), 1.0), but I'd have to check.) >> I think >> To fix this we could we could do one of two options: >> >> 1) Normalize the inputs >> total = 0 >> total += sum(pmf) > > Why not just total = sum(pmf)? > > Also, this isn't normalizing. Normalizing means scaling by the appropriate factor. > >> check if total is close to 1.0 >> then >> ... (same as code above) >> >> finally set cummassf[-1] = 1.0 >> >> 2) Scale the random variable by the total amount. In this case there >> is no need to take into account the floating point error. > > There very definitely is a need to take into account the floating point error! Adding an extra operation (a division) doesn't make the accumulated error go away, it makes it worse. > > I believe PEP 450 (for the statistics module) has a good discussion of accumulating errors in running sums, and how to deal with it. > > Meanwhile, from a very quick survey, it looks like many implementations allow integer weights (which you get directly out of Counter) and do exact rather than inexact math in those cases. That seems like a good idea. > >> 3) The last idea I had was to instead of returning the value we could >> return the index of which position to get as well as the actual value. >> index = 0 >> for x in cummassf: >> index += 1 >> if randVal < x: >> return (index, pmf[index]) > > Use enumerate; it's a builtin for a reason. > > Meanwhile, what use is returning the weight at all? What would a caller do with it? > > But again, why are you trying to write this from scratch if someone already wrote a working version? > >>> Finally, this is clearly a weighted randrange (with a fixed min of 0), but it claims to be a weighted choice. Obviously you can build the latter out of the former (e.g., take a val:weight mapping, call randweighted on the values, then do next(islice(keys, result, None)), but you do need to actually build it (and show what the API for it will be) if that's the motivating reason. (Also, I think an _unnormalized_ val:weight mapping might be a better API--after all, that means you can immediately call it on a Counter--but you probably want to look at what numpy, R, Mathematica, etc. do...) >>> >>> There was a ticket that got started before. It has since stalled because I believe the original author intended the library to get called over an over so they came up with generators which would be more efficient if passing in the same list. Otherwise there isn't really a point to using generators in my opinion. >>> >>> http://bugs.python.org/issue18844 >>> >>> >>> Why not just pick up from their original implementation, fix any bugs, and propose that, instead of starting over from scratch? >> I've never contributed to the python package before so I'm honestly >> not sure how to go about updating the previous ticket. I just knew >> how to find it. >> I did check out the source code from mercurial and started asking on >> irc about it. Someone there suggested I present my idea here. >> I wanted to give a simple version of what the code would perhaps look >> like in the python-ideas mailing list and also discuss the main api of >> what the function would do. I just believe it is a really useful >> function and having it available for many users would save time. > > Sure, but you don't seem to be presenting a simplified version of the code you found, but instead writing completely different code that doesn't work, doesn't have the API you want, and doesn't make it clear what you're trying to do. Again, why not just post working code with the right interface by copying the existing implementation or stripping it down? > > Meanwhile, there definitely is a point to the generators. It's very common to want to do N random choices in a row. For unweighted choice, that's fine; just call choice N times. But for weighted choice, there's a linear-time setup followed by a sublinear-time choice function, so just calling the whole thing N times will take O(NM) time, while doing the setup only once takes O(M+NlogM) or something. Most of the functions from NumPy and other languages that I found in a quick google do this by returning N values instead of 1, which works, but it seems like a generator is a much more pythonic way of implementing the same idea. > > Also, it doesn't seem to meaningfully complicate things. With your implementation (once you get it working), all you have to do is wrap the last bit in a "while True:" and change the "return" to "yield" and you're done. > > From Nikolaus at rath.org Wed Mar 23 16:54:41 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Wed, 23 Mar 2016 13:54:41 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: (Alexander Belopolsky's message of "Wed, 23 Mar 2016 13:48:25 -0400") References: <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> <87io0goq04.fsf@vostro.rath.org> <20160321002945.GT8022@ando.pearwood.info> <87poul40lf.fsf@thinkpad.rath.org> Message-ID: <87mvpo52zy.fsf@thinkpad.rath.org> On Mar 23 2016, Alexander Belopolsky wrote: > On Wed, Mar 23, 2016 at 12:31 PM, Nikolaus Rath wrote: > >> > and it formally acknowledges the work done by >> > others. >> >> Yeah, but the "others" don't benefit from it and don't care about it. > > What's the basis for this assertion? The responses on this list so far. > Even if you do a survey of all present and past contributors [...] I think it's best to work with the data that's available. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From alexander.belopolsky at gmail.com Wed Mar 23 17:01:00 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 23 Mar 2016 17:01:00 -0400 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <87mvpo52zy.fsf@thinkpad.rath.org> References: <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> <87io0goq04.fsf@vostro.rath.org> <20160321002945.GT8022@ando.pearwood.info> <87poul40lf.fsf@thinkpad.rath.org> <87mvpo52zy.fsf@thinkpad.rath.org> Message-ID: On Wed, Mar 23, 2016 at 4:54 PM, Nikolaus Rath wrote: > >> Yeah, but the "others" don't benefit from it and don't care about it. > > > > What's the basis for this assertion? > > The responses on this list so far. The first two responses to OP were +1's. That's enough to invalidate your assertion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbvsmo at gmail.com Wed Mar 23 17:26:52 2016 From: jbvsmo at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Bernardo?=) Date: Wed, 23 Mar 2016 18:26:52 -0300 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F07E78.9020309@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: On Mon, Mar 21, 2016 at 8:06 PM, Michel Desmoulin wrote: > Itertools is great, and some functions in it are more used than others: > > - islice; > - chain; > - dropwhile, takewhile; > I like how dropwhile and takewhile could be easily integrated with list comprehensions / generator expressions: [x for x in range(10) while x < 5] # takewhile [x for x in range(10) not while x < 5] # dropwhile [x for x in range(10) from x >= 5] # forward thinking dropwhile I believe this is almost plain english without creating new keywords. Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Mar 23 17:29:12 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 23 Mar 2016 22:29:12 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> Message-ID: <56F30AA8.1040607@mail.de> On 23.03.2016 20:06, Joao S. O. Bueno wrote: > As for adding different syntax for [empty use-case], > I am +0 > > For the idea of adding "except" clauses to > 'with', 'for' and 'while' (along with a "EmptyIteration" exception) > I am + 0.5 So it seems, you consider a new keyword for the sole purpose of "discovering never executed" less worthwhile than the "except" alternative. What is the reason for this? The potential extensibility? Readability? Btw. I can understand and confirm that the "ifempty" use-case is not as frequent as the length of this thread suggests. So, I would not mind a slightly longer syntax ("except EmptyIteration") as long as it handles ALL corner cases correctly. This actually is my main driver here. I know that there are two or three easy patterns that fail in some cases. If a new syntax would be proposed that cannot handle all of them, it's not worth it. However, the frequency by which people here and on python-list suggest not-always working solutions still gives me a reason to work on this. Best, Sven From srkunze at mail.de Wed Mar 23 17:30:15 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 23 Mar 2016 22:30:15 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <303B3A8E-5791-4E43-9CE6-5B679D8D02C6@selik.org> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <303B3A8E-5791-4E43-9CE6-5B679D8D02C6@selik.org> Message-ID: <56F30AE7.10101@mail.de> On 23.03.2016 19:19, Michael Selik wrote: >> is_empty = True >> for item in iterable: >> is_empty = False >> # do for each item >> if is_empty: >> # do exactly when iterable raises StopIteration on the first >> >> Maybe I'm missing something, but this seems to be an unnecessary addition. > It?s comparable in purpose to the ``else`` clause on a ``for`` or ``while``. It eliminates the need for a flag variable, like ``was_completed`` or ``is_empty``. It saves two lines of code, but more importantly avoids the accidental flip of True/False when setting the flag variable. Whether that warrants adding new syntax... I?m not convinced by the current proposals, but I can imagine finding a good keyword for the purpose. A keyword as good as ``else`` at least. Good point. Best, Sven From mike at selik.org Wed Mar 23 17:44:22 2016 From: mike at selik.org (Michael Selik) Date: Wed, 23 Mar 2016 21:44:22 +0000 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: On Wed, Mar 23, 2016, 5:27 PM Jo?o Bernardo wrote: > On Mon, Mar 21, 2016 at 8:06 PM, Michel Desmoulin < > desmoulinmichel at gmail.com> wrote: > >> Itertools is great, and some functions in it are more used than others: >> >> - islice; >> - chain; >> - dropwhile, takewhile; >> > > I like how dropwhile and takewhile could be easily integrated with list > comprehensions / generator expressions: > > [x for x in range(10) while x < 5] # takewhile > [x for x in range(10) not while x < 5] # dropwhile > [x for x in range(10) from x >= 5] # forward thinking dropwhile > > I believe this is almost plain english without creating new keywords. > They read well, except for the square brackets which to me imply consuming the entire iterator. Itertools takewhile will early exit, possibly leaving some values on the iterator. If these do consume the entire iterator, what's the difference with the ``if`` clause in a comprehension? -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Mar 23 17:44:54 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 23 Mar 2016 22:44:54 +0100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <3F5C43C1-83BB-42B9-AF6F-4E52BFA79B04@yahoo.com> Message-ID: fdatasync() on the file is not enough? Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Wed Mar 23 17:46:14 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 14:46:14 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> Message-ID: > On Mar 23, 2016, at 12:24, Michael Selik wrote: > >> On Wed, Mar 23, 2016 at 1:39 PM Andrew Barnert wrote: >> On Mar 23, 2016, at 10:13, Michael Selik wrote: >> > >> >> On Mar 23, 2016, at 6:37 AM, Chris Angelico wrote: >> >> >> >> On Wed, Mar 23, 2016 at 9:04 PM, Michel Desmoulin >> >> wrote: >> >>>> (Whether or not to make slice notation usable outside subscript >> >>>> operations could then be tackled as an independent question) >> >>>> >> >>>> For itertools.chain, it may make sense to simply promote it to the builtins. >> >>> >> >>> Same problem as with new keywords : it can be a problem with people >> >>> using chain as a var name. >> >> >> >> Less of a problem though - it'd only be an issue for people who (a) >> >> use chain as a var name, and (b) want to use the new shorthand. Their >> >> code will continue to work identically with itertools.chain (or not >> >> using it at all). With a new keyword, their code would instantly fail. >> > >> > I enjoy using ``chain`` as a variable name. It has many meanings outside of iteration tools. Three cheers for namespaces. >> >> As Chris just explained in the message you're replying to, this wouldn't affect you. I've used "vars" and "compile" and "dir" as local variables many times; as long as I don't need to call the builtin in the same scope, it's not a problem. The same would be true if you keep using "chain". Unless you want to chain iterables of your chains together, it'll never arise. > > Depends what you mean by "affect". It'll affect how I read my colleagues' code. I want to see ``from itertools import chain`` at the top of their modules if they're using chain in the sense of chaining iterables. The advantage of having a small set of builtins is that you know the entire set of builtins. If chain really is useful enough that it belongs as a builtin, you will very quickly adapt to reading that code, and it won't bother you or slow down your comprehension at all, any more than any other builtins do. Adding dozens of builtins would break that; adding one wouldn't. There's only room for a handful more builtins in the entire future life of Python, and the question of whether chain deserves to be one of them is a big question (and I suspect the answer is no). But I think it's the only real question, and adding more on top of it doesn't really help us get to the answer. > Some evidence... Agreed. And again, my own anecdotal experience is that chain.from_iterable is actually used (or sadly missed) more than chain itself, especially among novices, so I'm not advocating making chain a builtin unless someone proves me wrong on that. >> Still, I like adding chain (and/or flatten) to builtins a lot more than I like adding sequence behavior to some iterators, or adding a whole new kind of function-like slicing syntax to all iterables, or any of the other proposals here. > > I like the LazyList you wrote. If the need for operator-style slicing on iterators were great, I think we'd find a decent amount of usage of such a wrapper. Its absence is evidence that ``from itertools import islice`` is more pleasant. The one on my blog? I've never found a use for that in real code[0], hence why (IIRC) I never even put it on PyPI. But I use range all the time, and various other lazy sequences. The problem with LazyList isn't that it's lazy, but that it's a recursive cons-like structure, which is ultimately a different way to solve (mostly) the same set of problems that Python already has one obvious solution for (iterators), and an ill-fitting one at that (given that Python discourages unnecessary recursive algorithms). I've also written a more Python-style lazy list that wraps up caching an iterator in a list-like object. It's basically just a list and an iterator, along with a method to self.lst.extend(islice(self.it, index - len(self.lst)). The point of that is to show how easy it is to write, but how many different API decisions come up that could be reasonably resolved either way, so there's probably no one-size-fits-all design. Which implies that if there are apps that need something like that, they probably write it themselves. Anyway, if your point is that iterators having slicing would be a bad thing, I agree with you. But that doesn't necessarily mean that chain and islice as they are today is the best possible answer, just that it's better than adding operators to the iterator type (even if there were such a thing as "the iterator type", which there isn't). Using islice can still be clumsy, and I'm happy to see what alternatives or improvements people come up with. > As has been discussed many times, lazy sequences like range cause some confusion by providing both generator behavior and getitem/slicing. Perhaps it's better to keep the two varieties of iterables, lazy and non-lazy, more firmly separated by having ``lst[a:b]`` used exclusively for true sequences and ``islice(g, a, b)`` for generators. Definitely not. Even if breaking range and friends weren't a huge backward compat issue, it would weaken the language. And, meanwhile, you would gain absolutely nothing. You'd still have sets and dicts and third-party sorted trees and list iterators and itertools iterators and key views and so on, none of which are either sequences or generators. (Also, think about this parallel: do you want to say that dict key views shouldn't have set operators because they're lazy and therefore not "real sets"? If not, what's the difference?) The problem isn't that Python has things that are neither sequences nor generators, it's that the Python community has people who think it doesn't. Any time someone says range is (like) a generator, or any similar misleading myth, they need to be corrected with extreme prejudice before they set another batch of novices up for confusion. So, again: range does not provide anything like "generator behavior". It's not only repeatably iterable, it's also randomly-accessible, container-testable, reversible, and all the other things that are true of sequences. > Just yesterday I got frustrated by the difference of return types from subscripting bytes versus str while trying to keep my code 2/3 compatible: That's a completely separate problem. I think everyone agrees that there are design mistakes in the bytes class. As of 3.5, most of the ones that can be fixed have been, but some of them we're just unfortunately stuck with. None of those problems have to do with the iteration or sequence protocols. > A couple questions to help clarify the situation: > 1. Do you prefer ``a.union(b)`` or ``a | b``? When I'm doing stuff that's inherently mathematical-set-like, I use the operator. When I'm using sets purely as an optimization, I use the method.[1] But how does that relate to this thread? I think the implied assumption in the proposal is that people want to do "inherently sequence-like stuff" with iterators; if so, they _should_ want to spell it with brackets. I think the problem is that they're wrong to want that, not that they're trying to spell it wrong. > 2. Do you prefer ``datetime.now() + timedelta(days=5)`` or ``5.days.from_now``? Of course the former.[2] But the problem with "5.days" is nothing to do with this discussion. It's that it implies a bizarre ontology where natural numbers are things that know about timespans as intimately as they know about counting,[3] which is ridiculous.[4] If you really wanted numbers to know how to do "inherently day-like stuff", the Ruby way would make sense; you just shouldn't want that. --- [0]: I have found uses for it in comparing translated Lisp/ML/Haskell idioms to native Python ones, or in demonstrating things to other people, which is why I wrote it in the first place. [1]: Or sometimes even the classmethod set.union(a, b). In fact, sometimes I even write it out more explicitly--e.g., {x for x in a if x in b} in place of a & b emphasizes that I could easily change the {...} to [...] if I need to preserve order or duplicates in a. [2]: I like now() + days(5), or maybe even now() + 5*days, even more. But that's easy to add myself without modifying either int or timedelta, and without confusing my readers. [3]: Or, only slightly less accurately, it's like saying "5's days" instead of "5 days" in English. [4]: The Ruby answer to that is that writing an application is really about writing an app-specific language that lets you write the app itself a trivial single-page program, and in an OO context, that means extending types in app-specific ways. The number 5 in general doesn't know how to construct a timespan, but the number 5 in the context of a calendaring web application does. That's an arguably reasonable stance (Paul Graham has some great blog posts making the non-OO version of the same argument for Lisp/Arc), but it's blatantly obvious that it's directly opposed to the Python stance that the language should be simple enough that we can all immediately read each other's code. Python doesn't encourage indefinitely expanding core types for the same reason it doesn't encourage writing your own flow control statements. -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Wed Mar 23 17:54:19 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 23 Mar 2016 22:54:19 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: <56F3108B.9080802@gmail.com> Le 23/03/2016 22:26, Jo?o Bernardo a ?crit : > > On Mon, Mar 21, 2016 at 8:06 PM, Michel Desmoulin > > wrote: > > Itertools is great, and some functions in it are more used than others: > > - islice; > - chain; > - dropwhile, takewhile; > > > I like how dropwhile and takewhile could be easily integrated with list > comprehensions / generator expressions: > > [x for x in range(10) while x < 5] # takewhile > [x for x in range(10) not while x < 5] # dropwhile > [x for x in range(10) from x >= 5] # forward thinking dropwhile > > I believe this is almost plain english without creating new keywords. > > Regards > Nice. But if you need to combine both it's a long one liner. It could be combined with enumerate for islicing, but it would then end up huge. From srkunze at mail.de Wed Mar 23 17:55:32 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 23 Mar 2016 22:55:32 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: <56F310D4.9070503@mail.de> On 23.03.2016 22:44, Michael Selik wrote: > > > On Wed, Mar 23, 2016, 5:27 PM Jo?o Bernardo > wrote: > > On Mon, Mar 21, 2016 at 8:06 PM, Michel Desmoulin > > wrote: > > Itertools is great, and some functions in it are more used > than others: > > - islice; > - chain; > - dropwhile, takewhile; > > > I like how dropwhile and takewhile could be easily integrated with > list comprehensions / generator expressions: > > [x for x in range(10) while x < 5] # takewhile > [x for x in range(10) not while x < 5] # dropwhile > [x for x in range(10) from x >= 5] # forward thinking dropwhile > > I believe this is almost plain english without creating new keywords. > > > They read well, except for the square brackets which to me imply > consuming the entire iterator. Itertools takewhile will early exit, > possibly leaving some values on the iterator. If these do consume the > entire iterator, what's the difference with the ``if`` clause in a > comprehension? But isn't that their purpose? Not consuming. I think that would be easy to learn. +1 on this proposal. This could also prove useful to avoid for-else. :-) Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Mar 23 18:01:29 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 24 Mar 2016 09:01:29 +1100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> Message-ID: On Thu, Mar 24, 2016 at 8:46 AM, Andrew Barnert wrote: > The advantage of having a small set of builtins is that you know the entire > set of builtins. Python 3.6 has len(dir(builtins))==150, which is hardly small. Even removing the ones that don't begin with a lower-case letter (which removes a small number of special names like __package__, and a large number of exception classes) leaves 72. Can you name them all? ChrisA From abarnert at yahoo.com Wed Mar 23 18:11:21 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 15:11:21 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F2F9DC.5000203@mail.de> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F2F9DC.5000203@mail.de> Message-ID: <05E5C838-B3E7-482C-A771-686B325F0916@yahoo.com> On Mar 23, 2016, at 13:17, Sven R. Kunze wrote: > >> On 23.03.2016 00:56, Andrew Barnert via Python-ideas wrote: > >> (Especially since 90% of the time, when you need to do something special on empty, you explicitly have a sequence, not any iterable, so it's just "if not seq:".) > > Repeating that you think it will be a sequence will make it one. I've said that if be happy to see any counterexamples, where you really do need this with iterators. So far, nobody has provided one. Of course absence of proof isn't proof of absence, but "you can't prove that it's impossible anyone will ever need this" is not a good rationale for a language change. >> So really, this proposal is really just asking for syntactic sugar that complicates the language in exchange for making some already-understandable code a little more concise, which doesn't seem worth it. > > Did you even think what you just said? Almost everything in Python is "just syntactic sugar" compared to most other Turing-complete languages. > > To put it differently: every construct that abstracts away "goto" is "just syntactic sugar". There's a big difference between syntactic sugar that makes hard-to-follow code more readable, and syntactic sugar that only makes some already-understandable code a little more concise. The former may belong in the language, the latter very rarely does. You seem to think that there's no inherent cost to adding new features to a language. For example, you later say: > The only real reason against it so far: "it makes the language more complicated because I don't need it". Not entirely compelling but understandable. The more complicated the language is, the harder it is to keep it all in your head, to spot the control flow while skimming, to trace out the details when necessary, etc. Also, the more things you add, the more places there are for inconsistencies to creep in. (Of course it also makes the language harder to implement and maintain, but those are less important.) Only accepting changes that are actually worth the cost in increased complexity is a big part of what makes Python more readable than "dumping-ground" languages that have C-for, Python-for, while, do-while, until, repeat-until, and loop, some in both statement and expression variants, and some also writable in postfix form. >> The for loop bytecode would have to change to stash an "any values seen" flag somewhere such that if it sees StopIteration and hasn't seen any values, it converts that to an EmptyCollection. Or any of the other equivalents (e.g., the compiler could unroll the first PyIter_Next from loop from the rest of them to handle it specially). > > Something like that. > >> But this seems like it would add a lot of overhead and complexity to every loop whether desired or not. > > If the "for" does not have any empty-equivalent clauses, there is no need to introduce that overhead in the first place. Not true. The first implementation I suggested, putting EmptyCollection into every iterable, requires the overhead in every case. The second one, changing the way the existing bytecodes work, means making frame objects (or _something_) more complicated to enable stashing the flags, which affects every for loop. The third, unrolling the first PyObject_Iter, has to be done if there's any code that can inspect the current exception state, which can't be statically determined, so it has to be always done. If you have a _different_ implementation, I'm happy to hear it. I supplied all the versions I could think of because another critic (Stephen? I forget...) implied that what you wanted was impossible or incoherent, and it clearly isn't. But it may not be a good idea to let a critic of your idea come up with the implementation. :) > So we can conclude: > > 1) none overhead for regular "for"s > 2) less overhead for "for-ifempty" because it would be done in C and not in Python For which of the three implementations? I'm pretty sure all of them would have significant overhead. >>> If not, then how will this work? Is this a special kind of >>> exception-like process that *only* operates inside for loops? >>> >>> What will an explicit "raise NeverExecuted" do? >> Presumably that's the same question as what an explicit raise StopIteration does. Just as there's nothing stopping you from writing a __next__ method that raises StopIteration but then yields more values of called again, there's nothing stopping you from raising NeverExecuted pathologically, but you shouldn't do so. M >> >>>> So, independent of the initial "never executed loop body" use-case, one >>>> could also emulate the "else" clause by: >>>> >>>> for item in collection: >>>> # do for item >>>> except StopIteration: >>>> # do after the loop >>> That doesn't work, for two reasons: >>> >>> (1) Not all for-loops use iterators. The venerable old "sequence >>> protocol" is still supported for sequences that don't support __iter__. >>> So there may not be any StopIteration raised at all. >> I think there always is. >> >> IIRC, PyObject_Iter (the C API function used by iter and by for loops) actually constructs a sequence iterator object if the object doesn't have tp_iter (__iter__ for Python types) but does have tp_sequence (__getitem__ for Python types, but, e.g., dict has __getitem__ without having tp_sequence). And the for loop doesn't treat that sequence iterator any different from "real" iterators returned by __iter__; it just calls tp_next (__next__) until StopIteration. (And the "other half" of the old-style sequence protocol, that lets old-style sequences be reversed if they have a length, is similarly implemented by the C API underneath the reversed function.) >> >> I'm on my phone right now, so I can't double-check any of the details, but I'm 80% sure they're all at least pretty close... > > I think I would have to deal with the old protocol given the proposal is accepted. No, because, as I just explained, the old protocol is taken care of by wrapping old-style sequences in iterator objects, so as far as the for-loop code is concerned, they look identical to "new-style" iterables. >>> (2) Even if StopIteration is raised, the for-loop catches it (in a >>> manner of speaking) and consumes it. >>> >>> So to have this work, we would need to have the for-loop re-raise >>> StopIteration... but what happens if you don't include an except >>> StopIteration clause? Does every bare for-loop with no "except" now >>> print a traceback and halt processing? If not, why not? >> I think this could be made to work: a for loop without an except clause handles StopIteration the same as today (by jumping to the else clause), but one that does have one or more except clauses just treats it like a normal exception. >> >> Of course this would mean for/except/else is now legal but useless, which could be confusing ("why does my else clause no longer run when I add an 'except ValueError' clause?"). > > One could disallow "else" in case any "except" is defined. But that's just an extra rule to implement (and remember) for no real benefit. Why not just document that for/except/else is generally useless and shouldn't be written, and let linters flag it? >> More generally, I think the fact that for/except StopIteration is almost but not quite identical to plain for would be confusing more often than helpful. > > You bet how people think about "else". "So, 'else' is always executed after the for?" "Yes, but only when there is no 'break' executed in the 'for'" "... *thinking* ... okay ..." I don't understand your point here. Because there's already something in the language that you find confusing, that gives us free rein to add anything else to the language that people will find confusing? From abarnert at yahoo.com Wed Mar 23 18:17:22 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 15:17:22 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F2FBDB.9040507@mail.de> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F26B08.4060802@gmail.com> <2336A5F4-7A2E-4A1A-A16A-736F513BCE32@yahoo.com> <56F2FBDB.9040507@mail.de> Message-ID: On Mar 23, 2016, at 13:26, Sven R. Kunze wrote: > >> On 23.03.2016 16:53, Andrew Barnert via Python-ideas wrote: >> On Mar 23, 2016, at 03:08, Michel Desmoulin wrote: >> >>>> But I think it is a coherent proposal, even if it's not one I like. :) >>> And what do you think about adding except clauses to if ? with ? while ? >> Well, they don't have the problem that for has (for is already implicitly handling a specific exception in a specific way; none of them are), so they're coherent even without a solution to that problem. > > Could you describe what you mean by coherent? Someone earlier claimed that it's impossible to turn the vague proposal into any actual sensible semantics that could be implemented by Python. That would make it incoherent. But I don't believe it's true, so I tried to show how it could be defined well enough to be implementable. For the other three statements, I don't think that argument even comes up; I think the desired semantics are obvious enough that nobody can claim the proposal is incoherent. >> (With has the additional problem that it may not be immediately obvious on first glance whether the exit gets calls before or after the except clause, but if so, people can read the docs the first time they come across it and remember it after that.) >> >> But I also think they're even less necessary. They'd all be pure syntactic sugar for nesting the statement and a try/except, so we'd be making the language more complicated to learn and remember, > > I completely disagree here. > >> and encouraging more code that isn't backward compatible. > > What exactly is the problem with that? Everything discussed on python-ideas would be 3.6+ anyway. Yes, but it's still a cost, that has to be weighed against the benefits. Some new syntax (like "with" or "yield from") is so useful that it's worth encouraging new code to use the new features and not be backward compatible. But that doesn't mean every piece of syntax that might be nice in a brand-new language is so useful that it's worth doing that. It's a high bar, and just ignoring the bar doesn't help you meet it. >> That's not a huge cost, but the benefit isn't very big either. For _really_ short cases, I think we want except expressions (PEP 463); for longish code, there's nothing wrong with an explicit try; the range for which hiding an implicit try inside if and while would really improve things is small enough that I don't think the benefit outweighs the cost. > > That's your opinion here. And I find it interesting that others already have thought of the merging of try and if, for and while. It would definitely come in handy. > > But I understand from your previous responses that you'd rather use Assembler because it is has no unneeded syntactic sugar. ;-) Just kidding. You clearly don't understand my precious responses. If you don't see the difference between good syntactic sugar and useless syntactic sugar, then I can't explain to you why Python is better than assembler. :P >> But maybe some good examples of realistic 3-liners that are significantly improved by the change would convince me (and, more importantly, convince a majority of the others who are skeptical), so I'll keep an open mind. > > From what I can see regarding the if-except proposal, the improvement would be constant. So, either you like it (as I do) or you don't see them as necessary (as you do). > > There is nothing somebody can do to convince you that saving 1 line of code and 1 indentation level across 20 lines of code are good or bad. So you're refusing to provide examples because you're sure that no examples would convince anyone? In that case, I'm pretty sure your proposal is dead in the water, so I might as well stop responding. From mike at selik.org Wed Mar 23 18:18:26 2016 From: mike at selik.org (Michael Selik) Date: Wed, 23 Mar 2016 22:18:26 +0000 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F310D4.9070503@mail.de> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F310D4.9070503@mail.de> Message-ID: On Wed, Mar 23, 2016, 5:55 PM Sven R. Kunze wrote: > > > On 23.03.2016 22:44, Michael Selik wrote: > > > > On Wed, Mar 23, 2016, 5:27 PM Jo?o Bernardo < > jbvsmo at gmail.com> wrote: > >> On Mon, Mar 21, 2016 at 8:06 PM, Michel Desmoulin < >> desmoulinmichel at gmail.com> wrote: >> >>> Itertools is great, and some functions in it are more used than others: >>> >>> - islice; >>> - chain; >>> - dropwhile, takewhile; >>> >> >> I like how dropwhile and takewhile could be easily integrated with list >> comprehensions / generator expressions: >> >> [x for x in range(10) while x < 5] # takewhile >> [x for x in range(10) not while x < 5] # dropwhile >> [x for x in range(10) from x >= 5] # forward thinking dropwhile >> >> I believe this is almost plain english without creating new keywords. >> > > They read well, except for the square brackets which to me imply consuming > the entire iterator. Itertools takewhile will early exit, possibly leaving > some values on the iterator. If these do consume the entire iterator, > what's the difference with the ``if`` clause in a comprehension? > > > But isn't that their purpose? Not consuming. I think that would be easy to > learn. +1 on this proposal. > > This could also prove useful to avoid for-else. :-) > After the execution of the list comprehension using "while" if the condition triggered halfway, would there be anything left in the original iterator? If it was a generator expression, we'd say "Yes" immediately. As a list comprehension, I'm not sure. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Mar 23 18:20:30 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 23 Mar 2016 23:20:30 +0100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <56F21CF9.7000005@gmail.com> References: <56F21CF9.7000005@gmail.com> Message-ID: <56F316AE.7090302@mail.de> On 23.03.2016 05:35, Thomas Nyberg wrote: > Not to knock too much on R, but this kind of thing was one of the > things I always really hated about that language. Separating the > operations of converting a string and then > writing/printing/sending/etc. that string is just so much more flexible. and inconvenient. I am no social scientist but I share the sentiment. bash: echo "my string" >> my_file python: with open('my_file', 'w') as f: f.write('my string') Hmm. Python actually is best at avoiding special characters but regarding files, it's still on the level of C. > It reminds me of the many objects I worked with in R that had natural > print methods, but did not have natural conversions to strings (at > least nowhere near as easy as it should have been). There were so many > times that having the operations as separate steps (e.g. having print > just call call __str__() or whatever) is so much better. > Maybe, I am missed that but when did Nick talk about non-string to be written to a file? > I understand the argument that it feels more natural to certain > people, but it really incentivizes bad design imo. I disagree. It incentivizes good design as it forces you to prepare 1) your file name and 2) your data properly and 3) you don't "invent" self-proclaimed-but-not-really atomic writes. I can tell you from my experience with several aged Python developers that they regularly fail to implement atomic file operations. Just saying. > Definitely a big -1 from me. +1 from me. Best, Sven From tjreedy at udel.edu Wed Mar 23 18:25:55 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 23 Mar 2016 18:25:55 -0400 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F19E29.7090700@stoneleaf.us> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F19E29.7090700@stoneleaf.us> Message-ID: On 3/22/2016 3:34 PM, Ethan Furman wrote: > On 03/22/2016 11:49 AM, Terry Reedy wrote: >> I have never, ever, claimed that. Initially empty and empty after all >> items have been processed are different. > > This is the heart of the issue: *initially empty* and *empty after all > items are exhausted* are different, from the perspective of a particular loop. > but for/else and while/else treat them the same. Which is to say, each test is a test of current state, independent of history. It does not matter if the iterater was created empty, was emptied by previous code out or inside the current function, or emptied by the current loop. People who want to condition on history should record it. The idiom presented previously is a form of recording history. item = marker = object() # could be earlier in function body for item in iterable: ... else: if item = marker: # started loop empty ... -- Terry Jan Reedy From alexander.belopolsky at gmail.com Wed Mar 23 18:30:31 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 23 Mar 2016 18:30:31 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <56F316AE.7090302@mail.de> References: <56F21CF9.7000005@gmail.com> <56F316AE.7090302@mail.de> Message-ID: On Wed, Mar 23, 2016 at 6:20 PM, Sven R. Kunze wrote: > > bash: echo "my string" >> my_file > python: with open('my_file', 'w') as f: f.write('my string') ... > > I can tell you from my experience with several aged Python developers that > they regularly fail to implement atomic file operations. Just saying. What make you think your bash example implements an atomic write? It actually performs an append and therefore not equivalent to the python code that followed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Mar 23 18:32:38 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 23 Mar 2016 18:32:38 -0400 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <93075DD5-2590-47EB-A903-1769D7FFCAF0@selik.org> <56F14181.2000701@gmail.com> <56F16C64.6000606@gmail.com> <56F18636.40100@gmail.com> Message-ID: On 3/22/2016 4:03 PM, Andrew Barnert via Python-ideas wrote: > Range is a sequence, just like list and tuple, not an iterator, like > generator and list_iterator. So it supports slicing because sequences > generally support slicing. Not because it's so convenient and > expressive that it's worth making a special case, just because it > follows the normal rules. > > It's amazing to me that so many people still describe xrange/3.x > range as an iterator or generator, and then draw further incorrect > lessons from that. An example I recently ran across is http://python4kids.brendanscott.com/2016/03/19/python-for-kids-python-3-project-2/ Python4Kids uses range-as-list for the 2.x version. For the 3.x version, he is agonizing over explaing 'range-as-generator'. -- Terry Jan Reedy From abarnert at yahoo.com Wed Mar 23 18:38:02 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 15:38:02 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> Message-ID: <1F60A754-D9BA-4494-89F8-6F047B6A3F2D@yahoo.com> On Mar 23, 2016, at 15:01, Chris Angelico wrote: > >> On Thu, Mar 24, 2016 at 8:46 AM, Andrew Barnert wrote: >> The advantage of having a small set of builtins is that you know the entire >> set of builtins. > > Python 3.6 has len(dir(builtins))==150, which is hardly small. Even > removing the ones that don't begin with a lower-case letter (which > removes a small number of special names like __package__, and a large > number of exception classes) leaves 72. Can you name them all? Can you name all 25000 words of English that you know? There are 68 builtins listed in a nice table at the top of the builtin functions chapter in the docs. Take a look at that table. They're all (except maybe format and iter[1]) functions[2] that you can immediately recognize them when reading them, and recall when they're relevant to code you're writing, including know exactly what their interface is and what they do. If you see "any()" or "int()" in code, you don't need to turn to the docs to look up what it does. If you need to convert something to a debug string representation, you know to call "repr()". The fact that you might leave out "any", "int", or "repr" in a pop quiz demanding that you list all 68 of them in alphabetical order doesn't mean you don't know them. You certainly can't say the same is true for all functions and methods in the stdlib. --- [1]: I think everyone knows iter _exists_, but not everyone knows about its two-argument form. [2]: ... or types... From srkunze at mail.de Wed Mar 23 18:42:32 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 23 Mar 2016 23:42:32 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F26B08.4060802@gmail.com> <2336A5F4-7A2E-4A1A-A16A-736F513BCE32@yahoo.com> <56F2FBDB.9040507@mail.de> Message-ID: <56F31BD8.90101@mail.de> On 23.03.2016 23:17, Andrew Barnert wrote: > You clearly don't understand my precious responses. If you don't see the difference between good syntactic sugar and useless syntactic sugar, then I can't explain to you why Python is better than assembler. :P Good syntactic sugar is sweet. What means good to you? ;-) > So you're refusing to provide examples because you're sure that no > examples would convince anyone? No, I didn't say that. I don't need examples mostly because I see them each day. But because you asked, here you are: class MyClass: def mymethod(self): try: if my_condition: dosomething() elif another_condition(): makesomething() andanythingelse() else: butdontforgetaboutthis() except SomeException: pass finally: return None Would turn into: class MyClass: def mymethod(self): if my_condition: dosomething() elif another_condition(): makesomething() andanythingelse() else: butdontforgetaboutthis() except SomeException: pass finally: return None 1 line saved and 7 lines with 1 indentation level less (maybe even related to PEP8 and 80 chars line length). You might call it useless, I call it good. Regarding Python syntax, we are constantly going into territory where there is nothing much left to improve. The big "good syntactic sugar" is already done. However, I don't think we should stop here. 10 smaller improvements are as good one big one. Btw. I don't believe we could improve all Python code out there with this change, however when you are in such a situation you really appreciate it if you can do it. Especially when the surrounding source code already blows your mind; you are grateful for every single simplification (less lines, less indentation). I don't know what Python code you maintain but this would our lives easier. Best, Sven From desmoulinmichel at gmail.com Wed Mar 23 18:55:14 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 23 Mar 2016 23:55:14 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> Message-ID: <56F31ED2.2090100@gmail.com> Le 23/03/2016 22:26, Jo?o Bernardo a ?crit : > > On Mon, Mar 21, 2016 at 8:06 PM, Michel Desmoulin > > wrote: > > Itertools is great, and some functions in it are more used than others: > > - islice; > - chain; > - dropwhile, takewhile; > > > I like how dropwhile and takewhile could be easily integrated with list > comprehensions / generator expressions: > > [x for x in range(10) while x < 5] # takewhile > [x for x in range(10) not while x < 5] # dropwhile > [x for x in range(10) from x >= 5] # forward thinking dropwhile > > I believe this is almost plain english without creating new keywords. > > Regards > Nice. But if you need to combine both it's a long one liner. It could be combined with enumerate for islicing, but it would then end up huge. From rosuav at gmail.com Wed Mar 23 19:00:39 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 24 Mar 2016 10:00:39 +1100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <1F60A754-D9BA-4494-89F8-6F047B6A3F2D@yahoo.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> <1F60A754-D9BA-4494-89F8-6F047B6A3F2D@yahoo.com> Message-ID: On Thu, Mar 24, 2016 at 9:38 AM, Andrew Barnert wrote: > On Mar 23, 2016, at 15:01, Chris Angelico wrote: >> >>> On Thu, Mar 24, 2016 at 8:46 AM, Andrew Barnert wrote: >>> The advantage of having a small set of builtins is that you know the entire >>> set of builtins. >> >> Python 3.6 has len(dir(builtins))==150, which is hardly small. Even >> removing the ones that don't begin with a lower-case letter (which >> removes a small number of special names like __package__, and a large >> number of exception classes) leaves 72. Can you name them all? > > Can you name all 25000 words of English that you know? > > There are 68 builtins listed in a nice table at the top of the builtin functions chapter in the docs. Take a look at that table. They're all (except maybe format and iter[1]) functions[2] that you can immediately recognize them when reading them, and recall when they're relevant to code you're writing, including know exactly what their interface is and what they do. If you see "any()" or "int()" in code, you don't need to turn to the docs to look up what it does. If you need to convert something to a debug string representation, you know to call "repr()". The fact that you might leave out "any", "int", or "repr" in a pop quiz demanding that you list all 68 of them in alphabetical order doesn't mean you don't know them. > > You certainly can't say the same is true for all functions and methods in the stdlib. Fair enough, naming them all is a poor definition of "know". But the flip side (knowing what one means when you meet it) is also a poor definition. The real trick is knowing, when you're faced with a problem, what the solution is; and while most people here will be able to do that, I'm not sure that every Python programmer can. Some builtins are well known, like int and len; others, not so much - zip, hash, divmod. And then there are some (compile, bytearray, callable) that, if they were elsewhere in the stdlib, would barely change people's code. How many non-experts know exactly which function decorators are builtins and which are imported from functools? If you have to look them up, (a) there's not a lot of difference between built-in and stdlib, and (b) the builtins are already a bit crowded. My conclusion: itertools.chain is just as discoverable as builtins.chain, so the only real benefit would be playing around at the interactive prompt, which you can improve on with an auto script. -0.5 on making chain a builtin. ChrisA From breamoreboy at yahoo.co.uk Wed Mar 23 19:09:54 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 23 Mar 2016 23:09:54 +0000 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> <1F60A754-D9BA-4494-89F8-6F047B6A3F2D@yahoo.com> Message-ID: On 23/03/2016 23:00, Chris Angelico wrote: > On Thu, Mar 24, 2016 at 9:38 AM, Andrew Barnert wrote: >> On Mar 23, 2016, at 15:01, Chris Angelico wrote: >>> >>>> On Thu, Mar 24, 2016 at 8:46 AM, Andrew Barnert wrote: >>>> The advantage of having a small set of builtins is that you know the entire >>>> set of builtins. >>> >>> Python 3.6 has len(dir(builtins))==150, which is hardly small. Even >>> removing the ones that don't begin with a lower-case letter (which >>> removes a small number of special names like __package__, and a large >>> number of exception classes) leaves 72. Can you name them all? >> >> Can you name all 25000 words of English that you know? >> >> There are 68 builtins listed in a nice table at the top of the builtin functions chapter in the docs. Take a look at that table. They're all (except maybe format and iter[1]) functions[2] that you can immediately recognize them when reading them, and recall when they're relevant to code you're writing, including know exactly what their interface is and what they do. If you see "any()" or "int()" in code, you don't need to turn to the docs to look up what it does. If you need to convert something to a debug string representation, you know to call "repr()". The fact that you might leave out "any", "int", or "repr" in a pop quiz demanding that you list all 68 of them in alphabetical order doesn't mean you don't know them. >> >> You certainly can't say the same is true for all functions and methods in the stdlib. > > Fair enough, naming them all is a poor definition of "know". But the > flip side (knowing what one means when you meet it) is also a poor > definition. The real trick is knowing, when you're faced with a > problem, what the solution is; and while most people here will be able > to do that, I'm not sure that every Python programmer can. Some > builtins are well known, like int and len; others, not so much - zip, > hash, divmod. And then there are some (compile, bytearray, callable) > that, if they were elsewhere in the stdlib, would barely change > people's code. How many non-experts know exactly which function > decorators are builtins and which are imported from functools? If you > have to look them up, (a) there's not a lot of difference between > built-in and stdlib, and (b) the builtins are already a bit crowded. > > My conclusion: itertools.chain is just as discoverable as > builtins.chain, so the only real benefit would be playing around at > the interactive prompt, which you can improve on with an auto script. > -0.5 on making chain a builtin. > > ChrisA My conclusion is RTFM. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From rosuav at gmail.com Wed Mar 23 19:15:09 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 24 Mar 2016 10:15:09 +1100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> <1F60A754-D9BA-4494-89F8-6F047B6A3F2D@yahoo.com> Message-ID: On Thu, Mar 24, 2016 at 10:09 AM, Mark Lawrence wrote: > My conclusion is RTFM. Exactly. Once you have to RTFM, what's the difference between builtin and stdlib? ChrisA From desmoulinmichel at gmail.com Wed Mar 23 19:17:13 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Thu, 24 Mar 2016 00:17:13 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> <1F60A754-D9BA-4494-89F8-6F047B6A3F2D@yahoo.com> Message-ID: <56F323F9.1050700@gmail.com> Le 24/03/2016 00:15, Chris Angelico a ?crit : > On Thu, Mar 24, 2016 at 10:09 AM, Mark Lawrence wrote: >> My conclusion is RTFM. > > Exactly. Once you have to RTFM, what's the difference between builtin > and stdlib? The import in every modules and in the shell. But I agree, chain is used, I used it one time last year. Not enough to justify a built-in. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From abarnert at yahoo.com Wed Mar 23 19:18:07 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 23 Mar 2016 16:18:07 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> <1F60A754-D9BA-4494-89F8-6F047B6A3F2D@yahoo.com> Message-ID: On Mar 23, 2016, at 16:00, Chris Angelico wrote: > >> On Thu, Mar 24, 2016 at 9:38 AM, Andrew Barnert wrote: >>> On Mar 23, 2016, at 15:01, Chris Angelico wrote: >>> >>>> On Thu, Mar 24, 2016 at 8:46 AM, Andrew Barnert wrote: >>>> The advantage of having a small set of builtins is that you know the entire >>>> set of builtins. >>> >>> Python 3.6 has len(dir(builtins))==150, which is hardly small. Even >>> removing the ones that don't begin with a lower-case letter (which >>> removes a small number of special names like __package__, and a large >>> number of exception classes) leaves 72. Can you name them all? >> >> Can you name all 25000 words of English that you know? >> >> There are 68 builtins listed in a nice table at the top of the builtin functions chapter in the docs. Take a look at that table. They're all (except maybe format and iter[1]) functions[2] that you can immediately recognize them when reading them, and recall when they're relevant to code you're writing, including know exactly what their interface is and what they do. If you see "any()" or "int()" in code, you don't need to turn to the docs to look up what it does. If you need to convert something to a debug string representation, you know to call "repr()". The fact that you might leave out "any", "int", or "repr" in a pop quiz demanding that you list all 68 of them in alphabetical order doesn't mean you don't know them. >> >> You certainly can't say the same is true for all functions and methods in the stdlib. > > Fair enough, naming them all is a poor definition of "know". But the > flip side (knowing what one means when you meet it) is also a poor > definition. The real trick is knowing, when you're faced with a > problem, what the solution is; Well, readability is at least as important as writability. But yes, writability is also important, which is why I covered it: 'If you need to convert something to a debug string representation, you know to call "repr()".' > and while most people here will be able > to do that, I'm not sure that every Python programmer can. Sure, there's a certain level of competence below which you can't say people know all the builtins. There's also a certain level of competence below which you can't say people know every kind of statement in the syntax. But the point is that the level is low enough to be reasonably achievable. It's not only the hardcore experts that know almost all of the builtins. If we tripled the number of builtins, I don't think that would be true anymore. (Secondarily, the alphabetical chart in the docs is good enough at 68; at 200, we'd almost certainly need to reorganize it to make it useful.) > Some > builtins are well known, like int and len; others, not so much - zip, > hash, divmod. And then there are some (compile, bytearray, callable) > that, if they were elsewhere in the stdlib, would barely change > people's code. I don't think anyone would be horribly bothered by moving compile outside of builtins, but it has to be there for bootstrapping reasons. (Maybe that isn't true anymore, but it apparently was in 3.0.) For callable, that actually _was_ removed from builtins in 3.0, and added back pretty late in the 3.2 cycle because, as it turns out, people really did miss it more than you'd expect a priori. But anyway, the presence of 2 or 3 or even 10 functions out of those 68 that maybe shouldn't be there isn't license to add 100 more. If anything, it means we have to be a tiny bit more careful what we add there in the future. > My conclusion: itertools.chain is just as discoverable as > builtins.chain, so the only real benefit would be playing around at > the interactive prompt, which you can improve on with an auto script. > -0.5 on making chain a builtin. I don't think it's _just_ as discoverable. But I do think it's discoverable _enough_. (Plus, as I already mentioned, I think chain.from_iterable is more of a problem than chain, and moving chain to builtins wouldn't help that problem enough to be worth doing.) So I'm also -0.5. From srkunze at mail.de Wed Mar 23 19:19:53 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 24 Mar 2016 00:19:53 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <05E5C838-B3E7-482C-A771-686B325F0916@yahoo.com> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F2F9DC.5000203@mail.de> <05E5C838-B3E7-482C-A771-686B325F0916@yahoo.com> Message-ID: <56F32499.10202@mail.de> On 23.03.2016 23:11, Andrew Barnert wrote: > I've said that if be happy to see any counterexamples, where you really do need this with iterators. 1) you conveniently snipped the example I've given: somebody wrote functionality assuming a sequence (as innocently as most of the contributors to python-list and python-ideas); later the input sequence will be changed to an input iterator; "if len(collection)" may hide errors then 2) the colleague who basically initiated the whole discussion needed to implement some sort of "on-the-fly" converter pulling specific items from a third-party system A and pushing the converted items to another third-party system B. In case none items are there to be transferred and converted, a placeholder item still needs be pushed to system B. [yes, that is how somebody can make money; customers sometimes have interesting requirements] > The more complicated the language is, the harder it is to keep it all in your head, to spot the control flow while skimming, to trace out the details when necessary, etc. And that is exactly why I and other like to see such an addition. It's not just the idea of a single person. It reduces the amount of lines of code and (for the correct implementation of all corner cases) it has less indentation and is presumably faster. > Only accepting changes that are actually worth the cost in increased complexity is a big part of what makes Python more readable than "dumping-ground" languages that have C-for, Python-for, while, do-while, until, repeat-until, and loop, some in both statement and expression variants, and some also writable in postfix form. I understand that and want to keep it that way as you do. It seems that languages like Django and jinja2, which are 10x more concerned about feature reduction than Python, have {% empty %} or {% else %}. So, Python would not be the first language to support this feature. >>> The for loop bytecode would have to change to stash an "any values seen" flag somewhere such that if it sees StopIteration and hasn't seen any values, it converts that to an EmptyCollection. Or any of the other equivalents (e.g., the compiler could unroll the first PyIter_Next from loop from the rest of them to handle it specially). >> Something like that. >> >>> But this seems like it would add a lot of overhead and complexity to every loop whether desired or not. >> If the "for" does not have any empty-equivalent clauses, there is no need to introduce that overhead in the first place. > Not true. The first implementation I suggested, putting EmptyCollection into every iterable, requires the overhead in every case. The second one, changing the way the existing bytecodes work, means making frame objects (or _something_) more complicated to enable stashing the flags, which affects every for loop. The third, unrolling the first PyObject_Iter, has to be done if there's any code that can inspect the current exception state, which can't be statically determined, so it has to be always done. > > If you have a _different_ implementation, I'm happy to hear it. I supplied all the versions I could think of because another critic (Stephen? I forget...) implied that what you wanted was impossible or incoherent, and it clearly isn't. But it may not be a good idea to let a critic of your idea come up with the implementation. :) Not sure if that can be easily done, but as I understand it, there would be two alternative bytecode implementations available. Depending on whether the "for" has an "empty"-equivalent clause, the "emptified/except/slow for" is used. If the "for" doesn't have such clause, the "original" "for" is used. So, depending on the actual code, the bytecode is different. Does this makes sense? >> So we can conclude: >> >> 1) none overhead for regular "for"s >> 2) less overhead for "for-ifempty" because it would be done in C and not in Python > [snip] > No, because, as I just explained, the old protocol is taken care of by wrapping old-style sequences in iterator objects, so as far as the for-loop code is concerned, they look identical to "new-style" iterables. Oh, then I had interpreted what you said the wrong way. Thanks for explaining again. > But that's just an extra rule to implement (and remember) for no real benefit. Why not just document that for/except/else is generally useless and shouldn't be written, and let linters flag it? Good idea! > Because there's already something in the language that you find confusing, that gives us free rein to add anything else to the language that people will find confusing? I am open for suggestions. My argument here is that "except StopIteration" is not better/worse than "else" but the former is extensible. Best, Sven From srkunze at mail.de Wed Mar 23 19:27:29 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 24 Mar 2016 00:27:29 +0100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <56F21CF9.7000005@gmail.com> <56F316AE.7090302@mail.de> Message-ID: <56F32661.3060907@mail.de> On 23.03.2016 23:30, Alexander Belopolsky wrote: > > On Wed, Mar 23, 2016 at 6:20 PM, Sven R. Kunze > wrote: > > > bash: echo "my string" >> my_file > python: with open('my_file', 'w') as f: f.write('my string') > > ... > > > I can tell you from my experience with several aged Python > developers that they regularly fail to implement atomic file > operations. Just saying. > > > What make you think your bash example implements an atomic write? It > actually performs an append and therefore not equivalent to the python > code that followed. What makes you think that I think my bash example implements an atomic write? ;-) My point was the simplicity of the bash command. Make it Python as simple AND atomic here (with file handling) and people will drop bash immediately. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at yahoo.co.uk Wed Mar 23 19:27:11 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 23 Mar 2016 23:27:11 +0000 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56EEE82A.6050505@mail.de> References: <56EEE82A.6050505@mail.de> Message-ID: On 20/03/2016 18:12, Sven R. Kunze wrote: > Hi Python-ideas, > > this is a follow up from an thread of Python-List: > https://mail.python.org/pipermail/python-list/2016-March/705205.html > > What do you think about the following proposal? > > *Abstract* > Control structures such as "for", "while" and "try" feature some various > additional clauses being executed under certain conditions. This > proposal adds another extra clause to "for" and "while" in order to > provide an easy way to discover whether the body of a loop as never been > executed at all. > > *Motivation** > *Coming from the Web application development, we periodically can see a > benefit of a hypothetical "empty" clause of Python's for loop like this: I don't do hypothetical. Practicality beats purity. So having followed this thread I do not believe that language changes are needed. So why not add a paragraph to this https://docs.python.org/3/tutorial/controlflow.html#break-and-continue-statements-and-else-clauses-on-loops giving the cleanest solution from this thread for the way to write this Pythonically. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From breamoreboy at yahoo.co.uk Wed Mar 23 19:33:58 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 23 Mar 2016 23:33:58 +0000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <56F32661.3060907@mail.de> References: <56F21CF9.7000005@gmail.com> <56F316AE.7090302@mail.de> <56F32661.3060907@mail.de> Message-ID: On 23/03/2016 23:27, Sven R. Kunze wrote: > On 23.03.2016 23:30, Alexander Belopolsky wrote: >> >> On Wed, Mar 23, 2016 at 6:20 PM, Sven R. Kunze > > wrote: >> >> bash: echo "my string" >> my_file >> python: with open('my_file', 'w') as f: f.write('my string') >> >> ... >> >> I can tell you from my experience with several aged Python >> developers that they regularly fail to implement atomic file >> operations. Just saying. >> >> What make you think your bash example implements an atomic write? It >> actually performs an append and therefore not equivalent to the python >> code that followed. > > What makes you think that I think my bash example implements an atomic > write? ;-) > > My point was the simplicity of the bash command. Make it Python as > simple AND atomic here (with file handling) and people will drop bash > immediately. > > In four decades I've never once felt the need for a one-liner that writes a string to a file. How do you drop something that you've never used and have no interest in? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From srkunze at mail.de Wed Mar 23 19:42:50 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 24 Mar 2016 00:42:50 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F310D4.9070503@mail.de> Message-ID: <56F329FA.9080403@mail.de> On 23.03.2016 23:18, Michael Selik wrote: > > > On Wed, Mar 23, 2016, 5:55 PM Sven R. Kunze > wrote: > > > > On 23.03.2016 22:44, Michael Selik wrote: >> >> >> On Wed, Mar 23, 2016, 5:27 PM Jo?o Bernardo > > wrote: >> >> On Mon, Mar 21, 2016 at 8:06 PM, Michel Desmoulin >> > > wrote: >> >> Itertools is great, and some functions in it are more >> used than others: >> >> - islice; >> - chain; >> - dropwhile, takewhile; >> >> >> I like how dropwhile and takewhile could be easily integrated >> with list comprehensions / generator expressions: >> >> [x for x in range(10) while x < 5] # takewhile >> [x for x in range(10) not while x < 5] # dropwhile >> [x for x in range(10) from x >= 5] # forward thinking >> dropwhile >> >> I believe this is almost plain english without creating new >> keywords. >> >> >> They read well, except for the square brackets which to me imply >> consuming the entire iterator. Itertools takewhile will early >> exit, possibly leaving some values on the iterator. If these do >> consume the entire iterator, what's the difference with the >> ``if`` clause in a comprehension? > > But isn't that their purpose? Not consuming. I think that would be > easy to learn. +1 on this proposal. > > This could also prove useful to avoid for-else. :-) > > > After the execution of the list comprehension using "while" if the > condition triggered halfway, would there be anything left in the > original iterator? If it was a generator expression, we'd say "Yes" > immediately. As a list comprehension, I'm not sure. Let's define it this way. Feeling better? ;-) Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbvsmo at gmail.com Wed Mar 23 20:08:38 2016 From: jbvsmo at gmail.com (=?UTF-8?Q?Jo=C3=A3o_Bernardo?=) Date: Wed, 23 Mar 2016 21:08:38 -0300 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F310D4.9070503@mail.de> Message-ID: Jo?o Bernardo On Wed, Mar 23, 2016 at 7:18 PM, Michael Selik wrote: > >>> [x for x in range(10) while x < 5] # takewhile >>> [x for x in range(10) not while x < 5] # dropwhile >>> [x for x in range(10) from x >= 5] # forward thinking dropwhile >>> >>> I believe this is almost plain english without creating new keywords. >>> >> >> They read well, except for the square brackets which to me imply >> consuming the entire iterator. Itertools takewhile will early exit, >> possibly leaving some values on the iterator. If these do consume the >> entire iterator, what's the difference with the ``if`` clause in a >> comprehension? >> >> >> But isn't that their purpose? Not consuming. I think that would be easy >> to learn. +1 on this proposal. >> >> This could also prove useful to avoid for-else. :-) >> > > After the execution of the list comprehension using "while" if the > condition triggered halfway, would there be anything left in the original > iterator? If it was a generator expression, we'd say "Yes" immediately. > As a list comprehension, I'm not sure. > Dropwhile alone would consume the entire iterator for obvious reasons. Takewhile will stop when necessary, otherwise why implement it? You could nest them and still be readable: [x for x in range(10) from x > 3 while x < 7] Still quite readable. If this were an iterator like "foo = iter(range(10))", there will be items left in the end. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Mar 23 20:16:11 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 23 Mar 2016 17:16:11 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F323F9.1050700@gmail.com> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> <1F60A754-D9BA-4494-89F8-6F047B6A3F2D@yahoo.com> <56F323F9.1050700@gmail.com> Message-ID: <56F331CB.7000805@stoneleaf.us> On 03/23/2016 04:17 PM, Michel Desmoulin wrote: > Le 24/03/2016 00:15, Chris Angelico a ?crit : >> Exactly. Once you have to RTFM, what's the difference between builtin >> and stdlib? > > The import in every modules and in the shell. You know, that's a good point! It just has one problem: I use datetime.datetime and datetime.date a heck of a lot more than chain and chain.from_iterable; and os.getuid and os.setuid! and sys.argv, and collections.OrderedDict, and sys.version_info, and sys.modules, and most of the things in logging... So all those things should be builtin first. Then we can talk about chain. Chaining-all-my-favorite-imports-ly yrs, -- ~Ethan~ From steve at pearwood.info Wed Mar 23 23:10:30 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 24 Mar 2016 14:10:30 +1100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <3F5C43C1-83BB-42B9-AF6F-4E52BFA79B04@yahoo.com> Message-ID: <20160324031030.GL12526@ando.pearwood.info> On Wed, Mar 23, 2016 at 10:14:07AM -0700, Nathaniel Smith wrote: > POSIX doesn't guarantee anything whatsoever over power loss. Individual > filesystem implementations make somewhat stronger guarantees, but it's a > mess: > http://danluu.com/file-consistency/ [...] > I don't want to get sucked into a long debate about this; it's entirely > likely that adding something like that original recipe to the stdlib would > be an improvement, so long as it had *very* detailed docs explaining the > exact tradeoffs made. All I want to do is raise a cautionary flag that such > an effort would need to tread carefully :-) And then there are file media which lie to you, and return from a fsync before actually syncing, because that makes their benchmarks look good. I've seen file corruption on USB sticks that do this, including some otherwise "respectable" brands. Worst case I ever saw was a USB stick that (fortunately) had a blinking light to show when it was writing. It continuing writing for *eight minutes* (I timed it) after returning from fsync and the OS had unmounted the stick and it was nominally safe to remove. I'm told that some hard drives will do the same, although I've never knowingly seen it. -- Steve From rosuav at gmail.com Wed Mar 23 23:34:12 2016 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 24 Mar 2016 14:34:12 +1100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <20160324031030.GL12526@ando.pearwood.info> References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <3F5C43C1-83BB-42B9-AF6F-4E52BFA79B04@yahoo.com> <20160324031030.GL12526@ando.pearwood.info> Message-ID: On Thu, Mar 24, 2016 at 2:10 PM, Steven D'Aprano wrote: > And then there are file media which lie to you, and return from a fsync > before actually syncing, because that makes their benchmarks look good. > I've seen file corruption on USB sticks that do this, including some > otherwise "respectable" brands. Worst case I ever saw was a USB stick > that (fortunately) had a blinking light to show when it was writing. It > continuing writing for *eight minutes* (I timed it) after returning from > fsync and the OS had unmounted the stick and it was nominally safe to > remove. > > I'm told that some hard drives will do the same, although I've never > knowingly seen it. Yes, but not as much of it; usually with hard drives, it's a small amount of on-board cache, which tends to improve overall performance by batching writes. That can be disabled by the paranoid (which you should be). More commonly, SSDs do the same bare-faced lying that you noted with USB sticks. There was a time when I took the paranoid approach of eschewing SSDs even on my client computers because they're not trustworthy. I'm beginning to shift toward an even more paranoid approach of not trusting _any_ local storage - all my important data gets pushed to a remote server anyway (maybe GitHub, maybe something on the LAN, whatever), so it won't matter if a power failure corrupts stuff. But for a server with a database on it, you kinda need to solve this problem three ways at once, rather than depending on just one. ChrisA From njs at pobox.com Thu Mar 24 00:31:00 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 23 Mar 2016 21:31:00 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <20160324031030.GL12526@ando.pearwood.info> References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <3F5C43C1-83BB-42B9-AF6F-4E52BFA79B04@yahoo.com> <20160324031030.GL12526@ando.pearwood.info> Message-ID: On Wed, Mar 23, 2016 at 8:10 PM, Steven D'Aprano wrote: > On Wed, Mar 23, 2016 at 10:14:07AM -0700, Nathaniel Smith wrote: > >> POSIX doesn't guarantee anything whatsoever over power loss. Individual >> filesystem implementations make somewhat stronger guarantees, but it's a >> mess: >> http://danluu.com/file-consistency/ > [...] >> I don't want to get sucked into a long debate about this; it's entirely >> likely that adding something like that original recipe to the stdlib would >> be an improvement, so long as it had *very* detailed docs explaining the >> exact tradeoffs made. All I want to do is raise a cautionary flag that such >> an effort would need to tread carefully :-) > > And then there are file media which lie to you, and return from a fsync > before actually syncing, because that makes their benchmarks look good. > I've seen file corruption on USB sticks that do this, including some > otherwise "respectable" brands. Worst case I ever saw was a USB stick > that (fortunately) had a blinking light to show when it was writing. It > continuing writing for *eight minutes* (I timed it) after returning from > fsync and the OS had unmounted the stick and it was nominally safe to > remove. > > I'm told that some hard drives will do the same, although I've never > knowingly seen it. Not just USB sticks and hard drives -- whole operating systems too. Ones you might have heard of! Like... OS X. Their fsync() syscall is intentionally designed and documented to provide absolutely zero guarantees of anything. (From the man page: "[after calling fsync], if the drive loses power or the OS crashes, the application may find that only some or none of their data was written [...] This is not a theoretical edge case. This scenario is easily reproduced with real world workloads".) A real fsync call is still available, it's just hidden in fcntl. While we're speaking of it, Python's os.fsync on OS X should probably call that fcntl instead of fsync. Right now os.fsync is effectively a no-op on OS X. (Apparently this is https://bugs.python.org/issue11877 but that issue seems to have petered out years ago in confusion.) -n -- Nathaniel J. Smith -- https://vorpus.org From sjoerdjob at sjec.nl Thu Mar 24 02:39:55 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Thu, 24 Mar 2016 07:39:55 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F2F9DC.5000203@mail.de> References: <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F2F9DC.5000203@mail.de> Message-ID: <20160324063955.GA1890@sjoerdjob.com> Let me start by saying I'm not against the proposal, and think it could make a lot of sense when properly implemented. However, I would like to specificially respond to the following. On Wed, Mar 23, 2016 at 09:17:32PM +0100, Sven R. Kunze wrote: > On 23.03.2016 00:56, Andrew Barnert via Python-ideas wrote: > >The only question is whether any of them are obvious enough (even > >for novices). If not, we could argue whether any proposed change > >would be significantly _more_ obvious. And, if so, then the > >question is whether the change is worth the cost. But I think what > >we have is already obvious enough. > > It was reasonable for both Django and for jinja2 to add this. People > actually asked for it https://github.com/mitsuhiko/jinja2/issues/77 It would be good to remember that both Django and jinja2 are templating languages, and as such they impose different constraints on the user, as well as the sentiment that templates should be (nearly) devoid of all logic. Due to those constraints, adding an `empty` in the template language makes a lot more sense than in a general-purpose programming language. Not to mention the fact that representing 'no results' as an empty page is bad UX, but representing 'no results' from an API as an empty [list, generator, ...] is good design. Again, I like the proposal for the edge-cases it tries to solve (and sometimes I have seen use-cases where it makes sense), but please make sure you have better arguments and examples than 'look at Django and jinja2'. From sjoerdjob at sjec.nl Thu Mar 24 02:54:19 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Thu, 24 Mar 2016 07:54:19 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F13875.3030804@mail.de> References: <56EEE82A.6050505@mail.de> <20160321014433.GC12526@ando.pearwood.info> <56F13875.3030804@mail.de> Message-ID: <20160324065419.GB1890@sjoerdjob.com> On Tue, Mar 22, 2016 at 01:20:05PM +0100, Sven R. Kunze wrote: > Now, that you mention it. This might give birth to a completely > different idea. What about? > > > for item in collection: > # do for item > except EmptyCollection: > # do if collection is empty > except StopIteration: > # do after the loop > > Which basically merges try and for? What worries me about this is that the next question is going to be: Why not have this on `with` suites, `if` suites, and so on. I've already seen remarks symmilar to that as replies. Especially because of your remark 'Which basically merges try and for', which adds to my worries. Consider the following: for item in [0, 1, 2]: raise StopIteration except StopIteration: # do after the loop # Also after a `raise` inside the loop body? >From your statement 'merges try and for', it would seem as you now want the exception to be caught in this case as well. Now, maybe you did not intend it as such, and really intended to only catch the exceptions raised by the iterable itself. But that was not quite clear to begin with. Because of this (possible) confusion, I'd really suggest slapping on another keyword to the `for` statement over allowing `except` clauses with limited use. Another reason is that for item in [0, 1, 2]: # do something except StopIteration: pass would do exactly the same as the `for` loop without the `except` clause, (due to the `pass`). Now this does not break the principle of 'one obvious way to do it', because for Pythonistas with experience, the `except` is superfluous. But would a novice know? (Assuming he started not by reading the Python docs, but by working on code written by another programmer)? Yes, RTFM, but more often you learn by reading other peoples code. From srkunze at mail.de Thu Mar 24 04:55:09 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 24 Mar 2016 09:55:09 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <20160324065419.GB1890@sjoerdjob.com> References: <56EEE82A.6050505@mail.de> <20160321014433.GC12526@ando.pearwood.info> <56F13875.3030804@mail.de> <20160324065419.GB1890@sjoerdjob.com> Message-ID: <56F3AB6D.6090700@mail.de> On 24.03.2016 07:54, Sjoerd Job Postmus wrote: > What worries me about this is that the next question is going to be: Why > not have this on `with` suites, `if` suites, and so on. I've already > seen remarks symmilar to that as replies. My initial motivation is to support the "empty" use-case. I am not attached to any proposal as long as that is achieved. For one reason, I dislike to explain how to emulate the missing feature to others (especially when Django and jinja2 supports it) and for another I found usage for it for myself recently. [offtopic] I am curious now. What is the problem with providing exception handling to more than just try? Is it just because it complicates the language which you are used to? Or is there something really problematic I miss here? > Especially because of your > remark 'Which basically merges try and for', which adds to my worries. This was to make people understand it more easily. :-) > Consider the following: > > for item in [0, 1, 2]: > raise StopIteration > except StopIteration: > # do after the loop > # Also after a `raise` inside the loop body? > > From your statement 'merges try and for', it would seem as you now want > the exception to be caught in this case as well. That's interesting. Yes, that could be confusing. But what can we do? I cannot believe that we run out of ideas just for something that some simpler languages can handle easily. :-/ > Now, maybe you did not > intend it as such, and really intended to only catch the exceptions > raised by the iterable itself. But that was not quite clear to begin > with. Because of this (possible) confusion, I'd really suggest slapping > on another keyword to the `for` statement over allowing `except` clauses > with limited use. > > Another reason is that > > for item in [0, 1, 2]: > # do something > except StopIteration: > pass > > would do exactly the same as the `for` loop without the `except` clause, > (due to the `pass`). Now this does not break the principle of 'one > obvious way to do it', because for Pythonistas with experience, the > `except` is superfluous. But would a novice know? (Assuming he started > not by reading the Python docs, but by working on code written by > another programmer)? Yes, RTFM, but more often you learn by reading > other peoples code. Noted but I wouldn't give too much weight to this issue. As others noted "else" is equally confusing. Best, Sven From guettliml at thomas-guettler.de Thu Mar 24 09:54:38 2016 From: guettliml at thomas-guettler.de (=?UTF-8?Q?Thomas_G=c3=bcttler?=) Date: Thu, 24 Mar 2016 14:54:38 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56EEE82A.6050505@mail.de> References: <56EEE82A.6050505@mail.de> Message-ID: <56F3F19E.4090604@thomas-guettler.de> Am 20.03.2016 um 19:12 schrieb Sven R. Kunze: > Hi Python-ideas, > > .. > for item in my_iterator: > # do per item > empty: Yes, this looks good. What about this? {{{ for item in my_iterator: # do per item on empty: # this code gets executed if iterator was empty on break: # this code gets executed if the iteration was left by a "break" on notempty: # ... }}} Is there a case which I have forgotten? ... I have no clue how to implement this :-) Regards, Thomas -- Thomas Guettler http://www.thomas-guettler.de/ From desmoulinmichel at gmail.com Thu Mar 24 10:56:59 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Thu, 24 Mar 2016 15:56:59 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F331CB.7000805@stoneleaf.us> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F07E78.9020309@gmail.com> <56F26A13.2030006@gmail.com> <1B3394C2-9307-43C2-9D99-581A0D58CFC2@yahoo.com> <1F60A754-D9BA-4494-89F8-6F047B6A3F2D@yahoo.com> <56F323F9.1050700@gmail.com> <56F331CB.7000805@stoneleaf.us> Message-ID: <56F4003B.1020700@gmail.com> Le 24/03/2016 01:16, Ethan Furman a ?crit : > On 03/23/2016 04:17 PM, Michel Desmoulin wrote: >> Le 24/03/2016 00:15, Chris Angelico a ?crit : > >>> Exactly. Once you have to RTFM, what's the difference between builtin >>> and stdlib? >> >> The import in every modules and in the shell. > > You know, that's a good point! It just has one problem: I use > datetime.datetime and datetime.date a heck of a lot more than chain and > chain.from_iterable; and os.getuid and os.setuid! and sys.argv, and > collections.OrderedDict, and sys.version_info, and sys.modules, and most > of the things in logging... > > So all those things should be builtin first. Then we can talk about chain. > > Chaining-all-my-favorite-imports-ly yrs, > I agree on that. > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From k7hoven at gmail.com Thu Mar 24 11:02:16 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 24 Mar 2016 17:02:16 +0200 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: Message-ID: On Wed, Mar 23, 2016 at 5:06 AM, Nick Eubank wrote: [...] > Apparently there's something like this in the Path library > (https://docs.python.org/dev/library/pathlib.html#pathlib.Path.write_text) , > but I suspect most people have no idea about that method (I've been doing > python for years and this has always been a personal frustration, and I've > asked several others for better options and no one had one to offer), and it > seems like it would make much more sense as a string method. If someone has > a string they want to save to disk, I can't imagine anyone looking in the > Path library. > Maybe you know this, but Path.read_text and Path.write_text (and the bytes versions) were introduced in Python 3.5, released only half a year ago, so there has not really been time for their adoption. Path("mytextfile.txt").write_text(...) Path("mytextfile.txt").read_text() Besides being very readable and convenient, one of the nice things about this is that it helps avoid code like open("mytextfile.txt").write(text) which you see all the time, even if the file will be left open for some time after this in non-ref-counting implementations of Python. So, as mentioned before by others, all we need is the better interplay of pathlib with the rest of the stdlib and that people will learn about it. Personally, I now always use `pathlib` instead of `open` when I don't need to care about compatibility with old Python versions. Maybe some day, Path could be in builtins like open is now? - Koos From Nikolaus at rath.org Thu Mar 24 12:49:50 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Thu, 24 Mar 2016 09:49:50 -0700 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: (Alexander Belopolsky's message of "Wed, 23 Mar 2016 17:01:00 -0400") References: <56EB1502.5010102@stoneleaf.us> <56EB23DD.5050908@python.org> <20160317233254.GG8022@ando.pearwood.info> <87io0goq04.fsf@vostro.rath.org> <20160321002945.GT8022@ando.pearwood.info> <87poul40lf.fsf@thinkpad.rath.org> <87mvpo52zy.fsf@thinkpad.rath.org> Message-ID: <87y497izwx.fsf@thinkpad.rath.org> On Mar 23 2016, Alexander Belopolsky wrote: > On Wed, Mar 23, 2016 at 4:54 PM, Nikolaus Rath wrote: > >> >> Yeah, but the "others" don't benefit from it and don't care about it. >> > >> > What's the basis for this assertion? >> >> The responses on this list so far. > > The first two responses to OP were +1's. That's enough to invalidate your > assertion. I understood them as "I'd like to have citation information as a builtin", not "I would benefit from Python being cited". Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From steve at pearwood.info Thu Mar 24 13:12:54 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 25 Mar 2016 04:12:54 +1100 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <87y497izwx.fsf@thinkpad.rath.org> References: <87io0goq04.fsf@vostro.rath.org> <20160321002945.GT8022@ando.pearwood.info> <87poul40lf.fsf@thinkpad.rath.org> <87mvpo52zy.fsf@thinkpad.rath.org> <87y497izwx.fsf@thinkpad.rath.org> Message-ID: <20160324171254.GM12526@ando.pearwood.info> On Thu, Mar 24, 2016 at 09:49:50AM -0700, Nikolaus Rath wrote: > On Mar 23 2016, Alexander Belopolsky wrote: > > On Wed, Mar 23, 2016 at 4:54 PM, Nikolaus Rath wrote: > > > >> >> Yeah, but the "others" don't benefit from it and don't care about it. > >> > > >> > What's the basis for this assertion? > >> > >> The responses on this list so far. > > > > The first two responses to OP were +1's. That's enough to invalidate your > > assertion. > > I understood them as "I'd like to have citation information as a > builtin", not "I would benefit from Python being cited". This argument is a waste of everybody's time. Regardless of whether you, or anyone else, thinks that Python needs to be cited, or whether we get any benefit from that, *people will cite it no matter what we think*. If you read my original post in this thread, I link to examples of people asking how to cite Python, because they intend to cite it. When a scientist who has written a paper gets told by the journal editors to cite the software used, then regardless of what you or I think about the matter, the author will cite. If somebody doing their thesis is told by their supervisor to cite, they will cite. What you and I think about this is irrelevant. All we can do is help them, or not, according to whether we feel like being helpful or not. My vote is to be helpful, but I am satisfied that a "Citation" page in the documentation is enough for now. -- Steve From srkunze at mail.de Thu Mar 24 13:44:51 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 24 Mar 2016 18:44:51 +0100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F3F19E.4090604@thomas-guettler.de> References: <56EEE82A.6050505@mail.de> <56F3F19E.4090604@thomas-guettler.de> Message-ID: <56F42793.6070503@mail.de> On 24.03.2016 14:54, Thomas G?ttler wrote: > for item in my_iterator: > # do per item > on empty: > # this code gets executed if iterator was empty > on break: > # this code gets executed if the iteration was left by a "break" > on notempty: > # ... > Hmm, interesting. "on" would indeed be a distinguishing keyword to "except". So, "except" handles exceptions and "on" handles internal control flow (the iter protocol). Nice idea actually. > }}} > > Is there a case which I have forgotten? "on notbreak" I suppose. I think the downside here is that we would need to register another keyword "on". Is this used as a normal variable often? Not sure if "on empty" can be registered as a single keyword since it contains whitespace. Best, Sven From srkunze at mail.de Thu Mar 24 13:46:40 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 24 Mar 2016 18:46:40 +0100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: Message-ID: <56F42800.3050004@mail.de> On 24.03.2016 16:02, Koos Zevenhoven wrote: > So, as mentioned before by others, all we need is the better interplay > of pathlib with the rest of the stdlib and that people will learn > about it. Personally, I now always use `pathlib` instead of `open` > when I don't need to care about compatibility with old Python > versions. Maybe some day, Path could be in builtins like open is now? How do you handle Path <-> str conversion? Just living with the extra conversion step and don't bother with this ugliness? Best, Sven From stephen at xemacs.org Thu Mar 24 14:02:33 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 25 Mar 2016 03:02:33 +0900 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: <22260.11193.357854.285199@turnbull.sk.tsukuba.ac.jp> Andrew Barnert via Python-ideas writes: > I do it all the time in other languages when dealing with smallish > files. Python's very nice file-object concept, slant toward > iterator-based processing, and amazingly consistent ecosystem means > that the same issues don't apply, so I'd rarely do the same > thing. But for users migrating to Python from another language, or > using Python occasionally while primarily using another language, I > can see it being a lot more attractive. It occurs to me that we already have a perfectly appropriate builtin for the purpose anyway: print. Add a filename= keyword argument, and make use of both file= and filename= in the same call an error. If that's not the right answer, I don't see how str.to_file() can possibly be better. From stephen at xemacs.org Thu Mar 24 14:04:22 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 25 Mar 2016 03:04:22 +0900 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <05E5C838-B3E7-482C-A771-686B325F0916@yahoo.com> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F2F9DC.5000203@mail.de> <05E5C838-B3E7-482C-A771-686B325F0916@yahoo.com> Message-ID: <22260.11302.562367.583959@turnbull.sk.tsukuba.ac.jp> Andrew Barnert via Python-ideas writes: > If you have a _different_ implementation, I'm happy to hear it. I > supplied all the versions I could think of because another critic > (Stephen? I forget...) I presented a good try at a one-extra-line pure Python implementation (which is still good enough that I'm -1 on syntax, and waiting for a compelling use case for adding "for ... except"). It was Chris Angelico who shot a hole in it with locals().values(). > implied that what you wanted was impossible or incoherent, and it > clearly isn't. I don't think there was ever a claim that it was incoherent/impossible or even inefficient at C-level. The point of the mildly obscure[1] item = sentinel idiom was to show that syntax isn't needed to avoid changing the loop body. I don't even know if Chris A would disagree that I succeeded well enough. Footnotes: [1] I know that several people found the item = _sentinel approach unreadable. Which surprised me: I'm usually the one who takes hours to understand the TOOWTDI one-liner. Maybe my newt-ness got bettah! From ethan at stoneleaf.us Thu Mar 24 14:28:00 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 24 Mar 2016 11:28:00 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F19E29.7090700@stoneleaf.us> Message-ID: <56F431B0.9040106@stoneleaf.us> On 03/23/2016 03:25 PM, Terry Reedy wrote: > On 3/22/2016 3:34 PM, Ethan Furman wrote: >> This is the heart of the issue: *initially empty* and *empty after all >> items are exhausted* are different, > > from the perspective of a particular loop. From the perspective of many, many (most?) people. I use for loops and while loops *alot*, and I can count the times I have wanted to use the `else` clause in the meaning of "all my tests failed, so do this block" on the fingers of one hand. >> but for/else and while/else treat them the same. > > Which is to say, each test is a test of current state, independent of > history. By the way, I appreciate your explanations -- they help make a little more sense out of it -- but the behaviour of `for/else` and `while/else` is definitely a learned exception to the general rule of "truthy -> this thing is something" / "falsey -> there is nothing here". -- ~Ethan~ From p.f.moore at gmail.com Thu Mar 24 14:30:38 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 24 Mar 2016 18:30:38 +0000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <22260.11193.357854.285199@turnbull.sk.tsukuba.ac.jp> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <22260.11193.357854.285199@turnbull.sk.tsukuba.ac.jp> Message-ID: On 24 March 2016 at 18:02, Stephen J. Turnbull wrote: >> I do it all the time in other languages when dealing with smallish > > files. Python's very nice file-object concept, slant toward > > iterator-based processing, and amazingly consistent ecosystem means > > that the same issues don't apply, so I'd rarely do the same > > thing. But for users migrating to Python from another language, or > > using Python occasionally while primarily using another language, I > > can see it being a lot more attractive. > > It occurs to me that we already have a perfectly appropriate builtin > for the purpose anyway: print. Add a filename= keyword argument, and > make use of both file= and filename= in the same call an error. If > that's not the right answer, I don't see how str.to_file() can > possibly be better. Well, with print("A string", filename='foo.txt') what encoding should be used? What if the user wants control of the encoding (on Windows, it's far from uncommon to want UTF-8 instead of the system default, for example)? What about error handlers? The same questions apply for str.to_file() of course, although having extra arguments is likely to be less of an issue there. And yes, at some point the answer is "if the convenience function doesn't suit your needs, you can fall back to explicit file handling". But the trick is finding the *right* balance of convenience. And having a "simple" way of writing a string to a file that doesn't at least hint to the user that they should be thinking about encodings seems to me like something of an attractive nuisance. Paul From rosuav at gmail.com Thu Mar 24 16:06:45 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 25 Mar 2016 07:06:45 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <22260.11302.562367.583959@turnbull.sk.tsukuba.ac.jp> References: <56EEE82A.6050505@mail.de> <22255.43600.444180.152613@turnbull.sk.tsukuba.ac.jp> <22257.24630.142582.931259@turnbull.sk.tsukuba.ac.jp> <56F17F14.9090306@mail.de> <20160322232136.GJ12526@ando.pearwood.info> <4F65A80C-8313-4E02-ACB2-F94CAFD14126@yahoo.com> <56F2F9DC.5000203@mail.de> <05E5C838-B3E7-482C-A771-686B325F0916@yahoo.com> <22260.11302.562367.583959@turnbull.sk.tsukuba.ac.jp> Message-ID: On Fri, Mar 25, 2016 at 5:04 AM, Stephen J. Turnbull wrote: > The point of the mildly obscure[1] item = sentinel idiom was to show > that syntax isn't needed to avoid changing the loop body. I don't > even know if Chris A would disagree that I succeeded well enough. You succeeded well enough. I just like finding the obscure cases :) ChrisA From abarnert at yahoo.com Thu Mar 24 16:36:17 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 24 Mar 2016 13:36:17 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <22260.11193.357854.285199@turnbull.sk.tsukuba.ac.jp> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <22260.11193.357854.285199@turnbull.sk.tsukuba.ac.jp> Message-ID: On Mar 24, 2016, at 11:02, Stephen J. Turnbull wrote: > > Andrew Barnert via Python-ideas writes: > >> I do it all the time in other languages when dealing with smallish >> files. Python's very nice file-object concept, slant toward >> iterator-based processing, and amazingly consistent ecosystem means >> that the same issues don't apply, so I'd rarely do the same >> thing. But for users migrating to Python from another language, or >> using Python occasionally while primarily using another language, I >> can see it being a lot more attractive. > > It occurs to me that we already have a perfectly appropriate builtin > for the purpose anyway: print. Add a filename= keyword argument, and > make use of both file= and filename= in the same call an error. If > that's not the right answer, I don't see how str.to_file() can > possibly be better. That's not a bad idea. However, it's obvious how to extend str.to_file() to reads (str.from_file() classmethod), binary data (bytes.to_file()), atomic writes (str.to_file(path, atomic=True), non-default encodings (str.to_file(path, encoding='Latin-1', errors='replace')). With print, those are all less obvious. Maybe input() for reads, but how can it handle bytes? And, while you could toss on more parameters to print() for some of these, it already has a lot, and adding new parameters that are only legal if filename is specified seems like it'll be make the docs even harder to follow. Of course if the answers are "we don't want any of those", then being more limited is a good thing, not a problem. :) From k7hoven at gmail.com Thu Mar 24 17:06:59 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 24 Mar 2016 23:06:59 +0200 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <56F42800.3050004@mail.de> References: <56F42800.3050004@mail.de> Message-ID: On Thu, Mar 24, 2016 at 7:46 PM, Sven R. Kunze wrote: > On 24.03.2016 16:02, Koos Zevenhoven wrote: >> >> So, as mentioned before by others, all we need is the better interplay >> of pathlib with the rest of the stdlib and that people will learn >> about it. Personally, I now always use `pathlib` instead of `open` >> when I don't need to care about compatibility with old Python >> versions. Maybe some day, Path could be in builtins like open is now? > > How do you handle Path <-> str conversion? Just living with the extra > conversion step and don't bother with this ugliness? For me, it's most often str->Path, meaning things like Path("file.txt") or directory = Path("/path/to/some/directory") with (directory / "file.txt").open() as f: #do something with f with (directory / "otherfile.txt").open() as f: #do something with f Then, somewhat less often, I need to give a str as an argument to something. I then need an additional str(...) around the Path object. That just feels stupid and makes me start wishing Path was a subclass of str, and/or that Path was a builtin. Or even better, that you could do p"filename.txt", which would give you a Path string object. Has this been discussed? - Koos -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Mar 24 17:20:27 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 24 Mar 2016 14:20:27 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax Message-ID: I haven't seen enough in this conversation to persuade me that we need more built ins, but: I recently taught Intro to Python using py3 for the first time recently, and really noticed that while the most obvious difference (at the beginning level) is that print is a function now, that is only annoying, but didn't confuse anyone at all. (and Unicode because a non-issue -- yeah!) However, where I found things changed in a conceptual way is the movement from sequences to iterables in many places: zip() dict.keys(), etc.... I found that my approach of focusing a lot on sequences -- you can iterate over them, you can slice them, you can.... seems out of date now, and I think lead to confusion as folks got a bit more advanced. Py2 was all about sequences -- py3 is all about iterables, and I think I need to start teaching it that way -- start with iterables, and introduce sequences as a special case of iterables, rather than starting from sequences, and introducing iterables as a more advanced topic. I understand the focus on iterables -- for performance reasons if nothing else, but I think it does, in fact, make the language more complex. and I think we should think about making the language more iterable-focused. This certainly comes up with this discussion -- if Python is all about iterables, then some of itertools should be more discoverable and obvious. And it came up in another recent thred about a mechanism for doing something after a for loop that didn't loop -- in that case, for sequences, the idiom is obvious: if seq: do_the_for_loop else: do_somethign else since the loop wont have run. But when you plug in an arbitrary iterable into that, it doesn't work, and there is no easy, obvious, and robust idiom to replace that with. I don't know that that particular issue needs to be solved, but it makes my point -- there is much to be done to make python really about iterables, rather than sequences. For my part, I would like to see iterables look more like sequences -- I know it's going to be inefficient in many cases to index an iterable, but it i really less efficient that wrapping list() around it? One tiny example: I often need to parse text files, which I used to do with code like: for line in the_file.readlines(): .... then, particulary when debugging, I could do: for line in the_file.readlines()[:10]: .... and just work with the first ten lines.... but now that file objects ar iterable, I can do: for line in the_file: .... much nicer! but it breaks when I want to debug and try to do: for line in the_file[:10]: ... arrgg! files are not indexable!. (and by the way, not only for testing, but also when you really do want the next ten lines from the file -- maybe it's a header, or...) Sure, I have plenty of ways to work around this, but frankly, they are all a bit ugly, even if trivial. So maybe there should be some ability to index / slice iterables? But aside from that -- just the idea that looking at how to make iterable a more "natural" part of the language is a good thing. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu Mar 24 17:22:15 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 24 Mar 2016 17:22:15 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <56F42800.3050004@mail.de> Message-ID: On Thu, Mar 24, 2016 at 5:06 PM, Koos Zevenhoven wrote: > Then, somewhat less often, I need to give a str as an argument to > something. I then need an additional str(...) around the Path object. > That just feels stupid and makes me start wishing Path was a subclass of > str, and/or that Path was a builtin. ... Has this been discussed? > Yes, right in the PEP 428 where pathlib was proposed. [1] [1]: https://www.python.org/dev/peps/pep-0428/#id29 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Thu Mar 24 17:25:55 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Thu, 24 Mar 2016 17:25:55 -0400 Subject: [Python-ideas] Add citation() to site.py In-Reply-To: <20160324171254.GM12526@ando.pearwood.info> References: <87io0goq04.fsf@vostro.rath.org> <20160321002945.GT8022@ando.pearwood.info> <87poul40lf.fsf@thinkpad.rath.org> <87mvpo52zy.fsf@thinkpad.rath.org> <87y497izwx.fsf@thinkpad.rath.org> <20160324171254.GM12526@ando.pearwood.info> Message-ID: <56F45B63.1070602@sdamon.com> On 3/24/2016 13:12, Steven D'Aprano wrote: > My vote is to be helpful, but I am satisfied that a "Citation" page in > the documentation is enough for now. +1 to docs (which is think was close to the consensus for this thread anyways.) From chris.barker at noaa.gov Thu Mar 24 17:33:15 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 24 Mar 2016 14:33:15 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: <56F431B0.9040106@stoneleaf.us> References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F19E29.7090700@stoneleaf.us> <56F431B0.9040106@stoneleaf.us> Message-ID: >From the perspective of many, many (most?) people. I use for loops and > while loops *alot*, and I can count the times I have wanted to use the > `else` clause in the meaning of "all my tests failed, so do this block" on > the fingers of one hand. I"ve only used for .. else a handful of times, and honestly, probably forgot to use it when it would have been the right way to do something far more times :-) but I can't think of a SINGLE time when I wanted to know that the loop didn't run because it's iterable was empty. Which is my way of saying: I think adding something would make this use case more clear and easy to do, but it's a rare enough use case that it's not worth it. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Thu Mar 24 17:41:37 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 24 Mar 2016 23:41:37 +0200 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <56F42800.3050004@mail.de> Message-ID: On Thu, Mar 24, 2016 at 11:22 PM, Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > On Thu, Mar 24, 2016 at 5:06 PM, Koos Zevenhoven > wrote: > >> Then, somewhat less often, I need to give a str as an argument to >> something. I then need an additional str(...) around the Path object. >> That just feels stupid and makes me start wishing Path was a subclass of >> str, and/or that Path was a builtin. ... Has this been discussed? >> > > Yes, right in the PEP 428 where pathlib was proposed. [1] > > [1]: https://www.python.org/dev/peps/pep-0428/#id29 > Thanks. The pep indeed mentions that str (or other builtins like tuple) are not subclassed in pathlib, but couldn't find the explanation why. Well, it would add all kinds of attributes, many of which will just be confusing and not useful for paths. - Koos -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Mar 24 17:43:28 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 24 Mar 2016 14:43:28 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: semi OT, but: On Tue, Mar 22, 2016 at 8:33 PM, Andrew Barnert via Python-ideas < python-ideas at python.org> wrote: > Assuming that the long-term goal is to get everyone using pathlib, rather > than just abandoning it as a stdlib fossil of a nice idea that didn't work > out, +inf on that -- we really do need to get the entire stdlib to work with Path objects! -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Thu Mar 24 17:43:58 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 24 Mar 2016 14:43:58 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <22260.11193.357854.285199@turnbull.sk.tsukuba.ac.jp> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <22260.11193.357854.285199@turnbull.sk.tsukuba.ac.jp> Message-ID: <67C371E4-27E4-4FD6-A095-F2E64D5022C9@yahoo.com> I did a quick survey of other languages, cross-language frameworks, etc. to see what they offer, and I was surprised. To look at my quick survey (or correct me! I wasn't super-careful here, and may easily have taken misleading StackOverflow posts or the like at face value): https://goo.gl/9bJcAf Anyway, the original proposal in this thread is very close to Cocoa, but that turns out to be pretty uncommon. Most frameworks use free functions, not methods on either string or path objects. Most don't offer an option for atomic writes or exclusive locks--although some do them whether you want it or not. Most have an option for appending. All but one either let you specify the encoding or don't do test. Almost none of those were what I expected. From chris.barker at noaa.gov Thu Mar 24 18:07:30 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 24 Mar 2016 15:07:30 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: > It is a rare situation when you would want to write just a single string > to a file. > > > I do it all the time in other languages when dealing with smallish files. > Python's very nice file-object concept, slant toward iterator-based > processing, and amazingly consistent ecosystem means that the same issues > don't apply, so I'd rarely do the same thing. > also, Python's strings are immutable, so we really don't want to encourage people to build up a big string in memory anyway. and what's wrong with: open(a_path, 'w').write(the_string) short, simple one-liner. OK, non cPython garbage collection may leave that file open and dangling, but we're talking the quicky scripting data analysis type user -- the script will terminate soon enough. BTW, numpy does offer one stop ways to save and load arrays to a file, binary or text -- and that's a lot more useful than a simple string. especially the reading. oh, and for reading: string = open(path).read() I really don't get the point of all this. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Thu Mar 24 18:14:49 2016 From: mike at selik.org (Michael Selik) Date: Thu, 24 Mar 2016 22:14:49 +0000 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F19E29.7090700@stoneleaf.us> <56F431B0.9040106@stoneleaf.us> Message-ID: On Thu, Mar 24, 2016 at 5:34 PM Chris Barker wrote: > From the perspective of many, many (most?) people. I use for loops and >> while loops *alot*, and I can count the times I have wanted to use the >> `else` clause in the meaning of "all my tests failed, so do this block" on >> the fingers of one hand. > > > I"ve only used for .. else a handful of times, and honestly, probably > forgot to use it when it would have been the right way to do something far > more times :-) > > but I can't think of a SINGLE time when I wanted to know that the loop > didn't run because it's iterable was empty. > More specifically, a time when the loop didn't run because the iterable was empty AND the iterable didn't have a len to make that a simple check. Which suddenly gives me an idea... class Peeking: def __init__(self, iterator): self.it = iter(iterator) self.sentinel = object() self.buf = next(self.it, self.sentinel) def __iter__(self): return self def __next__(self): if self.buf is self.sentinel: raise StopIteration value = self.buf self.buf = next(self.it, self.sentinel) return value def __bool__(self): return self.buf is not self.sentinel If you find yourself in a situation where you want to check empty as if your iterator was a sequence (ie. len(seq) == 0 means False), give this wrapper a shot just before you do the loop. >>> bool(Peeking(range(0))) False >>> bool(Peeking(range(1))) True We could even make it a context manager to monkey-patch a buffering inside the context for the bool check, then undo the patch on leaving the context to stop buffering. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Thu Mar 24 18:21:45 2016 From: mike at selik.org (Michael Selik) Date: Thu, 24 Mar 2016 22:21:45 +0000 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F19E29.7090700@stoneleaf.us> <56F431B0.9040106@stoneleaf.us> Message-ID: On Thu, Mar 24, 2016 at 6:14 PM Michael Selik wrote: > On Thu, Mar 24, 2016 at 5:34 PM Chris Barker > wrote: > >> From the perspective of many, many (most?) people. I use for loops and >>> while loops *alot*, and I can count the times I have wanted to use the >>> `else` clause in the meaning of "all my tests failed, so do this block" on >>> the fingers of one hand. >> >> >> I"ve only used for .. else a handful of times, and honestly, probably >> forgot to use it when it would have been the right way to do something far >> more times :-) >> >> but I can't think of a SINGLE time when I wanted to know that the loop >> didn't run because it's iterable was empty. >> > > More specifically, a time when the loop didn't run because the iterable > was empty AND the iterable didn't have a len to make that a simple check. > > Which suddenly gives me an idea... > > class Peeking: > > def __init__(self, iterator): > self.it = iter(iterator) > self.sentinel = object() > self.buf = next(self.it, self.sentinel) > > def __iter__(self): > return self > > def __next__(self): > if self.buf is self.sentinel: > raise StopIteration > value = self.buf > self.buf = next(self.it, self.sentinel) > return value > > def __bool__(self): > return self.buf is not self.sentinel > > > If you find yourself in a situation where you want to check empty as if > your iterator was a sequence (ie. len(seq) == 0 means False), give this > wrapper a shot just before you do the loop. > > >>> bool(Peeking(range(0))) > False > >>> bool(Peeking(range(1))) > True > > We could even make it a context manager to monkey-patch a buffering inside > the context for the bool check, then undo the patch on leaving the context > to stop buffering. > Whoops, got the True/False accidentally flipped. Still, I think the example gets the point across. Aside: the PSF should probably buy the "self.it" domain name to take advantage of Gmail's auto-linking. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 24 18:39:28 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 24 Mar 2016 15:39:28 -0700 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> <56F19E29.7090700@stoneleaf.us> <56F431B0.9040106@stoneleaf.us> Message-ID: <56F46CA0.1020504@stoneleaf.us> On 03/24/2016 03:14 PM, Michael Selik wrote: > More specifically, a time when the loop didn't run because the iterable > was empty AND the iterable didn't have a len to make that a simple check. > > Which suddenly gives me an idea... > > class Peeking: ... You mean like this answer? http://stackoverflow.com/a/7976272/208880 -- ~Ethan~ From Nikolaus at rath.org Thu Mar 24 18:45:15 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Thu, 24 Mar 2016 15:45:15 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: (Chris Barker's message of "Thu, 24 Mar 2016 14:20:27 -0700") References: Message-ID: <87shzfijgk.fsf@thinkpad.rath.org> On Mar 24 2016, Chris Barker wrote: > And it came up in another recent thred about a mechanism for doing > something after a for loop that didn't loop -- in that case, for sequences, > the idiom is obvious: > > if seq: > do_the_for_loop > else: > do_somethign else since the loop wont have run. > > But when you plug in an arbitrary iterable into that, it doesn't work, and > there is no easy, obvious, and robust idiom to replace that with. Hmm. empty = True for stuff in seq: empty = False do_stuff if empty: do_something_else is two lines longer than the above (if you expand the do_the_for_loop) and seems pretty obvious and robust. What do you dislike about it? Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From chris.barker at noaa.gov Thu Mar 24 19:52:36 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 24 Mar 2016 16:52:36 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <87shzfijgk.fsf@thinkpad.rath.org> References: <87shzfijgk.fsf@thinkpad.rath.org> Message-ID: <849954148806281407@unknownmsgid> > Hmm. > > empty = True > for stuff in seq: > empty = False > do_stuff > if empty: > do_something_else > > is two lines longer than the above (if you expand the do_the_for_loop) > and seems pretty obvious and robust. What do you dislike about it? Not horrible --- but see the other thread if you want to see it discussed to death :-) -CHB From andrey.vlasovskikh at gmail.com Thu Mar 24 20:00:37 2016 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Fri, 25 Mar 2016 03:00:37 +0300 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: Upon further investigation of the problem I've come up with an alternative idea that looks simpler and yet still capable of finding most text/binary conversion errors. Here is a rendered Markdown version: https://gist.github.com/vlasovskikh/1a8d5effe95d5944b919 ## TL;DR * Introduce `typing.Text` for text data in Python 2+3 * `bytes`, `str`, `unicode`, `typing.Text` in type hints mean whatever they mean at runtime for Python 2 or 3 * Allow `str -> unicode` and `unicode -> str` promotions for Python 2 * Type checking for Python 2 *and* Python 3 actually finds most text/binary errors * A few false negatives for Python 2 are not worth special handling besides possible ad-hoc handling of non-ASCII literals conversions ## Summary for Python users If you want your code to be Python 2+3 compatible: * Write text/binary type hints in 2+3 compatible comments * Use `typing.Text` for text data, `bytes` for binary data * Use `str` only for rare cases of "native strings" * Don't use `unicode` since it's absent in Python 3 * Run a type checker for *both* Python 2 and Python 3 ## Summary for authors of type checkers The semantics of types `bytes`, `str`, `unicode`, `typing.Text` and the type checking rules for them should match the *runtime behavior* of these types in Python 2 and Python 3 depending on Python 2 or 3 modes. Using the runtime semantics for the types is easy to understand while it still allows to catch most errors. The Python 2+3 compatibility mode is just a sum of Python 2 and Python 3 warnings. Type checkers *should* promote `str`/`bytes` to `unicode`/`Text` and `unicode`/`Text` to `str`/`bytes` for Python 2. Most text/binary conversion errors can be found by running a type checker for Python 2 *and* for Python 3. ## typing.Text: Python 2+3 compatible type for text data The `typing.Text` type is a Python 2+3 compatible type for text data. It's defined as follows: if sys.version_info < (3,): Text = unicode else: Text = str For a Python 2+3 compatible type for binary data use `bytes` that is available in both 2 and 3. ## Implicit text/binary conversions In Python 2 text data is implicitly converted to binary data and vice versa using the ASCII encoding. Only if the data isn't ASCII-compatible, then a `UnicodeEncodeError` or a `UnicodeDecodeError` is raised. This results in many programs that aren't well-tested regarding non-ASCII data handling. In Python 3 converting text data to binary data always raises a `TypeError`. A type checker run in the Python 3 mode will find most of Python 2 implicit conversion errors. ## Checking for Python 2+3 compatibility In order to be Python 2+3 compatible a program has to pass *both* Python 2 and Python 3 type checking. In other words, the warnings found in the Python 2+3 compatible mode are a simple sum of Python 2 warnings and Python 3 warnings. ## Runtime type compatibility Here is a table of types whose values are compatible at runtime. Columns are the expected types, rows are the actual types: | Text | bytes | str | unicode --------+-------+-------+-------+--------- Text | . . | * F | * . | . F bytes | * F | . . | . F | * F str | * . | . F | . . | * F unicode | . F | * F | * F | . F Each cell contains two characters: the result in Python 2 and in Python 3 respectively. Abbreviations: * `.` ? types are compatible * `F` ? types are not compatible * `*` ? types are compatible, ignoring implicit ASCII conversions At runtime in Python 2 `str` is compatible with `unicode` and vice versa (ignoring possible implicit ASCII conversion errors). Using `unicode` in Python 3 is always an error since there is no `unicode` name in Python 3. As you can see from the table above, many implicit ASCII conversion errors in a Python 2 program can be found just by running a type checker in the Python 3 mode. The only problematic conversions that may result in errors are `Text` to `str` and vice versa in Python 2. Example 1. `Text` to `str` def foo(obj, x): # type: (Any, str) -> Any return getattr(obj, x) foo(..., u'??????') # False negative warning for non-ASCII in Python 2 Example 2. `str` to `Text` def foo(x): # type: (Text) -> Any return u'??????, ' + x foo('???') # False negative warning for non-ASCII in Python 2 For non-ASCII text literals passed to functions that expect `Text` or `str` in Python 2 a type checker can analyze the contents of the literal and show additional warnings based on this information. For non-ASCII data coming from sources other than literals this check would be more complicated. To summarize, with this type compatibility table in place, a type checker run for *both* Python 2 and Python 3 is able to find *almost all errors* related to text and binary data except for a few text to "native string" conversions and vice versa in Python 2. ## Current Mypy type compatibility (non-runtime semantics) Mypy implies `str` to `unicode` promotion for Python 2, but it doesn't promote `unicode` to `str`. Here is an example of a Python 2 program that is correct given the runtime type compatibility semantics shown in the table above, but is incorrect for Mypy: def foo(obj, x): # type: (Any, str) -> Any return getattr(obj, x) foo({}, u'upper') # False positive warning in Mypy for ASCII in Python 2 Here is the type compatibility table for the current version of Mypy: | Text | bytes | str | unicode --------+-------+-------+-------+--------- Text | . . | F F | F . | . F bytes | * F | . . | . F | * F str | * . | . F | . . | * F unicode | . F | F F | F F | . F Running the Mypy type checker in Python 2 mode *and* Python 3 mode for the same program would find almost all implicit ASCII conversion errors except for `str` to `Text` conversions. To summarize, the current Mypy type compatibility table covers almost all text and binary data handling errors when used for *both* Python 2 and Python 3. But it doesn't notice errors in "native string" to text conversions in Python 2 and produces *false warnings* for text to "native string" conversions in Python 2. -- Andrey Vlasovskikh Web: http://pirx.ru/ From abarnert at yahoo.com Thu Mar 24 21:07:13 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 24 Mar 2016 18:07:13 -0700 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: <14A0AA58-A4A1-411C-AB0A-9354ACE01B62@yahoo.com> On Mar 24, 2016, at 17:00, Andrey Vlasovskikh wrote: > > The only problematic conversions that may result in errors are `Text` to `str` > and vice versa in Python 2. So any time you use Text strings together with strings from sys.argv, sys.stdin/raw_input(), os.listdir(), ZipFile, csv.reader, etc., all of which are native str, they'll pass as valid in a 2+3 test, even though they're not actually valid in 2.x? From andrey.vlasovskikh at gmail.com Thu Mar 24 22:06:11 2016 From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh) Date: Fri, 25 Mar 2016 05:06:11 +0300 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: <14A0AA58-A4A1-411C-AB0A-9354ACE01B62@yahoo.com> References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> <14A0AA58-A4A1-411C-AB0A-9354ACE01B62@yahoo.com> Message-ID: <99FE5B65-D9D5-448B-9720-4344E2ED74D0@gmail.com> > 2016-03-25, ? 4:07, Andrew Barnert ???????(?): > > On Mar 24, 2016, at 17:00, Andrey Vlasovskikh wrote: >> >> The only problematic conversions that may result in errors are `Text` to `str` >> and vice versa in Python 2. > > So any time you use Text strings together with strings from sys.argv, sys.stdin/raw_input(), os.listdir(), ZipFile, csv.reader, etc., all of which are native str, they'll pass as valid in a 2+3 test, even though they're not actually valid in 2.x? Yes, these errors will go unnoticed, unfortunately. But this guarantees that there will be no false positive warnings related to text/binary types. And a model of text/binary types that matches the runtime semantics is easier for users. This kind of errors would have been more important to find if users had been expected to port their code from Python 3 back to Python 2 more often than from 2 to 3. Speaking of ways to actually find these errors, one idea discussed in the issue tracker of Mypy [1] was to have a separate _AsciiStr type for things that are certainly ASCII-compatible. However, treating all str values as non-ASCII by default would result in false positive warnings. We could have a reverse type, say, _NonAsciiStr (there should be a better name for that) not compatible with Text for things we know are non-ASCII for sure: * Non-ASCII str literals * Functions like those you mentioned above There will be false negatives in cases not covered by _NonAsciiStr, but at least there will be a way of documenting non-ASCII native str interfaces for the users who care about this kind of Python 2 errors. The downside is that _NonAsciiStr is harder to understand and apply correctly than str. [1]: https://github.com/python/typing/issues/19 -- Andrey Vlasovskikh Web: http://pirx.ru/ From ncoghlan at gmail.com Thu Mar 24 22:22:37 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 25 Mar 2016 12:22:37 +1000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On 25 March 2016 at 08:07, Chris Barker wrote: > what's wrong with: > > open(a_path, 'w').write(the_string) > > short, simple one-liner. > > OK, non cPython garbage collection may leave that file open and dangling, > but we're talking the quicky scripting data analysis type user -- the script > will terminate soon enough. One of the few downsides of Python's popularity as both a scripting language and an app development language is that a lot of tutorials are written for the latter, and in app development, relying on the GC for external resource cleanup isn't a great habit to get into. As a result, tutorials will introduce the deterministic cleanup form, and actively discourage use of the non-deterministic cleanup. Independent of the "Why not just rely on the GC?" question, though, we're also forcing the user to upgrade their mental model to achieve their objective. User model: "I want to save this data to disk at a particular location and be able to read it back later" By contrast, unpacking the steps in the one-liner: - open the nominated location for writing (with implied text encoding & error handling) - write the data to that location It's that switch from a 1-step process to a 2-step process that breaks flow, rather than the specifics of the wording in the code (Python 3 at least improves the hidden third step in the process by having the implied text encoding typically be UTF-8 rather than ASCII). Formulating the question this way does suggest a somewhat radical notion, though: what if we had a JSON-based save builtin that wrote UTF-8 encoded files based on json.dump()? That is, "save(a_path, the_string)" would attempt to do: with open(a_path, 'w', encoding='utf-8', errors='strict') as f: json.dump(the_string, f) While a corresponding "load(a_path)" would attempt to do: with open(a_path, 'r', encoding='utf-8', errors='strict') as f: return json.load(f) The format of the created files would be using a well-defined standard rather than raw data dumps (as well as covering more than just pre-serialised strings), and the degenerate case of saving a single string would just have quotation marks added to the beginning and end. If we later chose to define a "__save__" and "__load__" protocol, then json.dump/load would also be able to benefit. There'd also be a potential long term security benefit here, as folks are often prone to reaching for pickle to save data structures to disk, which creates an arbitrary code execution security risk when loading them again later. Explicitly favouring the less dangerous JSON as the preferred serialisation format can help nudge people towards safer practices without forcing them to consciously think through the security implications. Switching from the save/load builtins to manual file management would then be a matter of improving storage efficiency and speed of access, just as switching to any other kind of persistent data store would be (and a JSON based save/load would migrate very well to NoSQL style persistent data storage services). Cheers, Nick. P.S. I'm going to be mostly offline until late next week, but found this idea intriguing enough to share before I left. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From nickeubank at gmail.com Thu Mar 24 22:27:27 2016 From: nickeubank at gmail.com (Nick Eubank) Date: Fri, 25 Mar 2016 02:27:27 +0000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: Thanks Nick -- I really couldn't have said it better than "we're also forcing the user to upgrade their mental model to achieve their objective." Python is beautiful in part because it takes all sorts of idioms that are complicated in other languages and wraps them up in something simple and intuitive ("in" has to be my favorite builtin of all time). This feels like one of those few cases where Python still feels like C, and it's not at all clear to my why that needs to be the case. On Thu, Mar 24, 2016 at 7:22 PM Nick Coghlan wrote: > On 25 March 2016 at 08:07, Chris Barker wrote: > > what's wrong with: > > > > open(a_path, 'w').write(the_string) > > > > short, simple one-liner. > > > > OK, non cPython garbage collection may leave that file open and dangling, > > but we're talking the quicky scripting data analysis type user -- the > script > > will terminate soon enough. > > One of the few downsides of Python's popularity as both a scripting > language and an app development language is that a lot of tutorials > are written for the latter, and in app development, relying on the GC > for external resource cleanup isn't a great habit to get into. As a > result, tutorials will introduce the deterministic cleanup form, and > actively discourage use of the non-deterministic cleanup. > > Independent of the "Why not just rely on the GC?" question, though, > we're also forcing the user to upgrade their mental model to achieve > their objective. > > User model: "I want to save this data to disk at a particular location > and be able to read it back later" > > By contrast, unpacking the steps in the one-liner: > > - open the nominated location for writing (with implied text encoding > & error handling) > - write the data to that location > > It's that switch from a 1-step process to a 2-step process that breaks > flow, rather than the specifics of the wording in the code (Python 3 > at least improves the hidden third step in the process by having the > implied text encoding typically be UTF-8 rather than ASCII). > > Formulating the question this way does suggest a somewhat radical > notion, though: what if we had a JSON-based save builtin that wrote > UTF-8 encoded files based on json.dump()? > > That is, "save(a_path, the_string)" would attempt to do: > > with open(a_path, 'w', encoding='utf-8', errors='strict') as f: > json.dump(the_string, f) > > While a corresponding "load(a_path)" would attempt to do: > > with open(a_path, 'r', encoding='utf-8', errors='strict') as f: > return json.load(f) > > The format of the created files would be using a well-defined standard > rather than raw data dumps (as well as covering more than just > pre-serialised strings), and the degenerate case of saving a single > string would just have quotation marks added to the beginning and end. > If we later chose to define a "__save__" and "__load__" protocol, then > json.dump/load would also be able to benefit. > > There'd also be a potential long term security benefit here, as folks > are often prone to reaching for pickle to save data structures to > disk, which creates an arbitrary code execution security risk when > loading them again later. Explicitly favouring the less dangerous JSON > as the preferred serialisation format can help nudge people towards > safer practices without forcing them to consciously think through the > security implications. > > Switching from the save/load builtins to manual file management would > then be a matter of improving storage efficiency and speed of access, > just as switching to any other kind of persistent data store would be > (and a JSON based save/load would migrate very well to NoSQL style > persistent data storage services). > > Cheers, > Nick. > > P.S. I'm going to be mostly offline until late next week, but found > this idea intriguing enough to share before I left. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Mar 24 22:41:06 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 25 Mar 2016 12:41:06 +1000 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: Message-ID: On 25 March 2016 at 07:20, Chris Barker wrote: > for line in the_file[:10]: > ... > > arrgg! files are not indexable!. > > (and by the way, not only for testing, but also when you really do want the > next ten lines from the file -- maybe it's a header, or...) > > Sure, I have plenty of ways to work around this, but frankly, they are all a > bit ugly, even if trivial. > > So maybe there should be some ability to index / slice iterables? I like the concept of focusing on working with file iterators and file processing concepts like head, tail and cat as a concrete example - starting by designing a clean solution to a specific problem and generalising from there is almost always a much better approach than trying to design the general case without reference to specific examples. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Fri Mar 25 00:05:19 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 25 Mar 2016 15:05:19 +1100 Subject: [Python-ideas] Control Flow - Never Executed Loop Body In-Reply-To: References: <56EEE82A.6050505@mail.de> <20160321013234.GB12526@ando.pearwood.info> <20160321132952.GD12526@ando.pearwood.info> Message-ID: <20160325040519.GO12526@ando.pearwood.info> On Tue, Mar 22, 2016 at 02:49:44PM -0400, Terry Reedy wrote: > On 3/21/2016 9:29 AM, Steven D'Aprano wrote: > >On Sun, Mar 20, 2016 at 11:19:54PM -0400, Terry Reedy wrote: > >>On 3/20/2016 9:32 PM, Steven D'Aprano wrote: > >> > >>>I understood from the keyword that "else" ran *if the for block > >>>didn't*, i.e. when the loop iterator is empty. .................................^^^^^^^^ I just realised I typo-ed that. I meant "iterable", not iterator. For-loops can iterate over any iterable (sequences, sets, dicts etc) not just iterators (as I'm sure you know). I may have inadvertently given you the wrong impression, in which case, I apologise. > >>>A perfectly natural mistake to make, > > >>Why is the truth a mistake? > > > >It's not the truth. > > Yes it is. When the iterator is empty, possibly after many iterations, > and the iterator is tested, the else clause runs. I can only guess that you thought I was referring to the invisible, implementation-dependent iterator which is used internally by the for-loop. (See byte code below for what I mean.) That's fine, but it isn't what the topic of this discussion is about. It's irrelevent. Nobody should care about whether *that* iterator is exhausted or not, because that iterator is unreachable from Python code. What we might care about is the iterable "seq" used in (for example) "for x in seq". Sometimes people want to run a block of code *only* if the body of the for loop never runs, and some people mistakenly thought, or think, that the "else" clause does that. I know this for a fact because I was one of those people, and I have seen others make the same mistake. I can prove it is a mistake by running this code: seq = [2, 4] for x in seq: print("seq is not empty") # this is correct else: print("seq is empty") # this is wrong assert list(seq) == [] The assertion won't *always* fail, e.g. try passing "seq = []") but the fact that it *sometimes* fails proves that it is a mistake to interpret the "else" clause as "run only when the for loop doesn't run". Take this for-loop: def f(): for x in seq: print(x) else: print(1) In Python 2.7, dis.dis(f) gives: 2 0 SETUP_LOOP 24 (to 27) 3 LOAD_GLOBAL 0 (seq) 6 GET_ITER >> 7 FOR_ITER 11 (to 21) 10 STORE_FAST 0 (x) 3 13 LOAD_FAST 0 (x) 16 PRINT_ITEM 17 PRINT_NEWLINE 18 JUMP_ABSOLUTE 7 >> 21 POP_BLOCK 5 22 LOAD_CONST 1 (1) 25 PRINT_ITEM 26 PRINT_NEWLINE >> 27 LOAD_CONST 0 (None) 30 RETURN_VALUE My guess is that you thought I was referring to the iterator which is created by the call to GET_ITER, and iterated over by FOR_ITER. Mea culpa -- I was not. I meant to say iterable, not iterator, and in this example I would mean "seq". If you look at the byte code, you will see that there is no test for whether this internal iterator, or seq for that matter, is empty. When the for loop, the code unconditionally executes the else block. If the for loop contains a break (or a return, or a raise) execution may transfer to outside of the for...else altogether, skipping the else block, but there is still no test for emptiness. It's an unconditional jump: if you reach the break, you jump past the end of the for...else. > > The else block does *not* only run if the for block doesn't run. > > I have never, ever, claimed that. Initially empty and empty after all > items have been processed are different. Then I hope that we now are on the same page, and agree that: (1) "else" does NOT mean "execute this block only if the for block doesn't run", even if some people mistakenly did, or do, think so. (2) Some people want such a statement, and that's the topic for this discussion. (3) As for...else now exists, there is no testing whether or not the for-loop ITERABLE is empty, or whether the internal (inaccessible) ITERATOR is empty; furthermore, other implementations or versions may not even use an internal ITERATOR (e.g. Python 1.5 certainly doesn't). (4) "break" doesn't set a flag, it just jumps to the outside of for...else (although in principle this is implementation dependent, there is nothing in the semantics of for...else that *requires* a flag be set), or the iterable be tested for emptiness. -- Steve From desmoulinmichel at gmail.com Fri Mar 25 02:41:32 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Fri, 25 Mar 2016 07:41:32 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: Message-ID: <56F4DD9C.1040804@gmail.com> Le 25/03/2016 03:41, Nick Coghlan a ?crit : > On 25 March 2016 at 07:20, Chris Barker wrote: >> for line in the_file[:10]: >> ... >> >> arrgg! files are not indexable!. >> >> (and by the way, not only for testing, but also when you really do want the >> next ten lines from the file -- maybe it's a header, or...) >> >> Sure, I have plenty of ways to work around this, but frankly, they are all a >> bit ugly, even if trivial. >> >> So maybe there should be some ability to index / slice iterables? > > I like the concept of focusing on working with file iterators and file > processing concepts like head, tail and cat as a concrete example - > starting by designing a clean solution to a specific problem and > generalising from there is almost always a much better approach than > trying to design the general case without reference to specific > examples. Dully noted. I'll do that next time. The ability to accept callables in slicing, and allow slicing on more iterables would actually fit well in the file processing use case: # get all lines between the first command and the first blank line # then limit that result to 100 with open(p) as f: def is_comment(line): return line.startswith('#') def is_blank_line(line): return line.strip() for line in f[is_comment, is_blank_line][:100]: print(line) It's also very convenient for generator expressions: # get random numbers between 0 and 100000, and square them # remove all numbers you can't devide by 3 # then sample 100 of them numbers = (x * x for x in random.randint(100000) if x % 3 == 0) for x in numbers[:100]: print(x) > > Cheers, > Nick. > From leewangzhong+python at gmail.com Fri Mar 25 12:30:19 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Fri, 25 Mar 2016 12:30:19 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On Mar 24, 2016 10:22 PM, "Nick Coghlan" wrote: > Independent of the "Why not just rely on the GC?" question, though, > we're also forcing the user to upgrade their mental model to achieve > their objective. > > User model: "I want to save this data to disk at a particular location > and be able to read it back later" > > By contrast, unpacking the steps in the one-liner: > > - open the nominated location for writing (with implied text encoding > & error handling) > - write the data to that location > > It's that switch from a 1-step process to a 2-step process that breaks > flow, rather than the specifics of the wording in the code (Python 3 > at least improves the hidden third step in the process by having the > implied text encoding typically be UTF-8 rather than ASCII). > > Formulating the question this way does suggest a somewhat radical > notion, though: what if we had a JSON-based save builtin that wrote > UTF-8 encoded files based on json.dump()? I was going to suggest that, if str save/load were to be implemented, it could be made as a parallel to json.dump and json.load (as string module members?). Any new features like atomicity would be "backported" to json.dump/load. Same with pickle.load/dump. It's just str serialization, isn't it? If there is a save/load builtin (or dump/load), I don't see what's so special about JSON. (YAML is more like Python, including in readability.) I'd suggest that you can specify a "format" parameter (or with another name) to use it with pickle, json, yaml, etc., but that means two ways of doing things. Deprecate the original way? import json load("data.json", format=json) Without a format kwarg, Python might look at the file extension and try to guess the format. Modules could register extensions (is this magic?), or it could look for an object with the same name which has __dump__/__load__ (is this VERY magic?). I don't like either: You have to import pickle to load it, but you wouldn't have to name it explicitly to use it. I'd hate `import pickle; pickle.register(".p")`: this thread is for scripters and not application devs, so wordiness matters. Your proposal means I don't even have to import json to use json, and json needs explaining to a data scientist (who won't typically have web dev experience). -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Fri Mar 25 12:53:13 2016 From: Nikolaus at rath.org (Nikolaus Rath) Date: Fri, 25 Mar 2016 09:53:13 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F4DD9C.1040804@gmail.com> (Michel Desmoulin's message of "Fri, 25 Mar 2016 07:41:32 +0100") References: <56F4DD9C.1040804@gmail.com> Message-ID: <87a8lm4hza.fsf@vostro.rath.org> On Mar 25 2016, Michel Desmoulin wrote: > It's also very convenient for generator expressions: > > # get random numbers between 0 and 100000, and square them > # remove all numbers you can't devide by 3 > # then sample 100 of them > > numbers = (x * x for x in random.randint(100000) if x % 3 == 0) > for x in numbers[:100]: > print(x) If there has ever been an example that I hope to never, ever see in actual code than it's this one. If you have to make up stuff like that to justify a proposed feature, then that does not bode well. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From zachary.ware+pyideas at gmail.com Fri Mar 25 12:57:47 2016 From: zachary.ware+pyideas at gmail.com (Zachary Ware) Date: Fri, 25 Mar 2016 11:57:47 -0500 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: On Wed, Mar 23, 2016 at 1:45 AM, Nick Coghlan wrote: > Having some lower level convenience functions in io may make sense > (specifically io.write_bytes(path, data) and io.write_text(path, data, > encoding=None, errors=None), but the str builtin definitely isn't the > right place for the capability. In this vein, what about just adding io.read() and io.write() convenience functions that take the same arguments as io.open() (with an added 'data' parameter for write()) and do the obvious "open file, return file.read_or_write(), close file"? This doesn't answer the atomic write question, but that wasn't in the OP. The OP's 'string'.to_file('some_file') would just be io.write('string', 'some_file'), and str.from_file('some_file') would be io.read('some_file'). Granted, both functions are 3 lines of Python, but they might well be used enough to be worth adding. -- Zach From srkunze at mail.de Fri Mar 25 13:14:57 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Fri, 25 Mar 2016 18:14:57 +0100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <56F42800.3050004@mail.de> Message-ID: <56F57211.9090200@mail.de> On 24.03.2016 22:06, Koos Zevenhoven wrote: > Or even better, that you could do p"filename.txt", which would give > you a Path string object. Has this been discussed? Interesting. I had this thought half a year ago. Quite recently I mentioned this to Andrew in a private conversation. p'/etc/hosts' would make a perfect path which subclassed from str. The p-string idea has not been discussed in the PEP I think. The subclassing thing however was and I think its resolution was a mistake. The explanation, at least from my point of view, is a bit weird. Maybe, this can be re-discussed in a separate thread? Especially when different people think independently of the same issue and the same solution. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Mar 25 14:04:27 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 25 Mar 2016 11:04:27 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: <56F57DAB.2040607@stoneleaf.us> On 03/25/2016 09:57 AM, Zachary Ware wrote: > On Wed, Mar 23, 2016 at 1:45 AM, Nick Coghlan wrote: >> Having some lower level convenience functions in io may make sense >> (specifically io.write_bytes(path, data) and io.write_text(path, data, >> encoding=None, errors=None), but the str builtin definitely isn't the >> right place for the capability. > > In this vein, what about just adding io.read() and io.write() > convenience functions that take the same arguments as io.open() (with > an added 'data' parameter for write()) and do the obvious "open file, > return file.read_or_write(), close file"? This doesn't answer the > atomic write question, but that wasn't in the OP. The OP's > 'string'.to_file('some_file') would just be io.write('string', > 'some_file'), and str.from_file('some_file') would be > io.read('some_file'). Granted, both functions are 3 lines of Python, > but they might well be used enough to be worth adding. +1 (with extra keyword arguments for fun ;) -- ~Ethan~ From k7hoven at gmail.com Fri Mar 25 16:20:29 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 25 Mar 2016 22:20:29 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? Message-ID: So, let's start a new thread about how to deal with pathlib.Path objects, involving how/whether to convert between str and Path. On Fri, Mar 25, 2016 at 7:14 PM, Sven R. Kunze wrote: >> On 24.03.2016 22:06, Koos Zevenhoven wrote: >> >> Or even better, that you could do p"filename.txt", which would give you a >> Path string object. Has this been discussed? >> > > Interesting. I had this thought half a year ago. Quite recently I mentioned > this to Andrew in a private conversation. > Yeah, I first thought about this at some point last summer when I had written the little np package for creating numpy arrays as np[3,4,5] (which I am planning to polish and maybe advertise), probably also inspired by the f-string discussions on this list. And now that you asked me about dealing with Paths... Anyway, as you say, if different people have thought about this independently, maybe its worth discussing. > p'/etc/hosts' would make a perfect path which subclassed from str. > Yeah, and regarding the "`to_file()` method for strings" thread, one could simply do p"targetfile.txt".write_text(string_to_write) using the write_text method introduced to Path objects in 3.5. [And of course there's also read_text(), write_bytes(...) and read_bytes()] One downside to p-strings like this is that they don't help you when you have the path in a plain str variable (i.e. not a literal), which has led me to think Path perhaps should be in builtins. But one thing that just occured to me now is that, for instance in interactive sessions, you could do p"" / path_as_str instead of `import pathlib; pathlib.Path(path_as_str)`. Not sure if this is the most beautiful thing ever seen, but it would work for quick scripting at least. > The p-string idea has not been discussed in the PEP I think. The subclassing > thing however was and I think its resolution was a mistake. The explanation, > at least from my point of view, is a bit weird. > Yeah, having Path subclass str would immediately solve a lot of compatibility issues. One question is how large a part of those issues would be solved by just making the builtin open(...) and a few places here and there in the stdlib accept Path objects. Maybe there are implementation issues too, although having them lead to bad decisions would be sad. Anyway, I think the question of whether Path should subclass str is partly separate from the p-string concept. After all, b'byte strings' are not subclassed from str either. - Koos From p.f.moore at gmail.com Fri Mar 25 16:49:17 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 25 Mar 2016 20:49:17 +0000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: On 25 March 2016 at 16:57, Zachary Ware wrote: > On Wed, Mar 23, 2016 at 1:45 AM, Nick Coghlan wrote: >> Having some lower level convenience functions in io may make sense >> (specifically io.write_bytes(path, data) and io.write_text(path, data, >> encoding=None, errors=None), but the str builtin definitely isn't the >> right place for the capability. > > In this vein, what about just adding io.read() and io.write() > convenience functions that take the same arguments as io.open() (with > an added 'data' parameter for write()) and do the obvious "open file, > return file.read_or_write(), close file"? This doesn't answer the > atomic write question, but that wasn't in the OP. The OP's > 'string'.to_file('some_file') would just be io.write('string', > 'some_file'), and str.from_file('some_file') would be > io.read('some_file'). Granted, both functions are 3 lines of Python, > but they might well be used enough to be worth adding. +1. If we're going to implement this functionality, this seems like the "one obvious way", now that it's been pointed out. Paul From desmoulinmichel at gmail.com Fri Mar 25 17:56:07 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Fri, 25 Mar 2016 22:56:07 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: Message-ID: <56F5B3F7.40502@gmail.com> Le 25/03/2016 21:20, Koos Zevenhoven a ?crit : > So, let's start a new thread about how to deal with pathlib.Path > objects, involving how/whether to convert between str and Path. > > On Fri, Mar 25, 2016 at 7:14 PM, Sven R. Kunze wrote: >>> On 24.03.2016 22:06, Koos Zevenhoven wrote: >>> >>> Or even better, that you could do p"filename.txt", which would give you a >>> Path string object. Has this been discussed? >>> >> >> Interesting. I had this thought half a year ago. Quite recently I mentioned >> this to Andrew in a private conversation. >> > > Yeah, I first thought about this at some point last summer when I had > written the little np package for creating numpy arrays as np[3,4,5] > (which I am planning to polish and maybe advertise), probably also > inspired by the f-string discussions on this list. And now that you > asked me about dealing with Paths... > > Anyway, as you say, if different people have thought about this > independently, maybe its worth discussing. > >> p'/etc/hosts' would make a perfect path which subclassed from str. >> > > Yeah, and regarding the "`to_file()` method for strings" thread, one > could simply do > > p"targetfile.txt".write_text(string_to_write) It's nice. It remains me of the proposal for odereddict with o{}. There is a limited number of prefix we can use though. > > using the write_text method introduced to Path objects in 3.5. [And of > course there's also read_text(), write_bytes(...) and read_bytes()] > > One downside to p-strings like this is that they don't help you when > you have the path in a plain str variable (i.e. not a literal), which > has led me to think Path perhaps should be in builtins. But one thing > that just occured to me now is that, for instance in interactive > sessions, you could do > > p"" / path_as_str > > instead of `import pathlib; pathlib.Path(path_as_str)`. Not sure if > this is the most beautiful thing ever seen, but it would work for > quick scripting at least. > >> The p-string idea has not been discussed in the PEP I think. The subclassing >> thing however was and I think its resolution was a mistake. The explanation, >> at least from my point of view, is a bit weird. >> > > Yeah, having Path subclass str would immediately solve a lot of > compatibility issues. One question is how large a part of those issues > would be solved by just making the builtin open(...) and a few places > here and there in the stdlib accept Path objects. Maybe there are > implementation issues too, although having them lead to bad decisions > would be sad. Although path.py, which I have been using for years now (and still prefer to pathlib) subclass str and never caused any problem whatsoever. So really, we should pinpoint where it could be an issue and see if this is a frequent problem, because right now, it seems more a decision based on purity than practicality. > > Anyway, I think the question of whether Path should subclass str is > partly separate from the p-string concept. After all, b'byte strings' > are not subclassed from str either. > > - Koos > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From rosuav at gmail.com Fri Mar 25 17:58:42 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 26 Mar 2016 08:58:42 +1100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On Sat, Mar 26, 2016 at 3:30 AM, Franklin? Lee wrote: > I was going to suggest that, if str save/load were to be implemented, it > could be made as a parallel to json.dump and json.load (as string module > members?). Any new features like atomicity would be "backported" to > json.dump/load. Same with pickle.load/dump. It's just str serialization, > isn't it? > > If there is a save/load builtin (or dump/load), I don't see what's so > special about JSON. (YAML is more like Python, including in readability.) > I'd suggest that you can specify a "format" parameter (or with another name) > to use it with pickle, json, yaml, etc., but that means two ways of doing > things. Deprecate the original way? > > import json > load("data.json", format=json) The thing that's special about JSON is that nearly everyone recognizes the name. Thanks to JSON's extensive use in networked systems (courtesy of web browser <-> web server communication), pretty much everyone will be able to grok bidirectional JSON handling. Maybe YAML would be better, but if you publish something that uses JSON, nobody will be confused. Not sure that's enough justification to make it the default, but it is significant. > Without a format kwarg, Python might look at the file extension and try to > guess the format. Modules could register extensions (is this magic?), or it > could look for an object with the same name which has __dump__/__load__ (is > this VERY magic?). > > I don't like either: You have to import pickle to load it, but you wouldn't > have to name it explicitly to use it. I'd hate `import pickle; > pickle.register(".p")`: this thread is for scripters and not application > devs, so wordiness matters. You can slap those lines into sitecustomize.py though. Or third party Python distributors could slap those lines into sitecustomize.py, and you won't even be aware that stuff had to be registered. But I don't like the registration of extensions as a means of selecting an output format, generally. Too magical. ChrisA From desmoulinmichel at gmail.com Fri Mar 25 18:16:05 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Fri, 25 Mar 2016 23:16:05 +0100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <87a8lm4hza.fsf@vostro.rath.org> References: <56F4DD9C.1040804@gmail.com> <87a8lm4hza.fsf@vostro.rath.org> Message-ID: <56F5B8A5.6090604@gmail.com> Le 25/03/2016 17:53, Nikolaus Rath a ?crit : > On Mar 25 2016, Michel Desmoulin wrote: >> It's also very convenient for generator expressions: >> >> # get random numbers between 0 and 100000, and square them >> # remove all numbers you can't devide by 3 >> # then sample 100 of them >> >> numbers = (x * x for x in random.randint(100000) if x % 3 == 0) >> for x in numbers[:100]: >> print(x) > > If there has ever been an example that I hope to never, ever see in > actual code than it's this one. If you have to make up stuff like that > to justify a proposed feature, then that does not bode well. The (x * x for x in random.randint(100000) if x % 3 == 0) is just here to stand for "rich generators". The example is artificial, but every advanced python code out there use generator. The feature is on the for loop, which is simple and elegant. What you're doing here is trolling, really. > > > Best, > -Nikolaus > From rosuav at gmail.com Fri Mar 25 18:19:15 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 26 Mar 2016 09:19:15 +1100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F5B8A5.6090604@gmail.com> References: <56F4DD9C.1040804@gmail.com> <87a8lm4hza.fsf@vostro.rath.org> <56F5B8A5.6090604@gmail.com> Message-ID: On Sat, Mar 26, 2016 at 9:16 AM, Michel Desmoulin wrote: > Le 25/03/2016 17:53, Nikolaus Rath a ?crit : >> On Mar 25 2016, Michel Desmoulin wrote: >>> It's also very convenient for generator expressions: >>> >>> # get random numbers between 0 and 100000, and square them >>> # remove all numbers you can't devide by 3 >>> # then sample 100 of them >>> >>> numbers = (x * x for x in random.randint(100000) if x % 3 == 0) >>> for x in numbers[:100]: >>> print(x) >> >> If there has ever been an example that I hope to never, ever see in >> actual code than it's this one. If you have to make up stuff like that >> to justify a proposed feature, then that does not bode well. > > The (x * x for x in random.randint(100000) if x % 3 == 0) is just here > to stand for "rich generators". The example is artificial, but every > advanced python code out there use generator. > > The feature is on the for loop, which is simple and elegant. > > What you're doing here is trolling, really. No, it's not trolling. But there's a difficult balance to strike when giving examples of a new feature; too simple and there's no justification for the feature, too complex and it's a poor justification ("if you have to make up stuff like that"). Trying to see past the example, generalizing to code that you might yourself write, isn't always easy. ChrisA From ethan at stoneleaf.us Fri Mar 25 18:34:13 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 25 Mar 2016 15:34:13 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F5B8A5.6090604@gmail.com> References: <56F4DD9C.1040804@gmail.com> <87a8lm4hza.fsf@vostro.rath.org> <56F5B8A5.6090604@gmail.com> Message-ID: <56F5BCE5.40100@stoneleaf.us> On 03/25/2016 03:16 PM, Michel Desmoulin wrote: > The (x * x for x in random.randint(100000) if x % 3 == 0) is just here > to stand for "rich generators". The example is artificial, but every > advanced python code out there use generator. True, but without a useful comment it was easy to miss that this line was not the feature. > The feature is on the for loop, which is simple and elegant. True. > What you're doing here is trolling, really. Just because you don't like it doesn't make it trolling. -- ~Ethan~ From greg.ewing at canterbury.ac.nz Fri Mar 25 17:59:41 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 26 Mar 2016 10:59:41 +1300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: Message-ID: <56F5B4CD.90404@canterbury.ac.nz> >>>On 24.03.2016 22:06, Koos Zevenhoven wrote: >>> >>>Or even better, that you could do p"filename.txt", which would give you a >>>Path string object. That would tie Path objects deeply into the parser and compiler, which I'm not sure is a good idea. Also, it would be stretching the string-prefix concept considerably. Currently, the prefixes just represent different ways of specifying a string -- the end result is still always an instance of str. In this proposal, it would be a different type of object with greatly different behaviour. Not sure whether I feel positively or negatively about this, so +0j. -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 25 18:03:05 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 26 Mar 2016 11:03:05 +1300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: Message-ID: <56F5B599.10708@canterbury.ac.nz> Koos Zevenhoven wrote: >>>Or even better, that you could do p"filename.txt", which would give you a >>>Path string object. Has this been discussed? Also, this would give a privileged place to one particular path library. How do we choose which one? -- Greg From donald at stufft.io Fri Mar 25 19:56:09 2016 From: donald at stufft.io (Donald Stufft) Date: Fri, 25 Mar 2016 19:56:09 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F5B4CD.90404@canterbury.ac.nz> References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: > On Mar 25, 2016, at 5:59 PM, Greg Ewing wrote: > > Currently, the prefixes just represent different ways of specifying > a string -- the end result is still always an instance of str. While I?m not sure if the p?? string is a good idea, this isn?t exactly true. b?? creates a bytes object, not a str object. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rosuav at gmail.com Fri Mar 25 19:56:26 2016 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 26 Mar 2016 10:56:26 +1100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F5B4CD.90404@canterbury.ac.nz> References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: On Sat, Mar 26, 2016 at 8:59 AM, Greg Ewing wrote: > Also, it would be stretching the string-prefix concept considerably. > Currently, the prefixes just represent different ways of specifying > a string -- the end result is still always an instance of str. > In this proposal, it would be a different type of object with > greatly different behaviour. There's b"..." vs u"...", which do represent entirely different objects, plus f"..." which isn't even a literal at all, but more like a special syntax for an expression (it's more akin to a list display than to a string literal). So there is precedent. If Path objects had universal support in the stdlib *and* significant support in third-party libraries, they could be the one obvious way to do pretty much anything involving paths. At the moment, they're a cute [1] way of getting just slightly more functionality than strings give. The question is: Should syntax precede or follow extensive usage? ChrisA [1] Overloading division doesn't really do anything for you, other than the way it looks similar to the normal path sep. It's really more of a concatenation operation, which would normally be + not /. From ethan at stoneleaf.us Fri Mar 25 21:56:00 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 25 Mar 2016 18:56:00 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F5B599.10708@canterbury.ac.nz> References: <56F5B599.10708@canterbury.ac.nz> Message-ID: <56F5EC30.2010908@stoneleaf.us> On 03/25/2016 03:03 PM, Greg Ewing wrote: > Koos Zevenhoven wrote: >>>> Or even better, that you could do p"filename.txt", which would give >>>> you a >>>> Path string object. Has this been discussed? > > Also, this would give a privileged place to one particular > path library. How do we choose which one? Um, the one in the stdlib? ;) -- ~Ethan~ From k7hoven at gmail.com Sat Mar 26 06:35:06 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sat, 26 Mar 2016 12:35:06 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: On Sat, Mar 26, 2016 at 1:56 AM, Chris Angelico wrote: [...] > If Path objects had universal support in the stdlib *and* significant > support in third-party libraries, they could be the one obvious way to > do pretty much anything involving paths. True. I suppose many third-party libraries will just work when the stdlib functions they use [like open(...) and os.path.join(...)] have started accepting Path objects. Then, if a third-party library returns a string, one would need to convert it into a path. That is, unless you just directly pass it to open(...) or something so you don't care. How many reasonably things can a library do with paths without just passing them to stdlib functions? [...] > Should syntax precede or follow extensive usage? > > Does anything learned from the evolution of string formatting apply here? > [1] Overloading division doesn't really do anything for you, other > than the way it looks similar to the normal path sep. It's really more > of a concatenation operation, which would normally be + not /. > I agree that joining paths with / is the least interesting feature of pathlib.Path. Some of this can of course be done with os.path.* too, but here's the list of methods and properties on Path: ['absolute', 'anchor', 'as_posix', 'as_uri', 'chmod', 'cwd', 'drive', 'exists', 'expanduser', 'glob', 'group', 'home', 'is_absolute', 'is_block_device', 'is_char_device', 'is_dir', 'is_fifo', 'is_file', 'is_reserved', 'is_socket', 'is_symlink', 'iterdir', 'joinpath', 'lchmod', 'lstat', 'match', 'mkdir', 'name', 'open', 'owner', 'parent', 'parents', 'parts', 'read_bytes', 'read_text', 'relative_to', 'rename', 'replace', 'resolve', 'rglob', 'rmdir', 'root', 'samefile', 'stat', 'stem', 'suffix', 'suffixes', 'symlink_to', 'touch', 'unlink', 'with_name', 'with_suffix', 'write_bytes', 'write_text'] - Koos -------------- next part -------------- An HTML attachment was scrubbed... URL: From abarnert at yahoo.com Sat Mar 26 06:44:18 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 26 Mar 2016 03:44:18 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: Message-ID: On Mar 25, 2016, at 13:20, Koos Zevenhoven wrote: > > So, let's start a new thread about how to deal with pathlib.Path > objects, involving how/whether to convert between str and Path. > > On Fri, Mar 25, 2016 at 7:14 PM, Sven R. Kunze wrote: >>> On 24.03.2016 22:06, Koos Zevenhoven wrote: >>> >>> Or even better, that you could do p"filename.txt", which would give you a >>> Path string object. Has this been discussed? >> >> Interesting. I had this thought half a year ago. Quite recently I mentioned >> this to Andrew in a private conversation. > > Yeah, I first thought about this at some point last summer when I had > written the little np package for creating numpy arrays as np[3,4,5] > (which I am planning to polish and maybe advertise), probably also > inspired by the f-string discussions on this list. And now that you > asked me about dealing with Paths... > > Anyway, as you say, if different people have thought about this > independently, maybe its worth discussing. > >> p'/etc/hosts' would make a perfect path which subclassed from str. Having Path types subclass str would solve 90% of the problems with pathlib. Adding p-strings raises that to 91%; I don't think it's enough extra win to be worth it. The main reason not to use pathlib today is that almost nothing takes Path objects. A handful of functions (open, stat, access) have methods on the Path objects. But if you want to zlib.open a Path, or add one to a ZipFile, or pass one to json.load or etree.parse, or use it with any third-party library, you have to convert to str manually. And in the opposite direction, paths you get from sys.argv or WSGI or anywhere else are str, not Path, so you have to construct Path objects manually too. So, attempting to use Path all over the place means converting back and forth to str all over the place. And the benefit of having Path objects is so tiny that it's not worth the cost. Path seems to have been designed for a statically-typed language with implicit conversions. In fact, it's very similar to the Boost/C++ design. But that design works for C++ because every time you pass a path to a function that wants a string, or try to store a string in a path-typed variable, the compiler sees what you're doing, sees that the types are unrelated, and looks for a conversion operator or constructor so it can insert an implicit call. That doesn't, and can't, happen in Python. As mentioned in one of the other replies, pathlib's main PyPI predecessor does subclass str, and that makes it a lot more usable. >> The p-string idea has not been discussed in the PEP I think. The subclassing >> thing however was and I think its resolution was a mistake. The explanation, >> at least from my point of view, is a bit weird. I assume there are good arguments against making Path inherit from str. Unfortunately, those arguments don't seem to appear in the PEP. They really should. But, given that they don't, anyone proposing a change like this really should go back and look at the archived previous discussions, summarize and link the old arguments, and offer counter-arguments. > Anyway, I think the question of whether Path should subclass str is > partly separate from the p-string concept. After all, b'byte strings' > are not subclassed from str either. Well, adding p-strings but _not_ subclassing str seems to add almost nothing. It makes it easier to construct literal Path objects that you still can't use anywhere, so you end up writing "spam(str(p'eggs.txt'))" instead of just "spam('eggs.txt')"? But I agree that they can be separated, for the opposite reason: a str-subclassing Path is useful with or without p-strings. From desmoulinmichel at gmail.com Sat Mar 26 06:44:52 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sat, 26 Mar 2016 11:44:52 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: <56F66824.4090202@gmail.com> Le 26/03/2016 11:35, Koos Zevenhoven a ?crit : > On Sat, Mar 26, 2016 at 1:56 AM, Chris Angelico > wrote: > [...] > > If Path objects had universal support in the stdlib *and* significant > support in third-party libraries, they could be the one obvious way to > do pretty much anything involving paths. > > > True. I suppose many third-party libraries will just work when the > stdlib functions they use [like open(...) and os.path.join(...)] have > started accepting Path objects. Then, if a third-party library returns a > string, one would need to convert it into a path. That is, unless you > just directly pass it to open(...) or something so you don't care. How > many reasonably things can a library do with paths without just passing > them to stdlib functions? > > [...] > > Should syntax precede or follow extensive usage? > > > Does anything learned from the evolution of string formatting apply here? > > > [1] Overloading division doesn't really do anything for you, other > than the way it looks similar to the normal path sep. It's really more > of a concatenation operation, which would normally be + not /. > > > I agree that joining paths with / is the least interesting feature of > pathlib.Path. Quite the contratry: - it saves "+" so that "+" still returns a string if needed an not a Path object, preserving compatibility. - on the other hand, "/" is garantied to return a Path object, meaning you can chain the method you listed : Path('test/') / 'foo').method - it ensure your don't have double "/" around : >>> Path('test/') / 'foo' PosixPath('test/foo') >>> Path('test') / 'foo' PosixPath('test/foo') - it will insert "\" if you need it to. Dealing with "\" is really annoying because it's an escape caracter, so it abstract that for you. - it marks clearly in the code that you are doing a concatenation of directories and files, and not something like a prefix, a suffix. You can spot quickly the places in the code where you deal with hierarchy. It's a subtile but quite awesome feature. > > Some of this can of course be done with os.path.* too, but here's the > list of methods and properties on Path: > > ['absolute', 'anchor', 'as_posix', 'as_uri', 'chmod', 'cwd', 'drive', > 'exists', 'expanduser', 'glob', 'group', 'home', 'is_absolute', > 'is_block_device', 'is_char_device', 'is_dir', 'is_fifo', 'is_file', > 'is_reserved', 'is_socket', 'is_symlink', 'iterdir', 'joinpath', > 'lchmod', 'lstat', 'match', 'mkdir', 'name', 'open', 'owner', 'parent', > 'parents', 'parts', 'read_bytes', 'read_text', 'relative_to', 'rename', > 'replace', 'resolve', 'rglob', 'rmdir', 'root', 'samefile', 'stat', > 'stem', 'suffix', 'suffixes', 'symlink_to', 'touch', 'unlink', > 'with_name', 'with_suffix', 'write_bytes', 'write_text'] > > - Koos > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From desmoulinmichel at gmail.com Sat Mar 26 06:51:36 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sat, 26 Mar 2016 11:51:36 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: Message-ID: <56F669B8.7090405@gmail.com> Le 26/03/2016 11:44, Andrew Barnert via Python-ideas a ?crit : > On Mar 25, 2016, at 13:20, Koos Zevenhoven wrote: >> >> So, let's start a new thread about how to deal with pathlib.Path >> objects, involving how/whether to convert between str and Path. >> >> On Fri, Mar 25, 2016 at 7:14 PM, Sven R. Kunze wrote: >>>> On 24.03.2016 22:06, Koos Zevenhoven wrote: >>>> >>>> Or even better, that you could do p"filename.txt", which would give you a >>>> Path string object. Has this been discussed? >>> >>> Interesting. I had this thought half a year ago. Quite recently I mentioned >>> this to Andrew in a private conversation. >> >> Yeah, I first thought about this at some point last summer when I had >> written the little np package for creating numpy arrays as np[3,4,5] >> (which I am planning to polish and maybe advertise), probably also >> inspired by the f-string discussions on this list. And now that you >> asked me about dealing with Paths... >> >> Anyway, as you say, if different people have thought about this >> independently, maybe its worth discussing. >> >>> p'/etc/hosts' would make a perfect path which subclassed from str. > > Having Path types subclass str would solve 90% of the problems with pathlib. Adding p-strings raises that to 91%; I don't think it's enough extra win to be worth it. > > The main reason not to use pathlib today is that almost nothing takes Path objects. A handful of functions (open, stat, access) have methods on the Path objects. But if you want to zlib.open a Path, or add one to a ZipFile, or pass one to json.load or etree.parse, or use it with any third-party library, you have to convert to str manually. And in the opposite direction, paths you get from sys.argv or WSGI or anywhere else are str, not Path, so you have to construct Path objects manually too. So, attempting to use Path all over the place means converting back and forth to str all over the place. And the benefit of having Path objects is so tiny that it's not worth the cost. > > Path seems to have been designed for a statically-typed language with implicit conversions. In fact, it's very similar to the Boost/C++ design. But that design works for C++ because every time you pass a path to a function that wants a string, or try to store a string in a path-typed variable, the compiler sees what you're doing, sees that the types are unrelated, and looks for a conversion operator or constructor so it can insert an implicit call. That doesn't, and can't, happen in Python. > > As mentioned in one of the other replies, pathlib's main PyPI predecessor does subclass str, and that makes it a lot more usable. > >>> The p-string idea has not been discussed in the PEP I think. The subclassing >>> thing however was and I think its resolution was a mistake. The explanation, >>> at least from my point of view, is a bit weird. > > I assume there are good arguments against making Path inherit from str. Unfortunately, those arguments don't seem to appear in the PEP. They really should. But, given that they don't, anyone proposing a change like this really should go back and look at the archived previous discussions, summarize and link the old arguments, and offer counter-arguments. > >> Anyway, I think the question of whether Path should subclass str is >> partly separate from the p-string concept. After all, b'byte strings' >> are not subclassed from str either. > > Well, adding p-strings but _not_ subclassing str seems to add almost nothing. It makes it easier to construct literal Path objects that you still can't use anywhere, so you end up writing "spam(str(p'eggs.txt'))" instead of just "spam('eggs.txt')"? > +1 to everything. Remember that because of this, all projects I work on still use path.py. We prefer to add an additional dependency than to deal with the casting, because it spreads everywhere. And we are not a conservative crew, some projects use Python 3.5 and asyncio. Although honestly, p'/foo/bar'.write_text(stuff) would be awesome at some many levels. Can you imagine how scripting becomes easier because of this ? Sys admin ? Shell sessions ? > But I agree that they can be separated, for the opposite reason: a str-subclassing Path is useful with or without p-strings. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From abarnert at yahoo.com Sat Mar 26 06:59:57 2016 From: abarnert at yahoo.com (Andrew Barnert) Date: Sat, 26 Mar 2016 03:59:57 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: Message-ID: On Mar 25, 2016, at 13:20, Koos Zevenhoven wrote: > > So, let's start a new thread about how to deal with pathlib.Path > objects, involving how/whether to convert between str and Path As a different point: If we _don't_ either subclass str or find some way to make things magically work, is there any other way to start getting more uptake on pathlib? This may only be anecdotal, but from what I can tell, nobody is using it, because everyone who tries is put off by the need to convert back and forth everywhere. Last year, everyone agreed that it would be good if at least the stdlib accepted paths everywhere, which might prompt third party libs to start doing the same. But nobody's started writing patches to do that. And I'm not sure we'd want it to happen in an uncoordinated way anyway, since there are at least four different ways to do it, and if we pick among them arbitrarily for different parts of the stdlib, we'll have a huge mess, and possibly behavioral inconsistencies. The four ways I can think of are (in every function that currently takes a "path: str" argument, and should now take a "path: str | Path"): * path = str(path) * path = Path(path) * if isinstance(path, Path): ... else: ... * try: f = path.open('w') except AttributeError: open(path, 'w') It's also worth noting that all but the first require every module to depend on Path. Including C modules. And modules in the bootstrap. But the first version makes a bad guide for third-party code, because a lot of third-party code is dual-version/single-source libs, and you definitely don't want to unconditionally call str on a path argument that may be Unicode in 2.7 (or to unconditionally call a six-style Unicode function on a path argument that may be str in 2.7 on Linux). So, assuming we all want a future in which pathlib is actually used by people, and assuming str subclassing is out and there's no other magic bullet, how do we get there from here? From desmoulinmichel at gmail.com Sat Mar 26 07:02:58 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sat, 26 Mar 2016 12:02:58 +0100 Subject: [Python-ideas] Add a begins like wrapper to argparse In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: <56F66C62.6090703@gmail.com> I love argparse. I really do. But it's one of those lib forcing me to RTFM everytime I need it, for the last 10 years. We had pathlib added to the stdlib to help with open/os/shutil API and we could do the same with argparse. There are many libs helping with argparsing (clize, click, argh, docopt, etc), and I'd like to talk about the API of one of them, begins. You define the entry point of your script as a regular function you decorate: >>> import begin >>> @begin.start ... def run(name='Arther', quest='Holy Grail', colour='blue', *knights): ... "tis but a scratch!" @begin.start will do "if __name__ == '__main__'" for you (alhough you can opt out), and parse the function arguments then generate a argparse parser. When you call the script, it parses the arguments, and call your function with them passed as parameters. It also generates the --help: usage: example.py [-h] [-n NAME] [-q QUEST] [-c COLOUR] [knights [knights ...]] tis but a scratch! positional arguments: knights optional arguments: -h, --help show this help message and exit -n NAME, --name NAME (default: Arther) -q QUEST, --quest QUEST (default: Holy Grail) -c COLOUR, --colour COLOUR (default: blue) Of course, you can define more behavior using other decorators such as repetition, type, checks and sub commands. It's not a huge lib, but's it makes writting script out of the box a breeze. The only problem is : you usually don't want to do a pip install just to run a script. That's why I things it would be great to have something similar in the stdlib. What's more, it's really not going to evolve much (you can't get more old school than cmd), so having it not be updated for years after including it (or something like it) in the stdlib is not an issue. And we can even add some features that would be useful to benefit from the last Python goodness, such as an option to auto-spawn an event loop. This way you can create a quick script with just an async def function. Indeed, the boiler plate of getting the event loop, adding a coroutine and starting the loop is getting annoying very fast when you do a lot of quick things with any aio libs. From kaiser.yann at gmail.com Sat Mar 26 07:21:53 2016 From: kaiser.yann at gmail.com (Yann Kaiser) Date: Sat, 26 Mar 2016 11:21:53 +0000 Subject: [Python-ideas] Add a begins like wrapper to argparse In-Reply-To: <56F66C62.6090703@gmail.com> References: <56F5B4CD.90404@canterbury.ac.nz> <56F66C62.6090703@gmail.com> Message-ID: (Full disclosure: I'm the developer of clize, another argument parser) > you usually don't want to do a pip install just to run a script > I feel like this should be corrected instead. While I'm not advocating script developers auto-install packages, it'd be nice to help "I just want to ship a single 50-line file" people make better use of dependencies. What if they want to use `requests` (of which the maintainer is, as I understand it, leaning on the side of not including it in the stdlib) ? It's even more likely than wishing for a different argument parser. I'm sure there are other examples. Including another lib in the stdlib (of which there are already 3 alternatives in!) is merely delaying this general issue. -- Yann Kaiser kaiser.yann at gmail.com yann.kaiser at efrei.net +33 6 51 64 01 89 https://github.com/epsy -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Sat Mar 26 08:43:50 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Sat, 26 Mar 2016 08:43:50 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: Message-ID: <56F68406.3030103@sdamon.com> I have been watching this thread, and read the docs and the PEP, and... What is pathlib even for? Yes, its there to be an object oriented abstraction of a file system. Why? Why do I want this? Am I alone in wondering why this exists? Is it even worth improving? On 3/26/2016 06:59, Andrew Barnert via Python-ideas wrote: > On Mar 25, 2016, at 13:20, Koos Zevenhoven wrote: >> So, let's start a new thread about how to deal with pathlib.Path >> objects, involving how/whether to convert between str and Path > As a different point: > > If we _don't_ either subclass str or find some way to make things magically work, is there any other way to start getting more uptake on pathlib? This may only be anecdotal, but from what I can tell, nobody is using it, because everyone who tries is put off by the need to convert back and forth everywhere. > > Last year, everyone agreed that it would be good if at least the stdlib accepted paths everywhere, which might prompt third party libs to start doing the same. But nobody's started writing patches to do that. And I'm not sure we'd want it to happen in an uncoordinated way anyway, since there are at least four different ways to do it, and if we pick among them arbitrarily for different parts of the stdlib, we'll have a huge mess, and possibly behavioral inconsistencies. > > The four ways I can think of are (in every function that currently takes a "path: str" argument, and should now take a "path: str | Path"): > > * path = str(path) > * path = Path(path) > * if isinstance(path, Path): ... else: ... > * try: f = path.open('w') except AttributeError: open(path, 'w') > > It's also worth noting that all but the first require every module to depend on Path. Including C modules. And modules in the bootstrap. But the first version makes a bad guide for third-party code, because a lot of third-party code is dual-version/single-source libs, and you definitely don't want to unconditionally call str on a path argument that may be Unicode in 2.7 (or to unconditionally call a six-style Unicode function on a path argument that may be str in 2.7 on Linux). > > So, assuming we all want a future in which pathlib is actually used by people, and assuming str subclassing is out and there's no other magic bullet, how do we get there from here? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From jlehtosalo at gmail.com Sat Mar 26 08:55:11 2016 From: jlehtosalo at gmail.com (Jukka Lehtosalo) Date: Sat, 26 Mar 2016 12:55:11 +0000 Subject: [Python-ideas] Type hints for text/binary data in Python 2+3 code In-Reply-To: References: <09935613-2494-4A15-9CAE-CCE73957804E@gmail.com> Message-ID: On Fri, Mar 25, 2016 at 12:00 AM, Andrey Vlasovskikh < andrey.vlasovskikh at gmail.com> wrote: > Upon further investigation of the problem I've come up with an alternative > idea that looks simpler and yet still capable of finding most text/binary > conversion errors. > ... > ## TL;DR > > * Introduce `typing.Text` for text data in Python 2+3 > * `bytes`, `str`, `unicode`, `typing.Text` in type hints mean whatever they > mean at runtime for Python 2 or 3 > * Allow `str -> unicode` and `unicode -> str` promotions for Python 2 > I'm against this, as it would seem to make str and unicode pretty much the same type in Python 2, and thus Python 2 mode seems much weaker than necessary. I wrote a more detailed reply in the mypy issue tracker ( https://github.com/python/mypy/issues/1141#issuecomment-201799761). I'm not copying it all here since much of that is somewhat mypy-specific and related to the rest of the discussion on that issue, but I'll summarize my main points here. I prefer the idea of doing better type checking in Python 2 mode for str and unicode, though I suspect we need to implement a prototype to decide whether it will be practical. * Type checking for Python 2 *and* Python 3 actually finds most text/binary > errors > This may be true, but I'm worried about usability for Python 2 code bases. Also, the effort needed to pass type checking in both modes (which is likely pretty close to the effort of a full Python 3 migration, if the entire code will be annotated) might be impractical for a large Python 2 code base. ## Summary for authors of type checkers > > The semantics of types `bytes`, `str`, `unicode`, `typing.Text` and the > type > checking rules for them should match the *runtime behavior* of these types > in > Python 2 and Python 3 depending on Python 2 or 3 modes. Using the runtime > semantics for the types is easy to understand while it still allows to > catch > most errors. The Python 2+3 compatibility mode is just a sum of Python 2 > and > Python 3 warnings. > At least for mypy, the Python 2+3 compatibility mode would likely that twice as much CPU to run, which is a pretty high cost as type checking speed is one of the biggest open issues we have right now. ## Runtime type compatibility > ... > Each cell contains two characters: the result in Python 2 and in Python 3 > respectively. Abbreviations: > ... > * `*` ? types are compatible, ignoring implicit ASCII conversions > Am I reading this right if I understand this as "considered valid during type checking but may fail at runtime"? For non-ASCII text literals passed to functions that expect `Text` or `str` > in > Python 2 a type checker can analyze the contents of the literal and show > additional warnings based on this information. For non-ASCII data coming > from > sources other than literals this check would be more complicated. > I wonder what would the check look like in the latter case? I can't imagine how this would work for non-literals. Jukka -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Sat Mar 26 10:03:16 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sat, 26 Mar 2016 15:03:16 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F68406.3030103@sdamon.com> References: <56F68406.3030103@sdamon.com> Message-ID: <56F696A4.6080704@gmail.com> Le 26/03/2016 13:43, Alexander Walters a ?crit : > I have been watching this thread, and read the docs and the PEP, and... > > What is pathlib even for? > > Yes, its there to be an object oriented abstraction of a file system. > Why? Why do I want this? Because it makes a very common task (FS manipulation) much more natural. This is the same reason people like request over urllib or like @decorator over func = decorator(func) or unpacking vs manual item extract. In a directory of music files, you ave mp3 files you converted to other formats. Now you want to remove those conversion. Your task is to find all files having the same name as the ones with the mp3 ones, but with a different extension, remove them, then list the mp3 absolument path in a text file. This example has baked in a lot of the tasks you do when you use Python as a scripting language at the core of you job such as if it's your glue language, if you are a tester or for sysadmin tasks. But it helps also everybody that once in a file does a one off script. The traditional approach: import os import glob import sys root = os.path.abspath(sys.argv[1]) playlist = os.path.join(root, 'playlist.m3u8') with open(playlist, 'w') as f: for path in glob.glob(os.path.join(root, '*.mp3')): name, ext = os.path.splitext(os.path.basename(path)) for to_remove in glob.glob(os.path.join(root, name + '.*')): if not to_remove.endswith('mp3'): os.remove(to_remove) f.write(os.path.join(root, path) + "\n") Now with pathlib you don't have to wonder about whether the feature you are looking for is on "os", "os.path" or "glob" or "open". You don't have to deal the file opening for such a small script. You don't have to switch between functions and methods all the time and have to choose between nested function calls or intermediary variables. The pathlib version is way easier to figure out without knowing the stdlib by heart, it is one line shorter and one level of indent less. And you can discover it all from iPython with ".": import sys import pathlib root = pathlib.Path(sys.argv[1]) files = [] for path in root.glob('*.mp3'): name = str(path).replace(path.suffix, '.*') files.append(str(path.absolute())) for to_remove in root.glob(name): if to_remove.suffix != ".mp3": to_remove.unlink() (root / 'playlist.m3u8').write_text('\n'.join(files)) And this true while pathlib is a half backed written lib, since the competition (that existed before pathlib and that pathlib failed to inspire from), can do even shorter, easier and cleaner: import sys import path root = path.Path(sys.argv[1]).realpath() files = [] for p in root.glob('*.mp3'): name = p.namebase + '.*' files.append(p) for to_remove in root.glob(name): if to_remove.ext != ".mp3": to_remove.remove() (root / 'playlist.m3u8').write_lines(files) Because path.py: - inherit from str - has all the methods from os, not just a few cherry picked - has logical names for attributes and methods - have more utilities than pathlib or os So yes, if you do a lot of scripting, this is a must. It's also way easier for beginers to grasp. Am I alone in wondering why this exists? Is > it even worth improving? > > On 3/26/2016 06:59, Andrew Barnert via Python-ideas wrote: >> On Mar 25, 2016, at 13:20, Koos Zevenhoven wrote: >>> So, let's start a new thread about how to deal with pathlib.Path >>> objects, involving how/whether to convert between str and Path >> As a different point: >> >> If we _don't_ either subclass str or find some way to make things >> magically work, is there any other way to start getting more uptake on >> pathlib? This may only be anecdotal, but from what I can tell, nobody >> is using it, because everyone who tries is put off by the need to >> convert back and forth everywhere. >> >> Last year, everyone agreed that it would be good if at least the >> stdlib accepted paths everywhere, which might prompt third party libs >> to start doing the same. But nobody's started writing patches to do >> that. And I'm not sure we'd want it to happen in an uncoordinated way >> anyway, since there are at least four different ways to do it, and if >> we pick among them arbitrarily for different parts of the stdlib, >> we'll have a huge mess, and possibly behavioral inconsistencies. >> >> The four ways I can think of are (in every function that currently >> takes a "path: str" argument, and should now take a "path: str | Path"): >> >> * path = str(path) >> * path = Path(path) >> * if isinstance(path, Path): ... else: ... >> * try: f = path.open('w') except AttributeError: open(path, 'w') >> >> It's also worth noting that all but the first require every module to >> depend on Path. Including C modules. And modules in the bootstrap. But >> the first version makes a bad guide for third-party code, because a >> lot of third-party code is dual-version/single-source libs, and you >> definitely don't want to unconditionally call str on a path argument >> that may be Unicode in 2.7 (or to unconditionally call a six-style >> Unicode function on a path argument that may be str in 2.7 on Linux). >> >> So, assuming we all want a future in which pathlib is actually used by >> people, and assuming str subclassing is out and there's no other magic >> bullet, how do we get there from here? >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From tritium-list at sdamon.com Sat Mar 26 10:09:57 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Sat, 26 Mar 2016 10:09:57 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F696A4.6080704@gmail.com> References: <56F68406.3030103@sdamon.com> <56F696A4.6080704@gmail.com> Message-ID: <56F69835.206@sdamon.com> I had to read the traditional way to figure out what you were even trying with the other two, so I don't know how 'natural' it is. As long as the traditional way continues to be supported, and this is what pathlib is intended to do, I guess I don't care. I know I'm not going to use it. On 3/26/2016 10:03, Michel Desmoulin wrote: > > > Because it makes a very common task (FS manipulation) much more natural. > > This is the same reason people like request over urllib or like > @decorator over func = decorator(func) or unpacking vs manual item extract. > > In a directory of music files, you ave mp3 files you converted to other > formats. Now you want to remove those conversion. Your task is to find > all files having the same name as the ones with the mp3 ones, but with a > different extension, remove them, then list the mp3 absolument path in a > text file. > > This example has baked in a lot of the tasks you do when you use Python > as a scripting language at the core of you job such as if it's your glue > language, if you are a tester or for sysadmin tasks. But it helps also > everybody that once in a file does a one off script. > > The traditional approach: > > import os > import glob > import sys > > root = os.path.abspath(sys.argv[1]) > > playlist = os.path.join(root, 'playlist.m3u8') > > with open(playlist, 'w') as f: > > for path in glob.glob(os.path.join(root, '*.mp3')): > > name, ext = os.path.splitext(os.path.basename(path)) > > for to_remove in glob.glob(os.path.join(root, name + '.*')): > if not to_remove.endswith('mp3'): > os.remove(to_remove) > > f.write(os.path.join(root, path) + "\n") > > Now with pathlib you don't have to wonder about whether the feature you > are looking for is on "os", "os.path" or "glob" or "open". You don't > have to deal the file opening for such a small script. You don't have to > switch between functions and methods all the time and have to choose > between nested function calls or intermediary variables. > > The pathlib version is way easier to figure out without knowing the > stdlib by heart, it is one line shorter and one level of indent less. > And you can discover it all from iPython with ".": > > import sys > import pathlib > > root = pathlib.Path(sys.argv[1]) > > files = [] > for path in root.glob('*.mp3'): > > name = str(path).replace(path.suffix, '.*') > files.append(str(path.absolute())) > > for to_remove in root.glob(name): > if to_remove.suffix != ".mp3": > to_remove.unlink() > > (root / 'playlist.m3u8').write_text('\n'.join(files)) > > And this true while pathlib is a half backed written lib, since the > competition (that existed before pathlib and that pathlib failed to > inspire from), can do even shorter, easier and cleaner: > > import sys > import path > > root = path.Path(sys.argv[1]).realpath() > > files = [] > for p in root.glob('*.mp3'): > > name = p.namebase + '.*' > files.append(p) > > for to_remove in root.glob(name): > if to_remove.ext != ".mp3": > to_remove.remove() > > (root / 'playlist.m3u8').write_lines(files) > > Because path.py: > > - inherit from str > - has all the methods from os, not just a few cherry picked > - has logical names for attributes and methods > - have more utilities than pathlib or os > > So yes, if you do a lot of scripting, this is a must. It's also way > easier for beginers to grasp. > > > > > From p.f.moore at gmail.com Sat Mar 26 10:33:20 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 26 Mar 2016 14:33:20 +0000 Subject: [Python-ideas] Add a begins like wrapper to argparse In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> <56F66C62.6090703@gmail.com> Message-ID: On 26 March 2016 at 11:21, Yann Kaiser wrote: > I feel like this should be corrected instead. While I'm not advocating > script developers auto-install packages, it'd be nice to help "I just want > to ship a single 50-line file" people make better use of dependencies. You may want to look at zipapp (and in general the support that's been in Python since 2.6) for zipped applications. For pure-Python dependencies you can put them in a directory alongside your application script (renamed as __main__.py), add the dependency directory to sys.path, and zip it all up and have a standalone file runnable with Python. Paul From dk.titouan at gmail.com Sat Mar 26 12:30:19 2016 From: dk.titouan at gmail.com (titouan dk) Date: Sat, 26 Mar 2016 17:30:19 +0100 Subject: [Python-ideas] custom predicate for all() & any() Message-ID: Hi, I was wondering if the builtins functions all() & any() could ever accept a custom predicate: all(iterable, predicate=bool) any(iterable, predicate=bool) Something like Lodash.js functions: every(), ... ## Example 1 ages = [21, 13, 18] major = 18..__le__ allowed = all(ages, major) ## Example 2 people = [ {"name": "Mike", "age": 42}, {"name": "Josh", "age": 17} ] major = 18..__le__ allowed = all(people, {"age": major}) ## Consistency drawbacks The filter() function takes its arguments in the reverse order: filter(predicate_or_none, iterable) I would have write this function like this... filter(iterable, predicate=None) ... but this is not the case, anyway. ## Your opinion? What do you think about this? Good idea, could be a quick-win? Or requiring a lot of efforts for nothing? What about consistency drawbacks? Thanks for reading, Titouan de Kerautem -------------- next part -------------- An HTML attachment was scrubbed... URL: From clint.hepner at gmail.com Sat Mar 26 12:33:37 2016 From: clint.hepner at gmail.com (Clint Hepner) Date: Sat, 26 Mar 2016 12:33:37 -0400 Subject: [Python-ideas] custom predicate for all() & any() In-Reply-To: References: Message-ID: <1C405062-977D-4324-8A14-D37AAB752F7E@gmail.com> > On 2016 Mar 26 , at 12:30 p, titouan dk wrote: > > Hi, > > I was wondering if the builtins functions all() & any() could ever accept a custom predicate: > > all(iterable, predicate=bool) > any(iterable, predicate=bool) Just map your predicate over the iterable: from itertools import imap all(imap(bool, iterable)) From stephen at xemacs.org Sat Mar 26 13:00:21 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 27 Mar 2016 02:00:21 +0900 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: Message-ID: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> Chris Barker writes: > I understand the focus on iterables -- for performance reasons if > nothing else, but I think it does, in fact, make the language more > complex. and I think we should think about making the language more > iterable-focused. The *language* has always been iterable-focused. Python doesn't have any counting loops! range() is a builtin (and a necessary one), but not part of the language. for and while are part of the language, and they are both "do until done" loops, not "do n times" loops. I think the problem is more on the user (and teacher) side. We learn that a square has four sides, but perhaps it's more Pythonic(?) to think of a square as the process "do until here == start: forward 1 furlong; right 90". But *neither* (or *both*) is how the children I've observed think of it. They draw four sides, the first three quite straight, and the last one *warped as necessary* to join with the first. (And adult practice is even more varied: drawing two parallel sides first, then joining them at the ends. In the case of the Japanese and Chinese, a square has *three* sides: the left vertical is drawn, then the top and right are drawn as one stroke, and finally the horizontal base.[2]) People don't think like the algorithms that are convenient for us to teach computers. > And it came up in another recent thred about a mechanism for doing > something after a for loop that didn't loop -- in that case, for > sequences, the idiom is obvious: > > if seq: > do_the_for_loop > else: > do_somethign else since the loop wont have run. > > But when you plug in an arbitrary iterable into that, it doesn't > work, and there is no easy, obvious, and robust idiom to replace > that with. I don't know that that particular issue needs to be > solved, but it makes my point But does it? That thread never did present a real use case for "empty:" with an iterator. Evidently the OP has one, but we didn't get to see it. All of the realistic iterator cases I can think of are *dynamic*: eg RSS or Twitter feeds. In those cases, it's not that the iterator is empty, it's that it's in a wait state. Even an empty database cursor can be interpreted that way. (If you didn't expect updates, why are you using a database?) I am inclined to think this is a general point, that is, the problem is not *empty* iterators vs. *non-empty* ones. It's iterators that have produced values recently vs. those that haven't. The empty vs. non-empty distinction is a property of *sequences*, including buffers (which are associated with iterators). > but now that file objects ar iterable, I can do: > > for line in the_file: > .... > > much nicer! > > but it breaks when I want to debug and try to do: > > for line in the_file[:10]: > ... > > arrgg! files are not indexable!. No, they are *enumerable*: for i, line in enumerate(the_file): if i >= 10: break ... I guess your request to make iterators more friendly is a good part of why enumerate() got promoted to builtin. > So maybe there should be some ability to index / slice iterables? There's no way to index an iterator, except to enumerate it. Then, "to memoize or not to memoize, that is the question." Slicing makes more sense to me, but again the fact that your discarded data may or may not be valuable means that you need to make a choice between memoizing and not doing so. Putting that in the API is complexity, or perhaps even complication. If you just want head or tail, then takewhile or dropwhile from itertools is your friend. (I have no opinion -- not even -0 -- on whether promoting those functions to builtin is a good idea.) > But aside from that -- just the idea that looking at how to make > iterable a more "natural" part of the language is a good thing. I think it's from Zen and the Art of Motorcycle Maintenance (though Pirsig may have been quoting), but I once read the advice: "If you want to paint a perfect painting, make yourself perfect and then paint naturally." I think iterators are just "unnatural" to a lot of people, and will remain unnatural until people evolve. Which they may not! Real life "do until done" tasks are careers ("do ... until dead") or have timeouts ("do n times: break if done else ..."). In computation that would be analogous to scrolling a Twitter feed vs. grabbing a pageful. In the context of this discussion, a feed is something you wait for (and maybe timeout and complain to the operator if it blocks too long), while you can apply len() to pages. And you know which is which in all applications I've dealt with -- except for design of abstract programming language facilities like "for ... in". The point being that "real life" examples don't seem to help people's intuition on the Python versions. From tjreedy at udel.edu Sat Mar 26 13:04:43 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 26 Mar 2016 13:04:43 -0400 Subject: [Python-ideas] custom predicate for all() & any() In-Reply-To: <1C405062-977D-4324-8A14-D37AAB752F7E@gmail.com> References: <1C405062-977D-4324-8A14-D37AAB752F7E@gmail.com> Message-ID: On 3/26/2016 12:33 PM, Clint Hepner wrote: >> On 2016 Mar 26 , at 12:30 p, titouan dk wrote: >> I was wondering if the builtins functions all() & any() could ever accept a custom predicate: >> >> all(iterable, predicate=bool) >> any(iterable, predicate=bool) > > Just map your predicate over the iterable: > > from itertools import imap > all(imap(bool, iterable)) I agree. In 3.x, which is nearly always the subject of this list, map == (2.7) itertools.imap. So proposed all(iterable, predicate) == all(map(predicate, iterable)) -- Terry Jan Reedy From songofacandy at gmail.com Sat Mar 26 13:22:34 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Sat, 26 Mar 2016 17:22:34 +0000 Subject: [Python-ideas] custom predicate for all() & any() In-Reply-To: References: Message-ID: > > > ## Example 1 > > ages = [21, 13, 18] > major = 18..__le__ > > allowed = all(ages, major) > > allowed = all(18 <= a for a in ages) > ## Example 2 > > people = [ > {"name": "Mike", "age": 42}, > {"name": "Josh", "age": 17} > ] > major = 18..__le__ > > allowed = all(people, {"age": major}) > allowed = all(18 <= p['age'] for p in people) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat Mar 26 13:32:40 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 27 Mar 2016 02:32:40 +0900 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: Message-ID: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> Koos Zevenhoven writes: > One downside to p-strings like this is that they don't help you when > you have the path in a plain str variable (i.e. not a literal), IMO that kills this simple form of the idea, since hard-coding paths is a bad idea in many many programs, and paths are naturally derived from user input in a host of others. It becomes an attractive nuisance. How about generalizing f-strings, or perhaps giving p-strings (some of) the power of f-strings? From k7hoven at gmail.com Sat Mar 26 14:17:20 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sat, 26 Mar 2016 20:17:20 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sat, Mar 26, 2016 at 7:32 PM, Stephen J. Turnbull wrote: > Koos Zevenhoven writes: > > > How about generalizing f-strings, or perhaps giving p-strings (some > of) the power of f-strings? > > Just their combination: pf"{ path_as_string }" ? That would at least be very consistent. - Koos -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Mar 26 15:10:28 2016 From: brett at python.org (Brett Cannon) Date: Sat, 26 Mar 2016 19:10:28 +0000 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F5B4CD.90404@canterbury.ac.nz> References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: On Fri, 25 Mar 2016 at 16:50 Greg Ewing wrote: > >>>On 24.03.2016 22:06, Koos Zevenhoven wrote: > >>> > >>>Or even better, that you could do p"filename.txt", which would give you > a > >>>Path string object. > > That would tie Path objects deeply into the parser and compiler, > which I'm not sure is a good idea. > I'll go one step further than Greg and say that it's not a good idea. To make something like this work you have two options. One is you do what importlib does and very carefully construct it to only rely on built-in modules (and if you look at https://hg.python.org/cpython/file/default/Lib/pathlib.py you will notice that is no where near true ATM). Two, is Python gets a two-tier syntax system for before and after pathlib is imported and available. That would mean that all the dependencies of pathlib would need to not use this syntax until it's bootstrapped in. If you notice, both solutions are extremely painful to implement and radiate well past the code in pathilb.py and would potentially become hard to maintain. > > Also, it would be stretching the string-prefix concept considerably. > Currently, the prefixes just represent different ways of specifying > a string -- the end result is still always an instance of str. > In this proposal, it would be a different type of object with > greatly different behaviour. > Yep, that too. > > Not sure whether I feel positively or negatively about this, > so +0j. > I'm sure how I feel :) -1. I also want to mention two things. One, pathlib.path is a thing now and something most people are probably not aware of as an alternative to doing `str(path)`: https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path . And two, wrapping path inputs in `pathlib.Path(path)` is a small price to pay for a nicer API. As time passes it will be less and less of a thing as APIs will start to come up that explicitly accept either, and then eventually they will only accept pathlib. Plus if you do it at the edges of your own API then you can do the conversion early and only get out the string as necessary later on as needed (much like how you handle text/binary data boundaries). I get the desire of succinctness as Python spoils us for that, but much like __future__ statements it's really a small price to pay in the grand scheme of things in order to migrate to a better API. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gvanrossum at gmail.com Sat Mar 26 15:21:36 2016 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat, 26 Mar 2016 12:21:36 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: I'm not going to say how I feel. But I noticed one thing in this thread. These seems to be a growing difference between people who use Python for "serious" programming and those who consider themselves testers, QA integrators and the like (more power to them!). Maybe both groups can occasionally try to walk in the others' shoes. --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Sat Mar 26 15:22:23 2016 From: random832 at fastmail.com (Random832) Date: Sat, 26 Mar 2016 15:22:23 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F5B4CD.90404@canterbury.ac.nz> References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: <1459020143.980673.560411090.2373AF70@webmail.messagingengine.com> On Fri, Mar 25, 2016, at 17:59, Greg Ewing wrote: > >>>On 24.03.2016 22:06, Koos Zevenhoven wrote: > >>> > >>>Or even better, that you could do p"filename.txt", which would give you a > >>>Path string object. > > That would tie Path objects deeply into the parser and compiler, > which I'm not sure is a good idea. It's not obviously _not_ a good idea. Are float and complex more "worthy" of being tied into the parser and compiler than Path? From storchaka at gmail.com Sat Mar 26 15:29:34 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 26 Mar 2016 21:29:34 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: Message-ID: On 26.03.16 12:59, Andrew Barnert via Python-ideas wrote: > Last year, everyone agreed that it would be good if at least the stdlib accepted paths everywhere, which might prompt third party libs to start doing the same. But nobody's started writing patches to do that. And I'm not sure we'd want it to happen in an uncoordinated way anyway, since there are at least four different ways to do it, and if we pick among them arbitrarily for different parts of the stdlib, we'll have a huge mess, and possibly behavioral inconsistencies. There is an open issue and I'm working on the patch. I'm going to make major path-related functions in the stdlib accepting the Path object in 3.6. > The four ways I can think of are (in every function that currently takes a "path: str" argument, and should now take a "path: str | Path"): > > * path = str(path) > * path = Path(path) > * if isinstance(path, Path): ... else: ... > * try: f = path.open('w') except AttributeError: open(path, 'w') The official way: try: path = path.path except AttributeError: pass The advantage is that this works not with concrete Path implementation, but with any third-party Path class and other path-related objects like DirEntry that provide the "path" attribute. From k7hoven at gmail.com Sat Mar 26 15:53:13 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sat, 26 Mar 2016 21:53:13 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F696A4.6080704@gmail.com> References: <56F68406.3030103@sdamon.com> <56F696A4.6080704@gmail.com> Message-ID: On Sat, Mar 26, 2016 at 4:03 PM, Michel Desmoulin wrote: > [...] > Now with pathlib you don't have to wonder about whether the feature you > are looking for is on "os", "os.path" or "glob" or "open". You don't > have to deal the file opening for such a small script. You don't have to > switch between functions and methods all the time and have to choose > between nested function calls or intermediary variables. > > Here we completely agree. I used to do scripting of filesystem-intensive tasks with bash, because it felt more natural, partly because I was doing that before I started using Python for anything. Due to pathlib I now do this stuff in Python, which I of course like better for other reasons. Pretty much everything is there, and nicely auto-completed in editors. The fact that you can do / for joining paths is not necessarily better than a method (which exists too as .joinpath()), but it works and may be easy to remember. > The pathlib version is way easier to figure out without knowing the > stdlib by heart, it is one line shorter and one level of indent less. > And you can discover it all from iPython with ".": > > import sys > import pathlib > > root = pathlib.Path(sys.argv[1]) > > files = [] > for path in root.glob('*.mp3'): > > name = str(path).replace(path.suffix, '.*') > files.append(str(path.absolute())) > > for to_remove in root.glob(name): > if to_remove.suffix != ".mp3": > to_remove.unlink() > > (root / 'playlist.m3u8').write_text('\n'.join(files)) > > Or even cleaner, and hopefully easier to understand (not tested, sorry): import sys import pathlib root = pathlib.Path(sys.argv[1]) files = [f.absolute() for f in root.glob("*.mp3")] for mp3file in files: for f in root.glob(mp3file.stem + ".*"): if f.suffix != ".mp3": f.unlink() # remove non-mp3 version (root / "playlist.m3u8").write_text("\n".join(files)) Yes, some of the properties could have more logical names like .remove() instead of .unlink(), and there could be even more functionality. Subclassing from str would have obvious benefits, but I'm just not sure whether it's a good idea to inherit things like str.find, str.splitlines, str.title, ... So maybe the best thing is not to inherit from str but to make Path quack as much like a str-duck as possible (and vice versa?) without introducing confusing things? BTW, here's a link to issue #22570 (Better stdlib support for Path objects) http://www.psf.upfronthosting.co.za/issue22570 - Koos -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sat Mar 26 18:55:29 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 27 Mar 2016 11:55:29 +1300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F5EC30.2010908@stoneleaf.us> References: <56F5B599.10708@canterbury.ac.nz> <56F5EC30.2010908@stoneleaf.us> Message-ID: <56F71361.5010301@canterbury.ac.nz> Ethan Furman wrote: >> Also, this would give a privileged place to one particular >> path library. How do we choose which one? > > Um, the one in the stdlib? ;) Not everyone likes that one though. As things stand, you can easily substitute one of your own that's more to your liking. That won't be true if path objects get baked into the language syntax. -- Greg From leewangzhong+python at gmail.com Sat Mar 26 19:01:12 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Sat, 26 Mar 2016 19:01:12 -0400 Subject: [Python-ideas] custom predicate for all() & any() In-Reply-To: References: Message-ID: Others have already demonstrated using generators and `map` for the general case, but FYI, the following are equivalent: 1. `all(iterable)` 2. `all(map(bool, iterable))` Because these are equivalent: 1. `if obj:` 2. `if bool(obj):` On Mar 26, 2016 12:31 PM, "titouan dk" wrote: > Hi, > > I was wondering if the builtins functions all() & any() could ever accept > a custom predicate: > > all(iterable, predicate=bool) > any(iterable, predicate=bool) > > Something like Lodash.js functions: every(), ... > > ## Example 1 > > ages = [21, 13, 18] > major = 18..__le__ > > allowed = all(ages, major) > > ## Example 2 > > people = [ > {"name": "Mike", "age": 42}, > {"name": "Josh", "age": 17} > ] > major = 18..__le__ > > allowed = all(people, {"age": major}) > > ## Consistency drawbacks > > The filter() function takes its arguments in the reverse order: > > filter(predicate_or_none, iterable) > > I would have write this function like this... > > filter(iterable, predicate=None) > > ... but this is not the case, anyway. > > ## Your opinion? > > What do you think about this? > Good idea, could be a quick-win? > Or requiring a lot of efforts for nothing? > What about consistency drawbacks? > > > Thanks for reading, > > Titouan de Kerautem > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sat Mar 26 19:13:25 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 27 Mar 2016 12:13:25 +1300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: <56F71795.7040101@canterbury.ac.nz> Chris Angelico wrote: > There's b"..." vs u"...", which do represent entirely different > objects, That's true, but they're not entirely unrelated -- bytes objects are the py3 version of what str was in py2. Also, they're both pretty fundamental to the language, and they're essentially pure representations of data, with not much semantics attached. Path objects, on the other hand, come with a pile of semantics relating to a particular application area -- manipulation of the file system. Moreover, those semantics vary depending on what operating system you're running on. So it's hard so see them as being part of the language the way str and bytes are. This feels like a slippery slope to me. If we include special syntax for pathnames, why shouldn't we have it for dates and times? Regular expressions? URLs? JSON data? SQL queries? XML data? Where do we draw the line? > plus f"..." which isn't even a literal at all, but more like > a special syntax for an expression It still evaluates to a str, though. -- Greg From k7hoven at gmail.com Sat Mar 26 19:23:13 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 27 Mar 2016 01:23:13 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: On Sat, Mar 26, 2016 at 9:10 PM, Brett Cannon wrote: > > > On Fri, 25 Mar 2016 at 16:50 Greg Ewing > wrote: > >> >>>On 24.03.2016 22:06, Koos Zevenhoven wrote: >> >>> >> >>>Or even better, that you could do p"filename.txt", which would give >> you a >> >>>Path string object. >> >> That would tie Path objects deeply into the parser and compiler, >> which I'm not sure is a good idea. >> > > I'll go one step further than Greg and say that it's not a good idea. To > make something like this work you have two options. One is you do what > importlib does and very carefully construct it to only rely on built-in > modules (and if you look at > https://hg.python.org/cpython/file/default/Lib/pathlib.py you will notice > that is no where near true ATM). Two, is Python gets a two-tier syntax > system for before and after pathlib is imported and available. That would > mean that all the dependencies of pathlib would need to not use this > syntax until it's bootstrapped in. If you notice, both solutions are > extremely painful to implement and radiate well past the code in pathilb.py > and would potentially become hard to maintain. > As I understand it, this could be implemented by making the compiler essentially turn p"/path/to/whatever" into something like _make_path("/path/to/whatever"), where _make_path would be builtin and do something like this. def _make_path(str_path): import pathlib return pathlib.Path(str_path) Unless of course Path could be modified to import its dependencies on demand and put in builtins. Am I missing something crucial? > > I also want to mention two things. One, pathlib.path is a thing now and > something most people are probably not aware of as an alternative to doing > `str(path)`: > https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path . > I assume you meant to type pathlib.Path.path, so that Path("...").path == str(Path("...")). That's a good start, and I'm looking forward to Serhiy's patch for making the stdlib accept Paths. But if Path will not subclass str, we also need new stdlib functions that *return* Paths. - Koos -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sat Mar 26 19:38:58 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 27 Mar 2016 12:38:58 +1300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F68406.3030103@sdamon.com> References: <56F68406.3030103@sdamon.com> Message-ID: <56F71D92.9030605@canterbury.ac.nz> I'm wondering how often literal pathames are really used in actual programs. In my experience, most pathnames are received as arguments, read from config files, etc. Most literals are just fragments that get concatenated with existing pathnames, and that's already covered by operations between path objects and strings. -- Greg From k7hoven at gmail.com Sat Mar 26 19:51:04 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 27 Mar 2016 01:51:04 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F71795.7040101@canterbury.ac.nz> References: <56F5B4CD.90404@canterbury.ac.nz> <56F71795.7040101@canterbury.ac.nz> Message-ID: On Sun, Mar 27, 2016 at 1:13 AM, Greg Ewing wrote: > [...] > This feels like a slippery slope to me. If we include > special syntax for pathnames, why shouldn't we have it > for dates and times? Regular expressions? URLs? JSON > data? SQL queries? XML data? Where do we draw the line? > OT: To be honest, I do think it feels like URL:s are becoming (or have become) just as important as paths, and that pathlib.Path should in the future work with URLs just like it now works with windows and posix paths. The difference between "http://domain.xyz/" and "C:\\" is not huge. I also think there should be a Python type (stdlib or builtin), which handles JSON objects nicer than dicts do and has its own literal syntax. - Koos -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Mar 26 20:02:14 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 26 Mar 2016 17:02:14 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F71361.5010301@canterbury.ac.nz> References: <56F5B599.10708@canterbury.ac.nz> <56F5EC30.2010908@stoneleaf.us> <56F71361.5010301@canterbury.ac.nz> Message-ID: <56F72306.8090802@stoneleaf.us> On 03/26/2016 03:55 PM, Greg Ewing wrote: > Ethan Furman wrote: > >>> Also, this would give a privileged place to one particular >>> path library. How do we choose which one? >> >> Um, the one in the stdlib? ;) > > Not everyone likes that one though. As things stand, you > can easily substitute one of your own that's more to your > liking. That won't be true if path objects get baked into > the language syntax. Yeah, I'm one of those (primarily because it's a pain to work with). My point was just that baked-in syntax would naturally go with stdlib objects (and it would still not preclude using your preferred library). -- ~Ethan~ From k7hoven at gmail.com Sat Mar 26 20:33:36 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 27 Mar 2016 02:33:36 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: On Sun, Mar 27, 2016 at 1:23 AM, Koos Zevenhoven wrote: > On Sat, Mar 26, 2016 at 9:10 PM, Brett Cannon wrote: > >> [...] >> > I also want to mention two things. One, pathlib.path is a thing now and >> something most people are probably not aware of as an alternative to doing >> `str(path)`: >> https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path . >> > > I assume you meant to type pathlib.Path.path, so that Path("...").path == > str(Path("...")). That's a good start, and I'm looking forward to Serhiy's > patch for making the stdlib accept Paths. But if Path will not subclass > str, we also need new stdlib functions that *return* Paths. > > Well, just to reply to myself, here's a slightly crazy idea, which I'll mention before I realize that it's a bad idea: What if there was a another class, say StringPath, that inherits from both str and Path, which wraps another instance of Path, but is also a str. When you call its Path methods, it would delegate them to the wrapped Path object so that functions that now return paths as plain str could in future versions start returning that type? - Koos -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Mar 26 21:42:14 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 27 Mar 2016 12:42:14 +1100 Subject: [Python-ideas] PEP 484 evolution In-Reply-To: References: <4a3c35b1-0e6a-4825-b55a-0078a4022f48@googlegroups.com> <20160319132704.GL8022@ando.pearwood.info> <20160319164609.GN8022@ando.pearwood.info> <56EE0926.5070408@sdamon.com> Message-ID: <20160327014214.GP12526@ando.pearwood.info> On Sun, Mar 20, 2016 at 01:29:36PM +1100, Chris Angelico wrote: > On Sun, Mar 20, 2016 at 1:21 PM, Alexander Walters > wrote: > > I don't exactly see how stub files would be any different than tests in this > > regard. Unless I am totally misunderstanding something, the type hints only > > exist to make static analysis by tools (presumably in a test suite) easier. > > Why would a failing static analysis check go unnoticed and not the a test? > > Why is there no argument to integrate the test suite into the module under > > test, by the same logic? > > There are - look at things like doctest. However, tests tend to be > much longer-winded than type hints - even the most verbose of type > hints - so the cost of keeping them inline is far higher. In my experience, for every line of code I write, I will end up with ten lines of tests. Keeping *all* my tests in the same file as the code being tested would completely overwhelm the actual code. Nevertheless, it is appropriate to include some tests in place with the code: - doc tests, as Chris mentions above; - some applications may include a short "self-test" that the user can run from the command line to verify that it is working; - we might consider assertions as a kind of run time internal consistency test; - some languages have support for integrating tests in your code. For example, Eiffel makes extensive use of Design By Contract for testing, with "require", "ensure" and "invariant" keywords. https://archive.eiffel.com/doc/online/eiffel50/intro/language/invitation-07.html Rust allows you to mark a function as test code by annotating it with the "test" attribute. http://static.rust-lang.org/doc/master/book/testing.html D can integrate the function and its unit test in the same piece of code: http://dlang.org/spec/unittest.html Cobra includes both contracts and in-place unit tests: http://cobra-language.com/docs/quality/ -- Steve From wes.turner at gmail.com Sun Mar 27 01:25:26 2016 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 27 Mar 2016 00:25:26 -0500 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> <56F71795.7040101@canterbury.ac.nz> Message-ID: On Sat, Mar 26, 2016 at 6:51 PM, Koos Zevenhoven wrote: > On Sun, Mar 27, 2016 at 1:13 AM, Greg Ewing > wrote: > >> [...] >> This feels like a slippery slope to me. If we include >> special syntax for pathnames, why shouldn't we have it >> for dates and times? Regular expressions? URLs? JSON >> data? SQL queries? XML data? Where do we draw the line? >> > > OT: > > To be honest, I do think it feels like URL:s are becoming (or have become) > just as important as paths, and that pathlib.Path should in the future work > with URLs just like it now works with windows and posix paths. The > difference between "http://domain.xyz/" and "C:\\" is not huge. > Here's some preliminary research (w/ links) that may be of use here: https://www.reddit.com/r/Python/comments/1r7h1t/python_objects_for_working_with_urls_and_uris/ * URLObject (+1 from me) * rdflib.term.URIRef * fs.path Differences between fs paths and URIs: * urlencode IDK why I seem to remember having concluded that it would make sense to store paths as tuples / lists and join by os.path.sep > I also think there should be a Python type (stdlib or builtin), which > handles JSON objects nicer than dicts do and has its own literal syntax. > collections.OrderedDict almost works here, except that .keys() and .values() must be casted to e.g. lists (because otherwise the comparison is between KeysView / ValuesView ~iterators) to_json = _repr_json_ # "IPython repr method examples" # https://gist.github.com/westurner/be22dba8110be099a35e#file-ordereddefaultdict-py-L110 Then, if Path is not a subclass of str (or, py2 __builtin__.unicode, as in https://github.com/jaraco/path.py/blob/master/path.py ), comparisons between JSON-ified path objects will fail. > > - Koos > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Sun Mar 27 07:48:33 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sun, 27 Mar 2016 13:48:33 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F71D92.9030605@canterbury.ac.nz> References: <56F68406.3030103@sdamon.com> <56F71D92.9030605@canterbury.ac.nz> Message-ID: <56F7C891.6030704@gmail.com> Le 27/03/2016 00:38, Greg Ewing a ?crit : > I'm wondering how often literal pathames are really used > in actual programs. > > In my experience, most pathnames are received as arguments, > read from config files, etc. Most literals are just fragments > that get concatenated with existing pathnames, and that's > already covered by operations between path objects and strings. > That's because you make big programs. I used to think that as well but spending now half of my time as a trainer, I met many other ways to use Python: - financial analists: load recursively all those csv files in pandas for me please. - biologists : damn where did I put my data again ? ipython => numpy => ok not this one. Not this one. Not this one... - geomaticians : hum, it's too slow, what if I reduce my layers to get only bob shape files ? Let me write that again. - sysadmins : ok let's move all those around, it's becoming a mess here. I'll make real script later to prevent this for happening again. - sunday scipters : oh, I'll reorganize my music today; - students : I don't know what I'm doing, but I know I want to do it in this dir. - testers : what's the f***** ? let me fire a shell to explore this. Python is not just used by programmers. It's used by 1000 people for whom reconfigurability is a nice to have in the end, but first they want to explore stuff manually and easily. I've been spending a lot of time in docker containers lately, and a switched a lot between python and bash because Python is still less productive to use for files. I can see myself in the shell using p'' to replace open, os and the like. Just because it's faster, easier to remember, doesn't require import and it one "." away from giving me all I need to manipulate the file. ipython and ptpython already provide completion on file paths. From k7hoven at gmail.com Sun Mar 27 10:40:34 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Sun, 27 Mar 2016 17:40:34 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: On Sun, Mar 27, 2016 at 1:23 AM, Koos Zevenhoven wrote: > On Sat, Mar 26, 2016 at 9:10 PM, Brett Cannon wrote: > >> >> I also want to mention two things. One, pathlib.path is a thing now and >> something most people are probably not aware of as an alternative to doing >> `str(path)`: >> https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path . >> > > I assume you meant to type pathlib.Path.path, so that Path("...").path == > str(Path("...")). That's a good start, and I'm looking forward to Serhiy's > patch for making the stdlib accept Paths. But if Path will not subclass > str, we also need new stdlib functions that *return* Paths. > > Actually, now that .path is not out yet, would it make sense to call it Path.str or Path.strpath instead, and introduce the same thing on DirEntry and guarantee a str (instead of str or bytes as DirEntry.path now does)? Maybe that would lead to fewer broken implementations in third-party libraries too? - Koos -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sun Mar 27 13:36:26 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 28 Mar 2016 02:36:26 +0900 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> <56F71795.7040101@canterbury.ac.nz> Message-ID: <22264.6682.585845.192160@turnbull.sk.tsukuba.ac.jp> Koos Zevenhoven writes: > To be honest, I do think it feels like URL:s are becoming (or have become) > just as important as paths, and that pathlib.Path should in the future work > with URLs just like it now works with windows and posix paths. +1 ... to the concept, but: Is there a semantic difference between a RFC 3986 path component (Section 3.3), and a pathlib path? If there is, this could be a difficult project. (May as well start now, though!) From brett at python.org Sun Mar 27 13:50:38 2016 From: brett at python.org (Brett Cannon) Date: Sun, 27 Mar 2016 17:50:38 +0000 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: On Sat, 26 Mar 2016 at 16:23 Koos Zevenhoven wrote: > On Sat, Mar 26, 2016 at 9:10 PM, Brett Cannon wrote: > >> >> >> On Fri, 25 Mar 2016 at 16:50 Greg Ewing >> wrote: >> >>> >>>On 24.03.2016 22:06, Koos Zevenhoven wrote: >>> >>> >>> >>>Or even better, that you could do p"filename.txt", which would give >>> you a >>> >>>Path string object. >>> >>> That would tie Path objects deeply into the parser and compiler, >>> which I'm not sure is a good idea. >>> >> >> I'll go one step further than Greg and say that it's not a good idea. To >> make something like this work you have two options. One is you do what >> importlib does and very carefully construct it to only rely on built-in >> modules (and if you look at >> https://hg.python.org/cpython/file/default/Lib/pathlib.py you will >> notice that is no where near true ATM). Two, is Python gets a two-tier >> syntax system for before and after pathlib is imported and available. That >> would mean that all the dependencies of pathlib would need to not use this >> syntax until it's bootstrapped in. If you notice, both solutions are >> extremely painful to implement and radiate well past the code in pathilb.py >> and would potentially become hard to maintain. >> > > As I understand it, this could be implemented by making the compiler > essentially turn p"/path/to/whatever" into something like > _make_path("/path/to/whatever"), where _make_path would be builtin and do > something like this. > > def _make_path(str_path): > import pathlib > return pathlib.Path(str_path) > > Unless of course Path could be modified to import its dependencies on > demand and put in builtins. Am I missing something crucial? > The trick is whether that import statement will work when you call it. Import is currently the only bit of syntax that relies on Python code in CPython to work and making that happen took a lot of work. If you add in this concept of p-strings then suddenly we have two pieces of syntax that require Python code to work and on top of it p-strings would depend on import but also that we would then have to make sure to not have import depend on p-strings. And on top of it we would have to freeze pathlib and all of its dependencies which may or may not be preferable. My point is that it's not straight-forward and this proposal will have to consider some technical difficulties involved with it if it happens to go forward. -Brett > > >> >> I also want to mention two things. One, pathlib.path is a thing now and >> something most people are probably not aware of as an alternative to doing >> `str(path)`: >> https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path . >> > > I assume you meant to type pathlib.Path.path, so that Path("...").path == > str(Path("...")). That's a good start, and I'm looking forward to Serhiy's > patch for making the stdlib accept Paths. But if Path will not subclass > str, we also need new stdlib functions that *return* Paths. > > - Koos > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sun Mar 27 22:53:51 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Sun, 27 Mar 2016 19:53:51 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: Message-ID: <-4131081583774748380@unknownmsgid> > > There is an open issue and I'm working on the patch. I'm going to make major path-related functions in the stdlib accepting the Path object in 3.6. This is great! Thanks Serhiy. > try: > path = path.path > except AttributeError: > pass > > The advantage is that this works not with concrete Path implementation, but with any third-party Path class and other path-related objects like DirEntry that provide the "path" attribute. This sounds like the way to go, also. Could have saved ourselves some discussion :-) -CHB From chris.barker at noaa.gov Sun Mar 27 23:26:51 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Sun, 27 Mar 2016 20:26:51 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> Message-ID: <-3740384837840340487@unknownmsgid> >> I think we should think about making the language more >> iterable-focused. > > The *language* has always been iterable-focused. Python doesn't have > any counting loops! Sure. > range() is a builtin (and a necessary one), But range used to produce a list. So for lips were, in the beginning, about iterating, yes, but iterating through a sequence, not an arbitrary iterables. I may be wrong here, but I started using Python in version 1.5, and I'm pretty sure the iterator protocol did not exist then -- I know for sure I learned about it far later. And aside from range, there is dict.keys, zip, etc.... > I think the problem is more on the user (and teacher) side. Well sure - mostly this is about how we present the language, and I at least am making that switch. > That thread never did present a real use case for > "empty:" with an iterator. I agree here, actually. >> for line in the_file[:10]: >> ... >> >> arrgg! files are not indexable!. > > No, they are *enumerable*: > > for i, line in enumerate(the_file): > if i >= 10: break > ... > > I guess your request to make iterators more friendly is a good part of > why enumerate() got promoted to builtin. Sure, but you have to admit that the slicing notation is a lot cleaner. And I don't WANT to enumerate -/ I want > >> So maybe there should be some ability to index / slice iterables? > > There's no way to index an iterator, except to enumerate it. Then, > "to memoize or not to memoize, that is the question." Slicing makes > more sense to me, but again the fact that your discarded data may or > may not be valuable means that you need to make a choice between > memoizing and not doing so. Putting that in the API is complexity, or > perhaps even complication. If you just want head or tail, then > takewhile or dropwhile from itertools is your friend. (I have no > opinion -- not even -0 -- on whether promoting those functions to > builtin is a good idea.) > >> But aside from that -- just the idea that looking at how to make >> iterable a more "natural" part of the language is a good thing. > > I think it's from Zen and the Art of Motorcycle Maintenance (though > Pirsig may have been quoting), but I once read the advice: "If you > want to paint a perfect painting, make yourself perfect and then paint > naturally." I think iterators are just "unnatural" to a lot of > people, and will remain unnatural until people evolve. Which they may > not! > > Real life "do until done" tasks are careers ("do ... until dead") or > have timeouts ("do n times: break if done else ..."). In computation > that would be analogous to scrolling a Twitter feed vs. grabbing a > pageful. In the context of this discussion, a feed is something you > wait for (and maybe timeout and complain to the operator if it blocks > too long), while you can apply len() to pages. And you know which is > which in all applications I've dealt with -- except for design of > abstract programming language facilities like "for ... in". The point > being that "real life" examples don't seem to help people's intuition > on the Python versions. > > > From chris.barker at noaa.gov Sun Mar 27 23:30:47 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Sun, 27 Mar 2016 20:30:47 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <-3740384837840340487@unknownmsgid> References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> Message-ID: <2861663221964797505@unknownmsgid> Oops, hit send by accident. > >>> for line in the_file[:10]: >>> ... >>> >>> arrgg! files are not indexable!. >> >> No, they are *enumerable*: >> >> for i, line in enumerate(the_file): >> if i >= 10: break >> ... >> >> I guess your request to make iterators more friendly is a good part of >> why enumerate() got promoted to builtin. > > Sure, but you have to admit that the slicing notation is a lot > cleaner. And I don't WANT to enumerate -/ I want I want to iterate only the first n items -- that may be a common enough use case for a concise notation. >> takewhile or dropwhile from itertools is your friend. (I have no >> opinion -- not even -0 -- on whether promoting those functions to >> builtin is a good idea.) Well this thread started with those ideas .... My point is still : making working with iteratables as easy and natural as sequences is a good direction to go. -CHB >> >>> But aside from that -- just the idea that looking at how to make >>> iterable a more "natural" part of the language is a good thing. >> >> I think it's from Zen and the Art of Motorcycle Maintenance (though >> Pirsig may have been quoting), but I once read the advice: "If you >> want to paint a perfect painting, make yourself perfect and then paint >> naturally." I think iterators are just "unnatural" to a lot of >> people, and will remain unnatural until people evolve. Which they may >> not! >> >> Real life "do until done" tasks are careers ("do ... until dead") or >> have timeouts ("do n times: break if done else ..."). In computation >> that would be analogous to scrolling a Twitter feed vs. grabbing a >> pageful. In the context of this discussion, a feed is something you >> wait for (and maybe timeout and complain to the operator if it blocks >> too long), while you can apply len() to pages. And you know which is >> which in all applications I've dealt with -- except for design of >> abstract programming language facilities like "for ... in". The point >> being that "real life" examples don't seem to help people's intuition >> on the Python versions. >> >> >> From leewangzhong+python at gmail.com Mon Mar 28 10:45:27 2016 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Mon, 28 Mar 2016 10:45:27 -0400 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <2861663221964797505@unknownmsgid> References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> <2861663221964797505@unknownmsgid> Message-ID: I'm still very uneasy about how slicing is usually random access, and doesn't change how it indexes its elements from repeated use. It means that you have something very different for the same syntax. (No interest in `my_iterator(:)`? It might actually be compatible with existing syntax.) What about IterTool(foo).islice(dropuntil_f, take_while_g, filter_h) Keeps it as a single call but with explicit call syntax. Won't most uses stick something complicated (= long) in the slice? On Mar 27, 2016 11:31 PM, "Chris Barker - NOAA Federal" < chris.barker at noaa.gov> wrote: > Oops, hit send by accident. > > > > >>> for line in the_file[:10]: > >>> ... > >>> > >>> arrgg! files are not indexable!. > >> > >> No, they are *enumerable*: > >> > >> for i, line in enumerate(the_file): > >> if i >= 10: break > >> ... > >> > >> I guess your request to make iterators more friendly is a good part of > >> why enumerate() got promoted to builtin. > > > > Sure, but you have to admit that the slicing notation is a lot > > cleaner. And I don't WANT to enumerate -/ I want > I want to iterate only the first n items -- that may be a common > enough use case for a concise notation. > > >> takewhile or dropwhile from itertools is your friend. (I have no > >> opinion -- not even -0 -- on whether promoting those functions to > >> builtin is a good idea.) > > Well this thread started with those ideas .... > > My point is still : making working with iteratables as easy and > natural as sequences is a good direction to go. > > -CHB > > > > >> > >>> But aside from that -- just the idea that looking at how to make > >>> iterable a more "natural" part of the language is a good thing. > >> > >> I think it's from Zen and the Art of Motorcycle Maintenance (though > >> Pirsig may have been quoting), but I once read the advice: "If you > >> want to paint a perfect painting, make yourself perfect and then paint > >> naturally." I think iterators are just "unnatural" to a lot of > >> people, and will remain unnatural until people evolve. Which they may > >> not! > >> > >> Real life "do until done" tasks are careers ("do ... until dead") or > >> have timeouts ("do n times: break if done else ..."). In computation > >> that would be analogous to scrolling a Twitter feed vs. grabbing a > >> pageful. In the context of this discussion, a feed is something you > >> wait for (and maybe timeout and complain to the operator if it blocks > >> too long), while you can apply len() to pages. And you know which is > >> which in all applications I've dealt with -- except for design of > >> abstract programming language facilities like "for ... in". The point > >> being that "real life" examples don't seem to help people's intuition > >> on the Python versions. > >> > >> > >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Mar 28 11:04:55 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Mar 2016 02:04:55 +1100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> <2861663221964797505@unknownmsgid> Message-ID: On Tue, Mar 29, 2016 at 1:45 AM, Franklin? Lee wrote: > I'm still very uneasy about how slicing is usually random access, and > doesn't change how it indexes its elements from repeated use. It means that > you have something very different for the same syntax. >>> range(10, 100)[25:35] range(35, 45) It's a slice. Whether it's random access or not is pretty much unrelated to slicing; you get a slice of the underlying object, whatever that is. A slice of a sliceable iterator should be a sliced iterator. ChrisA From chris.barker at noaa.gov Mon Mar 28 11:20:04 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 28 Mar 2016 08:20:04 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> Message-ID: <7169445218378847930@unknownmsgid> The OP's > 'string'.to_file('some_file') would just be io.write('string', > 'some_file'), and str.from_file('some_file') would be > io.read('some_file'). Nick's observation about the mental model may be correct ( though I don't think so, frankly), but if it is, then this isn't any better than: open("some_path", 'w').write(string) " I need to open a file to write something to disk" isn't any harder to grok than "I need the io module to write something to disk". It would be nice to not need the 'w' flag, though, but probably way too dangerous to simply make an file object writable if the write method is called! Though I think the way this thread is going is correct - quick reading and writing to disk requires a serialization format - making the string is a heavier lift than writing it to disk for anything but very simple text. -CHB > Granted, both functions are 3 lines of Python, > but they might well be used enough to be worth adding. > > -- > Zach > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From chris.barker at noaa.gov Mon Mar 28 11:30:09 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 28 Mar 2016 08:30:09 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: <-7249072436132249471@unknownmsgid> > On Mar 24, 2016, at 7:22 PM, Nick Coghlan : what if we had a JSON-based save builtin that wrote > UTF-8 encoded files based on json.dump()? I've been think about this for a while, but would rather have a "pyson" format -- I.e. Python literals, rather than JSON. This would preserve the tuple vs list and integer vs float distinction, and allow more options for dictionary keys.(and sets?). Granted, you'd lose the interoperability, but for the quick saving and loading of data, it'd be pretty nice. There is also JSON pickle: https://jsonpickle.github.io Though as I understand it, it has the same security issues as pickle. But could we make a not-quite-as-complete pickle-like protocol that could save and load arbitrary objects, without ever running arbitrary code? -CHB From desmoulinmichel at gmail.com Mon Mar 28 11:44:15 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 28 Mar 2016 17:44:15 +0200 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <-7249072436132249471@unknownmsgid> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <-7249072436132249471@unknownmsgid> Message-ID: <56F9514F.2090304@gmail.com> Le 28/03/2016 17:30, Chris Barker - NOAA Federal a ?crit : >> On Mar 24, 2016, at 7:22 PM, Nick Coghlan : what if we had a JSON-based save builtin that wrote >> UTF-8 encoded files based on json.dump()? > > I've been think about this for a while, but would rather have a > "pyson" format -- I.e. Python literals, rather than JSON. This would > preserve the tuple vs list and integer vs float distinction, and allow > more options for dictionary keys.(and sets?). > > Granted, you'd lose the interoperability, but for the quick saving and > loading of data, it'd be pretty nice. > > There is also JSON pickle: > > https://jsonpickle.github.io > > Though as I understand it, it has the same security issues as pickle. > > But could we make a not-quite-as-complete pickle-like protocol that > could save and load arbitrary objects, without ever running arbitrary > code? If it's for quick data saving, the security is not an issue since the data will never comes from an attacker if you do a quick script. For other needs, where security is an issue, having a oneliner to dump some serialization is not going to do much of a difference. > > -CHB > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From chris.barker at noaa.gov Mon Mar 28 11:45:33 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Mar 2016 08:45:33 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On Thu, Mar 24, 2016 at 7:22 PM, Nick Coghlan wrote: > One of the few downsides of Python's popularity as both a scripting > language and an app development language is that a lot of tutorials > are written for the latter, and in app development, relying on the GC > for external resource cleanup isn't a great habit to get into. no, it's not. though now that think about it, while I understand that context managers are great as a universal and flexible way to ,well, mange context, for file objects themselves: open('a_file', 'w').write(some_stuff) Is really nice -- while we don't want to make reference-counting garbage collection a required part of the language in general, I wonder if it would be practical to make it part of the spec for SOME objects -- i.e kind of like you can define a given object to be a context manager, you could define certain objects to clean up after themselves when they go out of scope. It would sure be nice is this example ;; it would buy us that simple syntax, the atomic operation, etc... I have not thought this through AT ALL, but I wonder if one could avoid requiring a reference counting system by catching at the parsing stage that that the object created is never assigned to anything -- and this you know it's going to go out of scope as soon as that line is finished processing. In fact, if you have a non-reference counting garbage collection system, it might be helpful to special case all objects with very short lives, so you don't pile up a bunch of temporaries, etc that have to be collected later.... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Mar 28 11:50:03 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Mar 2016 08:50:03 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <56F9514F.2090304@gmail.com> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <-7249072436132249471@unknownmsgid> <56F9514F.2090304@gmail.com> Message-ID: On Mon, Mar 28, 2016 at 8:44 AM, Michel Desmoulin wrote: > > If it's for quick data saving, the security is not an issue since the > data will never comes from an attacker if you do a quick script. > that's why we have pickle already -- but a nice human-readable and editable form would be nice... Also -- when it comes to security it's a tough one -- people DO start with one thing, thinking "I will always trust this source" (if they think about it at all), then later expand the system to be a web service, or .. and oops! > For other needs, where security is an issue, having a one liner to dump > some serialization is not going to do much of a difference. > no -- I kind of mixed topics here -- my "safe json serialization" would be for web services, configuration, etc -- where security matters, but quick one-liner access is not so important -- though why not have one thing for multiple uses? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Mar 28 11:55:38 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Mar 2016 02:55:38 +1100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On Tue, Mar 29, 2016 at 2:45 AM, Chris Barker wrote: > no, it's not. though now that think about it, while I understand that > context managers are great as a universal and flexible way to ,well, mange > context, for file objects themselves: > > open('a_file', 'w').write(some_stuff) > > Is really nice -- while we don't want to make reference-counting garbage > collection a required part of the language in general, I wonder if it would > be practical to make it part of the spec for SOME objects -- i.e kind of > like you can define a given object to be a context manager, you could define > certain objects to clean up after themselves when they go out of scope. It > would sure be nice is this example ;; it would buy us that simple syntax, > the atomic operation, etc... The whole point of the context manager is that it defines the notion "go out of scope". But if you want the simple syntax, the easiest way is to create a helper: def write_file(filename, contents): with open(filename, "w") as f: f.write(contents) write_file('a_file', some_stuff) Or, if you're writing a quick one-off script, just use the line you have above, and don't worry about guaranteed closing. What's the worst that can happen? AIUI buffered data will be written out at process termination if not before, and of course the file will be closed. The main case where you absolutely must close the file promptly is when you plan to then open it again (in the same or another process), and a quick script will usually know if that's going to be a possibility. (How bad can it be? I suppose https://xkcd.com/292/ level bad. But not usually.) ChrisA From vgr255 at live.ca Mon Mar 28 11:54:52 2016 From: vgr255 at live.ca (=?UTF-8?Q?=C3=89manuel_Barry?=) Date: Mon, 28 Mar 2016 11:54:52 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: I wonder how reasonable it would be to add a new keyword to open that would .close() the file object upon a single write/read. Consider: data = open("foo.txt", "r", close_after_use=True).read() It doesn?t rely on garbage collection to work properly, is a fancy one-liner, and is obvious that you can only read or write from/to it once. Could use a better keyword for that, though. -Emanuel ~ If it doesn?t quack like a duck, add a quack() method ~ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Mar 28 12:13:22 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Mar 2016 09:13:22 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> <2861663221964797505@unknownmsgid> Message-ID: On Mon, Mar 28, 2016 at 7:45 AM, Franklin? Lee < leewangzhong+python at gmail.com> wrote: > I'm still very uneasy about how slicing is usually random access, and > doesn't change how it indexes its elements from repeated use. It means that > you have something very different for the same syntax. > I think only slightly different :-) On Mon, Mar 28, 2016 at 8:04 AM, Chris Angelico wrote: > >>> range(10, 100)[25:35] > range(35, 45) > > It's a slice. Whether it's random access or not is pretty much > unrelated to slicing; you get a slice of the underlying object, > whatever that is. A slice of a sliceable iterator should be a sliced > iterator. > sure -- but that works great for range, because is is an lazy evaluated sequence, which of course makes it iterable, but I think the trick is: arbitrary_iterable[10:20] should return an iterable that will return the 10th to the 20th item when iterated -- doable, but: for i in arbitrary_iterable[10:20]: pass will then have to call the underlying iterable 20 times -- if it's an iterator that isn't a sequence, its state will have been altered. so I'm not sure how to go about this -- but it would be nice. Also, arbitrary_iterable[-10:] would be essentially impossible. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 28 12:20:11 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Mar 2016 09:20:11 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <-7249072436132249471@unknownmsgid> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <-7249072436132249471@unknownmsgid> Message-ID: <56F959BB.1090001@stoneleaf.us> On 03/28/2016 08:30 AM, Chris Barker - NOAA Federal wrote: > But could we make a not-quite-as-complete pickle-like protocol that > could save and load arbitrary objects, without ever running arbitrary > code? We could use `ast.literal_eval` to parse the file. -- ~Ethan~ From chris.barker at noaa.gov Mon Mar 28 12:20:18 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Mar 2016 09:20:18 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On Mon, Mar 28, 2016 at 8:55 AM, Chris Angelico wrote: > The whole point of the context manager is that it defines the notion > "go out of scope". of course it does -- very useful but a lot of extra for a siple read this file then close it right away, for the casual scripting user :-) > But if you want the simple syntax, the easiest way > is to create a helper: > > def write_file(filename, contents): > with open(filename, "w") as f: > f.write(contents) > > write_file('a_file', some_stuff) > So I guess this comes around to the OPs point -- a helper like that should be built in, and easy and obvious to scripting user. I don't think it should be astring method, but... Or, if you're writing a quick one-off script, just use the line you have above, and don't worry about guaranteed closing. And that comes back around to my original post on this thread -- we should just tell scripting users to do that :-) but I think it was Nick's point that Python is used for scripting and "real" system development, so it's hard to communicate best practices for both... given that, I'm kind of liking "write_file" It solves Nick's mental model issue" " want to write some stuff to a file", rather than "I want to create a file object, then write stuff to it" -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 28 12:23:14 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Mar 2016 09:23:14 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <7169445218378847930@unknownmsgid> References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <7169445218378847930@unknownmsgid> Message-ID: <56F95A72.2080009@stoneleaf.us> On 03/28/2016 08:20 AM, Chris Barker - NOAA Federal wrote: > The OP's >> 'string'.to_file('some_file') would just be io.write('string', >> 'some_file'), and str.from_file('some_file') would be >> io.read('some_file'). > > Nick's observation about the mental model may be correct ( though I > don't think so, frankly), but if it is, then this isn't any better > than: > > open("some_path", 'w').write(string) > > " I need to open a file to write something to disk" isn't any harder > to grok than "I need the io module to write something to disk". The benefit here is that the `io` module is a natural place to put a file `read` and `write` function, and they don't need to be built-in. -- ~Ethan~ From rosuav at gmail.com Mon Mar 28 12:22:56 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Mar 2016 03:22:56 +1100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> <2861663221964797505@unknownmsgid> Message-ID: On Tue, Mar 29, 2016 at 3:13 AM, Chris Barker wrote: > On Mon, Mar 28, 2016 at 8:04 AM, Chris Angelico wrote: > >> >> >>> range(10, 100)[25:35] >> range(35, 45) >> >> It's a slice. Whether it's random access or not is pretty much >> unrelated to slicing; you get a slice of the underlying object, >> whatever that is. A slice of a sliceable iterator should be a sliced >> iterator. > > > sure -- but that works great for range, because is is an lazy evaluated > sequence, which of course makes it iterable, but I think the trick is: > > > arbitrary_iterable[10:20] > > should return an iterable that will return the 10th to the 20th item when > iterated -- doable, but: > > for i in arbitrary_iterable[10:20]: > pass > > will then have to call the underlying iterable 20 times -- if it's an > iterator that isn't a sequence, its state will have been altered. > > so I'm not sure how to go about this -- but it would be nice. I don't think arbitrary iterables should be sliceable. Arbitrary *iterators* can be, because that operation has well-defined semantics (just tie in with itertools.islice); the state of the underlying iterator *will* be changed. > Also, arbitrary_iterable[-10:] > > would be essentially impossible. I suppose it could be something like: iter(collections.deque(arbitrary_iterable, maxlen=10)) but that's not what people will normally expect. Much better for various iterable types to define their own slice handling. ChrisA From desmoulinmichel at gmail.com Mon Mar 28 12:23:28 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 28 Mar 2016 18:23:28 +0200 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> <2861663221964797505@unknownmsgid> Message-ID: <56F95A80.2090709@gmail.com> Le 28/03/2016 18:13, Chris Barker a ?crit : > On Mon, Mar 28, 2016 at 7:45 AM, Franklin? Lee > > > wrote: > > I'm still very uneasy about how slicing is usually random access, > and doesn't change how it indexes its elements from repeated use. It > means that you have something very different for the same syntax. > > I think only slightly different :-) > > On Mon, Mar 28, 2016 at 8:04 AM, Chris Angelico > wrote: > > > >>> range(10, 100)[25:35] > range(35, 45) > > It's a slice. Whether it's random access or not is pretty much > unrelated to slicing; you get a slice of the underlying object, > whatever that is. A slice of a sliceable iterator should be a sliced > iterator. > > > sure -- but that works great for range, because is is an lazy evaluated > sequence, which of course makes it iterable, but I think the trick is: > > > arbitrary_iterable[10:20] > > should return an iterable that will return the 10th to the 20th item > when iterated -- doable, but: > > for i in arbitrary_iterable[10:20]: > pass > > will then have to call the underlying iterable 20 times -- if it's an > iterator that isn't a sequence, its state will have been altered. > > so I'm not sure how to go about this -- but it would be nice. > > Also, arbitrary_iterable[-10:] > > would be essentially impossible. It would be possible but would basically be equivalent to list(arbitrary_iterable)[-10:] and would load everything in memory, at best rendering generators useless, at worst exploding on a infinite data stream. But I think it would be ok to raise ValueError any negative indices by default, and then let containers such as list/tuple define custom behavior for them. Note that arbitrary_iterable[:-10] or arbitrary_iterable[-10] would be possibl, with collections.deque you buffer at most 10 elements in memory, but I'm not sure if it would be preferable to fordid all negative values, or have an special case. It's handy but confusing. Also, what do you think about the callable passed as an index from the first examples ? > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From chris.barker at noaa.gov Mon Mar 28 12:26:04 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Mar 2016 09:26:04 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On Mon, Mar 28, 2016 at 8:54 AM, ?manuel Barry wrote: > I wonder how reasonable it would be to add a new keyword to open that > would .close() the file object upon a single write/read. Consider: > > > > data = open("foo.txt", "r", close_after_use=True).read() > > it makes it a tiny bit shorter than using "with", but doesn't solve Nick's mental model issue -- the user still needs to be thinking about the fact that they are creating a file object and that it needs to be closed when you are done with it. and I"m not sure how you would define "use" -- any i.o. opeartion? i.e.: infile = open("foo.txt", "r", close_after_use=True) first_line = infile.readline() now what is the state of infile? a closed file object? exactly why context managers were introduced: with open("foo.txt", "r", close_after_use=True) as infile: first_line = infile.readline() something_else... now it's pretty clear that infile is no longer a useful object. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 28 12:29:27 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Mar 2016 09:29:27 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> <2861663221964797505@unknownmsgid> Message-ID: <56F95BE7.7080607@stoneleaf.us> On 03/28/2016 09:13 AM, Chris Barker wrote: > On Mon, Mar 28, 2016 at 7:45 AM, Franklin? Lee wrote: > >> I'm still very uneasy about how slicing is usually random access, >> and doesn't change how it indexes its elements from repeated use. It >> means that you have something very different for the same syntax. > > I think only slightly different :-) > On Mon, Mar 28, 2016 at 8:04 AM, Chris Angelico wrote: > >> >>> range(10, 100)[25:35] >> range(35, 45) >> >> It's a slice. Whether it's random access or not is pretty much >> unrelated to slicing; you get a slice of the underlying object, >> whatever that is. A slice of a sliceable iterator should be a sliced >> iterator. > > sure -- but that works great for range, because is is an lazy evaluated > sequence, which of course makes it iterable, but I think the trick is: > > arbitrary_iterable[10:20] > > should return an iterable that will return the 10th to the 20th item > when iterated Iter*ables* are easy to slice (think list, tuple, range, etc.). Iter*ators* are the tricky ones. And yes, there is no way to slice an iterator without changing its state. -- ~Ethan~ From ethan at stoneleaf.us Mon Mar 28 12:32:01 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Mar 2016 09:32:01 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F95A80.2090709@gmail.com> References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> <2861663221964797505@unknownmsgid> <56F95A80.2090709@gmail.com> Message-ID: <56F95C81.2000501@stoneleaf.us> On 03/28/2016 09:23 AM, Michel Desmoulin wrote: > It would be possible but would basically be equivalent to > list(arbitrary_iterable)[-10:] and would load everything in memory, at > best rendering generators useless, at worst exploding on a infinite data > stream. > > But I think it would be ok to raise ValueError any negative indices by > default, and then let containers such as list/tuple define custom > behavior for them. Agreed. And already the default behaviour. > Also, what do you think about the callable passed as an index from the > first examples ? I like it, but it will be a pain to implement. (At least, it would be for me. ;) -- ~Ethan~ From vgr255 at live.ca Mon Mar 28 12:32:00 2016 From: vgr255 at live.ca (=?UTF-8?Q?=C3=89manuel_Barry?=) Date: Mon, 28 Mar 2016 12:32:00 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On Mon, Mar 28, 2016 at 8:54 AM, ?manuel Barry < vgr255 at live.ca> wrote: I wonder how reasonable it would be to add a new keyword to open that would .close() the file object upon a single write/read. Consider: data = open("foo.txt", "r", close_after_use=True).read() it makes it a tiny bit shorter than using "with", but doesn't solve Nick's mental model issue -- the user still needs to be thinking about the fact that they are creating a file object and that it needs to be closed when you are done with it. and I"m not sure how you would define "use" -- any i.o. opeartion? i.e.: infile = open("foo.txt", "r", close_after_use=True) first_line = infile.readline() now what is the state of infile? a closed file object? Sure. The keyword really could benefit from a better name, but it might solve some part of the issue. It doesn?t solve the mental model issue though, but to be fair, open() should always be wrapped in a context manager (IMO, anyway). -Emanuel exactly why context managers were introduced: with open("foo.txt", "r", close_after_use=True) as infile: first_line = infile.readline() something_else... now it's pretty clear that infile is no longer a useful object. -CHB -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Mar 28 12:34:09 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Mar 2016 03:34:09 +1100 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <56F95BE7.7080607@stoneleaf.us> References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> <2861663221964797505@unknownmsgid> <56F95BE7.7080607@stoneleaf.us> Message-ID: On Tue, Mar 29, 2016 at 3:29 AM, Ethan Furman wrote: > Iter*ables* are easy to slice (think list, tuple, range, etc.). > > Iter*ators* are the tricky ones. And yes, there is no way to slice an > iterator without changing its state. *Some* iterables are easy to slice. Some are not. A dictionary is perfectly iterable, but it doesn't make a lot of sense to slice it. Are you slicing based on keys, or sequence (like paginating the dict), or something else? That's why iterable slicing needs to be handled by the type itself - a range knows how to produce a subrange, a list knows how to produce a sublist. ChrisA From ethan at stoneleaf.us Mon Mar 28 12:38:31 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Mar 2016 09:38:31 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: <56F95E07.10102@stoneleaf.us> On 03/28/2016 09:32 AM, ?manuel Barry wrote: > On Mon, Mar 28, 2016 at 8:54 AM, ?manuel Barry wrote: >> >> I wonder how reasonable it would be to add a new keyword to open >> that would .close() the file object upon a single write/read. Consider: >> >> data = open("foo.txt", "r", close_after_use=True).read() > > it makes it a tiny bit shorter than using "with", but doesn't solve > Nick's mental model issue -- the user still needs to be thinking about > the fact that they are creating a file object and that it needs to be > closed when you are done with it. > > and I"m not sure how you would define "use" -- any i.o. opeartion? i.e.: > > infile = open("foo.txt", "r", close_after_use=True) > first_line = infile.readline() > > now what is the state of infile? a closed file object? Good point. At best this would be an attractive nuisance. Better to have the `io.read` and `io.write` short cuts. -- ~Ethan~ From chris.barker at noaa.gov Mon Mar 28 12:30:56 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Mar 2016 09:30:56 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <56F95A72.2080009@stoneleaf.us> References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <7169445218378847930@unknownmsgid> <56F95A72.2080009@stoneleaf.us> Message-ID: On Mon, Mar 28, 2016 at 9:23 AM, Ethan Furman wrote: > 'string'.to_file('some_file') would just be io.write('string', >>> 'some_file'), and str.from_file('some_file') would be >>> io.read('some_file'). >>> >> >> Nick's observation about the mental model may be correct ( though I >> don't think so, frankly), but if it is, then this isn't any better >> than: >> >> open("some_path", 'w').write(string) >> >> " I need to open a file to write something to disk" isn't any harder >> to grok than "I need the io module to write something to disk". >> > > The benefit here is that the `io` module is a natural place to put a file > `read` and `write` function, and they don't need to be built-in. I'm coming around to this actually -- not because it's easier to grok than: data = open('some_file').read() but because it could be an atomic operation, and close the file properly. and Is easier to grok than: with open('some_file') as infile: data = infile.read() and heck, it would even keep the door open to being made a built-in in the future. Maybe it's time to hear from the OP on this one -- after all, most of us in the discussion are already really familiar with the options! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Mar 28 12:40:29 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 28 Mar 2016 09:40:29 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On Mon, Mar 28, 2016 at 9:32 AM, ?manuel Barry wrote: > and I"m not sure how you would define "use" -- any i.o. opeartion? i.e.: > > > > infile = open("foo.txt", "r", close_after_use=True) > first_line = infile.readline() > > > > now what is the state of infile? a closed file object? > > Sure. The keyword really could benefit from a better name, but it might > solve some part of the issue. It doesn?t solve the mental model issue > though, but to be fair, open() should always be wrapped in a context > manager (IMO, anyway). > then we don't need anyting else :-) but the name of the keyword is not that point here -- the issue is what does it mean to "use" a file object? using "with" let's the user clearly define, when they are done with the object. and if a reference to the object is not stored anywhere, then you can be sure the user doesn't expect to be abel to use it again. but: data = open("foo.txt", "r", close_after_use=True).read() and infile = open("foo.txt", "r", close_after_use=True) data = infile.read() look exactly the same to the file object itself, it has no idea when the user is done with it. in cPython, the reference counter knows that teh file object has no references to it when that first line is done running, so it can clean up and delete the object -- but without a reference counting system, who knows when it will get cleaned up? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 28 12:45:23 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Mar 2016 09:45:23 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> <2861663221964797505@unknownmsgid> <56F95BE7.7080607@stoneleaf.us> Message-ID: <56F95FA3.4020000@stoneleaf.us> On 03/28/2016 09:34 AM, Chris Angelico wrote: > On Tue, Mar 29, 2016 at 3:29 AM, Ethan Furman wrote: >> Iter*ables* are easy to slice (think list, tuple, range, etc.). >> >> Iter*ators* are the tricky ones. And yes, there is no way to slice an >> iterator without changing its state. > > *Some* iterables are easy to slice. Some are not. A dictionary is > perfectly iterable, but it doesn't make a lot of sense to slice it. > Are you slicing based on keys, or sequence (like paginating the dict), > or something else? That's why iterable slicing needs to be handled by > the type itself - a range knows how to produce a subrange, a list > knows how to produce a sublist. From the glossary [https://docs.python.org/3/glossary.html] iterable An object capable of returning its members one at a time. Very good point, Chris, thanks for the correction. So the key point then is that objects with a `__getitem__` that is based on offsets `0` to `len(obj)-1` are easily sequenced, all others may as well be wrapped in `iter()` calls. -- ~Ethan~ From alexander.belopolsky at gmail.com Mon Mar 28 13:24:54 2016 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 28 Mar 2016 13:24:54 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: On Mon, Mar 28, 2016 at 12:40 PM, Chris Barker wrote: > but: > > data = open("foo.txt", "r", close_after_use=True).read() > > and > > infile = open("foo.txt", "r", close_after_use=True) > data = infile.read() > > look exactly the same to the file object itself, it has no idea when the > user is done with it. It is my understanding that under the "close_after_use" proposal, infile will be closed by .read() in both cases. Still, I don't like this idea. I puts action specification (close) too far from where the action is taken. The two-line example already makes me uncomfortable and imagine if infile is passed through several layers of function calls before infile.read() is called. I would rather see read_and_close() and write_and_close() convenience methods for file objects: data = open("foo.txt").read_and_close() -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Mar 28 13:35:15 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 28 Mar 2016 18:35:15 +0100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <7169445218378847930@unknownmsgid> <56F95A72.2080009@stoneleaf.us> Message-ID: On 28 March 2016 at 17:30, Chris Barker wrote: > and heck, it would even keep the door open to being made a built-in in the > future. What's the huge advantage of being a builtin? Sure, I can understand the benefit of wanting it in the stdlib over being a 3rd party dependency or a "simple recipe you can add to your own code"[1], but what is the huge deal about avoiding an import? If it's about interactive use, you have PYTHONSTARTUP, or the config options of your environment of choice (for example, IPython). For scripts, a one-line import is certainly not something to worry about. And for teaching, you can't get very far in Python without knowing about imports, so it's not like you're needing an advanced concept here. Paul From nickeubank at gmail.com Mon Mar 28 14:16:31 2016 From: nickeubank at gmail.com (Nick Eubank) Date: Mon, 28 Mar 2016 18:16:31 +0000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: Just to make sure I follow: Chris, you're proposing a modification to garbage collection to clean up `open()` if `open()` is used by not closed and not assigned to anything so that `open(file).write(stuff)` is safe? That sounds brilliant to me, and totally meets my goal of having a "one-step" method for writing to disk that doesn't require users to develop a two-step mental model. If it's feasible (I have nothing but the most basic understanding of how the garbage collector works), that would totally obviate my desire to have a `to_file()` method on strings. On Mon, Mar 28, 2016 at 8:46 AM Chris Barker wrote: > On Thu, Mar 24, 2016 at 7:22 PM, Nick Coghlan wrote: > >> One of the few downsides of Python's popularity as both a scripting >> language and an app development language is that a lot of tutorials >> are written for the latter, and in app development, relying on the GC >> for external resource cleanup isn't a great habit to get into. > > > no, it's not. though now that think about it, while I understand that > context managers are great as a universal and flexible way to ,well, mange > context, for file objects themselves: > > open('a_file', 'w').write(some_stuff) > > Is really nice -- while we don't want to make reference-counting garbage > collection a required part of the language in general, I wonder if it would > be practical to make it part of the spec for SOME objects -- i.e kind of > like you can define a given object to be a context manager, you could > define certain objects to clean up after themselves when they go out of > scope. It would sure be nice is this example ;; it would buy us that simple > syntax, the atomic operation, etc... > > I have not thought this through AT ALL, but I wonder if one could avoid > requiring a reference counting system by catching at the parsing stage that > that the object created is never assigned to anything -- and this you know > it's going to go out of scope as soon as that line is finished processing. > > In fact, if you have a non-reference counting garbage collection system, > it might be helpful to special case all objects with very short lives, so > you don't pile up a bunch of temporaries, etc that have to be collected > later.... > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 28 14:30:45 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Mar 2016 11:30:45 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <7169445218378847930@unknownmsgid> <56F95A72.2080009@stoneleaf.us> Message-ID: <56F97855.6020105@stoneleaf.us> On 03/28/2016 10:35 AM, Paul Moore wrote: > On 28 March 2016 at 17:30, Chris Barker wrote: >> and heck, it would even keep the door open to being made a built-in in the >> future. > > What's the huge advantage of being a builtin? Sure, I can understand > the benefit of wanting it in the stdlib over being a 3rd party > dependency or a "simple recipe you can add to your own code"[1], but > what is the huge deal about avoiding an import? Personally, I have no idea. The way some folks are talking about how hard typing in `import blah` is, you'd think they had to write a metaclass to get the job done. __prepare__-__new__-__init__-ly yrs -- ~Ethan~ From desmoulinmichel at gmail.com Mon Mar 28 14:31:20 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 28 Mar 2016 20:31:20 +0200 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <56F96F46.5020306@gmail.com> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <56F96F46.5020306@gmail.com> Message-ID: <56F97878.1020503@gmail.com> Woops, sent it to only one person. Le 28/03/2016 19:52, Michel Desmoulin a ?crit : > > > Le 28/03/2016 19:24, Alexander Belopolsky a ?crit : >> >> On Mon, Mar 28, 2016 at 12:40 PM, Chris Barker > > wrote: >> >> but: >> >> data = open("foo.txt", "r", close_after_use=True).read() >> >> and >> >> infile = open("foo.txt", "r", close_after_use=True) >> data = infile.read() >> >> look exactly the same to the file object itself, it has no idea when >> the user is done with it. >> >> >> It is my understanding that under the "close_after_use" proposal, infile >> will be closed by .read() in both cases. >> >> Still, I don't like this idea. I puts action specification (close) too >> far from where the action is taken. The two-line example already makes >> me uncomfortable and imagine if infile is passed through several layers >> of function calls before infile.read() is called. >> >> I would rather see read_and_close() and write_and_close() convenience >> methods for file objects: >> >> data = open("foo.txt").read_and_close() > > All those area already part of the stdlib though: > >>>> import pathlib >>>> pathlib.Path('/tmp/foo').write_text('bar') > 3 >>>> pathlib.Path('/tmp/foo').read_text() > 'bar' > > Hence the other discussion about making a p-string, so you can have Path > object literal notation and do: > >>>> p'/tmp/foo'.write_text('bar') > > An no imports. > > I like the idea of having path literals (alhough I wish for a much > improved pathlib), it would be handy for scripting, shell sessions, data > mangling, etc. > > But progamming is now less and less about files : web API, phones APIS, > DB, ESB... So I'm wondering if it's not trying to fight a century old > battle. > > >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> From tjreedy at udel.edu Mon Mar 28 17:46:27 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 28 Mar 2016 17:46:27 -0400 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: <-3740384837840340487@unknownmsgid> References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> Message-ID: On 3/27/2016 11:26 PM, Chris Barker - NOAA Federal wrote: > But range used to produce a list. So for lips were, in the beginning, > about iterating, yes, but iterating through a sequence, not an > arbitrary iterables. I may be wrong here, but I started using Python > in version 1.5, and I'm pretty sure the iterator protocol did not > exist then -- I know for sure I learned about it far later. The current iterator protocol, based on __iter__, __next__, and StopIteration, was added in 2.2. The original iteration protocol, now called the 'sequence protocol', was based on __getitem__(count) and IndexError. Except for not requiring a __length__ method, it required that what we now call an 'iterable' pretend to be a sequence. It is still supported by iter(). https://docs.python.org/3/library/functions.html#iter With the additional changes to builtins in 3.0, there has definitely been a shift of emphasis from lists and sequences to iterables and iterators. --- General response to thread: In spite of this, I am not in favor of trying to force-fit itertools, in particular islice, into syntax. Iterables for which slicing makes sense are free to directly support non-destructive slicing in a __getitem__ method. It does not bother me that destructive slicing of iterators requires a different syntax. Because of the difference between non-destructive and destructive slicing, the goal of making the same code work for sequences and iterators cannot work beyond a single slice. Suppose one wants to process the first 5 and then the next 5 items of an iterable. The necessary code is different for sequences and iterators. from itertools import islice rs = range(10) ri = iter(rs) print(list(islice(rs, 0, 5))) print(list(islice(rs, 0, 5))) print() print(list(islice(ri, 0, 5))) print(list(islice(ri, 0, 5))) # works for iterators print() ri = iter(rs) print(list(islice(rs, 0, 5))) print(list(islice(rs, 5, 10))) # works for sequences print() print(list(islice(ri, 0, 5))) print(list(islice(ri, 5, 10))) # [0, 1, 2, 3, 4] [0, 1, 2, 3, 4] [0, 1, 2, 3, 4] [5, 6, 7, 8, 9] [0, 1, 2, 3, 4] [5, 6, 7, 8, 9] [0, 1, 2, 3, 4] [] How anyone written and experimented with using a 'syntax-iterator' class to test the idea? Has such to uploaded to PyPI? import itertools class Iter: # syntax-intergrated iterator def __init__(self, iterable): self.it = iter(iterable) def __iter__(self): return self def __next__(self): return next(self.it) def __getitem__(self, what): if isistance(what, slice): return itertools.islice( what.start or 0, what.stop, what.step or 1) def __add__(self, other): return itertools.chain(self, other) # for more that two iterables, chain(a, b, c, ...) will run faster ... Then, in the current idiom for a function of an iterable: def f(args, iterable, args): it = iter(iterable) one could replace 'iter' with 'Iter' and proceed to use proposed iterator syntax within the function. -- Terry Jan Reedy From chris.barker at noaa.gov Mon Mar 28 20:27:57 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 28 Mar 2016 17:27:57 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <7169445218378847930@unknownmsgid> <56F95A72.2080009@stoneleaf.us> Message-ID: <-4482701274830018179@unknownmsgid> > What's the huge advantage of being a builtin? Sure, I can understand > the benefit of wanting it in the stdlib over being a 3rd party > dependency or a "simple recipe you can add to your own code"[1], but > what is the huge deal about avoiding an import? Well, I'm not the OP here, but: > If it's about interactive use, you have PYTHONSTARTUP, or the config > options of your environment of choice (for example, IPython). Well, I find I want to cut and paste code to-from interactive environment enough, and use enough different systems, that I never customize my environment like that. > For > scripts, a one-line import is certainly not something to worry about. > And for teaching, you can't get very far in Python without knowing > about imports, so it's not like you're needing an advanced concept > here. No a very big deal, no -- but neither is using with .... :-) So forget I said that, and let's consider putting it in the io module. -CHB > > Paul From chris.barker at noaa.gov Mon Mar 28 20:48:58 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 28 Mar 2016 17:48:58 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: <6264907126635056595@unknownmsgid> Just to make sure I follow: Chris, you're proposing a modification to garbage collection to clean up `open()` if `open()` is used by not closed and not assigned to anything so that `open(file).write(stuff)` is safe? Well, I wouldn't call it a proposal, more a random inspiration that popped up when thinking about this thread. That sounds brilliant to me, and totally meets my goal of having a "one-step" method for writing to disk that doesn't require users to develop a two-step mental model. If it's feasible (I have nothing but the most basic understanding of how the garbage collector works), that would totally obviate my desire to have a `to_file()` method on strings. IIUC, in the current cPython implementation, you do indeed have that -- I've used it for years, long before context managers existed, and still do for quickie scripts. CPython uses a reference counting scheme: each time an object is referenced, its count is increased, each lost reference, and it is decreased. When the count goes to zero, the object is deleted. So in: Data = open(file name).read() The file object is created, given a refount of one. Then the read() method is called, creating a string, then bound to the name Data. When the next line is reached, the recount of the file object is reduced to zero, and the object is deleted. And the internal file pointer is closed before deleting the object. The "trick" is that the Python language spec does not require this kind of garbage collection. So jython, or pypy, or ironPython may not clean up that file object right away, it could hang around in an open and unflushed state until the garbage collector gets around to cleaning it up. But for short scripts, it'll get cleaned up at the end of the script anyway. The thing is that while I, at least, think it's a fine practice for scripting, it's really not a good idea for larger system development. Thus the general advise to use "with". As for any proposal, it dawned on me that while we don't want Python to require any particular garbage collecting scheme, I wonder if it would be possible (or desirable) to specify that temporary objects used on one line of code, and never referenced any other way, get deleted right away. It's actually very common to create a lot of temporaries on a line of code -- when you chain operations or methods. So it might be a small performance tweak worth doing (or it may not :-) ) -CHB -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Mon Mar 28 21:04:21 2016 From: mike at selik.org (Michael Selik) Date: Tue, 29 Mar 2016 01:04:21 +0000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <6264907126635056595@unknownmsgid> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <6264907126635056595@unknownmsgid> Message-ID: Brevity may be the soul of wit, but I am completely happy without a convenience function to open and read/write, then close a file. In production, the ``with`` statement seems concise enough. If I'm messing around at the prompt we can already write ``text = open(filename).read()``. We can also write ``open(filename, 'w').write(text)`` and not worry about closing the file in the basic Python prompt. On Mon, Mar 28, 2016 at 8:49 PM Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > > Just to make sure I follow: Chris, you're proposing a modification to > garbage collection to clean up `open()` if `open()` is used by not closed > and not assigned to anything so that `open(file).write(stuff)` is safe? > > > Well, I wouldn't call it a proposal, more a random inspiration that popped > up when thinking about this thread. > > That sounds brilliant to me, and totally meets my goal of having a > "one-step" method for writing to disk that doesn't require users to develop > a two-step mental model. If it's feasible (I have nothing but the most > basic understanding of how the garbage collector works), that would totally > obviate my desire to have a `to_file()` method on strings. > > > IIUC, in the current cPython implementation, you do indeed have that -- > I've used it for years, long before context managers existed, and still do > for quickie scripts. > > CPython uses a reference counting scheme: each time an object is > referenced, its count is increased, each lost reference, and it is > decreased. When the count goes to zero, the object is deleted. So in: > > Data = open(file name).read() > > The file object is created, given a refount of one. Then the read() method > is called, creating a string, then bound to the name Data. When the next > line is reached, the recount of the file object is reduced to zero, and the > object is deleted. And the internal file pointer is closed before deleting > the object. > > The "trick" is that the Python language spec does not require this kind of > garbage collection. So jython, or pypy, or ironPython may not clean up that > file object right away, it could hang around in an open and unflushed state > until the garbage collector gets around to cleaning it up. > > But for short scripts, it'll get cleaned up at the end of the script > anyway. > > The thing is that while I, at least, think it's a fine practice for > scripting, it's really not a good idea for larger system development. > > Thus the general advise to use "with". > > As for any proposal, it dawned on me that while we don't want Python to > require any particular garbage collecting scheme, I wonder if it would be > possible (or desirable) to specify that temporary objects used on one line > of code, and never referenced any other way, get deleted right away. > > It's actually very common to create a lot of temporaries on a line of code > -- when you chain operations or methods. So it might be a small performance > tweak worth doing (or it may not :-) ) > > -CHB > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 28 22:21:47 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 28 Mar 2016 19:21:47 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> Message-ID: <56F9E6BB.6040400@stoneleaf.us> [Sorry, Nick, didn't mean to just send it to you.] On 03/28/2016 11:16 AM, Nick Eubank wrote: > Just to make sure I follow: Chris, you're proposing a modification to > garbage collection to clean up `open()` if `open()` is used by not > closed and not assigned to anything so that `open(file).write(stuff)` is > safe? Even if he is, I don't see it happening. Modifying the garbage collector to act sooner on one particular type of object when we already have good, if very mildly inconvenient, solutions is not going to be a high-priority item -- especially if it has /any/ negative impact on performance. -- ~Ethan~ From srkunze at mail.de Tue Mar 29 03:11:37 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 09:11:37 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> Message-ID: <56FA2AA9.5050102@mail.de> On 26.03.2016 19:17, Koos Zevenhoven wrote: > On Sat, Mar 26, 2016 at 7:32 PM, Stephen J. Turnbull > > wrote: > > Koos Zevenhoven writes: > > > How about generalizing f-strings, or perhaps giving p-strings (some > of) the power of f-strings? > > That'll be a good idea. > Just their combination: pf"{ path_as_string }" ? That would at least > be very consistent. Could be make implicit: p'{my_variable}' -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Mar 29 03:22:44 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 09:22:44 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA2AA9.5050102@mail.de> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> Message-ID: <56FA2D44.8080707@mail.de> On 29.03.2016 09:11, Sven R. Kunze wrote: > On 26.03.2016 19:17, Koos Zevenhoven wrote: >> On Sat, Mar 26, 2016 at 7:32 PM, Stephen J. Turnbull >> wrote: >> >> Koos Zevenhoven writes: >> >> >> How about generalizing f-strings, or perhaps giving p-strings (some >> of) the power of f-strings? >> >> > > That'll be a good idea. > >> Just their combination: pf"{ path_as_string }" ? That would at least >> be very consistent. > > Could be make implicit: p'{my_variable}' Thinking more about it, that would even come in nicely with path separators: p'/{whereever}/{youwant}/{togo}' That'll finally an easy-to-write and readable path generation. Great idea, Stephen. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Mar 29 03:25:11 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 09:25:11 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F71D92.9030605@canterbury.ac.nz> References: <56F68406.3030103@sdamon.com> <56F71D92.9030605@canterbury.ac.nz> Message-ID: <56FA2DD7.5070206@mail.de> On 27.03.2016 00:38, Greg Ewing wrote: > I'm wondering how often literal pathames are really used > in actual programs. Don't underestimate what made Python great. Exploration is very important. > In my experience, most pathnames are received as arguments, > read from config files, etc. Most literals are just fragments > that get concatenated with existing pathnames, and that's > already covered by operations between path objects and strings. True, matured programs usually work this way. But as we all know, agile is better than waterfall, so we might start with literals and replacing them with config variables later when needed. Best, Sven From rosuav at gmail.com Tue Mar 29 03:27:09 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Mar 2016 18:27:09 +1100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA2D44.8080707@mail.de> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> Message-ID: On Tue, Mar 29, 2016 at 6:22 PM, Sven R. Kunze wrote: > Thinking more about it, that would even come in nicely with path separators: > > p'/{whereever}/{youwant}/{togo}' > > That'll finally an easy-to-write and readable path generation. Great idea, > Stephen. Agreed, although this introduces a new edge case: what if 'togo' contains a slash? Should this raise an exception, or should it be interpreted as a multi-part path component? Arguments can be put for both sides. But it's pretty cool. ChrisA From srkunze at mail.de Tue Mar 29 03:31:22 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 09:31:22 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F68406.3030103@sdamon.com> <56F696A4.6080704@gmail.com> Message-ID: <56FA2F4A.9060505@mail.de> On 26.03.2016 20:53, Koos Zevenhoven wrote: > Yes, some of the properties could have more logical names like > .remove() instead of .unlink(), and there could be even more > functionality. Subclassing from str would have obvious benefits, but > I'm just not sure whether it's a good idea to inherit things like > str.find, str.splitlines, str.title, ... > > So maybe the best thing is not to inherit from str but to make Path > quack as much like a str-duck as possible (and vice versa?) without > introducing confusing things? I would not like this limitation. Paths are strings (even for the os). It's just that Python devs are terribly lazy so, they want a string deluxe instead of a plain string in case a string represents a path. A lot of companies encode data right into the path of some files. So, when dealing with such situation, I would very like to have all capabilities of str available. Everything else would be frustrating IMHO. Best, Sven From sjoerdjob at sjec.nl Tue Mar 29 03:46:34 2016 From: sjoerdjob at sjec.nl (Sjoerd Job Postmus) Date: Tue, 29 Mar 2016 09:46:34 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> Message-ID: <20160329074634.GA25618@sjoerdjob.com> On Tue, Mar 29, 2016 at 06:27:09PM +1100, Chris Angelico wrote: > On Tue, Mar 29, 2016 at 6:22 PM, Sven R. Kunze wrote: > > Thinking more about it, that would even come in nicely with path separators: > > > > p'/{whereever}/{youwant}/{togo}' > > > > That'll finally an easy-to-write and readable path generation. Great idea, > > Stephen. > > Agreed, although this introduces a new edge case: what if 'togo' > contains a slash? Should this raise an exception, or should it be > interpreted as a multi-part path component? Arguments can be put for > both sides. As we all know, togo is user input containing ../../../../../../../../../../../../../../../etc/passwd I agree that both sides can get arguments. However, if it does raise an exception, you'd have to go through hoops to make it work with components that might contain a slash. The other way around, however, is p'/{whereever}/{youwant}/{nosl(togo)}' where 'nosl' (or something similar) is a function that checks that the string 'togo' matches some sensible expectations: - Does not contain a '/'. - Does not equal '..' One of the codebases I worked at also had to work with relative paths, and we had a similar helper function which we used for os.path.join( whereever, youwant, nosl(togo), ) It hurts me to say this, but in this case it is probably better to slap on extra security later, than to make it too secure a default. An alternative might be to look at what frameworks like Django do w.r.t. XSS prevention, and do something like p'/{whereever}/{youwant}/{togo:|safe}' However I'm not really a fan of that. > > But it's pretty cool. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From p.f.moore at gmail.com Tue Mar 29 03:49:00 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Mar 2016 08:49:00 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA2D44.8080707@mail.de> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> Message-ID: On 29 March 2016 at 08:22, Sven R. Kunze wrote: > Thinking more about it, that would even come in nicely with path separators: > > p'/{whereever}/{youwant}/{togo}' > > That'll finally an easy-to-write and readable path generation. Great idea, > Stephen. P(wherever)/youwant/togo is as good, if not better - just "from pathlib import Path as P" (or use Path(wherever) if you don't like unnecessary 1-letter aliases...) Paul From srkunze at mail.de Tue Mar 29 03:50:24 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 09:50:24 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F5B3F7.40502@gmail.com> References: <56F5B3F7.40502@gmail.com> Message-ID: <56FA33C0.4000902@mail.de> On 25.03.2016 22:56, Michel Desmoulin wrote: > Although path.py, which I have been using for years now (and still > prefer to pathlib) subclass str and never caused any problem whatsoever. > So really, we should pinpoint where it could be an issue and see if this > is a frequent problem, because right now, it seems more a decision based > on purity than practicality. I agree. The PEP says: """ No confusion with builtins ---------------------------------- In this proposal, the path classes do not derive from a builtin type. This contrasts with some other Path class proposals which were derived from str . They also do not pretend to implement the sequence protocol: if you want a path to act as a sequence, you have to lookup a dedicated attribute (the parts attribute). Not behaving like one of the basic builtin types also minimizes the potential for confusion if a path is combined by accident with genuine builtin types. """" I have to admit I cannot follow these statements but they should have appeared to be necessary back then. As experience shows the PyPI module fared far better here. I am great a fan of theory over practice as what has been proven in theory cannot be proven wrong in practice. However, this only holds if we talk about hard proof. For soft things like "human interaction", "mutual understanding" or "potential for confusion", only the practice of many many people can "prove" what's useful, what's "practical". Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From itetteh34 at hotmail.com Tue Mar 29 03:49:47 2016 From: itetteh34 at hotmail.com (isaac tetteh) Date: Tue, 29 Mar 2016 02:49:47 -0500 Subject: [Python-ideas] MySQL GUI Message-ID: Hello, I was looking at creating a gui in Python to connect to a mysql database to do SIDU. What will be the best GUI package to use. Thanks :) Sent from my iPhone From p.f.moore at gmail.com Tue Mar 29 03:51:14 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Mar 2016 08:51:14 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA2F4A.9060505@mail.de> References: <56F68406.3030103@sdamon.com> <56F696A4.6080704@gmail.com> <56FA2F4A.9060505@mail.de> Message-ID: On 29 March 2016 at 08:31, Sven R. Kunze wrote: > Paths are strings (even for the os). On Unix, paths are bytestrings, not strings (in Python 3, string = Unicode string), but on Windows they are Unicode strings. Having an abstraction over this OS difference is useful. Paul From srkunze at mail.de Tue Mar 29 03:58:11 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 09:58:11 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <20160329074634.GA25618@sjoerdjob.com> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <20160329074634.GA25618@sjoerdjob.com> Message-ID: <56FA3593.30807@mail.de> On 29.03.2016 09:46, Sjoerd Job Postmus wrote: > p'/{whereever}/{youwant}/{togo:|safe}' However I'm not really a fan of > that. But you made a good point. Format strings support format specifiers. Let's use them. :) Best, Sven From srkunze at mail.de Tue Mar 29 04:05:12 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 10:05:12 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F71361.5010301@canterbury.ac.nz> References: <56F5B599.10708@canterbury.ac.nz> <56F5EC30.2010908@stoneleaf.us> <56F71361.5010301@canterbury.ac.nz> Message-ID: <56FA3738.4020604@mail.de> On 26.03.2016 23:55, Greg Ewing wrote: > Ethan Furman wrote: > >>> Also, this would give a privileged place to one particular >>> path library. How do we choose which one? >> >> Um, the one in the stdlib? ;) > > Not everyone likes that one though. As things stand, you > can easily substitute one of your own that's more to your > liking. That won't be true if path objects get baked into > the language syntax. I'd rather submit bug reports and help improving a commonly used library than re-inventing the wheel. If the help is not wanted, that's another thing. But I consider working together important. Furthermore, file paths are an essential part of programming even of todays operating systems. As are strings, numbers btw. Best, Sven From srkunze at mail.de Tue Mar 29 04:14:16 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 10:14:16 +0200 Subject: [Python-ideas] Add a begins like wrapper to argparse In-Reply-To: <56F66C62.6090703@gmail.com> References: <56F5B4CD.90404@canterbury.ac.nz> <56F66C62.6090703@gmail.com> Message-ID: <56FA3958.2050700@mail.de> On 26.03.2016 12:02, Michel Desmoulin wrote: > I love argparse. I really do. But it's one of those lib forcing me to > RTFM everytime I need it, for the last 10 years. You may want to look at: https://pypi.python.org/pypi/argh#examples It's dead simple. Each function can become a cmdline command since it just re-uses the functions' signatures. :) Best, Sven From srkunze at mail.de Tue Mar 29 04:16:21 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 10:16:21 +0200 Subject: [Python-ideas] Add a begins like wrapper to argparse In-Reply-To: <56FA3958.2050700@mail.de> References: <56F5B4CD.90404@canterbury.ac.nz> <56F66C62.6090703@gmail.com> <56FA3958.2050700@mail.de> Message-ID: <56FA39D5.5010401@mail.de> Forget about it. My thunderbird again screwed up my threading view. On 29.03.2016 10:14, Sven R. Kunze wrote: > On 26.03.2016 12:02, Michel Desmoulin wrote: >> I love argparse. I really do. But it's one of those lib forcing me to >> RTFM everytime I need it, for the last 10 years. > > You may want to look at: https://pypi.python.org/pypi/argh#examples > > It's dead simple. Each function can become a cmdline command since it > just re-uses the functions' signatures. :) > > > Best, > Sven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From srkunze at mail.de Tue Mar 29 04:44:54 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 10:44:54 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> <56F71795.7040101@canterbury.ac.nz> Message-ID: <56FA4086.7020004@mail.de> On 27.03.2016 00:51, Koos Zevenhoven wrote: > On Sun, Mar 27, 2016 at 1:13 AM, Greg Ewing > > wrote: > > [...] > This feels like a slippery slope to me. If we include > special syntax for pathnames, why shouldn't we have it > for dates and times? Regular expressions? URLs? JSON > data? SQL queries? XML data? Where do we draw the line? > Just because you want to draw a line does not necessarily mean, we want to draw one. ;) I don't think we (as the Python community) can allow ourselves to stand still. We need to have tools that are simple and available. If YAML is the new standard for everything, Python should move. If $ before variable names are the new standard (I hope they never will again), Python should move. I understand that there are preservers who don't want anything to change. So, compromises will be necessary. > OT: > > To be honest, I do think it feels like URL:s are becoming (or have > become) just as important as paths, and that pathlib.Path should in > the future work with URLs just like it now works with windows and > posix paths. The difference between "http://domain.xyz/" and "C:\\" is > not huge. I also think there should be a Python type (stdlib or > builtin), which handles JSON objects nicer than dicts do and has its > own literal That even occurred to me after we talked about the p-string (mainly because I am working in this field, so I basically need both file paths and URIs). I agree with the URI/IRI idea. It feels natural and sometimes it is necessary to extract specific parts from an URL according to RFC 3986 or 3987. So, +1 from me. Just for the record: "Path" might not be the most correct wording. There is a "file://" scheme which identifies locally located files. So, paths are basically a subset of URLs speaking functionality-wise. Thus, a better/more generic name would be "URL", "URI", "Link" or the like in order to avoid confusing of later generations. However, I think I could live with Path. Another thought: requesting URLs. Basically the same as p'/etc/hosts'.write_text(secret). It's really important to have a dead simple library which is able to work with URLs. So, if I could do: p'https://mysite.com/{page}'.get() that'll be awesome. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Mar 29 04:55:05 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 10:55:05 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <22264.6682.585845.192160@turnbull.sk.tsukuba.ac.jp> References: <56F5B4CD.90404@canterbury.ac.nz> <56F71795.7040101@canterbury.ac.nz> <22264.6682.585845.192160@turnbull.sk.tsukuba.ac.jp> Message-ID: <56FA42E9.8010501@mail.de> On 27.03.2016 19:36, Stephen J. Turnbull wrote: > Koos Zevenhoven writes: > > > To be honest, I do think it feels like URL:s are becoming (or have become) > > just as important as paths, and that pathlib.Path should in the future work > > with URLs just like it now works with windows and posix paths. > > +1 > > ... to the concept, but: Is there a semantic difference between a RFC > 3986 path component (Section 3.3), and a pathlib path? If there is, > this could be a difficult project. (May as well start now, though!) I would interpret the path of pathlib as a subset of functionality of path of a URI. """ The path component contains data, usually organized in hierarchical form [...] serves to identify a resource within the scope of the URI's scheme and naming authority (if any). """ URI's scheme for path of pathlib could implicitly be: "file://" Relative paths of pathlib are a subset of relative references (Section 4.1). The only practical issue, I can think of is, how to distinguish (in the sense of avoiding hidden bugs) file paths and url paths. And name clashes when communicating with your fellow programmers: "I got that url path working" "the url path or the url path path?" Best, Sven From srkunze at mail.de Tue Mar 29 05:00:11 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 11:00:11 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F68406.3030103@sdamon.com> <56F696A4.6080704@gmail.com> <56FA2F4A.9060505@mail.de> Message-ID: <56FA441B.4070908@mail.de> On 29.03.2016 09:51, Paul Moore wrote: > On 29 March 2016 at 08:31, Sven R. Kunze wrote: >> Paths are strings (even for the os). > On Unix, paths are bytestrings, not strings Yep. > (in Python 3, string = Unicode string), Do you mean str? When reading the docs, let's say: https://docs.python.org/3.5/library/stdtypes.html and search for "string" you find a lot of places where "string" is used and both "str" and "bytes" are working. ;-) > but on Windows they are Unicode strings. Having an > abstraction over this OS difference is useful. Indeed true. Best, Sven From srkunze at mail.de Tue Mar 29 05:03:17 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 11:03:17 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: <56FA44D5.2020905@mail.de> On 26.03.2016 20:21, Guido van Rossum wrote: > > I'm not going to say how I feel. But I noticed one thing in this > thread. These seems to be a growing difference between people who use > Python for "serious" programming and those who consider themselves > testers, QA integrators and the like (more power to them!). Maybe both > groups can occasionally try to walk in the others' shoes. > +1 From p.f.moore at gmail.com Tue Mar 29 05:09:03 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Mar 2016 10:09:03 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA4086.7020004@mail.de> References: <56F5B4CD.90404@canterbury.ac.nz> <56F71795.7040101@canterbury.ac.nz> <56FA4086.7020004@mail.de> Message-ID: On 29 March 2016 at 09:44, Sven R. Kunze wrote: > > I don't think we (as the Python community) can allow ourselves to stand > still. We need to have tools that are simple and available. If YAML is the > new standard for everything, Python should move. If $ before variable names > are the new standard (I hope they never will again), Python should move. > > I understand that there are preservers who don't want anything to change. > So, compromises will be necessary. I agree 100% that Python should not stand still. However, every addition to the core language involves a backward compatibility cost that must be very carefully weighed. If we built XML into the language back when that was the standard, then we'd now be looking at adding YAML, but we wouldn't be able to remove XML without significant pain. So a conservative approach *to core language syntax* is entirely reasonable. Having said that, Python has a very good means of adding new *functionality* in the form of modules, and an very good path for gradually standardising modules in the form of personal library -> PyPI package -> stdlib package. It seems to me that this route is appropriate here. At the moment, pathlib has been included as a stdlib library, based on the experience gained from the various PyPI packages developed beforehand. There still seem to be some reservations in the community over the adoption of pathlib, and the first thing we need to do is to resolve those, before we even think about promoting the library to syntax (after all, regular expressions have been round *forever*, and still aren't built in syntax, in spite of the precedents in Perl, Javascript, and other languages - why are Path objects so much more deserving of a fast track?) I'm much more in favour of discussions on how to address the perceived shortcomings of pathlib over alternatives like path.py and pylib's path object. If it's all about subclassing str, then let's by all means reopen that debate - making Path a str subclass is something that could be done as a stdlib change if the decision made in the original PEP was agreed to be flawed (I'm not sure it was, personally, but I'm open to a discussion on that, as the need to convert to and from strings certainly is a nuisance). > To be honest, I do think it feels like URL:s are becoming (or have become) > just as important as paths, and that pathlib.Path should in the future work > with URLs just like it now works with windows and posix paths. The > difference between "http://domain.xyz/" and "C:\\" is not huge. I also think > there should be a Python type (stdlib or builtin), which handles JSON > objects nicer than dicts do and has its own literal So again, if you have personal code you find cleaner and more usable for this job, publish it on PyPI, publicise it and gain support for it, and then propose it for stdlib inclusion. This also has the advantage of being useful for Python 3.4/3.5 users (and you can even support Python 2.7 if you wish). I've no idea how we would ensure that a built in syntax for URLs was correctly designed without having got design experience with the feature as a library module. Paul From srkunze at mail.de Tue Mar 29 05:10:17 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 11:10:17 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: <56FA4679.4080605@mail.de> On 27.03.2016 19:50, Brett Cannon wrote: > The trick is whether that import statement will work when you call it. > Import is currently the only bit of syntax that relies on Python code > in CPython to work and making that happen took a lot of work. If you > add in this concept of p-strings then suddenly we have two pieces of > syntax that require Python code to work and on top of it p-strings > would depend on import but also that we would then have to make sure > to not have import depend on p-strings. And on top of it we would have > to freeze pathlib and all of its dependencies which may or may not be > preferable. > > My point is that it's not straight-forward and this proposal will have > to consider some technical difficulties involved with it if it happens > to go forward. What solution can you propose to solve this? Can this procedure be generalized and smoothed in order to reduce that "lot of work" and to some extend enable a simpler language evolution? Or would it be better to leave the bootstrapping infrastructure alone? Best, Sven From srkunze at mail.de Tue Mar 29 05:35:41 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 11:35:41 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> <56F71795.7040101@canterbury.ac.nz> <56FA4086.7020004@mail.de> Message-ID: <56FA4C6D.7010207@mail.de> On 29.03.2016 11:09, Paul Moore wrote: > On 29 March 2016 at 09:44, Sven R. Kunze wrote: >> I don't think we (as the Python community) can allow ourselves to stand >> still. We need to have tools that are simple and available. If YAML is the >> new standard for everything, Python should move. If $ before variable names >> are the new standard (I hope they never will again), Python should move. >> >> I understand that there are preservers who don't want anything to change. >> So, compromises will be necessary. > I agree 100% that Python should not stand still. However, every > addition to the core language involves a backward compatibility cost > that must be very carefully weighed. If we built XML into the language > back when that was the standard, then we'd now be looking at adding > YAML, but we wouldn't be able to remove XML without significant pain. > > So a conservative approach *to core language syntax* is entirely reasonable. > > Having said that, Python has a very good means of adding new > *functionality* in the form of modules, and an very good path for > gradually standardising modules in the form of personal library -> > PyPI package -> stdlib package. > > It seems to me that this route is appropriate here. At the moment, > pathlib has been included as a stdlib library, based on the experience > gained from the various PyPI packages developed beforehand. There > still seem to be some reservations in the community over the adoption > of pathlib, and the first thing we need to do is to resolve those, > before we even think about promoting the library to syntax (after all, > regular expressions have been round *forever*, and still aren't built > in syntax, in spite of the precedents in Perl, Javascript, and other > languages - why are Path objects so much more deserving of a fast > track?) Age is no indicator for "deserving". Because you are asking specifically for regexes, I might be able to give an explanation. Regular expressions have, let's say, the tendency to grow. It's more or less code, so they grow naturally. Readability, however, diminishes extremely fast with regexes when compared to regular code. So, speaking from experience (and the moaning and groaning of the fellow team members), they might have been a good idea back then, nowadays we see they disadvantages more clearly. In short, we have far less usage of regular expressions than we see for paths (from what I can see). If this speaks _for_ paths, I cannot say. But waiting 20 years for this to happen might be a bit long. And hey, I wouldn't mind a builtin type for regular expressions either if this makes things considerably easier. :) > I'm much more in favour of discussions on how to address the perceived > shortcomings of pathlib over alternatives like path.py and pylib's > path object. If it's all about subclassing str, then let's by all > means reopen that debate - making Path a str subclass is something > that could be done as a stdlib change if the decision made in the > original PEP was agreed to be flawed (I'm not sure it was, personally, > but I'm open to a discussion on that, as the need to convert to and > from strings certainly is a nuisance). > >> To be honest, I do think it feels like URL:s are becoming (or have become) >> just as important as paths, and that pathlib.Path should in the future work >> with URLs just like it now works with windows and posix paths. The >> difference between "http://domain.xyz/" and "C:\\" is not huge. I also think >> there should be a Python type (stdlib or builtin), which handles JSON >> objects nicer than dicts do and has its own literal > So again, if you have personal code you find cleaner and more usable > for this job, publish it on PyPI, publicise it and gain support for > it, and then propose it for stdlib inclusion. This also has the > advantage of being useful for Python 3.4/3.5 users (and you can even > support Python 2.7 if you wish). > > I've no idea how we would ensure that a built in syntax for URLs was > correctly designed without having got design experience with the > feature as a library module. Are you addressing me? The way you quoted makes it seem as if I had written those lines. This is not true. Btw. I totally agree with the experience-proven way: module -> stdlib module -> syntax. Btw2. we already have design experience in this field: requests + URLObject. Not sure if you've ever used them, but they are amazing (mainly they work and don't get into your way). Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Tue Mar 29 05:43:13 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 29 Mar 2016 11:43:13 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA33C0.4000902@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> Message-ID: <56FA4E31.3010908@gmail.com> Le 29/03/2016 09:50, Sven R. Kunze a ?crit : > On 25.03.2016 22:56, Michel Desmoulin wrote: >> Although path.py, which I have been using for years now (and still >> prefer to pathlib) subclass str and never caused any problem whatsoever. >> So really, we should pinpoint where it could be an issue and see if this >> is a frequent problem, because right now, it seems more a decision based >> on purity than practicality. > > I agree. The PEP says: > > """ > No confusion with builtins > ---------------------------------- > In this proposal, the path classes do not derive from a builtin type. > This contrasts with some other Path class proposals which were derived > from str . They also do not pretend to implement the sequence protocol: > if you want a path to act as a sequence, you have to lookup a dedicated > attribute (the parts attribute). > > Not behaving like one of the basic builtin types also minimizes the > potential for confusion if a path is combined by accident with genuine > builtin types. > """" > > I have to admit I cannot follow these statements but they should have > appeared to be necessary back then. As experience shows the PyPI module > fared far better here. > > > I am great a fan of theory over practice as what has been proven in > theory cannot be proven wrong in practice. However, this only holds if > we talk about hard proof. For soft things like "human interaction", > "mutual understanding" or "potential for confusion", only the practice > of many many people can "prove" what's useful, what's "practical". > > +1. Can somebody defend, with practical examples, the imperative to refuse to inherit from str ? And weight it against the benefits of it ? We have been doing it for years, and so far it's been really, really nice. I can't recall problem I had because of it ever. I can recall numerous editions of my code because I forgot str() on pathlib.Path. > Best, > Sven > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From p.f.moore at gmail.com Tue Mar 29 05:46:12 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Mar 2016 10:46:12 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA4C6D.7010207@mail.de> References: <56F5B4CD.90404@canterbury.ac.nz> <56F71795.7040101@canterbury.ac.nz> <56FA4086.7020004@mail.de> <56FA4C6D.7010207@mail.de> Message-ID: On 29 March 2016 at 10:35, Sven R. Kunze wrote: > Age is no indicator for "deserving". Absolutely. But you do need experience over time to *know* if something is deserving. Many other languages have builtin regexes - Python proved (by not rushing to make them built in) that it wasn't needed. > Are you addressing me? The way you quoted makes it seem as if I had written > those lines. This is not true. Bah. My mail client (or a previous post) may have messed up the quoting - it seemed to me that what I was replying to was your original comment. My apologies if it was someone else's. > Btw. I totally agree with the experience-proven way: module -> stdlib module > -> syntax. > Btw2. we already have design experience in this field: requests + URLObject. > Not sure if you've ever used them, but they are amazing (mainly they work > and don't get into your way). I've used requests - it is indeed excellent (although I don't believe it treats URLs as anything other than strings - if it does, I've never used that part of it). I've not used URLObject, I'll take a look at that (although I don't have much need for more than casual URL parsing in my code). Paul From k7hoven at gmail.com Tue Mar 29 06:24:48 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 29 Mar 2016 13:24:48 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <20160329074634.GA25618@sjoerdjob.com> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <20160329074634.GA25618@sjoerdjob.com> Message-ID: On Tue, Mar 29, 2016 at 10:49 AM, Paul Moore wrote: > On 29 March 2016 at 08:22, Sven R. Kunze wrote: > > Thinking more about it, that would even come in nicely with path > separators: > > > > p'/{whereever}/{youwant}/{togo}' > > > > That'll finally an easy-to-write and readable path generation. Great > idea, > > Stephen. > > P(wherever)/youwant/togo > > is as good, if not better - just "from pathlib import Path as P" (or > use Path(wherever) if you don't like unnecessary 1-letter aliases...) Yes. I assume you meant P('/') /wherever/youwant/togo. Or with p-strings, p'/' /wherever/youwant/togo I'm not necessarily against Sven's version either, but my first thoughts about the power of combined pf strings would be something like this (assuming Sven's implicit f): all_data = [p'data{i}.txt'.read_text() for i in range(1, 307)] which would read the contents of data1.txt, data2.txt, ..., and data306.txt into a list. And below, to the security issue raised by Sjoerd. On Tue, Mar 29, 2016 at 10:46 AM, Sjoerd Job Postmus wrote: > On Tue, Mar 29, 2016 at 06:27:09PM +1100, Chris Angelico wrote: > > On Tue, Mar 29, 2016 at 6:22 PM, Sven R. Kunze wrote: > > > Thinking more about it, that would even come in nicely with path > separators: > > > > > > p'/{whereever}/{youwant}/{togo}' > > > > > > That'll finally an easy-to-write and readable path generation. Great > idea, > > > Stephen. > > > > Agreed, although this introduces a new edge case: what if 'togo' > > contains a slash? Should this raise an exception, or should it be > > interpreted as a multi-part path component? Arguments can be put for > > both sides. > > As we all know, togo is user input containing > ../../../../../../../../../../../../../../../etc/passwd > > How about p'/' /wherever/youwant//togo That is, the floordiv operator could be used to prevent 'togo' from going up the directory tree with "../../" or "/etc/passwd". The // would thus restrict the user (who provides `togo`) into /wherever/youwant/ and it its subdirectories. -Koos -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Mar 29 06:36:38 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Mar 2016 11:36:38 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA4E31.3010908@gmail.com> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> Message-ID: On 29 March 2016 at 10:43, Michel Desmoulin wrote: > +1. Can somebody defend, with practical examples, the imperative to > refuse to inherit from str ? And weight it against the benefits of it ? I don't intend to get sucked into the role of "defender of not inheriting", but one point that should probably be considered is that on Unix, it is possible to have paths which are *not* valid Python strings. How would those work in an inherit-from-str environment? I suspect that using surrogateescape is the answer here, but I don't know. Also, has anyone *converted* code from os.path manipulation of strings to a str-subclass path object like path.py? If you write your code from scratch intending to use path.py it's probably easier to avoid objects getting silently downgraded from path to str than if you're retrofitting a path library to legacy code. That's a consideration that is more applicable to the standard library than to a 3rd party module. But it may be that now is the right time to revisit the choice to not inherit from str. I don't know, personally. I'm not even sure if a pathlib.Path that inherited from str is something I'd use more often than I currently do. For open source work, the need to support Python 2 means I can't use pathlib (without adding the backport as a dependency). For personal code, it's just a case of mental inertia... Paul From p.f.moore at gmail.com Tue Mar 29 06:41:07 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Mar 2016 11:41:07 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <20160329074634.GA25618@sjoerdjob.com> Message-ID: On 29 March 2016 at 11:24, Koos Zevenhoven wrote: >> As we all know, togo is user input containing >> ../../../../../../../../../../../../../../../etc/passwd >> > > How about > > p'/' /wherever/youwant//togo > > That is, the floordiv operator could be used to prevent 'togo' from going up > the directory tree with "../../" or "/etc/passwd". The // would thus > restrict the user (who provides `togo`) into /wherever/youwant/ and it its > subdirectories. There have been a number of previous threads about the security of path objects (and for that matter of os.path) and "sandboxing" paths to disallow traversing "up" out of a particular directory. I've no idea whether the floordiv operator would work for this purpose (I'm not a security expert, and most of my personal applications can afford to take a very lax view of such things, luckily) but it's an interesting idea. Potentially, it's a bit easy to miss, which may be a problem. Paul From desmoulinmichel at gmail.com Tue Mar 29 06:57:02 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 29 Mar 2016 12:57:02 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> Message-ID: <56FA5F7E.3070001@gmail.com> Le 29/03/2016 12:36, Paul Moore a ?crit : > On 29 March 2016 at 10:43, Michel Desmoulin wrote: >> +1. Can somebody defend, with practical examples, the imperative to >> refuse to inherit from str ? And weight it against the benefits of it ? > > I don't intend to get sucked into the role of "defender of not > inheriting", but one point that should probably be considered is that > on Unix, it is possible to have paths which are *not* valid Python > strings. How would those work in an inherit-from-str environment? I > suspect that using surrogateescape is the answer here, but I don't > know. How does str(path) does it ? Because ultimatly, we pass strings to open() and os.stuff. So we need it as strings anyway. > > Also, has anyone *converted* code from os.path manipulation of strings > to a str-subclass path object like path.py? If you write your code > from scratch intending to use path.py it's probably easier to avoid > objects getting silently downgraded from path to str than if you're > retrofitting a path library to legacy code. That's a consideration > that is more applicable to the standard library than to a 3rd party > module. I do it all the time. I'm a path.py/begin/request/arrow/pytest addict and try to convert coworkers as I meet them, so for path.py I'll usually take a small part of their project with intensive FS manipulations and demonstrate how path.py integrate in it. It's doesn't take a lot to convince them, the one good reason people don't want to is the additional dependency, which I understand perfectly. The only hard part to sell is the need to always pass unicode strings to Path(), because in Python 2.7 it tries to automatically convert b'' to u"" with the usual UnicodeDecodeError pitfalls. It's a good habit anyway, yet people are attached to their shortcuts : it's the same with arrow because of the default to UTC instead of local time. it's the same with pytest which requires python setup.py develop and explicit -s. request and begin are is easier since urllib and argparse are so hard. Like with all changes, you need a little adaptation. But all in all, HTTP requests, FS manipulation, datetime handling, unit tests and parameter parsing are so common tasks it make sense to have tools that gets the job done in a few lines and gets out of the way. You want to concentrate of the real value of your code, not those. Now I was sad when I learned request would never be part of the stdlib, and I'm still sad asyncio doesn't have a good way to do a simple GET with the stdlib, even more with http2 comming our way. But I get it: the web moves was, the stdlib moves slowly. It's not True with FS manipulations, they are are old as it gets, and we can improve on them without fearing to be left behind. > > But it may be that now is the right time to revisit the choice to not > inherit from str. I don't know, personally. I'm not even sure if a > pathlib.Path that inherited from str is something I'd use more often > than I currently do. For open source work, the need to support Python > 2 means I can't use pathlib (without adding the backport as a > dependency). For personal code, it's just a case of mental inertia... > > Paul > From srkunze at mail.de Tue Mar 29 07:04:27 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 13:04:27 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> Message-ID: <56FA613B.6090706@mail.de> On 29.03.2016 12:36, Paul Moore wrote: > I don't intend to get sucked into the role of "defender of not > inheriting", but one point that should probably be considered is that > on Unix, it is possible to have paths which are *not* valid Python > strings. I don't know how other system developers are working in this field but that might be something serious I encounter from time to time. As a developer and API designer, one has two options (off the top of my head): 1) design and code with Unicode/UTF-8 as file names 2) deal with arbitrary bytes in file names I like 1) more because it gives me the ability to extract useful data/text from paths, which as far as I can tell is quite some usecase. 2) is more like a black box variant to me, but then operations like "/" or "+" or "suffix" etc. have far less meaning (at least to me). > How would those work in an inherit-from-str environment? I > suspect that using surrogateescape is the answer here, but I don't > know. Could you explain what this means? > Also, has anyone *converted* code from os.path manipulation of strings > to a str-subclass path object like path.py? If you write your code > from scratch intending to use path.py it's probably easier to avoid > objects getting silently downgraded from path to str than if you're > retrofitting a path library to legacy code. That's a consideration > that is more applicable to the standard library than to a 3rd party > module. Good point. It's quite problematic to convert a large system at once, so a transition is necessary. Maybe, that is a reason why more people want path->str. > But it may be that now is the right time to revisit the choice to not > inherit from str. I don't know, personally. I'm not even sure if a > pathlib.Path that inherited from str is something I'd use more often > than I currently do. It's only one of its warts. In order to get a greater mass to make the transition, most of these issues needs to be removed. Quite recently, there has been a discussion to add something to pathlib (I can't remember now) but it failed to get through because of "purity over practicability". A wasted chance IIRC. That thing was exactly what the Zen warns about. > For open source work, the need to support Python > 2 means I can't use pathlib (without adding the backport as a > dependency). For personal code, it's just a case of mental inertia... Adding the dependency works for me. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Mar 29 07:15:19 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 29 Mar 2016 13:15:19 +0200 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <7169445218378847930@unknownmsgid> References: <6BC67AA2-411E-4913-88BC-A3485E521544@yahoo.com> <7169445218378847930@unknownmsgid> Message-ID: 2016-03-28 17:20 GMT+02:00 Chris Barker - NOAA Federal : > The OP's >> 'string'.to_file('some_file') would just be io.write('string', >> 'some_file'), and str.from_file('some_file') would be >> io.read('some_file'). > > Nick's observation about the mental model may be correct ( though I > don't think so, frankly), but if it is, then this isn't any better > than: > > open("some_path", 'w').write(string) (Sorry, I didn't follow the whole thread, and I'm not sure that it's best place for my answer.) A quick reminder: this pattern raises a ResourceWarning to remind you that the state of the file is unknown. Flush to disk? Maybe. Maybe not. It's a good practice to run Python with -Wd to see this DeprecationWarning and ResourceWarning. Use -Werror to ensure that you handle correctly *all* warnings ;-) Victor From steve at pearwood.info Tue Mar 29 07:21:33 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 29 Mar 2016 22:21:33 +1100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA5F7E.3070001@gmail.com> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> Message-ID: <20160329112132.GS12526@ando.pearwood.info> On Tue, Mar 29, 2016 at 12:57:02PM +0200, Michel Desmoulin wrote: > > > Le 29/03/2016 12:36, Paul Moore a ?crit : > > I don't intend to get sucked into the role of "defender of not > > inheriting", but one point that should probably be considered is that > > on Unix, it is possible to have paths which are *not* valid Python > > strings. How would those work in an inherit-from-str environment? I > > suspect that using surrogateescape is the answer here, but I don't > > know. > > How does str(path) does it ? Because ultimatly, we pass strings to > open() and os.stuff. So we need it as strings anyway. Actually, open and os.* will accept bytes paths, at least on Linux. Unless I'm missing something, pathlib doesn't support bytes filenames. So the question doesn't come up. Any path you can use with pathlib is a valid Python string, because you can only create paths using Python strings. So if I have a Linux file b'\xD8\x01', I can't create a path object to work with it. -- Steve From srkunze at mail.de Tue Mar 29 07:22:19 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 13:22:19 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA5F7E.3070001@gmail.com> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> Message-ID: <56FA656B.5070707@mail.de> offtopic On 29.03.2016 12:57, Michel Desmoulin wrote: > Now I was sad when I learned request would never be part of the stdlib, Looking at http://docs.python-requests.org/en/master/dev/philosophy/#standard-library Seems to be true. I wonder if it would be possible/necessary for the stdlib to include a set of libraries marked as "featured". Not like "provisional" but more like "You should use this one and be aware that it may upgrade faster than normal stdlib modules". Basically like a "pull module". Just a third-party module pulled from the source and upgraded with each Python version. That would maintain the maintainers flexibility and the increase the availability of these "defacto stdlib" modules. Just wondering. Best, Sven From srkunze at mail.de Tue Mar 29 07:24:20 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 13:24:20 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <20160329112132.GS12526@ando.pearwood.info> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <20160329112132.GS12526@ando.pearwood.info> Message-ID: <56FA65E4.60308@mail.de> On 29.03.2016 13:21, Steven D'Aprano wrote: > So if I have a Linux file b'\xD8\x01', I can't create a path object to > work with it. I actually like this limitation. It forces a cleaner API and cleaner code as it is the one way. From eric at trueblade.com Tue Mar 29 08:06:24 2016 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 29 Mar 2016 08:06:24 -0400 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <56F9E6BB.6040400@stoneleaf.us> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <56F9E6BB.6040400@stoneleaf.us> Message-ID: <56FA6FC0.30306@trueblade.com> On 3/28/2016 10:21 PM, Ethan Furman wrote: > [Sorry, Nick, didn't mean to just send it to you.] > > On 03/28/2016 11:16 AM, Nick Eubank wrote: > >> Just to make sure I follow: Chris, you're proposing a modification to >> garbage collection to clean up `open()` if `open()` is used by not >> closed and not assigned to anything so that `open(file).write(stuff)` is >> safe? > > Even if he is, I don't see it happening. Modifying the garbage > collector to act sooner on one particular type of object when we already > have good, if very mildly inconvenient, solutions is not going to be a > high-priority item -- especially if it has /any/ negative impact on > performance. I agree this is not likely to ever happen. It's especially problematic that we'd be making this a requirement for implementations other than CPython. Eric. From victor.stinner at gmail.com Tue Mar 29 08:08:04 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 29 Mar 2016 14:08:04 +0200 Subject: [Python-ideas] custom predicate for all() & any() In-Reply-To: References: Message-ID: 2016-03-26 17:30 GMT+01:00 titouan dk : > major = 18..__le__ Minor remark: using this syntax hack, you get float.__le__. I suggest to write directly int.__le__ (or (18).__le__ is you really like getting unbound methods). Victor From p.f.moore at gmail.com Tue Mar 29 08:10:52 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Mar 2016 13:10:52 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA65E4.60308@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <20160329112132.GS12526@ando.pearwood.info> <56FA65E4.60308@mail.de> Message-ID: On 29 March 2016 at 12:24, Sven R. Kunze wrote: > On 29.03.2016 13:21, Steven D'Aprano wrote: >> >> So if I have a Linux file b'\xD8\x01', I can't create a path object to >> work with it. > > > I actually like this limitation. It forces a cleaner API and cleaner code as > it is the one way. However, it does mean that certain use cases are not supported by pathlib. One of the constant issues with core/stdlib development is having to consider how the design affects people with needs you don't personally have. This one's not such a big problem "pathlib doesn't support non-Unicode filenames" is a simple enough to express limitation. But that doesn't help the poor guy debugging an issue with his filesystem-scanning program where a user has a mangled non-Unicode filename somewhere in the bowels of his profile - does he declare that user's situation is unsupported, or does he abandon pathlib in favour of the old os APIs? (And if the , what's the likelihood From p.f.moore at gmail.com Tue Mar 29 08:11:53 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Mar 2016 13:11:53 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <20160329112132.GS12526@ando.pearwood.info> <56FA65E4.60308@mail.de> Message-ID: Bah, unexpected keyboard shortcut - sorry for the premature send... On 29 March 2016 at 13:10, Paul Moore wrote: > However, it does mean that certain use cases are not supported by > pathlib. One of the constant issues with core/stdlib development is > having to consider how the design affects people with needs you don't > personally have. This one's not such a big problem "pathlib doesn't > support non-Unicode filenames" is a simple enough to express > limitation. But that doesn't help the poor guy debugging an issue with > his filesystem-scanning program where a user has a mangled non-Unicode > filename somewhere in the bowels of his profile - does he declare that > user's situation is unsupported, or does he abandon pathlib in favour > of the old os APIs? (And if the , what's the likelihood (And either way, what's the likelihood of him ever wanting to use pathlib again?) Paul From eric at trueblade.com Tue Mar 29 08:24:13 2016 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 29 Mar 2016 08:24:13 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56F71795.7040101@canterbury.ac.nz> References: <56F5B4CD.90404@canterbury.ac.nz> <56F71795.7040101@canterbury.ac.nz> Message-ID: <56FA73ED.6000804@trueblade.com> On 3/26/2016 7:13 PM, Greg Ewing wrote: > Path objects, on the other hand, come with a pile of > semantics relating to a particular application area -- > manipulation of the file system. Moreover, those > semantics vary depending on what operating system you're > running on. So it's hard so see them as being part of > the language the way str and bytes are. > > This feels like a slippery slope to me. If we include > special syntax for pathnames, why shouldn't we have it > for dates and times? Regular expressions? URLs? JSON > data? SQL queries? XML data? Where do we draw the line? PEP 501 tries to address these issues, with i-strings. There were other proposals on python-ideas relating to arbitrary string prefixes when PEPs 498 and 501 were being discussed. Eric. From ethan at stoneleaf.us Tue Mar 29 08:28:38 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 29 Mar 2016 05:28:38 -0700 Subject: [Python-ideas] MySQL GUI In-Reply-To: References: Message-ID: <56FA74F6.2080103@stoneleaf.us> On 03/29/2016 12:49 AM, isaac tetteh wrote: > I was looking at creating a gui in Python to connect to a mysql database to do SIDU. > What will be the best GUI package to use. Greetings! This list is for discussion of ideas to improve Python the language. Your question will do better on the Python List, which is about pretty much any question that has anything to do with Python: https://mail.python.org/mailman/listinfo/python-list -- ~Ethan~ From srkunze at mail.de Tue Mar 29 08:31:19 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 14:31:19 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <20160329112132.GS12526@ando.pearwood.info> <56FA65E4.60308@mail.de> Message-ID: <56FA7597.60202@mail.de> Patching your mails: On 29.03.2016 14:10, Paul Moore wrote: > However, it does mean that certain use cases are not supported by > pathlib. One of the constant issues with core/stdlib development is > having to consider how the design affects people with needs you don't > personally have. This one's not such a big problem "pathlib doesn't > support non-Unicode filenames" is a simple enough to express > limitation. But that doesn't help the poor guy debugging an issue with > his filesystem-scanning program where a user has a mangled non-Unicode > filename somewhere in the bowels of his profile - does he declare that > user's situation is unsupported, or does he abandon pathlib in favour > of the old os APIs? Perfect example. As I said, I've encountered this situation and most likely will encounter it again. We have mostly control over the file names. So, we might be able to fix a problematic file name. The issue you are bringing up (which is perfectly valid) is when you as a developer don't have control over the file name (aka fixing them). In this regard, I might be pragmatic and say: "let's see what happens". If this is a frequent issue, then this is one of those warts pathlib needs to loose in order to become mainstream. But why not open another thread with a solution to this issue as has Koos done with p-strings? > (And either way, what's the likelihood of him ever wanting to use > pathlib again?) You mean like myself? As I wanted to use pathlib but found myself writing awkward code with it? (btw. that was not the path->str issue) That as well needs to be fixed but it's no argument for delaying other improvements, right? Best, Sven From rosuav at gmail.com Tue Mar 29 08:33:43 2016 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 29 Mar 2016 23:33:43 +1100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <20160329112132.GS12526@ando.pearwood.info> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <20160329112132.GS12526@ando.pearwood.info> Message-ID: On Tue, Mar 29, 2016 at 10:21 PM, Steven D'Aprano wrote: > Unless I'm missing something, pathlib doesn't support bytes filenames. > So the question doesn't come up. Any path you can use with pathlib is a > valid Python string, because you can only create paths using Python > strings. > > So if I have a Linux file b'\xD8\x01', I can't create a path object to > work with it. Actually you can. Here's an annotated interactive session illustrating byte smuggling: >>> # Create a file using a bytes name ... with open(b'\xD8\x01',"w") as f: f.write("Demo text\n") ... 10 >>> import os, pathlib >>> # By default, you get text strings. ... os.listdir() ['\udcd8\x01'] >>> os.listdir(".") ['\udcd8\x01'] >>> # But you can request byte strings. ... os.listdir(b".") [b'\xd8\x01'] >>> # pathlib works with text strings the same way. ... list(pathlib.Path(".").glob("*")) [PosixPath('\udcd8\x01')] >>> # And you can cast that to str and open it. ... open(str(pathlib.Path("\uDCD8\x01"))).read() 'Demo text\n' So it looks like the only missing piece of the puzzle is a way to construct a Path from a bytes, which would do the same transformation of \xD8 to \uDCD8. ChrisA From p.f.moore at gmail.com Tue Mar 29 09:08:43 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Mar 2016 14:08:43 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA7597.60202@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <20160329112132.GS12526@ando.pearwood.info> <56FA65E4.60308@mail.de> <56FA7597.60202@mail.de> Message-ID: On 29 March 2016 at 13:31, Sven R. Kunze wrote: > But why not open another thread with a solution to this issue as has Koos > done with p-strings? Agreed, that's fine. I personally don't have any solutions to offer, so I'll wait to comment on any that do arise (as I've been doing on the p-string proposal). >> (And either way, what's the likelihood of him ever wanting to use >> pathlib again?) > > You mean like myself? As I wanted to use pathlib but found myself writing > awkward code with it? (btw. that was not the path->str issue) > > That as well needs to be fixed but it's no argument for delaying other > improvements, right? Certainly. However, I don't think p-strings are an improvement - I'm not suggesting delaying them because other things need to be fixed, I'm suggesting that the whole proposal shouldn't be implemented because it's not a good idea. If you want custom string-like syntaxes, PEP 501 is where you should be getting involved. Adding a single-purpose custom string prefix is a proposal that *should* be delayed until the fate of a more general proposal of which it's a subset is decided. Because once syntax is added it's nigh-on impossible to get rid of. Paul From ethan at stoneleaf.us Tue Mar 29 09:11:05 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 29 Mar 2016 06:11:05 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> Message-ID: <56FA7EE9.6040501@stoneleaf.us> On 03/29/2016 03:36 AM, Paul Moore wrote: > On 29 March 2016 at 10:43, Michel Desmoulin wrote: >> +1. Can somebody defend, with practical examples, the imperative to >> refuse to inherit from str ? And weight it against the benefits of it ? I cannot. Pathlib seems far too complicated to me. (And this coming from the guy whose antipathy [1] path library has both bytes and unicode inheritance and supports 2.7 and 3.3+ in the same code base.) > I don't intend to get sucked into the role of "defender of not > inheriting", but one point that should probably be considered is that > on Unix, it is possible to have paths which are *not* valid Python > strings. How would those work in an inherit-from-str environment? I > suspect that using surrogateescape is the answer here, but I don't > know. With antipathy's Path you get either a bPath if the incoming object was from bytes (str in 2.x) or a uPath if it was str (unicode in 2.x). The only problem I have ever hit is with older libraries that were doing type-checks instead of isinstance-checks on their incoming arguments. > Also, has anyone *converted* code from os.path manipulation of strings > to a str-subclass path object like path.py? If you write your code > from scratch intending to use path.py it's probably easier to avoid > objects getting silently downgraded from path to str than if you're > retrofitting a path library to legacy code. That's a consideration > that is more applicable to the standard library than to a 3rd party > module. If I have to do any other work with the code, then I convert the os.path portions to Path first. It just makes it so much clearer. > But it may be that now is the right time to revisit the choice to not > inherit from str. Is it still marked provisional? The pain of using it with any other portion of the stdlib that uses paths makes it a non-starter for me. -- ~Ethan~ [1] https://pypi.python.org/pypi/antipathy/ From desmoulinmichel at gmail.com Tue Mar 29 09:09:28 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 29 Mar 2016 15:09:28 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA7597.60202@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <20160329112132.GS12526@ando.pearwood.info> <56FA65E4.60308@mail.de> <56FA7597.60202@mail.de> Message-ID: <56FA7E88.1070309@gmail.com> Le 29/03/2016 14:31, Sven R. Kunze a ?crit : > Patching your mails: > > On 29.03.2016 14:10, Paul Moore wrote: >> However, it does mean that certain use cases are not supported by >> pathlib. One of the constant issues with core/stdlib development is >> having to consider how the design affects people with needs you don't >> personally have. This one's not such a big problem "pathlib doesn't >> support non-Unicode filenames" is a simple enough to express >> limitation. But that doesn't help the poor guy debugging an issue with >> his filesystem-scanning program where a user has a mangled non-Unicode >> filename somewhere in the bowels of his profile - does he declare that >> user's situation is unsupported, or does he abandon pathlib in favour >> of the old os APIs? > > Perfect example. > > As I said, I've encountered this situation and most likely will > encounter it again. We have mostly control over the file names. So, we > might be able to fix a problematic file name. The issue you are bringing > up (which is perfectly valid) is when you as a developer don't have > control over the file name (aka fixing them). > > In this regard, I might be pragmatic and say: "let's see what happens". > If this is a frequent issue, then this is one of those warts pathlib > needs to loose in order to become mainstream. yes. It's a very rare use case. If you create a program that needs to be handling this, you still got the old API. For the 99% of the rest of the world, you have the modern one. Let's not try to make pathlib handle everything, it's a convenience wrapper, not a silver bullet. > > But why not open another thread with a solution to this issue as has > Koos done with p-strings? > >> (And either way, what's the likelihood of him ever wanting to use >> pathlib again?) > > You mean like myself? As I wanted to use pathlib but found myself > writing awkward code with it? (btw. that was not the path->str issue) > > That as well needs to be fixed but it's no argument for delaying other > improvements, right? Exactly. pathlib needs improvements, and there is plenty of prior art to make suggestions about this, but if it's a central API, it will be on the spotlight to do so. > > > Best, > Sven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From srkunze at mail.de Tue Mar 29 09:15:21 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 15:15:21 +0200 Subject: [Python-ideas] custom predicate for all() & any() In-Reply-To: References: <1C405062-977D-4324-8A14-D37AAB752F7E@gmail.com> Message-ID: <56FA7FE9.60507@mail.de> On 26.03.2016 18:04, Terry Reedy wrote: > So proposed all(iterable, predicate) == all(map(predicate, iterable)) I am no native English speaker. Is predicate the right word for this? Two observations: 1) I recently saw a similar request to another "aggregate" function. 2) I can comprehend why all(map(predicate, iterable)) is harder to write. However, somehow I feel all(iterable, predicate) is harder to maintain. Because of 2) I am -1 on this. :) One could further argue that we need a another "predicate" for filtering: all(iterable, map, filter) etc. I get the feeling we would rebuilding things over and over again: list comprehensions, filter, map, itertools, for-loops etc. What about "sum", "max" etc? They all could need map + filter. Additional thought: I think the underlying issue is the number of parentheses involved. I myself somehow avoid nesting too many function calls into one line just because of that; not because of the complexity involved but because it looks strange. Of course the number of parentheses involved is an indicator of the complexity. Not sure if there is another way of handling this; maybe piping or something. Would be great if we could avoid writing ))) or ))))). Best, Sven From random832 at fastmail.com Tue Mar 29 09:36:52 2016 From: random832 at fastmail.com (Random832) Date: Tue, 29 Mar 2016 09:36:52 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <20160329112132.GS12526@ando.pearwood.info> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <20160329112132.GS12526@ando.pearwood.info> Message-ID: <1459258612.1631456.562476698.6B1CAE83@webmail.messagingengine.com> On Tue, Mar 29, 2016, at 07:21, Steven D'Aprano wrote: > Unless I'm missing something, pathlib doesn't support bytes filenames. > So the question doesn't come up. Any path you can use with pathlib is a > valid Python string, because you can only create paths using Python > strings. > > So if I have a Linux file b'\xD8\x01', I can't create a path object to > work with it. Any reason you can't use surrogateescape and "\uDCD8\x01"? From srkunze at mail.de Tue Mar 29 09:40:31 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 15:40:31 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <20160329112132.GS12526@ando.pearwood.info> <56FA65E4.60308@mail.de> <56FA7597.60202@mail.de> Message-ID: <56FA85CF.2050706@mail.de> On 29.03.2016 15:08, Paul Moore wrote: > On 29 March 2016 at 13:31, Sven R. Kunze wrote: >> That as well needs to be fixed but it's no argument for delaying other >> improvements, right? > Certainly. However, I don't think p-strings are an improvement - I'm > not suggesting delaying them because other things need to be fixed, > I'm suggesting that the whole proposal shouldn't be implemented > because it's not a good idea. Alright. Got it. :) p-strings are not the most pressing issue of pathlib I assume. Wrapping an (f-)string into a Path might work for now although I would love see the convenient p-string in some future Python version. > If you want custom string-like syntaxes, PEP 501 is where you should > be getting involved. Adding a single-purpose custom string prefix is a > proposal that *should* be delayed until the fate of a more general > proposal of which it's a subset is decided. Because once syntax is > added it's nigh-on impossible to get rid of. Agreed. First things first. Note on PEP 501. Not sure if I like it (sorry Nick) but to me it feels a bit artificial/impracticable. Its examples seem very awkward to me. This additional pair of parentheses kills it for me. runquery & runcommand should do the wrapping themselves. If done, i-strings look like f-strings. runquery(sql(i"SELECT {column} FROM {table};")) runcommand(sh(i"cat {filename}")) This means for p-string: Path(f'/whereever/{youwant}/{togo}') == p'/whereever/{youwant}/{togo}' So, if Path is powerful enough, I think most people can live with that for now (but those parentheses .... don't we already have the quotation marks ... anyway) ;-) Best, Sven From random832 at fastmail.com Tue Mar 29 09:33:11 2016 From: random832 at fastmail.com (Random832) Date: Tue, 29 Mar 2016 09:33:11 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA4C6D.7010207@mail.de> References: <56F5B4CD.90404@canterbury.ac.nz> <56F71795.7040101@canterbury.ac.nz> <56FA4086.7020004@mail.de> <56FA4C6D.7010207@mail.de> Message-ID: <1459258391.1630604.562474258.218101B7@webmail.messagingengine.com> On Tue, Mar 29, 2016, at 05:35, Sven R. Kunze wrote: > Because you are asking specifically for regexes, I might be able to give > an explanation. > > Regular expressions have, let's say, the tendency to grow. It's more or > less code, so they grow naturally. > Readability, however, diminishes extremely fast with regexes when > compared to regular code. ... > And hey, I wouldn't mind a builtin type for regular expressions either > if this makes things considerably easier. :) Is it time again to bring up my regex AST idea? From srkunze at mail.de Tue Mar 29 09:51:18 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 15:51:18 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <1459258391.1630604.562474258.218101B7@webmail.messagingengine.com> References: <56F5B4CD.90404@canterbury.ac.nz> <56F71795.7040101@canterbury.ac.nz> <56FA4086.7020004@mail.de> <56FA4C6D.7010207@mail.de> <1459258391.1630604.562474258.218101B7@webmail.messagingengine.com> Message-ID: <56FA8856.9060603@mail.de> On 29.03.2016 15:33, Random832 wrote: > On Tue, Mar 29, 2016, at 05:35, Sven R. Kunze wrote: >> Because you are asking specifically for regexes, I might be able to give >> an explanation. >> >> Regular expressions have, let's say, the tendency to grow. It's more or >> less code, so they grow naturally. >> Readability, however, diminishes extremely fast with regexes when >> compared to regular code. > ... >> And hey, I wouldn't mind a builtin type for regular expressions either >> if this makes things considerably easier. :) > Is it time again to bring up my regex AST idea? Own thread? ;-) From itetteh34 at hotmail.com Tue Mar 29 10:31:44 2016 From: itetteh34 at hotmail.com (isaac tetteh) Date: Tue, 29 Mar 2016 09:31:44 -0500 Subject: [Python-ideas] MySQL GUI In-Reply-To: <56FA74F6.2080103@stoneleaf.us> References: <56FA74F6.2080103@stoneleaf.us> Message-ID: Thank you :) Sent from my iPhone > On Mar 29, 2016, at 7:28 AM, Ethan Furman wrote: > >> On 03/29/2016 12:49 AM, isaac tetteh wrote: >> >> I was looking at creating a gui in Python to connect to a mysql database to do SIDU. >> What will be the best GUI package to use. > > Greetings! > > This list is for discussion of ideas to improve Python the language. > > Your question will do better on the Python List, which is about pretty much any question that has anything to do with Python: > > https://mail.python.org/mailman/listinfo/python-list > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From k7hoven at gmail.com Tue Mar 29 10:42:04 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 29 Mar 2016 17:42:04 +0300 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? Message-ID: The 'Working with Path objects: p-strings?' thread spawned a discussion about how URLs (or more generally URIs) and Paths should work together. I suggest we move that discussion to this new thread. The concept is 'explained' below in this email and quotes, but a little bit of discussion happened in the other thread too. While I think that the decisions about p-strings (or a-strings for addresses or whatever they should be) should keep URIs in mind, it is premature to add the Path+URI fusion into the stdlib. I agree with Paul Moore that this URL stuff should be on PyPI first. It could even be library that monkey patches pathlib to accept URIs. Or a URI library that instantiates Path objects when appropriate. Then there could be a smooth transition into the stdlib some day. See all the stuff below: On Tue, Mar 29, 2016 at 11:44 AM, Sven R. Kunze wrote: > > On 27.03.2016 00:51, Koos Zevenhoven wrote: >> >> OT: >> >> To be honest, I do think it feels like URL:s are becoming (or have become) just as important as paths, and that pathlib.Path should in the future work with URLs just like it now works with windows and posix paths. The difference between "http://domain.xyz/" and "C:\\" is not huge. I also think there should be a Python type (stdlib or builtin), which handles JSON objects nicer than dicts do and has its own literal >> > > That even occurred to me after we talked about the p-string (mainly because I am working in this field, so I basically need both file paths and URIs). > Again, that you say you thought about it too perhaps means it's worth discussing :). > > Just for the record: "Path" might not be the most correct wording. There is a "file://" scheme which identifies locally located files. So, paths are basically a subset of URLs speaking functionality-wise. Thus, a better/more generic name would be "URL", "URI", "Link" or the like in order to avoid confusing of later generations. However, I think I could live with Path. > Yes, these are concerns that should be considered if/when deciding whether to make URI/URLs a subclass of Path or the other way around, or something else. Anyway, since Path(...) already instantiates different subclasses based on the situation, having it instantiate a URI in some cases would not be completely unnatural. As suggested by Stephen, I've been looking into RFC 3986 as a whole, and it seems that making instantiating both URIs and fs paths from p-strings does not seem completely impossible. Some points below (you can skip past them if you have to, there's more general discussion at the end): - Only some URIs (or even URLs) can be reliably distinguished from file paths. However, those that contain '://' could be automatically turned into URI objects by p-strings [or Path(...)]. I suspect that would cover the majority of use cases. (The unambiguous cases would be exactly those URIs that contain an 'authority' component -- these always begin with 'scheme://' while other's don't) - If we want allow URIs without an 'authority' component, like mailto:someone at domain.com', they should be explicitly instantiated as URI objects. - Some terminology: There are indeed 'URI's and 'relative references'. Relative references are essentially the URI-equivalent of relative paths. Then there are 'URI references' which can be either 'URIs' or 'relative references' (kinda like if you consider general paths that can be absolute or relative paths, as is done in pathlib). - Instantiating relative URI references with Path(...) or p-strings may cause issues, because they might get turned into Windows paths and the like. It does seem like this could be worked around by for instance making another class like "RelativePath" or "RelativeRef", but there are some questions about when/how these should be instantiated. This may lead to a need slight backwards incompatibilities if implemented within pathlib. - "Queries" like '?this=that' after the path component have a special role in URIs, but in file system paths they can be parts of the file (or even directory) name. This might again be ambiguous when using relative paths / references. This could perhaps be dealt with by requiring more explicit handling when joining relative paths / references together. - "Fragments" like '#what'. This is essentially the same issue as with queries above and should be solved the same way. Anyway, both may be present at the same time. - '..' and '.' in relative paths / references. In URIs, there's a difference between 'scheme://foo/bar/' and 'scheme://foo/bar'. Merging the relative reference './baz' to the former gives 'scheme://foo/baz' while merging it to the latter gives 'scheme://foo/bar/baz'. I kinda wish the same thing was the standard with filesystem paths too. - Percent encoding of URIs: quite obvious -- should not be done before it is unambiguous that we deal with an URI. Perhaps it should be done only when the resource is accessed or when the URI is exported to a plain str or bytes etc. I suppose this is matter of what we would want in the repr. - I may still have missed or forgotten something. So, also with paths, especially relative ones, a library should "resist the temptation to guess", and carry around all the information until the context becomes unambiguous. For instance, when merging a relative reference with an explicit URI, the ambiguities about ?query and #fragment and about resolving the merged path disappear. > Another thought: requesting URLs. Basically the same as p'/etc/hosts'.write_text(secret). It's really important to have a dead simple library which is able to work with URLs. So, if I could do: > > p'https://mysite.com/{page}'.get() > Good idea. When I suggested extending Paths (and p-strings) to work with URLs, I indeed meant that it would be an instance of (a subclass of) Path, so that you do the same as with filesystem path objects: p'https://mysite.com/somepage.html'.read_text() or (p'https://mysite.com' / page).read_text() But who knows what we might end up with if we go down this path. An I mean a metaphorical path here, not necessarily Path :). Whatever it is, it probably can't be added to the stdlib right away. Still, we could take some measures regarding the language and stdlib now, to prepare for the future. -Koos From desmoulinmichel at gmail.com Tue Mar 29 11:07:42 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 29 Mar 2016 17:07:42 +0200 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: Message-ID: <56FA9A3E.9020602@gmail.com> Le 29/03/2016 16:42, Koos Zevenhoven a ?crit : > The 'Working with Path objects: p-strings?' thread spawned a > discussion about how URLs (or more generally URIs) and Paths should > work together. I suggest we move that discussion to this new thread. > The concept is 'explained' below in this email and quotes, but a > little bit of discussion happened in the other thread too. > > While I think that the decisions about p-strings (or a-strings for > addresses or whatever they should be) should keep URIs in mind, it is > premature to add the Path+URI fusion into the stdlib. I agree with > Paul Moore that this URL stuff should be on PyPI first. It could even > be library that monkey patches pathlib to accept URIs. Or a URI > library that instantiates Path objects when appropriate. Then there > could be a smooth transition into the stdlib some day. > > See all the stuff below: > > On Tue, Mar 29, 2016 at 11:44 AM, Sven R. Kunze wrote: >> >> On 27.03.2016 00:51, Koos Zevenhoven wrote: >>> >>> OT: >>> >>> To be honest, I do think it feels like URL:s are becoming (or have become) just as important as paths, and that pathlib.Path should in the future work with URLs just like it now works with windows and posix paths. The difference between "http://domain.xyz/" and "C:\\" is not huge. I also think there should be a Python type (stdlib or builtin), which handles JSON objects nicer than dicts do and has its own literal >>> >> >> That even occurred to me after we talked about the p-string (mainly because I am working in this field, so I basically need both file paths and URIs). >> > > Again, that you say you thought about it too perhaps means it's worth > discussing :). > >> >> Just for the record: "Path" might not be the most correct wording. There is a "file://" scheme which identifies locally located files. So, paths are basically a subset of URLs speaking functionality-wise. Thus, a better/more generic name would be "URL", "URI", "Link" or the like in order to avoid confusing of later generations. However, I think I could live with Path. >> > > Yes, these are concerns that should be considered if/when deciding > whether to make URI/URLs a subclass of Path or the other way around, > or something else. Anyway, since Path(...) already instantiates > different subclasses based on the situation, having it instantiate a > URI in some cases would not be completely unnatural. > > As suggested by Stephen, I've been looking into RFC 3986 as a whole, > and it seems that making instantiating both URIs and fs paths from > p-strings does not seem completely impossible. Some points below (you > can skip past them if you have to, there's more general discussion at > the end): > > - Only some URIs (or even URLs) can be reliably distinguished from > file paths. However, those that contain '://' could be automatically > turned into URI objects by p-strings [or Path(...)]. I suspect that > would cover the majority of use cases. > > (The unambiguous cases would be exactly those URIs that contain an > 'authority' component -- these always begin with 'scheme://' while > other's don't) > > - If we want allow URIs without an 'authority' component, like > mailto:someone at domain.com', they should be explicitly instantiated as > URI objects. > > - Some terminology: There are indeed 'URI's and 'relative references'. > Relative references are essentially the URI-equivalent of relative > paths. Then there are 'URI references' which can be either 'URIs' or > 'relative references' (kinda like if you consider general paths that > can be absolute or relative paths, as is done in pathlib). > > - Instantiating relative URI references with Path(...) or p-strings > may cause issues, because they might get turned into Windows paths and > the like. It does seem like this could be worked around by for > instance making another class like "RelativePath" or "RelativeRef", > but there are some questions about when/how these should be > instantiated. This may lead to a need slight backwards > incompatibilities if implemented within pathlib. > > - "Queries" like '?this=that' after the path component have a special > role in URIs, but in file system paths they can be parts of the file > (or even directory) name. This might again be ambiguous when using > relative paths / references. This could perhaps be dealt with by > requiring more explicit handling when joining relative paths / > references together. > > - "Fragments" like '#what'. This is essentially the same issue as with > queries above and should be solved the same way. Anyway, both may be > present at the same time. > > - '..' and '.' in relative paths / references. In URIs, there's a > difference between 'scheme://foo/bar/' and 'scheme://foo/bar'. Merging > the relative reference './baz' to the former gives 'scheme://foo/baz' > while merging it to the latter gives 'scheme://foo/bar/baz'. I kinda > wish the same thing was the standard with filesystem paths too. > > - Percent encoding of URIs: quite obvious -- should not be done before > it is unambiguous that we deal with an URI. Perhaps it should be done > only when the resource is accessed or when the URI is exported to a > plain str or bytes etc. I suppose this is matter of what we would want > in the repr. > > - I may still have missed or forgotten something. > > So, also with paths, especially relative ones, a library should > "resist the temptation to guess", and carry around all the information > until the context becomes unambiguous. For instance, when merging a > relative reference with an explicit URI, the ambiguities about ?query > and #fragment and about resolving the merged path disappear. > >> Another thought: requesting URLs. Basically the same as p'/etc/hosts'.write_text(secret). It's really important to have a dead simple library which is able to work with URLs. So, if I could do: >> >> p'https://mysite.com/{page}'.get() >> > > Good idea. When I suggested extending Paths (and p-strings) to work > with URLs, I indeed meant that it would be an instance of (a subclass > of) Path, so that you do the same as with filesystem path objects: > > p'https://mysite.com/somepage.html'.read_text() > > or > > (p'https://mysite.com' / page).read_text() > > But who knows what we might end up with if we go down this path. An I > mean a metaphorical path here, not necessarily Path :). Whatever it > is, it probably can't be added to the stdlib right away. Still, we > could take some measures regarding the language and stdlib now, to > prepare for the future. Yes but then there is a scope problem: are we providing just the parsing or also convenience method to access the ressource. E.G, you suggested: url('http://foo.com').get() For a ftp url, what would you do ? Ssh ? Why path would have them and not Http. Why http and not ftp ? Why ftp and not mailto: ? And if we do implement get() for http, then urllib ? Or request ? But then what about http 2 ? What about asyncio ? This needs to be sorted out first. Alhough, I do think URLS are very important, as I'm a web dev, integrating p"http://foo.com'.get() seems dangerous. We don't know how the web is going to move, and it's moving fast, while the stdlib is slow. > > -Koos > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From k7hoven at gmail.com Tue Mar 29 11:17:02 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 29 Mar 2016 18:17:02 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <20160329074634.GA25618@sjoerdjob.com> Message-ID: On Tue, Mar 29, 2016 at 1:41 PM, Paul Moore wrote: > On 29 March 2016 at 11:24, Koos Zevenhoven wrote: >>> As we all know, togo is user input containing >>> ../../../../../../../../../../../../../../../etc/passwd >>> >> >> How about >> >> p'/' /wherever/youwant//togo >> >> That is, the floordiv operator could be used to prevent 'togo' from going up >> the directory tree with "../../" or "/etc/passwd". The // would thus >> restrict the user (who provides `togo`) into /wherever/youwant/ and it its >> subdirectories. > > There have been a number of previous threads about the security of > path objects (and for that matter of os.path) and "sandboxing" paths > to disallow traversing "up" out of a particular directory. > > I've no idea whether the floordiv operator would work for this purpose > (I'm not a security expert, and most of my personal applications can > afford to take a very lax view of such things, luckily) but it's an > interesting idea. Potentially, it's a bit easy to miss, which may be a > problem. Actually, after having looked into RFC 3986s, I'd say using | instead of // for the path sandboxing operator may be better for at least two reasons: 1a. Because // is used in relative references for URIs to mean essentially 'go all the way up to scheme://', which is quite opposite to the concept of sandboxing. 1b. Because | kind of looks like a wall. 2. Because | has lower precedence than /, meaning that dontmesswiththis | relative / paths == dontmesswiththis | (relative / paths) while dontmesswiththis // relative / paths == (dontmesswiththis // relative) / paths So unless (dontmesswiththis // relative) somehow remembers the sandboxing barrier, the | would do the right thing and // the wrong thing. Then of course you might want it to remember the barrier anyway. I won't open the pandora's box in front of myself now regarding the old threads you mention. But if someone else wants to do that and/or there is something to discuss regarding this, I suggest to move the sandboxing discussion to a new thread. -Koos From ethan at stoneleaf.us Tue Mar 29 11:31:46 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 29 Mar 2016 08:31:46 -0700 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: Message-ID: <56FA9FE2.9060206@stoneleaf.us> On 03/29/2016 07:42 AM, Koos Zevenhoven wrote: > While I think that the decisions about p-strings (or a-strings for > addresses or whatever they should be) should keep URIs in mind, it is > premature to add the Path+URI fusion into the stdlib. I agree with > Paul Moore that this URL stuff should be on PyPI first. It could even > be library that monkey patches pathlib to accept URIs. Or a URI > library that instantiates Path objects when appropriate. Then there > could be a smooth transition into the stdlib some day. Pathlib is already complicated; unless we would be doing the same types of operations, and have the same mental model, there would be no point in trying to support URIs with Pathlib. If (a big if) we do add URI support, there would be no danger of mixing file paths with URI paths as you must specify which one you want: - WindowsPath - PosixPath - UriPath -- ~Ethan~ From k7hoven at gmail.com Tue Mar 29 11:48:24 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 29 Mar 2016 18:48:24 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA656B.5070707@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <56FA656B.5070707@mail.de> Message-ID: On Tue, Mar 29, 2016 at 2:22 PM, Sven R. Kunze wrote: > offtopic > > I wonder if it would be possible/necessary for the stdlib to include a set > of libraries marked as "featured". Not like "provisional" but more like "You > should use this one and be aware that it may upgrade faster than normal > stdlib modules". > > Basically like a "pull module". Just a third-party module pulled from the > source and upgraded with each Python version. That would maintain the > maintainers flexibility and the increase the availability of these "defacto > stdlib" modules. > +1. It is a different thing to find a random module on PyPI and depend on it, than to depend on something that is ~officially recommended. Or maybe some kind of package reputation system on PyPI? More threads? :) - Koos From p.f.moore at gmail.com Tue Mar 29 11:52:18 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Mar 2016 16:52:18 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <56FA656B.5070707@mail.de> Message-ID: On 29 March 2016 at 16:48, Koos Zevenhoven wrote: > On Tue, Mar 29, 2016 at 2:22 PM, Sven R. Kunze wrote: >> offtopic >> >> I wonder if it would be possible/necessary for the stdlib to include a set >> of libraries marked as "featured". Not like "provisional" but more like "You >> should use this one and be aware that it may upgrade faster than normal >> stdlib modules". >> >> Basically like a "pull module". Just a third-party module pulled from the >> source and upgraded with each Python version. That would maintain the >> maintainers flexibility and the increase the availability of these "defacto >> stdlib" modules. >> > > +1. > > It is a different thing to find a random module on PyPI and depend on > it, than to depend on something that is ~officially recommended. Or > maybe some kind of package reputation system on PyPI? More threads? :) Been discussed a few times previously (look for things like "sumo distribution", "breaking out the stdlib", "recommended packages"). So yes, again, should be a new thread and should also start with a bit of research into previous discussions. Paul From srkunze at mail.de Tue Mar 29 11:54:51 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 17:54:51 +0200 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: Message-ID: <56FAA54B.30408@mail.de> That's some post. Thanks a lot for collecting all that stuff. On 29.03.2016 16:42, Koos Zevenhoven wrote: >> That even occurred to me after we talked about the p-string (mainly because I am working in this field, so I basically need both file paths and URIs). > Again, that you say you thought about it too perhaps means it's worth > discussing :). :) > Yes, these are concerns that should be considered if/when deciding > whether to make URI/URLs a subclass of Path or the other way around, > or something else. Anyway, since Path(...) already instantiates > different subclasses based on the situation, having it instantiate a > URI in some cases would not be completely unnatural. You are right of course. I thought too narrow in this case. > As suggested by Stephen, I've been looking into RFC 3986 as a whole, > and it seems that making instantiating both URIs and fs paths from > p-strings does not seem completely impossible. Some points below (you > can skip past them if you have to, there's more general discussion at > the end): Well done. > - Only some URIs (or even URLs) can be reliably distinguished from > file paths. However, those that contain '://' could be automatically > turned into URI objects by p-strings [or Path(...)]. I suspect that > would cover the majority of use cases. I agree. 'http' and 'https' would make the majority of schemes, when it comes to the Web. 'ftp', 'ssh' and 'mailto' might follow. > (The unambiguous cases would be exactly those URIs that contain an > 'authority' component -- these always begin with 'scheme://' while > other's don't) > > - If we want allow URIs without an 'authority' component, like > mailto:someone at domain.com', they should be explicitly instantiated as > URI objects. > > - Some terminology: There are indeed 'URI's and 'relative references'. > Relative references are essentially the URI-equivalent of relative > paths. Then there are 'URI references' which can be either 'URIs' or > 'relative references' (kinda like if you consider general paths that > can be absolute or relative paths, as is done in pathlib). > > - Instantiating relative URI references with Path(...) or p-strings > may cause issues, because they might get turned into Windows paths and > the like. It does seem like this could be worked around by for > instance making another class like "RelativePath" or "RelativeRef", > but there are some questions about when/how these should be > instantiated. This may lead to a need slight backwards > incompatibilities if implemented within pathlib. > > - "Queries" like '?this=that' after the path component have a special > role in URIs, but in file system paths they can be parts of the file > (or even directory) name. This might again be ambiguous when using > relative paths / references. This could perhaps be dealt with by > requiring more explicit handling when joining relative paths / > references together. > > - "Fragments" like '#what'. This is essentially the same issue as with > queries above and should be solved the same way. Anyway, both may be > present at the same time. > > - '..' and '.' in relative paths / references. In URIs, there's a > difference between 'scheme://foo/bar/' and 'scheme://foo/bar'. Merging > the relative reference './baz' to the former gives 'scheme://foo/baz' > while merging it to the latter gives 'scheme://foo/bar/baz'. I kinda > wish the same thing was the standard with filesystem paths too. All of this makes me think that it MIGHT be better to leave the decision of whether it's a *real path*, a *URL* or a *URL path* to the user. Not sure if we can handle this lazily but I CAN imagine some "confusion" *sounding like PEP 428*. If we can found an unambiguous solution, that'll be awesome and would simplify a lot. > - Percent encoding of URIs: quite obvious -- should not be done before > it is unambiguous that we deal with an URI. Perhaps it should be done > only when the resource is accessed or when the URI is exported to a > plain str or bytes etc. I suppose this is matter of what we would want > in the repr. Good point. URIs should be able to handle both inputs. So, we would need to decide on a canonical form. > - I may still have missed or forgotten something. > > So, also with paths, especially relative ones, a library should > "resist the temptation to guess", and carry around all the information > until the context becomes unambiguous. For instance, when merging a > relative reference with an explicit URI, the ambiguities about ?query > and #fragment and about resolving the merged path disappear. Another point for "let the user decide". >> Another thought: requesting URLs. Basically the same as p'/etc/hosts'.write_text(secret). It's really important to have a dead simple library which is able to work with URLs. So, if I could do: >> >> p'https://mysite.com/{page}'.get() >> > Good idea. When I suggested extending Paths (and p-strings) to work > with URLs, I indeed meant that it would be an instance of (a subclass > of) Path, so that you do the same as with filesystem path objects: > > p'https://mysite.com/somepage.html'.read_text() > > or > > (p'https://mysite.com' / page).read_text() Ah, I see. Well, that's one approach. On the other hand, I can imagine a lot of people willing to do a "PUT", "DELETE" or "POST" (and the rather unknown other ones). It seems to me that a one-to-one mapping would be easier here instead of retrofitting. Although read_text might come in handy as an alias for "GET". :) That is when you don't care if you read locally or remotely. So, I can see room for this. > But who knows what we might end up with if we go down this path. An I > mean a metaphorical path here, not necessarily Path :). Let's see where this path lead us. ;) Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Tue Mar 29 12:17:15 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 29 Mar 2016 19:17:15 +0300 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: <56FA9A3E.9020602@gmail.com> References: <56FA9A3E.9020602@gmail.com> Message-ID: On Tue, Mar 29, 2016 at 6:07 PM, Michel Desmoulin wrote: > > url('http://foo.com').get() > > For a ftp url, what would you do ? > > Ssh ? > > Why path would have them and not Http. Why http and not ftp ? Why ftp > and not mailto: ? That's the nice thing about URI:s. They tell you if it's http or ftp, so the library can decide how to do a read_text() or whatever, if it is something that makes sense for that kind of URI. Already with filesystem Paths you have situations where you can't do some operation, for instance because of permissions, even if the Path points to something that exists, and still those methods exist. They will just fail. That's life. Trying to read_text() on a mailto URI should fail too. > > And if we do implement get() for http, then urllib ? Or request ? But > then what about http 2 ? What about asyncio ? Yes, the asyncio / blocking io is a whole other issue. In fact, I started a thread about that almost a year ago, but I think my timing was really bad, since the async/await PEP 492 was just about to be accepted. > This needs to be sorted out first. > > Alhough, I do think URLS are very important, as I'm a web dev, > integrating p"http://foo.com'.get() seems dangerous. We don't know how > the web is going to move, and it's moving fast, while the stdlib is slow. I completely agree it's important to try to look into the future. However, as long as we believe the meaning of read_text() or get() will not change, how much harm can we do? I'm sure reading a text file or query from a URL is not going to disappear any time soon. - Koos From k7hoven at gmail.com Tue Mar 29 12:44:41 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 29 Mar 2016 19:44:41 +0300 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: <56FAA54B.30408@mail.de> References: <56FAA54B.30408@mail.de> Message-ID: On Tue, Mar 29, 2016 at 6:54 PM, Sven R. Kunze wrote: > That's some post. Thanks a lot for collecting all that stuff. > > On 29.03.2016 16:42, Koos Zevenhoven wrote: >> [.....] >> Good idea. When I suggested extending Paths (and p-strings) to work >> with URLs, I indeed meant that it would be an instance of (a subclass >> of) Path, so that you do the same as with filesystem path objects: >> >> p'https://mysite.com/somepage.html'.read_text() >> >> or >> >> (p'https://mysite.com' / page).read_text() >> > > Ah, I see. Well, that's one approach. > > On the other hand, I can imagine a lot of people willing to do a "PUT", > "DELETE" or "POST" (and the rather unknown other ones). It seems to me that > a one-to-one mapping would be easier here instead of retrofitting. > Yeah, if subclassing Path, when the URI object gets instantiated, then the object can have all kinds of specialized methods and behavior. I mean, when the address string starts with 'https://' [*], it can instantiate a HttpsURI object or something, which provides Path stuff like .read_text() as well as more https-specific methods. [*] "https://" does not make a whole lot of sense in (the beginning of) a file-system path, even if pathlib.Path currently pretends that it does. > Although read_text might come in handy as an alias for "GET". :) > That is when you don't care if you read locally or remotely. So, I can see > room for this. Yes :). - Koos From random832 at fastmail.com Tue Mar 29 12:49:27 2016 From: random832 at fastmail.com (Random832) Date: Tue, 29 Mar 2016 12:49:27 -0400 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: <56FAA54B.30408@mail.de> Message-ID: <1459270167.1678328.562687826.62592AC5@webmail.messagingengine.com> On Tue, Mar 29, 2016, at 12:44, Koos Zevenhoven wrote: > > Although read_text might come in handy as an alias for "GET". :) > > That is when you don't care if you read locally or remotely. So, I can see > > room for this. > > Yes :). What about a read_json? Should the underlying request set Accept types accordingly (read_json demands text/json or application/json, read_text demands text/plain? what if I want HTML? Need another method for that?) What if content-transfer-encoding isn't set, or the data contains bytes that aren't valid for it? From k7hoven at gmail.com Tue Mar 29 13:12:17 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 29 Mar 2016 20:12:17 +0300 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: <56FA9FE2.9060206@stoneleaf.us> References: <56FA9FE2.9060206@stoneleaf.us> Message-ID: On Tue, Mar 29, 2016 at 6:31 PM, Ethan Furman wrote: > On 03/29/2016 07:42 AM, Koos Zevenhoven wrote: > >> While I think that the decisions about p-strings (or a-strings for >> addresses or whatever they should be) should keep URIs in mind, it is >> premature to add the Path+URI fusion into the stdlib. I agree with >> Paul Moore that this URL stuff should be on PyPI first. It could even >> be library that monkey patches pathlib to accept URIs. Or a URI >> library that instantiates Path objects when appropriate. Then there >> could be a smooth transition into the stdlib some day. > > > Pathlib is already complicated; unless we would be doing the same types of > operations, and have the same mental model, there would be no point in > trying to support URIs with Pathlib. Agreed. I think the main benefits besides flexibility would indeed be to be able to use the same mental model (and added syntax?) with Paths and URIs as long as you do operations that make sense with the different 'addresses'. If you need something more specific, the subclasses can still add whatever features make sense for the given scheme/protocol. - Koos From desmoulinmichel at gmail.com Tue Mar 29 13:13:40 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 29 Mar 2016 19:13:40 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <56FA656B.5070707@mail.de> Message-ID: <56FAB7C4.6060904@gmail.com> Le 29/03/2016 17:48, Koos Zevenhoven a ?crit : > On Tue, Mar 29, 2016 at 2:22 PM, Sven R. Kunze wrote: >> offtopic >> >> I wonder if it would be possible/necessary for the stdlib to include a set >> of libraries marked as "featured". Not like "provisional" but more like "You >> should use this one and be aware that it may upgrade faster than normal >> stdlib modules". >> >> Basically like a "pull module". Just a third-party module pulled from the >> source and upgraded with each Python version. That would maintain the >> maintainers flexibility and the increase the availability of these "defacto >> stdlib" modules. >> > > +1. > > It is a different thing to find a random module on PyPI and depend on > it, than to depend on something that is ~officially recommended. Or > maybe some kind of package reputation system on PyPI? More threads? :) Stop growing the stdlib and make Python a distribution is a topic I already read here. It may be time to revive it again. Last time, it was deeply linked with packaging and battery included philosophy. > > - Koos > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From mike at selik.org Tue Mar 29 13:48:17 2016 From: mike at selik.org (Michael Selik) Date: Tue, 29 Mar 2016 17:48:17 +0000 Subject: [Python-ideas] custom predicate for all() & any() In-Reply-To: <56FA7FE9.60507@mail.de> References: <1C405062-977D-4324-8A14-D37AAB752F7E@gmail.com> <56FA7FE9.60507@mail.de> Message-ID: On Tue, Mar 29, 2016 at 9:15 AM Sven R. Kunze wrote: > On 26.03.2016 18:04, Terry Reedy wrote: > > So proposed all(iterable, predicate) == all(map(predicate, iterable)) > > I am no native English speaker. Is predicate the right word for this? > I'm not sure being a native speaker would help in this case. :-) One reasonably common definition of "predicate": a function that tests a condition on a value and returns True/False. Some folks define it more broadly, but then the word loses its usefulness as distinct from other kinds of functions. The usage of "predicate" here comes from predicate logic, or more specifically the usage of the term in languages like Lisp. Where Python says ``'42'.isnumeric()`` a lisper might say ``(numberp 42)``, the "p" suffix indicating that it's a predicate function. https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node69.html > I get the feeling we would rebuilding things over and over again: list > comprehensions, filter, map, itertools, for-loops etc. > Certainly feels like it. > I think the underlying issue is the number of parentheses involved. I > myself somehow avoid nesting too many function calls into one line just > because of that; not because of the complexity involved but because it > looks strange. Same here. But I don't mind the occasional throwaway variable. When it's a multi-step process, a good name for an intermediate stage helps me think. > Of course the number of parentheses involved is an > indicator of the complexity. Not sure if there is another way of > handling this; maybe piping or something. Would be great if we could > avoid writing ))) or ))))). > http://xkcd.com/297/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Mar 29 14:32:53 2016 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 29 Mar 2016 13:32:53 -0500 Subject: [Python-ideas] custom predicate for all() & any() In-Reply-To: References: <1C405062-977D-4324-8A14-D37AAB752F7E@gmail.com> <56FA7FE9.60507@mail.de> Message-ID: On Mar 29, 2016 12:49 PM, "Michael Selik" wrote: > > > > On Tue, Mar 29, 2016 at 9:15 AM Sven R. Kunze wrote: >> >> On 26.03.2016 18:04, Terry Reedy wrote: >> > So proposed all(iterable, predicate) == all(map(predicate, iterable)) >> >> I am no native English speaker. Is predicate the right word for this? > > > I'm not sure being a native speaker would help in this case. :-) > > One reasonably common definition of "predicate": a function that tests a condition on a value and returns True/False. Some folks define it more broadly, but then the word loses its usefulness as distinct from other kinds of functions. The usage of "predicate" here comes from predicate logic, or more specifically the usage of the term in languages like Lisp. Where Python says ``'42'.isnumeric()`` a lisper might say ``(numberp 42)``, the "p" suffix indicating that it's a predicate function. > > https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node69.html > >> >> I get the feeling we would rebuilding things over and over again: list >> comprehensions, filter, map, itertools, for-loops etc. > > > Certainly feels like it. * "predicate" seems to be the common term * (predicate, iterable) seems to be the, at least in Python, common function signature * itertools.ifilter (predicate, iterable) https://docs.python.org/2/library/itertools.html#itertools.ifilter * python3: filter(function, iterable)) https://docs.python.org/3/library/functions.html#filter * toolz: *(predicate, iterable) * funcy.ifilter(pred, seq) http://funcy.readthedocs.org/en/stable/seqs.html#transform-and-filter ... "iterable" / sequence > >> >> I think the underlying issue is the number of parentheses involved. I >> myself somehow avoid nesting too many function calls into one line just >> because of that; not because of the complexity involved but because it >> looks strange. > > > Same here. But I don't mind the occasional throwaway variable. When it's a multi-step process, a good name for an intermediate stage helps me think. much easier to debug with e.g. ipdb, pdb, when there are intermediate variables. IDK if there are any less references to track without intermediate variables? > >> >> Of course the number of parentheses involved is an >> indicator of the complexity. Not sure if there is another way of >> handling this; * git commit * :PymodeLintAuto #vim * git diff * git commit ... or gg=G #vim > maybe piping or something. Would be great if we could >> avoid writing ))) or ))))). > > > http://xkcd.com/297/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Mar 29 14:35:01 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 20:35:01 +0200 Subject: [Python-ideas] custom predicate for all() & any() In-Reply-To: References: <1C405062-977D-4324-8A14-D37AAB752F7E@gmail.com> <56FA7FE9.60507@mail.de> Message-ID: <56FACAD5.5060600@mail.de> On 29.03.2016 19:48, Michael Selik wrote: > > > On Tue, Mar 29, 2016 at 9:15 AM Sven R. Kunze > wrote: > > On 26.03.2016 18:04, Terry Reedy wrote: > > So proposed all(iterable, predicate) == all(map(predicate, > iterable)) > > I am no native English speaker. Is predicate the right word for this? > > > I'm not sure being a native speaker would help in this case. :-) > > One reasonably common definition of "predicate": a function that tests > a condition on a value and returns True/False. Some folks define it > more broadly, but then the word loses its usefulness as distinct from > other kinds of functions. The usage of "predicate" here comes from > predicate logic, or more specifically the usage of the term in > languages like Lisp. Where Python says ``'42'.isnumeric()`` a lisper > might say ``(numberp 42)``, the "p" suffix indicating that it's a > predicate function. > > https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node69.html I see. That was why I would associate "predicate" more with filter than with map. > I get the feeling we would rebuilding things over and over again: list > comprehensions, filter, map, itertools, for-loops etc. > > > Certainly feels like it. :) > I think the underlying issue is the number of parentheses involved. I > myself somehow avoid nesting too many function calls into one line > just > because of that; not because of the complexity involved but because it > looks strange. > > > Same here. But I don't mind the occasional throwaway variable. When > it's a multi-step process, a good name for an intermediate stage helps > me think. Yep. That is how it should be done then. > Of course the number of parentheses involved is an > indicator of the complexity. Not sure if there is another way of > handling this; maybe piping or something. Would be great if we could > avoid writing ))) or ))))). > > > http://xkcd.com/297/ :D -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Mar 29 14:38:22 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 20:38:22 +0200 Subject: [Python-ideas] custom predicate for all() & any() In-Reply-To: References: <1C405062-977D-4324-8A14-D37AAB752F7E@gmail.com> <56FA7FE9.60507@mail.de> Message-ID: <56FACB9E.5010200@mail.de> On 29.03.2016 20:32, Wes Turner wrote: > * "predicate" seems to be the common term > > * (predicate, iterable) seems to be the, at least in Python, common > function signature > > * itertools.ifilter (predicate, iterable) > > https://docs.python.org/2/library/itertools.html#itertools.ifilter > > * python3: filter(function, iterable)) > > https://docs.python.org/3/library/functions.html#filter > > * toolz: *(predicate, iterable) > > * funcy.ifilter(pred, seq) > > http://funcy.readthedocs.org/en/stable/seqs.html#transform-and-filter > > ... "iterable" / sequence > filtering, yes; not mapping > >> Of course the number of parentheses involved is an > > >> indicator of the complexity. Not sure if there is another way of > >> handling this; > > * git commit > * :PymodeLintAuto #vim > * git diff > * git commit > > ... or gg=G #vim > PyCharm? Best, Sven From brett at python.org Tue Mar 29 14:51:49 2016 From: brett at python.org (Brett Cannon) Date: Tue, 29 Mar 2016 18:51:49 +0000 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FA4E31.3010908@gmail.com> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> Message-ID: On Tue, 29 Mar 2016 at 02:45 Michel Desmoulin wrote: > > > Le 29/03/2016 09:50, Sven R. Kunze a ?crit : > > On 25.03.2016 22:56, Michel Desmoulin wrote: > >> Although path.py, which I have been using for years now (and still > >> prefer to pathlib) subclass str and never caused any problem whatsoever. > >> So really, we should pinpoint where it could be an issue and see if this > >> is a frequent problem, because right now, it seems more a decision based > >> on purity than practicality. > > > > I agree. The PEP says: > > > > """ > > No confusion with builtins > > ---------------------------------- > > In this proposal, the path classes do not derive from a builtin type. > > This contrasts with some other Path class proposals which were derived > > from str . They also do not pretend to implement the sequence protocol: > > if you want a path to act as a sequence, you have to lookup a dedicated > > attribute (the parts attribute). > > > > Not behaving like one of the basic builtin types also minimizes the > > potential for confusion if a path is combined by accident with genuine > > builtin types. > > """" > > > > I have to admit I cannot follow these statements but they should have > > appeared to be necessary back then. As experience shows the PyPI module > > fared far better here. > > > > > > I am great a fan of theory over practice as what has been proven in > > theory cannot be proven wrong in practice. However, this only holds if > > we talk about hard proof. For soft things like "human interaction", > > "mutual understanding" or "potential for confusion", only the practice > > of many many people can "prove" what's useful, what's "practical". > > > > > > +1. Can somebody defend, with practical examples, the imperative to > refuse to inherit from str ? And weight it against the benefits of it ? > > We have been doing it for years, and so far it's been really, really > nice. I can't recall problem I had because of it ever. I can recall > numerous editions of my code because I forgot str() on pathlib.Path. > I may regret this, but I will give this a shot. :) I wrote a blog post at http://www.snarky.ca/why-pathlib-path-doesn-t-inherit-from-str because my response was getting a bit long. I have pasted it below in CommonMark. ----- Over on [python-ideas](https://mail.python.org/mailman/listinfo/python-ideas) a discussion has broken out about somehow trying to make `p'/some/path/to/a/file` return an [instance of `pathlib.Path`]( https://docs.python.org/3/library/pathlib.html#pathlib.Path). This led to a splinter discussion as to why `pathlib.Path` doesn't inherit from `str`? I figured instead of burying my response to this question in the thread I'd blog about it to try and explain one approach to API design. I think the key question in all of this is whether paths are semantically equivalent to a sequence of characters? Obviously the answer is "no" since paths have structure, directly represent something like files and directories, etc. Now paths do have a serialized representation as strings -- at least most of the time, but I'm ignoring Linux and the crazy situation of binary paths -- which is why C APIs take in `char *` as the representation of paths (and because C tends to make one try to stick with the types that are part of the C standard). So not all strings represent paths, but all paths can be represented by strings. I think we can all agree with that. OK, so if all paths can be represented as strings, why don't we just make `pathlib.Path` subclass `str`? Well, as I said earlier, not all strings are paths. You can't concatenate a string representing a path with some other random string and expect to get back a string that still represents a valid string (and I'm not talking about "valid" as in "the file doesn't exist", I'm talking about "syntactically not possible"). This is what [PEP 428]( https://www.python.org/dev/peps/pep-0428/) is talking about when it says: > Not behaving like one of the basic builtin types [list str] also minimizes the potential for confusion if a path is combined by accident with genuine builtin types [like str]. Now at this point someone someone will start saying, "but Brett, '[practicality beats purity](https://www.python.org/dev/peps/pep-0020/)' and all of those pre-existing, old OS APIs want a string as an argument!" And you're right, in terms of practicality it would be easier for `pathlib.Path` to inherit from `str` ... for the short/medium term. One thing people are forgetting is that "[explicit is better than implicit]( https://www.python.org/dev/peps/pep-0020/)" as well and as with most design decisions there's not one clear answer. Implicitly treating a path as a string currently works, but that's just because we have inherited a suboptimal representation for paths from C. Now if you use an explicit representation of paths like `pathlib.Path` then you gain semantic separation and understanding of what you are working with. You also avoid any issues that come with implicit `str` compatibility as I pointed out earlier. And before anyone says, "so what?", think about this: Python 2/3. While I'm sure some will say I'm overblowing the comparison, but Python 3 came about because the implicit compatibility of binary and textual data in Python 2 caused major headaches to the point that we made a backwards-incompatible change that has caused widespread ramifications (but which we seem to be coming out on the other side of). By not inheriting from `str`, `pathlib.Path` has avoided a similar potential issue from the get-go. Or another way of looking at it is to ask why doesn't `dict` inherit from `str` so that it's easier to make it easier to stick in the body of an HTTP response so it can be implicitly treated as JSON? If you going, "eww, no because they are different types" then you at least understand the argument I'm trying to make (if you're going "I like that idea", then I think [Perl]( https://www.perl.org/) might be more to your liking than Python and that's fine since the two languages just have different designs). Now back to that "practicality beats purity" side of this. Obviously there are lots of APIs out there that take a string as an argument for a file path. And I understand people don't like using `str(path)` or the upcoming/new [`getattr(path, 'path', path)` idiom]( https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path) to work around the limitations forced upon them by other APIs that only take a `str` because they occasionally forget it. For this point, I have two answers. One is still "explicit is better than implicit" and you should have tests to begin with to catch when you forget to convert your `pathlib.Path` objects to `str` as needed. I'm just one of those programmers who's willing to type a bit more for easy-to-read code that's less error-prone within reason (and I obviously view this as reasonable). But more importantly, why don't you work with the code and/or projects that are forcing you to convert to `str` to start accepting `pathlib` objects as well for those same APIs? If projects start to work on making `pathlib` objects acceptable anywhere a path is accepted then once Python 3.4 is the oldest version of Python that is supported by the project then they can start considering dropping support for `str` as paths in new releases (obviously this also includes dropping support for Python 2 unless people use [pathlib2](https://pypi.python.org/pypi/pathlib2/)). And I fully admit the stdlib is not exempt from a place that needs updating, so for `importlib` I have [opened an issue](http://bugs.python.org/issue26667) to update it to show this isn't just talk on my end (unfortunately there's some unique bootstrapping problems to import where stuff like `sys.path` probably can't be updated since that has to exist before you can import `pathlib` and that sort of thing ripples out, but I'm hoping I can update at least some things and this is a very unique case of potentially needing to stick with `str`). Hopefully if people start asking for `pathlib` support from projects they will add it, eventually leading to an alleviation of the desire to have `pathlib.Path` inherit from `str`. -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Tue Mar 29 14:59:02 2016 From: random832 at fastmail.com (Random832) Date: Tue, 29 Mar 2016 14:59:02 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> Message-ID: <1459277942.1710166.562817970.6C518420@webmail.messagingengine.com> On Tue, Mar 29, 2016, at 14:51, Brett Cannon wrote: > OK, so if all paths can be represented as strings, why don't we just make > `pathlib.Path` subclass `str`? Well, as I said earlier, not all strings > are > paths. You can't concatenate a string representing a path with some other > random string and expect to get back a string that still represents a > valid > string I assume you mean a valid path. You can't add an int to a float and expect to get an integer back, either. If you're adding it to a random string you're clearly not using it in a path context, and the type of the result won't be Path anyway. From ethan at stoneleaf.us Tue Mar 29 15:22:00 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 29 Mar 2016 12:22:00 -0700 Subject: [Python-ideas] Working with pathlib.Path objects In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> Message-ID: <56FAD5D8.2020001@stoneleaf.us> On 03/29/2016 11:51 AM, Brett Cannon wrote: > I wrote a blog post at > http://www.snarky.ca/why-pathlib-path-doesn-t-inherit-from-str because > my response was getting a bit long. I have pasted it below in CommonMark. Thanks for that. For myself at least, I also find the complexity of pathlib completely unnecessary: I see the structure of paths as loose at best -- at least in regards to the kinds of things we do with paths: - changing file names either in part or in whole, - changing directory names or making new directories from portions of the names of previous directories, - looking for directory or file names that have some substring as part of their names - etc. And yes, it is possible to incorrectly combine two path pieces (or a path and a string) together and get something that is semantically invalid -- but you can do that with any thing more complex than a random jumble of characters: - sentences - names - phrases - paragraphs - etc. But oh well -- what's done is done and we're probably stuck with it now. -- ~Ethan~ From mike at selik.org Tue Mar 29 16:51:45 2016 From: mike at selik.org (Michael Selik) Date: Tue, 29 Mar 2016 16:51:45 -0400 Subject: [Python-ideas] Package reputation system In-Reply-To: <56FAB7C4.6060904@gmail.com> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FA5F7E.3070001@gmail.com> <56FA656B.5070707@mail.de> <56FAB7C4.6060904@gmail.com> Message-ID: <4DD01140-B429-4FD0-8164-C965210A2282@selik.org> In days of yore, before package managers, people used to download source code and read it. Maybe not all of it, but enough to feel not terribly scared when running that code. In modern times, with centralized package repositories and convenient installer tools, we want a better way to know "what?s the right package to use for this task?" It's a "first-world problem" in a sense. There are too many products in my supermarket aisle! On a personal note, I have on occasion spent twenty minutes choosing a toothpaste at Target. If I care enough, I'll take a moment to look at how many downloads have been counted recently, how many issues there are (usually on GitHub), how many contributors, etc. I'll read the docs. I might even poke around in the source. I'll also check Google rankings to see if people are chatting about the module and linking to it. I'm not sure if there's a good centralized solution to this problem, but it's a question many people are asking: How do I know which non-stdlib module to use? Back at Georgia Tech, my professor [0] once told me that the way to get rich is to invent an index. He was referring to Richard Florida's "Creative Class" book and the subsequent "Creativity Index" consulting that Florida provided to various municipalities. People who score high on the index pay you to speak. People who score low on the index pay you to consult. There are a few companies who sell a Python package reputation service, along with some distribution tools. Continuum's Anaconda, Enthought's Canopy, and ActiveState's ActivePython come to mind. There's clearly value in helping people answer this question. [0] http://www.coa.gatech.edu/people/david-sawicki > On Mar 29, 2016, at 1:13 PM, Michel Desmoulin wrote: > Le 29/03/2016 17:48, Koos Zevenhoven a ?crit : >> On Tue, Mar 29, 2016 at 2:22 PM, Sven R. Kunze wrote: >>> offtopic >>> >>> I wonder if it would be possible/necessary for the stdlib to include a set >>> of libraries marked as "featured". Not like "provisional" but more like "You >>> should use this one and be aware that it may upgrade faster than normal >>> stdlib modules". >>> >>> Basically like a "pull module". Just a third-party module pulled from the >>> source and upgraded with each Python version. That would maintain the >>> maintainers flexibility and the increase the availability of these "defacto >>> stdlib" modules. >>> >> >> +1. >> >> It is a different thing to find a random module on PyPI and depend on >> it, than to depend on something that is ~officially recommended. Or >> maybe some kind of package reputation system on PyPI? More threads? :) > > > Stop growing the stdlib and make Python a distribution is a topic I > already read here. It may be time to revive it again. > > Last time, it was deeply linked with packaging and battery included > philosophy. From srkunze at mail.de Tue Mar 29 17:00:08 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 29 Mar 2016 23:00:08 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> Message-ID: <56FAECD8.3030605@mail.de> On 29.03.2016 20:51, Brett Cannon wrote: > I may regret this, but I will give this a shot. :) Never, Brett. I think everybody here appreciate a decent discussion. Who knows maybe, somebody advocating for path->str misses an important piece which in turn misses the PEP to describe clearly. > > I wrote a blog post at > http://www.snarky.ca/why-pathlib-path-doesn-t-inherit-from-str because > my response was getting a bit long. I have pasted it below in CommonMark. Thanks +1 So, let's see: > Over on > [python-ideas](https://mail.python.org/mailman/listinfo/python-ideas) > a discussion has broken out about somehow trying to make > `p'/some/path/to/a/file` return an [instance of > `pathlib.Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path). > This led to a splinter discussion as to why `pathlib.Path` doesn't > inherit from `str`? I figured instead of burying my response to this > question in the thread I'd blog about it to try and explain one > approach to API design. > > I think the key question in all of this is whether paths are > semantically equivalent to a sequence of characters? Obviously the > answer is "no" since paths have structure, directly represent > something like files and directories, etc. Now paths do have a > serialized representation as strings -- at least most of the time, but > I'm ignoring Linux and the crazy situation of binary paths -- which is > why C APIs take in `char *` as the representation of paths (and > because C tends to make one try to stick with the types that are part > of the C standard). So not all strings represent paths, but all paths > can be represented by strings. I think we can all agree with that. > > OK, so if all paths can be represented as strings, why don't we just > make `pathlib.Path` subclass `str`? Well, as I said earlier, not all > strings are paths. You can't concatenate a string representing a path > with some other random string and expect to get back a string that > still represents a valid string (and I'm not talking about "valid" as > in "the file doesn't exist", I'm talking about "syntactically not > possible"). This is what [PEP > 428](https://www.python.org/dev/peps/pep-0428/) is talking about when > it says: > > > Not behaving like one of the basic builtin types [list str] also > minimizes the potential for confusion if a path is combined by > accident with genuine builtin types [like str]. Valid string? I assume you mean path, right? > Now at this point someone someone will start saying, "but Brett, > '[practicality beats > purity](https://www.python.org/dev/peps/pep-0020/)' and all of those > pre-existing, old OS APIs want a string as an argument!" And you're > right, in terms of practicality it would be easier for `pathlib.Path` > to inherit from `str` ... for the short/medium term. One thing people > are forgetting is that "[explicit is better than > implicit](https://www.python.org/dev/peps/pep-0020/)" as well and as > with most design decisions there's not one clear answer. Implicitly > treating a path as a string currently works, but that's just because > we have inherited a suboptimal representation for paths from C. Now if > you use an explicit representation of paths like `pathlib.Path` then > you gain semantic separation and understanding of what you are working > with. You also avoid any issues that come with implicit `str` > compatibility as I pointed out earlier. I agree with the concatenation issue when using plain strings for paths. However, something that I cannot leave uncommented is "suboptimal representation for paths". What would your optimal representation for paths look like? I cannot believe that the current representation is so bad and has been for so long and nobody, really nobody, has anything done about it. > And before anyone says, "so what?", think about this: Python 2/3. > While I'm sure some will say I'm overblowing the comparison, but > Python 3 came about because the implicit compatibility of binary and > textual data in Python 2 caused major headaches to the point that we > made a backwards-incompatible change that has caused widespread > ramifications (but which we seem to be coming out on the other side > of). By not inheriting from `str`, `pathlib.Path` has avoided a > similar potential issue from the get-go. Or another way of looking at > it is to ask why doesn't `dict` inherit from `str` so that it's easier > to make it easier to stick in the body of an HTTP response so it can > be implicitly treated as JSON? If you going, "eww, no because they are > different types" then you at least understand the argument I'm trying > to make (if you're going "I like that idea", then I think > [Perl](https://www.perl.org/) might be more to your liking than Python > and that's fine since the two languages just have different designs). I think most "practicality beats purity" folks don't want that either. They are just bloody lazy. They actually want the benefits of both, the pure datastructure with its convenience methods and the dirty str-like thing with its convenience methods. People don't like it when somebody takes one bag of convenience away from them if they easily can have both. ;-) Better sell it differently. So, let's read on. ;) > Now back to that "practicality beats purity" side of this. Obviously > there are lots of APIs out there that take a string as an argument for > a file path. And I understand people don't like using `str(path)` or > the upcoming/new [`getattr(path, 'path', path)` > idiom](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path) > to work around the limitations forced upon them by other APIs that > only take a `str` because they occasionally forget it. For this point, > I have two answers. Sorry to interrupt your here but you missed one important piece (didn't we agree on selling this better?): people do nasty little string manipulations/analysis/regex with their paths for whatever reasons. Right now, it's easier than ever because, well, because paths are strings. :) Selling pathlib with the "getattr(path, 'path', path)" idiom is not going to help those fellows. I mean, are you serious? It looks ridiculous. :-/ Especially the point that a "path" has a "path" is not really getting into my head. Either a "path" is a "path" or it better be named "file/directory" which has a "path". path.path vs file.path/dir.path You need to work on your selling skills, Brett. ;-) > One is still "explicit is better than implicit" and you should have > tests to begin with to catch when you forget to convert your > `pathlib.Path` objects to `str` as needed. I'm just one of those > programmers who's willing to type a bit more for easy-to-read code > that's less error-prone within reason (and I obviously view this as > reasonable). > > But more importantly, why don't you work with the code and/or projects > that are forcing you to convert to `str` to start accepting `pathlib` > objects as well for those same APIs? If projects start to work on > making `pathlib` objects acceptable anywhere a path is accepted then > once Python 3.4 is the oldest version of Python that is supported by > the project then they can start considering dropping support for `str` > as paths in new releases (obviously this also includes dropping > support for Python 2 unless people use > [pathlib2](https://pypi.python.org/pypi/pathlib2/) > ). And I fully admit the > stdlib is not exempt from a place that needs updating, so for > `importlib` I have [opened an > issue](http://bugs.python.org/issue26667) to update it to show this > isn't just talk on my end (unfortunately there's some unique > bootstrapping problems to import where stuff like `sys.path` probably > can't be updated since that has to exist before you can import > `pathlib` and that sort of thing ripples out, but I'm hoping I can > update at least some things and this is a very unique case of > potentially needing to stick with `str`). Hopefully if people start > asking for `pathlib` support from projects they will add it, > eventually leading to an alleviation of the desire to have > `pathlib.Path` inherit from `str`. I think updating the stdlib looks sufficiently simple to start hacking CPython. So, I would be willing to help here once the github repo is up and running. :) Nice post, Brett. Seems you covered the most important point quite well and also gave an explanation of why people request path->str. Best, Sven From greg.ewing at canterbury.ac.nz Tue Mar 29 17:30:02 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 30 Mar 2016 10:30:02 +1300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> Message-ID: <56FAF3DA.4090200@canterbury.ac.nz> Paul Moore wrote: > > P(wherever)/youwant/togo If P were a suitable seed object, this could be P/wherever/youwant/togo Why do we think we need special syntax for this? If it's just to avoid having to import stuff, that can be addressed by adding to the builtins. -- Greg From brett at python.org Tue Mar 29 17:37:23 2016 From: brett at python.org (Brett Cannon) Date: Tue, 29 Mar 2016 21:37:23 +0000 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <1459277942.1710166.562817970.6C518420@webmail.messagingengine.com> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <1459277942.1710166.562817970.6C518420@webmail.messagingengine.com> Message-ID: On Tue, 29 Mar 2016 at 11:59 Random832 wrote: > On Tue, Mar 29, 2016, at 14:51, Brett Cannon wrote: > > OK, so if all paths can be represented as strings, why don't we just make > > `pathlib.Path` subclass `str`? Well, as I said earlier, not all strings > > are > > paths. You can't concatenate a string representing a path with some other > > random string and expect to get back a string that still represents a > > valid > > string > > I assume you mean a valid path. > Yep, it's a typo. > > You can't add an int to a float and expect to get an integer back, > either. Sure, but that's because int and float are both part of the same type hierarchy (https://www.python.org/dev/peps/pep-3141/). > If you're adding it to a random string you're clearly not using > it in a path context, and the type of the result won't be Path anyway. > You're assuming it's always clear. If that was always true then no one would ever find a bug in their code when porting to Python 3. We can all assume we won't screw up, but the fact bugs exist shows that's not true. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Mar 29 17:39:54 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 29 Mar 2016 14:39:54 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FAF3DA.4090200@canterbury.ac.nz> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> Message-ID: <56FAF62A.9020104@stoneleaf.us> On 03/29/2016 02:30 PM, Greg Ewing wrote: > Paul Moore wrote: >> >> P(wherever)/youwant/togo > > If P were a suitable seed object, this could be > > P/wherever/youwant/togo An interesting idea, but how would you differentiate between relative and absolute paths? -- ~Ethan~ From k7hoven at gmail.com Tue Mar 29 17:47:39 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 30 Mar 2016 00:47:39 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FAF62A.9020104@stoneleaf.us> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> Message-ID: On Mar 30, 2016 00:39, "Ethan Furman" wrote: > > On 03/29/2016 02:30 PM, Greg Ewing wrote: >> >> Paul Moore wrote: >>> >>> >>> P(wherever)/youwant/togo >> >> >> If P were a suitable seed object, this could be >> >> P/wherever/youwant/togo > > > An interesting idea, but how would you differentiate between relative and absolute paths? > Indeed interesting. It would currently work with P = Path(''), kinda like my earlier p'' / ... example. It works even with relative paths, although they would not look like relative ones because of the first /. How short can a name be to be acceptable in builtins?-) K (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Mar 29 17:57:09 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 29 Mar 2016 22:57:09 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FAECD8.3030605@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> Message-ID: On 29 March 2016 at 22:00, Sven R. Kunze wrote: > However, something that I cannot leave uncommented is "suboptimal > representation for paths". What would your optimal representation for paths > look like? I cannot believe that the current representation is so bad and > has been for so long and nobody, really nobody, has anything done about it. Well, pathlib.Path :-) The point here is that C's char* representation is a serialisation of a path object, just like 123 is a serialisation of an integer object. We don't object to having to convert user input to a string if we need to, why object to having to convert it to a Path if appropriate? Paul From brett at python.org Tue Mar 29 18:03:14 2016 From: brett at python.org (Brett Cannon) Date: Tue, 29 Mar 2016 22:03:14 +0000 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FAECD8.3030605@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> Message-ID: On Tue, 29 Mar 2016 at 14:00 Sven R. Kunze wrote: > On 29.03.2016 20:51, Brett Cannon wrote: > > I may regret this, but I will give this a shot. :) > > Never, Brett. I think everybody here appreciate a decent discussion. Who > knows maybe, somebody advocating for path->str misses an important piece > which in turn misses the PEP to describe clearly. > > > > > I wrote a blog post at > > http://www.snarky.ca/why-pathlib-path-doesn-t-inherit-from-str because > > my response was getting a bit long. I have pasted it below in CommonMark. > > Thanks +1 > > So, let's see: > > > Over on > > [python-ideas](https://mail.python.org/mailman/listinfo/python-ideas) > > a discussion has broken out about somehow trying to make > > `p'/some/path/to/a/file` return an [instance of > > `pathlib.Path`]( > https://docs.python.org/3/library/pathlib.html#pathlib.Path). > > This led to a splinter discussion as to why `pathlib.Path` doesn't > > inherit from `str`? I figured instead of burying my response to this > > question in the thread I'd blog about it to try and explain one > > approach to API design. > > > > I think the key question in all of this is whether paths are > > semantically equivalent to a sequence of characters? Obviously the > > answer is "no" since paths have structure, directly represent > > something like files and directories, etc. Now paths do have a > > serialized representation as strings -- at least most of the time, but > > I'm ignoring Linux and the crazy situation of binary paths -- which is > > why C APIs take in `char *` as the representation of paths (and > > because C tends to make one try to stick with the types that are part > > of the C standard). So not all strings represent paths, but all paths > > can be represented by strings. I think we can all agree with that. > > > > OK, so if all paths can be represented as strings, why don't we just > > make `pathlib.Path` subclass `str`? Well, as I said earlier, not all > > strings are paths. You can't concatenate a string representing a path > > with some other random string and expect to get back a string that > > still represents a valid string (and I'm not talking about "valid" as > > in "the file doesn't exist", I'm talking about "syntactically not > > possible"). This is what [PEP > > 428](https://www.python.org/dev/peps/pep-0428/) is talking about when > > it says: > > > > > Not behaving like one of the basic builtin types [list str] also > > minimizes the potential for confusion if a path is combined by > > accident with genuine builtin types [like str]. > > Valid string? I assume you mean path, right? > Yes. > > > Now at this point someone someone will start saying, "but Brett, > > '[practicality beats > > purity](https://www.python.org/dev/peps/pep-0020/)' and all of those > > pre-existing, old OS APIs want a string as an argument!" And you're > > right, in terms of practicality it would be easier for `pathlib.Path` > > to inherit from `str` ... for the short/medium term. One thing people > > are forgetting is that "[explicit is better than > > implicit](https://www.python.org/dev/peps/pep-0020/)" as well and as > > with most design decisions there's not one clear answer. Implicitly > > treating a path as a string currently works, but that's just because > > we have inherited a suboptimal representation for paths from C. Now if > > you use an explicit representation of paths like `pathlib.Path` then > > you gain semantic separation and understanding of what you are working > > with. You also avoid any issues that come with implicit `str` > > compatibility as I pointed out earlier. > > I agree with the concatenation issue when using plain strings for paths. > > However, something that I cannot leave uncommented is "suboptimal > representation for paths". What would your optimal representation for > paths look like? I don't know, but I do like pathlib.Path more than a plain string. > I cannot believe that the current representation is so > bad and has been for so long and nobody, really nobody, has anything > done about it. > I never said it was bad, just suboptimal. If you want you can say using strings as paths is mediocre. I really don't want to argue about the label, just that pathlib.Path is a better representation of a path than a plain string and that strings on their own are nothing that special in terms of representing a string that demands we defend it on its own grounds as such. > > > And before anyone says, "so what?", think about this: Python 2/3. > > While I'm sure some will say I'm overblowing the comparison, but > > Python 3 came about because the implicit compatibility of binary and > > textual data in Python 2 caused major headaches to the point that we > > made a backwards-incompatible change that has caused widespread > > ramifications (but which we seem to be coming out on the other side > > of). By not inheriting from `str`, `pathlib.Path` has avoided a > > similar potential issue from the get-go. Or another way of looking at > > it is to ask why doesn't `dict` inherit from `str` so that it's easier > > to make it easier to stick in the body of an HTTP response so it can > > be implicitly treated as JSON? If you going, "eww, no because they are > > different types" then you at least understand the argument I'm trying > > to make (if you're going "I like that idea", then I think > > [Perl](https://www.perl.org/) might be more to your liking than Python > > and that's fine since the two languages just have different designs). > > I think most "practicality beats purity" folks don't want that either. > They are just bloody lazy. They actually want the benefits of both, the > pure datastructure with its convenience methods and the dirty str-like > thing with its convenience methods. > > People don't like it when somebody takes one bag of convenience away > from them if they easily can have both. ;-) > That's fine, but I also don't want to have the worry of latent bugs lurking in code either due to implicit type compatibility and so that clashes in this instance. > > Better sell it differently. So, let's read on. ;) > > > Now back to that "practicality beats purity" side of this. Obviously > > there are lots of APIs out there that take a string as an argument for > > a file path. And I understand people don't like using `str(path)` or > > the upcoming/new [`getattr(path, 'path', path)` > > idiom]( > https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.path) > > to work around the limitations forced upon them by other APIs that > > only take a `str` because they occasionally forget it. For this point, > > I have two answers. > > Sorry to interrupt your here but you missed one important piece (didn't > we agree on selling this better?): people do nasty little string > manipulations/analysis/regex with their paths for whatever reasons. > Right now, it's easier than ever because, well, because paths are > strings. :) > Sure, but that doesn't mean the situation can't be improved. You can also get the string out of Path objects to hack on, but that doesn't mean you should be shipping paths around in your code as strings either. I'm not saying you *can't* use strings to represent a path, just that it isn't the best option available from a higher-level perspective. > > Selling pathlib with the "getattr(path, 'path', path)" idiom is not > going to help those fellows. I mean, are you serious? It looks > ridiculous. :-/ > I disagree. > > Especially the point that a "path" has a "path" is not really getting > into my head. Either a "path" is a "path" or it better be named > "file/directory" which has a "path". > > path.path vs file.path/dir.path > > You need to work on your selling skills, Brett. ;-) > > > One is still "explicit is better than implicit" and you should have > > tests to begin with to catch when you forget to convert your > > `pathlib.Path` objects to `str` as needed. I'm just one of those > > programmers who's willing to type a bit more for easy-to-read code > > that's less error-prone within reason (and I obviously view this as > > reasonable). > > > > But more importantly, why don't you work with the code and/or projects > > that are forcing you to convert to `str` to start accepting `pathlib` > > objects as well for those same APIs? If projects start to work on > > making `pathlib` objects acceptable anywhere a path is accepted then > > once Python 3.4 is the oldest version of Python that is supported by > > the project then they can start considering dropping support for `str` > > as paths in new releases (obviously this also includes dropping > > support for Python 2 unless people use > > [pathlib2](https://pypi.python.org/pypi/pathlib2/) > > ). And I fully admit the > > stdlib is not exempt from a place that needs updating, so for > > `importlib` I have [opened an > > issue](http://bugs.python.org/issue26667) to update it to show this > > isn't just talk on my end (unfortunately there's some unique > > bootstrapping problems to import where stuff like `sys.path` probably > > can't be updated since that has to exist before you can import > > `pathlib` and that sort of thing ripples out, but I'm hoping I can > > update at least some things and this is a very unique case of > > potentially needing to stick with `str`). Hopefully if people start > > asking for `pathlib` support from projects they will add it, > > eventually leading to an alleviation of the desire to have > > `pathlib.Path` inherit from `str`. > > I think updating the stdlib looks sufficiently simple to start hacking > CPython. So, I would be willing to help here once the github repo is up > and running. :) > I suspect simple fixes like updating the docs and using `getattr(p, 'path', p)` in the right places will be the kind of quick fixes the GitHub migration will encourage. -Brett > > > Nice post, Brett. Seems you covered the most important point quite well > and also gave an explanation of why people request path->str. > > > Best, > Sven > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Mar 29 18:06:19 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 30 Mar 2016 11:06:19 +1300 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: Message-ID: <56FAFC5B.3000008@canterbury.ac.nz> Koos Zevenhoven wrote: > - Only some URIs (or even URLs) can be reliably distinguished from > file paths. However, those that contain '://' could be automatically > turned into URI objects by p-strings [or Path(...)]. No, they couldn't. "hello://world" is a perfectly valid unix pathname (albeit slightly redundant due to the double slash). I would not want Path() deciding that I really meant it to be a URI. -- Greg From ethan at stoneleaf.us Tue Mar 29 18:07:07 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 29 Mar 2016 15:07:07 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> Message-ID: <56FAFC8B.9060805@stoneleaf.us> On 03/29/2016 02:47 PM, Koos Zevenhoven wrote: > On Mar 30, 2016 00:39, "Ethan Furman" wrote: >> On 03/29/2016 02:30 PM, Greg Ewing wrote: >>> If P were a suitable seed object, this could be >>> >>> P/wherever/youwant/togo >> >> An interesting idea, but how would you differentiate between relative >> and absolute paths? > > Indeed interesting. It would currently work with P = Path(''), kinda > like my earlier p'' / ... example. It works even with relative paths, > although they would not look like relative ones because of the first /. Please provide an example of both a relative and an absolute path construction using `P`. -- ~Ethan~ From k7hoven at gmail.com Tue Mar 29 18:10:52 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 30 Mar 2016 01:10:52 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FAFC8B.9060805@stoneleaf.us> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> Message-ID: On Mar 30, 2016 01:07, "Ethan Furman" wrote: > > On 03/29/2016 02:47 PM, Koos Zevenhoven wrote: > >> On Mar 30, 2016 00:39, "Ethan Furman" wrote: >>> >>> On 03/29/2016 02:30 PM, Greg Ewing wrote: > > >>>> If P were a suitable seed object, this could be >>>> >>>> P/wherever/youwant/togo >>> >>> >>> An interesting idea, but how would you differentiate between relative >>> and absolute paths? >> >> >> Indeed interesting. It would currently work with P = Path(''), kinda >> like my earlier p'' / ... example. It works even with relative paths, >> although they would not look like relative ones because of the first /. > > > Please provide an example of both a relative and an absolute path construction using `P`. > P/'relative/path' P/'/absolute/path' -K / mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Mar 29 18:10:41 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 29 Mar 2016 15:10:41 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <56FA6FC0.30306@trueblade.com> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <56F9E6BB.6040400@stoneleaf.us> <56FA6FC0.30306@trueblade.com> Message-ID: On Tue, Mar 29, 2016 at 5:06 AM, Eric V. Smith wrote: > > Even if he is, I don't see it happening. Modifying the garbage > > collector to act sooner on one particular type of object when we already > > have good, if very mildly inconvenient, solutions is not going to be a > > high-priority item -- especially if it has /any/ negative impact on > > performance. > > I agree this is not likely to ever happen. It's especially problematic > that we'd be making this a requirement for implementations other than > CPython. > Again -- this is a thought experiment, not a proposal, but: It's only relevant for other implementations -- cPython already cleans up the file object (any object) when it is no longer referenced. And I do move the thought experiment along -- it was never specific to file objects, but rather: In the case of temporary objects that are never referred to outside a single line of code -- delete it right away. in theory, this could help all implementations leave a little less garbage lying around. Whether that is possible or desirable, I have no idea. -CHB > > Eric. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Mar 29 18:16:10 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 29 Mar 2016 15:16:10 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> Message-ID: <56FAFEAA.2010503@stoneleaf.us> On 03/29/2016 03:10 PM, Koos Zevenhoven wrote: > On Mar 30, 2016 01:07, "Ethan Furman" wrote: >> Please provide an example of both a relative and an absolute path >> construction using `P`. > > P/'relative/path' > > P/'/absolute/path' Thank you. So under this system a relative path appears with one leading slash, and an absolute path appears with two leading slashes. Can't say I'm a fan. :( -- ~Ethan~ From mike at selik.org Tue Mar 29 18:17:05 2016 From: mike at selik.org (Michael Selik) Date: Tue, 29 Mar 2016 22:17:05 +0000 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: <56FAFC5B.3000008@canterbury.ac.nz> References: <56FAFC5B.3000008@canterbury.ac.nz> Message-ID: On Tue, Mar 29, 2016 at 6:06 PM Greg Ewing wrote: > Koos Zevenhoven wrote: > > - Only some URIs (or even URLs) can be reliably distinguished from > > file paths. However, those that contain '://' could be automatically > > turned into URI objects by p-strings [or Path(...)]. > > No, they couldn't. "hello://world" is a perfectly valid unix > pathname (albeit slightly redundant due to the double slash). > I would not want Path() deciding that I really meant it to > be a URI. > > Good point. Even worse: $ mkdir ftp://www.example.com $ tree . ??? ftp:/ ??? www.example.com/ 2 directories, 0 files -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Mar 29 18:17:09 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 29 Mar 2016 15:17:09 -0700 Subject: [Python-ideas] Integrate some itertools into the Python syntax In-Reply-To: References: <22262.49189.676872.388182@turnbull.sk.tsukuba.ac.jp> <-3740384837840340487@unknownmsgid> Message-ID: On Mon, Mar 28, 2016 at 2:46 PM, Terry Reedy wrote: > With the additional changes to builtins in 3.0, there has definitely been > a shift of emphasis from lists and sequences to iterables and iterators. > Exactly my point -- and I think there ARE place in the language where we could make that more obviou san natural -- maybe not this place though. > Because of the difference between non-destructive and destructive slicing, This is a great way to describe the difference. Note that there is also a distinction between destructive and nondestructive looping: for item in an_iterable: do_something(item) if some_condition: break may leave an_iterable alone, and may leave it in a different state depending on if an_iterable is a sequence or an actual iterator. Even so -- I agree that it's probably not a good idea to re-use the slicing syntax for "destructive slicing." -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Tue Mar 29 18:32:33 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 30 Mar 2016 01:32:33 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FAFEAA.2010503@stoneleaf.us> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> Message-ID: >> On Mar 30, 2016 01:07, "Ethan Furman" wrote: > > >>> Please provide an example of both a relative and an absolute path >>> construction using `P`. >> >> >> P/'relative/path' >> >> P/'/absolute/path' > > > Thank you. > > So under this system a relative path appears with one leading slash, and an absolute path appears with two leading slashes. > > Can't say I'm a fan. :( > I'll have to agree. But to be fair, Greg wrote something like 'a suitable seed object' so we could for ex. have P=PathSeed(), and use another operator. P@'rel/path' P@'/abs/path' Dunno. -K at mobile -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Mar 29 18:44:23 2016 From: brett at python.org (Brett Cannon) Date: Tue, 29 Mar 2016 22:44:23 +0000 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <56F9E6BB.6040400@stoneleaf.us> <56FA6FC0.30306@trueblade.com> Message-ID: On Tue, 29 Mar 2016 at 15:11 Chris Barker wrote: > On Tue, Mar 29, 2016 at 5:06 AM, Eric V. Smith wrote: > >> > Even if he is, I don't see it happening. Modifying the garbage >> > collector to act sooner on one particular type of object when we already >> > have good, if very mildly inconvenient, solutions is not going to be a >> > high-priority item -- especially if it has /any/ negative impact on >> > performance. >> >> I agree this is not likely to ever happen. It's especially problematic >> that we'd be making this a requirement for implementations other than >> CPython. >> > > Again -- this is a thought experiment, not a proposal, but: > > It's only relevant for other implementations -- cPython already cleans up > the file object (any object) when it is no longer referenced. > > And I do move the thought experiment along -- it was never specific to > file objects, but rather: > > In the case of temporary objects that are never referred to outside a > single line of code -- delete it right away. > > in theory, this could help all implementations leave a little less garbage > lying around. > > Whether that is possible or desirable, I have no idea. > It's not desirable to dictate what various Python implementations must make sure that their garbage collector supports direct collection like this for this specific case rather than staying general. It would very much be a shift in the definition of the Python project where we have purposefully avoided dictating how objects need to behave in terms of garbage collection. -Brett > > -CHB > > > > > > > >> >> Eric. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Mar 29 18:49:47 2016 From: brett at python.org (Brett Cannon) Date: Tue, 29 Mar 2016 22:49:47 +0000 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> Message-ID: On Tue, 29 Mar 2016 at 15:41 Koos Zevenhoven wrote: > > >> On Mar 30, 2016 01:07, "Ethan Furman" wrote: > > > > > >>> Please provide an example of both a relative and an absolute path > >>> construction using `P`. > >> > >> > >> P/'relative/path' > >> > >> P/'/absolute/path' > > > > > > Thank you. > > > > So under this system a relative path appears with one leading slash, and > an absolute path appears with two leading slashes. > > > > Can't say I'm a fan. :( > > > > I'll have to agree. But to be fair, Greg wrote something like 'a suitable > seed object' so we could for ex. have P=PathSeed(), and use another > operator. > > P@'rel/path' > > P@'/abs/path' > > Dunno. > If you are constructing a new object then you don't need to limit yourself to a single one. You could use A for absolute paths and R for relative paths: R/'relative'/'path' A/'absolute'/'path' Or even P.absolute and P.relative: P.relative/'relative'/'path' P.absolute/'absolute'/'path' And you could even go as far as make one the default and the other not to promote one over the other: P.relative/'relative'/'path' P/'absolute'/'path' -Brett > -K at mobile > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Mar 29 22:22:05 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 30 Mar 2016 13:22:05 +1100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <56F9E6BB.6040400@stoneleaf.us> <56FA6FC0.30306@trueblade.com> Message-ID: On Wed, Mar 30, 2016 at 9:10 AM, Chris Barker wrote: > It's only relevant for other implementations -- cPython already cleans up > the file object (any object) when it is no longer referenced. > > And I do move the thought experiment along -- it was never specific to file > objects, but rather: > > In the case of temporary objects that are never referred to outside a single > line of code -- delete it right away. The problem is that there's no way to be sure. For instance, compare these lines of code: from threading import Thread open("marker").write("Ha") Thread(target=threadfunc).start() One of them has finished with the object completely. The other most certainly has not. Python prefers, in the case of ambiguity, to require explicit directives. ChrisA From rosuav at gmail.com Tue Mar 29 22:51:08 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 30 Mar 2016 13:51:08 +1100 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: <56FAFC5B.3000008@canterbury.ac.nz> Message-ID: On Wed, Mar 30, 2016 at 9:17 AM, Michael Selik wrote: > On Tue, Mar 29, 2016 at 6:06 PM Greg Ewing > wrote: >> >> Koos Zevenhoven wrote: >> > - Only some URIs (or even URLs) can be reliably distinguished from >> > file paths. However, those that contain '://' could be automatically >> > turned into URI objects by p-strings [or Path(...)]. >> >> No, they couldn't. "hello://world" is a perfectly valid unix >> pathname (albeit slightly redundant due to the double slash). >> I would not want Path() deciding that I really meant it to >> be a URI. >> > > Good point. Even worse: > > $ mkdir ftp://www.example.com > $ tree > . > ??? ftp:/ > ??? www.example.com/ > > 2 directories, 0 files How is that "even worse"? It's the exact same thing. (You might need "mkdir -p" to make this work, as it'll need to create more than one directory.) You have a directory called "ftp:", and inside that, a directory called "www.example.com". rosuav at sikorsky:~/tmp$ mkdir -p ftp://www.example.com rosuav at sikorsky:~/tmp$ find . . ./ftp: ./ftp:/www.example.com rosuav at sikorsky:~/tmp$ But this is a case where the weird edge cases can be dealt with specially, IMO. There are a *lot* of programs which cannot easily handle a file name that begins with a hyphen - the solution is to force a different interpretation, either with a double-hyphen "end of options", or by using "./-rf" as the file name. The same could be done here; in the extremely rare situation where you actually want to start your path with "http:" and then another directory, you have three options: 1) Path("./http://www.example.com") 2) Path("http:/www.example.com") 3) Path("file://http://www.example.com") For scripts that need 100% dependable parsing, the third option will be guaranteed to work. For normal usage, compressing the double slash to a single one will have the right effect AND canonicalize the name. This should be safe. ChrisA From random832 at fastmail.com Tue Mar 29 23:23:54 2016 From: random832 at fastmail.com (Random832) Date: Tue, 29 Mar 2016 23:23:54 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <1459277942.1710166.562817970.6C518420@webmail.messagingengine.com> Message-ID: <1459308234.15606.563184226.63614D5B@webmail.messagingengine.com> On Tue, Mar 29, 2016, at 17:37, Brett Cannon wrote: > > You can't add an int to a float and expect to get an integer back, > > either. > > > Sure, but that's because int and float are both part of the same type > hierarchy (https://www.python.org/dev/peps/pep-3141/). ...so would str and path, in the scenario being discussed. The argument presented here against allowing paths to be strings is also an argument against allowing ints to be Numbers. From random832 at fastmail.com Tue Mar 29 23:25:33 2016 From: random832 at fastmail.com (Random832) Date: Tue, 29 Mar 2016 23:25:33 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FAF62A.9020104@stoneleaf.us> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> Message-ID: <1459308333.15976.563186042.556CA823@webmail.messagingengine.com> On Tue, Mar 29, 2016, at 17:39, Ethan Furman wrote: > On 03/29/2016 02:30 PM, Greg Ewing wrote: > > Paul Moore wrote: > >> > >> P(wherever)/youwant/togo > > > > If P were a suitable seed object, this could be > > > > P/wherever/youwant/togo > > An interesting idea, but how would you differentiate between relative > and absolute paths? Path.ROOT/absolute/path Path.DOT/relative/path From stephen at xemacs.org Wed Mar 30 00:06:07 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 30 Mar 2016 13:06:07 +0900 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: <56FAFC5B.3000008@canterbury.ac.nz> Message-ID: <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> Chris Angelico writes: > 1) Path("./http://www.example.com") > 2) Path("http:/www.example.com") > 3) Path("file://http://www.example.com") > > For scripts that need 100% dependable parsing, the third option will > be guaranteed to work. No, the third should crap out with a syntax error on the colon, see [1], which does not allow a port spec at all, and RFC 3986, which doesn't allow colon in the host name ([1] references RFC 3986 for the syntax of the host name). Specifying the host to a "file:" URI gives locally-defined behavior (eg, a Windows share), but in the most recent attempt to deal with exactly these issues[1], it is legal. The correct syntaxes per [1] and RFC 3986 are 4) Path("file:///http://www.example.com") 5) Path("file://localhost/http://www.example.com") 6) Path("file://[127.0.0.1]/http://www.example.com") 7) Path("file://[::1]/http://www.example.com") As far as I can tell the colon in "http:" is RFC 3986-legal, since it has no URI syntactic meaning in the path component. This isn't as easy as it looks (which is why people are trying to delegate it to something they think of as "simple"). There's an additional problem with trying to cram URIs and Path together, which is that in a file system, "/a/b/symlink/../c" may refer to any file system object depending on symlink's target which is unknown, while as an URI path it refers to whatever "/a/b/c" refers to, and nothing else. (This is the semantic glitch I was thinking of earlier.) This means that URIs can be canonicalized syntactically, while doing so with file system paths is risky. Footnotes: [1] https://tools.ietf.org/html/draft-ietf-appsawg-file-scheme-06 From rosuav at gmail.com Wed Mar 30 00:23:32 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 30 Mar 2016 15:23:32 +1100 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> References: <56FAFC5B.3000008@canterbury.ac.nz> <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Mar 30, 2016 at 3:06 PM, Stephen J. Turnbull wrote: > Chris Angelico writes: > > > 3) Path("file://http://www.example.com") > > > > For scripts that need 100% dependable parsing, the third option will > > be guaranteed to work. > > No, the third should crap out with a syntax error on the colon... > > The correct syntaxes per [1] and RFC 3986 are > > 4) Path("file:///http://www.example.com") Oops, my bad - I forgot about the third slash. It comes to the same thing, though; for most paths, you can deduce that a prefix "http://" implies that it's not a file path, and for the rare case when you do mean that, you can explicitly adorn it. (This is a good reminder that code and specs should not be created by one person firing off an email. This needs someone - preferably multiple people - checking the appropriate specs. Get it right, folks, don't trust me!) > There's an additional problem with trying to cram URIs and Path > together, which is that in a file system, "/a/b/symlink/../c" may > refer to any file system object depending on symlink's target which is > unknown, while as an URI path it refers to whatever "/a/b/c" refers > to, and nothing else. (This is the semantic glitch I was thinking of > earlier.) > > This means that URIs can be canonicalized syntactically, while doing > so with file system paths is risky. Or there are two operations: canonicalizing by components, and rendering a "true path", which requires file system access (stat every component). ChrisA From chris.barker at noaa.gov Wed Mar 30 00:33:31 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 29 Mar 2016 21:33:31 -0700 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <56F9E6BB.6040400@stoneleaf.us> <56FA6FC0.30306@trueblade.com> Message-ID: <-5085290531960812972@unknownmsgid> >> In the case of temporary objects that are never referred to outside a single >> line of code -- delete it right away. > > The problem is that there's no way to be sure. For instance, compare > these lines of code: > > from threading import Thread > > open("marker").write("Ha") > Thread(target=threadfunc).start() > > One of them has finished with the object completely. The other most > certainly has not. Interesting -- now I'm curious -- how does cPython know that the reference count of that Thread object needs to be increased So it won't be deleted? Does the start() method call incref() on itself? -CHB > > Python prefers, in the case of ambiguity, to require explicit directives. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Wed Mar 30 00:40:44 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 30 Mar 2016 15:40:44 +1100 Subject: [Python-ideas] `to_file()` method for strings In-Reply-To: <-5085290531960812972@unknownmsgid> References: <7DDCB676-2AA3-42AB-807C-D7C1F0BA293C@yahoo.com> <56F9E6BB.6040400@stoneleaf.us> <56FA6FC0.30306@trueblade.com> <-5085290531960812972@unknownmsgid> Message-ID: On Wed, Mar 30, 2016 at 3:33 PM, Chris Barker - NOAA Federal wrote: >>> In the case of temporary objects that are never referred to outside a single >>> line of code -- delete it right away. >> >> The problem is that there's no way to be sure. For instance, compare >> these lines of code: >> >> from threading import Thread >> >> open("marker").write("Ha") >> Thread(target=threadfunc).start() >> >> One of them has finished with the object completely. The other most >> certainly has not. > > Interesting -- now I'm curious -- how does cPython know that the > reference count of that Thread object needs to be increased So it > won't be deleted? Does the start() method call incref() on itself? There'll be another reference somewhere. Presumably a new stack is allocated or something, and that's what keeps a reference. It doesn't matter _what_ keeps the ref; all that matters is that there is one, and that the code line itself doesn't show that. Hence you need to explicitly say "open this, and close it when you're done". ChrisA From stephen at xemacs.org Wed Mar 30 03:42:34 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 30 Mar 2016 16:42:34 +0900 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: <56FAFC5B.3000008@canterbury.ac.nz> <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> Message-ID: <22267.33642.881307.298560@turnbull.sk.tsukuba.ac.jp> Chris Angelico writes: > On Wed, Mar 30, 2016 at 3:06 PM, Stephen J. Turnbull wrote: > > This means that URIs can be canonicalized syntactically, while doing > > so with file system paths is risky. > > Or there are two operations: canonicalizing by components, and > rendering a "true path", which requires file system access (stat every > component). That's not to my taste because while you do need to make that choice for filesystem paths, it's always safe (in the sense of "you're buggy, not me!") to canonicalize a URI. Also, RFC 3896 explicitly refuses to require URIs to make any physical sense, while it's an error for a filesystem path to refer to a non-existent object. In other words, URIs are abstract syntax to which you can assign whatever semantics you want (as long as they are compatible with the syntax), while filesystem paths are actual paths in a graph that is instantiated in hardware storage. I suspect you won't have a problem with that distinction, but ISTM that it's the exact opposite of the way the "let's derive Path from str" (or from "BaseString" or whatever) crowd want to think (filesystem paths are a subset of strings). From victor.stinner at gmail.com Wed Mar 30 03:49:53 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 30 Mar 2016 09:49:53 +0200 Subject: [Python-ideas] Add a developer mode to Python: -X dev command line option Message-ID: Hi, When I develop on CPython, I'm always building Python in debug mode using ./configure --with-pydebug. This mode enables a *lot* of extra checks which helps me to detect bugs earlier. The debug mode makes Python much slower and so is not the default. "python3" in Linux distributions is a binary compiled in release mode. When they also provide a binary compiled in debug mode, you will probably have issues to use your existing C extensions, since they all of them are compiled in release mode which is not compatible (you must recompile C extensions in debug mode). I propose to add a "development mode" to Python, to get a few checks to detect bugs earlier: a new -X dev command line option. Example: python3.6 -X dev script.py Checks enabled by this flag must: * Not flood the stdout/stderr (ex: don't write one message per second) * Have a low overhead in term of CPU and memory (ex: max 10%) I propose to enable: * Show DeprecationWarning and ResourceWarning warnings: python -Wd * Show BytesWarning warning: python -b * Enable Python assertions (assert) and set __debug__ to True: remove (or just ignore) -O or -OO command line arguments * faulthandler to get a Python traceback on segfault and fatal errors: python -X faulthandler * Debug hooks on Python memory allocators: PYTHONMALLOC=debug For example, I consider that enabling tracemalloc is too expensive (CPU & memory) and must not be enabled in -X dev. I wrote a proof-of-concept: if -X dev is used, executable Python once again with more parameters. Basically, replace: python3.6 -X dev ... with PYTHONMALLOC=debug python3.6 -Wd -b -X faulthandler ... The list of checks can be extended later. For example, we may enable the debug mode of asyncio: PYTHONASYNCIODEBUG=1. The scope of the "developer mode" is unclear to me. Many modules (ex: asyncio) already have a debug mode. Would it be a bad idea to enable *all* debug modes of *all* modules? For example, http.client has a debug level: do you expect debug traces of the HTTP client when you develop your application? IMHO the scope must be well defined: don't modify modules of the stdlib, only enable more debug flags in the Python core (warnings, unicode, memory allocators, etc.). Maybe we even a need -X dev=strict which would be stricter: * use -Werror instead of -Wd: raise an exception when a warning is emitted * use -bb instead of -b: get BytesWarning exceptions * Replace "inconsistent use of tabs and spaces in indentation" warning with an error in the Python parser * etc. Again, this list can be extended later. Or maybe we need multiple level to control the quantity of debug traces, warnings, ... written into stdout/stderr? In my experience, the problem with converting warnings to errors is that you don't control the code of your whole application. It's common that third party modules raise DeprecationWarning, ResourceWarning, etc. Even some modules of the Python stdlib raise DeprecatingWarning! For example, I recall that distutils raises a DeprecationWarning on Python 3.4 when importing the imp module. "Similar" idea in other programming languages: * Perl: http://perldoc.perl.org/strict.html * PHP: https://secure.php.net/manual/fr/function.error-reporting.php Victor From p.f.moore at gmail.com Wed Mar 30 03:55:33 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 30 Mar 2016 08:55:33 +0100 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: <56FAFC5B.3000008@canterbury.ac.nz> <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> Message-ID: On 30 March 2016 at 05:23, Chris Angelico wrote: >> 4) Path("file:///http://www.example.com") > > Oops, my bad - I forgot about the third slash. It comes to the same > thing, though; for most paths, you can deduce that a prefix "http://" > implies that it's not a file path, and for the rare case when you do > mean that, you can explicitly adorn it. I presume you're deliberately ignoring the fact that most paths come from variables, not from literals, and many come from user input (sometimes even unintended user input such as a "*" expanded by the shell)? It's easy enough to rewrite literals to be unambiguous, but in order to do so for arbitrary input you need to basically implement (part of) a URI parser... Paul From victor.stinner at gmail.com Wed Mar 30 04:21:20 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 30 Mar 2016 10:21:20 +0200 Subject: [Python-ideas] Add a developer mode to Python: -X dev command line option In-Reply-To: References: Message-ID: 2016-03-30 9:49 GMT+02:00 Victor Stinner : > I propose to add a "development mode" to Python, to get a few checks > to detect bugs earlier: a new -X dev command line option. Example: > > python3.6 -X dev script.py > (...) I posted my patch on the bug tracker if you would like to try it: https://bugs.python.org/issue26670 Victor From srkunze at mail.de Wed Mar 30 04:45:30 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 30 Mar 2016 10:45:30 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> Message-ID: <56FB922A.6090607@mail.de> On 29.03.2016 23:57, Paul Moore wrote: > On 29 March 2016 at 22:00, Sven R. Kunze wrote: >> However, something that I cannot leave uncommented is "suboptimal >> representation for paths". What would your optimal representation for paths >> look like? I cannot believe that the current representation is so bad and >> has been for so long and nobody, really nobody, has anything done about it. > Well, pathlib.Path :-) > > The point here is that C's char* representation is a serialisation of > a path object, just like 123 is a serialisation of an integer object. > We don't object to having to convert user input to a string if we need > to, why object to having to convert it to a Path if appropriate? I think there is a misunderstanding here. Let me quote myself: '''I think most "practicality beats purity" folks don't want that either. They are just bloody lazy. They actually want the benefits of both, the pure datastructure with its convenience methods and the dirty str-like thing with its convenience methods.''' The desire is not "str vs. Path"; its "Path + str". (at least from what I can tell) Best, Sven From wes.turner at gmail.com Wed Mar 30 04:56:01 2016 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 30 Mar 2016 03:56:01 -0500 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FB922A.6090607@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> Message-ID: On Mar 30, 2016 3:47 AM, "Sven R. Kunze" wrote: > > On 29.03.2016 23:57, Paul Moore wrote: >> >> On 29 March 2016 at 22:00, Sven R. Kunze wrote: >>> >>> However, something that I cannot leave uncommented is "suboptimal >>> representation for paths". What would your optimal representation for paths >>> look like? I cannot believe that the current representation is so bad and >>> has been for so long and nobody, really nobody, has anything done about it. >> >> Well, pathlib.Path :-) >> >> The point here is that C's char* representation is a serialisation of >> a path object, just like 123 is a serialisation of an integer object. >> We don't object to having to convert user input to a string if we need >> to, why object to having to convert it to a Path if appropriate? > > > I think there is a misunderstanding here. Let me quote myself: > > '''I think most "practicality beats purity" folks don't want that either. They are just bloody lazy. They actually want the benefits of both, the pure datastructure with its convenience methods and the dirty str-like thing with its convenience methods.''' > > The desire is not "str vs. Path"; its "Path + str". (at least from what I can tell) so, like str but without +, %, .format ? class pathstr(str): __add__ __mod__ format python3: pathstr(str)? python2: pathstr(unicode)? > > > Best, > Sven > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Mar 30 05:15:05 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 30 Mar 2016 11:15:05 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> Message-ID: <56FB9919.8020308@mail.de> On 30.03.2016 10:56, Wes Turner wrote: > so, like str but without +, %, .format ? > > class pathstr(str): > __add__ > __mod__ > format > > python3: pathstr(str)? > python2: pathstr(unicode)? > I am not sure I understand. Where is the Path in your code? To make it more explicit: like "all methods of Path" + "all methods of str". And no, format, % and + are nice to have in case you need them; so why removing them? Best, Sven From p.f.moore at gmail.com Wed Mar 30 05:20:22 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 30 Mar 2016 10:20:22 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FB922A.6090607@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> Message-ID: On 30 March 2016 at 09:45, Sven R. Kunze wrote: > On 29.03.2016 23:57, Paul Moore wrote: >> >> On 29 March 2016 at 22:00, Sven R. Kunze wrote: >>> >>> However, something that I cannot leave uncommented is "suboptimal >>> representation for paths". What would your optimal representation for >>> paths >>> look like? I cannot believe that the current representation is so bad and >>> has been for so long and nobody, really nobody, has anything done about >>> it. >> >> Well, pathlib.Path :-) >> >> The point here is that C's char* representation is a serialisation of >> a path object, just like 123 is a serialisation of an integer object. >> We don't object to having to convert user input to a string if we need >> to, why object to having to convert it to a Path if appropriate? > > > I think there is a misunderstanding here. Let me quote myself: > > '''I think most "practicality beats purity" folks don't want that either. > They are just bloody lazy. They actually want the benefits of both, the pure > datastructure with its convenience methods and the dirty str-like thing with > its convenience methods.''' > > The desire is not "str vs. Path"; its "Path + str". (at least from what I > can tell) I understand all that. I thought you were referring to Brett's comment that C's "char *" format was "suboptimal" and asking what would be an optimal representation (you said "your optimal representation", meaning Brett's, so only he can answer to that but I took the question more generally as what would be *an* optimal representation). My reply was intended to say that I view something like pathlib.Path as optimal. I don't personally see "working like a string" as being optimal, precisely because it mixes the "structured object" level with the "serialised representation" level. That's something Brett's article clarified for me, so I suspect he'd have the same view. I doubt the people arguing for Path being a subclass of string would be swayed by an argument like the above, though, as they would view it as "purity over practicality". Personally, I find that careful separation of concerns is a practicality benefit, in terms of maintainability and clarity of code - a lesson I learned from having to do far too many bytes/unicode migrations where separating out the concepts was a vital step. Anyway, this is pretty off topic, so I'll leave it there. Paul From srkunze at mail.de Wed Mar 30 05:58:24 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 30 Mar 2016 11:58:24 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> Message-ID: <56FBA340.50301@mail.de> On 30.03.2016 00:03, Brett Cannon wrote: > > I cannot believe that the current representation is so > bad and has been for so long and nobody, really nobody, has anything > done about it. > > > [...] pathlib.Path is a better representation of a path than a plain > string and that strings on their own are nothing that special in terms > of representing a string that demands we defend it on its own grounds > as such. I agree with the first part. I disagree with the second (assuming you mean "representing a path"). Strings are extremely useful for informal working like exploration, debugging etc. > I think most "practicality beats purity" folks don't want that either. > They are just bloody lazy. They actually want the benefits of > both, the > pure datastructure with its convenience methods and the dirty str-like > thing with its convenience methods. > > People don't like it when somebody takes one bag of convenience away > from them if they easily can have both. ;-) > > > That's fine, but I also don't want to have the worry of latent bugs > lurking in code either due to implicit type compatibility and so that > clashes in this instance. Could you give an example here? > Sure, but that doesn't mean the situation can't be improved. You can > also get the string out of Path objects to hack on, but that doesn't > mean you should be shipping paths around in your code as strings > either. I'm not saying you *can't* use strings to represent a path, > just that it isn't the best option available from a higher-level > perspective. Let me think of a recommendation here. I personally find the proposed idiom quite cumbersome and ugly but that's just me. However, it is basically not better than "latent bugs lurking in code" due to inheritance. Wrapping stuff up just because we don't know if it's a string or a path is always a bad idea. In pathlibs current form, this would give use a recommendation for application developers: 1) pass a path if you need path methods 2) pass a path.path if you want to do string manipulations 3) don't try both at the same time Would this makes sense? > Selling pathlib with the "getattr(path, 'path', path)" idiom is not > going to help those fellows. I mean, are you serious? It looks > ridiculous. :-/ > > > I disagree. That's fine but that's something I would call toiled paper programming. Wrapping things up until it doesn't smell anymore. Seems like here I am the purist. ;-) So, having only one place (the stdlib) dealing with this inconvenience would make things less worse I suppose. > I suspect simple fixes like updating the docs and using `getattr(p, > 'path', p)` in the right places will be the kind of quick fixes the > GitHub migration will encourage. I look forward to it. :) Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Mar 30 06:05:55 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 30 Mar 2016 12:05:55 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <1459308333.15976.563186042.556CA823@webmail.messagingengine.com> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <1459308333.15976.563186042.556CA823@webmail.messagingengine.com> Message-ID: <56FBA503.2070307@mail.de> On 30.03.2016 05:25, Random832 wrote: > On Tue, Mar 29, 2016, at 17:39, Ethan Furman wrote: >> >> An interesting idea, but how would you differentiate between relative >> and absolute paths? > Path.ROOT/absolute/path > Path.DOT/relative/path Very interesting. I like how it looks with variables. But I have to admit I don't like the look so much with regards to string literals. This is where the format string really shines. Best, Sven From rosuav at gmail.com Wed Mar 30 06:07:00 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 30 Mar 2016 21:07:00 +1100 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: <56FAFC5B.3000008@canterbury.ac.nz> <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Mar 30, 2016 at 6:55 PM, Paul Moore wrote: > On 30 March 2016 at 05:23, Chris Angelico wrote: >>> 4) Path("file:///http://www.example.com") >> >> Oops, my bad - I forgot about the third slash. It comes to the same >> thing, though; for most paths, you can deduce that a prefix "http://" >> implies that it's not a file path, and for the rare case when you do >> mean that, you can explicitly adorn it. > > I presume you're deliberately ignoring the fact that most paths come > from variables, not from literals, and many come from user input > (sometimes even unintended user input such as a "*" expanded by the > shell)? It's easy enough to rewrite literals to be unambiguous, but in > order to do so for arbitrary input you need to basically implement > (part of) a URI parser... Not quite; what I'm saying is that *any* file path can be made unambiguous by prepending "file:///", thus guaranteeing that it will be parsed correctly. I'm not sure that this is necessarily a good thing, but it's in response to the objection that a magic prefix "http://" would introduce an impossibility. ChrisA From rosuav at gmail.com Wed Mar 30 06:13:51 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 30 Mar 2016 21:13:51 +1100 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: <22267.33642.881307.298560@turnbull.sk.tsukuba.ac.jp> References: <56FAFC5B.3000008@canterbury.ac.nz> <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> <22267.33642.881307.298560@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Mar 30, 2016 at 6:42 PM, Stephen J. Turnbull wrote: > Chris Angelico writes: > > On Wed, Mar 30, 2016 at 3:06 PM, Stephen J. Turnbull wrote: > > > > This means that URIs can be canonicalized syntactically, while doing > > > so with file system paths is risky. > > > > Or there are two operations: canonicalizing by components, and > > rendering a "true path", which requires file system access (stat every > > component). > > That's not to my taste because while you do need to make that choice > for filesystem paths, it's always safe (in the sense of "you're buggy, > not me!") to canonicalize a URI. Yes, this is true; for a non-file-system URI, the "true path" would simply be identical to the component-based canonicalization. > Also, RFC 3896 explicitly refuses to require URIs to make any physical > sense, while it's an error for a filesystem path to refer to a > non-existent object. In other words, URIs are abstract syntax to > which you can assign whatever semantics you want (as long as they are > compatible with the syntax), while filesystem paths are actual paths > in a graph that is instantiated in hardware storage. Hmm. Not sure what you mean by that error - how else could you create a new file, than by identifying a path that does not exist? You're absolutely right that URIs have no specific physical meaning, yet they do have definite semantic meaning; an HTTP client can (and should) resolve relative URLs itself, rather than simply sticking "../../static/main.css" on the end of the current URL. > I suspect you won't have a problem with that distinction, but ISTM > that it's the exact opposite of the way the "let's derive Path from > str" (or from "BaseString" or whatever) crowd want to think > (filesystem paths are a subset of strings). > Even if a file system path is a special form of string, this will work. It's just that the semantics would have to be defined sans canonicalization, with two methods that do that. But I am liking the line of thinking (not posted in this exact thread, so I'm not sure who said it - sorry!) that Paths are as abstract as integers and text strings, and that they should have no particular tie to their mechanical implementation. The string "/home/rosuav/foo.py" should be more like a "path display" (by analogy with lists etc) than an actual Path. ChrisA From mal at egenix.com Wed Mar 30 06:26:36 2016 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 30 Mar 2016 12:26:36 +0200 Subject: [Python-ideas] Add a developer mode to Python: -X dev command line option In-Reply-To: References: Message-ID: <56FBA9DC.4070603@egenix.com> On 30.03.2016 09:49, Victor Stinner wrote: > I propose to add a "development mode" to Python, to get a few checks > to detect bugs earlier: a new -X dev command line option. Example: > > python3.6 -X dev script.py > > ... > > The scope of the "developer mode" is unclear to me. Many modules (ex: > asyncio) already have a debug mode. Would it be a bad idea to enable > *all* debug modes of *all* modules? For example, http.client has a > debug level: do you expect debug traces of the HTTP client when you > develop your application? > > IMHO the scope must be well defined: don't modify modules of the > stdlib, only enable more debug flags in the Python core (warnings, > unicode, memory allocators, etc.). I'm not sure whether this would make things easier for the majority of developers, e.g. someone not writing C extensions would likely not be interested in debugging memory allocations or segfaults, someone spending more time on numerics wouldn't bother with bytes warnings, etc. I usually use wrappers or custom Python builds for debugging purposes, which are then specialized for the specific task I need them for. Those are quick to write and more explicit in what they enable or not. I wouldn't want to rely on a specific set of flags which may change from Python release to Python release. So overall, I guess adding a debug howto of some sort showing and explaining the purpose of the various options would be more helpful to developers than a meta-option which triggers some subset of other available options. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Wed Mar 30 06:45:24 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 30 Mar 2016 11:45:24 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FBA340.50301@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FBA340.50301@mail.de> Message-ID: On 30 March 2016 at 10:58, Sven R. Kunze wrote: >> That's fine, but I also don't want to have the worry of latent bugs lurking >> in code either due to implicit type compatibility and so that clashes in >> this instance. > > Could you give an example here? def fix_path(p): # Old code, possibly in a module. # Quite possibly it does a lot more than # simply calling abspath, so it might not # be easy to review return os.path.abspath(p) # Elsewhere p = Path(sys.argv[1]) if not p.is_absolute(): p = fix_path(p) # Now a relative p is no longer a Path object. Later in your code, possibly somewhere # miles away if you rely heavily on paths being interchangeable with strings, you get # an AttributeError when you try to use a Path method on p. Obviously the issue here is trivial to spot. But in real code, it could be a mess of complex logic, with a single os.path call somewhere in an obscure conditional branch. Possibly fix_path has even been updated to work with pathlib, but that one os.path call was accidentally missed and there's no test covering it. Before you say so, of course this is a made up scenario, and of course it'll likely be rare. The question isn't whether such things are common, so much as whether the cost of requiring explicit conversion to string is worth the benefit of avoiding situations like this. And that's a hard judgement call. For casual scripters, and people doing research, the answer is "clearly not". For teaching new users, probably not (new users won't make this sort of mistake, but equally having to teach what a string subclass is relatively early is maybe an issue). For people writing mission critical software, very definitely yes (as a pip developer, I'd hate to switch pip over to a str-subclass path object, whereas switching to a non-subclass one is much more conceivable[1]). Paul [1] Backward compatibility, not wanting another dependency (on the pathlib backport) and the fact that it's change for no actual benefit mean we'd probably not bother anyway, but that's a different decision. From p.f.moore at gmail.com Wed Mar 30 06:49:35 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 30 Mar 2016 11:49:35 +0100 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: <56FAFC5B.3000008@canterbury.ac.nz> <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> Message-ID: On 30 March 2016 at 11:07, Chris Angelico wrote: > On Wed, Mar 30, 2016 at 6:55 PM, Paul Moore wrote: >> On 30 March 2016 at 05:23, Chris Angelico wrote: >>>> 4) Path("file:///http://www.example.com") >>> >>> Oops, my bad - I forgot about the third slash. It comes to the same >>> thing, though; for most paths, you can deduce that a prefix "http://" >>> implies that it's not a file path, and for the rare case when you do >>> mean that, you can explicitly adorn it. >> >> I presume you're deliberately ignoring the fact that most paths come >> from variables, not from literals, and many come from user input >> (sometimes even unintended user input such as a "*" expanded by the >> shell)? It's easy enough to rewrite literals to be unambiguous, but in >> order to do so for arbitrary input you need to basically implement >> (part of) a URI parser... > > Not quite; what I'm saying is that *any* file path can be made > unambiguous by prepending "file:///", thus guaranteeing that it will > be parsed correctly. I'm not sure that this is necessarily a good > thing, but it's in response to the objection that a magic prefix > "http://" would introduce an impossibility. I don't know if that's true for Windows paths. You'd need to switch backslashes to slashes, for a start (and *not* do that on Unix, where backslash is a valid filename character, albeit a silly one to use). And the URL syntax for drive letters is at best inconsistent across applications, and at worst undefined (I don't know if the standards define it, but if they do, I do know that not all applications follow the same rules). Paul From wes.turner at gmail.com Wed Mar 30 06:52:41 2016 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 30 Mar 2016 05:52:41 -0500 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> Message-ID: On Wed, Mar 30, 2016 at 4:20 AM, Paul Moore wrote: > On 30 March 2016 at 09:45, Sven R. Kunze wrote: > > On 29.03.2016 23:57, Paul Moore wrote: > >> > >> On 29 March 2016 at 22:00, Sven R. Kunze wrote: > >>> > >>> However, something that I cannot leave uncommented is "suboptimal > >>> representation for paths". What would your optimal representation for > >>> paths > >>> look like? I cannot believe that the current representation is so bad > and > >>> has been for so long and nobody, really nobody, has anything done about > >>> it. > >> > >> Well, pathlib.Path :-) > >> > >> The point here is that C's char* representation is a serialisation of > >> a path object, just like 123 is a serialisation of an integer object. > >> We don't object to having to convert user input to a string if we need > >> to, why object to having to convert it to a Path if appropriate? > > > > > > I think there is a misunderstanding here. Let me quote myself: > > > > '''I think most "practicality beats purity" folks don't want that either. > > They are just bloody lazy. They actually want the benefits of both, the > pure > > datastructure with its convenience methods and the dirty str-like thing > with > > its convenience methods.''' > > > > The desire is not "str vs. Path"; its "Path + str". (at least from what I > > can tell) > > I understand all that. I thought you were referring to Brett's comment > that C's "char *" format was "suboptimal" and asking what would be an > optimal representation (you said "your optimal representation", > meaning Brett's, so only he can answer to that but I took the question > more generally as what would be *an* optimal representation). My reply > was intended to say that I view something like pathlib.Path as > optimal. > Really, these are actual graph paths sequences Because: * When you have to abspath, normpath, check for '../.../../.$var./..', you have to parse it by splitting on /, anyway, which indicates that a list/tuple would be more optimal in some use cases. > > I don't personally see "working like a string" as being optimal, > precisely because it mixes the "structured object" level with the > "serialised representation" level. That's something Brett's article > clarified for me, so I suspect he'd have the same view. > " To e.g. rsync, this is a different thing: ./path/ ./path > > I doubt the people arguing for Path being a subclass of string would > be swayed by an argument like the above, though, as they would view it > as "purity over practicality". Personally, I find that careful > separation of concerns is a practicality benefit, in terms of > maintainability and clarity of code - a lesson I learned from having > to do far too many bytes/unicode migrations where separating out the > concepts was a vital step. > Practically, because path.py subclasses unicode in python2 and str in python 3, path.py should work with all existing standard library methods: https://github.com/jaraco/path.py/blob/master/path.py > > Anyway, this is pretty off topic, so I'll leave it there. > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From neatnate at gmail.com Wed Mar 30 08:36:17 2016 From: neatnate at gmail.com (Nathan Schneider) Date: Wed, 30 Mar 2016 13:36:17 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <1459308333.15976.563186042.556CA823@webmail.messagingengine.com> Message-ID: On Wed, Mar 30, 2016 at 4:25 AM, Random832 wrote: > On Tue, Mar 29, 2016, at 17:39, Ethan Furman wrote: > > On 03/29/2016 02:30 PM, Greg Ewing wrote: > > > Paul Moore wrote: > > >> > > >> P(wherever)/youwant/togo > > > > > > If P were a suitable seed object, this could be > > > > > > P/wherever/youwant/togo > > > > An interesting idea, but how would you differentiate between relative > > and absolute paths? > > Path.ROOT/absolute/path > Path.DOT/relative/path > How about: Path/absolute/path Path%relative/path [think: % as the Unix shell prompt] I also like the idea of being able to do Path/~homesubdir/path as an alias for Path/'~'/homesubdir/path Apart from the above, the user would have to know to quote special path characters like ~ and . and trailing / to avoid a syntax error: Path/'~' Path/'..'/relative/path Path/absolute/path/'/' [in rare cases where the trailing slash is necessary] Nathan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Wed Mar 30 08:46:30 2016 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 30 Mar 2016 21:46:30 +0900 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: <56FAFC5B.3000008@canterbury.ac.nz> <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> Message-ID: <22267.51878.159924.38717@turnbull.sk.tsukuba.ac.jp> Paul Moore writes: > And the URL syntax for drive letters is at best inconsistent across > applications, and at worst undefined (I don't know if the standards > define it, but if they do, I do know that not all applications > follow the same rules). AFAICS it only matters for the file scheme, for which see https://tools.ietf.org/html/draft-ietf-appsawg-file-scheme. That draft describes how several apps handle drive letters. From k7hoven at gmail.com Wed Mar 30 08:46:20 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 30 Mar 2016 15:46:20 +0300 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> References: <56FAFC5B.3000008@canterbury.ac.nz> <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Mar 30, 2016 at 7:06 AM, Stephen J. Turnbull wrote: [...] > The correct syntaxes per [1] and RFC 3986 are > > 4) Path("file:///http://www.example.com") > 5) Path("file://localhost/http://www.example.com") > 6) Path("file://[127.0.0.1]/http://www.example.com") > 7) Path("file://[::1]/http://www.example.com") > Even if correct, these do not refer to "http:/www.example.com", but to "/http:/www.example.com". An URI with a relative path would not make a lot of sense, because its meaning would depend on the context, which is against. Then again, all file system paths are 'relative' with respect to the file system you are working in. Also, while RFC 3986 is not super clear about this, I think '//' inside a URI path component may cause problems. IIUC this leads to a zero-length path segment '' in between the two slashes. It might work though if it it just gets passed forward to the file system in the end. I don't know if that can 'officially' be normalized to a single slash though. "URIs that identify in relation to the end-user's local context should only be used when the context itself is a defining aspect of the resource, such as when an on-line help manual refers to a file on the end- user's file system (e.g., "file:///etc/hosts")." - RFC 3986 > As far as I can tell the colon in "http:" is RFC 3986-legal, since it > has no URI syntactic meaning in the path component. That's right; per RFC 3986, colons are allowed in a URI path component, even if it is disallowed in *the first path segment* of a *relative reference*, which I assume is to make relative references unambiguous as *URI references* which can be URIs or relative references. That is, a URI reference "mailto:email at address.com" is a mailto-URL and not a relative reference equivalent to "./mailto:email at address.com". So basically, if you want to express the (ridiculous) path 'http:/www.example.com' as a relative reference, you'd need to do './http:/www.example.com'. >This isn't as > easy as it looks (which is why people are trying to delegate it to > something they think of as "simple"). > > There's an additional problem with trying to cram URIs and Path > together, which is that in a file system, "/a/b/symlink/../c" may > refer to any file system object depending on symlink's target which is > unknown, while as an URI path it refers to whatever "/a/b/c" refers > to, and nothing else. (This is the semantic glitch I was thinking of > earlier.) This is an interesting issue, because the behavior is not implemented consistently: k7hoven at pomelo ~ % mkdir -p foo/bar k7hoven at pomelo ~ % ln -s foo/bar baz k7hoven at pomelo ~ % cd baz/.. k7hoven at pomelo ~ % cd baz k7hoven at pomelo ~/baz % cd .. k7hoven at pomelo ~ % echo "am I in foo/ or in ~/ ?" > baz/../question.txt k7hoven at pomelo ~ % cat question.txt cat: question.txt: No such file or directory k7hoven at pomelo ~ % cat foo/question.txt am I in foo/ or in ~/ ? > This means that URIs can be canonicalized syntactically, while doing > so with file system paths is risky. And that URI normalization should not be done automatically, especially if it is not clear if it's an URI or not. Then sometimes you also want to do scheme-specific normalization. -Koos From k7hoven at gmail.com Wed Mar 30 09:30:15 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 30 Mar 2016 16:30:15 +0300 Subject: [Python-ideas] URLs/URIs + pathlib.Path + literal syntax = ? In-Reply-To: References: <56FAFC5B.3000008@canterbury.ac.nz> <22267.20655.16002.43709@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, Mar 30, 2016 at 10:55 AM, Paul Moore wrote: > the fact that most paths come > from variables, not from literals, and many come from user input > (sometimes even unintended user input such as a "*" expanded by the > shell)? It's easy enough to rewrite literals to be unambiguous, but in > order to do so for arbitrary input you need to basically implement > (part of) a URI parser... > If a user input (or config file) says "http://www.example.com/page.html", what does the user want? The http URL or the relative file-system path "http:/www.example.com/page.html"? How often are users *explicitly* asked for a *file-system path*? And if they notice that 'http://...' does not by default refer to the file system, who will complain? Maybe someone can imagine a realistic situation where interpreting "scheme://..." interpreted as an URI causes problems? - Koos From chris.barker at noaa.gov Wed Mar 30 10:26:51 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 30 Mar 2016 07:26:51 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FB922A.6090607@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> Message-ID: <-6058951228755932973@unknownmsgid> >> . They actually want the benefits of both, the pure datastructure with its convenience methods and the dirty str-like thing with its convenience methods.''' > > The desire is not "str vs. Path"; its "Path + str". (at least from what I can tell) I don't think so. ( though I agree with the lazy part). People want paths to be a strings so that they will work with all the code that already works with strings. That's the primary motivator. If early versions of Python had a path object, there'd be no problem. But whatever happened to duck typing? Paths don't need to BE strings. Rather, everything that needs a path needs to accept anything that acts like a path. I suppose a __path__ magic method would be the "proper" way to do this, but it's really not necessary, it can be handled by the libs themselves. I think there is a fine plan in place, but if more flexibility is required, something like numpy's asarray() would be handy -- it passes actual Numpy arrays through untouched, and makes various efforts to make an array out of anything else. It's remarkably easy and effective to write functions that take virtually anything remotely array like -- lists, tuples, array.arrays, custom objects, etc. -CHB From k7hoven at gmail.com Wed Mar 30 10:55:50 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 30 Mar 2016 17:55:50 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> Message-ID: On Wed, Mar 30, 2016 at 1:49 AM, Brett Cannon wrote: > > On Tue, 29 Mar 2016 at 15:41 Koos Zevenhoven wrote: >> >> I'll have to agree. But to be fair, Greg wrote something like 'a suitable >> seed object' so we could for ex. have P=PathSeed(), and use another >> operator. >> >> P@'rel/path' >> >> P@'/abs/path' >> >> Dunno. > [...] > Or even P.absolute and P.relative: > > P.relative/'relative'/'path' > P.absolute/'absolute'/'path' > > And you could even go as far as make one the default and the other not to > promote one over the other: > > P.relative/'relative'/'path' > P/'absolute'/'path' > The trick is, again, that the path is often in a variable coming from user input etc. and you don't (want to) deal with whether it is relative or absolute. This also still has a slash before relative paths too. Another thing is that, while / looks very nice, it makes you have to use parentheses when you want to directly call a method on the path: (somedir / relative_file_path).method() We have been making examples with /many/'path'/'objects'/joined_together. But how often do people really join more than two paths at once? Sometimes, in interactive sessions, I have wished for this to work: somedir(relative_file_path).method() ...which is kind of stupid ;) - Koos From p.f.moore at gmail.com Wed Mar 30 11:18:41 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 30 Mar 2016 16:18:41 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <-6058951228755932973@unknownmsgid> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> Message-ID: On 30 March 2016 at 15:26, Chris Barker - NOAA Federal wrote: >>> . They actually want the benefits of both, the pure datastructure with its convenience methods and the dirty str-like thing with its convenience methods.''' >> >> The desire is not "str vs. Path"; its "Path + str". (at least from what I can tell) > > I don't think so. ( though I agree with the lazy part). I'm not sure I follow your point here. > People want paths to be a strings so that they will work with all the code > that already works with strings. Correct. That's the prime motivation. But you then say > But whatever happened to duck typing? Paths don't need to BE strings. > Rather, everything that needs a path needs to accept anything that > acts like a path. But "all the code that already works with strings" doesn't do that. If we're allowed to change that code then a simple patharg = getattr(patharg, 'path', patharg) is sufficient to work with path objects or strings. Of course this doesn't address functions that *return* paths (as strings). There the caller has to wrap the return value in Path(). Or the function changes to return Path objects, which won't be backward compatible (whether that matters depends on what the code is). > I suppose a __path__ magic method would be the "proper" way to do > this, but it's really not necessary, it can be handled by the libs > themselves. Well, the 'path' attribute is basically the same as a __path__ magic method. (That may be what you mean by "can be handled by the libs" - I'm not sure how you'd handle it *other* than in the libraries that currently assume a string). > I think there is a fine plan in place, but if more flexibility is > required, something like numpy's asarray() would be handy -- it passes > actual Numpy arrays through untouched, and makes various efforts to > make an array out of anything else. > > It's remarkably easy and effective to write functions that take > virtually anything remotely array like -- lists, tuples, array.arrays, > custom objects, etc. Again, this seems to be backwards - the problem isn't treating things that aren't paths as paths (just do p = Path(p) and that's sorted - paths are immutable so there's no need for the complexity that I imagine asarray has to manage). The problem is existing code that expects and/or returns strings. Until that is changed or replaced with code that expects and returns Path objects, people will always need to convert to and from strings. And of course code that has to deal with *both* Path and string objects (for compatibility reasons) can easily enough handle anything it receives, but has a decision to make about what to return - if it returns Path objects, it won't be backward compatible. But if it returns string objects, we'll never get away from the need to convert strings to paths in our code at some point. As a simple case in point, what should the appdirs module (https://pypi.python.org/pypi/appdirs) do? Continue to return strings, and the user needs to wrap them in Path(), or switch to returning Path objects and break backward compatibility? Or maintain a messy API that caters (somehow) for both possibilities? It may be that we're talking at cross purposes here. I'm thinking very much of users working with library code on from PyPI and things like that. You may be looking at this from the perspective of a user who controls the majority of the code they are working with. I'm not sure. A transition like this is never simple, and libraries pretty much have to wait for end user code to switch first. And paths as string subclasses doesn't really count as switching, it simply makes it easier to ignore the issue. Arguably being able to pass a path object to code that expects a string means that you don't have to change your code twice, first to handle path objects because the library code doesn't, then again to switch back when the library code is fixed. But then there is zero motivation for the library code to change. And if code *accepting* paths doesn't change, then code *producing* paths won't either. And we're stuck where we are at the moment, with the string representation as the lowest common denominator representation of a path, and everyone converting back and forth if they want the richer Path interface (or deciding it's not worth it, and sticking to the old os.path routines...). Paul From srkunze at mail.de Wed Mar 30 11:32:16 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 30 Mar 2016 17:32:16 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> Message-ID: <56FBF180.4010203@mail.de> On 30.03.2016 16:55, Koos Zevenhoven wrote: > The trick is, again, that the path is often in a variable coming from > user input etc. and you don't (want to) deal with whether it is > relative or absolute. This also still has a slash before relative > paths too. > > Another thing is that, while / looks very nice, it makes you have to > use parentheses when you want to directly call a method on the path: > > (somedir / relative_file_path).method() > > We have been making examples with > /many/'path'/'objects'/joined_together. But how often do people really > join more than two paths at once? > > Sometimes, in interactive sessions, I have wished for this to work: > > somedir(relative_file_path).method() > > ...which is kind of stupid ;) This seems all to speak for p-strings. :) Best, Sven From ethan at stoneleaf.us Wed Mar 30 12:36:02 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 30 Mar 2016 09:36:02 -0700 Subject: [Python-ideas] Add a developer mode to Python: -X dev command line option In-Reply-To: <56FBA9DC.4070603@egenix.com> References: <56FBA9DC.4070603@egenix.com> Message-ID: <56FC0072.8010208@stoneleaf.us> On 03/30/2016 03:26 AM, M.-A. Lemburg wrote: > So overall, I guess adding a debug howto of some sort showing > and explaining the purpose of the various options would be more > helpful to developers than a meta-option which triggers some subset > of other available options. +1 A good doc would much more useful in figuring out - what options are available - what the options do -- ~Ethan~ From ethan at stoneleaf.us Wed Mar 30 12:49:12 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 30 Mar 2016 09:49:12 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> Message-ID: <56FC0388.8050801@stoneleaf.us> On 03/30/2016 08:18 AM, Paul Moore wrote: > And of course code that has to deal with *both* Path and string > objects (for compatibility reasons) can easily enough handle anything > it receives, but has a decision to make about what to return - if it > returns Path objects, it won't be backward compatible. But if it > returns string objects, we'll never get away from the need to convert > strings to paths in our code at some point. A solution here is to return what you receive: - str coming in? str going out - path coming in? path going out Of course, this gets muddied when multiple path/str arguments are accepted coming in. -- ~Ethan~ From ethan at stoneleaf.us Wed Mar 30 12:56:05 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 30 Mar 2016 09:56:05 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> Message-ID: <56FC0525.8070608@stoneleaf.us> On 03/30/2016 07:55 AM, Koos Zevenhoven wrote: > On Wed, Mar 30, 2016 at 1:49 AM, Brett Cannon wrote: >> Or even P.absolute and P.relative: >> >> P.relative/'relative'/'path' >> P.absolute/'absolute'/'path' Which involves even more typing and is less readable. > Sometimes, in interactive sessions, I have wished for this to work: > > somedir(relative_file_path).method() > > ...which is kind of stupid ;) Do you mean: base = Path('source/antipathy') base('test').listdir() I don't know about pathlib, but with antipathy's Path [1] you can do: base.listdir('test') -- ~Ethan~ [1] Yes, that one is mine. The step is to make it pathlib compatible, which will involve, at a minimum, renaming some attributes. From k7hoven at gmail.com Wed Mar 30 13:51:53 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 30 Mar 2016 20:51:53 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FC0525.8070608@stoneleaf.us> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> <56FC0525.8070608@stoneleaf.us> Message-ID: On Wed, Mar 30, 2016 at 7:56 PM, Ethan Furman wrote: > On 03/30/2016 07:55 AM, Koos Zevenhoven wrote: >> >> Sometimes, in interactive sessions, I have wished for this to work: >> >> somedir(relative_file_path).method() >> >> ...which is kind of stupid ;) > > > Do you mean: > > base = Path('source/antipathy') > base('test').listdir() Yes, I think. That is, equivalent to (base / 'test').listdir() > > I don't know about pathlib, but with antipathy's Path [1] you can do: > > base.listdir('test') > I have wished for base.mkdir("subdir") too, but the addition to pathlib would be backwards incompatible for people that give the mode argument as a positional argument. > [1] Yes, that one is mine. The step is to make it pathlib compatible, which > will involve, at a minimum, renaming some attributes. I wonder if it is still possible to rename pathlib attributes (and keep the old ones as deprecated aliases for a while), now that the adoption of pathlib is still limited. -Koos From jcgoble3 at gmail.com Wed Mar 30 14:02:39 2016 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Wed, 30 Mar 2016 14:02:39 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> <56FC0525.8070608@stoneleaf.us> Message-ID: (I have no interest in this subject of this thread, but wanted to point this out.) On Wed, Mar 30, 2016 at 1:51 PM, Koos Zevenhoven wrote: > I have wished for base.mkdir("subdir") too, but the addition to > pathlib would be backwards incompatible for people that give the mode > argument as a positional argument. > >> [snip] > > I wonder if it is still possible to rename pathlib attributes (and > keep the old ones as deprecated aliases for a while), now that the > adoption of pathlib is still limited. pathlib is still provisional (PEP 411), thus any changes can be made if deemed necessary, including backward incompatible changes. The bar for such changes is described by PEP 411 as "quite high", but they can occur. From ethan at stoneleaf.us Wed Mar 30 14:10:56 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 30 Mar 2016 11:10:56 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> <56FC0525.8070608@stoneleaf.us> Message-ID: <56FC16B0.2020201@stoneleaf.us> On 03/30/2016 10:51 AM, Koos Zevenhoven wrote: > I wonder if it is still possible to rename pathlib attributes (and > keep the old ones as deprecated aliases for a while), now that the > adoption of pathlib is still limited. The `.path` attribute, which seems a silly name, I look at as being necessary for compatibility with DirEntry. -- ~Ethan~ From k7hoven at gmail.com Wed Mar 30 14:24:05 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 30 Mar 2016 21:24:05 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B4CD.90404@canterbury.ac.nz> Message-ID: On Wed, Mar 30, 2016 at 6:18 PM, Paul Moore wrote: > > Of course this doesn't address functions that *return* paths (as > strings). There the caller has to wrap the return value in Path(). Or > the function changes to return Path objects, which won't be backward > compatible (whether that matters depends on what the code is). > So, below was a though of how to deal with this: StringPath, which is both a str and a Path (and/or perhaps PurePath). On Sun, Mar 27, 2016 at 2:33 AM, Koos Zevenhoven wrote: > > Well, just to reply to myself, here's a slightly crazy idea, which I'll > mention before I realize that it's a bad idea: > > What if there was a another class, say StringPath, that inherits from both > str and Path, which wraps another instance of Path, but is also a str. When > you call its Path methods, it would delegate them to the wrapped Path object > so that functions that now return paths as plain str could in future > versions start returning that type? While I still think it's a crazy idea, I'm not 100% convinced that it is the wrong thing to do, because there are already many kinds of Path objects. So a wrote a super hacky implementation of StringPath. Instances of something like this could be returned from functions that now return paths in strings. Here's a little demo with the toy implementation: >>> p = StringPath("foo/bar/baz") >>> p StringPath(PosixPath('foo/bar/baz')) >>> isinstance(p, pathlib.Path) True >>> isinstance(p, str) True >>> str(p) 'foo/bar/baz' >>> p + 'hello' 'foo/bar/bazhello' >>> p / 'hello' PosixPath('foo/bar/baz/hello') >>> 'hello' / p PosixPath('hello/foo/bar/baz') >>> p.split('r') ['foo/ba', '/baz'] >>> pathlib.Path(p) PosixPath('foo/bar/baz') So it's a str, but as soon as you do something Path-like with it, it gives you a Path back. But for anyone who thinks they have a string, it's a string, except for its repr. -Koos P.S. The implementation I used here is actually completely ridiculous, but if you want to look at it, it's here: https://gist.github.com/k7hoven/defb7f2eb9ccd9dbd0be0063a475058e From brett at python.org Wed Mar 30 14:26:57 2016 From: brett at python.org (Brett Cannon) Date: Wed, 30 Mar 2016 18:26:57 +0000 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FC16B0.2020201@stoneleaf.us> References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> <56FC0525.8070608@stoneleaf.us> <56FC16B0.2020201@stoneleaf.us> Message-ID: On Wed, 30 Mar 2016 at 11:10 Ethan Furman wrote: > On 03/30/2016 10:51 AM, Koos Zevenhoven wrote: > > > I wonder if it is still possible to rename pathlib attributes (and > > keep the old ones as deprecated aliases for a while), now that the > > adoption of pathlib is still limited. > > The `.path` attribute, which seems a silly name, I look at as being > necessary for compatibility with DirEntry. > I don't remember the exact issue that tracked the discussion, but there was back-and-forth on that attribute name and in the end Guido chose `path`. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Mar 30 14:33:28 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 30 Mar 2016 11:33:28 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> <56FC0525.8070608@stoneleaf.us> <56FC16B0.2020201@stoneleaf.us> Message-ID: <56FC1BF8.3020805@stoneleaf.us> On 03/30/2016 11:26 AM, Brett Cannon wrote: > On Wed, 30 Mar 2016 at 11:10 Ethan Furman wrote: >> The `.path` attribute, which seems a silly name, I look at as being >> necessary for compatibility with DirEntry. > > I don't remember the exact issue that tracked the discussion, but there > was back-and-forth on that attribute name and in the end Guido chose > `path`. Hmmm.... so - Path.path -> str - DirEntry.path -> str Perhaps DirEntry should grow a .Path attribute to a Path from. ;) -- ~Ethan~ From k7hoven at gmail.com Wed Mar 30 14:45:24 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 30 Mar 2016 21:45:24 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> <56FC0525.8070608@stoneleaf.us> <56FC16B0.2020201@stoneleaf.us> Message-ID: On Wed, Mar 30, 2016 at 9:26 PM, Brett Cannon wrote: > >> >> The `.path` attribute, which seems a silly name, I look at as being >> necessary for compatibility with DirEntry. > > > I don't remember the exact issue that tracked the discussion, but there was > back-and-forth on that attribute name and in the end Guido chose `path`. This one: https://bugs.python.org/issue22570 See my comment below (from this thread, but there were no reactions). I'm not sure if that option was ever considered. Can we let DirEntry dictate the future of Python? On Sun, Mar 27, 2016 at 5:40 PM, Koos Zevenhoven wrote: > > Actually, now that .path is not out yet, would it make sense to call it > Path.str or Path.strpath instead, and introduce the same thing on DirEntry > and guarantee a str (instead of str or bytes as DirEntry.path now does)? > Maybe that would lead to fewer broken implementations in third-party > libraries too? -Koos From k7hoven at gmail.com Wed Mar 30 19:29:24 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 31 Mar 2016 02:29:24 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <22262.51128.781599.727450@turnbull.sk.tsukuba.ac.jp> <56FA2AA9.5050102@mail.de> <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> <56FC0525.8070608@stoneleaf.us> <56FC16B0.2020201@stoneleaf.us> Message-ID: On Wed, Mar 30, 2016 at 9:45 PM, Koos Zevenhoven wrote: > > See my comment below (from this thread, but there were no reactions). > I'm not sure if that option was ever considered. Can we let DirEntry > dictate the future of Python? > BTW, don't get me wrong, I'm sure os.scandir is great, even if I have not used it :). What I mean is that it's easier to change something that was added more recently, i.e. when there are no old codebases that depend on it. I just think we should get these things right, before it's too late. > On Sun, Mar 27, 2016 at 5:40 PM, Koos Zevenhoven wrote: >> >> Actually, now that .path is not out yet, would it make sense to call it >> Path.str or Path.strpath instead, and introduce the same thing on DirEntry >> and guarantee a str (instead of str or bytes as DirEntry.path now does)? >> Maybe that would lead to fewer broken implementations in third-party >> libraries too? > > -Koos From ethan at stoneleaf.us Wed Mar 30 20:02:25 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 30 Mar 2016 17:02:25 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> <56FC0525.8070608@stoneleaf.us> <56FC16B0.2020201@stoneleaf.us> Message-ID: <56FC6911.3090901@stoneleaf.us> Koos Zevenhoven wrote: > > See my comment below (from this thread, but there were no reactions). > I'm not sure if that option was ever considered. Can we let DirEntry > dictate the future of Python? No. On the other hand, we'd need a /really/ good reason to change it now. > Actually, now that .path is not out yet, would it make sense to call it > Path.str or Path.strpath instead, and introduce the same thing on DirEntry > and guarantee a str (instead of str or bytes as DirEntry.path now does)? > Maybe that would lead to fewer broken implementations in third-party > libraries too? DirEntry needs to return the same type it was given, and one can give it both bytes and strings. Your first point /may/ be valid. The big question is: is DirEntry provisional? I don't think it is. In which case, we're stuck with .path on DirEntry, and may as well use the same name for the same idea on pathlib.Path. -- ~Ethan~ From random832 at fastmail.com Wed Mar 30 20:17:50 2016 From: random832 at fastmail.com (Random832) Date: Wed, 30 Mar 2016 20:17:50 -0400 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FC6911.3090901@stoneleaf.us> References: <56FA2D44.8080707@mail.de> <56FAF3DA.4090200@canterbury.ac.nz> <56FAF62A.9020104@stoneleaf.us> <56FAFC8B.9060805@stoneleaf.us> <56FAFEAA.2010503@stoneleaf.us> <56FC0525.8070608@stoneleaf.us> <56FC16B0.2020201@stoneleaf.us> <56FC6911.3090901@stoneleaf.us> Message-ID: <1459383470.3382342.564175914.61B59156@webmail.messagingengine.com> On Wed, Mar 30, 2016, at 20:02, Ethan Furman wrote: > Koos Zevenhoven wrote: > > > > See my comment below (from this thread, but there were no reactions). > > I'm not sure if that option was ever considered. Can we let DirEntry > > dictate the future of Python? > > No. On the other hand, we'd need a /really/ good reason to change it > now. Just change it in Python 4000 ;) From srkunze at mail.de Thu Mar 31 05:09:54 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 31 Mar 2016 11:09:54 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FC0388.8050801@stoneleaf.us> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> <56FC0388.8050801@stoneleaf.us> Message-ID: <56FCE962.40700@mail.de> On 30.03.2016 18:49, Ethan Furman wrote: > A solution here is to return what you receive: > > - str coming in? str going out > - path coming in? path going out > > Of course, this gets muddied when multiple path/str arguments are > accepted coming in. I have bad memories about this kind of approach. It reminds me of the peculiarity of os(.path)-related functions in Python 2 that return either bytes or unicode depending on the argument's type. That's utterly confusing and I remember us hunting bugs related to this for hours (basically looking for "where does this byte string come from?"). Best, Sven From p.f.moore at gmail.com Thu Mar 31 05:15:53 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 31 Mar 2016 10:15:53 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FCE962.40700@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> <56FC0388.8050801@stoneleaf.us> <56FCE962.40700@mail.de> Message-ID: On 31 March 2016 at 10:09, Sven R. Kunze wrote: > On 30.03.2016 18:49, Ethan Furman wrote: >> >> A solution here is to return what you receive: >> >> - str coming in? str going out >> - path coming in? path going out >> >> Of course, this gets muddied when multiple path/str arguments are accepted >> coming in. > > I have bad memories about this kind of approach. It reminds me of the > peculiarity of os(.path)-related functions in Python 2 that return either > bytes or unicode depending on the argument's type. > > That's utterly confusing and I remember us hunting bugs related to this for > hours (basically looking for "where does this byte string come from?"). Any suggestions for a better design? It might be nice for someone to write a blog entry or posting somewhere on "how to upgrade your 3rd party library to support pathlib" and one of the harder questions will be "how do I decide whether to return a string or a path?" If we had a good recommendation for this, it might help adoption. Paul From srkunze at mail.de Thu Mar 31 05:46:38 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 31 Mar 2016 11:46:38 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> <56FC0388.8050801@stoneleaf.us> <56FCE962.40700@mail.de> Message-ID: <56FCF1FE.7000903@mail.de> On 31.03.2016 11:15, Paul Moore wrote: > Any suggestions for a better design? It might be nice for someone to > write a blog entry or posting somewhere on "how to upgrade your 3rd > party library to support pathlib" and one of the harder questions will > be "how do I decide whether to return a string or a path?" If we had a > good recommendation for this, it might help adoption. Good point. This discussion has been long already so it might make sense to write something down in a concise manner. I am not sure if I can make a good suggestion here because I am still trapped at the point where I need to sort out if a path more like a dict or another structured datatype, or if it is more a monolithic object. That will be the next blog post topic for me. "how do I decide whether to return a string or a path?" << very good question btw; my initial thought would be; make two functions (maybe the same name in two different namespaces), so it's crystal clear what this function takes a arguments and what it returns. But I need to think more about this. Best, Sven From p.f.moore at gmail.com Thu Mar 31 06:59:45 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 31 Mar 2016 11:59:45 +0100 Subject: [Python-ideas] Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?] Message-ID: On 31 March 2016 at 10:46, Sven R. Kunze wrote: > On 31.03.2016 11:15, Paul Moore wrote: >> >> Any suggestions for a better design? It might be nice for someone to >> write a blog entry or posting somewhere on "how to upgrade your 3rd >> party library to support pathlib" and one of the harder questions will >> be "how do I decide whether to return a string or a path?" If we had a >> good recommendation for this, it might help adoption. > > Good point. This discussion has been long already so it might make sense to > write something down in a concise manner. > > I am not sure if I can make a good suggestion here because I am still > trapped at the point where I need to sort out if a path more like a dict or > another structured datatype, or if it is more a monolithic object. That will > be the next blog post topic for me. I'm only talking here about supporting the *existing* pathlib object. Discussing alternative path object options doesn't help promote adoption of the stdlib path object, which is in all honesty the only thing I'm interested in. > "how do I decide whether to return a string or a path?" << very good > question btw; my initial thought would be; make two functions (maybe the > same name in two different namespaces), so it's crystal clear what this > function takes a arguments and what it returns. But I need to think more > about this. So, coming at this from the perspective of a library developer: 1. Which interface (the str or the path one) gets the "obvious" name? Backward compatibility pretty much dictates it'd be the str version. 2. How do I support Python 2.7? And Python 3.3/3.4 (where the ".path" attribute doesn't exist)? 3. What's my transition plan for the future when I deprecate the str interface and only support path objects? 4. What will my users have to do if they want to use Path objects in their code? And how do I minimise their pain in doing so? So it's a pretty hard issue, to think about adding pathlib support if you *return* paths. (It's essentially trivial if you only *consume* paths - just add "patharg = getattr(patharg, 'path', patharg)" at the top of your APIs, and you're done - apart from testing, documentation, etc, of course :-)) But without library adoption, *users* bear the pain of adapting everything to Path objects. And that's a real barrier to adoption. Maybe once the stdlib uses path objects more consistently (there's a *lot* of modules that could do with accepting Path or str arguments equally, and one or two like tempfile that create path names), it will be easier to argue for wider pathlib support in 3rd party libraries. Paul From k7hoven at gmail.com Thu Mar 31 07:31:58 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 31 Mar 2016 14:31:58 +0300 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FCF1FE.7000903@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> <56FC0388.8050801@stoneleaf.us> <56FCE962.40700@mail.de> <56FCF1FE.7000903@mail.de> Message-ID: On Thu, Mar 31, 2016 at 12:46 PM, Sven R. Kunze wrote: > On 31.03.2016 11:15, Paul Moore wrote: >> >> Any suggestions for a better design? It might be nice for someone to >> write a blog entry or posting somewhere on "how to upgrade your 3rd >> party library to support pathlib" and one of the harder questions will >> be "how do I decide whether to return a string or a path?" If we had a >> good recommendation for this, it might help adoption. > True. When we do have the answer, we need to let people know about it. > Good point. This discussion has been long already so it might make sense to > write something down in a concise manner. > > I am not sure if I can make a good suggestion here because I am still > trapped at the point where I need to sort out if a path more like a dict or > another structured datatype, or if it is more a monolithic object. That will > be the next blog post topic for me. > While discussing, pondering and experimenting with this, I have formed a quite clear view on how I think we should move forward with this. I'm working on a proposal that attempts to not be in conflict with any of the goals. I hope it is not in conflict with your present thoughts about structured/vs monolithic objects either :) > "how do I decide whether to return a string or a path?" << very good > question btw; my initial thought would be; make two functions (maybe the > same name in two different namespaces), so it's crystal clear what this > function takes a arguments and what it returns. But I need to think more > about this. > > Best, > Sven > From k7hoven at gmail.com Thu Mar 31 07:55:09 2016 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 31 Mar 2016 14:55:09 +0300 Subject: [Python-ideas] Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?] In-Reply-To: References: Message-ID: On Thu, Mar 31, 2016 at 1:59 PM, Paul Moore wrote: > On 31 March 2016 at 10:46, Sven R. Kunze wrote: >> On 31.03.2016 11:15, Paul Moore wrote: >>> >>> Any suggestions for a better design? It might be nice for someone to >>> write a blog entry or posting somewhere on "how to upgrade your 3rd >>> party library to support pathlib" and one of the harder questions will >>> be "how do I decide whether to return a string or a path?" If we had a >>> good recommendation for this, it might help adoption. >> >> Good point. This discussion has been long already so it might make sense to >> write something down in a concise manner. >> >> I am not sure if I can make a good suggestion here because I am still >> trapped at the point where I need to sort out if a path more like a dict or >> another structured datatype, or if it is more a monolithic object. That will >> be the next blog post topic for me. > > I'm only talking here about supporting the *existing* pathlib object. > Discussing alternative path object options doesn't help promote > adoption of the stdlib path object, which is in all honesty the only > thing I'm interested in. > >> "how do I decide whether to return a string or a path?" << very good >> question btw; my initial thought would be; make two functions (maybe the >> same name in two different namespaces), so it's crystal clear what this >> function takes a arguments and what it returns. But I need to think more >> about this. > > So, coming at this from the perspective of a library developer: > > 1. Which interface (the str or the path one) gets the "obvious" name? > Backward compatibility pretty much dictates it'd be the str version. > 2. How do I support Python 2.7? And Python 3.3/3.4 (where the ".path" > attribute doesn't exist)? > 3. What's my transition plan for the future when I deprecate the str > interface and only support path objects? > 4. What will my users have to do if they want to use Path objects in > their code? And how do I minimise their pain in doing so? > > So it's a pretty hard issue, to think about adding pathlib support if > you *return* paths. (It's essentially trivial if you only *consume* > paths - just add "patharg = getattr(patharg, 'path', patharg)" at the > top of your APIs, and you're done - apart from testing, documentation, > etc, of course :-)) > > But without library adoption, *users* bear the pain of adapting > everything to Path objects. And that's a real barrier to adoption. > Maybe once the stdlib uses path objects more consistently (there's a > *lot* of modules that could do with accepting Path or str arguments > equally, and one or two like tempfile that create path names), it will > be easier to argue for wider pathlib support in 3rd party libraries. > > Paul I agree about pretty much everything here. My upcoming proposal addresses most of this, but I will add more words about writing code that also works in older Python versions. Thanks! - Koos From p.f.moore at gmail.com Thu Mar 31 09:09:07 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 31 Mar 2016 14:09:07 +0100 Subject: [Python-ideas] Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?] In-Reply-To: References: Message-ID: On 31 March 2016 at 12:55, Koos Zevenhoven wrote: > My upcoming proposal addresses most of this I'll wait to see what you propose, but understand that I'm not talking in this thread about changing pathlib significantly (certainly not in terms of making Path objects "act like" strings). I'm personally not interested in considering any model other than str for the textual representation of paths, and pathlib.Path for the rich path object. I'm interested in trying to promote the existing approach, not in changing it before it's had a chance to prove itself. I no longer see the idea of pathlib.Path subclassing (or otherwise acting transparently as) str as an important goal, as it only really affects passing paths to libraries that expect str, which is the direction that's a trivial library change to fix (and I think we *do* need to encourage libraries to make changes to support pathlib, rather than letting them ignore it). In fact, my first proposal would be to encourage people to raise issues against their favourite 3rd party libraries asking for pathlib support for arguments representing paths. Something like the following should make it clear to the project that it's not a big issue to do so: In function xxx, please support passing a pathlib.Path for argument bar. This can be done simply by adding a line bar = getattr(bar, 'path', bar) at the top of the function, which will support pathlib from Python 3.5.2 and 3.4.5 onwards. I do not think that there is any need to support older versions of pathlib. Hmm, I've just noticed that the ".path" attribute is new in 3.5.2, which is not released yet. Maybe we should wait until it is before promoting this approach :-) I do still prefer this approach, though, as it doesn't require importing pathlib, and it doesn't accept other types of object the way str(patharg) does. Paul From srkunze at mail.de Thu Mar 31 10:43:07 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 31 Mar 2016 16:43:07 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> <56FC0388.8050801@stoneleaf.us> <56FCE962.40700@mail.de> <56FCF1FE.7000903@mail.de> Message-ID: <56FD377B.7000204@mail.de> On 31.03.2016 13:31, Koos Zevenhoven wrote: > On Thu, Mar 31, 2016 at 12:46 PM, Sven R. Kunze wrote: >> I am not sure if I can make a good suggestion here because I am still >> trapped at the point where I need to sort out if a path more like a dict or >> another structured datatype, or if it is more a monolithic object. That will >> be the next blog post topic for me. > While discussing, pondering and experimenting with this, I have formed > a quite clear view on how I think we should move forward with this. > I'm working on a proposal that attempts to not be in conflict with any > of the goals. I hope it is not in conflict with your present thoughts > about structured/vs monolithic objects either :) Not at all. The following post it just a reflection of this whole discussion to make it clear to myself and hopefully to others about where several thoughts emerged from and why this whole topic started in the first place. Basically, a structured collection of thoughts, mainly mine: http://srkunze.blogspot.com/2016/03/what-is-path.html I pasted the interesting meat below: *What is a path in the first place?* Brett Cannon made a good point of why PEP 428 (that's the one which introduced pathlib as a stdlib module) deliberately chose Path not to inherit from string. I pondered over it for a while and saw that from his perspective a path is actually not a string but rather a complex data-structure consisting of parts with distinct meanings: literally the steps (of which a path consists) to a resource. The string which represents a path as most people know it is just that: a representation of a more complex object, just like a dict or a list. Let me make this a bit clearer. I think we agree on the following: writing down 21 characters in a row is a string, right? So, what about these 21 characters? {1: 11, 2: 12, 3: 13} If you see that in a Python program (and presumably in many other modern programming languages), you associate that with a dictionary, mapping, hash, etc. So, these 21 characters are a mere representation of a complex object with a very rich functionality. The following paragraphs summarizes what makes the discussion about paths and strings to hard. Depending on whom you ask there are different interpretations of what a path actually is. *Paths as complex objects and strings as their representation* Let's put this analogy to work with paths. If you come from a language that treats strings as file paths (like Python), you can imagine and categorize the facilities of pathlib like so: * pure path - operating on the path string * concrete path - operating on files corresponding to the given path The classic "extract the file extension" issue is done easily with the pure path methods. Writing to a file is also easily done with concrete path operations. So, it seems paths are pretty complex objects with some internal structure and a lot of functionality. *Paths as monolithic object for addressing resources* The previous interpretation is not the only one. Despite all the fine functionality of extracting file extensions, concatenating parts to a larger path, etc., building a path is not an end in itself. When you got a path, it addresses a resource on a machine. When doing so for reading or writing that resource, you actually don't care about whether the path consists of parts or not. To you, it's a monolithic structure, an address. But, you might say, each part of a path represents a directory in hierarchical file systems. Sure that is true for many file systems but not for all. Moreover, how often do you really care about the underlying directory structure? It needs to be there to make things work, of course. When it's there, you mostly don't care. How often do you need to create a subtree in an directory in order to create a single config file? I encounter this once in a while and to be honest: it sucks. ```touch /home/me/on/your/ssd.conf``` will fail if the directory "on/your/" has not been created by somebody before me. Especially for me, as a Web developer, it's quite hard to understand what purpose this restriction serves. Within a Web application the hierarchy of URLs is an emergent property not a prerequisite. Users of git are accustomed to not-committing directories in. Why? Because it's unnecessary and the directory structure is again emerging from the files names themselves (aka from the content). This said, it's rather cumbersome to attribute semantics to the parts of a string that happens to be separated by "/" or "\". At least to me, a path made of one piece. *What about security then?* One can further argue that Web development and git repositories are different here. There is a clear boundary where a path can lead. A URL path cannot address a foreign resource on another domain. git file paths are contained within the repository root. See the common theme? There is a container from where the path of a resource cannot escape. If you have a complete file system available at your fingertips, a lot harm can be done when malicious user input is concatenated unattendedly as a subpath; actually to address a resource within a container but misused to gain access to the complete file system. I cannot say if the container pattern would work for everybody but it's definitely worth exploring as there are some prominent working examples out there. *Conclusion* I really like pathlib since it solves many frequently asked questions in the right way once and for all. But I don't like using it as an argument again inheriting paths from strings saying paths have internal structure in contrast to strings. At least to me, they do not. That, on the other hand, does not necessarily mean inheriting path from string is a good idea but it makes it no worse one than it was before. That's it. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 31 10:59:51 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 31 Mar 2016 07:59:51 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FD377B.7000204@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> <56FC0388.8050801@stoneleaf.us> <56FCE962.40700@mail.de> <56FCF1FE.7000903@mail.de> <56FD377B.7000204@mail.de> Message-ID: <56FD3B67.1000507@stoneleaf.us> On 03/31/2016 07:43 AM, Sven R. Kunze wrote: > I pasted the interesting meat below: Some interesting insights! Thanks for sharing. -- ~Ethan~ From p.f.moore at gmail.com Thu Mar 31 11:02:22 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 31 Mar 2016 16:02:22 +0100 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FD377B.7000204@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> <56FC0388.8050801@stoneleaf.us> <56FCE962.40700@mail.de> <56FCF1FE.7000903@mail.de> <56FD377B.7000204@mail.de> Message-ID: That's a very interesting perspective, and one I'd not fully appreciated. Thanks for posting it. There's a lot I still have to think about in your post, but there's one point I'd like to comment on already: On 31 March 2016 at 15:43, Sven R. Kunze wrote: > When doing so for reading or writing that resource, you actually don't care > about whether the path consists of parts or not. While that's true for *existing* resources (files) it's not so true for when you're creating a file. Like it or not, the filesystem imposes a set of container-containee relationships that is the directory structure. Your mental model (the application domain view) may not care about this, but at the "path" level, we're representing the filesystem structure, where directories are real, and so a/b/c being a set of 3 nested objects, 2 directories and a leaf object (which may be a file or itself a directory) is the reality being modeled. So I'd suggest that your "resource address" is a higher level abstraction, layered on top of the filesystem. In many ways, it's a URI - and as such may map to a filesystem path or to something else. This also clarifies (at least to me) why I am uncomfortable about the idea of merging pathlib and URIs - they are at 2 different abstraction levels. It's somewhat interesting to me that the "resource address" level of abstraction is a higher level that the "filesystem path" level, but has *less* structure. That seems reasonable to me (you abstract away the details). And it makes the string representation at the "lowest level" the really odd case, as it has *no* structure, and yet it's representing a detail level of object structure. Maybe that's an indication of why it's useful to have a structured Path object at this level rather than working solely with strings? Paul From wes.turner at gmail.com Thu Mar 31 11:36:11 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 31 Mar 2016 10:36:11 -0500 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> <56FC0388.8050801@stoneleaf.us> <56FCE962.40700@mail.de> <56FCF1FE.7000903@mail.de> <56FD377B.7000204@mail.de> Message-ID: On Mar 31, 2016 10:02 AM, "Paul Moore" wrote: > > That's a very interesting perspective, and one I'd not fully > appreciated. Thanks for posting it. > > There's a lot I still have to think about in your post, but there's > one point I'd like to comment on already: > > On 31 March 2016 at 15:43, Sven R. Kunze wrote: > > When doing so for reading or writing that resource, you actually don't care > > about whether the path consists of parts or not. > > While that's true for *existing* resources (files) it's not so true > for when you're creating a file. Like it or not, the filesystem > imposes a set of container-containee relationships that is the > directory structure. Your mental model (the application domain view) > may not care about this, but at the "path" level, we're representing > the filesystem structure, where directories are real, and so a/b/c > being a set of 3 nested objects, 2 directories and a leaf object > (which may be a file or itself a directory) is the reality being > modeled. > > So I'd suggest that your "resource address" is a higher level > abstraction, layered on top of the filesystem. In many ways, it's a > URI - and as such may map to a filesystem path or to something else. > This also clarifies (at least to me) why I am uncomfortable about the > idea of merging pathlib and URIs - they are at 2 different abstraction > levels. > > It's somewhat interesting to me that the "resource address" level of > abstraction is a higher level that the "filesystem path" level, but > has *less* structure. That seems reasonable to me (you abstract away > the details). And it makes the string representation at the "lowest > level" the really odd case, as it has *no* structure, and yet it's > representing a detail level of object structure. Maybe that's an > indication of why it's useful to have a structured Path object at this > level rather than working solely with strings? it would seem that the utility here includes having a @memoize-d parsed copy of the path; and then not doubling forward/backslashes ... and then checking for '../' traversals and any chroot-like relpath predicates w/ e.g. fnmatch); and not shell quoting None = gethostname() os.path.sep.join(['file:', None, '/home/user/']) os.path.sep.join(['file:', None, ('/home', '/user/')]) p = Path('/home/user/') p.parts # ... URLObject + path.py > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Mar 31 11:40:27 2016 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 31 Mar 2016 10:40:27 -0500 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> <56FC0388.8050801@stoneleaf.us> <56FCE962.40700@mail.de> <56FCF1FE.7000903@mail.de> <56FD377B.7000204@mail.de> Message-ID: On Mar 31, 2016 10:36 AM, "Wes Turner" wrote: > > > On Mar 31, 2016 10:02 AM, "Paul Moore" wrote: > > > > That's a very interesting perspective, and one I'd not fully > > appreciated. Thanks for posting it. > > > > There's a lot I still have to think about in your post, but there's > > one point I'd like to comment on already: > > > > On 31 March 2016 at 15:43, Sven R. Kunze wrote: > > > When doing so for reading or writing that resource, you actually don't care > > > about whether the path consists of parts or not. > > > > While that's true for *existing* resources (files) it's not so true > > for when you're creating a file. Like it or not, the filesystem > > imposes a set of container-containee relationships that is the > > directory structure. Your mental model (the application domain view) > > may not care about this, but at the "path" level, we're representing > > the filesystem structure, where directories are real, and so a/b/c > > being a set of 3 nested objects, 2 directories and a leaf object > > (which may be a file or itself a directory) is the reality being > > modeled. > > > > So I'd suggest that your "resource address" is a higher level > > abstraction, layered on top of the filesystem. In many ways, it's a > > URI - and as such may map to a filesystem path or to something else. > > This also clarifies (at least to me) why I am uncomfortable about the > > idea of merging pathlib and URIs - they are at 2 different abstraction > > levels. > > > > It's somewhat interesting to me that the "resource address" level of > > abstraction is a higher level that the "filesystem path" level, but > > has *less* structure. That seems reasonable to me (you abstract away > > the details). And it makes the string representation at the "lowest > > level" the really odd case, as it has *no* structure, and yet it's > > representing a detail level of object structure. Maybe that's an > > indication of why it's useful to have a structured Path object at this > > level rather than working solely with strings? > > it would seem that the utility here includes having a @memoize-d parsed copy of the path; and then not doubling forward/backslashes ... and then checking for '../' traversals and any chroot-like relpath predicates w/ e.g. fnmatch); and not shell quoting > > None = gethostname() > os.path.sep.join(['file:', None, '/home/user/']) > os.path.sep.join(['file:', None, ('/home', '/user/')]) > > p = Path('/home/user/') > p.parts > > # ... URLObject + path.py URLObject has a .path attr for getattr, too. - char*: bytestring, unicode/str > > > > > Paul > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Thu Mar 31 11:56:00 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 31 Mar 2016 17:56:00 +0200 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> <56FC0388.8050801@stoneleaf.us> <56FCE962.40700@mail.de> <56FCF1FE.7000903@mail.de> <56FD377B.7000204@mail.de> Message-ID: <56FD4890.9070005@mail.de> On 31.03.2016 17:02, Paul Moore wrote: > That's a very interesting perspective, and one I'd not fully > appreciated. Thanks for posting it. You're welcome. > There's a lot I still have to think about in your post, but there's > one point I'd like to comment on already: > > On 31 March 2016 at 15:43, Sven R. Kunze wrote: >> When doing so for reading or writing that resource, you actually don't care >> about whether the path consists of parts or not. > While that's true for *existing* resources (files) it's not so true > for when you're creating a file. Like it or not, the filesystem > imposes a set of container-containee relationships that is the > directory structure. Your mental model (the application domain view) > may not care about this, but at the "path" level, we're representing > the filesystem structure, where directories are real, and so a/b/c > being a set of 3 nested objects, 2 directories and a leaf object > (which may be a file or itself a directory) is the reality being > modeled. > > So I'd suggest that your "resource address" is a higher level > abstraction, layered on top of the filesystem. In many ways, it's a > URI - and as such may map to a filesystem path or to something else. > This also clarifies (at least to me) why I am uncomfortable about the > idea of merging pathlib and URIs - they are at 2 different abstraction > levels. > > It's somewhat interesting to me that the "resource address" level of > abstraction is a higher level that the "filesystem path" level, but > has *less* structure. That seems reasonable to me (you abstract away > the details). And it makes the string representation at the "lowest > level" the really odd case, as it has *no* structure, and yet it's > representing a detail level of object structure. Maybe that's an > indication of why it's useful to have a structured Path object at this > level rather than working solely with strings? That all makes perfect sense now. I didn't expect such a quick and enlightening response. That's awesome. Thanks a lot. That might also influence how we look at the other thread about including better URL handling to Python. It's just another abstraction level and the main question is: is Python ready for that level or better use a third party library for it. Best, Sven From ethan at stoneleaf.us Thu Mar 31 12:03:32 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 31 Mar 2016 09:03:32 -0700 Subject: [Python-ideas] Working with Path objects: p-strings? In-Reply-To: <56FD4890.9070005@mail.de> References: <56F5B3F7.40502@gmail.com> <56FA33C0.4000902@mail.de> <56FA4E31.3010908@gmail.com> <56FAECD8.3030605@mail.de> <56FB922A.6090607@mail.de> <-6058951228755932973@unknownmsgid> <56FC0388.8050801@stoneleaf.us> <56FCE962.40700@mail.de> <56FCF1FE.7000903@mail.de> <56FD377B.7000204@mail.de> <56FD4890.9070005@mail.de> Message-ID: <56FD4A54.6070602@stoneleaf.us> On 03/31/2016 08:56 AM, Sven R. Kunze wrote: > That might also influence how we look at the other thread about > including better URL handling to Python. It's just another abstraction > level and the main question is: is Python ready for that level or better > use a third party library for it. The actual questions are: 1) Should URL abstraction be merged in to pathlib? 2) If no, does it /need/ to be in the stdlib? -- ~Ethan~ From mahanmarwat at gmail.com Thu Mar 31 12:29:36 2016 From: mahanmarwat at gmail.com (Mahan Marwat) Date: Thu, 31 Mar 2016 21:29:36 +0500 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions In-Reply-To: References: Message-ID: ---------- Forwarded message ---------- From: Mahan Marwat Date: Thu, Mar 31, 2016 at 9:09 PM Subject: Make parenthesis optional in parameterless functions definitions To: python-ideas at python.org Hi, I have an idea of making parenthesis optional for functions having no parameters. i.e def greet: # note the missing parenthesis print('hello') The less awkward characters we have, the more readable our code will be (Beautiful is better then ugly). Some people argued that function definition with parenthesis seems to them natural. But actually it seems to us natural, because we have been used to it a-lot. IMHO parenthesisless functions definitions are natural and readable. In Python we have already adopted this. i.e parenthesis are optional in `if` statements, in class definition, in tuple etc if x == 1: # parenthesis are optional here pass class Greet: # now we don't explicitly inherit from `object` pass Tuple = 1, 2, 3 # parenthesis are optional here too Please, give your opinion. Thanks, Adnan Khan -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Thu Mar 31 12:47:53 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 31 Mar 2016 18:47:53 +0200 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions In-Reply-To: References: Message-ID: <56FD54B9.5060803@mail.de> On 31.03.2016 18:29, Mahan Marwat wrote: > > ---------- Forwarded message ---------- > From: *Mahan Marwat* > > Date: Thu, Mar 31, 2016 at 9:09 PM > Subject: Make parenthesis optional in parameterless functions definitions > To: python-ideas at python.org > > > Hi, > > I have an idea of making parenthesis optional for functions having no > parameters. i.e > > def greet: # note the missing parenthesis > print('hello') > > The less awkward characters we have, the more readable our code will > be (Beautiful is better then ugly). Some people argued that function > definition with parenthesis seems to them natural. But actually it > seems to us natural, because we have been used to it a-lot. IMHO > parenthesisless functions definitions are natural and readable. > > In Python we have already adopted this. i.e parenthesis are optional > in `if` statements, in class definition, in tuple etc > > if x == 1: # parenthesis are optional here > pass > > class Greet: # now we don't explicitly inherit from `object` > pass > > Tuple = 1, 2, 3 # parenthesis are optional here too > > Please, give your opinion. +1 [side-note: although I admit that it would not change much where we work. We mostly use classes and there you need to have "self" as minimum parameter. But the idea is actually quite nice.] Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Thu Mar 31 13:27:58 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Thu, 31 Mar 2016 13:27:58 -0400 Subject: [Python-ideas] Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?] In-Reply-To: References: Message-ID: <56FD5E1E.8060904@sdamon.com> I think this is missing step zero. What does pathlib even offer me, the library developer? Why should I use it? I cannot be the only person who does not understand why it exists, let alone why I should be using it in my libraries. On 3/31/2016 06:59, Paul Moore wrote: > 1. Which interface (the str or the path one) gets the "obvious" name? > Backward compatibility pretty much dictates it'd be the str version. > 2. How do I support Python 2.7? And Python 3.3/3.4 (where the ".path" > attribute doesn't exist)? > 3. What's my transition plan for the future when I deprecate the str > interface and only support path objects? > 4. What will my users have to do if they want to use Path objects in > their code? And how do I minimise their pain in doing so? From tritium-list at sdamon.com Thu Mar 31 13:30:47 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Thu, 31 Mar 2016 13:30:47 -0400 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions In-Reply-To: References: Message-ID: <56FD5EC7.7060909@sdamon.com> While it would be consistent, and if you suggested it in 1991 probably should have been included... are the two extra keystrokes really a problem? On 3/31/2016 12:29, Mahan Marwat wrote: > > ---------- Forwarded message ---------- > From: *Mahan Marwat* > > Date: Thu, Mar 31, 2016 at 9:09 PM > Subject: Make parenthesis optional in parameterless functions definitions > To: python-ideas at python.org > > > Hi, > > I have an idea of making parenthesis optional for functions having no > parameters. i.e > > def greet: # note the missing parenthesis > print('hello') > > The less awkward characters we have, the more readable our code will > be (Beautiful is better then ugly). Some people argued that function > definition with parenthesis seems to them natural. But actually it > seems to us natural, because we have been used to it a-lot. IMHO > parenthesisless functions definitions are natural and readable. > > In Python we have already adopted this. i.e parenthesis are optional > in `if` statements, in class definition, in tuple etc > > if x == 1: # parenthesis are optional here > pass > > class Greet: # now we don't explicitly inherit from `object` > pass > > Tuple = 1, 2, 3 # parenthesis are optional here too > > Please, give your opinion. > > Thanks, Adnan Khan > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 31 13:32:11 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 31 Mar 2016 10:32:11 -0700 Subject: [Python-ideas] Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?] In-Reply-To: <56FD5E1E.8060904@sdamon.com> References: <56FD5E1E.8060904@sdamon.com> Message-ID: <56FD5F1B.20108@stoneleaf.us> On 03/31/2016 10:27 AM, Alexander Walters wrote: > I think this is missing step zero. What does pathlib even offer me, the > library developer? Why should I use it? I cannot be the only person > who does not understand why it exists, let alone why I should be using > it in my libraries. Easier path manipulations. A better user experience for those users that use pathlib. Much pain on your part until the stdlib fully supports pathlib.Paths. -- ~Ethan~ From tritium-list at sdamon.com Thu Mar 31 13:36:34 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Thu, 31 Mar 2016 13:36:34 -0400 Subject: [Python-ideas] Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?] In-Reply-To: <56FD5F1B.20108@stoneleaf.us> References: <56FD5E1E.8060904@sdamon.com> <56FD5F1B.20108@stoneleaf.us> Message-ID: <56FD6022.2030702@sdamon.com> On 3/31/2016 13:32, Ethan Furman wrote: > On 03/31/2016 10:27 AM, Alexander Walters wrote: > >> I think this is missing step zero. What does pathlib even offer me, the >> library developer? Why should I use it? I cannot be the only person >> who does not understand why it exists, let alone why I should be using >> it in my libraries. > > Easier path manipulations. Are path manipulations hard? I have never run into anything that I couldn't solve trivially with os.path. Is this really all this is for? > > A better user experience for those users that use pathlib. In my experience, a rounding error of users. > > Much pain on your part until the stdlib fully supports pathlib.Paths. Considering my last two counterpoints, this is likely to kill any motivation to use the library. If you want to promote pathlib use, really sell pathlib's benefits. the patches to the standard library will come later if people actually want to use it. > > -- > ~Ethan~ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ethan at stoneleaf.us Thu Mar 31 13:56:03 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 31 Mar 2016 10:56:03 -0700 Subject: [Python-ideas] Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?] In-Reply-To: <56FD6022.2030702@sdamon.com> References: <56FD5E1E.8060904@sdamon.com> <56FD5F1B.20108@stoneleaf.us> <56FD6022.2030702@sdamon.com> Message-ID: <56FD64B3.2060209@stoneleaf.us> On 03/31/2016 10:36 AM, Alexander Walters wrote: > On 3/31/2016 13:32, Ethan Furman wrote: >> On 03/31/2016 10:27 AM, Alexander Walters wrote: >> >>> I think this is missing step zero. What does pathlib even offer me, the >>> library developer? Why should I use it? I cannot be the only person >>> who does not understand why it exists, let alone why I should be using >>> it in my libraries. >> >> Easier path manipulations. > > Are path manipulations hard? I have never run into anything that I > couldn't solve trivially with os.path. Is this really all this is for? I wouldn't call the contortions I have had to go through to use os.path.* functions "trivial". YMMV. > If you want to promote pathlib use, really sell pathlib's benefits. the > patches to the standard library will come later if people actually want > to use it. Two points: 0) I personally have no interest in promoting pathlib -- I wrote my own path library. 1) Part of promoting pathlib is getting the stdlib to use it; otherwise it's a pretty high pain point doing all the string <-> Path conversions oneself. Again, YMMV. I also think creating pathlib was only half the job -- it should have been integrated into the stdlib at the same time. -- ~Ethan~ From steve at pearwood.info Thu Mar 31 13:57:41 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 1 Apr 2016 04:57:41 +1100 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions In-Reply-To: References: Message-ID: <20160331175741.GU12526@ando.pearwood.info> On Thu, Mar 31, 2016 at 09:29:36PM +0500, Mahan Marwat wrote: > I have an idea of making parenthesis optional for functions having no > parameters. i.e > > def greet: # note the missing parenthesis > print('hello') -1 I don't think that the benefit (two fewer characters to type) is worth the effort of learning the special case. Right now, the rule is simple: the def keyword ALWAYS needs parentheses after the name of the function, regardless of whether there is one argumemt, two arguments, twenty arguments, or zero arguments. Why treat zero as special? > The less awkward characters we have, the more readable our code will be > (Beautiful is better then ugly). I don't think that function declarations missing the parens are more beautiful than them with the parens. The Zen also says: Special cases aren't special enough to break the rules. Why do you think that zero-argument functions are special enough to justify breaking the rules? Just to save two characters? > Some people argued that function > definition with parenthesis seems to them natural. But actually it seems to > us natural, because we have been used to it a-lot. IMHO parenthesisless > functions definitions are natural and readable. When I was a beginner, I found that it was very helpful that I *always* needed to use parens after functions: def func(): ... # later result = func() Otherwise I would forget the parens, and get a syntax error: def func: ... or a strange and mysterious (well, mysterious to me, as a beginner) error: result = func # oops, forgot to call the func answer = result + 1 # TypeError So I think that is a positive feature that we have to write parens after function declarations, even for zero-argument functions. I think it would make Python a lesser language to make that optional: - one more special case to memorise; - loss of a good, helpful reminder for beginners that parens are needed for both function definitions and function calls. And for what gain? Saving a couple of characters. It's not worth it. > In Python we have already adopted this. i.e parenthesis are optional in > `if` statements, in class definition, in tuple etc I think you are confused about these. Except for the class definition, where it is optional for backwards compatibility with Python 2 "classic classes", in all the other cases it isn't so much that parens are optional as that parens are not part of the syntax. You can *add* parens nearly everywhere, where they just act to group expressions: > if x == 1: # parenthesis are optional here > pass if (x) == (1): # parens are just used for grouping expressions if (x == 1): # still just used for grouping expressions In both cases, it's a waste of time to group the expressions, but it's not an error to do so. > class Greet: # now we don't explicitly inherit from `object` > pass This is a good example. In my opinion, I'm not actually happy that classes will *implicitly* inherit from object. But it is necessary for backwards compatibility. > Tuple = 1, 2, 3 # parenthesis are optional here too Again, the parens are not part of the syntax for tuples, except for the zero-length tuple (), so they're only used for grouping. -- Steven From tjreedy at udel.edu Thu Mar 31 14:06:39 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 31 Mar 2016 14:06:39 -0400 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions In-Reply-To: References: Message-ID: On 3/31/2016 12:29 PM, Mahan Marwat wrote: Forwarded from Adnan Khan > I have an idea of making parenthesis optional for functions having no > parameters. i.e I presume you mean only in function definitions and not in function calls. > def greet: # note the missing parenthesis > print('hello') -1 This will lead people to think even more often than they do now that they can omit () in the call. > In Python we have already adopted this. i.e parenthesis are optional in > `if` statements, in class definition, in tuple etc Irrelevant. *Every* expression can be optionally enclosed in (). The () used to delimit the scope of operators is somewhat different from identifier(args) used to indicate a call and the scope of the argument list. > if x == 1: # parenthesis are optional here > pass They are optional because they are irrelevant. The above is the same as 'if ((((((((x==1)))))))):pass'. The extraneous ()s are deleted. However, greet() != greet(()) as the inner () is not deleted but passed as an empty tuple, which raises TypeError. -- Terry Jan Reedy From stefan at bytereef.org Thu Mar 31 14:17:04 2016 From: stefan at bytereef.org (Stefan Krah) Date: Thu, 31 Mar 2016 18:17:04 +0000 (UTC) Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions References: Message-ID: Mahan Marwat writes: > I have an idea of making parenthesis optional for functions having no parameters. i.e > > def greet: # note the missing parenthesis > ? ? print('hello') > This is an interesting idea, but it does not play well with the concept of function evaluation and higher order functions. Evaluation of functions is triggered when they're applied to something: >>> def greet(): ... print("hello") ... >>> lst = [greet, greet] >>> for f in lst: ... f() # function application ... hello hello If a function does not take any arguments, it must be eagerly evaluated when it is stored in a list. There is an exception: OCaml allows argument-less *methods*: let greeterInTheKingdomOfNouns = object method greet = print_string "hello\n" end;; val greeterInTheKingdomOfNouns : < greet : unit > = # greeterInTheKingdomOfNouns#greet;; hello - : unit = () This is different though: Here one sends the message "greet" to the object "greeterInTheKingdomOfNouns", so function application is replaced by message passing. In short, these argument-less functions are amusing but not so useful in practice. Stefan Krah From desmoulinmichel at gmail.com Thu Mar 31 15:52:33 2016 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Thu, 31 Mar 2016 21:52:33 +0200 Subject: [Python-ideas] Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?] In-Reply-To: <56FD6022.2030702@sdamon.com> References: <56FD5E1E.8060904@sdamon.com> <56FD5F1B.20108@stoneleaf.us> <56FD6022.2030702@sdamon.com> Message-ID: <56FD8001.4090008@gmail.com> Le 31/03/2016 19:36, Alexander Walters a ?crit : > > > On 3/31/2016 13:32, Ethan Furman wrote: >> On 03/31/2016 10:27 AM, Alexander Walters wrote: >> >>> I think this is missing step zero. What does pathlib even offer me, the >>> library developer? Why should I use it? I cannot be the only person >>> who does not understand why it exists, let alone why I should be using >>> it in my libraries. >> >> Easier path manipulations. > > Are path manipulations hard? I have never run into anything that I > couldn't solve trivially with os.path. Is this really all this is for? Slicing is not hard. But we like listing[a:b] better than slice(listing, a, b). Decorating is not hard. But we like @decorator better than func = decorator(func) Try/except is not hard, but we like dict.get('foo', default) better than user KeyError. Path are not hard, but I like better (p / 'foo').absolute() than os.path.realpath(os.path.join(p, 'foo')). I like not having to know if I need to import to look for something in glop, shutils, open or os just for my quick ipython session. Your program is the sum of its parts, and every time you make something a tiny bit clearer, cleaner, easier to use and read, it's a win. Python is fantastic, because it's full of small details making your program simple and easy to write, while clear to read. The FS manipulation API is showing it's age, people are using alternatives from pypi because they like it better, it's been notice and pathlib has been integrated in 3.5. Now pathlib needs improving. But it's here. > >> >> A better user experience for those users that use pathlib. > > In my experience, a rounding error of users. > >> >> Much pain on your part until the stdlib fully supports pathlib.Paths. > > Considering my last two counterpoints, this is likely to kill any > motivation to use the library. > > If you want to promote pathlib use, really sell pathlib's benefits. the > patches to the standard library will come later if people actually want > to use it. > >> >> -- >> ~Ethan~ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From srkunze at mail.de Thu Mar 31 16:41:07 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 31 Mar 2016 22:41:07 +0200 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions In-Reply-To: <56FD5EC7.7060909@sdamon.com> References: <56FD5EC7.7060909@sdamon.com> Message-ID: <56FD8B63.1000207@mail.de> On 31.03.2016 19:30, Alexander Walters wrote: > While it would be consistent, and if you suggested it in 1991 probably > should have been included... are the two extra keystrokes really a > problem? Four on my keyboard. ;-) I think the keystrokes are less important than the visual clutter one have with those parentheses. Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Mar 31 16:44:16 2016 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 01 Apr 2016 09:44:16 +1300 Subject: [Python-ideas] Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?] In-Reply-To: References: Message-ID: <56FD8C20.5050000@canterbury.ac.nz> Paul Moore wrote: > In function xxx, please support passing a pathlib.Path for > argument bar. This can be done > simply by adding a line > bar = getattr(bar, 'path', bar) This is a rather ugly piece of code to have to sprinkle all over any library that deals with pathnames. I don't get a good feeling about this approach. It seems like it will be a spreading infection that only gets worse over time. -- Greg From random832 at fastmail.com Thu Mar 31 16:56:59 2016 From: random832 at fastmail.com (Random832) Date: Thu, 31 Mar 2016 16:56:59 -0400 Subject: [Python-ideas] Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?] In-Reply-To: <56FD8C20.5050000@canterbury.ac.nz> References: <56FD8C20.5050000@canterbury.ac.nz> Message-ID: <1459457819.2388350.565131546.07B04A2F@webmail.messagingengine.com> On Thu, Mar 31, 2016, at 16:44, Greg Ewing wrote: > Paul Moore wrote: > > In function xxx, please support passing a pathlib.Path for > > argument bar. This can be done > > simply by adding a line > > bar = getattr(bar, 'path', bar) > > This is a rather ugly piece of code to have to sprinkle > all over any library that deals with pathnames. > > I don't get a good feeling about this approach. It seems > like it will be a spreading infection that only gets > worse over time. If os and io (and open builtin) supported it, I bet that 90% of other modules wouldn't need to do anything to support it. io/open might not even need to explicitly support it. Duck typing = a filename is anything that can be passed to open / some os functions. From srkunze at mail.de Thu Mar 31 16:49:47 2016 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 31 Mar 2016 22:49:47 +0200 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions In-Reply-To: References: Message-ID: <56FD8D6B.9040000@mail.de> On 31.03.2016 20:06, Terry Reedy wrote: > >> def greet: # note the missing parenthesis >> print('hello') > > -1 This will lead people to think even more often than they do now > that they can omit () in the call. Interesting that you mentioned it. Doesn't Ruby handle it this way? Let's see how this would look like in Python: def distance of point1, point2: # Pythagoras point1 = (3, 1) point2 = (1, 4) print distance of point1, point2 Hmmm. Although I like the lightness of this, it's somewhat confusing, isn't? Best, Sven -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Mar 31 17:18:25 2016 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 31 Mar 2016 14:18:25 -0700 Subject: [Python-ideas] Promoting the adoption of pathlib [Was: Working with Path objects: p-strings?] In-Reply-To: <56FD8C20.5050000@canterbury.ac.nz> References: <56FD8C20.5050000@canterbury.ac.nz> Message-ID: <56FD9421.5000200@stoneleaf.us> On 03/31/2016 01:44 PM, Greg Ewing wrote: > Paul Moore wrote: >> bar = getattr(bar, 'path', bar) > > This is a rather ugly piece of code to have to sprinkle > all over any library that deals with pathnames. > > I don't get a good feeling about this approach. It seems > like it will be a spreading infection that only gets > worse over time. Agreed. And as Random832 alluded to, if the stdlib supported pathlib.Path, the only libraries that would need to care would be those that do path manipulations without os.path.* help. -- ~Ethan~ From joshua.morton13 at gmail.com Thu Mar 31 17:24:42 2016 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Thu, 31 Mar 2016 21:24:42 +0000 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions In-Reply-To: <56FD8D6B.9040000@mail.de> References: <56FD8D6B.9040000@mail.de> Message-ID: (Let's hope I'm doing this right) ? Hmmm. Although I like the lightness of this, it's somewhat confusing, isn't? It would also lead to parse errors, or alternatively parentheses being required sometimes in the case of function calls as arguments Take as an example a function with two functions being called as the arguments: f(x(y, z), a(b, c)) with parentheses, however without you get f of x of y z a of b c in the second case its ambiguous if f is a function on 3 arguments or two. In other words, is the function f(x(y, z), a(b, c)) or f(x(y, z), a(b), c)? This is also all against the Zen. There should be one-- and preferably only one --obvious way to do it. Regards, Josh On Thu, Mar 31, 2016 at 4:57 PM Sven R. Kunze wrote: > On 31.03.2016 20:06, Terry Reedy wrote: > > > def greet: # note the missing parenthesis > print('hello') > > > -1 This will lead people to think even more often than they do now that > they can omit () in the call. > > > Interesting that you mentioned it. Doesn't Ruby handle it this way? > > Let's see how this would look like in Python: > > > def distance of point1, point2: > # Pythagoras > > point1 = (3, 1) > point2 = (1, 4) > print distance of point1, point2 > > > Hmmm. Although I like the lightness of this, it's somewhat confusing, > isn't? > > > Best, > Sven > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Mar 31 18:27:26 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 1 Apr 2016 00:27:26 +0200 Subject: [Python-ideas] The next major Python version will be Python 8 Message-ID: Hi, Python 3 becomes more and more popular and is close to a dangerous point where it can become popular that Python 2. The PSF decided that it's time to elaborate a new secret plan to ensure that Python users suffer again with a new major release breaking all their legacy code. The PSF is happy to announce that the new Python release will be Python 8! Why the version 8? It's just to be greater than Perl 6 and PHP 7, but it's also a mnemonic for PEP 8. By the way, each minor release will now multiply the version by 2. With Python 8 released in 2016 and one release every two years, we will beat Firefox 44 in 2022 (Python 64) and Windows 2003 in 2032 (Python 2048). A major release requires a major change to justify a version bump: the new killer feature is that it's no longer possible to import a module which does not respect the PEP 8. It ensures that all your code is pure. Example: $ python8 -c 'import keyword' Lib/keyword.py:16:1: E122 continuation line missing indentation or outdented Lib/keyword.py:16:1: E265 block comment should start with '# ' Lib/keyword.py:50:1: E122 continuation line missing indentation or outdented (...) ImportError: no pep8, no glory Good news: since *no* module of the current standard library of Python 3 respect the PEP 8, the standard library will be simplified to one unique module, which is new in Python 8: pep8. The standard library will move to the Python Cheeseshop (PyPI), to reply to an old and popular request. DON'T PANIC! You are still able to import your legacy code into Python 8, you just have to rename all your modules to add a "_noqa" suffix to the filename. For example, rename utils.py to utils_noqa.py. A side effect is that you have to update all imports. For example, replace "import django" with "import django_noqa". After a study of the PSF, it's a best option to split again the Python community and make sure that all users are angry. The plan is that in 10 years, at least 50% of the 77,000 packages on the Python cheeseshop will be updated to get the "_noqa" tag. After 2020, the PSF will start to sponsor trolls to harass users of the legacy Python 3 to force them to migrate to Python 8. Python 8 is a work-in-progress (it's still an alpha version), the standard library was not removed yet. Hopefully, trying to import any module of the standard library fails. Don't hesitate to propose more ideas to make Python 8 more incompatible with Python 3! Note: The change is already effective in the default branch of Python: https://hg.python.org/cpython/rev/9aedec2dbc01 Have fun, Victor From ben+python at benfinney.id.au Thu Mar 31 19:22:40 2016 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 01 Apr 2016 10:22:40 +1100 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions References: <56FD5EC7.7060909@sdamon.com> Message-ID: <8537r69qrj.fsf@benfinney.id.au> Alexander Walters writes: > are the two extra keystrokes really a problem? Who said the keystrokes are a problem? Not the original poster. Instead, the issue raised is of readability: > On 3/31/2016 12:29, Mahan Marwat wrote: > > The less awkward characters we have, the more readable our code will > > be (Beautiful is better then ugly). Some people argued that function > > definition with parenthesis seems to them natural. But actually it > > seems to us natural, because we have been used to it a-lot. IMHO > > parenthesisless functions definitions are natural and readable. No mention of keystrokes. I'm endlessly disappointed that discussions of ?too much noise in the code? are mis-interpreted as *only* about writing, not about reading. -- \ ?Always do right. This will gratify some people, and astonish | `\ the rest.? ?Mark Twain | _o__) | Ben Finney From mahanmarwat at gmail.com Thu Mar 31 20:12:56 2016 From: mahanmarwat at gmail.com (Mahan Marwat) Date: Fri, 1 Apr 2016 05:12:56 +0500 Subject: [Python-ideas] Make parenthesis optional in parameterless functions definitions In-Reply-To: References: <56FD8D6B.9040000@mail.de> Message-ID: > Alexander Walters wrote: > > While it would be consistent, and if you suggested it in 1991 probably > should have been included... are the two extra keystrokes really a problem? I born in 1994. So, no chance of suggestion in 1991. It is not about the keystrokes. Its about the visual cluttering. (and yes it can save two keystrokes too and also four on Seven R. Kuze keyboard :-) ) As I have been stated in the Subject. My idea is only about function definition, not function calling. > On Thu, Mar 31, 2016 at 09:29:36PM +0500, Mahan Marwat wrote: > > I have an idea of making parenthesis optional for functions having no > > parameters. i.e > > > > def greet: # note the missing parenthesis > > print('hello') > -1 > I don't think that the benefit (two fewer characters to type) is worth > the effort of learning the special case. Right now, the rule is simple: > the def keyword ALWAYS needs parentheses after the name of the function, > regardless of whether there is one argumemt, two arguments, twenty > arguments, or zero arguments. Why treat zero as special? For readability (Readability counts). > The less awkward characters we have, the more readable our code will be > > (Beautiful is better then ugly). > I don't think that function declarations missing the parens are more > beautiful than them with the parens. Some people argued that function definition with parenthesis seems to them natural. But actually it seems to us natural, because we have been used to it a-lot. IMHO parenthesisless functions definitions are natural and readable. The Zen also says: > Special cases aren't special enough to break the rules. > Why do you think that zero-argument functions are special enough to > justify breaking the rules? Just to save two characters? I think empty parenthesis for no apparent reason does not make any sense. When I was a beginner, I found that it was very helpful that I *always* > needed to use parens after functions: We are beginner only for once and expert for life. Between you can asked in your novice level, that why there are empty parenthesis if a function don't have parameters? -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Mar 31 20:27:22 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 1 Apr 2016 11:27:22 +1100 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions In-Reply-To: <56FD8D6B.9040000@mail.de> References: <56FD8D6B.9040000@mail.de> Message-ID: <20160401002722.GV12526@ando.pearwood.info> On Thu, Mar 31, 2016 at 10:49:47PM +0200, Sven R. Kunze wrote: > On 31.03.2016 20:06, Terry Reedy wrote: > > > >>def greet: # note the missing parenthesis > >> print('hello') > > > >-1 This will lead people to think even more often than they do now > >that they can omit () in the call. > > Interesting that you mentioned it. Doesn't Ruby handle it this way? > > Let's see how this would look like in Python: > > > def distance of point1, point2: > # Pythagoras > > point1 = (3, 1) > point2 = (1, 4) > print distance of point1, point2 I don't know that Ruby allows function calls like that. It doesn't work in Ruby 1.8, which is the most recent version I have installed: steve at orac:~$ irb irb(main):001:0> def foo(x) irb(main):002:1> return x+1 irb(main):003:1> end => nil irb(main):004:0> foo(7) => 8 irb(main):005:0> foo of 7 NoMethodError: undefined method `of' for main:Object from (irb):5 from :0 irb(main):006:0> However, Hypertalk, and other similar "XTalk" languages, do. Function calls in Hypertalk generally have a long form and a short form. The long form will be something like: total = the sum of field "expenses" while the short form is the more familiar: total = sum(field "expenses") Although Hypercard only allowed the long form if there was exactly one argument. > Hmmm. Although I like the lightness of this, it's somewhat confusing, isn't? In Hypertalk, it worked very well. But I wouldn't think it would be a good fit to Python. In a previous email, Sven also wrote: > I think the keystrokes are less important than the visual clutter > one have with those parentheses. I don't think they introduce "visual clutter" to the function. I think they make it explicit and clear that the function has no arguments. That's not clutter. When I was learning Python, I found it hard to remember when I needed parens and when I didn't, because the inconsistency between class and def confused me. I would write `class X()` or `def spam` and get a syntax error (this was back in Python 1.5). My own experience tells me strongly that the inconsistency between: class X: # must not use parens in Python 1.5 class Y(X): # must use parens and the inconsistency between class and def was harmful. If I could, I'd make parens mandatory for both. So I think that making parens optional for functions doesn't simplify the syntax so much as make the syntax *more complicated* and therefore harder to learn. And I do not agree that the empty parens are "clutter" or make the function definition harder to read. -- Steve From steve at pearwood.info Thu Mar 31 20:43:29 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 1 Apr 2016 11:43:29 +1100 Subject: [Python-ideas] Make parenthesis optional in parameterless functions definitions In-Reply-To: References: <56FD8D6B.9040000@mail.de> Message-ID: <20160401004329.GW12526@ando.pearwood.info> On Fri, Apr 01, 2016 at 05:12:56AM +0500, Mahan Marwat wrote: > > I don't think that the benefit (two fewer characters to type) is worth > > the effort of learning the special case. Right now, the rule is simple: > > the def keyword ALWAYS needs parentheses after the name of the function, > > regardless of whether there is one argumemt, two arguments, twenty > > arguments, or zero arguments. Why treat zero as special? > > For readability (Readability counts). What makes you think that removing the parens increases readability? I think it *decreases* readability, because now there is a inconstency between the zero argument case and every single other case. Now the reader looks at function definitions and every single function definition has the same pattern: def name ( parameter-list ) : (where the parameter-list might have one parameter, or ten, or zero). With your proposal the reader will *nearly always* see the consistent pattern: def name ( parameter-list ) : but very occasionally, maybe one time in a hundred functions, or a thousand, see: def name : and be surprised. Even if it is just for a millisecond, that doesn't help readability, it hurts it. > > The less awkward characters we have, the more readable our code will be > > > (Beautiful is better then ugly). > > I don't think that function declarations missing the parens are more > > beautiful than them with the parens. > > Some people argued that function definition with parenthesis seems to them > natural. But actually it seems to us natural, because we have been used to > it a-lot. IMHO parenthesisless functions definitions are natural and > readable. Yes yes, you said those exact same words before. I disagree the first time you said it, and I still disagree now. > The Zen also says: > > Special cases aren't special enough to break the rules. > > Why do you think that zero-argument functions are special enough to > > justify breaking the rules? Just to save two characters? > > I think empty parenthesis for no apparent reason does not make any sense. Of course it makes sense. The parameter list is empty. How do you get an empty list? With list delimiters with nothing inside them. Are you confused by "thelist = []"? Function parameter lists use round brackets instead of square brackets, but otherwise are the same. An empty parameter list is indicated by empty parameter list delimiters. > > When I was a beginner, I found that it was very helpful that I > > *always* needed to use parens after functions: > > We are beginner only for once and expert for life. If I claim to be an expert, but am confused by an empty parameter list, then I shouldn't be surprised if others wonder how expert I actually am. This matter boils down to a question of taste. You apparently don't like the look of "def spam()", I do. I think my experience supports the current requirement, you think that it hurts readability, I don't. Unless you can give some objective evidence that it hurts readability, you aren't going to convince me. > Between you can asked in your novice level, that why there are empty > parenthesis if a function don't have parameters? Because the list of parameters is empty. -- Steve From python-ideas at mgmiller.net Thu Mar 31 22:29:41 2016 From: python-ideas at mgmiller.net (Mike Miller) Date: Thu, 31 Mar 2016 19:29:41 -0700 Subject: [Python-ideas] Make parenthesis optional in parameterless functions definitions In-Reply-To: <20160401004329.GW12526@ando.pearwood.info> References: <56FD8D6B.9040000@mail.de> <20160401004329.GW12526@ando.pearwood.info> Message-ID: <56FDDD15.40901@mgmiller.net> Hmm, The fact that the class definition doesn't require empty parentheses works against a few of these arguments. I like Mahan's idea and don't find it confusing, though I agree it isn't a big pain point either way. -Mike On 2016-03-31 17:43, Steven D'Aprano wrote: >> >parenthesis if a function don't have parameters? > Because the list of parameters is empty. > From jsbueno at python.org.br Thu Mar 31 22:37:53 2016 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Thu, 31 Mar 2016 23:37:53 -0300 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions In-Reply-To: <20160331175741.GU12526@ando.pearwood.info> References: <20160331175741.GU12526@ando.pearwood.info> Message-ID: On 31 March 2016 at 14:57, Steven D'Aprano wrote: > On Thu, Mar 31, 2016 at 09:29:36PM +0500, Mahan Marwat wrote: > >> I have an idea of making parenthesis optional for functions having no >> parameters. i.e >> >> def greet: # note the missing parenthesis >> print('hello') > > -1 > > I don't think that the benefit (two fewer characters to type) is worth > the effort of learning the special case. Right now, the rule is simple: > the def keyword ALWAYS needs parentheses after the name of the function, > regardless of whether there is one argumemt, two arguments, twenty > arguments, or zero arguments. Why treat zero as special? Because class definitions already do so? So, if it is possible to omit parentheses when inheriting from the default object when declaring a class, not needing parenthesis for default parameterless functions would not be an exception - it would be generalizing the idea of "Python admits less clutter". For that, I'd think of this a good idea - but I don't like changing the idea syntax in such a fundamental way - so I am +0 on this thing. I think this can be an interesting discussion - but I dislike people taking a ride on this to suggest omitting parentheses on function calls as well - that is totally broken. :-) js -><- From rosuav at gmail.com Thu Mar 31 23:00:26 2016 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 1 Apr 2016 14:00:26 +1100 Subject: [Python-ideas] Fwd: Make parenthesis optional in parameterless functions definitions In-Reply-To: References: <20160331175741.GU12526@ando.pearwood.info> Message-ID: On Fri, Apr 1, 2016 at 1:37 PM, Joao S. O. Bueno wrote: > On 31 March 2016 at 14:57, Steven D'Aprano wrote: >> On Thu, Mar 31, 2016 at 09:29:36PM +0500, Mahan Marwat wrote: >> >>> I have an idea of making parenthesis optional for functions having no >>> parameters. i.e >>> >>> def greet: # note the missing parenthesis >>> print('hello') >> >> -1 >> >> I don't think that the benefit (two fewer characters to type) is worth >> the effort of learning the special case. Right now, the rule is simple: >> the def keyword ALWAYS needs parentheses after the name of the function, >> regardless of whether there is one argumemt, two arguments, twenty >> arguments, or zero arguments. Why treat zero as special? > > Because class definitions already do so? Class definitions aren't quite the same thing. A function *must* have a set of arguments, and a function call *must* have a set of parameters. But you can think about a class definition without ever thinking about inheritance; the fact that all objects are instances of 'object' doesn't mean you have to say so every time you construct a class. When I explain classes and the "is-a" relationship to my students, it's easier to pretend that classes completely stand alone than to dig into the fact that, in Python's world, your Dog (subclass of Animal) is actually closely related to ValueError (subclass of Exception). If Python (or a style guide) mandated that classes MUST declare what they're subclassing, it would want to say "(object)", not "()". No class in Python 3 can truly have an empty inheritance list. ChrisA