From larry at hastings.org Sat Feb 1 04:23:52 2014 From: larry at hastings.org (Larry Hastings) Date: Fri, 31 Jan 2014 19:23:52 -0800 Subject: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4) In-Reply-To: <52E88F2D.9030500@stoneleaf.us> References: <52E65110.7070904@hastings.org> <20140127125651.GE3915@ando> <52E65B5D.6060502@hastings.org> <52E74901.1070008@hastings.org> <20140128123714.GI3915@ando> <52E7BC2F.2080207@stoneleaf.us> <52E86C8A.8090706@hastings.org> <52E88F2D.9030500@stoneleaf.us> Message-ID: <52EC68C8.7090301@hastings.org> On 01/28/2014 09:18 PM, Ethan Furman wrote: > On 01/28/2014 06:50 PM, Larry Hastings wrote: >> I think the "times behaves differently when passed by name versus >> passed by position" behavior falls exactly into this >> category, and its advice on how to handle it is sound. > > I don't agree with this. This is a bug. Somebody going through (for > example) a code review and making minor changes so the code is more > readable shouldn't have to be afraid that [inserting | removing] the > keyword in the function call is going to *drastically* [1] change the > behavior. I understand the need for a cycle of deprecation [2], but > not fixing it in 3.5 is folly. It's a bug. But it's also a longstanding bug, having been a part of Python since 2.7. Python is the language that cares about backwards-compatibility--bugs and all. If your code runs on version X.Y, it should run without modification on version X.(Y+Z) where Z is a positive integer. Therefore it would be inappropriate to remove the "times=-1 when passed by keyword repeats indefinitely" behavior without at /least/ a full deprecation cycle. Personally I'd prefer to leave the behavior in, undocumented and deprecated, until Python 4.0. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Feb 1 07:26:23 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 1 Feb 2014 17:26:23 +1100 Subject: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4) In-Reply-To: <52EC68C8.7090301@hastings.org> References: <52E65110.7070904@hastings.org> <20140127125651.GE3915@ando> <52E65B5D.6060502@hastings.org> <52E74901.1070008@hastings.org> <20140128123714.GI3915@ando> <52E7BC2F.2080207@stoneleaf.us> <52E86C8A.8090706@hastings.org> <52E88F2D.9030500@stoneleaf.us> <52EC68C8.7090301@hastings.org> Message-ID: <20140201062622.GP3799@ando> On Fri, Jan 31, 2014 at 07:23:52PM -0800, Larry Hastings wrote: > Python is the language that cares about backwards-compatibility--bugs > and all. If your code runs on version X.Y, it should run without > modification on version X.(Y+Z) where Z is a positive integer. > > Therefore it would be inappropriate to remove the "times=-1 when passed > by keyword repeats indefinitely" behavior without at /least/ a full > deprecation cycle. That is a reasonable and responsible position to take. > Personally I'd prefer to leave the behavior in, > undocumented and deprecated, until Python 4.0. It's one thing to avoid removing things which do no harm, but this is a bug which does harm. Anyone who refactors repeat(x, count) to repeat(x, times=count) (or vice versa) is in for a nasty surprise if count ever becomes negative. If anyone out there relies on this bug, they can get the same behaviour easily, and end up with better code: # Before. repeat(x, times=-1) # Magic to make x repeat forever. # After. repeat(x) # Non-magic fully documented way of getting x to repeat forever. Given support for times=None at the same time, then it's easy to support the use-case of "sometimes I want an infinite repeat, sometimes I don't, but I won't know until runtime": # Before. values = (100, -1) # Repeat 100 times, or forever. repeat(x, times=random.choice(values)) # After. values = (100, None) # Repeat 100 times, or forever. repeat(x, times=random.choice(values)) I can see that delaying for a depreciation cycle is the responsible thing to do. I don't think delaying fixing this until some hypothetical distant Python 4000 is a good idea. -- Steven From ethan at stoneleaf.us Sat Feb 1 07:04:43 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 31 Jan 2014 22:04:43 -0800 Subject: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4) In-Reply-To: <52EC68C8.7090301@hastings.org> References: <52E65110.7070904@hastings.org> <20140127125651.GE3915@ando> <52E65B5D.6060502@hastings.org> <52E74901.1070008@hastings.org> <20140128123714.GI3915@ando> <52E7BC2F.2080207@stoneleaf.us> <52E86C8A.8090706@hastings.org> <52E88F2D.9030500@stoneleaf.us> <52EC68C8.7090301@hastings.org> Message-ID: <52EC8E7B.7020600@stoneleaf.us> On 01/31/2014 07:23 PM, Larry Hastings wrote: > On 01/28/2014 09:18 PM, Ethan Furman wrote: >> On 01/28/2014 06:50 PM, Larry Hastings wrote: >>> I think the "times behaves differently when passed by name versus passed by position" behavior falls exactly into this >>> category, and its advice on how to handle it is sound. >> >> I don't agree with this. This is a bug. Somebody going through (for example) a code review and making minor changes >> so the code is more readable shouldn't have to be afraid that [inserting | removing] the keyword in the function call >> is going to *drastically* [1] change the behavior. I understand the need for a cycle of deprecation [2], but not >> fixing it in 3.5 is folly. > > It's a bug. But it's also a longstanding bug, having been a part of Python since 2.7. > > Python is the language that cares about backwards-compatibility--bugs and all. If your code runs on version X.Y, it > should run without modification on version X.(Y+Z) where Z is a positive integer. So we only fix bugs that don't work at all? By which I mean, if the interpreter doesn't crash, we don't fix it? > Therefore it would be inappropriate to remove the "times=-1 when passed by keyword repeats indefinitely" behavior > without at /least/ a full deprecation cycle. Personally I'd prefer to leave the behavior in, undocumented and > deprecated, until Python 4.0. Well, at least we are agreed on a deprecation cycle. :) -- ~Ethan~ From ncoghlan at gmail.com Sat Feb 1 13:20:24 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 1 Feb 2014 22:20:24 +1000 Subject: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4) In-Reply-To: <52EC8E7B.7020600@stoneleaf.us> References: <52E65110.7070904@hastings.org> <20140127125651.GE3915@ando> <52E65B5D.6060502@hastings.org> <52E74901.1070008@hastings.org> <20140128123714.GI3915@ando> <52E7BC2F.2080207@stoneleaf.us> <52E86C8A.8090706@hastings.org> <52E88F2D.9030500@stoneleaf.us> <52EC68C8.7090301@hastings.org> <52EC8E7B.7020600@stoneleaf.us> Message-ID: On 1 February 2014 16:04, Ethan Furman wrote: > On 01/31/2014 07:23 PM, Larry Hastings wrote: >> Python is the language that cares about backwards-compatibility--bugs and >> all. If your code runs on version X.Y, it >> should run without modification on version X.(Y+Z) where Z is a positive >> integer. > > > So we only fix bugs that don't work at all? By which I mean, if the > interpreter doesn't crash, we don't fix it? No, we make a judgment call based on the severity and likelihood of encountering the bug, the consequences of encountering it, and the difficulty of working around it after you *do* encounter it. In this case: * Likelihood: low (using keyword arguments with simple APIs like this is not a common idiom, and you have to specifically pass -1 to trigger misbehaviour) * Consequence: application hang. Minimal chance of silent data corruption, high chance of being caught in testing if it's going to be encountered at all. Effectively equivalent impact arises when passing large integer values. * Security impact: negligible. If you're passing untrusted input to the second argument of repeat() at all, you're already exposed, independent of this bug. * Workaround: don't use keyword arguments or else prevalidate the input (the latter can also handle the large integer problem) * Potential backwards compatibility issue?: Yes, as users could be relying on this as a data driven trigger for infinite repetition and the most convenient currently available alternative spelling is rather awkward (involving *args). * Requires a new feature to fix properly?: Yes, as "None" should be added as a replacement, less error prone, data driven trigger for infinite repetition for deprecating or removing the current behaviour. Assessment: * this is a likely rare, high impact, easy to detect, easy to workaround failure where fixing it involves both adding a new feature and potentially breaking currently working code Conclusion: * add the new, supported feature that provides equivalent functionality (accepting None) in 3.5 * deprecate the problematic behaviour in 3.5 and then start treating it as equivalent to the positional argument handling in 3.6+ It's a complex balance between wanting to fix things that are broken in the interpreter and standard library and not breaking currently working end user code. As general rules of thumb: - for maintenance releases and a new feature release approaching release candidate status, err on the side of "preserving backwards compatibility is paramount" and "no new user visible features allowed" - for a new feature release after feature freeze, "no new user visible features allowed" still applies, but there is still scope to make the case for new deprecations and potentially even arguable backwards compatibility breaks that involve adding a note to the porting guide in the "What's New" document - for a new feature release before feature freeze, new features are allowed, as are new deprecations and notes in the porting guide - security issues that are deemed to be the interpreter's responsibility to resolve can trump all of this and lead to the inclusion of new features in maintenance releases. However, even then the new behaviour is likely to be opt-in in the old releases and opt-out in the next feature release (for example, hash randomisation). The "Porting to Python X.Y" guides can be a useful source of examples of previous changes that have been judged as posing too great a risk of breaking end user code to implement in a maintenance release, even when they're to resolve bugs. This is the one for 3.4: http://docs.python.org/dev/whatsnew/3.4.html#porting-to-python-3-4 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From martin at v.loewis.de Sat Feb 1 16:54:32 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 01 Feb 2014 16:54:32 +0100 Subject: [Python-Dev] Add PyType_GetSlot In-Reply-To: <52E86B8F.2060500@hastings.org> References: <52E76A09.40700@v.loewis.de> <52E86B8F.2060500@hastings.org> Message-ID: <52ED18B8.5000307@v.loewis.de> Am 29.01.14 03:46, schrieb Larry Hastings: > So this would be a new public ABI function? Correct. > Would it be 100% new code, or would you need to refactor code internally > to achieve it? See the patch - it's new code. > Also, just curious: what is typeslots.h used for? I tried searching for > a couple of those macros, and their only appearance in trunk was their > definition. It's meant for use by extension modules. See xxlimited.c:Xxo_Type_slots for an application. Regards, Martin From ethan at stoneleaf.us Sat Feb 1 18:33:10 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 01 Feb 2014 09:33:10 -0800 Subject: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4) In-Reply-To: References: <52E65110.7070904@hastings.org> <20140127125651.GE3915@ando> <52E65B5D.6060502@hastings.org> <52E74901.1070008@hastings.org> <20140128123714.GI3915@ando> <52E7BC2F.2080207@stoneleaf.us> <52E86C8A.8090706@hastings.org> <52E88F2D.9030500@stoneleaf.us> <52EC68C8.7090301@hastings.org> <52EC8E7B.7020600@stoneleaf.us> Message-ID: <52ED2FD6.5010006@stoneleaf.us> On 02/01/2014 04:20 AM, Nick Coghlan wrote: > > No, we make a judgment call based on the severity and likelihood of > encountering the bug, the consequences of encountering it, and the > difficulty of working around it after you *do* encounter it. Thanks for the explanation. -- ~Ethan~ From ethan at stoneleaf.us Sat Feb 1 18:36:50 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 01 Feb 2014 09:36:50 -0800 Subject: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4) In-Reply-To: References: <52E65110.7070904@hastings.org> <20140127125651.GE3915@ando> <52E65B5D.6060502@hastings.org> <52E74901.1070008@hastings.org> <20140128123714.GI3915@ando> <52E7BC2F.2080207@stoneleaf.us> <52E86C8A.8090706@hastings.org> <52E88F2D.9030500@stoneleaf.us> <52EC68C8.7090301@hastings.org> <52EC8E7B.7020600@stoneleaf.us> Message-ID: <52ED30B2.5050908@stoneleaf.us> On 02/01/2014 04:20 AM, Nick Coghlan wrote: [snip very nice summary] So you agree that the bug should be fixed. So do I. My disagreement with Larry is that he would leave the bug in until Py4k. -- ~Ethan~ From brett at python.org Sat Feb 1 19:20:48 2014 From: brett at python.org (Brett Cannon) Date: Sat, 1 Feb 2014 13:20:48 -0500 Subject: [Python-Dev] [Python-checkins] cpython: whatsnew: move of reload, update new windows-only ssl functions entry. In-Reply-To: <3fGj0k1zTdz7Llp@mail.python.org> References: <3fGj0k1zTdz7Llp@mail.python.org> Message-ID: On Sat, Feb 1, 2014 at 12:27 PM, r.david.murray wrote: > http://hg.python.org/cpython/rev/b3f034f5000f > changeset: 88884:b3f034f5000f > parent: 88882:19d81cc213d7 > user: R David Murray > date: Sat Feb 01 12:27:07 2014 -0500 > summary: > whatsnew: move of reload, update new windows-only ssl functions entry. > > files: > Doc/whatsnew/3.4.rst | 10 ++++++++-- > 1 files changed, 8 insertions(+), 2 deletions(-) > > > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > --- a/Doc/whatsnew/3.4.rst > +++ b/Doc/whatsnew/3.4.rst > @@ -765,6 +765,10 @@ > it will normally be desirable to override the default implementation > for performance reasons. (Contributed by Brett Cannon in :issue:`18072`.) > > +The :func:`~importlib.reload` function has been moved from :mod:`imp` > +to :mod:`importlib`. The :func:`mod.reload` name is retained for > +backward compatibility, but is deprecated. > + > That wording seems confusing to me. It makes it seem like importlib.reload is deprecated when in fact it's the imp module itself. -Brett > > inspect > ------- > @@ -1062,8 +1066,10 @@ > list of the loaded ``CA`` certificates. (Contributed by Christian Heimes > in > and :issue:`18147`.) > > -Add :func:`ssl.enum_cert_store` to retrieve certificates and CRL from > Windows' > -cert store. (Contributed by Christian Heimes in :issue:`17134`.) > +Two new windows-only functions, :func:`~ssl.enum_certificates` and > +:func:`~ssl.enum_crls` provide the ability to retrieve certificates, > +certificate information, and CRLs from the Windows cert store. > (Contributed > +by Christian Heimes in :issue:`17134`.) > > Support for server-side SNI using the new > :meth:`ssl.SSLContext.set_servername_callback` method. > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Feb 1 22:50:10 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 2 Feb 2014 08:50:10 +1100 Subject: [Python-Dev] Negative times behaviour in itertools.repeat for Python maintenance releases (2.7, 3.3 and maybe 3.4) In-Reply-To: References: <52E65B5D.6060502@hastings.org> <52E74901.1070008@hastings.org> <20140128123714.GI3915@ando> <52E7BC2F.2080207@stoneleaf.us> <52E86C8A.8090706@hastings.org> <52E88F2D.9030500@stoneleaf.us> <52EC68C8.7090301@hastings.org> <52EC8E7B.7020600@stoneleaf.us> Message-ID: <20140201215010.GQ3799@ando> On Sat, Feb 01, 2014 at 10:20:24PM +1000, Nick Coghlan wrote: > On 1 February 2014 16:04, Ethan Furman wrote: > > So we only fix bugs that don't work at all? By which I mean, if the > > interpreter doesn't crash, we don't fix it? > > No, we make a judgment call based on the severity and likelihood of > encountering the bug, the consequences of encountering it, and the > difficulty of working around it after you *do* encounter it. Nice summary Nick, however there is one slight error: > In this case: > > * Likelihood: low (using keyword arguments with simple APIs like this > is not a common idiom, and you have to specifically pass -1 to trigger > misbehaviour) It's not just -1, its any negative value: py> from itertools import repeat py> it = repeat('a', -17) # Correct. py> next(it) Traceback (most recent call last): File "", line 1, in StopIteration py> it = repeat('a', times=-17) # Bug, repeats forever. py> next(it) 'a' [...] > Conclusion: > > * add the new, supported feature that provides equivalent > functionality (accepting None) in 3.5 > > * deprecate the problematic behaviour in 3.5 and then start treating > it as equivalent to the positional argument handling in 3.6+ Seems reasonable to me. Thanks again for the nice summary. -- Steven From steve at pearwood.info Sun Feb 2 02:06:42 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 2 Feb 2014 12:06:42 +1100 Subject: [Python-Dev] Guidance regarding what counts as breaking backwards compatibility Message-ID: <20140202010642.GR3799@ando> Hi all, Over on the Python-ideas list, there's a thread about the new statistics module, and as the author of that module, I'm looking for a bit of guidance regarding backwards compatibility. Specifically two issues: (1) With numeric code, what happens if the module become more[1] accurate in the future? Does that count as breaking backwards compatibility? E.g. Currently I use a particular algorithm for calculating variance. Suppose that for a particular data set, this algorithm is accurate to (say) seven decimal places: # Python 3.4 variance(some_data) == 1.23456700001 Later, I find a better algorithm, which improves the accuracy of the result: # Python 3.5 or 3.6 variance(some_data) == 1.23456789001 Would this count as breaking backwards compatibility? If so, how should I handle this? I don't claim that the current implementation of the statistics module is optimal, as far as precision and accuracy is concerned. It may improve in the future. Or would that count as a bug-fix? "Variance function was inaccurate, now less wrong", perhaps. I suppose the math module has the same issue, except that it just wraps the C libraries, which are mature and stable and unlikely to change. The random module has a similar issue: http://docs.python.org/3/library/random.html#notes-on-reproducibility (2) Mappings[2] are iterable. That means that functions which expect sequences or iterators may also operate on mappings by accident. For example, sum({1: 100, 2: 200}) returns 3. If one wanted to reserve the opportunity to handle mappings specifically in the future, without being locked in by backwards-compatibility, how should one handle it? a) document that behaviour with mappings is unsupported and may change in the future; b) raise a warning when passed a mapping, but still iterate over it; c) raise an exception and refuse to iterate over the mapping; d) something else? Question (2) is of course a specific example of a more general question, to what degree is the library author responsible for keeping backwards compatibility under circumstances which are not part of the intended API, but just work by accident? [1] Or, for the sake of the argument, less accurate. [2] And sets. -- Steven From tjreedy at udel.edu Sun Feb 2 03:11:22 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 01 Feb 2014 21:11:22 -0500 Subject: [Python-Dev] Guidance regarding what counts as breaking backwards compatibility In-Reply-To: <20140202010642.GR3799@ando> References: <20140202010642.GR3799@ando> Message-ID: On 2/1/2014 8:06 PM, Steven D'Aprano wrote: > Hi all, > > Over on the Python-ideas list, there's a thread about the new statistics > module, and as the author of that module, I'm looking for a bit of > guidance regarding backwards compatibility. Specifically two issues: > > > (1) With numeric code, what happens if the module become more[1] > accurate in the future? Does that count as breaking backwards > compatibility? > > E.g. Currently I use a particular algorithm for calculating variance. > Suppose that for a particular data set, this algorithm is accurate to > (say) seven decimal places: > > # Python 3.4 > variance(some_data) == 1.23456700001 > > Later, I find a better algorithm, which improves the accuracy of the > result: > > # Python 3.5 or 3.6 > variance(some_data) == 1.23456789001 > > > Would this count as breaking backwards compatibility? If so, how should > I handle this? I don't claim that the current implementation of the > statistics module is optimal, as far as precision and accuracy is > concerned. It may improve in the future. > > Or would that count as a bug-fix? "Variance function was inaccurate, now > less wrong", perhaps. That is my inclination. > I suppose the math module has the same issue, except that it just wraps > the C libraries, which are mature and stable and unlikely to change. Because C libraries differ, math results differ even in the same version, so they can certainly change (hopefully improve) in future versions. I think the better analogy is cmath, which I believe is more than just a wrapper. > The random module has a similar issue: > > http://docs.python.org/3/library/random.html#notes-on-reproducibility > > > (2) Mappings[2] are iterable. That means that functions which expect > sequences or iterators may also operate on mappings by accident. I think 'accident' is the key. (Working with sets is not an accident.) Anyone who really wants the mean of keys should be explicit: mean(d.keys()) > example, sum({1: 100, 2: 200}) returns 3. If one wanted to reserve the > opportunity to handle mappings specifically in the future, without being > locked in by backwards-compatibility, how should one handle it? > > a) document that behaviour with mappings is unsupported and may > change in the future; I think the doc should in any case specify the proper domain. In this case, I think it should exclude mappings: 'non-empty non-mapping iterable of numbers', or 'an iterable of numbers that is neither empty nor a mapping'. That makes the behavior at best undefined and subject to change. There should also be a caveat about mixing types, especially Decimals, if not one already. Perhaps rewrite the above as 'an iterable that is neither empty nor a mapping of numbers that are mutually summable'. > b) raise a warning when passed a mapping, but still iterate over it; > > c) raise an exception and refuse to iterate over the mapping; This, if possible. An empty iterable will raise at '/ 0'. Most anything that is not an iterable of number will eventually raise at '/ n' Testing both that an exception is raised and that it is one we want is why why unittest has assertRaises. > Question (2) is of course a specific example of a more general > question, to what degree is the library author responsible for keeping > backwards compatibility under circumstances which are not part of the > intended API, but just work by accident? > [1] Or, for the sake of the argument, less accurate. > > [2] And sets. -- Terry Jan Reedy From ncoghlan at gmail.com Sun Feb 2 03:14:28 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 2 Feb 2014 12:14:28 +1000 Subject: [Python-Dev] Guidance regarding what counts as breaking backwards compatibility In-Reply-To: <20140202010642.GR3799@ando> References: <20140202010642.GR3799@ando> Message-ID: On 2 February 2014 11:06, Steven D'Aprano wrote: > Hi all, > > Over on the Python-ideas list, there's a thread about the new statistics > module, and as the author of that module, I'm looking for a bit of > guidance regarding backwards compatibility. Specifically two issues: > > > (1) With numeric code, what happens if the module become more[1] > accurate in the future? Does that count as breaking backwards > compatibility? > > E.g. Currently I use a particular algorithm for calculating variance. > Suppose that for a particular data set, this algorithm is accurate to > (say) seven decimal places: > > # Python 3.4 > variance(some_data) == 1.23456700001 > > Later, I find a better algorithm, which improves the accuracy of the > result: > > # Python 3.5 or 3.6 > variance(some_data) == 1.23456789001 > > > Would this count as breaking backwards compatibility? If so, how should > I handle this? I don't claim that the current implementation of the > statistics module is optimal, as far as precision and accuracy is > concerned. It may improve in the future. For this kind of case, we tend to cover it in the "Porting to Python X.Y" section of the What's New guide. User code *shouldn't* care about this kind of change, but it *might*, so we split the difference and say "It's OK in a feature release, but not in a maintenance release". There have been multiple changes along these lines in our floating handling as Tim Peters, Mark Dickinson et al have made various improvements to reduce platform dependent behaviour (especially around overflow handling, numeric precision, infinity and NaN handling, etc). However, we also sometimes have module specific disclaimers - the decimal module, for example, has an explicit caveat that updates to the General Decimal Arithmetic Specification will be treated as bug fixes, even if they would normally not be allowed in maintenance releases. For a non-math related example, a comment from Michael Foord at the PyCon US 2013 sprints made me realise that the implementation of setting the __wrapped__ attribute in functools was just flat out broken - when applied multiple times it was supposed to create a chain of references that eventually terminated in a callable without the attribute set, but due to the bug every layer actually referred directly to the innermost callable (the one without the attribute set). Unfortunately, the docs I wrote for it were also ambiguous, so a lot of folks (including Michael) assumed it was working as intended. I have fixed the bug in 3.4, but there *is* a chance it will break introspection code that assumed the old behaviour was intentional and doesn't correctly unravel __wrapped__ chains. > Or would that count as a bug-fix? "Variance function was inaccurate, now > less wrong", perhaps. > > I suppose the math module has the same issue, except that it just wraps > the C libraries, which are mature and stable and unlikely to change. They may look that way *now*, but that's only after Tim, Mark et al did a lot of work on avoiding platform specific issues and inconsistencies > The random module has a similar issue: > > http://docs.python.org/3/library/random.html#notes-on-reproducibility I think a disclaimer in the statistics module similar to the ones in the math module and this one in the random module would be appropriate - one of the key purposes of the library/language reference is to let us distinguish between "guaranteed behaviour user code can rely on" and "implementation details that user code should not assume will remain unchanged forever". In this case, it would likely be appropriate to point out that the algorithms used internally may change over time, thus potentially changing the error bounds in the module output. > (2) Mappings[2] are iterable. That means that functions which expect > sequences or iterators may also operate on mappings by accident. For > example, sum({1: 100, 2: 200}) returns 3. If one wanted to reserve the > opportunity to handle mappings specifically in the future, without being > locked in by backwards-compatibility, how should one handle it? > > a) document that behaviour with mappings is unsupported and may > change in the future; > > b) raise a warning when passed a mapping, but still iterate over it; > > c) raise an exception and refuse to iterate over the mapping; > > d) something else? > > > Question (2) is of course a specific example of a more general > question, to what degree is the library author responsible for keeping > backwards compatibility under circumstances which are not part of the > intended API, but just work by accident? In this particular case, I consider having a single API treat mappings differently from other iterables is a surprising anti-pattern and providing a separate API specifically for mappings is clearer (cf format() vs format_map()). However, if you want to preserve maximum flexibility, the best near term option is typically c (just disallow the input you haven't decided how to handle yet entirely), but either a or b would also be an acceptable way of achieving the same end (they're just less user friendly, since they let people do something that you're already considering changing in the future). In the case where you allow an API to escape into the wild without even a documented caveat that it isn't covered by the normal standard library backwards compatibility guarantees, then you're pretty much stuck. This is why you tend to see newer stdlib APIs often exposing functions that return private object types, rather than exposing the object type itself: exposed a function just promises a callable() API, while exposing a class directly promises a lot more in terms of supporting inheritance, isinstance(), issubclass(), etc, which can make future evolution of that API substantially more difficult. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sun Feb 2 13:29:21 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 2 Feb 2014 12:29:21 +0000 Subject: [Python-Dev] Guidance regarding what counts as breaking backwards compatibility In-Reply-To: References: <20140202010642.GR3799@ando> Message-ID: On 2 February 2014 02:11, Terry Reedy wrote: >> example, sum({1: 100, 2: 200}) returns 3. If one wanted to reserve the >> opportunity to handle mappings specifically in the future, without being >> locked in by backwards-compatibility, how should one handle it? >> >> a) document that behaviour with mappings is unsupported and may >> change in the future; > > > I think the doc should in any case specify the proper domain. In this case, > I think it should exclude mappings: 'non-empty non-mapping iterable of > numbers', or 'an iterable of numbers that is neither empty nor a mapping'. > That makes the behavior at best undefined and subject to change. There > should also be a caveat about mixing types, especially Decimals, if not one > already. Perhaps rewrite the above as 'an iterable that is neither empty nor > a mapping of numbers that are mutually summable'. Generally, my view would be that users should not rely on undocumented behaviour. But as documentation is sometimes less than 100% precise, it's worth in a case like this, documenting that a particular behaviour is not defined (yet). Then, picking and implementing the desired behaviour will be a new feature and hence totally acceptable. Changing the way mappings are treated in a bugfix release (as opposed to a feature release) is unlikely to be acceptable no matter how you do it, as there's no way you can reasonably deliberately implement one behaviour now and claim it's a bug later :-) Paul From rdmurray at bitdance.com Sun Feb 2 19:08:48 2014 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 02 Feb 2014 13:08:48 -0500 Subject: [Python-Dev] [Python-checkins] cpython: whatsnew: move of reload, update new windows-only ssl functions entry. In-Reply-To: References: <3fGj0k1zTdz7Llp@mail.python.org> Message-ID: <20140202180848.C8A0C250499@webabinitio.net> On Sat, 01 Feb 2014 13:20:48 -0500, Brett Cannon wrote: > On Sat, Feb 1, 2014 at 12:27 PM, r.david.murray > wrote: > > > http://hg.python.org/cpython/rev/b3f034f5000f > > changeset: 88884:b3f034f5000f > > parent: 88882:19d81cc213d7 > > user: R David Murray > > date: Sat Feb 01 12:27:07 2014 -0500 > > summary: > > whatsnew: move of reload, update new windows-only ssl functions entry. > > > > files: > > Doc/whatsnew/3.4.rst | 10 ++++++++-- > > 1 files changed, 8 insertions(+), 2 deletions(-) > > > > > > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > > --- a/Doc/whatsnew/3.4.rst > > +++ b/Doc/whatsnew/3.4.rst > > @@ -765,6 +765,10 @@ > > it will normally be desirable to override the default implementation > > for performance reasons. (Contributed by Brett Cannon in :issue:`18072`.) > > > > +The :func:`~importlib.reload` function has been moved from :mod:`imp` > > +to :mod:`importlib`. The :func:`mod.reload` name is retained for > > +backward compatibility, but is deprecated. > > + > > > > That wording seems confusing to me. It makes it seem like importlib.reload > is deprecated when in fact it's the imp module itself. I committed that thinking I'd finished working on that entry, but I hadn't. The updated text reads: The reload() function has been moved from imp to importlib as part of the imp module deprecation. Is that clearer? --David From brett at python.org Sun Feb 2 21:10:32 2014 From: brett at python.org (Brett Cannon) Date: Sun, 2 Feb 2014 15:10:32 -0500 Subject: [Python-Dev] [Python-checkins] cpython: whatsnew: move of reload, update new windows-only ssl functions entry. In-Reply-To: <20140202180848.C8A0C250499@webabinitio.net> References: <3fGj0k1zTdz7Llp@mail.python.org> <20140202180848.C8A0C250499@webabinitio.net> Message-ID: On Feb 2, 2014 1:08 PM, "R. David Murray" wrote: > > On Sat, 01 Feb 2014 13:20:48 -0500, Brett Cannon wrote: > > On Sat, Feb 1, 2014 at 12:27 PM, r.david.murray > > wrote: > > > > > http://hg.python.org/cpython/rev/b3f034f5000f > > > changeset: 88884:b3f034f5000f > > > parent: 88882:19d81cc213d7 > > > user: R David Murray > > > date: Sat Feb 01 12:27:07 2014 -0500 > > > summary: > > > whatsnew: move of reload, update new windows-only ssl functions entry. > > > > > > files: > > > Doc/whatsnew/3.4.rst | 10 ++++++++-- > > > 1 files changed, 8 insertions(+), 2 deletions(-) > > > > > > > > > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > > > --- a/Doc/whatsnew/3.4.rst > > > +++ b/Doc/whatsnew/3.4.rst > > > @@ -765,6 +765,10 @@ > > > it will normally be desirable to override the default implementation > > > for performance reasons. (Contributed by Brett Cannon in :issue:`18072`.) > > > > > > +The :func:`~importlib.reload` function has been moved from :mod:`imp` > > > +to :mod:`importlib`. The :func:`mod.reload` name is retained for > > > +backward compatibility, but is deprecated. > > > + > > > > > > > That wording seems confusing to me. It makes it seem like importlib.reload > > is deprecated when in fact it's the imp module itself. > > I committed that thinking I'd finished working on that entry, but I > hadn't. The updated text reads: > > The reload() function has been moved from imp to importlib as part > of the imp module deprecation. > > Is that clearer? Yep! And thanks for catching the missing entry. -Brett > > --David -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sun Feb 2 23:38:19 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 2 Feb 2014 23:38:19 +0100 Subject: [Python-Dev] News from asyncio In-Reply-To: References: Message-ID: Latest asyncio update: it has a new asyncio.subprocess submodule which provides a high-level API to control subprocesses, similar to subprocess.Popen but using asyncio event loop (and so is asynchronous). It solves the following old and tricky issue: http://bugs.python.org/issue12187 "subprocess.wait() with a timeout uses polling on POSIX" "yield from asyncio.wait_for(proc.wait(), timeout)" is no more a busy loop ;-) Victor From cs at zip.com.au Sun Feb 2 23:52:46 2014 From: cs at zip.com.au (Cameron Simpson) Date: Mon, 3 Feb 2014 09:52:46 +1100 Subject: [Python-Dev] News from asyncio In-Reply-To: References: Message-ID: <20140202225246.GA56490@cskk.homeip.net> On 27Jan2014 09:15, Devin Jeanpierre wrote: > On Mon, Jan 27, 2014 at 5:21 AM, Victor Stinner > wrote: > > - asyncio.IncompleReadError.expected is the total expected size, not > > the remaining size > > Why not be consistent with the meaning of > http.client.IncompleteRead.expected? The current meaning can be > recovered via len(e.partial) + e.expected. Which you could also expose as a property name ".requested", since this value must have been thought useful or Victor wouldn't have defined .expected to be it:-) Seems reasonable, for all that the above computation is trivial. Just a thought, -- Cameron Simpson Artificial intelligence won't make a micrometer out of a monkeywrench. - Rick Gordon From victor.stinner at gmail.com Mon Feb 3 11:22:22 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 3 Feb 2014 11:22:22 +0100 Subject: [Python-Dev] The online documentation is no more updated? Message-ID: Hi, I modified the Python documentaton (asyncio module) 6 days ago, and my changes are not online yet: http://docs.python.org/dev/library/asyncio-eventloop.html#running-subprocesses Is it a problem with the server generating the documentation? By the way, would it be possible to regenerate the documentation at each commit, or maybe every hours, instead of having to wait 24h to see the modified documentation online? Thanks. Victor From larry at hastings.org Mon Feb 3 15:43:31 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 03 Feb 2014 06:43:31 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go Message-ID: <52EFAB13.4020405@hastings.org> A quick summary of the context: currently in CPython 3.4, a builtin function can publish its "signature" as a specially encoded line at the top of its docstring. CPython internally detects this line inside PyCFunctionObject.__doc__ and skips past it, and there's a new getter at PyCFunctionObject.__text_signature__ that returns just this line. As an example, the signature for os.stat looks like this: sig=($module, path, *, dir_fd=None, follow_symlinks=True) The convention is, if you have this signature, you shouldn't have your docstring start with a handwritten signature like 3.3 and before. help() on a callable displays the signature automatically if it can, so if you *also* had a handwritten signature, help() would show two signatures. That would look dumb. ----- So here's the problem. Let's say you want to write an extension that will work with Python 3.3 and 3.4, using the stable ABI. If you don't add this line, then in 3.4 you won't have introspection information, drat. But if you *do* add this line, your docstring will look mildly stupid in 3.3, because it'll have this unsightly "sig=(" line at the top. And it *won't* have a nice handwritten docstring. (And if you added both a sig= signature *and* a handwritten signature, in 3.4 it would display both. That would also look dumb.) I can't figure out any way to salvage this "first line of the docstring" approach. So I think we have to abandon it, and do this the hard way: extend the PyMethodDef structure. I propose three different variations. I prefer B, but I'm guessing Guido would prefer the YAGNI approach, which is A: A: We create a PyMethodDefEx structure with an extra field: "const char *signature". We add a new METH_SIGNATURE (maybe just METH_SIG?) flag to the flags, indicating that this is an extended structure. When iterating over the PyMethodDefs, we know how far to advance the pointer based on this flag. B: Same as A, but we add three unused pointers (void *reserved1 etc) to PyMethodDefEx to give us some room to grow. C: Same as A, but we add two fields to PyMethodDefEx. The second new field identifies the "version" of the structure, telling us its size somehow. Like the lStructSize field of the OPENFILENAME structure in Win32. I suspect YAGNI. ----- But that only fixes part of the problem. Our theoretical extension that wants to be binary-compatible with 3.3 and 3.4 still has a problem: how can they support signatures? They can't give PyMethodDefEx structures to 3.3, it will blow up. But if they don't use PyMethodDefEx, they can't have signatures. Solution: we write a function (which users would have to copy into their extension) that gives a PyMethodDefEx array to 3.4+, but converts it into a PyMethodDef array for 3.3. The tricky part there: what do we do about the docstring? The convention for builtins is to have the first line(s) contain a handwritten signature. But you *don't* want that if you provide a signature, because help() will read that signature and automatically render this first line for you. I can suggest four options here, and of these I like P best: M: Don't do anything. Docstrings with real signature information and a handwritten signature in the docstring will show two signatures in 3.4+, docstrings without any handwritten signature won't display their signature in help in 3.3. (Best practice for modules compiled for 3.4+ is probably: skip the handwritten signature. Users would have to do without in 3.3.) N: Leave the handwritten signature in the docstring, then when registering for 3.4+ add a second flag called METH_33_COMPAT that means "when displaying help for this function, don't automatically generate that first line." O: Have the handwritten signature in the docstring. When registering the function for 3.3, have the PyMethodDef docstring point to the it starting at the signature. When registering the function for 3.4+, have the docstring in the PyMethodDefEx point to the first byte after the handwritten signature. Note that automatically skipping the signature with a heuristic is mildly complicated, so this may be hard to get right. P: Have the handwritten signature in the docstring, and have separate static PyMethodDef and PyMethodDefEx arrays. The PyMethodDef docstring points to the docstring like normal. The PyMethodDefEx docstring field points to the first byte after the handwritten signature. This makes the registration "function" very simple: if it's 3.3 or before, use the PyMethodDef array, if it's 3.4+ use the PyMethodDefEx array. (Argument Clinic could theoretically automate coding some or all of this.) It's late and my brain is only working so well. I'd be interested in other approaches if people can suggest something good. Sorry about the mess, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Feb 3 16:08:15 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 3 Feb 2014 10:08:15 -0500 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52EFAB13.4020405@hastings.org> References: <52EFAB13.4020405@hastings.org> Message-ID: <20140203100815.0959f878@limelight.wooz.org> On Feb 03, 2014, at 06:43 AM, Larry Hastings wrote: >But that only fixes part of the problem. Our theoretical extension that >wants to be binary-compatible with 3.3 and 3.4 still has a problem: how can >they support signatures? They can't give PyMethodDefEx structures to 3.3, it >will blow up. But if they don't use PyMethodDefEx, they can't have >signatures. Can't an extension writer #ifdef around this? Yeah, it's ugly, but it's a pretty standard approach for making C extensions multi-version compatible. -Barry From benjamin at python.org Mon Feb 3 16:20:33 2014 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 03 Feb 2014 07:20:33 -0800 Subject: [Python-Dev] The online documentation is no more updated? In-Reply-To: References: Message-ID: <1391440833.10742.78692409.3D136F71@webmail.messagingengine.com> On Mon, Feb 3, 2014, at 02:22 AM, Victor Stinner wrote: > Hi, > > I modified the Python documentaton (asyncio module) 6 days ago, and my > changes are not online yet: > http://docs.python.org/dev/library/asyncio-eventloop.html#running-subprocesses > > Is it a problem with the server generating the documentation? Hopefully fixed now. > > By the way, would it be possible to regenerate the documentation at > each commit, or maybe every hours, instead of having to wait 24h to > see the modified documentation online? It was twice a day before, I've now switched the build to be every hour. From phil at riverbankcomputing.com Mon Feb 3 16:13:21 2014 From: phil at riverbankcomputing.com (Phil Thompson) Date: Mon, 03 Feb 2014 15:13:21 +0000 Subject: [Python-Dev] =?utf-8?q?=5FPyUnicode=5FCheckConsistency=28=29_too_?= =?utf-8?q?strict=3F?= Message-ID: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> _PyUnicode_CheckConsistency() checks that the contents of the string matches the _KIND of the string. However it does this in a very strict manner, ie. that the contents *exactly* match the _KIND rather than just detecting an inconsistency between the contents and the _KIND. For example, a string created with a maxchar of 255 (ie. a Latin-1 string) must contain at least one character in the range 128-255 otherwise you get an assertion failure. As it stands, when converting Latin-1 strings in my C extension module I must first check each character and specify a maxchar of 127 if the strings happens to only contain ASCII characters. What is the reasoning behind the checks being so strict? Phil From victor.stinner at gmail.com Mon Feb 3 16:35:51 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 3 Feb 2014 16:35:51 +0100 Subject: [Python-Dev] _PyUnicode_CheckConsistency() too strict? In-Reply-To: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> References: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> Message-ID: 2014-02-03 Phil Thompson : > For example, a string created with a maxchar of 255 (ie. a Latin-1 string) > must contain at least one character in the range 128-255 otherwise you get > an assertion failure. Yes, it's the specification of the PEP 393. > As it stands, when converting Latin-1 strings in my C extension module I > must first check each character and specify a maxchar of 127 if the strings > happens to only contain ASCII characters. Use PyUnicode_FromKindAndData(PyUnicode_1BYTE_KIND, latin1_str, length) which computes the kind for you. > What is the reasoning behind the checks being so strict? Different Python functions rely on the exact kind to compare strings. For example, if you search a latin1 substring in an ASCII string, the search returns immediatly instead of searching in the string. A latin1 string cannot be found in an ASCII string. The main reason in the PEP 393 itself, a string must be compact to not waste memory. Victor From phil at riverbankcomputing.com Mon Feb 3 16:44:27 2014 From: phil at riverbankcomputing.com (Phil Thompson) Date: Mon, 03 Feb 2014 15:44:27 +0000 Subject: [Python-Dev] =?utf-8?q?=5FPyUnicode=5FCheckConsistency=28=29_too_?= =?utf-8?q?strict=3F?= In-Reply-To: References: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> Message-ID: On 03-02-2014 3:35 pm, Victor Stinner wrote: > 2014-02-03 Phil Thompson : >> For example, a string created with a maxchar of 255 (ie. a Latin-1 >> string) >> must contain at least one character in the range 128-255 otherwise >> you get >> an assertion failure. > > Yes, it's the specification of the PEP 393. > >> As it stands, when converting Latin-1 strings in my C extension >> module I >> must first check each character and specify a maxchar of 127 if the >> strings >> happens to only contain ASCII characters. > > Use PyUnicode_FromKindAndData(PyUnicode_1BYTE_KIND, latin1_str, > length) which computes the kind for you. > >> What is the reasoning behind the checks being so strict? > > Different Python functions rely on the exact kind to compare strings. > For example, if you search a latin1 substring in an ASCII string, the > search returns immediatly instead of searching in the string. A > latin1 > string cannot be found in an ASCII string. > > The main reason in the PEP 393 itself, a string must be compact to > not > waste memory. > > Victor Are you saying that code will fail if a particular Latin-1 string just happens not to contains any character greater than 127? I would be very surprised if that was the case. If it isn't the case then I think that particular check shouldn't be made. Phil From victor.stinner at gmail.com Mon Feb 3 17:04:08 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 3 Feb 2014 17:04:08 +0100 Subject: [Python-Dev] _PyUnicode_CheckConsistency() too strict? In-Reply-To: References: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> Message-ID: 2014-02-03 Phil Thompson : > Are you saying that code will fail if a particular Latin-1 string just > happens not to contains any character greater than 127? PyUnicode_FromKindAndData(PyUnicode_1BYTE_KIND, latin1_str, length) accepts latin1 and ASCII strings. It computes the maximum code point and then use ASCII or latin1 unicode string. Victor From phil at riverbankcomputing.com Mon Feb 3 17:10:03 2014 From: phil at riverbankcomputing.com (Phil Thompson) Date: Mon, 03 Feb 2014 16:10:03 +0000 Subject: [Python-Dev] =?utf-8?q?=5FPyUnicode=5FCheckConsistency=28=29_too_?= =?utf-8?q?strict=3F?= In-Reply-To: References: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> Message-ID: <4261dee9f666192596da735c096cbd5e@www.riverbankcomputing.com> On 03-02-2014 4:04 pm, Victor Stinner wrote: > 2014-02-03 Phil Thompson : >> Are you saying that code will fail if a particular Latin-1 string >> just >> happens not to contains any character greater than 127? > > PyUnicode_FromKindAndData(PyUnicode_1BYTE_KIND, latin1_str, length) > accepts latin1 and ASCII strings. It computes the maximum code point > and then use ASCII or latin1 unicode string. > > Victor That doesn't answer my original question, that just works around the use case I presented. To restate... Why is a Latin-1 string considered inconsistent just because it doesn't happen to contain any characters in the range 128-255? Phil From larry at hastings.org Mon Feb 3 17:04:43 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 03 Feb 2014 08:04:43 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <20140203100815.0959f878@limelight.wooz.org> References: <52EFAB13.4020405@hastings.org> <20140203100815.0959f878@limelight.wooz.org> Message-ID: <52EFBE1B.9060606@hastings.org> On 02/03/2014 07:08 AM, Barry Warsaw wrote: > On Feb 03, 2014, at 06:43 AM, Larry Hastings wrote: > >> But that only fixes part of the problem. Our theoretical extension that >> wants to be binary-compatible with 3.3 and 3.4 still has a problem: how can >> they support signatures? They can't give PyMethodDefEx structures to 3.3, it >> will blow up. But if they don't use PyMethodDefEx, they can't have >> signatures. > Can't an extension writer #ifdef around this? Yeah, it's ugly, but it's a > pretty standard approach for making C extensions multi-version compatible. For source compatibility, yes. But I thought the point of the binary ABI was to allow compiling a single extension that worked unmodified with multiple versions of Python. If we simply don't support that, then an ifdef would be fine. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at bytereef.org Mon Feb 3 17:05:48 2014 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 3 Feb 2014 17:05:48 +0100 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52EFAB13.4020405@hastings.org> References: <52EFAB13.4020405@hastings.org> Message-ID: <20140203160548.GA9906@sleipnir.bytereef.org> Larry Hastings wrote: > So here's the problem. Let's say you want to write an extension that will work > with Python 3.3 and 3.4, using the stable ABI. If you don't add this line, > then in 3.4 you won't have introspection information, drat. But if you *do* > add this line, your docstring will look mildly stupid in 3.3, because it'll > have this unsightly "sig=(" line at the top. And it *won't* have a nice > handwritten docstring. (And if you added both a sig= signature *and* a > handwritten signature, in 3.4 it would display both. That would also look > dumb.) I think we may slowly get into PEP territory here. Just imagine that we settle on X, then decide at a later point to have a standard way of adding type annotations, then find that X does not work because of (unknown). I'm mentioning this because signatures get really interesting for me if they contain type information. Stefan Krah From victor.stinner at gmail.com Mon Feb 3 17:36:39 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 3 Feb 2014 17:36:39 +0100 Subject: [Python-Dev] The online documentation is no more updated? In-Reply-To: <1391440833.10742.78692409.3D136F71@webmail.messagingengine.com> References: <1391440833.10742.78692409.3D136F71@webmail.messagingengine.com> Message-ID: 2014-02-03 Benjamin Peterson : >> Is it a problem with the server generating the documentation? > > Hopefully fixed now. The documentation is still outdated. For example, I don't see the new subprocess page in http://docs.python.org/dev/library/asyncio.html > It was twice a day before, I've now switched the build to be every hour. Nice. I'm just waiting until it works :-) Victor From p.f.moore at gmail.com Mon Feb 3 17:38:35 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 3 Feb 2014 16:38:35 +0000 Subject: [Python-Dev] _PyUnicode_CheckConsistency() too strict? In-Reply-To: <4261dee9f666192596da735c096cbd5e@www.riverbankcomputing.com> References: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> <4261dee9f666192596da735c096cbd5e@www.riverbankcomputing.com> Message-ID: On 3 February 2014 16:10, Phil Thompson wrote: > That doesn't answer my original question, that just works around the use > case I presented. > > To restate... > > Why is a Latin-1 string considered inconsistent just because it doesn't > happen to contain any characters in the range 128-255? Butting in here (sorry) but I thought what Victor was trying to say is that being able to say that a string marked as Latin1 "kind" definitely has characters >127 allows the code to optimise some tests (for example, two strings cannot be equal if their kinds differ). Obviously, requiring this kind of constraint makes it somewhat harder for user code to construct string objects that conform to the spec. That's why the PyUnicode_FromKindAndData function has the convenience feature of doing the check and setting the kind correctly for you - you should use that rather than trying to get the details right yourself.. Paul. From brett at python.org Mon Feb 3 18:01:11 2014 From: brett at python.org (Brett Cannon) Date: Mon, 3 Feb 2014 12:01:11 -0500 Subject: [Python-Dev] Guidance regarding what counts as breaking backwards compatibility In-Reply-To: References: <20140202010642.GR3799@ando> Message-ID: On Sat, Feb 1, 2014 at 9:14 PM, Nick Coghlan wrote: > On 2 February 2014 11:06, Steven D'Aprano wrote: > > Hi all, > > > > Over on the Python-ideas list, there's a thread about the new statistics > > module, and as the author of that module, I'm looking for a bit of > > guidance regarding backwards compatibility. Specifically two issues: > > > > > > (1) With numeric code, what happens if the module become more[1] > > accurate in the future? Does that count as breaking backwards > > compatibility? > > > > E.g. Currently I use a particular algorithm for calculating variance. > > Suppose that for a particular data set, this algorithm is accurate to > > (say) seven decimal places: > > > > # Python 3.4 > > variance(some_data) == 1.23456700001 > > > > Later, I find a better algorithm, which improves the accuracy of the > > result: > > > > # Python 3.5 or 3.6 > > variance(some_data) == 1.23456789001 > > > > > > Would this count as breaking backwards compatibility? If so, how should > > I handle this? I don't claim that the current implementation of the > > statistics module is optimal, as far as precision and accuracy is > > concerned. It may improve in the future. > > For this kind of case, we tend to cover it in the "Porting to Python > X.Y" section of the What's New guide. User code *shouldn't* care about > this kind of change, but it *might*, so we split the difference and > say "It's OK in a feature release, but not in a maintenance release". > There have been multiple changes along these lines in our floating > handling as Tim Peters, Mark Dickinson et al have made various > improvements to reduce platform dependent behaviour (especially around > overflow handling, numeric precision, infinity and NaN handling, etc). > I agree with Nick that it's a feature release change, not a bugfix one. I think the key way to rationalize this particular case is the use of the words "better" and "improves". Notice you never said "broken" or "wrong", just that you were making the estimate better. Since the previous behaviour was not fundamentally broken and going to cause errors in correct code it then should only go in a feature release. -Brett > > However, we also sometimes have module specific disclaimers - the > decimal module, for example, has an explicit caveat that updates to > the General Decimal Arithmetic Specification will be treated as bug > fixes, even if they would normally not be allowed in maintenance > releases. > > For a non-math related example, a comment from Michael Foord at the > PyCon US 2013 sprints made me realise that the implementation of > setting the __wrapped__ attribute in functools was just flat out > broken - when applied multiple times it was supposed to create a chain > of references that eventually terminated in a callable without the > attribute set, but due to the bug every layer actually referred > directly to the innermost callable (the one without the attribute > set). Unfortunately, the docs I wrote for it were also ambiguous, so a > lot of folks (including Michael) assumed it was working as intended. I > have fixed the bug in 3.4, but there *is* a chance it will break > introspection code that assumed the old behaviour was intentional and > doesn't correctly unravel __wrapped__ chains. > > > Or would that count as a bug-fix? "Variance function was inaccurate, now > > less wrong", perhaps. > > > > I suppose the math module has the same issue, except that it just wraps > > the C libraries, which are mature and stable and unlikely to change. > > They may look that way *now*, but that's only after Tim, Mark et al > did a lot of work on avoiding platform specific issues and > inconsistencies > > > The random module has a similar issue: > > > > http://docs.python.org/3/library/random.html#notes-on-reproducibility > > I think a disclaimer in the statistics module similar to the ones in > the math module and this one in the random module would be appropriate > - one of the key purposes of the library/language reference is to let > us distinguish between "guaranteed behaviour user code can rely on" > and "implementation details that user code should not assume will > remain unchanged forever". > > In this case, it would likely be appropriate to point out that the > algorithms used internally may change over time, thus potentially > changing the error bounds in the module output. > > > (2) Mappings[2] are iterable. That means that functions which expect > > sequences or iterators may also operate on mappings by accident. For > > example, sum({1: 100, 2: 200}) returns 3. If one wanted to reserve the > > opportunity to handle mappings specifically in the future, without being > > locked in by backwards-compatibility, how should one handle it? > > > > a) document that behaviour with mappings is unsupported and may > > change in the future; > > > > b) raise a warning when passed a mapping, but still iterate over it; > > > > c) raise an exception and refuse to iterate over the mapping; > > > > d) something else? > > > > > > Question (2) is of course a specific example of a more general > > question, to what degree is the library author responsible for keeping > > backwards compatibility under circumstances which are not part of the > > intended API, but just work by accident? > > In this particular case, I consider having a single API treat mappings > differently from other iterables is a surprising anti-pattern and > providing a separate API specifically for mappings is clearer (cf > format() vs format_map()). > > However, if you want to preserve maximum flexibility, the best near > term option is typically c (just disallow the input you haven't > decided how to handle yet entirely), but either a or b would also be > an acceptable way of achieving the same end (they're just less user > friendly, since they let people do something that you're already > considering changing in the future). > > In the case where you allow an API to escape into the wild without > even a documented caveat that it isn't covered by the normal standard > library backwards compatibility guarantees, then you're pretty much > stuck. This is why you tend to see newer stdlib APIs often exposing > functions that return private object types, rather than exposing the > object type itself: exposed a function just promises a callable() API, > while exposing a class directly promises a lot more in terms of > supporting inheritance, isinstance(), issubclass(), etc, which can > make future evolution of that API substantially more difficult. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phil at riverbankcomputing.com Mon Feb 3 18:16:03 2014 From: phil at riverbankcomputing.com (Phil Thompson) Date: Mon, 03 Feb 2014 17:16:03 +0000 Subject: [Python-Dev] =?utf-8?q?=5FPyUnicode=5FCheckConsistency=28=29_too_?= =?utf-8?q?strict=3F?= In-Reply-To: References: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> <4261dee9f666192596da735c096cbd5e@www.riverbankcomputing.com> Message-ID: On 03-02-2014 4:38 pm, Paul Moore wrote: > On 3 February 2014 16:10, Phil Thompson > wrote: >> That doesn't answer my original question, that just works around the >> use >> case I presented. >> >> To restate... >> >> Why is a Latin-1 string considered inconsistent just because it >> doesn't >> happen to contain any characters in the range 128-255? > > Butting in here (sorry) but I thought what Victor was trying to say > is > that being able to say that a string marked as Latin1 "kind" > definitely has characters >127 allows the code to optimise some tests > (for example, two strings cannot be equal if their kinds differ). So there *is* code that will fail if a particular Latin-1 string just happens not to contains any character greater than 127? > Obviously, requiring this kind of constraint makes it somewhat harder > for user code to construct string objects that conform to the spec. > That's why the PyUnicode_FromKindAndData function has the convenience > feature of doing the check and setting the kind correctly for you - > you should use that rather than trying to get the details right > yourself.. > > Paul. I see now... The docs for PyUnicode_FromKindAndData() say... "Create a new Unicode object *with* the given kind" ...and so I didn't think is was useful to me. If they said... "Create a new Unicode object *from* the given kind" ...then I might have got it. Thanks - I'm happy now. Phil From solipsis at pitrou.net Mon Feb 3 18:35:39 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Feb 2014 18:35:39 +0100 Subject: [Python-Dev] _PyUnicode_CheckConsistency() too strict? References: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> <4261dee9f666192596da735c096cbd5e@www.riverbankcomputing.com> Message-ID: <20140203183539.45f0ac0d@fsol> On Mon, 03 Feb 2014 16:10:03 +0000 Phil Thompson wrote: > > Why is a Latin-1 string considered inconsistent just because it doesn't > happen to contain any characters in the range 128-255? Because as Victor said, it allows for some optimization shortcuts (e.g. a non-ASCII latin1 string cannot be equal to an ASCII string - no need for a memcmp). Regards Antoine. From guido at python.org Mon Feb 3 18:46:25 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Feb 2014 09:46:25 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52EFAB13.4020405@hastings.org> References: <52EFAB13.4020405@hastings.org> Message-ID: Larry, Can you summarize why neither of the two schemes you tried so far worked? AFAIR the original scheme was to support the 3.3-compatible syntax; there was some kind of corner-case problem with this, so you switched to the new "sig=..." syntax, but obviously this has poor compatibility with 3.3. Can you remind us of what the corner-case was? How bad would it be if we decided to just live with it or if we added a new flag bit (only recognized by 3.4) to disambiguate corner-cases? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Feb 3 18:52:00 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Feb 2014 09:52:00 -0800 Subject: [Python-Dev] _PyUnicode_CheckConsistency() too strict? In-Reply-To: <20140203183539.45f0ac0d@fsol> References: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> <4261dee9f666192596da735c096cbd5e@www.riverbankcomputing.com> <20140203183539.45f0ac0d@fsol> Message-ID: Can we provide a convenience API (or even a few lines of code one could copy+paste) that determines if a particular 8-bit string should have max-char equal to 127 or 255? I can easily imagine a number of use cases where this would come in handy (e.g. a list of strings produced by translation, or strings returned in Latin-1 by some other non-Python C-level API) -- and let's not get into a debate about whether UTF-8 wouldn't be better, I can also easily imagine legacy APIs where that isn't (yet) an option. On Mon, Feb 3, 2014 at 9:35 AM, Antoine Pitrou wrote: > On Mon, 03 Feb 2014 16:10:03 +0000 > Phil Thompson wrote: > > > > Why is a Latin-1 string considered inconsistent just because it doesn't > > happen to contain any characters in the range 128-255? > > Because as Victor said, it allows for some optimization shortcuts (e.g. > a non-ASCII latin1 string cannot be equal to an ASCII string - no need > for a memcmp). > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Feb 3 18:36:49 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 03 Feb 2014 09:36:49 -0800 Subject: [Python-Dev] _PyUnicode_CheckConsistency() too strict? In-Reply-To: References: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> <4261dee9f666192596da735c096cbd5e@www.riverbankcomputing.com> Message-ID: <52EFD3B1.6050909@stoneleaf.us> On 02/03/2014 09:16 AM, Phil Thompson wrote: > > So there *is* code that will fail if a particular Latin-1 string just > happens not to contains any character greater than 127? Yes, because if it does not contain a character > 127 it is not a latin-1 string as far as Python is concerned. -- ~Ethan~ From phil at riverbankcomputing.com Mon Feb 3 19:10:27 2014 From: phil at riverbankcomputing.com (Phil Thompson) Date: Mon, 03 Feb 2014 18:10:27 +0000 Subject: [Python-Dev] =?utf-8?q?=5FPyUnicode=5FCheckConsistency=28=29_too_?= =?utf-8?q?strict=3F?= In-Reply-To: References: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> <4261dee9f666192596da735c096cbd5e@www.riverbankcomputing.com> <20140203183539.45f0ac0d@fsol> Message-ID: On 03-02-2014 5:52 pm, Guido van Rossum wrote: > Can we provide a convenience API (or even a few lines of code one > could copy+paste) that determines if a particular 8-bit string > should? have max-char equal to 127 or 255? I can easily imagine a > number of use cases where this would come in handy (e.g. a list of > strings produced by translation, or strings returned in Latin-1 by > some other non-Python C-level API) -- and lets not get into a debate > about whether UTF-8 wouldnt be better, I can also easily imagine > legacy APIs where that isnt (yet) an option. For my particular use case PyUnicode_FromKindAndData() (once I'd interpreted the docs correctly) should have made such code unnecessary. However I've just discovered that it doesn't support surrogates in UCS2 so I'm going to have to roll my own anyway. Phil From ethan at stoneleaf.us Mon Feb 3 19:08:06 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 03 Feb 2014 10:08:06 -0800 Subject: [Python-Dev] _PyUnicode_CheckConsistency() too strict? In-Reply-To: References: <48b32b773ca7370a8db8f1060d17be2b@www.riverbankcomputing.com> <4261dee9f666192596da735c096cbd5e@www.riverbankcomputing.com> <20140203183539.45f0ac0d@fsol> Message-ID: <52EFDB06.2020005@stoneleaf.us> On 02/03/2014 09:52 AM, Guido van Rossum wrote: > Can we provide a convenience API (or even a few lines of code one could copy+paste) that determines if a particular > 8-bit string should have max-char equal to 127 or 255? Isn't that what this is? ============================================================ PyObject* PyUnicode_FromKindAndData( int kind, const void *buffer, Py_ssize_t size) Create a new Unicode object with the given kind (possible values are PyUnicode_1BYTE_KIND etc., as returned by PyUnicode_KIND()). The buffer must point to an array of size units of 1, 2 or 4 bytes per character, as given by the kind. ============================================================ -- ~Ethan~ From benjamin at python.org Mon Feb 3 20:07:41 2014 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 03 Feb 2014 11:07:41 -0800 Subject: [Python-Dev] The online documentation is no more updated? In-Reply-To: References: <1391440833.10742.78692409.3D136F71@webmail.messagingengine.com> Message-ID: <1391454461.24324.78797401.67EF6FEA@webmail.messagingengine.com> On Mon, Feb 3, 2014, at 08:36 AM, Victor Stinner wrote: > 2014-02-03 Benjamin Peterson : > >> Is it a problem with the server generating the documentation? > > > > Hopefully fixed now. > > The documentation is still outdated. For example, I don't see the new > subprocess page in > http://docs.python.org/dev/library/asyncio.html Finally fixed. From vic.scherba at gmail.com Mon Feb 3 20:41:22 2014 From: vic.scherba at gmail.com (=?KOI8-R?B?98nL1M/SIP3F0sLB?=) Date: Mon, 3 Feb 2014 23:41:22 +0400 Subject: [Python-Dev] BugFix for "time" native module in Python 2.7 Message-ID: Hi, guys! I have found a bug in module "time" function "sleep" in Python 2.7 under Windows XP / Server 2003 and lower. I fix this bug locally. But how could I send you hg patch or pull request or, maybe, commit to sandbox? P.S.: Sorry for my English :-) Best Regards, Victor Scherba, C++ developer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at timgolden.me.uk Mon Feb 3 20:55:22 2014 From: mail at timgolden.me.uk (Tim Golden) Date: Mon, 03 Feb 2014 19:55:22 +0000 Subject: [Python-Dev] BugFix for "time" native module in Python 2.7 In-Reply-To: References: Message-ID: <52EFF42A.50606@timgolden.me.uk> On 03/02/2014 19:41, ?????? ????? wrote: > Hi, guys! > > I have found a bug in module "time" function "sleep" in Python 2.7 under > Windows XP / Server 2003 and lower. > I fix this bug locally. But how could I send you hg patch or pull > request or, maybe, commit to sandbox? The best thing is to raise an bug on our issue tracker: http://bugs.python.org/ and attach the patch together with a test which reproduces the issue TJG From brett at python.org Mon Feb 3 20:55:10 2014 From: brett at python.org (Brett Cannon) Date: Mon, 3 Feb 2014 14:55:10 -0500 Subject: [Python-Dev] BugFix for "time" native module in Python 2.7 In-Reply-To: References: Message-ID: On Mon, Feb 3, 2014 at 2:41 PM, ?????? ????? wrote: > Hi, guys! > > I have found a bug in module "time" function "sleep" in Python 2.7 under > Windows XP / Server 2003 and lower. > I fix this bug locally. But how could I send you hg patch or pull request > or, maybe, commit to sandbox? > Please file bugs and patches at bugs.python.org. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pydev at gmail.com Mon Feb 3 20:56:06 2014 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Mon, 3 Feb 2014 13:56:06 -0600 Subject: [Python-Dev] BugFix for "time" native module in Python 2.7 In-Reply-To: References: Message-ID: Hi Victor, On Mon, Feb 3, 2014 at 1:41 PM, ?????? ????? wrote: > Hi, guys! > > I have found a bug in module "time" function "sleep" in Python 2.7 under > Windows XP / Server 2003 and lower. > I fix this bug locally. But how could I send you hg patch or pull request > or, maybe, commit to sandbox? Please open an issue on the bug tracker (http://bugs.python.org/) and attach your patch. > P.S.: Sorry for my English :-) Your English is better than that of most (American) teenagers I know, don't worry :) -- Zach From solipsis at pitrou.net Mon Feb 3 20:56:50 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Feb 2014 20:56:50 +0100 Subject: [Python-Dev] BugFix for "time" native module in Python 2.7 References: Message-ID: <20140203205650.7cadf6c2@fsol> Hi Victor, You can find instructions here to produce a patch: http://docs.python.org/devguide/ The first thing to do would be to create an issue on http://bugs.python.org/, and post your patch there. Regards Antoine. On Mon, 3 Feb 2014 23:41:22 +0400 ?????? ????? wrote: > Hi, guys! > > I have found a bug in module "time" function "sleep" in Python 2.7 under > Windows XP / Server 2003 and lower. > I fix this bug locally. But how could I send you hg patch or pull request > or, maybe, commit to sandbox? > > P.S.: Sorry for my English :-) > > Best Regards, > Victor Scherba, > C++ developer. > From victor.stinner at gmail.com Mon Feb 3 20:58:56 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 3 Feb 2014 20:58:56 +0100 Subject: [Python-Dev] The online documentation is no more updated? In-Reply-To: <1391454461.24324.78797401.67EF6FEA@webmail.messagingengine.com> References: <1391440833.10742.78692409.3D136F71@webmail.messagingengine.com> <1391454461.24324.78797401.67EF6FEA@webmail.messagingengine.com> Message-ID: 2014-02-03 Benjamin Peterson : >> The documentation is still outdated. For example, I don't see the new >> subprocess page in >> http://docs.python.org/dev/library/asyncio.html > > Finally fixed. Thanks. Victor From tseaver at palladion.com Mon Feb 3 21:02:24 2014 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 03 Feb 2014 15:02:24 -0500 Subject: [Python-Dev] BugFix for "time" native module in Python 2.7 In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/03/2014 02:41 PM, ?????? ????? wrote: > Hi, guys! > > I have found a bug in module "time" function "sleep" in Python 2.7 > under Windows XP / Server 2003 and lower. I fix this bug locally. But > how could I send you hg patch or pull request or, maybe, commit to > sandbox? > > P.S.: Sorry for my English :-) Create an issue on the Python tracker: http://bugs.python.org/ and attach your patch there. See the Python Developers' Guide for more details: http://docs.python.org/devguide/ Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlLv9dAACgkQ+gerLs4ltQ6JdQCfbQ5lFmW9OQceVZJFW8Zs3fIz 0m4An3ttIZbdWtVLtOtloMfsrnW4nanE =Cd+P -----END PGP SIGNATURE----- From tjreedy at udel.edu Mon Feb 3 21:55:46 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 03 Feb 2014 15:55:46 -0500 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52EFAB13.4020405@hastings.org> References: <52EFAB13.4020405@hastings.org> Message-ID: On 2/3/2014 9:43 AM, Larry Hastings wrote: > > > A quick summary of the context: currently in CPython 3.4, a builtin > function can publish its "signature" as a specially encoded line at the > top of its docstring. CPython internally detects this line inside > PyCFunctionObject.__doc__ and skips past it, and there's a new getter at > PyCFunctionObject.__text_signature__ that returns just this line. As an > example, the signature for os.stat looks like this: > > sig=($module, path, *, dir_fd=None, follow_symlinks=True) > > The convention is, if you have this signature, you shouldn't have your > docstring start with a handwritten signature like 3.3 and before. > help() on a callable displays the signature automatically if it can, so > if you *also* had a handwritten signature, help() would show two > signatures. That would look dumb. > > ----- > > So here's the problem. Let's say you want to write an extension that > will work with Python 3.3 and 3.4, using the stable ABI. If you don't > add this line, then in 3.4 you won't have introspection information, > drat. But if you *do* add this line, your docstring will look mildly > stupid in 3.3, because it'll have this unsightly "sig=(" line at the > top. And it *won't* have a nice handwritten docstring. (And if you > added both a sig= signature *and* a handwritten signature, in 3.4 it > would display both. That would also look dumb.) I think the solution adopted should be future-oriented (ie, clean in the future) even if the cost is slight awkwardness in 3.3. To me, an temporary 'unsightly' extra 'sig=' at the start of some docstrings, in one release, is better that any of the permanent contortions you propose to avoid it. For 3.3.5 Idle, I could add a check to detect and remove 'sig=' from calltips, but I would not consider it a disaster for it to appear with earlier versions. In 3.3.5 (assuming no change is possible for 3.3.4), help (or pydoc) could make the same check and deletion. As with calltips, help is for interactive viewing by humans. [snip] > O: Have the handwritten signature in the docstring. When registering > the function for 3.3, have the PyMethodDef docstring point to the it > starting at the signature. When registering the function for 3.4+, have > the docstring in the PyMethodDefEx point to the first byte after the > handwritten signature. Note that automatically skipping the signature > with a heuristic is mildly complicated, so this may be hard to get right. The old convention builtins was a one line handwritten signature followed by a blank line. For Python functions, one line describing the return value. The 'heuristic' for Idle was to grab the first line of the docstring. If than ended in mid-sentence because someone did not follow the convention, too bad. The newer convention for builtins is multiple lines followed by a blank line. So I recently changed the heuristic to all lines up to the first blank, but with a limit of 5 (needed for bytes), as protection against doctrings that start with a long paragraph. -- Terry Jan Reedy From zachary.ware+pydev at gmail.com Mon Feb 3 22:10:08 2014 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Mon, 3 Feb 2014 15:10:08 -0600 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52EFAB13.4020405@hastings.org> References: <52EFAB13.4020405@hastings.org> Message-ID: On Mon, Feb 3, 2014 at 8:43 AM, Larry Hastings wrote: > > > A quick summary of the context: currently in CPython 3.4, a builtin function > can publish its "signature" as a specially encoded line at the top of its > docstring. CPython internally detects this line inside > PyCFunctionObject.__doc__ and skips past it, and there's a new getter at > PyCFunctionObject.__text_signature__ that returns just this line. As an > example, the signature for os.stat looks like this: > > sig=($module, path, *, dir_fd=None, follow_symlinks=True) > > The convention is, if you have this signature, you shouldn't have your > docstring start with a handwritten signature like 3.3 and before. help() on > a callable displays the signature automatically if it can, so if you *also* > had a handwritten signature, help() would show two signatures. That would > look dumb. > > ----- > > So here's the problem. Let's say you want to write an extension that will > work with Python 3.3 and 3.4, using the stable ABI. If you don't add this > line, then in 3.4 you won't have introspection information, drat. But if > you *do* add this line, your docstring will look mildly stupid in 3.3, > because it'll have this unsightly "sig=(" line at the top. And it *won't* > have a nice handwritten docstring. (And if you added both a sig= signature > *and* a handwritten signature, in 3.4 it would display both. That would > also look dumb.) What about just choosing a marker value that is somewhat less unsightly? "signature = (", or "parameters: (", or something (better) to that effect? It may not be beautiful in 3.3, but we can at least make it make sense. -- Zach From greg at krypto.org Mon Feb 3 23:06:48 2014 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 3 Feb 2014 14:06:48 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52EFBE1B.9060606@hastings.org> References: <52EFAB13.4020405@hastings.org> <20140203100815.0959f878@limelight.wooz.org> <52EFBE1B.9060606@hastings.org> Message-ID: On Mon, Feb 3, 2014 at 8:04 AM, Larry Hastings wrote: > On 02/03/2014 07:08 AM, Barry Warsaw wrote: > > On Feb 03, 2014, at 06:43 AM, Larry Hastings wrote: > > > But that only fixes part of the problem. Our theoretical extension that > wants to be binary-compatible with 3.3 and 3.4 still has a problem: how can > they support signatures? They can't give PyMethodDefEx structures to 3.3, it > will blow up. But if they don't use PyMethodDefEx, they can't have > signatures. > > Can't an extension writer #ifdef around this? Yeah, it's ugly, but it's a > pretty standard approach for making C extensions multi-version compatible. > > > For source compatibility, yes. But I thought the point of the binary ABI > was to allow compiling a single extension that worked unmodified with > multiple versions of Python. If we simply don't support that, then an > ifdef would be fine. > Wouldn't your proposal to extend the PyMethodDef structure would require ifdef's and make it impossible to include the type information in something compiled against the 3.3 headers that you want to use in 3.4 without recompiling? If you don't like seeing an sig= at the front of the docstring couldn't you just move it to the end of the docstring. I don't think messiness in docstrings when running something read for 3.4 under 3.3 is a big deal. [side note] I consider it CRAZY for anyone to load a binary extension module compiled for one version in a later version of Python. People do it, I know, but they're insane. I wish we didn't bother trying to support that crap. I know this isn't going to change in 3.4. Just ranting. [/side note] -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 3 23:26:05 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 3 Feb 2014 23:26:05 +0100 Subject: [Python-Dev] The docstring hack for signature information has to go References: <52EFAB13.4020405@hastings.org> Message-ID: <20140203232605.71dd64f6@fsol> On Mon, 03 Feb 2014 06:43:31 -0800 Larry Hastings wrote: > > A: We create a PyMethodDefEx structure with an extra field: "const char > *signature". We add a new METH_SIGNATURE (maybe just METH_SIG?) flag to > the flags, indicating that this is an extended structure. When > iterating over the PyMethodDefs, we know how far to advance the pointer > based on this flag. How do you create an array that mixes PyMethodDefs and PyMethodDefExs together? It sounds like METH_SIGNATURE is the wrong mechanism. Instead, you may want a tp_methods_ex as well as as a Py_TPFLAGS_HAVE_METHODS_EX. > B: Same as A, but we add three unused pointers (void *reserved1 etc) to > PyMethodDefEx to give us some room to grow. Note that this constrains future growth to only add pointer fields, unless you also add a couple of long fields. But at least it sounds workable. > C: Same as A, but we add two fields to PyMethodDefEx. The second new > field identifies the "version" of the structure, telling us its size > somehow. Like the lStructSize field of the OPENFILENAME structure in > Win32. I suspect YAGNI. That doesn't work. The size of elements of a C array is constant, so you can't "mix and match" PyMethodDefExs of different versions with different sizes each. > Solution: we write a function (which users would have to copy into their > extension) that gives a PyMethodDefEx array to 3.4+, but converts it > into a PyMethodDef array for 3.3. The tricky part there: what do we do > about the docstring? The convention for builtins is to have the first > line(s) contain a handwritten signature. But you *don't* want that if > you provide a signature, because help() will read that signature and > automatically render this first line for you. Uh... If you write a "conversion function", you may as well make it so it converts the "sig=" line to a plain signature line in 3.3, which avoids the issue entirely. (and how would that conversion function be shipped to the user anyway? Python 3.3 and the stable ABI don't have it) Regards Antoine. From larry at hastings.org Tue Feb 4 02:29:06 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 03 Feb 2014 17:29:06 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: References: <52EFAB13.4020405@hastings.org> Message-ID: <52F04262.4010806@hastings.org> On 02/03/2014 09:46 AM, Guido van Rossum wrote: > Can you summarize why neither of the two schemes you tried so far worked? Certainly. In the first attempt, the signature looked like this: (arguments)\n The "(arguments)" part of the string was 100% compatible with Python syntax. So much so that I didn't write my own parser. Instead, I would take the whole line, strip off the \n, prepend it with "def ", append it with ": pass", and pass in the resulting string to ast.parse(). This had the advantage of looking great if the signature was not mechanically separated from the rest of the docstring: it looked like the old docstring with the handwritten signature on top. The problem: false positives. This is also exactly the traditional format for handwritten signatures. The function in C that mechanically separated the signature from the rest of the docstring had a simple heuristic: if the docstring started with "(", it assumed it had a valid signature and separated it from the rest of the docstring. But most of the functions in CPython passed this test, which resulted in complaints like "help(open) eats first line": http://bugs.python.org/issue20075 I opened an issue, writing a long impassioned plea to change this syntax: http://bugs.python.org/issue20326 Which we did. In the second attempt, the signature looked like this: sig=(arguments)\n In other words, the same as the first attempt, but with "sig=" instead of the name of the function. Since you never see docstrings that start with "sig=" in the wild, the false positives dropped to zero. I also took the opportunity to modify the signature slightly. Signatures were a little inconsistent about whether they specified the "self" parameter or not, so there were some complicated heuristics in inspect.Signature about when to keep or omit the first argument. In the new format I made this more explicit: if the first argument starts with a dollar sign ("$"), that means "this is a special first argument" (self for methods, module for module-level callables, type for class methods and __new__). That removed all the guesswork from inspect.Signature; now it works great. (In case you're wondering: I still use ast.parse to parse the signature, I just strip out the "$" first.) I want to mention: we anticipate modifying the syntax further in 3.5, adding square brackets around parameters to indicate "optional groups". This all has caused no problems so far. But my panicky email last night was me realizing a problem we may see down the road. To recap: if a programmer writes a module using the binary ABI, in theory they can use it with different Python versions without modification. If this programmer added Python 3.4+ compatible signatures, they'd have to insert this "sig=(" line at the top of their docstring. The downside: Python 3.3 doesn't understand that this is a signature and would happily display it to the user as part of help(). > How bad would it be if we decided to just live with it or if we added > a new flag bit (only recognized by 3.4) to disambiguate corner-cases? A new flag might solve the problem cheaply. Let's call it METH_SIG, set in the flags portion of the PyMethodDef. It would mean "This docstring contains a computer-readable signature". One could achieve source compatibility with 3.3 easily by adding "#ifndef METH_SIG / #define METH_SIG 0 / #endif"; the next version of 3.3 could add that itself. We could then switch back to the original approach of "(", so the signature would look presentable when displayed to the user. It would still have the funny dollar-sign, a la "$self" or "$module" or "$type", but perhaps users could live with that. Though perhaps this time maybe the end delimiter should be two newlines in a row, so that we can text-wrap long signature lines to enhance their readability if/when they get shown to users. I have two caveats: A: for binary compatibility, would Python 3.3 be allergic to this unfamiliar flag in PyMethodDef? Or does it ignore flags it doesn't explicitly look for? B: I had to modify four (or was it five?) different types in Python to add support for mechanically separating the __text_signature__. Although all of them originally started with a PyMethodDef structure, I'm not sure that all of them carry the "flags" parameter around with them. We might have to add a "flags" to a couple of these. Fortunately I believe they're all part of Py_LIMITED_API. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Feb 4 05:19:26 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Feb 2014 20:19:26 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52F04262.4010806@hastings.org> References: <52EFAB13.4020405@hastings.org> <52F04262.4010806@hastings.org> Message-ID: Hmm... I liked the original scheme because it doesn't come out so badly if some tool doesn't special-case the first line of the docstring at all. (I have to fess up that I wrote such a tool for a limited case not too long ago, and I wrote it to search for a blank line if the docstring starts with followed by '('.) Adding a flag sounds harmless, all the code I could find that looks at them just checks whether specific flags it knows about are set. But why do you even need a flag? Reading issue 20075 where the complaint started, it really feels that the change was an overreaction to a very minimal problem. A few docstrings appear truncated. Big deal. We can rewrite the ones that are reported as broken (either by adjusting the docstring to not match the patter or by adjusting it to match the pattern better, depending on the case). Tons of docstrings contain incorrect info, we just fix them when we notice the issue, we don't declare the language broken. On Mon, Feb 3, 2014 at 5:29 PM, Larry Hastings wrote: > > > On 02/03/2014 09:46 AM, Guido van Rossum wrote: > > Can you summarize why neither of the two schemes you tried so far worked? > > > Certainly. > > In the first attempt, the signature looked like this: > > (arguments)\n > > The "(arguments)" part of the string was 100% compatible with Python > syntax. So much so that I didn't write my own parser. Instead, I would > take the whole line, strip off the \n, prepend it with "def ", append it > with ": pass", and pass in the resulting string to ast.parse(). > > This had the advantage of looking great if the signature was not > mechanically separated from the rest of the docstring: it looked like the > old docstring with the handwritten signature on top. > > The problem: false positives. This is also exactly the traditional format > for handwritten signatures. The function in C that mechanically separated > the signature from the rest of the docstring had a simple heuristic: if the > docstring started with "(", it assumed it had a valid > signature and separated it from the rest of the docstring. But most of the > functions in CPython passed this test, which resulted in complaints like > "help(open) eats first line": > > http://bugs.python.org/issue20075 > > I opened an issue, writing a long impassioned plea to change this syntax: > > http://bugs.python.org/issue20326 > > Which we did. > > > In the second attempt, the signature looked like this: > > sig=(arguments)\n > > In other words, the same as the first attempt, but with "sig=" instead of > the name of the function. Since you never see docstrings that start with > "sig=" in the wild, the false positives dropped to zero. > > I also took the opportunity to modify the signature slightly. Signatures > were a little inconsistent about whether they specified the "self" > parameter or not, so there were some complicated heuristics in > inspect.Signature about when to keep or omit the first argument. In the > new format I made this more explicit: if the first argument starts with a > dollar sign ("$"), that means "this is a special first argument" (self for > methods, module for module-level callables, type for class methods and > __new__). That removed all the guesswork from inspect.Signature; now it > works great. (In case you're wondering: I still use ast.parse to parse the > signature, I just strip out the "$" first.) > > I want to mention: we anticipate modifying the syntax further in 3.5, > adding square brackets around parameters to indicate "optional groups". > > This all has caused no problems so far. But my panicky email last night > was me realizing a problem we may see down the road. To recap: if a > programmer writes a module using the binary ABI, in theory they can use it > with different Python versions without modification. If this programmer > added Python 3.4+ compatible signatures, they'd have to insert this "sig=(" > line at the top of their docstring. The downside: Python 3.3 doesn't > understand that this is a signature and would happily display it to the > user as part of help(). > > > > How bad would it be if we decided to just live with it or if we added a > new flag bit (only recognized by 3.4) to disambiguate corner-cases? > > > A new flag might solve the problem cheaply. Let's call it METH_SIG, set > in the flags portion of the PyMethodDef. It would mean "This docstring > contains a computer-readable signature". One could achieve source > compatibility with 3.3 easily by adding "#ifndef METH_SIG / #define > METH_SIG 0 / #endif"; the next version of 3.3 could add that itself. We > could then switch back to the original approach of "(", > so the signature would look presentable when displayed to the user. It > would still have the funny dollar-sign, a la "$self" or "$module" or > "$type", but perhaps users could live with that. Though perhaps this time > maybe the end delimiter should be two newlines in a row, so that we can > text-wrap long signature lines to enhance their readability if/when they > get shown to users. > > I have two caveats: > > A: for binary compatibility, would Python 3.3 be allergic to this > unfamiliar flag in PyMethodDef? Or does it ignore flags it doesn't > explicitly look for? > > B: I had to modify four (or was it five?) different types in Python to add > support for mechanically separating the __text_signature__. Although all > of them originally started with a PyMethodDef structure, I'm not sure that > all of them carry the "flags" parameter around with them. We might have to > add a "flags" to a couple of these. Fortunately I believe they're all part > of Py_LIMITED_API. > > > */arry* > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Tue Feb 4 05:40:28 2014 From: Nikolaus at rath.org (Nikolaus Rath) Date: Mon, 03 Feb 2014 20:40:28 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52F04262.4010806@hastings.org> (Larry Hastings's message of "Mon, 03 Feb 2014 17:29:06 -0800") References: <52EFAB13.4020405@hastings.org> <52F04262.4010806@hastings.org> Message-ID: <87txcfpk1v.fsf@vostro.rath.org> Larry Hastings writes: > In the second attempt, the signature looked like this: > > sig=(arguments)\n > [...] > This all has caused no problems so far. But my panicky email last > night was me realizing a problem we may see down the road. To recap: > if a programmer writes a module using the binary ABI, in theory they > can use it with different Python versions without modification. If > this programmer added Python 3.4+ compatible signatures, they'd have > to insert this "sig=(" line at the top of their docstring. The > downside: Python 3.3 doesn't understand that this is a signature and > would happily display it to the user as part of help(). I think this is not a bug, it's a feature. Since 3.3 users don't have the special signature parser either, this gives them exactly the information they need and without any duplication. The only drawback is in the cosmetic "sig=" prefix -- but that's the right amount of non-intrusive, kind nudging to get people to eventually update. >> How bad would it be if we decided to just live with it or if we >> added a new flag bit (only recognized by 3.4) to disambiguate >> corner-cases? > > A new flag might solve the problem cheaply. Let's call it METH_SIG, > set in the flags portion of the PyMethodDef. It would mean "This > docstring contains a computer-readable signature". One could achieve > source compatibility with 3.3 easily by adding "#ifndef METH_SIG / > #define METH_SIG 0 / #endif"; the next version of 3.3 could add that > itself. We could then switch back to the original approach of > "(", so the signature would look presentable when [...] That much effort to fix a purely cosmetic problem showing up only in older releases? Note that it's going to be a while until machine generated signatures have actually trickled down to end-users, so it's not as if every 3.3 installation would suddenly show different docstrings for all modules. Just my $0.02 of course. Best, Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ?Time flies like an arrow, fruit flies like a Banana.? From larry at hastings.org Tue Feb 4 10:12:58 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Feb 2014 01:12:58 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: References: <52EFAB13.4020405@hastings.org> <52F04262.4010806@hastings.org> Message-ID: <52F0AF1A.1060005@hastings.org> On 02/03/2014 08:19 PM, Guido van Rossum wrote: > But why do you even need a flag? Reading issue 20075 where the > complaint started, it really feels that the change was an overreaction > to a very minimal problem. I'll cop to that. I'm pretty anxious about trying to "get it right". My worry was (and is) that this hiding-the-signature-in-the-docstring approach is a cheap hack, and it will have unexpected and undesirable side-effects that will in retrospect seem obvious. This is FUD I admit. But it seems to me if we did it "the right way", with a "PyMethodDefEx", we'd be able to do a lot better job predicting the side-effects. > A few docstrings appear truncated. Big deal. We can rewrite the ones > that are reported as broken (either by adjusting the docstring to not > match the patter or by adjusting it to match the pattern better, > depending on the case). Tons of docstrings contain incorrect info, we > just fix them when we notice the issue, we don't declare the language > broken. I don't think #20075 touches on it, but my biggest concern was third-party modules. If you maintain a Python module, you very well might compile for 3.4 only to find that the first line of your docstrings have mysteriously vanished. You'd have to be very current on changes in Python 3.4 to know what was going on. It seemed like an overly efficient way of pissing off external module maintainers. (Why would they vanish? The mechanical separator for __doc__ vs __text_signature__ would accept them, but unless they're 100% compatible Python ast.parse will reject them. So they'd get stripped from your docstring, but you wouldn't get a valid signature in return.) I'd feel much better with an explicit flag--explicit is better than implicit, after all. But here's a reminder, to make it easier for you to say "no". That would mean adding an explicit flag to all the objects which support a signature hidden in the docstring: * PyTypeObject (has tp_flags, we've used 18 of 32 bits by my count) * PyMethodDef (has ml_flags, 7 of 32 bits in use) * PyMethodDescrObject (reuses PyMethodDef) * PyWrapperDescrObject (has d_base->flags, 1 of 32 bits in use) * wrapperobject (reuses PyWrapperDescrObject) Argument Clinic would write the PyMethodDefs, so we'd get those for free. The last three all originate from a PyMethodDef, so when we copied out the docstring pointer we could also propagate the flag. But we'd have to add the flag by hand to the PyTypeObjects. If you won't let me have a flag, can I at least have a more-clever marker? How about this: (...)\n \n Yes, the last four characters are right-parenthesis, newline, space, and newline. Benefits: * The odds of finding *that* in the wild seem remote. * If this got displayed as help in 3.3 the user would never notice the space. For the record, here are things that may be in the signature that aren't legal Python syntax and therefore might be surprising: * "self" parameters (and "module" and "type") are prefixed with '$'. * Positional-only parameters will soon be delimited by '/', just as keyword-only parameters are currently delimited by '*'. (Hasn't happened yet. Needs to happen for 3.4, in order for inspect.Signature to be accurate.) //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Tue Feb 4 10:41:15 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 04 Feb 2014 10:41:15 +0100 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52F0AF1A.1060005@hastings.org> References: <52EFAB13.4020405@hastings.org> <52F04262.4010806@hastings.org> <52F0AF1A.1060005@hastings.org> Message-ID: Am 04.02.2014 10:12, schrieb Larry Hastings: > If you won't let me have a flag, can I at least have a more-clever marker? How > about this: > > (...)\n \n > > Yes, the last four characters are right-parenthesis, newline, space, and > newline. Benefits: > > * The odds of finding *that* in the wild seem remote. > * If this got displayed as help in 3.3 the user would never notice the space. Clever, but due to the "hidden" space it also increases the frustration factor for people trying to find out "why is this accepted as a signature and not this". I don't think a well-chosen visible separator is worse off, such as "--\n". Georg From larry at hastings.org Tue Feb 4 10:55:12 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Feb 2014 01:55:12 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: References: <52EFAB13.4020405@hastings.org> Message-ID: <52F0B900.9090301@hastings.org> On 02/03/2014 12:55 PM, Terry Reedy wrote: > I think the solution adopted should be future-oriented (ie, clean in > the future) even if the cost is slight awkwardness in 3.3. Just a minor point: I keep saying 3.3, but I kind of mean "3.2 through 3.3". I believe the binary ABI shipped with 3.2. However, in practice I suspect there are few installations that * are still on 3.2, and * will ever use binary-ABI-clean third-party modules compiled against 3.4+ that contain signatures. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue Feb 4 10:56:40 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Feb 2014 01:56:40 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: References: <52EFAB13.4020405@hastings.org> Message-ID: <52F0B958.9080204@hastings.org> On 02/03/2014 01:10 PM, Zachary Ware wrote: > What about just choosing a marker value that is somewhat less > unsightly? "signature = (", or "parameters: (", or something (better) > to that effect? It may not be beautiful in 3.3, but we can at least > make it make sense. It's a reasonable enough idea, and we could consider it if we stick with something like "sig=". However, see later in the thread where Guido says to return to the old somewhat-ambiguous form with the function name. ;-) //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue Feb 4 10:59:29 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Feb 2014 01:59:29 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: References: <52EFAB13.4020405@hastings.org> <20140203100815.0959f878@limelight.wooz.org> <52EFBE1B.9060606@hastings.org> Message-ID: <52F0BA01.20504@hastings.org> On 02/03/2014 02:06 PM, Gregory P. Smith wrote: > Wouldn't your proposal to extend the PyMethodDef structure would > require ifdef's and make it impossible to include the type information > in something compiled against the 3.3 headers that you want to use in > 3.4 without recompiling? It might use #ifdefs. However, my proposal was forwards-compatible. When iterating over the methoddef array passed in with a type, if the PyMethodDef flags parameter had METH_SIGNATURE set, I'd advance by sizeof(PyMethodDefEx) bytes, otherwise I'd advance by sizeof(PyMethodDef) bytes. Modules compiled against 3.3 would not have the flag set, therefore I'd advance by the right amount, therefore they should be fine. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue Feb 4 11:18:08 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Feb 2014 02:18:08 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <20140203232605.71dd64f6@fsol> References: <52EFAB13.4020405@hastings.org> <20140203232605.71dd64f6@fsol> Message-ID: <52F0BE60.2010001@hastings.org> On 02/03/2014 02:26 PM, Antoine Pitrou wrote: > How do you create an array that mixes PyMethodDefs and PyMethodDefExs > together? You're right, you wouldn't be able to. For my PyMethodDefEx proposal, the entire array would have to be one way or the other. > It sounds like METH_SIGNATURE is the wrong mechanism. > Instead, you may want a tp_methods_ex as well as as a > Py_TPFLAGS_HAVE_METHODS_EX. You may well be right. We'd already need a flag on the type object anyway, to indicate "tp_doc start with a signature". So if we had such a flag, it could do double-duty to also indicate "tp_methods points to PyMethodDefEx structures". My only concern here: __text_signature__ is supported on five internal objects: PyCFunctionObject, PyTypeObject, PyMethodDescr_Type, _PyMethodWrapper_Type, and PyWrapperDescr_Type. I'm not certain that all of those carry around their own pointer back to their original type object. If you went off the "self" parameter, you wouldn't have one if you were an unbound method. And you might get the wrong answer if the user bound you to a different class, or if you were accessed through a subclass. (I say "might" not to mean "it could happen sometimes", but rather "I don't know what the correct answer is".) > Note that this constrains future growth to only add pointer fields, > unless you also add a couple of long fields. But at least it sounds > workable. Ah, in the back of my mind I meant to say "add some unused union {void *p; long i;} fields". Though in practice I don't think we support any platforms where sizeof(long) > sizeof(void *). > Uh... If you write a "conversion function", you may as well make it so > it converts the "sig=" line to a plain signature line in 3.3, which > avoids the issue entirely. Yeah, that's an improvement, though it makes the conversion function a lot more complicated, and presumably uses more memory. > (and how would that conversion function be shipped to the user anyway? > Python 3.3 and the stable ABI don't have it) As a C function in a text file, that they'd have to copy into their program. I know it's ugly. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue Feb 4 11:21:51 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Feb 2014 02:21:51 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: References: <52EFAB13.4020405@hastings.org> <52F04262.4010806@hastings.org> <52F0AF1A.1060005@hastings.org> Message-ID: <52F0BF3F.2000500@hastings.org> On 02/04/2014 01:41 AM, Georg Brandl wrote: > Clever, but due to the "hidden" space it also increases the frustration factor > for people trying to find out "why is this accepted as a signature and not this". > > I don't think a well-chosen visible separator is worse off, such as "--\n". I could live with that. To be explicit: the signature would then be of the form (" at the front. If it found it it'd scan forwards in the docstring for ")\n--\n". If it found *that*, then it would declare success. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue Feb 4 10:51:26 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Feb 2014 01:51:26 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <20140203160548.GA9906@sleipnir.bytereef.org> References: <52EFAB13.4020405@hastings.org> <20140203160548.GA9906@sleipnir.bytereef.org> Message-ID: <52F0B81E.4070300@hastings.org> On 02/03/2014 08:05 AM, Stefan Krah wrote: > I think we may slowly get into PEP territory here. Just imagine that > we settle on X, then decide at a later point to have a standard way of > adding type annotations, then find that X does not work because of (unknown). > > I'm mentioning this because signatures get really interesting for me > if they contain type information. I simultaneously share your interest, and also suspect that maybe Python is the wrong language for that. After all, Python has always been about duck-typing. Even if it did happen, it won't be for quite a while yet. The logical mechanism for type information in pure Python is annotations, and afaik they're not getting any large-scale real-world use for type annotating. (If I'm misinformed I'd love to hear counterexamples.) //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Feb 4 13:14:30 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 4 Feb 2014 13:14:30 +0100 Subject: [Python-Dev] The docstring hack for signature information has to go References: <52EFAB13.4020405@hastings.org> <52F04262.4010806@hastings.org> <52F0AF1A.1060005@hastings.org> <52F0BF3F.2000500@hastings.org> Message-ID: <20140204131430.1cdcf91a@fsol> On Tue, 04 Feb 2014 02:21:51 -0800 Larry Hastings wrote: > On 02/04/2014 01:41 AM, Georg Brandl wrote: > > Clever, but due to the "hidden" space it also increases the frustration factor > > for people trying to find out "why is this accepted as a signature and not this". > > > > I don't think a well-chosen visible separator is worse off, such as "--\n". > > I could live with that. To be explicit: the signature would then be of > the form > > > The scanning function would look for "(" at the > front. If it found it it'd scan forwards in the docstring for > ")\n--\n". If it found *that*, then it would declare success. This would have to be checked for layout regressions. If the docstring is formatted using a ReST-to-HTML converter, what will be the effect? Regards Antoine. From solipsis at pitrou.net Tue Feb 4 13:17:59 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 4 Feb 2014 13:17:59 +0100 Subject: [Python-Dev] __doc__ regression Message-ID: <20140204131759.50ed8101@fsol> Hello, Following the previous clinic thread, I realize that the latest signature improvements actually entail a regression in __doc__. Compare: $ ./python -c "print(dict.fromkeys.__doc__)" Returns a new dict with keys from iterable and values equal to value. $ python3.3 -c "print(dict.fromkeys.__doc__)" dict.fromkeys(S[,v]) -> New dict with keys from S and values equal to v. v defaults to None. As you see the signature string has vanished from the __doc__ contents. This means that any tool directly processing __doc__ attributes to generate (e.g.) automatic API docs will produce less useful docs. I think the signature should be restored, and a smarter way of reconciling __doc__ and the signature for help() should be found. Regards Antoine. From larry at hastings.org Tue Feb 4 14:08:28 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Feb 2014 05:08:28 -0800 Subject: [Python-Dev] __doc__ regression In-Reply-To: <20140204131759.50ed8101@fsol> References: <20140204131759.50ed8101@fsol> Message-ID: <52F0E64C.2040609@hastings.org> On 02/04/2014 04:17 AM, Antoine Pitrou wrote: > As you see the signature string has vanished from the __doc__ contents. > This means that any tool directly processing __doc__ attributes to > generate (e.g.) automatic API docs will produce less useful docs. Why couldn't these tools use inspect.Signature? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Feb 4 14:19:38 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 4 Feb 2014 14:19:38 +0100 Subject: [Python-Dev] __doc__ regression In-Reply-To: <52F0E64C.2040609@hastings.org> References: <20140204131759.50ed8101@fsol> <52F0E64C.2040609@hastings.org> Message-ID: 2014-02-04 Larry Hastings : > Why couldn't these tools use inspect.Signature? inspect.Signature was added in Python 3.3. Python 2 is still widely used, and some Linux distro only provide Python 3.2. By the way, help(dict.fromkeys) looks to use __doc__, not the signature, because the prototype is also missing. It's a regression. Victor From solipsis at pitrou.net Tue Feb 4 14:24:57 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 4 Feb 2014 14:24:57 +0100 Subject: [Python-Dev] __doc__ regression References: <20140204131759.50ed8101@fsol> <52F0E64C.2040609@hastings.org> Message-ID: <20140204142457.16a6bf5e@fsol> On Tue, 04 Feb 2014 05:08:28 -0800 Larry Hastings wrote: > On 02/04/2014 04:17 AM, Antoine Pitrou wrote: > > As you see the signature string has vanished from the __doc__ contents. > > This means that any tool directly processing __doc__ attributes to > > generate (e.g.) automatic API docs will produce less useful docs. > > Why couldn't these tools use inspect.Signature? Because it's new in 3.3? Most Python code has been written before that. Regards Antoine. From lucas.vacek at gmail.com Tue Feb 4 15:03:18 2014 From: lucas.vacek at gmail.com (Lukas Vacek) Date: Tue, 4 Feb 2014 14:03:18 +0000 Subject: [Python-Dev] expat symbols should be namespaced in pyexpat (Issue19186 - no progress in four months) Message-ID: Hi everyone, Just wondering if anyone has looked into http://bugs.python.org/issue19186- priority has been changed to critical four months ago but nothing has happened since. I think it would be nice to get this sorted before python3.4 release Cheers, Lucas -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Tue Feb 4 15:37:35 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 04 Feb 2014 23:37:35 +0900 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: References: <52EFAB13.4020405@hastings.org> <52F04262.4010806@hastings.org> <52F0AF1A.1060005@hastings.org> Message-ID: <87iosv54gg.fsf@uwakimon.sk.tsukuba.ac.jp> Georg Brandl writes: > I don't think a well-chosen visible separator is worse off, such as "--\n". Don't you mean "-- \n"? L'Ancien Mail-guique-ly y'rs, From ncoghlan at gmail.com Tue Feb 4 15:44:52 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 5 Feb 2014 00:44:52 +1000 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52EFBE1B.9060606@hastings.org> References: <52EFAB13.4020405@hastings.org> <20140203100815.0959f878@limelight.wooz.org> <52EFBE1B.9060606@hastings.org> Message-ID: On 4 February 2014 02:04, Larry Hastings wrote: > On 02/03/2014 07:08 AM, Barry Warsaw wrote: > > On Feb 03, 2014, at 06:43 AM, Larry Hastings wrote: > > But that only fixes part of the problem. Our theoretical extension that > wants to be binary-compatible with 3.3 and 3.4 still has a problem: how can > they support signatures? They can't give PyMethodDefEx structures to 3.3, > it > will blow up. But if they don't use PyMethodDefEx, they can't have > signatures. > > Can't an extension writer #ifdef around this? Yeah, it's ugly, but it's a > pretty standard approach for making C extensions multi-version compatible. > > > For source compatibility, yes. But I thought the point of the binary ABI > was to allow compiling a single extension that worked unmodified with > multiple versions of Python. If we simply don't support that, then an ifdef > would be fine. Then the solution appears straightforward to me: Python 3.4 will not support providing introspection information through the stable ABI. If you want to provide signature info for your C extension without an odd first line in your 3.3 docstring, you must produce version specific binaries (which allows #ifdef hackery). Then PEP 457 can address this properly for 3.5 along with the other issues it needs to cover. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From amilani at izsvenezie.it Tue Feb 4 16:15:09 2014 From: amilani at izsvenezie.it (Milani Adelaide) Date: Tue, 4 Feb 2014 16:15:09 +0100 Subject: [Python-Dev] test whatever failed Message-ID: <9EE3DB6458618749A38D4E0287711CE7382F638EAB@exchangesrv.elect.local> Dear support, we tried to install Phyton-3.3.3 but we got some problems with "make test". Test whatever failed. We trid also "make testall" but we got: Ran 45 tests in 7.897s FAILED (failures=25, skipped=15) test test_gdb failed make: *** [testall] Errore 1 Please could you help us? Thanks Adelaide Adelaide Milani, PhD student Division of Comparative Biomedical Sciences. OIE/FAO and National Reference Laboratory for Avian Influenza and Newcastle Disease FAO Reference Centre for Rabies OIE Collaborating Centre for Infectious Diseases at the Human- Animal Interface Istituto Zooprofilattico Sperimentale delle Venezie Viale dell'Universit?, 10, 35020, Legnaro (Padova), Italy Tel: 0039 049 8084381 Fax 0039 049 8084360 From g.brandl at gmx.net Tue Feb 4 18:30:18 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 04 Feb 2014 18:30:18 +0100 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <20140204131430.1cdcf91a@fsol> References: <52EFAB13.4020405@hastings.org> <52F04262.4010806@hastings.org> <52F0AF1A.1060005@hastings.org> <52F0BF3F.2000500@hastings.org> <20140204131430.1cdcf91a@fsol> Message-ID: Am 04.02.2014 13:14, schrieb Antoine Pitrou: > On Tue, 04 Feb 2014 02:21:51 -0800 > Larry Hastings wrote: >> On 02/04/2014 01:41 AM, Georg Brandl wrote: >> > Clever, but due to the "hidden" space it also increases the frustration factor >> > for people trying to find out "why is this accepted as a signature and not this". >> > >> > I don't think a well-chosen visible separator is worse off, such as "--\n". >> >> I could live with that. To be explicit: the signature would then be of >> the form >> >> > >> The scanning function would look for "(" at the >> front. If it found it it'd scan forwards in the docstring for >> ")\n--\n". If it found *that*, then it would declare success. > > This would have to be checked for layout regressions. If the docstring > is formatted using a ReST-to-HTML converter, what will be the effect? The "--" will be added after the signature in the same paragraph. However, I don't think this is a valid concern: if you process signatures as ReST you will already have to deal with lots of markup errors (e.g. due to unpaired "*" and "**"). Tools that extract the docstrings and treat them specially (such as Sphinx) will adapt anyway. Georg From g.brandl at gmx.net Tue Feb 4 18:34:34 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 04 Feb 2014 18:34:34 +0100 Subject: [Python-Dev] __doc__ regression In-Reply-To: References: <20140204131759.50ed8101@fsol> <52F0E64C.2040609@hastings.org> Message-ID: Am 04.02.2014 14:19, schrieb Victor Stinner: > 2014-02-04 Larry Hastings : >> Why couldn't these tools use inspect.Signature? > > inspect.Signature was added in Python 3.3. Python 2 is still widely > used, and some Linux distro only provide Python 3.2. Well, Python 2 won't be able to introspect C modules written and compiled for Python 3.x anyway. And __text_signature__ can be used by any Python 3.x with no loss of expressivity (handling strings in both cases). Georg From brett at python.org Tue Feb 4 19:16:59 2014 From: brett at python.org (Brett Cannon) Date: Tue, 4 Feb 2014 13:16:59 -0500 Subject: [Python-Dev] test whatever failed In-Reply-To: <9EE3DB6458618749A38D4E0287711CE7382F638EAB@exchangesrv.elect.local> References: <9EE3DB6458618749A38D4E0287711CE7382F638EAB@exchangesrv.elect.local> Message-ID: This is a mailing list for the development *of* Python, not *with* Python. The best way to get help would be to email python-list at python.org or python-help at python.org. On Tue, Feb 4, 2014 at 10:15 AM, Milani Adelaide wrote: > Dear support, > we tried to install Phyton-3.3.3 but we got some problems with "make > test". Test whatever failed. > > We trid also "make testall" but we got: > > Ran 45 tests in 7.897s > > FAILED (failures=25, skipped=15) > test test_gdb failed > make: *** [testall] Errore 1 > > Please could you help us? > Thanks > Adelaide > > > Adelaide Milani, PhD student > Division of Comparative Biomedical Sciences. > OIE/FAO and National Reference Laboratory for Avian Influenza and > Newcastle Disease > FAO Reference Centre for Rabies > OIE Collaborating Centre for Infectious Diseases at the Human- Animal > Interface > Istituto Zooprofilattico Sperimentale delle Venezie > Viale dell'Universit?, 10, 35020, Legnaro (Padova), Italy > Tel: 0039 049 8084381 > Fax 0039 049 8084360 > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Feb 4 23:08:36 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 04 Feb 2014 17:08:36 -0500 Subject: [Python-Dev] expat symbols should be namespaced in pyexpat (Issue19186 - no progress in four months) In-Reply-To: References: Message-ID: On 2/4/2014 9:03 AM, Lukas Vacek wrote: > Hi everyone, > > Just wondering if anyone has looked into > http://bugs.python.org/issue19186 - priority has been changed to > critical four months ago but nothing has happened since. > > I think it would be nice to get this sorted before python3.4 release Benjamin applied patch and closed issue. -- Terry Jan Reedy From larry at hastings.org Tue Feb 4 23:49:01 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Feb 2014 14:49:01 -0800 Subject: [Python-Dev] __doc__ regression In-Reply-To: References: <20140204131759.50ed8101@fsol> <52F0E64C.2040609@hastings.org> Message-ID: <52F16E5D.3070005@hastings.org> On 02/04/2014 05:19 AM, Victor Stinner wrote: > 2014-02-04 Larry Hastings : >> Why couldn't these tools use inspect.Signature? > inspect.Signature was added in Python 3.3. Python 2 is still widely > used, and some Linux distro only provide Python 3.2. > > By the way, help(dict.fromkeys) looks to use __doc__, not the > signature, because the prototype is also missing. > > It's a regression. In 3.4, inspect.getfullargspec and inspect.getargspec are being reimplemented using inspect.Signature. I don't understand your bringing up Python 2 and Python 3.2. Are there programs that run under Python 2 and Python 3.2 that examine docstrings from Python 3.4? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed Feb 5 01:43:12 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Feb 2014 16:43:12 -0800 Subject: [Python-Dev] __doc__ regression In-Reply-To: <20140204142457.16a6bf5e@fsol> References: <20140204131759.50ed8101@fsol> <52F0E64C.2040609@hastings.org> <20140204142457.16a6bf5e@fsol> Message-ID: <52F18920.9000203@hastings.org> On 02/04/2014 05:24 AM, Antoine Pitrou wrote: > On Tue, 04 Feb 2014 05:08:28 -0800 > Larry Hastings wrote: >> On 02/04/2014 04:17 AM, Antoine Pitrou wrote: >>> As you see the signature string has vanished from the __doc__ contents. >>> This means that any tool directly processing __doc__ attributes to >>> generate (e.g.) automatic API docs will produce less useful docs. >> Why couldn't these tools use inspect.Signature? > Because it's new in 3.3? Most Python code has been written before that. inspect.getargspec and inspect.getfullargspec in 3.4 use inspect.Signature to get the signatures of builtins. Does this address your concerns? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed Feb 5 04:20:21 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 04 Feb 2014 19:20:21 -0800 Subject: [Python-Dev] Decision on the docstring hack (was Re: The docstring hack for signature information has to go) In-Reply-To: References: <52EFAB13.4020405@hastings.org> <52F04262.4010806@hastings.org> <52F0AF1A.1060005@hastings.org> <52F0BF3F.2000500@hastings.org> <20140204131430.1cdcf91a@fsol> Message-ID: <52F1ADF5.50506@hastings.org> In the absence of a ruling from above I'm making a decision. I say we keep the docstring hack, but go back to the human-readable version, only this time with a *much* stricter signature. That should reduce the number of false positives to zero. Furthermore, I say we don't add any flags--adding a flag to the callable would mean doing it three different ways, and adding a flag to the type might not work in all cases. And finally, I agree with Nick, that publishing signatures shall not be a supported API for Python 3.4. If external modules publish signatures, and we break their support in 3.5, it's on them. The rules enforced on the signature will be as per Georg's suggestion but made slightly more stringent: The signature must start with "(". The signature must end with ")\n--\n\n". The signature may contain newlines, but must not contain empty lines. The signature may have non-Python syntax in it, such as "$" indicating a self parameter, "/" to indicate positional-only parameters, and "[" and "]" to indicate optional groups (though I don't expect 3.4 will ship with support for this last one). Here it is in non-working pseudo-code: s = function.docstring if not s.startswith(function.name + "(") return failure end = s.find(")\n--\n\n") if end < 0: return failure newline = s.find("n\n") if newline > 0 and newline < end: return failure return success (The actual implementation will be in C, in find_signature in Objects/typeobject.c.) Expect a patch in about 24 hours, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Feb 5 05:48:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 5 Feb 2014 14:48:05 +1000 Subject: [Python-Dev] Decision on the docstring hack (was Re: The docstring hack for signature information has to go) In-Reply-To: <52F1ADF5.50506@hastings.org> References: <52EFAB13.4020405@hastings.org> <52F04262.4010806@hastings.org> <52F0AF1A.1060005@hastings.org> <52F0BF3F.2000500@hastings.org> <20140204131430.1cdcf91a@fsol> <52F1ADF5.50506@hastings.org> Message-ID: Sounds like a good compromise to me - and we certainly have a lot of interesting experience to feed into the design of PEP 457 for 3.5 ;) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Feb 5 14:52:38 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 05 Feb 2014 14:52:38 +0100 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52EFAB13.4020405@hastings.org> References: <52EFAB13.4020405@hastings.org> Message-ID: <52F24226.4010903@v.loewis.de> Am 03.02.14 15:43, schrieb Larry Hastings: > A: We create a PyMethodDefEx structure with an extra field: "const char > *signature". We add a new METH_SIGNATURE (maybe just METH_SIG?) flag to > the flags, indicating that this is an extended structure. When > iterating over the PyMethodDefs, we know how far to advance the pointer > based on this flag. > > B: Same as A, but we add three unused pointers (void *reserved1 etc) to > PyMethodDefEx to give us some room to grow. > > C: Same as A, but we add two fields to PyMethodDefEx. The second new > field identifies the "version" of the structure, telling us its size > somehow. Like the lStructSize field of the OPENFILENAME structure in > Win32. I suspect YAGNI. D: Add a new type slot for method signatures. This would be a tp_signature field, along with a new slot id Py_tp_signature. The signature field itself could be struct PyMethodSignature { char *method_name; char *method_signature; }; Regards, Martin From g.brandl at gmx.net Wed Feb 5 17:04:12 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 05 Feb 2014 17:04:12 +0100 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52F24226.4010903@v.loewis.de> References: <52EFAB13.4020405@hastings.org> <52F24226.4010903@v.loewis.de> Message-ID: Am 05.02.2014 14:52, schrieb "Martin v. L?wis": > Am 03.02.14 15:43, schrieb Larry Hastings: >> A: We create a PyMethodDefEx structure with an extra field: "const char >> *signature". We add a new METH_SIGNATURE (maybe just METH_SIG?) flag to >> the flags, indicating that this is an extended structure. When >> iterating over the PyMethodDefs, we know how far to advance the pointer >> based on this flag. >> >> B: Same as A, but we add three unused pointers (void *reserved1 etc) to >> PyMethodDefEx to give us some room to grow. >> >> C: Same as A, but we add two fields to PyMethodDefEx. The second new >> field identifies the "version" of the structure, telling us its size >> somehow. Like the lStructSize field of the OPENFILENAME structure in >> Win32. I suspect YAGNI. > > D: Add a new type slot for method signatures. This would be a > tp_signature field, along with a new slot id Py_tp_signature. The > signature field itself could be > > struct PyMethodSignature { > char *method_name; > char *method_signature; > }; Mostly unrelated question while seeing the "char *" here: do we (or do we want to) support non-ASCII names for functions implemented in C? Georg From brett at python.org Wed Feb 5 17:25:07 2014 From: brett at python.org (Brett Cannon) Date: Wed, 5 Feb 2014 11:25:07 -0500 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: References: <52EFAB13.4020405@hastings.org> <52F24226.4010903@v.loewis.de> Message-ID: On Wed, Feb 5, 2014 at 11:04 AM, Georg Brandl wrote: > Am 05.02.2014 14:52, schrieb "Martin v. L?wis": > > Am 03.02.14 15:43, schrieb Larry Hastings: > >> A: We create a PyMethodDefEx structure with an extra field: "const char > >> *signature". We add a new METH_SIGNATURE (maybe just METH_SIG?) flag to > >> the flags, indicating that this is an extended structure. When > >> iterating over the PyMethodDefs, we know how far to advance the pointer > >> based on this flag. > >> > >> B: Same as A, but we add three unused pointers (void *reserved1 etc) to > >> PyMethodDefEx to give us some room to grow. > >> > >> C: Same as A, but we add two fields to PyMethodDefEx. The second new > >> field identifies the "version" of the structure, telling us its size > >> somehow. Like the lStructSize field of the OPENFILENAME structure in > >> Win32. I suspect YAGNI. > > > > D: Add a new type slot for method signatures. This would be a > > tp_signature field, along with a new slot id Py_tp_signature. The > > signature field itself could be > > > > struct PyMethodSignature { > > char *method_name; > > char *method_signature; > > }; > > Mostly unrelated question while seeing the "char *" here: do we (or do we > want to) support non-ASCII names for functions implemented in C? > Extension modules names being non-ASCII being discussed in http://bugs.python.org/issue20485 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy.kloth at gmail.com Wed Feb 5 20:27:24 2014 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Wed, 5 Feb 2014 12:27:24 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Close #20053: ignore default pip config settings In-Reply-To: <3fJR0P0KWbz7Lkt@mail.python.org> References: <3fJR0P0KWbz7Lkt@mail.python.org> Message-ID: On Tue, Feb 4, 2014 at 6:03 AM, nick.coghlan wrote: > http://hg.python.org/cpython/rev/1b8ba1346e67 > changeset: 88955:1b8ba1346e67 > user: Nick Coghlan > date: Tue Feb 04 23:02:36 2014 +1000 > summary: > Close #20053: ignore default pip config settings > > ensurepip now sets PIP_CONFIG_FILE to os.devnull before > import pip from the wheel file. This also ensures venv > ignores the default settings when bootstrapping pip. > > files: > Lib/ensurepip/__init__.py | 9 ++++-- > Lib/test/test_ensurepip.py | 16 +++++++++++ > Lib/test/test_venv.py | 37 +++++++++++++++++++------ > Misc/NEWS | 3 ++ > 4 files changed, 53 insertions(+), 12 deletions(-) This changeset caused the Windows buildbots to go red. Excerpt from one of the logs: ====================================================================== FAIL: test_with_pip (test.test_venv.EnsurePipTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_venv.py", line 349, in test_with_pip self.assertEqual(err, "") AssertionError: 'C:\\Users\\Buildbot\\AppData\\Local\\Temp[57 chars]\r\n' != '' - C:\Users\Buildbot\AppData\Local\Temp\tmpkwelerus\Scripts\python_d.exe: No module named pip -- Jeremy Kloth From ncoghlan at gmail.com Wed Feb 5 23:41:29 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 6 Feb 2014 08:41:29 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Close #20053: ignore default pip config settings In-Reply-To: References: <3fJR0P0KWbz7Lkt@mail.python.org> Message-ID: On 6 Feb 2014 05:27, "Jeremy Kloth" wrote: > > On Tue, Feb 4, 2014 at 6:03 AM, nick.coghlan wrote: > > http://hg.python.org/cpython/rev/1b8ba1346e67 > > changeset: 88955:1b8ba1346e67 > > user: Nick Coghlan > > date: Tue Feb 04 23:02:36 2014 +1000 > > summary: > > Close #20053: ignore default pip config settings > > > > ensurepip now sets PIP_CONFIG_FILE to os.devnull before > > import pip from the wheel file. This also ensures venv > > ignores the default settings when bootstrapping pip. > > > > files: > > Lib/ensurepip/__init__.py | 9 ++++-- > > Lib/test/test_ensurepip.py | 16 +++++++++++ > > Lib/test/test_venv.py | 37 +++++++++++++++++++------ > > Misc/NEWS | 3 ++ > > 4 files changed, 53 insertions(+), 12 deletions(-) > > This changeset caused the Windows buildbots to go red. Excerpt from > one of the logs: > > ====================================================================== > FAIL: test_with_pip (test.test_venv.EnsurePipTest) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\buildbot.python.org \3.x.kloth-win64\build\lib\test\test_venv.py", > line 349, in test_with_pip > self.assertEqual(err, "") > AssertionError: 'C:\\Users\\Buildbot\\AppData\\Local\\Temp[57 chars]\r\n' != '' > - C:\Users\Buildbot\AppData\Local\Temp\tmpkwelerus\Scripts\python_d.exe: > No module named pip Yeah, I noticed that yesterday :( I already reopened the issue, and flagged it as affecting the buildbots. Unfortunately, the only idea I've had for a possible culprit turned out not to be the problem (I added a separate test specifically to check the relevant cross-platform assumption, and that passed on Windows, but this test still failed) See the last few comments on http://bugs.python.org/issue20053 for details. Cheers, Nick. > > -- > Jeremy Kloth -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Thu Feb 6 04:19:29 2014 From: larry at hastings.org (Larry Hastings) Date: Wed, 05 Feb 2014 19:19:29 -0800 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: <52F24226.4010903@v.loewis.de> References: <52EFAB13.4020405@hastings.org> <52F24226.4010903@v.loewis.de> Message-ID: <52F2FF41.7040204@hastings.org> On 02/05/2014 05:52 AM, "Martin v. L?wis" wrote: > D: Add a new type slot for method signatures. This would be a > tp_signature field, along with a new slot id Py_tp_signature. The > signature field itself could be > > struct PyMethodSignature { > char *method_name; > char *method_signature; > }; That should work too, though we'd also have to add a md_signature field to module objects. It would probably be best to merge the signature into the callable object anyway. Otherwise we'd have to go look up the signature using __name__ and __self__ / __objclass__ on demand. Maybe that isn't such a big deal, but it gets a little worse: as far as I can tell, there's no attribute on a type object one can use to find the module it lives in. So in this situation: >>> import _pickle >>> import inspect >>> inspect.signature(_pickle.Pickler) How could inspect.signature figure out that the "Pickler" type object lives in the "_pickle" module? My best guess: parsing the __qualname__, which is pretty ugly. Also, keeping the signature as a reasonably-human-readable preface to the docstring means that, if we supported this for third-party modules, they could be binary ABI compatible with 3.3 and still have something approximating the hand-written signature at the top of their docstring. Cheers, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Feb 6 20:53:00 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 06 Feb 2014 11:53:00 -0800 Subject: [Python-Dev] ReST documents Message-ID: <52F3E81C.3070108@stoneleaf.us> My apologies if this is a stupid question (actually, two stupid questions! ;) , but: - is there someplace I can look to see what all the meta commands mean? Things like :class: or .. index:: pair: class; constructor - when I click on a link (using FireFox 16.0.1) instead of showing the title of that section (such as IntEnum or object.__init__) it shows the paragraph immediately following. I think it would be more helpful to show the title as well. Is this an issue with how we are creating the docs, or with FireFox? -- ~Ethan~ From berker.peksag at gmail.com Thu Feb 6 22:51:18 2014 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Thu, 6 Feb 2014 23:51:18 +0200 Subject: [Python-Dev] ReST documents In-Reply-To: <52F3E81C.3070108@stoneleaf.us> References: <52F3E81C.3070108@stoneleaf.us> Message-ID: On Thu, Feb 6, 2014 at 9:53 PM, Ethan Furman wrote: > My apologies if this is a stupid question (actually, two stupid questions! > ;) , but: > > - is there someplace I can look to see what all the meta commands mean? > Things like :class: or .. index:: pair: class; constructor Hi, I use the following links as reference: - http://docs.python.org/devguide/documenting.html#id3 - http://sphinx-doc.org/domains.html#the-python-domain - http://sphinx-doc.org/markup/inline.html#xref-syntax --Berker From ethan at stoneleaf.us Thu Feb 6 23:04:34 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 06 Feb 2014 14:04:34 -0800 Subject: [Python-Dev] ReST documents In-Reply-To: References: <52F3E81C.3070108@stoneleaf.us> Message-ID: <52F406F2.1010109@stoneleaf.us> On 02/06/2014 01:51 PM, Berker Peksa? wrote: > > I use the following links as reference: > > - http://docs.python.org/devguide/documenting.html#id3 > - http://sphinx-doc.org/domains.html#the-python-domain > - http://sphinx-doc.org/markup/inline.html#xref-syntax Many thanks! -- ~Ethan~ From terri at zone12.com Fri Feb 7 08:09:18 2014 From: terri at zone12.com (Terri Oda) Date: Thu, 06 Feb 2014 23:09:18 -0800 Subject: [Python-Dev] Google Summer of Code 2014: Looking for core python mentors! Message-ID: <52F4869E.9050900@zone12.com> For anyone who completely isn't familiar with this: Google Summer of Code is a program where Google pays students to work on open source projects for the summer. The motto goes: "Flip bits, not burgers." Google Summer of Code applications are open for mentoring organizations, and I've already put in Python's application. You can get more info here: https://wiki.python.org/moin/SummerOfCode/2014 What I'd like to know is if anyone's got any good ideas for core python projects this year and are interested in mentoring. We always have lots of students who'd like to work on Python, but right now we don't have any mentors who are available and interested in helping students make contributions to the language (We do, however, have mentors for various python sub-projects). Let me know if this might be interesting to you. I'm happy to answer any questions, and I've got experienced mentors who aren't core python devs but who'd be willing to pair up so you wouldn't have to go it alone, just let me know! Terri PS - I'm also looking for new sub-projects and other mentors -- just ask if you want to take part this year! From status at bugs.python.org Fri Feb 7 18:07:45 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 7 Feb 2014 18:07:45 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140207170745.ECC15568FA@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-01-31 - 2014-02-07) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4501 (+17) closed 27813 (+65) total 32314 (+82) Open issues with patches: 2045 Issues opened (65) ================== #12915: Add inspect.locate and inspect.resolve http://bugs.python.org/issue12915 reopened by vinay.sajip #16779: Fix compiler warning when building extension modules on 64-bit http://bugs.python.org/issue16779 reopened by skrah #19363: Python 2.7's future_builtins.map is not compatible with Python http://bugs.python.org/issue19363 reopened by Gareth.Rees #20053: venv and ensurepip are affected by default pip config file http://bugs.python.org/issue20053 reopened by ncoghlan #20461: Argument Clinic included return converters hard code use of `` http://bugs.python.org/issue20461 opened by brett.cannon #20462: Python IDLE auto closed when creating a new file or open exist http://bugs.python.org/issue20462 opened by Bishop #20463: sqlite dumpiter dumps invalid script when virtual tables are u http://bugs.python.org/issue20463 opened by Ronny.Pfannschmidt #20464: Update distutils sample config file in Doc/install/index.rst http://bugs.python.org/issue20464 opened by zach.ware #20465: Upgrade SQLite to 3.8.3 with 3.4.0 Windows and OS X installers http://bugs.python.org/issue20465 opened by ned.deily #20466: Example in Doc/extending/embedding.rst fails to compile cleanl http://bugs.python.org/issue20466 opened by zach.ware #20467: Confusing wording about __init__ http://bugs.python.org/issue20467 opened by BreamoreBoy #20468: resource module documentation is incorrect http://bugs.python.org/issue20468 opened by Kurt.Rose #20469: ssl.getpeercert() should include extensions http://bugs.python.org/issue20469 opened by oninoshiko #20475: pystone.py in 3.4 still uses time.clock(), even though it's ma http://bugs.python.org/issue20475 opened by pfalcon #20476: If new email policies are used, default message factory should http://bugs.python.org/issue20476 opened by r.david.murray #20478: Avoid inadvertently special casing Counter in statistics modul http://bugs.python.org/issue20478 opened by ncoghlan #20479: Efficiently support weight/frequency mappings in the statistic http://bugs.python.org/issue20479 opened by ncoghlan #20480: Add ipaddress property to get name of reverse DNS PTR record http://bugs.python.org/issue20480 opened by leonn #20481: Clarify type coercion rules in statistics module http://bugs.python.org/issue20481 opened by ncoghlan #20482: smtplib.SMTP.sendmail: improve exception message http://bugs.python.org/issue20482 opened by yegle #20483: Missing network resource checks in test_urllib2 & test_smtplib http://bugs.python.org/issue20483 opened by ncoghlan #20484: test_pydoc can alter execution environment causing subsequent http://bugs.python.org/issue20484 opened by ned.deily #20485: Enable non-ASCII extension module names http://bugs.python.org/issue20485 opened by Suzumizaki #20486: msilib: can't close opened database http://bugs.python.org/issue20486 opened by gentoo90 #20487: Odd words in unittest.mock document. http://bugs.python.org/issue20487 opened by naoki #20490: Show clear error message on circular import http://bugs.python.org/issue20490 opened by cool-RR #20491: textwrap: Non-breaking space not honored http://bugs.python.org/issue20491 opened by kunkku #20493: asyncio: OverflowError('timeout is too large') http://bugs.python.org/issue20493 opened by haypo #20495: test_read_pty_output() hangs on FreeBSD 7.2 buildbot http://bugs.python.org/issue20495 opened by haypo #20496: function definition tutorial encourages bad practice http://bugs.python.org/issue20496 opened by aisaac #20497: Unclear word in socket document. http://bugs.python.org/issue20497 opened by naoki #20499: Rounding errors with statistics.variance http://bugs.python.org/issue20499 opened by oscarbenjamin #20500: assertion failed when passing an exception object to sys.exit http://bugs.python.org/issue20500 opened by xdegaye #20501: fileinput module will read whole file into memory when using f http://bugs.python.org/issue20501 opened by gromgull #20503: super behavioru and abstract base classes (either implementati http://bugs.python.org/issue20503 opened by Gerrit.Holl #20504: cgi.FieldStorage, multipart, missing Content-Length http://bugs.python.org/issue20504 opened by srittau #20505: Remove resolution from selectors and granularity from asyncio http://bugs.python.org/issue20505 opened by haypo #20506: Command to display all available Import Library http://bugs.python.org/issue20506 opened by arupchakrav #20507: TypeError from str.join has no message http://bugs.python.org/issue20507 opened by Gareth.Rees #20508: IndexError from ipaddress._BaseNetwork.__getitem__ has no mess http://bugs.python.org/issue20508 opened by Gareth.Rees #20510: Test cases in test_sys don't match the comments http://bugs.python.org/issue20510 opened by Gareth.Rees #20511: asyncio: StreamReader should use bytearray for its internal bu http://bugs.python.org/issue20511 opened by yselivanov #20513: Python 2.7. Script interruption on logoff from 0 session under http://bugs.python.org/issue20513 opened by vScherba #20515: Null pointer dereference in tkinter module http://bugs.python.org/issue20515 opened by christian.heimes #20516: Concurrent.futures base concurrency improvement (with patch) http://bugs.python.org/issue20516 opened by glangford #20517: Support errors with two filenames for errno exceptions http://bugs.python.org/issue20517 opened by larry #20518: multiple inheritance + C extension = possibly unexpected __bas http://bugs.python.org/issue20518 opened by Martin.Teichmann #20519: Replace uuid ctypes usage with an extension module. http://bugs.python.org/issue20519 opened by gustavo #20520: Readline test in test_codecs is broken http://bugs.python.org/issue20520 opened by serhiy.storchaka #20521: [PATCH] Cleanup for "dis" module documentation http://bugs.python.org/issue20521 opened by svenberkvens #20523: global .pdbrc on windows 7 not reachable out of the box http://bugs.python.org/issue20523 opened by mbyt #20524: format error messages should provide context information http://bugs.python.org/issue20524 opened by r.david.murray #20525: Got compiler warning when compiling readline module http://bugs.python.org/issue20525 opened by vajrasky #20526: python: Modules/gcmodule.c:379: visit_decref: Assertion `((gc) http://bugs.python.org/issue20526 opened by haypo #20529: Unittest displays ResourceWarning warnings when running tests, http://bugs.python.org/issue20529 opened by Bernt.R??skar.Brenna #20530: Change the text signature format (again) to be more robust http://bugs.python.org/issue20530 opened by larry #20531: TypeError in e-mail.parser when non-ASCII is present http://bugs.python.org/issue20531 opened by jason.coombs #20532: Mark all tests which use _testcapi as CPython only http://bugs.python.org/issue20532 opened by serhiy.storchaka #20534: Enum tests fail with pickle protocol 4 http://bugs.python.org/issue20534 opened by ethan.furman #20535: run_tests.py: Work around issues 20355 and 20361 http://bugs.python.org/issue20535 opened by Arfrever #20536: statistics._decimal_to_ratio() produces non-integer ratio http://bugs.python.org/issue20536 opened by skrah #20537: logging exc_info parameter should accept exception instances http://bugs.python.org/issue20537 opened by yselivanov #20538: Segfault in UTF-7 incremental decoder http://bugs.python.org/issue20538 opened by serhiy.storchaka #20539: math.factorial may throw OverflowError http://bugs.python.org/issue20539 opened by ncoghlan #20540: Python 3.3/3.4 regression in multiprocessing manager ? http://bugs.python.org/issue20540 opened by Irvin.Probst Most recent 15 issues with no replies (15) ========================================== #20536: statistics._decimal_to_ratio() produces non-integer ratio http://bugs.python.org/issue20536 #20535: run_tests.py: Work around issues 20355 and 20361 http://bugs.python.org/issue20535 #20526: python: Modules/gcmodule.c:379: visit_decref: Assertion `((gc) http://bugs.python.org/issue20526 #20523: global .pdbrc on windows 7 not reachable out of the box http://bugs.python.org/issue20523 #20521: [PATCH] Cleanup for "dis" module documentation http://bugs.python.org/issue20521 #20516: Concurrent.futures base concurrency improvement (with patch) http://bugs.python.org/issue20516 #20513: Python 2.7. Script interruption on logoff from 0 session under http://bugs.python.org/issue20513 #20511: asyncio: StreamReader should use bytearray for its internal bu http://bugs.python.org/issue20511 #20508: IndexError from ipaddress._BaseNetwork.__getitem__ has no mess http://bugs.python.org/issue20508 #20504: cgi.FieldStorage, multipart, missing Content-Length http://bugs.python.org/issue20504 #20503: super behavioru and abstract base classes (either implementati http://bugs.python.org/issue20503 #20500: assertion failed when passing an exception object to sys.exit http://bugs.python.org/issue20500 #20497: Unclear word in socket document. http://bugs.python.org/issue20497 #20496: function definition tutorial encourages bad practice http://bugs.python.org/issue20496 #20486: msilib: can't close opened database http://bugs.python.org/issue20486 Most recent 15 issues waiting for review (15) ============================================= #20540: Python 3.3/3.4 regression in multiprocessing manager ? http://bugs.python.org/issue20540 #20537: logging exc_info parameter should accept exception instances http://bugs.python.org/issue20537 #20535: run_tests.py: Work around issues 20355 and 20361 http://bugs.python.org/issue20535 #20534: Enum tests fail with pickle protocol 4 http://bugs.python.org/issue20534 #20532: Mark all tests which use _testcapi as CPython only http://bugs.python.org/issue20532 #20530: Change the text signature format (again) to be more robust http://bugs.python.org/issue20530 #20525: Got compiler warning when compiling readline module http://bugs.python.org/issue20525 #20523: global .pdbrc on windows 7 not reachable out of the box http://bugs.python.org/issue20523 #20521: [PATCH] Cleanup for "dis" module documentation http://bugs.python.org/issue20521 #20520: Readline test in test_codecs is broken http://bugs.python.org/issue20520 #20519: Replace uuid ctypes usage with an extension module. http://bugs.python.org/issue20519 #20516: Concurrent.futures base concurrency improvement (with patch) http://bugs.python.org/issue20516 #20515: Null pointer dereference in tkinter module http://bugs.python.org/issue20515 #20513: Python 2.7. Script interruption on logoff from 0 session under http://bugs.python.org/issue20513 #20511: asyncio: StreamReader should use bytearray for its internal bu http://bugs.python.org/issue20511 Top 10 most discussed issues (10) ================================= #20480: Add ipaddress property to get name of reverse DNS PTR record http://bugs.python.org/issue20480 21 msgs #20485: Enable non-ASCII extension module names http://bugs.python.org/issue20485 17 msgs #20481: Clarify type coercion rules in statistics module http://bugs.python.org/issue20481 14 msgs #20053: venv and ensurepip are affected by default pip config file http://bugs.python.org/issue20053 13 msgs #20185: Derby #17: Convert 49 sites to Argument Clinic across 13 files http://bugs.python.org/issue20185 12 msgs #20475: pystone.py in 3.4 still uses time.clock(), even though it's ma http://bugs.python.org/issue20475 10 msgs #15216: Support setting the encoding on a text stream after creation http://bugs.python.org/issue15216 9 msgs #20505: Remove resolution from selectors and granularity from asyncio http://bugs.python.org/issue20505 9 msgs #20520: Readline test in test_codecs is broken http://bugs.python.org/issue20520 9 msgs #16296: Patch to fix building on Win32/64 under VS 2010 http://bugs.python.org/issue16296 8 msgs Issues closed (62) ================== #2226: Small _abcoll Bugs / Oddities http://bugs.python.org/issue2226 closed by ncoghlan #7538: HP-UX 11.11 GCC build fails to build modules http://bugs.python.org/issue7538 closed by Arfrever #7717: Compilation fixes for IRIX http://bugs.python.org/issue7717 closed by berker.peksag #11117: Implementing Async IO http://bugs.python.org/issue11117 closed by yselivanov #12187: subprocess.wait() with a timeout uses polling on POSIX http://bugs.python.org/issue12187 closed by haypo #14515: tempfile.TemporaryDirectory documented as returning object but http://bugs.python.org/issue14515 closed by r.david.murray #14665: faulthandler prints tracebacks in reverse order http://bugs.python.org/issue14665 closed by ncoghlan #14854: faulthandler: fatal error with "SystemError: null argument to http://bugs.python.org/issue14854 closed by haypo #17159: Remove explicit type check from inspect.Signature.from_functio http://bugs.python.org/issue17159 closed by yselivanov #17162: Py_LIMITED_API needs a PyType_GenericDealloc http://bugs.python.org/issue17162 closed by loewis #18801: inspect.classify_class_attrs() misclassifies object.__new__() http://bugs.python.org/issue18801 closed by yselivanov #18891: Master patch for content manager addition to email package http://bugs.python.org/issue18891 closed by r.david.murray #19186: expat symbols should be namespaced in pyexpat again http://bugs.python.org/issue19186 closed by python-dev #19320: Tkinter tests ran with wantobjects is false http://bugs.python.org/issue19320 closed by serhiy.storchaka #19334: test_asyncio hanging for 1 hour (non-AIX version) http://bugs.python.org/issue19334 closed by haypo #19652: test_asyncio: test_subprocess_send_signal() hangs on buildbot http://bugs.python.org/issue19652 closed by haypo #19761: test_tk fails on OS X with multiple test case failures with bo http://bugs.python.org/issue19761 closed by serhiy.storchaka #19863: Missing function attributes in 2.7 docs. http://bugs.python.org/issue19863 closed by mark.dickinson #19920: TarFile.list() fails on some files http://bugs.python.org/issue19920 closed by serhiy.storchaka #20029: asyncio.SubprocessProtocol is missing http://bugs.python.org/issue20029 closed by haypo #20102: shutil._make_zipfile possible resource leak http://bugs.python.org/issue20102 closed by python-dev #20136: Logging: StreamHandler does not use OS line separator. http://bugs.python.org/issue20136 closed by vinay.sajip #20137: Logging: RotatingFileHandler computes string length instead of http://bugs.python.org/issue20137 closed by vinay.sajip #20162: Test test_hash_distribution fails on RHEL 6.5 / ppc64 http://bugs.python.org/issue20162 closed by haypo #20311: epoll.poll(timeout) and PollSelector.select(timeout) must roun http://bugs.python.org/issue20311 closed by haypo #20319: concurrent.futures.wait() can block forever even if Futures ha http://bugs.python.org/issue20319 closed by bquinlan #20354: tracemalloc causes segfault in "make profile-opt" http://bugs.python.org/issue20354 closed by haypo #20363: BytesWarnings triggerred by test suite http://bugs.python.org/issue20363 closed by serhiy.storchaka #20368: Tkinter: handle the null character http://bugs.python.org/issue20368 closed by serhiy.storchaka #20400: Add create_read_pipe_protocol/create_write_pipe_protocol to as http://bugs.python.org/issue20400 closed by haypo #20404: Delayed exception using non-text encodings with TextIOWrapper http://bugs.python.org/issue20404 closed by python-dev #20412: Enum and IntEnum classes are not defined in the documentation http://bugs.python.org/issue20412 closed by python-dev #20423: io.StringIO newline param has wrong default http://bugs.python.org/issue20423 closed by pitrou #20426: Compiling a regex with re.DEBUG should force a recompile http://bugs.python.org/issue20426 closed by pitrou #20435: Discrepancy between io.StringIO and _pyio.StringIO with univer http://bugs.python.org/issue20435 closed by pitrou #20445: HAVE_BROKEN_NICE detected incorrectly due to configure.ac typo http://bugs.python.org/issue20445 closed by George.Kouryachy #20452: test_timeout_rounding() of test_asyncio fails on "x86 Ubuntu S http://bugs.python.org/issue20452 closed by haypo #20455: test_asyncio hangs on Windows http://bugs.python.org/issue20455 closed by haypo #20456: Argument Clinic rollup patch, 2014/01/31 http://bugs.python.org/issue20456 closed by larry #20460: Wrong markup in c-api/arg.rst http://bugs.python.org/issue20460 closed by zach.ware #20470: "Fatal Python error: Bus error" on the SPARC Solaris 10 buildb http://bugs.python.org/issue20470 closed by haypo #20471: test_signature_on_class() of test_inspect fails on "AMD64 Free http://bugs.python.org/issue20471 closed by yselivanov #20472: test_write_pty() of test_asyncio fails on "x86 Tiger 3.x" buil http://bugs.python.org/issue20472 closed by haypo #20473: inspect.Signature no longer handles builtin classes correctly http://bugs.python.org/issue20473 closed by yselivanov #20474: test_socket failures on OS X due to fixed "expected failures" http://bugs.python.org/issue20474 closed by ned.deily #20477: Add examples of using the new contentmanager api to the email http://bugs.python.org/issue20477 closed by r.david.murray #20488: importlib title and introduction are out of date http://bugs.python.org/issue20488 closed by brett.cannon #20489: help() fails for zlib Compress and Decompress objects http://bugs.python.org/issue20489 closed by serhiy.storchaka #20492: test_tcl is failing on windows 2.7 http://bugs.python.org/issue20492 closed by serhiy.storchaka #20494: Backport of http://bugs.python.org/issue11849 to Python 2.7 (M http://bugs.python.org/issue20494 closed by python-dev #20498: Update StringIO newline tests http://bugs.python.org/issue20498 closed by serhiy.storchaka #20502: Context.create_decimal_from_float() inconsistent precision for http://bugs.python.org/issue20502 closed by mark.dickinson #20509: logging.config.fileConfig() docs could link to the config file http://bugs.python.org/issue20509 closed by python-dev #20512: Python3.3 segfaults when using big5hkscs encoding http://bugs.python.org/issue20512 closed by ned.deily #20514: Errno 10013 - ipython notebook http://bugs.python.org/issue20514 closed by ned.deily #20522: pandas version 0.13.0 does not give warning when summing float http://bugs.python.org/issue20522 closed by mark.dickinson #20527: multiprocessing.Queue deadlocks after ???reader??? process dea http://bugs.python.org/issue20527 closed by sbt #20528: fileinput module will read whole file into memory when using f http://bugs.python.org/issue20528 closed by r.david.murray #20533: bug in float arithmetic? http://bugs.python.org/issue20533 closed by zach.ware #20541: os.path.exists() gives wrong answer for Windows special files http://bugs.python.org/issue20541 closed by ncoghlan #20542: Assertion failure in test_readline http://bugs.python.org/issue20542 closed by ncoghlan #1028088: Cookies without values are silently ignored (by design?) http://bugs.python.org/issue1028088 closed by berker.peksag From francismb at email.de Sat Feb 8 12:37:55 2014 From: francismb at email.de (francis) Date: Sat, 08 Feb 2014 12:37:55 +0100 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <20140207170745.ECC15568FA@psf.upfronthosting.co.za> References: <20140207170745.ECC15568FA@psf.upfronthosting.co.za> Message-ID: <52F61713.8000800@email.de> On 02/07/2014 06:07 PM, Python tracker wrote: > Open issues with patches: 2045 Has somebody done a graphic of that data againsttime? Regards, francis From guido at python.org Sun Feb 9 00:32:23 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Feb 2014 15:32:23 -0800 Subject: [Python-Dev] Fwd: [python-tulip] Need help to finish asyncio documentation In-Reply-To: References: Message-ID: We could really use more help reviewing and finishing asyncio's docs! ---------- Forwarded message ---------- From: Victor Stinner Date: Sat, Feb 8, 2014 at 2:38 PM Subject: [python-tulip] Need help to finish asyncio documentation To: python-tulip Hi, I wrote most parts of the documentation of the asyncio module, but I'm not sure that anyone already read it yet. Can you please at least take at look? http://docs.python.org/dev/library/asyncio.html Tell me if the documentation needs more examples. I don't want to add whole applications, only very short examples to explain one feature or concept, or show how to use one specific API. I just realized that add/remove_reader/writer() methods of the event loop were not documented, sock_recv/sendall/accept/connect() methods neither. I documented them. There are still many functions which only have "XXX" for documentation. If you would like to contribute, send patches on .rst files. The source of the documentation is in the Doc/library/ directory of CPython repository: http://hg.python.org/cpython/ Files asyncio-*.rst: http://hg.python.org/cpython/file/default/Doc/library Victor -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Feb 9 00:47:02 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 9 Feb 2014 00:47:02 +0100 Subject: [Python-Dev] [python-tulip] Need help to finish asyncio documentation References: Message-ID: <20140209004702.464e83d0@fsol> On Sat, 8 Feb 2014 15:32:23 -0800 Guido van Rossum wrote: > We could really use more help reviewing and finishing asyncio's docs! Well, it's probably difficult for people to help when they are not acquainted with the details of asyncio's functioning :-) Regards Antoine. From guido at python.org Sun Feb 9 00:54:03 2014 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Feb 2014 15:54:03 -0800 Subject: [Python-Dev] [python-tulip] Need help to finish asyncio documentation In-Reply-To: <20140209004702.464e83d0@fsol> References: <20140209004702.464e83d0@fsol> Message-ID: On Sat, Feb 8, 2014 at 3:47 PM, Antoine Pitrou wrote: > On Sat, 8 Feb 2014 15:32:23 -0800 > Guido van Rossum wrote: > > We could really use more help reviewing and finishing asyncio's docs! > > Well, it's probably difficult for people to help when they are not > acquainted with the details of asyncio's functioning :-) > Very funny. :-) But seriously: as a potential reviewer, you can try to write a simple asyncio app from scratch using the docs available, and report (preferably on the python-tulip list) any issues you encountered, whether they be English spelling or grammar (if those are your thing), confusing on incorrect documentation, missing or buggy examples, and so on. Plenty of people so far have contributed *code*, even without documentation (apart from PEP 3156), and it should be just as easy to contribute *docs*. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sun Feb 9 01:00:30 2014 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 09 Feb 2014 00:00:30 +0000 Subject: [Python-Dev] Fwd: [python-tulip] Need help to finish asyncio documentation In-Reply-To: References: Message-ID: <52F6C51E.5020607@mrabarnett.plus.com> On 2014-02-08 23:32, Guido van Rossum wrote: > We could really use more help reviewing and finishing asyncio's docs! > Some spelling mistakes: http://docs.python.org/dev/library/asyncio.html mimicks http://docs.python.org/dev/library/asyncio-task.html returing nummber http://docs.python.org/dev/library/asyncio-protocol.html correspondong http://docs.python.org/dev/library/asyncio-stream.html Sublclass http://docs.python.org/dev/library/asyncio-subprocess.html subproces signale > ---------- Forwarded message ---------- > From: *Victor Stinner* > > Date: Sat, Feb 8, 2014 at 2:38 PM > Subject: [python-tulip] Need help to finish asyncio documentation > To: python-tulip > > > > Hi, > > I wrote most parts of the documentation of the asyncio module, but I'm > not sure that anyone already read it yet. Can you please at least take > at look? > http://docs.python.org/dev/library/asyncio.html > > Tell me if the documentation needs more examples. I don't want to add > whole applications, only very short examples to explain one feature or > concept, or show how to use one specific API. > > I just realized that add/remove_reader/writer() methods of the event > loop were not documented, sock_recv/sendall/accept/connect() methods > neither. I documented them. > > There are still many functions which only have "XXX" for documentation. > > If you would like to contribute, send patches on .rst files. The > source of the documentation is in the Doc/library/ directory of > CPython repository: > http://hg.python.org/cpython/ > > Files asyncio-*.rst: > http://hg.python.org/cpython/file/default/Doc/library > > Victor > > > > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/python%40mrabarnett.plus.com > > > > No virus found in this message. > Checked by AVG - www.avg.com > Version: 2014.0.4259 / Virus Database: 3697/7075 - Release Date: 02/08/14 > From victor.stinner at gmail.com Sun Feb 9 01:05:46 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 9 Feb 2014 01:05:46 +0100 Subject: [Python-Dev] Fwd: [python-tulip] Need help to finish asyncio documentation In-Reply-To: <52F6C51E.5020607@mrabarnett.plus.com> References: <52F6C51E.5020607@mrabarnett.plus.com> Message-ID: 2014-02-09 1:00 GMT+01:00 MRAB : > Some spelling mistakes: Please, try to write a patch or it will be hard to merge fixes. Victor From brian at python.org Sun Feb 9 01:38:26 2014 From: brian at python.org (Brian Curtin) Date: Sat, 8 Feb 2014 18:38:26 -0600 Subject: [Python-Dev] Fwd: [python-tulip] Need help to finish asyncio documentation In-Reply-To: <52F6C51E.5020607@mrabarnett.plus.com> References: <52F6C51E.5020607@mrabarnett.plus.com> Message-ID: On Sat, Feb 8, 2014 at 6:00 PM, MRAB wrote: > On 2014-02-08 23:32, Guido van Rossum wrote: >> >> We could really use more help reviewing and finishing asyncio's docs! >> > Some spelling mistakes: > > http://docs.python.org/dev/library/asyncio.html > mimicks > > http://docs.python.org/dev/library/asyncio-task.html > returing > nummber > > http://docs.python.org/dev/library/asyncio-protocol.html > correspondong > > http://docs.python.org/dev/library/asyncio-stream.html > Sublclass > > http://docs.python.org/dev/library/asyncio-subprocess.html > subproces > signale Fixed: http://hg.python.org/cpython/rev/3cfaeb788e00 - thanks! From martin at v.loewis.de Sun Feb 9 12:48:14 2014 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 09 Feb 2014 12:48:14 +0100 Subject: [Python-Dev] The docstring hack for signature information has to go In-Reply-To: References: <52EFAB13.4020405@hastings.org> <52F24226.4010903@v.loewis.de> Message-ID: <52F76AFE.80706@v.loewis.de> Am 05.02.14 17:04, schrieb Georg Brandl: > Mostly unrelated question while seeing the "char *" here: do we (or do we > want to) support non-ASCII names for functions implemented in C? I didn't try, but I think it should work. methodobject.c:meth_get__name__ uses PyUnicode_FromString, which in turn decodes from UTF-8. Regards, Martin From nitikaagarwal18 at gmail.com Sun Feb 9 17:34:56 2014 From: nitikaagarwal18 at gmail.com (Nitika) Date: Sun, 9 Feb 2014 21:34:56 +0500 Subject: [Python-Dev] Getting Introduced with the Community and Looking forward to Contribute to the Project as part of Gsoc 2014 Message-ID: Hi, Firstly I, Nitika would like to introduce myself to the developers of this commmunity. I am currently pursuing Computer Science and Engineering at Netaji Subhash Institute of Technology INDIA. I have been following the discussions on the mailing list for past few months. I had got an indepth knowledge of the tools used for the development of open source software. My skills include Programming languages: GNU C/C++, Python, Javascript, Version control systems Git/Github and SVN. I am also aware with many of the web development application tools and content management systems. I am an ambitous person willing to learn other technologies when need in future. I choose this organisation for the contribution as the work the community does looks very appealing and interesting to me and useful for the whole community. The skills used by the organisation for the development of open source software matches with my skills to much extent. I had installed the required necessary tools and setup the environment in my system needed for the contribution to open source software. I had got myself aware of the source to some extent and had forked on my github account. I had also read the contribution, development tutorial on the community webpage. I would like to seek the mentors help in the selection of the project for which the idea has been proposed for the Gsoc and someone is willing to mentor the project idea. I will highly appreciate with any kind of help, guidance from the mentors to help me move forward to contribute to the software starting off with some easy tasks and then moving levels higher so as to make a strong proposal for the upcoming Google Summer of Code 2014. [1] Github : https://github.com/NitikaAgarwal [2] IRC : nitika18 *--Nitika Agarwal* -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Sun Feb 9 19:48:34 2014 From: brian at python.org (Brian Curtin) Date: Sun, 9 Feb 2014 12:48:34 -0600 Subject: [Python-Dev] Getting Introduced with the Community and Looking forward to Contribute to the Project as part of Gsoc 2014 In-Reply-To: References: Message-ID: On Sun, Feb 9, 2014 at 10:34 AM, Nitika wrote: > I had got myself aware of the source to some extent and had forked on my github account. The python source isn't forked in your github. A Github mirror of the Mercurial repository (hg.python.org) is available at https://github.com/python/cpython From techtonik at gmail.com Mon Feb 10 07:23:51 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 10 Feb 2014 09:23:51 +0300 Subject: [Python-Dev] PyPI offline, status is ok Message-ID: http://status.python.org/ shows all green https://pypi.python.org/pypi/gazest shows Error 503 backend read error backend read error Guru Meditation: XID: 2792709923 ________________________________ Varnish cache server https://pypi.python.org/pypi/ shows XID: 4199593736 -- anatoly t. From rosuav at gmail.com Mon Feb 10 11:42:19 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 10 Feb 2014 21:42:19 +1100 Subject: [Python-Dev] PyPI offline, status is ok In-Reply-To: References: Message-ID: On Mon, Feb 10, 2014 at 5:23 PM, anatoly techtonik wrote: > http://status.python.org/ shows all green > > https://pypi.python.org/pypi/gazest shows > > Error 503 backend read error > > backend read error > > Guru Meditation: > > XID: 2792709923 Working for me. But then, your email only just came through, and it's dated four hours ago, so maybe python-dev was also having trouble. Or maybe your ISP was having trouble. Hard to say now. ChrisA From techtonik at gmail.com Mon Feb 10 12:11:20 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 10 Feb 2014 14:11:20 +0300 Subject: [Python-Dev] PyPI offline, status is ok In-Reply-To: References: Message-ID: On Mon, Feb 10, 2014 at 1:42 PM, Chris Angelico wrote: > On Mon, Feb 10, 2014 at 5:23 PM, anatoly techtonik wrote: >> http://status.python.org/ shows all green >> >> https://pypi.python.org/pypi/gazest shows >> >> Error 503 backend read error >> >> backend read error >> >> Guru Meditation: >> >> XID: 2792709923 > > Working for me. But then, your email only just came through, and it's > dated four hours ago, so maybe python-dev was also having trouble. Or > maybe your ISP was having trouble. Hard to say now. ISP for sure won't run Varnish, which is enabled for Python Warehouse here: https://github.com/python/psf-chef/blob/14bbebdbbc4f7ccc65e0478458cab433594f2cfa/cookbooks/psf-pypi/recipes/warehouse.rb From rowen at uw.edu Mon Feb 10 21:47:00 2014 From: rowen at uw.edu (Russell E. Owen) Date: Mon, 10 Feb 2014 12:47:00 -0800 Subject: [Python-Dev] Fwd: [python-tulip] Need help to finish asyncio documentation References: Message-ID: In article , Guido van Rossum wrote: > We could really use more help reviewing and finishing asyncio's docs! ... > http://docs.python.org/dev/library/asyncio.html I think the documentation desperately needs an overview. I find it very confusing without one. I guess the links to PEPs are intended to be that, but it would be much more helpful to include the overview in the documentation. I think a bit of re-ordering of topics could also help: - One of the first things presented is a discussion of event loop policy, even though most users will never need to change the policy. Couldn't this go at the end? - It tells how to create and set event loops, with no hint as to why one would want to do that. Is it common to have multiple event loops? A related oddity: what is the point of BaseEventLoop.run_until_complete(future)? It sounds interesting but obscure (unless it applies to one of many event loops). In other words: if it was possible to show basic usage of event loops early, then discuss how to do the fancy stuff (new event loops, new even loop policies later) that might help. BaseEventLoop.stop() says: "Every callback scheduled before stop() is called will run." but it does not say WHEN it will run -- immediately due to the stop, or normally as if stop was never called? That said, I am very excited to have this functionality. I have been sticking with Python 2 for now, but this feature may entice me to make the switch. -- Russell > > ---------- Forwarded message ---------- > From: Victor Stinner > Date: Sat, Feb 8, 2014 at 2:38 PM > Subject: [python-tulip] Need help to finish asyncio documentation > To: python-tulip > > > Hi, > > I wrote most parts of the documentation of the asyncio module, but I'm > not sure that anyone already read it yet. Can you please at least take > at look? > http://docs.python.org/dev/library/asyncio.html > > Tell me if the documentation needs more examples. I don't want to add > whole applications, only very short examples to explain one feature or > concept, or show how to use one specific API. > > I just realized that add/remove_reader/writer() methods of the event > loop were not documented, sock_recv/sendall/accept/connect() methods > neither. I documented them. > > There are still many functions which only have "XXX" for documentation. > > If you would like to contribute, send patches on .rst files. The > source of the documentation is in the Doc/library/ directory of > CPython repository: > http://hg.python.org/cpython/ > > Files asyncio-*.rst: > http://hg.python.org/cpython/file/default/Doc/library > > Victor From georg at python.org Mon Feb 10 21:55:40 2014 From: georg at python.org (Georg Brandl) Date: Mon, 10 Feb 2014 21:55:40 +0100 Subject: [Python-Dev] [RELEASED] Python 3.3.4 Message-ID: <52F93CCC.70304@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I'm very happy to announce the release of Python 3.3.4. Python 3.3.4 includes several security fixes and over 120 bug fixes compared to the Python 3.3.3 release. This release fully supports OS X 10.9 Mavericks. In particular, this release fixes an issue that could cause previous versions of Python to crash when typing in interactive mode on OS X 10.9. Python 3.3 includes a range of improvements of the 3.x series, as well as easier porting between 2.x and 3.x. In total, almost 500 API items are new or improved in Python 3.3. For a more extensive list of changes in the 3.3 series, see http://docs.python.org/3.3/whatsnew/3.3.html To download Python 3.3.4 visit: http://www.python.org/download/releases/3.3.4/ This is a production release, please report any bugs to http://bugs.python.org/ Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.3's contributors) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlL5PMwACgkQN9GcIYhpnLCv4wCePNVqwsOYCHdJBix2bKk4PNpK GBoAnRML2x6obCssnUJe5xwuUZYw8ZSY =+/Nz -----END PGP SIGNATURE----- From guido at python.org Mon Feb 10 22:21:44 2014 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Feb 2014 13:21:44 -0800 Subject: [Python-Dev] Fwd: [python-tulip] Need help to finish asyncio documentation In-Reply-To: References: Message-ID: On Mon, Feb 10, 2014 at 12:47 PM, Russell E. Owen wrote: > In article > , > Guido van Rossum wrote: > > > We could really use more help reviewing and finishing asyncio's docs! > ... > > http://docs.python.org/dev/library/asyncio.html > > I think the documentation desperately needs an overview. I find it very > confusing without one. I guess the links to PEPs are intended to be > that, but it would be much more helpful to include the overview in the > documentation. > Thanks very much -- this is the kind of thing that's hard to notice as the author (of either the code or the docs :-). > I think a bit of re-ordering of topics could also help: > - One of the first things presented is a discussion of event loop > policy, even though most users will never need to change the policy. > Couldn't this go at the end? > Indeed. > - It tells how to create and set event loops, with no hint as to why one > would want to do that. Is it common to have multiple event loops? Shouldn't be that common (but more common than creating or even using the policy). > A related oddity: what is the point of > BaseEventLoop.run_until_complete(future)? It sounds interesting but > obscure (unless it applies to one of many event loops). > That's actually pretty common in a main() program. See several examples in the Tulip repo, e.g. http://code.google.com/p/tulip/source/browse/examples/crawl.py#851 > In other words: if it was possible to show basic usage of event loops > early, then discuss how to do the fancy stuff (new event loops, new even > loop policies later) that might help. > > BaseEventLoop.stop() says: > "Every callback scheduled before stop() is called will run." > but it does not say WHEN it will run -- immediately due to the stop, or > normally as if stop was never called? > Yeah, this is confusing -- it only applies to callbacks scheduled with call_soon(), not for those scheduled with call_later() or call_at() (and for those scheduled with call_soon_threadsafe() all bets are off, of course). it may not even be such a good idea to make this promise (though it's also in PEP 3156) -- the promise constrains the behavior of stop(). The implementation is kind of cute though: call_soon() adds things to a FIFO and the event loop picks items to run off this FIFO -- and stop() just adds a sentinel item to the FIFO. The FIFO behavior implements the promise. Also notice that if a callback schedules a further callback using call_soon(), the latter will end up in the FIFO *after* the sentinel item, so it won't run. It's also important to understand that a stopped loop can always be resumed with no ill effects -- the sentinel having been popped off the FIFO, the loop will continue to run until it is stopped again. (But there's also a bit if an issue here if you stop it twice in a row -- the second stop() call will remain in the FIFO.) Finally, scheduling something with call_later(0, ) is *not* the same as call_soon(). call_later() *always* creates a scheduler item which will require a separate "tick" of the loop before it gets moved to the above-mentioned FIFO (a.k.a. the ready queue). Perhaps the following is helpful (though it gives more implementation detail than we should probably put in the docs). The event loop repeats the following steps forever: 1. Wait for I/O ready with a deadline equal to the earliest scheduled item (zero if the ready queue is non-empty, infinity if nothing is scheduled) 2. Add I/O handlers that should be called to the ready queue 3. Move scheduled items that are now dueue to the ready queue 4. Run everything that is *currently* in the ready queue When the sentinel from stop() is found in the ready queue, the loop stops, leaving any remaining items in the ready queue. When the loop is restarted, we start from the top, with a deadline equal to zero. The other useful thing to understand is the data structures used by the event loop: - a ready queue (the aforementioned FIFO) - a heapq of scheduled items, sorted by the time they are due - a data structure to keep track of I/O callbacks (typically a selector, except when using IOCP on Windows) Much hand-wringing went into deciding what to do with callbacks put into the ready queue by call_soon() while step 4 above is running. The decision was *not* to run these immediately but to loop back to the top, using a deadline of zero for the I/O wait. This favors responding to I/O over running callbacks, but still runs the first "wave" of callbacks immediately without checking back for more I/O. > That said, I am very excited to have this functionality. I have been > sticking with Python 2 for now, but this feature may entice me to make > the switch. > That was part of the point. :-) > -- Russell > > > > > > ---------- Forwarded message ---------- > > From: Victor Stinner > > Date: Sat, Feb 8, 2014 at 2:38 PM > > Subject: [python-tulip] Need help to finish asyncio documentation > > To: python-tulip > > > > > > Hi, > > > > I wrote most parts of the documentation of the asyncio module, but I'm > > not sure that anyone already read it yet. Can you please at least take > > at look? > > http://docs.python.org/dev/library/asyncio.html > > > > Tell me if the documentation needs more examples. I don't want to add > > whole applications, only very short examples to explain one feature or > > concept, or show how to use one specific API. > > > > I just realized that add/remove_reader/writer() methods of the event > > loop were not documented, sock_recv/sendall/accept/connect() methods > > neither. I documented them. > > > > There are still many functions which only have "XXX" for documentation. > > > > If you would like to contribute, send patches on .rst files. The > > source of the documentation is in the Doc/library/ directory of > > CPython repository: > > http://hg.python.org/cpython/ > > > > Files asyncio-*.rst: > > http://hg.python.org/cpython/file/default/Doc/library > > > > Victor > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Tue Feb 11 08:43:15 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 10 Feb 2014 23:43:15 -0800 Subject: [Python-Dev] [RELEASED] Python 3.4.0 release candidate 1 Message-ID: <52F9D493.6090100@hastings.org> On behalf of the Python development team, I'm delighted to announce the first release candidate of Python 3.4. This is a preview release, and its use is not recommended for production settings. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series include: * PEP 428, a "pathlib" module providing object-oriented filesystem paths * PEP 435, a standardized "enum" module * PEP 436, a build enhancement that will help generate introspection information for builtins * PEP 442, improved semantics for object finalization * PEP 443, adding single-dispatch generic functions to the standard library * PEP 445, a new C API for implementing custom memory allocators * PEP 446, changing file descriptors to not be inherited by default in subprocesses * PEP 450, a new "statistics" module * PEP 451, standardizing module metadata for Python's module import system * PEP 453, a bundled installer for the *pip* package manager * PEP 454, a new "tracemalloc" module for tracing Python memory allocations * PEP 456, a new hash algorithm for Python strings and binary data * PEP 3154, a new and improved protocol for pickled objects * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O Python 3.4 is now in "feature freeze", meaning that no new features will be added. The final release is projected for mid-March 2014. To download Python 3.4.0rc1 visit: http://www.python.org/download/releases/3.4.0/ Please consider trying Python 3.4.0rc1 with your code and reporting any new issues you notice to: http://bugs.python.org/ Enjoy! -- Larry Hastings, Release Manager larry at hastings.org (on behalf of the entire python-dev team and 3.4's contributors) From victor.stinner at gmail.com Tue Feb 11 11:16:29 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 11 Feb 2014 11:16:29 +0100 Subject: [Python-Dev] [RELEASED] Python 3.4.0 release candidate 1 In-Reply-To: <52F9D493.6090100@hastings.org> References: <52F9D493.6090100@hastings.org> Message-ID: Hi, It would be nice to give also the link to the whole changelog in your emails and on the website: http://docs.python.org/3.4/whatsnew/changelog.html Congrats for your RC1 release :-) It's always hard to make developers stop addings "new minor" changes before the final version :-) Victor 2014-02-11 8:43 GMT+01:00 Larry Hastings : > > On behalf of the Python development team, I'm delighted to announce > the first release candidate of Python 3.4. > > This is a preview release, and its use is not recommended for > production settings. > > Python 3.4 includes a range of improvements of the 3.x series, including > hundreds of small improvements and bug fixes. Major new features and > changes in the 3.4 release series include: > > * PEP 428, a "pathlib" module providing object-oriented filesystem paths > * PEP 435, a standardized "enum" module > * PEP 436, a build enhancement that will help generate introspection > information for builtins > * PEP 442, improved semantics for object finalization > * PEP 443, adding single-dispatch generic functions to the standard library > * PEP 445, a new C API for implementing custom memory allocators > * PEP 446, changing file descriptors to not be inherited by default > in subprocesses > * PEP 450, a new "statistics" module > * PEP 451, standardizing module metadata for Python's module import system > * PEP 453, a bundled installer for the *pip* package manager > * PEP 454, a new "tracemalloc" module for tracing Python memory allocations > * PEP 456, a new hash algorithm for Python strings and binary data > * PEP 3154, a new and improved protocol for pickled objects > * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O > > Python 3.4 is now in "feature freeze", meaning that no new features will be > added. The final release is projected for mid-March 2014. > > > To download Python 3.4.0rc1 visit: > > http://www.python.org/download/releases/3.4.0/ > > > Please consider trying Python 3.4.0rc1 with your code and reporting any > new issues you notice to: > > http://bugs.python.org/ > > > Enjoy! > > -- > Larry Hastings, Release Manager > larry at hastings.org > (on behalf of the entire python-dev team and 3.4's contributors) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From mcepl at redhat.com Tue Feb 11 12:03:07 2014 From: mcepl at redhat.com (=?UTF-8?q?Mat=C4=9Bj=20Cepl?=) Date: Tue, 11 Feb 2014 12:03:07 +0100 Subject: [Python-Dev] [PATCH] Add an authorization header to the initial request. Message-ID: <1392116587-29676-1-git-send-email-mcepl@redhat.com> Many websites (e.g. GitHub API) on the Internet are intentionally not following RFC with regards to the Basic Authorization and require Authorization header in the initial request and they never return 401 error. Therefore it is not possible to authorize with such websites just using urllib2.py HTTPBasicAuthHandler as described in documentation. However, RFC 2617, end of section 2 allows pre-authorization in the initial request: > A client MAY preemptively send the corresponding Authorization > header with requests for resources in that space without > receipt of another challenge from the server. (RFC also suggests preauthorization of proxy requests, but that is not part of this patch, however it could be trivially added) Also, generating HTTP BasicAuth header has been refactored into special method of AbstractBasicAuthHandler. Suggested fix for bug# 19494 This is my first attempt to contribute to Python itself, so please be gentle with me. Yes, I know that I miss unit tests and port to other branches of Python (this is against 2.7), but I would like first some feedback to see what I am missing (aside from the mentioned). Mat?j Cepl --- Lib/urllib2.py | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/Lib/urllib2.py b/Lib/urllib2.py index aadeb73..a5feb03 100644 --- a/Lib/urllib2.py +++ b/Lib/urllib2.py @@ -848,6 +848,18 @@ class AbstractBasicAuthHandler: def reset_retry_count(self): self.retried = 0 + def generate_auth_header(self, host, req, realm): + user, pw = self.passwd.find_user_password(realm, host) + if pw is not None: + raw = "%s:%s" % (user, pw) + auth = 'Basic %s' % base64.b64encode(raw).strip() + if req.headers.get(self.auth_header, None) == auth: + return None + req.add_unredirected_header(self.auth_header, auth) + return req + else: + return None + def http_error_auth_reqed(self, authreq, host, req, headers): # host may be an authority (without userinfo) or a URL with an # authority @@ -875,14 +887,10 @@ class AbstractBasicAuthHandler: return response def retry_http_basic_auth(self, host, req, realm): - user, pw = self.passwd.find_user_password(realm, host) - if pw is not None: - raw = "%s:%s" % (user, pw) - auth = 'Basic %s' % base64.b64encode(raw).strip() - if req.headers.get(self.auth_header, None) == auth: - return None - req.add_unredirected_header(self.auth_header, auth) - return self.parent.open(req, timeout=req.timeout) + req = self.generate_auth_header(host, req, realm) + + if req is not None: + self.parent.open(req, timeout=req.timeout) else: return None @@ -898,6 +906,17 @@ class HTTPBasicAuthHandler(AbstractBasicAuthHandler, BaseHandler): self.reset_retry_count() return response + def http_request(self, req): + host = req.get_host() + + new_req = self.generate_auth_header(host, req, None) + if new_req is not None: + req = new_req + + return req + + https_request = http_request + class ProxyBasicAuthHandler(AbstractBasicAuthHandler, BaseHandler): -- 1.8.5.2.192.g7794a68 From tjreedy at udel.edu Tue Feb 11 13:27:41 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 11 Feb 2014 07:27:41 -0500 Subject: [Python-Dev] [PATCH] Add an authorization header to the initial request. In-Reply-To: <1392116587-29676-1-git-send-email-mcepl@redhat.com> References: <1392116587-29676-1-git-send-email-mcepl@redhat.com> Message-ID: On 2/11/2014 6:03 AM, Mat?j Cepl wrote: > Suggested fix for bug# 19494 > > This is my first attempt to contribute to Python itself, so please be > gentle with me. Yes, I know that I miss unit tests and port to other > branches of Python (this is against 2.7), but I would like first some > feedback to see what I am missing (aside from the mentioned). Please upload your patch to the issue. If it gets ignored there for a month, join the mentorship list and request a review. -- Terry Jan Reedy From drobinow at gmail.com Tue Feb 11 14:06:53 2014 From: drobinow at gmail.com (David Robinow) Date: Tue, 11 Feb 2014 08:06:53 -0500 Subject: [Python-Dev] [RELEASED] Python 3.4.0 release candidate 1 In-Reply-To: References: <52F9D493.6090100@hastings.org> Message-ID: On Tue, Feb 11, 2014 at 5:45 AM, Terry Reedy wrote: > On 2/11/2014 5:13 AM, Terry Reedy wrote: >> ... >> I installed 64 bit 3.3.4 yesterday with no problem. I reran it today in >> repair mode and again, no problem. >> >> With 64 bit 3.4.0, I get >> "There is a problem with this Windows Installer package. A program >> required for the install to complete could not be run." >> >> No, the generic message does not bother to say *which* program :-(. >> >> 34 bit 3.4.0 installed fine. > > > I wrote too soon. > > Python 3.4.0rc1 (v3.4.0rc1:5e088cea8660, Feb 11 2014, 05:54:25) [MSC >>>> import tkinter > Traceback ... > import _tkinter > ImportError: DLL load failed: %1 is not a valid Win32 application. > > So tkinter, Idle, turtle fail and the corresponding tests get skipped. 32 bit and 64 bit both work for me. Windows 7. From mcepl at redhat.com Tue Feb 11 14:04:54 2014 From: mcepl at redhat.com (=?UTF-8?Q?Mat=C4=9Bj?= Cepl) Date: Tue, 11 Feb 2014 14:04:54 +0100 Subject: [Python-Dev] [PATCH] Add an authorization header to the initial request. In-Reply-To: References: <1392116587-29676-1-git-send-email-mcepl@redhat.com> Message-ID: <20140211130458.9958640A8A@wycliff.ceplovi.cz> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2014-02-11, 12:27 GMT, you wrote: >> This is my first attempt to contribute to Python itself, so >> please be gentle with me. Yes, I know that I miss unit tests >> and port to other branches of Python (this is against 2.7), >> but I would like first some feedback to see what I am missing >> (aside from the mentioned). > > Please upload your patch to the issue. If it gets ignored there for a > month, join the mentorship list and request a review. It is there (http://bugs.python.org/file34031/0001-Add-an-authorization-header-to-the-initial-request.patch), but given I am the only on Nose list of the bug, I thought it necessary to make a noise here. Mat?j -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iD8DBQFS+h/14J/vJdlkhKwRAoOAAJ9nbR2LI+OPC6B/6LkpFvOnF5B2OwCdFgMg dhTv8f3u9d+Qmmukpmo2b9Y= =lj/m -----END PGP SIGNATURE----- From eric at trueblade.com Tue Feb 11 14:39:50 2014 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 11 Feb 2014 08:39:50 -0500 Subject: [Python-Dev] [PATCH] Add an authorization header to the initial request. In-Reply-To: <20140211130458.9958640A8A@wycliff.ceplovi.cz> References: <1392116587-29676-1-git-send-email-mcepl@redhat.com> <20140211130458.9958640A8A@wycliff.ceplovi.cz> Message-ID: <52FA2826.8080309@trueblade.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/11/2014 08:04 AM, Mat?j Cepl wrote: > On 2014-02-11, 12:27 GMT, you wrote: >>> This is my first attempt to contribute to Python itself, so >>> please be gentle with me. Yes, I know that I miss unit tests >>> and port to other branches of Python (this is against 2.7), but >>> I would like first some feedback to see what I am missing >>> (aside from the mentioned). > >> Please upload your patch to the issue. If it gets ignored there >> for a month, join the mentorship list and request a review. > > It is there > (http://bugs.python.org/file34031/0001-Add-an-authorization-header-to-the-initial-request.patch), > but given I am the only on Nose list of the bug, I thought it > necessary to make a noise here. For people looking for it, that's issue 19494: http://bugs.python.org/issue19494 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJS+igLAAoJENxauZFcKtNxIioH/RRm1qtQrYDlxzqia1dVegdM dyiD85zVgspmIp1EnszhH2X2QSIAtE3AQgBSmYG7isMZbrGGyiItwFlYLJElgbQg b+rGStCxsVhUEauzPHq6gMpqf8Nmfw+NsS3itZ0M0T41H7G7pEhi4Yep3ruqJ1Rp 21wNVMzck/9Zj8p5YDVncDESotODjNN2HeQDg5drsmROvzMW0nQidoXqfaS+7GXV GTIp4sjW4tJi5771Ob8hR9riEHNU9fQ12hf1z/IwNrsaTHToXa6PAhjGcmLez5Vj Vs66dQYtSki45hP5opNyBaEaaV+9dajs8fiktOeVyGA46UwJ7qyQN3SMNKGW4gg= =jiOU -----END PGP SIGNATURE----- From guido at python.org Tue Feb 11 18:30:03 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Feb 2014 09:30:03 -0800 Subject: [Python-Dev] Is the PIP requirement too strict? Message-ID: I don't happen to have OpenSSL configured on my OSX dev box right now. This usually leads to some warnings during the build stage and obviously various ssl-based tests don't work, but I can still get other stuff done. But with the latest repo, "make install" refuses to complete -- it ends fatally as follows: if test "xupgrade" != "xno" ; then \ case upgrade in \ upgrade) ensurepip="--upgrade" ;; \ install|*) ensurepip="" ;; \ esac; \ ./python.exe -E -m ensurepip \ $ensurepip --root=/ ; \ fi Traceback (most recent call last): File "/Users/guido/cpython/Lib/runpy.py", line 171, in _run_module_as_main "__main__", mod_spec) File "/Users/guido/cpython/Lib/runpy.py", line 86, in _run_code exec(code, run_globals) File "/Users/guido/cpython/Lib/ensurepip/__main__.py", line 4, in ensurepip._main() File "/Users/guido/cpython/Lib/ensurepip/__init__.py", line 203, in _main default_pip=args.default_pip, File "/Users/guido/cpython/Lib/ensurepip/__init__.py", line 74, in bootstrap _require_ssl_for_pip() File "/Users/guido/cpython/Lib/ensurepip/__init__.py", line 23, in _require_ssl_for_pip raise RuntimeError(_MISSING_SSL_MESSAGE) RuntimeError: pip 1.5.2 requires SSL/TLS make: *** [install] Error 1 Can this failure be suppressed in the Makefile (given that I know what I'm doing)? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Feb 11 19:15:30 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 11 Feb 2014 13:15:30 -0500 Subject: [Python-Dev] Is the PIP requirement too strict? In-Reply-To: References: Message-ID: On Feb 11, 2014, at 12:30 PM, Guido van Rossum wrote: > I don't happen to have OpenSSL configured on my OSX dev box right now. This usually leads to some warnings during the build stage and obviously various ssl-based tests don't work, but I can still get other stuff done. But with the latest repo, "make install" refuses to complete -- it ends fatally as follows: > > if test "xupgrade" != "xno" ; then \ > case upgrade in \ > upgrade) ensurepip="--upgrade" ;; \ > install|*) ensurepip="" ;; \ > esac; \ > ./python.exe -E -m ensurepip \ > $ensurepip --root=/ ; \ > fi > Traceback (most recent call last): > File "/Users/guido/cpython/Lib/runpy.py", line 171, in _run_module_as_main > "__main__", mod_spec) > File "/Users/guido/cpython/Lib/runpy.py", line 86, in _run_code > exec(code, run_globals) > File "/Users/guido/cpython/Lib/ensurepip/__main__.py", line 4, in > ensurepip._main() > File "/Users/guido/cpython/Lib/ensurepip/__init__.py", line 203, in _main > default_pip=args.default_pip, > File "/Users/guido/cpython/Lib/ensurepip/__init__.py", line 74, in bootstrap > _require_ssl_for_pip() > File "/Users/guido/cpython/Lib/ensurepip/__init__.py", line 23, in _require_ssl_for_pip > raise RuntimeError(_MISSING_SSL_MESSAGE) > RuntimeError: pip 1.5.2 requires SSL/TLS > make: *** [install] Error 1 > > Can this failure be suppressed in the Makefile (given that I know what I'm doing)? > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io So right now pip doesn?t work without TLS, we?re working on that and our 1.6 release should have that. I *thought* that Nick (I think?) had made it so that you just didn?t get pip if you didn?t have TLS enabled, but apparently not. You can suppress this by doing ``ENSUREPIP=no make install``, but probably this should just print a warning instead of dying when TLS isn?t available. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From guido at python.org Tue Feb 11 19:36:07 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Feb 2014 10:36:07 -0800 Subject: [Python-Dev] Is the PIP requirement too strict? In-Reply-To: References: Message-ID: Thanks, I discovered ENSUREPIP=no right after posting, but agreed it should print an intelligible error message instead of giving a traceback. On Tue, Feb 11, 2014 at 10:15 AM, Donald Stufft wrote: > On Feb 11, 2014, at 12:30 PM, Guido van Rossum wrote: > > I don't happen to have OpenSSL configured on my OSX dev box right now. > This usually leads to some warnings during the build stage and obviously > various ssl-based tests don't work, but I can still get other stuff done. > But with the latest repo, "make install" refuses to complete -- it ends > fatally as follows: > > if test "xupgrade" != "xno" ; then \ > case upgrade in \ > upgrade) ensurepip="--upgrade" ;; \ > install|*) ensurepip="" ;; \ > esac; \ > ./python.exe -E -m ensurepip \ > $ensurepip --root=/ ; \ > fi > Traceback (most recent call last): > File "/Users/guido/cpython/Lib/runpy.py", line 171, in > _run_module_as_main > "__main__", mod_spec) > File "/Users/guido/cpython/Lib/runpy.py", line 86, in _run_code > exec(code, run_globals) > File "/Users/guido/cpython/Lib/ensurepip/__main__.py", line 4, in > > ensurepip._main() > File "/Users/guido/cpython/Lib/ensurepip/__init__.py", line 203, in _main > default_pip=args.default_pip, > File "/Users/guido/cpython/Lib/ensurepip/__init__.py", line 74, in > bootstrap > _require_ssl_for_pip() > File "/Users/guido/cpython/Lib/ensurepip/__init__.py", line 23, in > _require_ssl_for_pip > raise RuntimeError(_MISSING_SSL_MESSAGE) > RuntimeError: pip 1.5.2 requires SSL/TLS > make: *** [install] Error 1 > > Can this failure be suppressed in the Makefile (given that I know what I'm > doing)? > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > > > So right now pip doesn't work without TLS, we're working on that and our > 1.6 release > should have that. I *thought* that Nick (I think?) had made it so that you > just didn't get pip > if you didn't have TLS enabled, but apparently not. > > You can suppress this by doing ``ENSUREPIP=no make install``, but probably > this should > just print a warning instead of dying when TLS isn't available. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From cjwelborn at live.com Tue Feb 11 21:18:21 2014 From: cjwelborn at live.com (Christopher Welborn) Date: Tue, 11 Feb 2014 14:18:21 -0600 Subject: [Python-Dev] bugs.python.org/review (DEBUG = True)? Message-ID: Is there a reason bugs.python.org/review is running in DEBUG mode right now? I was just poking around and came across the debug 'DoesNotExist' error page when accessing: http://bugs.python.org/review/rss/reviews/cjwelborn Then, instead of a 404 I get the 'Page not found' debug page. So, just glancing at the acceptable url patterns I see one that will give you a xsrf token (if you send the appropriate headers, which is easy). I'm not a web expert, so I don't know how dangerous it really is. But the error messages were really helpful (in a bad way?). This turned up an actual error. I'm pretty sure it's an actual error, though I could be wrong. It might be an 'expected error'. When accessing http://bugs.python.org/review/account , I got: KeyError at /review/account: 'AUTH_DOMAIN' /home/roundup/trackers/tracker/rietveld/codereview/views.py in account: domain = os.environ['AUTH_DOMAIN'] Maybe someone is doing some testing and I'm just wasting people's time. I thought it would be better to say something, so if it is a mistake someone can fix it. -- \?\ /?/\ \ \/??\/ / / Christopher Welborn (cj) \__/\__/ / cjwelborn at live?com \__/\__/ http://welbornprod.com From ncoghlan at gmail.com Tue Feb 11 22:24:03 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 12 Feb 2014 07:24:03 +1000 Subject: [Python-Dev] Is the PIP requirement too strict? In-Reply-To: References: Message-ID: Right, I think this is a genuine bug in the ensurepip CLI mode. http://bugs.python.org/issue19744 was still pending with Tim (to see if it solved the original problem), but we need to more gracefully handle the error at the CLI level as well. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at acm.org Tue Feb 11 23:22:42 2014 From: nad at acm.org (Ned Deily) Date: Tue, 11 Feb 2014 14:22:42 -0800 Subject: [Python-Dev] Is the PIP requirement too strict? References: Message-ID: In article , Guido van Rossum wrote: > I don't happen to have OpenSSL configured on my OSX dev box right now. This > usually leads to some warnings during the build stage and obviously various > ssl-based tests don't work, but I can still get other stuff done. As a side note, you shouldn't have to configure OpenSSL to build any of the current branches of Python on OS X systems. It should build gracefully with the Apple-supplied headers and libssl/libcrypto, modulo possible deprecation warnings in the recent releases of OS X. -- Ned Deily, nad at acm.org From guido at python.org Wed Feb 12 01:26:06 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Feb 2014 16:26:06 -0800 Subject: [Python-Dev] Is the PIP requirement too strict? In-Reply-To: References: Message-ID: To follow up, Ned diagnosed this for me off-list. The cause was my recent upgrade to Mavericks, which causes the Xcode 5 command line tools to be installed differently. In case others have the same issue, the solution was to run: xcode-select --install (and accept the dialog box it pops up). This is a common issue with Mavericks and open-source projects (not just Python); Ned will add something to the README. --Guido On Tue, Feb 11, 2014 at 2:22 PM, Ned Deily wrote: > In article > , > Guido van Rossum wrote: > > I don't happen to have OpenSSL configured on my OSX dev box right now. > This > > usually leads to some warnings during the build stage and obviously > various > > ssl-based tests don't work, but I can still get other stuff done. > > As a side note, you shouldn't have to configure OpenSSL to build any of > the current branches of Python on OS X systems. It should build > gracefully with the Apple-supplied headers and libssl/libcrypto, modulo > possible deprecation warnings in the recent releases of OS X. > > -- > Ned Deily, > nad at acm.org > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From d2mp1a9 at newsguy.com Wed Feb 12 17:30:20 2014 From: d2mp1a9 at newsguy.com (Bob Hanson) Date: Wed, 12 Feb 2014 08:30:20 -0800 Subject: [Python-Dev] MSI installer won't install on WinXP-SP2 (was Re: [RELEASED] Python 3.4.0 release candidate 1) References: <52F9D493.6090100@hastings.org> Message-ID: [32-bit Windows XP-SP2] Python 3.4.0rc1's MSI installer won't install on my machine. Error message (typed in from screenshot): ''' There is a problem with this Windows Installer package. A program required for this install to complete could not be run. Contact your support personnel or package vendor. ''' I tried to install twice with the default options as well as once with the pip option deselected -- all errored out with the above message. (And yes, I did delete the entire Python34 directory before trying to install. :-) ) As I experienced trouble with Python 3.4.0b2's installer, not sure if the problem is here or with the installer. (But b2 did install and seem to be okay.) Bob Hanson From victor.stinner at gmail.com Wed Feb 12 17:42:49 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 12 Feb 2014 17:42:49 +0100 Subject: [Python-Dev] MSI installer won't install on WinXP-SP2 (was Re: [RELEASED] Python 3.4.0 release candidate 1) In-Reply-To: References: <52F9D493.6090100@hastings.org> Message-ID: Hi, 2014-02-12 17:30 GMT+01:00 Bob Hanson : > [32-bit Windows XP-SP2] > > Python 3.4.0rc1's MSI installer won't install on my machine. I justed tested Python 3.4.0rc1 MSI installer on Windows XP SP3 (32-bit): Python was installed successfully. Victor From d2mp1a9 at newsguy.com Wed Feb 12 18:10:47 2014 From: d2mp1a9 at newsguy.com (Bob Hanson) Date: Wed, 12 Feb 2014 09:10:47 -0800 Subject: [Python-Dev] MSI installer won't install on WinXP-SP2 (was Re: [RELEASED] Python 3.4.0 release candidate 1) References: <52F9D493.6090100@hastings.org> Message-ID: On Wed, 12 Feb 2014 17:42:49 +0100, Victor Stinner wrote: > 2014-02-12 17:30 GMT+01:00 Bob Hanson : > > > [32-bit Windows XP-SP2] > > > > Python 3.4.0rc1's MSI installer won't install on my machine. > > I justed tested Python 3.4.0rc1 MSI installer on Windows XP SP3 > (32-bit): Python was installed successfully. Does this mean that Python no longer supports XP-SP2? Bob Hanson From victor.stinner at gmail.com Wed Feb 12 18:15:43 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 12 Feb 2014 18:15:43 +0100 Subject: [Python-Dev] MSI installer won't install on WinXP-SP2 (was Re: [RELEASED] Python 3.4.0 release candidate 1) In-Reply-To: References: <52F9D493.6090100@hastings.org> Message-ID: 2014-02-12 18:10 GMT+01:00 Bob Hanson : > Does this mean that Python no longer supports XP-SP2? I don't know what it means. I don't have access to Windows XP SP2 to test. By the way, why not upgrading to SP3? :-) I read that you installed the beta2 before. You should maybe make sure that Python 3.4 beta 2 has been fully uninstalled. Why did you remove the directory manually? Why not uninstall using the Control Panel? Victor From cjwelborn at live.com Wed Feb 12 18:49:20 2014 From: cjwelborn at live.com (Christopher Welborn) Date: Wed, 12 Feb 2014 11:49:20 -0600 Subject: [Python-Dev] MSI installer won't install on WinXP-SP2 (was Re: [RELEASED] Python 3.4.0 release candidate 1) In-Reply-To: References: <52F9D493.6090100@hastings.org> Message-ID: On 02/12/2014 10:30 AM, Bob Hanson wrote: > [32-bit Windows XP-SP2] > > Python 3.4.0rc1's MSI installer won't install on my machine. > > Error message (typed in from screenshot): > > ''' > There is a problem with this Windows Installer package. A program > required for this install to complete could not be run. Contact > your support personnel or package vendor. > ''' > > Bob Hanson > I've seen an error message similar to this one, because I didn't run the installer as admin. ('Run as Administrator...'). Not sure if its the same thing here though. -- \?\ /?/\ \ \/??\/ / / Christopher Welborn (cj) \__/\__/ / cjwelborn at live?com \__/\__/ http://welbornprod.com From d2mp1a9 at newsguy.com Wed Feb 12 19:03:26 2014 From: d2mp1a9 at newsguy.com (Bob Hanson) Date: Wed, 12 Feb 2014 10:03:26 -0800 Subject: [Python-Dev] MSI installer won't install on WinXP-SP2 (was Re: [RELEASED] Python 3.4.0 release candidate 1) References: <52F9D493.6090100@hastings.org> Message-ID: TL;DR: Solved. Thanks, Victor, for prodding me to jump through all those Windows hoops. ;-) [More comments interleaved below:] On Wed, 12 Feb 2014 18:15:43 +0100, Victor Stinner wrote: > 2014-02-12 18:10 GMT+01:00 Bob Hanson : > > By the way, why not upgrading to SP3? :-) Lots of reasons. One, though, IIRC, MS does not allow downloads of such anymore. (More importantly to me, I don't like the EULA for SP3+ and don't want to implicitly agree to it.) > I read that you installed the beta2 before. You should maybe make sure > that Python 3.4 beta 2 has been fully uninstalled. Why did you remove > the directory manually? Why not uninstall using the Control Panel? 'Twas long years of fighting with the Add/Remove Programs applet when working with Python and other OSS which made me do the "naughty" thing today. In my experience, particularly if one has third-party packages installed by a variety of means, the Control Panel applet often screws things up royally when "removing" things. (Also, I'd just read about Terry's install problems -- usually, I just install overtop the prior version and the MSI installer does the right thing. Today, though, with the above paragraph in mind, and with Terry's problems, I just decided to "shortcut" things.) Following Victor's questioning, I put the dir back, went to CP and did the Add/Remove thing, and then reinstalled rc1 and it "worked." Perhaps a caution in the docs (release notes?) or some such should be added -- as far as I know, I used to be able to delete Python dirs with abandon before installing a different version. It *does* seem that Python and Windows is becoming a sadder story -- I know, use Linux. :-) Meanwhile, my main Python remains 2.5.4 -- a very good vintage. Anyway, thanks again, Victor, for encouraging me to do the obstacle course yet again. :-) Bob Hanson From d2mp1a9 at newsguy.com Wed Feb 12 19:08:27 2014 From: d2mp1a9 at newsguy.com (Bob Hanson) Date: Wed, 12 Feb 2014 10:08:27 -0800 Subject: [Python-Dev] MSI installer won't install on WinXP-SP2 (was Re: [RELEASED] Python 3.4.0 release candidate 1) References: <52F9D493.6090100@hastings.org> Message-ID: On Wed, 12 Feb 2014 11:49:20 -0600, Christopher Welborn wrote: > I've seen an error message similar to this one, because I didn't run > the installer as admin. ('Run as Administrator...'). Not sure if its > the same thing here though. Thanks for the suggestion, Christopher. However, see my last reply to Victor up-thread -- the problem is now solved, I think. (I had to use the Windows Add/Remove programs applet to fix things.) Bob Hanson From ncoghlan at gmail.com Wed Feb 12 21:38:38 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 13 Feb 2014 06:38:38 +1000 Subject: [Python-Dev] MSI installer won't install on WinXP-SP2 (was Re: [RELEASED] Python 3.4.0 release candidate 1) In-Reply-To: References: <52F9D493.6090100@hastings.org> Message-ID: On 13 Feb 2014 04:06, "Bob Hanson" wrote: > > TL;DR: Solved. Thanks, Victor, for prodding me to jump through > all those Windows hoops. ;-) > > [More comments interleaved below:] > > On Wed, 12 Feb 2014 18:15:43 +0100, Victor Stinner wrote: > > > 2014-02-12 18:10 GMT+01:00 Bob Hanson : > > > > By the way, why not upgrading to SP3? :-) > > Lots of reasons. One, though, IIRC, MS does not allow downloads > of such anymore. > > (More importantly to me, I don't like the EULA for SP3+ and don't > want to implicitly agree to it.) > > > I read that you installed the beta2 before. You should maybe make sure > > that Python 3.4 beta 2 has been fully uninstalled. Why did you remove > > the directory manually? Why not uninstall using the Control Panel? > > 'Twas long years of fighting with the Add/Remove Programs applet > when working with Python and other OSS which made me do the > "naughty" thing today. In my experience, particularly if one has > third-party packages installed by a variety of means, the Control > Panel applet often screws things up royally when "removing" > things. > > (Also, I'd just read about Terry's install problems -- usually, I > just install overtop the prior version and the MSI installer does > the right thing. Today, though, with the above paragraph in mind, > and with Terry's problems, I just decided to "shortcut" things.) > > Following Victor's questioning, I put the dir back, went to CP > and did the Add/Remove thing, and then reinstalled rc1 and it > "worked." > > Perhaps a caution in the docs (release notes?) or some such > should be added -- as far as I know, I used to be able to delete > Python dirs with abandon before installing a different version. Just deleting the Python directory hasn't been the right thing to do in a very long time - it leaves cruft in the registry at the very least (that will confuse other tools into thinking Python is still installed), and since Python 3.3 will also leave the Python Launcher for Windows installed. Using Add/Remove programs instead of just deleting directories is a general Windows administration requirement for MSI installers, so there's no reason to add a specific note. > > It *does* seem that Python and Windows is becoming a sadder story > -- I know, use Linux. :-) Actually, we've been substantially improving the Windows integration to make things easier for new users, which is the main reason the Windows installer has become substantially less tolerant of corruption of the registry state (which is effectively what occurs when you just delete the Python directory instead of uninstalling it through Add/Remove programs). Python 3.4 now not only provides the "py" multi-version launcher and the "pip" package installer, selecting the path modification component during installation means you don't even need to adjust PATH yourself any more. The primary beneficiaries of the increasing adoption of the pre-built wheel binary packaging format are also Windows users (Linux users and Mac OS X users that do not use the python.org binaries currently still need to build from source themselves, or bootstrap the cross-platform conda tools). > > Meanwhile, my main Python remains 2.5.4 -- a very good vintage. If that's your baseline, then yes, Python was likely still quite tolerant of bypassing the uninstallation process back then. That hasn't been the case for a long time. Regards, Nick. > > Anyway, thanks again, Victor, for encouraging me to do the > obstacle course yet again. :-) > > Bob Hanson > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Feb 12 22:26:26 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 12 Feb 2014 21:26:26 +0000 Subject: [Python-Dev] MSI installer won't install on WinXP-SP2 (was Re: [RELEASED] Python 3.4.0 release candidate 1) In-Reply-To: References: <52F9D493.6090100@hastings.org> Message-ID: On 12 February 2014 20:38, Nick Coghlan wrote: >> It *does* seem that Python and Windows is becoming a sadder story >> -- I know, use Linux. :-) > > Actually, we've been substantially improving the Windows integration to make > things easier for new users, which is the main reason the Windows installer > has become substantially less tolerant of corruption of the registry state > (which is effectively what occurs when you just delete the Python directory > instead of uninstalling it through Add/Remove programs). Agreed, I feel that the user's experience of Python on Windows has been steadily improving. Thanks to everyone for this. The one thing that I sometimes miss is a "portable" distribution (i.e., unzip and run, do any PATH manipulation etc that you want yourself). But it's easy to make one yourself, so no real issue there. Paul From ethan at stoneleaf.us Thu Feb 13 00:10:25 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 12 Feb 2014 15:10:25 -0800 Subject: [Python-Dev] [Python-checkins] cpython: asyncio.events: Use __slots__ in Handle and TimerHandle In-Reply-To: <3fPZZz2sJyz7LjR@mail.python.org> References: <3fPZZz2sJyz7LjR@mail.python.org> Message-ID: <52FBFF61.1070508@stoneleaf.us> On 02/12/2014 02:02 PM, yury.selivanov wrote: > http://hg.python.org/cpython/rev/920304e1f36b > changeset: 89175:920304e1f36b > user: Yury Selivanov > date: Wed Feb 12 17:01:52 2014 -0500 > summary: > asyncio.events: Use __slots__ in Handle and TimerHandle > > files: > Lib/asyncio/events.py | 4 ++++ > 1 files changed, 4 insertions(+), 0 deletions(-) > > > diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py > --- a/Lib/asyncio/events.py > +++ b/Lib/asyncio/events.py > @@ -19,6 +19,8 @@ > class Handle: > """Object returned by callback registration methods.""" > > + __slots__ = ['_callback', '_args', '_cancelled'] > + > def __init__(self, callback, args): > assert not isinstance(callback, Handle), 'A Handle is not a callback' > self._callback = callback > @@ -46,6 +48,8 @@ > class TimerHandle(Handle): > """Object returned by timed callback registration methods.""" > > + __slots__ = ['_when'] > + > def __init__(self, when, callback, args): > assert when is not None > super().__init__(callback, args) Apologies up front if these are stupid questions, but: Why __slots__? Are we going to have so many of these things that memory is an issue? The asserts in the code -- those are not for checking user input, correct? -- ~Ethan~ From guido at python.org Thu Feb 13 00:36:15 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 12 Feb 2014 15:36:15 -0800 Subject: [Python-Dev] [Python-checkins] cpython: asyncio.events: Use __slots__ in Handle and TimerHandle In-Reply-To: <52FBFF61.1070508@stoneleaf.us> References: <3fPZZz2sJyz7LjR@mail.python.org> <52FBFF61.1070508@stoneleaf.us> Message-ID: On Wed, Feb 12, 2014 at 3:10 PM, Ethan Furman wrote: > On 02/12/2014 02:02 PM, yury.selivanov wrote: > >> http://hg.python.org/cpython/rev/920304e1f36b >> changeset: 89175:920304e1f36b >> user: Yury Selivanov >> date: Wed Feb 12 17:01:52 2014 -0500 >> summary: >> asyncio.events: Use __slots__ in Handle and TimerHandle >> >> files: >> Lib/asyncio/events.py | 4 ++++ >> 1 files changed, 4 insertions(+), 0 deletions(-) >> >> >> diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py >> --- a/Lib/asyncio/events.py >> +++ b/Lib/asyncio/events.py >> @@ -19,6 +19,8 @@ >> class Handle: >> """Object returned by callback registration methods.""" >> >> + __slots__ = ['_callback', '_args', '_cancelled'] >> + >> def __init__(self, callback, args): >> assert not isinstance(callback, Handle), 'A Handle is not a >> callback' >> self._callback = callback >> @@ -46,6 +48,8 @@ >> class TimerHandle(Handle): >> """Object returned by timed callback registration methods.""" >> >> + __slots__ = ['_when'] >> + >> def __init__(self, when, callback, args): >> assert when is not None >> super().__init__(callback, args) >> > > Apologies up front if these are stupid questions, but: > > Why __slots__? Are we going to have so many of these things that memory > is an issue > There's one of these created for every callback -- which includes every time 'yield from' blocks for a Future. The savings are primarily in allocation cost (no dicts to allocate). > The asserts in the code -- those are not for checking user input, correct? > Yeah, this class is never instantiated directly by the user. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Feb 13 00:42:15 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 12 Feb 2014 18:42:15 -0500 Subject: [Python-Dev] MSI installer won't install on WinXP-SP2 (was Re: [RELEASED] Python 3.4.0 release candidate 1) In-Reply-To: References: <52F9D493.6090100@hastings.org> Message-ID: On 2/12/2014 3:38 PM, Nick Coghlan wrote: > Just deleting the Python directory hasn't been the right thing to do in > a very long time - it leaves cruft in the registry at the very least > (that will confuse other tools into thinking Python is still installed), > and since Python 3.3 will also leave the Python Launcher for Windows > installed. > > Using Add/Remove programs instead of just deleting directories is a > general Windows administration requirement for MSI installers, so > there's no reason to add a specific note. In windows 7, Add/Remove is not Programs and Features. Looking there, I found an entry for .0b2 that should have been removed by the .0b3 install, but was not. When I tried to uninstall .0b2, I got the same box about not being able to run something. I downloaded the .0b2 installer from www.python.org/ftp/python/3.4.0/, repaired the .0b2 installation, and then uninstalled it, and the entry disappeared. I was then able to install 64bit .0c1. So I conclude that a glitch in the .0b3 installation, leaving a bit of .0b2, made .0c1 unable to uninstall .0b3 properly. -- Terry Jan Reedy From jessica.mckellar at gmail.com Thu Feb 13 17:08:23 2014 From: jessica.mckellar at gmail.com (Jessica McKellar) Date: Thu, 13 Feb 2014 11:08:23 -0500 Subject: [Python-Dev] CPython Google Summer of Code mentors needed: deadline is tomorrow Message-ID: Hi folks, Terri Oda's original message to this list about CPython's participation in Google Summer of Code 2014 is at the end of this email. If you'd like to see CPython participate in Google Summer of Code 2014, we need* at least 2 people* to say they are interested in mentoring by the end of today, February 13th. *Are you interested in mentoring? Please speak up!* This is not a binding commitment; this is just a statement of interest. As a previous GSoC mentor, I'd say the time commitment is around 10 hours / week per student, and this is best split across 2 mentors. GSoC FAQ: https://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2014/help_page The current umbrella project pages: https://wiki.python.org/moin/SummerOfCode/2014#Project_Ideas Last year's umbrella project pages: https://wiki.python.org/moin/SummerOfCode/2013#Project_Ideas Last year's CPython project ideas: https://wiki.python.org/moin/SummerOfCode/2013/python-core Regards, -Jessica === Terri's original message: For anyone who completely isn't familiar with this: Google Summer of Code is a program where Google pays students to work on open source projects for the summer. The motto goes: "Flip bits, not burgers." Google Summer of Code applications are open for mentoring organizations, and I've already put in Python's application. You can get more info here: https://wiki.python.org/moin/SummerOfCode/2014 What I'd like to know is if anyone's got any good ideas for core python projects this year and are interested in mentoring. We always have lots of students who'd like to work on Python, but right now we don't have any mentors who are available and interested in helping students make contributions to the language (We do, however, have mentors for various python sub-projects). Let me know if this might be interesting to you. I'm happy to answer any questions, and I've got experienced mentors who aren't core python devs but who'd be willing to pair up so you wouldn't have to go it alone, just let me know! Terri PS - I'm also looking for new sub-projects and other mentors -- just ask if you want to take part this year! -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Feb 13 17:31:13 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 13 Feb 2014 11:31:13 -0500 Subject: [Python-Dev] CPython Google Summer of Code mentors needed: deadline is tomorrow In-Reply-To: References: Message-ID: <52FCF351.7020704@udel.edu> On 2/13/2014 11:08 AM, Jessica McKellar wrote: > Hi folks, > > Terri Oda's original message to this list about CPython's participation > in Google Summer of Code 2014 is at the end of this email. > > If you'd like to see CPython participate in Google Summer of Code 2014, > we need*at least 2 people* to say they are interested in mentoring by > the end of today, February 13th. > > *Are you interested in mentoring? Please speak up!* > This is not a binding commitment; this is just a statement of interest. > As a previous GSoC mentor, I'd say the time commitment is around 10 > hours / week per student, and this is best split across 2 mentors. I am interested. Last summer I was unofficially a subject-matter committer-review mentor for the two GSOC students who worked on Idle tests. They were officially mentored by Todd Rovito, who handled everything else. I can imagine doing something similar (Idle project, with co-mentor) again. -- Terry Jan Reedy From solipsis at pitrou.net Thu Feb 13 19:07:10 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 13 Feb 2014 19:07:10 +0100 Subject: [Python-Dev] CPython Google Summer of Code mentors needed: deadline is tomorrow References: Message-ID: <20140213190710.05b53b31@fsol> Hello Jessica, On Thu, 13 Feb 2014 11:08:23 -0500 Jessica McKellar wrote: > Hi folks, > > Terri Oda's original message to this list about CPython's participation in > Google Summer of Code 2014 is at the end of this email. > > If you'd like to see CPython participate in Google Summer of Code 2014, we > need* at least 2 people* to say they are interested in mentoring by the end > of today, February 13th. > > *Are you interested in mentoring? Please speak up!* > > This is not a binding commitment; this is just a statement of interest. As > a previous GSoC mentor, I'd say the time commitment is around 10 hours / > week per student, and this is best split across 2 mentors. Ok, you can count me as non-bindingly interested in co-mentoring :-) cheers Antoine. From p.f.moore at gmail.com Thu Feb 13 21:53:10 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 13 Feb 2014 20:53:10 +0000 Subject: [Python-Dev] Possible major bug with zipimport on Windows in Python 3.3.4 Message-ID: Can someone please take a look at http://bugs.python.org/issue20621 for me? It appears that there is a significant problem with zipimport in 3.3.4. It's causing breakage in virtualenv but I've isolated the issue to what I think is Python itself. Basically, until this bug is fixed, virtualenv won't work on Windows with Python 3.3.4, as far as I can tell. Thanks, Paul From p.f.moore at gmail.com Thu Feb 13 21:58:56 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 13 Feb 2014 20:58:56 +0000 Subject: [Python-Dev] Possible major bug with zipimport on Windows in Python 3.3.4 In-Reply-To: <84841BB3-CAE7-44DE-A854-A6428DEC743D@stufft.io> References: <84841BB3-CAE7-44DE-A854-A6428DEC743D@stufft.io> Message-ID: On 13 February 2014 20:54, Donald Stufft wrote: > > On Feb 13, 2014, at 3:53 PM, Paul Moore wrote: > >> Can someone please take a look at http://bugs.python.org/issue20621 >> for me? It appears that there is a significant problem with zipimport >> in 3.3.4. It's causing breakage in virtualenv but I've isolated the >> issue to what I think is Python itself. >> >> Basically, until this bug is fixed, virtualenv won't work on Windows >> with Python 3.3.4, as far as I can tell. > > Does it affect 3.4? Just checked that. No, it's not an issue in 3.4rc0. Nor is it in 3.3.3, so it's a regression in the 3.3 series only. Paul From donald at stufft.io Thu Feb 13 21:54:13 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 13 Feb 2014 15:54:13 -0500 Subject: [Python-Dev] Possible major bug with zipimport on Windows in Python 3.3.4 In-Reply-To: References: Message-ID: <84841BB3-CAE7-44DE-A854-A6428DEC743D@stufft.io> On Feb 13, 2014, at 3:53 PM, Paul Moore wrote: > Can someone please take a look at http://bugs.python.org/issue20621 > for me? It appears that there is a significant problem with zipimport > in 3.3.4. It's causing breakage in virtualenv but I've isolated the > issue to what I think is Python itself. > > Basically, until this bug is fixed, virtualenv won't work on Windows > with Python 3.3.4, as far as I can tell. > > Thanks, > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Does it affect 3.4? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris at simplistix.co.uk Fri Feb 14 09:04:29 2014 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 14 Feb 2014 08:04:29 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError Message-ID: <52FDCE0D.5090402@simplistix.co.uk> Hi All, Sending this to python-dev as I'm wondering if this was considered when the choice to have objects of different types raise a TypeError when ordered... So, the concrete I case I have is implementing stable ordering for the python Range objects that psycopg2 uses. These have 3 attributes that can either be None or, for sake of argument, a numeric value. To implement __lt__ in Python 2, I could do: def __lt__(self, other): if not isinstance(other, Range): return True return ((self._lower, self._upper, self._bounds) < (other._lower, other._upper, other._bounds)) Because None < 1 raises a TypeError, in Python 3 I have to do: def __lt__(self, other): if not isinstance(other, Range): return NotImplemented for attr in '_lower', '_upper', '_bounds': self_value = getattr(self, attr) other_value = getattr(other, attr) if self_value == other_value: pass elif self_value is None: return True elif other_value is None: return False else: return self_value < other_value return False Am I missing something? How can I get this method down to a sane size? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Fri Feb 14 10:29:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Feb 2014 19:29:05 +1000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <52FDCE0D.5090402@simplistix.co.uk> References: <52FDCE0D.5090402@simplistix.co.uk> Message-ID: On 14 February 2014 18:04, Chris Withers wrote: > > Am I missing something? How can I get this method down to a sane size? The easiest way is usually to normalise the attributes to a sensible numeric value depending on where you want "None" to sort (positive and negative infinity floating point values often work well in the case of numeric data, but a custom AlwaysLess or AlwaysGreater type works for arbitrary data). You can either do that dynamically, or else cache the normalised values when the attributes are set. Python 2 used to guess, Python 3 makes developers decide how they want None to be handled in the context of ordering operations. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Fri Feb 14 10:44:28 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 14 Feb 2014 20:44:28 +1100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <52FDCE0D.5090402@simplistix.co.uk> References: <52FDCE0D.5090402@simplistix.co.uk> Message-ID: On Fri, Feb 14, 2014 at 7:04 PM, Chris Withers wrote: > > To implement __lt__ in Python 2, I could do: > > def __lt__(self, other): > if not isinstance(other, Range): > return True > return ((self._lower, self._upper, self._bounds) < > (other._lower, other._upper, other._bounds)) > > Because None < 1 raises a TypeError, in Python 3 I have to do: > > def __lt__(self, other): > if not isinstance(other, Range): > return NotImplemented > for attr in '_lower', '_upper', '_bounds': > self_value = getattr(self, attr) > other_value = getattr(other, attr) > if self_value == other_value: > pass > elif self_value is None: > return True > elif other_value is None: > return False > else: > return self_value < other_value > return False > > Am I missing something? How can I get this method down to a sane size? Can you be certain that all your values are either None or positive integers? If so, try this: def __lt__(self, other): if not isinstance(other, Range): return True # or NotImplemented, not sure why this change return ((self._lower or 0, self._upper or 0, self._bounds or 0) < (other._lower or 0, other._upper or 0, other._bounds or 0)) That'll treat all Nones as 0, and compare them accordingly. If you can't depend on them being positive (eg if you need None to be less than 0 rather than equal - even more so if you need it to be less than negative numbers), you'll need to more explicitly check. But this is nice and tidy, if it works. ChrisA From regebro at gmail.com Fri Feb 14 10:46:50 2014 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 14 Feb 2014 10:46:50 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <52FDCE0D.5090402@simplistix.co.uk> References: <52FDCE0D.5090402@simplistix.co.uk> Message-ID: On Fri, Feb 14, 2014 at 9:04 AM, Chris Withers wrote: > Hi All, > > Sending this to python-dev as I'm wondering if this was considered when the > choice to have objects of different types raise a TypeError when ordered... > > So, the concrete I case I have is implementing stable ordering for the > python Range objects that psycopg2 uses. These have 3 attributes that can > either be None or, for sake of argument, a numeric value. > > To implement __lt__ in Python 2, I could do: > > def __lt__(self, other): > if not isinstance(other, Range): > return True > return ((self._lower, self._upper, self._bounds) < > (other._lower, other._upper, other._bounds)) > > Because None < 1 raises a TypeError, in Python 3 I have to do: > > def __lt__(self, other): > if not isinstance(other, Range): > return NotImplemented > for attr in '_lower', '_upper', '_bounds': > self_value = getattr(self, attr) > other_value = getattr(other, attr) > if self_value == other_value: > pass > elif self_value is None: > return True > elif other_value is None: > return False > else: > return self_value < other_value > return False > > Am I missing something? How can I get this method down to a sane size? It was considered. It's not obvious where you want "None" to appear in your ordering, so you will have to implement this by yourself. I can't come up with anything obviously shorter. Or, well, this, but it's not exactly pretty: def __lt__(self, other): if not isinstance(other, Range): return NotImplemented ninf = float('-inf') # So None is lower than anything else. return ((self._lower if self._lower is not None else ninf, self._upper if self._upper is not None else ninf, self._bounds if self._bounds is not None else ninf) < ( other._lower if other._lower is not None else ninf, other._upper if other._upper is not None else ninf, other._bounds if other._bounds is not None else ninf)) Or maybe: def __lt__(self, other): if not isinstance(other, Range): return NotImplemented s = (self._lower, self._upper, self._bounds) if None is s: return True o = (other._lower, other._upper, other._bounds) if None in o: return False return s < o From solipsis at pitrou.net Fri Feb 14 11:02:02 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 14 Feb 2014 11:02:02 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError References: <52FDCE0D.5090402@simplistix.co.uk> Message-ID: <20140214110202.3ce2257a@fsol> On Fri, 14 Feb 2014 10:46:50 +0100 Lennart Regebro wrote: > > > > Sending this to python-dev as I'm wondering if this was considered when the > > choice to have objects of different types raise a TypeError when ordered... > > > > So, the concrete I case I have is implementing stable ordering for the > > python Range objects that psycopg2 uses. These have 3 attributes that can > > either be None or, for sake of argument, a numeric value. > > [...] > > It was considered. It's not obvious where you want "None" to appear in > your ordering, so you will have to implement this by yourself. I can't > come up with anything obviously shorter. I have to agree with Lennart. The fact that SQL defines an order for NULL and other values doesn't mean it's obvious or right in any way (I never remember which way it goes). Regards Antoine. From ncoghlan at gmail.com Fri Feb 14 11:13:43 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Feb 2014 20:13:43 +1000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <20140214110202.3ce2257a@fsol> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> Message-ID: On 14 February 2014 20:02, Antoine Pitrou wrote: > On Fri, 14 Feb 2014 10:46:50 +0100 > Lennart Regebro wrote: >> > >> > Sending this to python-dev as I'm wondering if this was considered when the >> > choice to have objects of different types raise a TypeError when ordered... >> > >> > So, the concrete I case I have is implementing stable ordering for the >> > python Range objects that psycopg2 uses. These have 3 attributes that can >> > either be None or, for sake of argument, a numeric value. >> > > [...] >> >> It was considered. It's not obvious where you want "None" to appear in >> your ordering, so you will have to implement this by yourself. I can't >> come up with anything obviously shorter. > > I have to agree with Lennart. The fact that SQL defines an order for > NULL and other values doesn't mean it's obvious or right in any way (I > never remember which way it goes). SQL doesn't define an order for NULL, it's more like a "quiet NaN" - if either operand is NULL, the result is also NULL. (I ran into this recently, in the context of "NULL == value" and "NULL != value" both being effectively false). We could theoretically do something similar, but the NULL handling in SQL is a frequent source of bugs in filter definitions, so that really doesn't sound like a good idea. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Fri Feb 14 11:20:34 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 14 Feb 2014 11:20:34 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> Message-ID: <20140214112034.78c9a02e@fsol> On Fri, 14 Feb 2014 20:13:43 +1000 Nick Coghlan wrote: > On 14 February 2014 20:02, Antoine Pitrou wrote: > > On Fri, 14 Feb 2014 10:46:50 +0100 > > Lennart Regebro wrote: > >> > > >> > Sending this to python-dev as I'm wondering if this was considered when the > >> > choice to have objects of different types raise a TypeError when ordered... > >> > > >> > So, the concrete I case I have is implementing stable ordering for the > >> > python Range objects that psycopg2 uses. These have 3 attributes that can > >> > either be None or, for sake of argument, a numeric value. > >> > > > [...] > >> > >> It was considered. It's not obvious where you want "None" to appear in > >> your ordering, so you will have to implement this by yourself. I can't > >> come up with anything obviously shorter. > > > > I have to agree with Lennart. The fact that SQL defines an order for > > NULL and other values doesn't mean it's obvious or right in any way (I > > never remember which way it goes). > > SQL doesn't define an order for NULL, it's more like a "quiet NaN" - > if either operand is NULL, the result is also NULL. (I ran into this > recently, in the context of "NULL == value" and "NULL != value" both > being effectively false). Hmm, it seems you're right, but I'm quite sure some DBMSes have a consistent way of ordering NULLs when using ORDER BY on a nullable column. Regards Antoine. From p.f.moore at gmail.com Fri Feb 14 11:30:33 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 14 Feb 2014 10:30:33 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <20140214112034.78c9a02e@fsol> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> Message-ID: On 14 February 2014 10:20, Antoine Pitrou wrote: > Hmm, it seems you're right, but I'm quite sure some DBMSes have a > consistent way of ordering NULLs when using ORDER BY on a nullable > column. ORDER BY xxx [NULLS FIRST|LAST] is the syntax in Oracle, with (IIRC) NULLS LAST as default. But I agree, this is not an argument in favour of doing the same in Python. Paul From rosuav at gmail.com Fri Feb 14 11:34:08 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 14 Feb 2014 21:34:08 +1100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <20140214112034.78c9a02e@fsol> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> Message-ID: On Fri, Feb 14, 2014 at 9:20 PM, Antoine Pitrou wrote: > Hmm, it seems you're right, but I'm quite sure some DBMSes have a > consistent way of ordering NULLs when using ORDER BY on a nullable > column. Yes, and I believe it's part of the SQL-92 spec. Certainly here's PostgreSQL's take on the matter: http://www.postgresql.org/docs/9.3/static/sql-select.html#SQL-ORDERBY In short, NULL is by default considered greater than anything else (last in an ascending sort, first in a descending sort), but this can be inverted. Oddly enough, the syntax is NULLS FIRST vs NULLS LAST, not NULLS LOW vs NULLS HIGH. ChrisA From ncoghlan at gmail.com Fri Feb 14 11:42:13 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Feb 2014 20:42:13 +1000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> Message-ID: On 14 February 2014 20:30, Paul Moore wrote: > On 14 February 2014 10:20, Antoine Pitrou wrote: >> Hmm, it seems you're right, but I'm quite sure some DBMSes have a >> consistent way of ordering NULLs when using ORDER BY on a nullable >> column. > > ORDER BY xxx [NULLS FIRST|LAST] is the syntax in Oracle, with (IIRC) > NULLS LAST as default. But I agree, this is not an argument in favour > of doing the same in Python. IIRC, MySQL and PostgreSQL sort them in the opposite order from each other (or it's possibly just Teiid's PostgreSQL emulation that is the opposite of MySQL). Either way, it nicely illustrates *why* we didn't grant None an exemption from the "no implicit cross-type ordering operations" in 3.x. It does make *adapting* to other models like SQL a bit more painful though. That's the main reason I still occasionally wonder if "AlwaysMin" and "AlwaysMax" singletons might make sense, although that approach has problems of its own (specifically, it's hard to actually use without breaking the clean NULL -> None mapping and in many cases +/- infinity already work fine). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Fri Feb 14 11:54:35 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 14 Feb 2014 21:54:35 +1100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> Message-ID: On Fri, Feb 14, 2014 at 9:42 PM, Nick Coghlan wrote: > IIRC, MySQL and PostgreSQL sort them in the opposite order from each > other Ouch! You're right: http://dev.mysql.com/doc/refman/5.7/en/working-with-null.html """When doing an ORDER BY, NULL values are presented first if you do ORDER BY ... ASC and last if you do ORDER BY ... DESC.""" Not configurable in MySQL. It's apparently not part of the standard, according to Wikipedia: http://en.wikipedia.org/wiki/Order_by So definitely SQL's handling of NULL should not be any sort of guide as regards Python's treatment of None. ChrisA From phd at phdru.name Fri Feb 14 11:59:05 2014 From: phd at phdru.name (Oleg Broytman) Date: Fri, 14 Feb 2014 11:59:05 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> Message-ID: <20140214105905.GA3341@phdru.name> On Fri, Feb 14, 2014 at 09:54:35PM +1100, Chris Angelico wrote: > So definitely SQL's handling of NULL should not be any sort of guide > as regards Python's treatment of None. Why not? Just make the order different for CPython and PyPy, Cython and Jython. ;-) Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From yoavglazner at gmail.com Fri Feb 14 12:28:10 2014 From: yoavglazner at gmail.com (yoav glazner) Date: Fri, 14 Feb 2014 13:28:10 +0200 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <20140214105905.GA3341@phdru.name> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <20140214105905.GA3341@phdru.name> Message-ID: On Feb 14, 2014 1:13 PM, "Oleg Broytman" wrote: > > On Fri, Feb 14, 2014 at 09:54:35PM +1100, Chris Angelico wrote: > > So definitely SQL's handling of NULL should not be any sort of guide > > as regards Python's treatment of None. > > Why not? Just make the order different for CPython and PyPy, Cython > and Jython. ;-) > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. Unicode + bytes explodes in order to get the error sooner.i.e Not waiting for the killer 8bit string. So None<1 is the opposite ! sort(somelist) is a ticking bomb in py3 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Feb 14 12:34:11 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 14 Feb 2014 12:34:11 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <20140214112034.78c9a02e@fsol> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> Message-ID: <52FDFF33.40703@egenix.com> On 14.02.2014 11:20, Antoine Pitrou wrote: > On Fri, 14 Feb 2014 20:13:43 +1000 > Nick Coghlan wrote: >> On 14 February 2014 20:02, Antoine Pitrou wrote: >>> On Fri, 14 Feb 2014 10:46:50 +0100 >>> Lennart Regebro wrote: >>>>> >>>>> Sending this to python-dev as I'm wondering if this was considered when the >>>>> choice to have objects of different types raise a TypeError when ordered... >>>>> >>>>> So, the concrete I case I have is implementing stable ordering for the >>>>> python Range objects that psycopg2 uses. These have 3 attributes that can >>>>> either be None or, for sake of argument, a numeric value. >>>>> >>> [...] >>>> >>>> It was considered. It's not obvious where you want "None" to appear in >>>> your ordering, so you will have to implement this by yourself. I can't >>>> come up with anything obviously shorter. >>> >>> I have to agree with Lennart. The fact that SQL defines an order for >>> NULL and other values doesn't mean it's obvious or right in any way (I >>> never remember which way it goes). >> >> SQL doesn't define an order for NULL, it's more like a "quiet NaN" - >> if either operand is NULL, the result is also NULL. (I ran into this >> recently, in the context of "NULL == value" and "NULL != value" both >> being effectively false). > > Hmm, it seems you're right, but I'm quite sure some DBMSes have a > consistent way of ordering NULLs when using ORDER BY on a nullable > column. They do, but it's not consistent: MS SQL Server: orders NULLs first (in table order; stable sort) Sybase ASE: orders NULLs first (in arbitrary order) Oracle: orders NULLs last (in arbitrary order) PostgreSQL: orders NULLs last (in arbitrary order) IBM DB2: orders NULLs last (in table order; stable sort) Reference: tests done with actual databases. A note about consistency: None is always ordered first in Python 2, so there is a precedent. And since Python's list.sort() is stable, Python 2 is in the same camp as MS SQL Server. >From Python-2.7.6/Objects/object.c: /* None is smaller than anything */ if (v == Py_None) return -1; if (w == Py_None) return 1; IMO, it was a mistake to have None return a TypeError in comparisons, since it makes many typical data operations fail, e.g. Python2: >>> l = [1,2,None,4,5,None,6] >>> l.sort() >>> l [None, None, 1, 2, 4, 5, 6] Python3: >>> l = [1,2,None,4,5,None,6] >>> l.sort() Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: NoneType() < int() -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 14 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From __peter__ at web.de Fri Feb 14 13:01:13 2014 From: __peter__ at web.de (Peter Otten) Date: Fri, 14 Feb 2014 13:01:13 +0100 Subject: [Python-Dev] Function to handle None in sort operation, was Re: python 3 niggle: None < 1 raises TypeError References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> Message-ID: M.-A. Lemburg wrote: > IMO, it was a mistake to have None return a TypeError in > comparisons, since it makes many typical data operations > fail, e.g. > > Python2: >>>> l = [1,2,None,4,5,None,6] >>>> l.sort() >>>> l > [None, None, 1, 2, 4, 5, 6] > > Python3: >>>> l = [1,2,None,4,5,None,6] >>>> l.sort() > Traceback (most recent call last): > File "", line 1, in > TypeError: unorderable types: NoneType() < int() While it is trivial to fix >>> sorted([1,2,None,4,5,None,6], ... key=lambda x: (x is None, x)) [1, 2, 4, 5, 6, None, None] maybe the key should be given a name like functools.none_first/none_last in order to offer a standard approach? On the other hand I'm not sure I like none_last(x) < none_last(y) From regebro at gmail.com Fri Feb 14 16:41:14 2014 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 14 Feb 2014 16:41:14 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <52FDFF33.40703@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> Message-ID: What I think is the biggest nitpick here is that you can't make a NULL object so that NULL is None evaluates as True. If you could, you could then easily make this NULL object evaluate as less than everything except itself and None, and use that consistently. But since this NULL is not None, that breaks any backwards compatibility if your SQL library mapped NULL to None in earlier versions. From status at bugs.python.org Fri Feb 14 18:07:10 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 14 Feb 2014 18:07:10 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140214170710.D766F56A81@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-02-07 - 2014-02-14) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4524 (+23) closed 27875 (+62) total 32399 (+85) Open issues with patches: 2052 Issues opened (66) ================== #16314: Support xz compression in distutils http://bugs.python.org/issue16314 reopened by Arfrever #20320: select.select(timeout) and select.kqueue.control(timeout) must http://bugs.python.org/issue20320 reopened by haypo #20543: ** operator does not overflow to inf http://bugs.python.org/issue20543 opened by Keith.Randall #20544: Use specific asserts in operator tests http://bugs.python.org/issue20544 opened by serhiy.storchaka #20545: Use specific asserts in unicode tests http://bugs.python.org/issue20545 opened by serhiy.storchaka #20547: Use specific asserts in bigmem tests http://bugs.python.org/issue20547 opened by serhiy.storchaka #20548: Use specific asserts in warnings and exceptions tests http://bugs.python.org/issue20548 opened by serhiy.storchaka #20550: Use specific asserts in collections tests http://bugs.python.org/issue20550 opened by serhiy.storchaka #20551: Use specific asserts in decimal tests http://bugs.python.org/issue20551 opened by serhiy.storchaka #20552: Use specific asserts in bytes tests http://bugs.python.org/issue20552 opened by serhiy.storchaka #20554: Use specific asserts in argparse and optparse tests http://bugs.python.org/issue20554 opened by serhiy.storchaka #20556: Use specific asserts in threading tests http://bugs.python.org/issue20556 opened by serhiy.storchaka #20557: Use specific asserts in io tests http://bugs.python.org/issue20557 opened by serhiy.storchaka #20558: ECONNRESET value in logging.config is valid with Linux [distro http://bugs.python.org/issue20558 opened by yaneurabeya #20559: urllib/http fail to sanitize a non-ascii url http://bugs.python.org/issue20559 opened by Dubslow #20560: tkFileFilter.c: ostypeCount not initialized? http://bugs.python.org/issue20560 opened by terry.reedy #20562: sqlite3 returns result set with doubled first entry http://bugs.python.org/issue20562 opened by jwezel #20564: test_threadsignals.py "failed" on OpenBSD because too slow (> http://bugs.python.org/issue20564 opened by rpointel #20565: Update Tcl/Tk version for buildbots http://bugs.python.org/issue20565 opened by serhiy.storchaka #20567: test_idle causes test_ttk_guionly 'can't invoke "event" comman http://bugs.python.org/issue20567 opened by ned.deily #20568: Pass --default-install to ensurepip in the Windows installers http://bugs.python.org/issue20568 opened by ncoghlan #20569: IDLE : Add clipboard history feature http://bugs.python.org/issue20569 opened by sahutd #20570: Bundle pip 1.5.3 in Python 3.4rc2 http://bugs.python.org/issue20570 opened by ncoghlan #20572: subprocess.Popen.wait() undocumented "endtime" parameter http://bugs.python.org/issue20572 opened by giampaolo.rodola #20573: "built-in repr()" function link on the repr module documentati http://bugs.python.org/issue20573 opened by rutsky #20574: Implement incremental decoder for cp65001 http://bugs.python.org/issue20574 opened by haypo #20575: Type handling policy for the statistics module http://bugs.python.org/issue20575 opened by oscarbenjamin #20577: IDLE: Remove FormatParagraph's width setting from config dialo http://bugs.python.org/issue20577 opened by taleinat #20578: BufferedIOBase.readinto1 is missing http://bugs.python.org/issue20578 opened by nikratio #20579: OS X IDLE keyboard accelerators fail or misbehave wi http://bugs.python.org/issue20579 opened by ned.deily #20580: IDLE should support platform-specific default config defaults http://bugs.python.org/issue20580 opened by ned.deily #20581: Incorrect behaviour of super() in a metaclass-created subclass http://bugs.python.org/issue20581 opened by jdetrey #20582: socket.getnameinfo() does not document flags http://bugs.python.org/issue20582 opened by roysmith #20584: On FreeBSD, signal.NSIG is smaller than biggest signal value http://bugs.python.org/issue20584 opened by jgehrcke #20585: urllib2 unrelease KQUEUE on Mac OSX 10.9+ http://bugs.python.org/issue20585 opened by Andrew.Gross #20586: Argument Clinic: functions with valid sig but no docstring hav http://bugs.python.org/issue20586 opened by zach.ware #20587: sqlite3 converter not being called http://bugs.python.org/issue20587 opened by kmk #20589: pathlib.owner() and pathlib.group() raise ImportError on Windo http://bugs.python.org/issue20589 opened by spiralx #20596: Support for alternate wcstok syntax for Windows compilers http://bugs.python.org/issue20596 opened by Jeffrey.Armstrong #20597: PATH_MAX already defined on some Windows compilers http://bugs.python.org/issue20597 opened by Jeffrey.Armstrong #20598: argparse docs: '7'.split() is confusing magic http://bugs.python.org/issue20598 opened by guettli #20600: test_create_server_ssl_verify_failed() failure on "PPC64 AIX 3 http://bugs.python.org/issue20600 opened by haypo #20601: tracing and tests that raise an exception in a SIGALRM handler http://bugs.python.org/issue20601 opened by xdegaye #20602: sys.flags and sys.float_info disappear at shutdown http://bugs.python.org/issue20602 opened by pitrou #20603: sys.path disappears at shutdown http://bugs.python.org/issue20603 opened by pitrou #20604: Missing invalid mode in error message of socket.makefile. http://bugs.python.org/issue20604 opened by kushou #20606: Operator Documentation Example doesn't work http://bugs.python.org/issue20606 opened by silent.bat #20607: multiprocessing cx_Freeze windows GUI bug (& easy fixes) http://bugs.python.org/issue20607 opened by mark #20608: 'SyntaxError: invalid token' is unfriendly http://bugs.python.org/issue20608 opened by mgedmin #20609: Always running kill_python breaks building x64 on 32-bit Windo http://bugs.python.org/issue20609 opened by sdeibel #20610: Documentation with two slashes in URL links to v2.6.5c2 http://bugs.python.org/issue20610 opened by Daniel.Ellis #20611: socket.create_connection() doesn't handle EINTR properly http://bugs.python.org/issue20611 opened by flox #20612: cElementTree has problems with StringIO object containing unic http://bugs.python.org/issue20612 opened by amcone #20613: Wrong line information in bytecode generated by compiler modul http://bugs.python.org/issue20613 opened by arnau #20614: test.script_helper should copy SYSTEMROOT environment variable http://bugs.python.org/issue20614 opened by haypo #20615: Replace PyDict_GetItem() with PyDict_GetItemWithError() http://bugs.python.org/issue20615 opened by haypo #20616: Add tracemalloc.Traceback.format() method http://bugs.python.org/issue20616 opened by haypo #20617: test_ssl does not use mock http://bugs.python.org/issue20617 opened by xdegaye #20620: Update the min()/max() docs for the new default argument http://bugs.python.org/issue20620 opened by rhettinger #20621: Issue with zipimport in 3.3.4 and 3.4.0rc1 http://bugs.python.org/issue20621 opened by pmoore #20622: Python3.3 venv pip fails to run if path contains spaces http://bugs.python.org/issue20622 opened by FeralBytes #20623: Run test_htmlparser with unbuffered source http://bugs.python.org/issue20623 opened by ezio.melotti #20624: Clarify recommendation to inherit from Exception http://bugs.python.org/issue20624 opened by mark.dickinson #20625: Argument names in __annotations__ are not mangled for function http://bugs.python.org/issue20625 opened by jonasw #20626: Manager documentation unclear about lists and thread safeness http://bugs.python.org/issue20626 opened by Irvin.Probst #20627: Add context manager support to xmlrpc.client.ServerProxy http://bugs.python.org/issue20627 opened by brett.cannon Most recent 15 issues with no replies (15) ========================================== #20627: Add context manager support to xmlrpc.client.ServerProxy http://bugs.python.org/issue20627 #20626: Manager documentation unclear about lists and thread safeness http://bugs.python.org/issue20626 #20625: Argument names in __annotations__ are not mangled for function http://bugs.python.org/issue20625 #20624: Clarify recommendation to inherit from Exception http://bugs.python.org/issue20624 #20623: Run test_htmlparser with unbuffered source http://bugs.python.org/issue20623 #20622: Python3.3 venv pip fails to run if path contains spaces http://bugs.python.org/issue20622 #20620: Update the min()/max() docs for the new default argument http://bugs.python.org/issue20620 #20617: test_ssl does not use mock http://bugs.python.org/issue20617 #20613: Wrong line information in bytecode generated by compiler modul http://bugs.python.org/issue20613 #20610: Documentation with two slashes in URL links to v2.6.5c2 http://bugs.python.org/issue20610 #20604: Missing invalid mode in error message of socket.makefile. http://bugs.python.org/issue20604 #20603: sys.path disappears at shutdown http://bugs.python.org/issue20603 #20602: sys.flags and sys.float_info disappear at shutdown http://bugs.python.org/issue20602 #20580: IDLE should support platform-specific default config defaults http://bugs.python.org/issue20580 #20579: OS X IDLE keyboard accelerators fail or misbehave wi http://bugs.python.org/issue20579 Most recent 15 issues waiting for review (15) ============================================= #20616: Add tracemalloc.Traceback.format() method http://bugs.python.org/issue20616 #20614: test.script_helper should copy SYSTEMROOT environment variable http://bugs.python.org/issue20614 #20609: Always running kill_python breaks building x64 on 32-bit Windo http://bugs.python.org/issue20609 #20607: multiprocessing cx_Freeze windows GUI bug (& easy fixes) http://bugs.python.org/issue20607 #20606: Operator Documentation Example doesn't work http://bugs.python.org/issue20606 #20604: Missing invalid mode in error message of socket.makefile. http://bugs.python.org/issue20604 #20601: tracing and tests that raise an exception in a SIGALRM handler http://bugs.python.org/issue20601 #20597: PATH_MAX already defined on some Windows compilers http://bugs.python.org/issue20597 #20596: Support for alternate wcstok syntax for Windows compilers http://bugs.python.org/issue20596 #20589: pathlib.owner() and pathlib.group() raise ImportError on Windo http://bugs.python.org/issue20589 #20586: Argument Clinic: functions with valid sig but no docstring hav http://bugs.python.org/issue20586 #20577: IDLE: Remove FormatParagraph's width setting from config dialo http://bugs.python.org/issue20577 #20574: Implement incremental decoder for cp65001 http://bugs.python.org/issue20574 #20569: IDLE : Add clipboard history feature http://bugs.python.org/issue20569 #20568: Pass --default-install to ensurepip in the Windows installers http://bugs.python.org/issue20568 Top 10 most discussed issues (10) ================================= #19255: Don't "wipe" builtins at shutdown http://bugs.python.org/issue19255 14 msgs #20572: subprocess.Popen.wait() undocumented "endtime" parameter http://bugs.python.org/issue20572 13 msgs #20414: Python 3.4 has two Overlapped types http://bugs.python.org/issue20414 12 msgs #20564: test_threadsignals.py "failed" on OpenBSD because too slow (> http://bugs.python.org/issue20564 10 msgs #20053: venv and ensurepip are affected by default pip config file http://bugs.python.org/issue20053 9 msgs #20539: math.factorial may throw OverflowError http://bugs.python.org/issue20539 9 msgs #20596: Support for alternate wcstok syntax for Windows compilers http://bugs.python.org/issue20596 9 msgs #17159: Remove explicit type check from inspect.Signature.from_functio http://bugs.python.org/issue17159 8 msgs #19494: urllib2.HTTPBasicAuthHandler (or urllib.request.HTTPBasicAuthH http://bugs.python.org/issue19494 8 msgs #20320: select.select(timeout) and select.kqueue.control(timeout) must http://bugs.python.org/issue20320 8 msgs Issues closed (59) ================== #4708: os.pipe should return inheritable descriptors (Windows) http://bugs.python.org/issue4708 closed by haypo #6501: Fatal error on startup with invalid PYTHONIOENCODING http://bugs.python.org/issue6501 closed by haypo #12296: Minor clarification in devguide http://bugs.python.org/issue12296 closed by ezio.melotti #13414: test_strftime failed on OpenBSD http://bugs.python.org/issue13414 closed by rpointel #14983: email.generator should always add newlines after closing bound http://bugs.python.org/issue14983 closed by r.david.murray #16983: header parsing could apply postel's law to encoded words insid http://bugs.python.org/issue16983 closed by r.david.murray #17671: io.BufferedRWPair can use uninitialized members http://bugs.python.org/issue17671 closed by serhiy.storchaka #18805: ipaddress netmask/hostmask parsing bugs http://bugs.python.org/issue18805 closed by ncoghlan #19465: selectors: provide a helper to choose a selector using constra http://bugs.python.org/issue19465 closed by gvanrossum #19680: Help missing for exec and print http://bugs.python.org/issue19680 closed by ezio.melotti #19751: test_sys: sys.hash_info.algorithm failure on SPARC Solaris bui http://bugs.python.org/issue19751 closed by haypo #19772: str serialization of Message object may mutate the payload and http://bugs.python.org/issue19772 closed by r.david.murray #19856: shutil.move() can't move a directory in non-empty directory on http://bugs.python.org/issue19856 closed by serhiy.storchaka #19906: Typo in urllib documentation http://bugs.python.org/issue19906 closed by ezio.melotti #20013: imaplib behaviour when deleting selected folder http://bugs.python.org/issue20013 closed by r.david.murray #20065: Python-3.3.3/Modules/socketmodule.c:1660:14: error: 'CAN_RAW' http://bugs.python.org/issue20065 closed by neologix #20089: email.message_from_string no longer working in Python 3.4 http://bugs.python.org/issue20089 closed by r.david.murray #20139: Python installer does not install a "pip" command (just "pip3" http://bugs.python.org/issue20139 closed by ncoghlan #20213: Change the install location of _sysconfigdata.py http://bugs.python.org/issue20213 closed by thomas-petazzoni #20288: HTMLParse handing of non-numeric charrefs broken http://bugs.python.org/issue20288 closed by ezio.melotti #20406: Use application icon for IDLE http://bugs.python.org/issue20406 closed by terry.reedy #20437: Use Py_CLEAR to safe clear attributes http://bugs.python.org/issue20437 closed by serhiy.storchaka #20462: Python IDLE auto closed when creating a new file or open exist http://bugs.python.org/issue20462 closed by terry.reedy #20478: Avoid inadvertently special casing Counter in statistics modul http://bugs.python.org/issue20478 closed by ncoghlan #20481: Clarify type coercion rules in statistics module http://bugs.python.org/issue20481 closed by python-dev #20495: test_read_pty_output() hangs on FreeBSD 7.2 buildbot http://bugs.python.org/issue20495 closed by haypo #20500: assertion failed when passing an exception object to sys.exit http://bugs.python.org/issue20500 closed by python-dev #20505: Remove resolution from selectors and granularity from asyncio http://bugs.python.org/issue20505 closed by haypo #20515: Null pointer dereference in tkinter module http://bugs.python.org/issue20515 closed by serhiy.storchaka #20517: Support errors with two filenames for errno exceptions http://bugs.python.org/issue20517 closed by larry #20526: python: Modules/gcmodule.c:379: visit_decref: Assertion `((gc) http://bugs.python.org/issue20526 closed by haypo #20529: Unittest displays ResourceWarning warnings when running tests, http://bugs.python.org/issue20529 closed by serhiy.storchaka #20530: Change the text signature format (again) to be more robust http://bugs.python.org/issue20530 closed by larry #20531: TypeError in e-mail.parser when non-ASCII is present http://bugs.python.org/issue20531 closed by r.david.murray #20532: Mark all tests which use _testcapi as CPython only http://bugs.python.org/issue20532 closed by serhiy.storchaka #20536: statistics._decimal_to_ratio() produces non-integer ratio http://bugs.python.org/issue20536 closed by python-dev #20538: Segfault in UTF-7 incremental decoder http://bugs.python.org/issue20538 closed by serhiy.storchaka #20540: Python 3.3/3.4 regression in multiprocessing manager ? http://bugs.python.org/issue20540 closed by pitrou #20546: Use specific asserts in int tests http://bugs.python.org/issue20546 closed by serhiy.storchaka #20549: Use specific asserts in mailbox, smtplib and poplib tests http://bugs.python.org/issue20549 closed by serhiy.storchaka #20553: Use specific asserts in ipaddress tests http://bugs.python.org/issue20553 closed by serhiy.storchaka #20555: Use specific asserts in urllib related tests http://bugs.python.org/issue20555 closed by serhiy.storchaka #20561: Decimal handling error in statistics module http://bugs.python.org/issue20561 closed by ncoghlan #20563: Mark ipaddress as stable? http://bugs.python.org/issue20563 closed by python-dev #20566: asyncio as_completed() thrashes adding and removing callbacks http://bugs.python.org/issue20566 closed by gvanrossum #20571: test_codecs currently failing on several Windows buildbots http://bugs.python.org/issue20571 closed by haypo #20576: Default signatures for slots should be positional-only http://bugs.python.org/issue20576 closed by larry #20583: Hide underscored parameters http://bugs.python.org/issue20583 closed by serhiy.storchaka #20588: Code generated by asdl_c.py not C89-compliant http://bugs.python.org/issue20588 closed by python-dev #20590: spam http://bugs.python.org/issue20590 closed by benjamin.peterson #20591: Let "braces" be a constant http://bugs.python.org/issue20591 closed by SilentGhost #20592: test_frozen fails on 3.4.0rc1, windows http://bugs.python.org/issue20592 closed by terry.reedy #20593: test_importhooks fails on 3.4 http://bugs.python.org/issue20593 closed by brett.cannon #20594: [PATCH] fail to compile posixmodule due to name clash with pos http://bugs.python.org/issue20594 closed by python-dev #20595: C89 compliance issue in Python/getargs.c http://bugs.python.org/issue20595 closed by python-dev #20599: test_cleanup() of test_builtin failed http://bugs.python.org/issue20599 closed by haypo #20605: test_socket (testGetaddrinfo) failing on OS X 10.6.8 (32-bit) http://bugs.python.org/issue20605 closed by ned.deily #20618: Unable to pip install pyjasperclient http://bugs.python.org/issue20618 closed by r.david.murray #20619: tracemalloc changes col_offset attribute for _ast.arg objects http://bugs.python.org/issue20619 closed by python-dev From chris.barker at noaa.gov Fri Feb 14 18:19:55 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 14 Feb 2014 09:19:55 -0800 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> Message-ID: On Fri, Feb 14, 2014 at 2:42 AM, Nick Coghlan wrote: > That's the main reason I still occasionally wonder if > "AlwaysMin" and "AlwaysMax" singletons might make sense, although that > approach has problems of its own (specifically, it's hard to actually > use without breaking the clean NULL -> None mapping and in many cases > +/- infinity already work fine). only for floating point -- it would be nice to have them for other types: integers, datetimes, any others that make sense.... ( In know I've wanted +-inf and NaN for integers forever, though what I really want is hardware support) But do you do that with a generic type or a special one for each type? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Feb 14 18:55:46 2014 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 14 Feb 2014 17:55:46 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <52FDCE0D.5090402@simplistix.co.uk> References: <52FDCE0D.5090402@simplistix.co.uk> Message-ID: <52FE58A2.6000703@mrabarnett.plus.com> On 2014-02-14 08:04, Chris Withers wrote: > Hi All, > > Sending this to python-dev as I'm wondering if this was considered when > the choice to have objects of different types raise a TypeError when > ordered... > > So, the concrete I case I have is implementing stable ordering for the > python Range objects that psycopg2 uses. These have 3 attributes that > can either be None or, for sake of argument, a numeric value. > > To implement __lt__ in Python 2, I could do: > > def __lt__(self, other): > if not isinstance(other, Range): > return True > return ((self._lower, self._upper, self._bounds) < > (other._lower, other._upper, other._bounds)) > > Because None < 1 raises a TypeError, in Python 3 I have to do: > > def __lt__(self, other): > if not isinstance(other, Range): > return NotImplemented > for attr in '_lower', '_upper', '_bounds': > self_value = getattr(self, attr) > other_value = getattr(other, attr) > if self_value == other_value: > pass > elif self_value is None: > return True > elif other_value is None: > return False > else: > return self_value < other_value > return False > > Am I missing something? How can I get this method down to a sane size? > How about this: def make_key(item): return [(x is not None, x or 0) for i in [item._lower, item._upper, item._bounds]] def __lt__(self, other): if not isinstance(other, Range): return NotImplemented return make_key(self) < make_key(other) It'll make None less than any number. From chris.barker at noaa.gov Fri Feb 14 18:14:53 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 14 Feb 2014 09:14:53 -0800 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> Message-ID: On Fri, Feb 14, 2014 at 1:29 AM, Nick Coghlan wrote: > On 14 February 2014 18:04, Chris Withers wrote: > > > > Am I missing something? How can I get this method down to a sane size? > > The easiest way is usually to normalise the attributes to a sensible > numeric value depending on where you want "None" to sort (positive and > negative infinity floating point values often work well in the case of > numeric data, but a custom AlwaysLess or AlwaysGreater type works for > arbitrary data). This is actually a pretty common case -- the question here is: Is this really None? or does None have a special meaning. It sounds like this is a wrapper for a PostgreSQL range object, in which case None isn't really right, there must be a +-infinity concept there -- how does postgress handle it? (is this a generic range, or a datetime range? I'm not up on it). If this means what I think it does, None is really a hack, both because of al lteh specia case code required for sorting, comparing, etc, but also because it's missing information -- the two ends of the range should really have different meanings. I've dealt with this for Python datetime via a pari fo custom InfDateTime objects one for plu infiinity, one for minus. Then we can use these in ranges and the reest of teh code can jsut do simple comparisons, sorting etc. Of course, floating point numbers have inf and -inf, which is really useful. I hadn't though of Nick's generic AlwaysLess or AlwaysGreater type -- if that means what I think it does -- i,e, it would compare appropriately to ANY other object, that would actually be a nice addition to the standard library. I had been thinking that the standard library datetime could use these, but why not just use generic ones? (though it could get a bit tricky -- what would AlwaysGreater > float('inf") evaluate to? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 14 22:41:21 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 15 Feb 2014 08:41:21 +1100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> Message-ID: On Sat, Feb 15, 2014 at 4:14 AM, Chris Barker wrote: > (though it could get a bit tricky -- what would AlwaysGreater > float('inf") > evaluate to? > It'd be true. AlwaysGreater is greater than infinity. It is greater than float("nan"). It is greater than the entire concept of floating point. It is greater than everything imaginable... except for a nice MLT - a mutton, lettuce, and tomato sandwich... oh. Where things get tricky is when you compare two AlwaysGreater objects. Or even compare one against itself. It has to be greater. Great. ChrisA From ijmorlan at uwaterloo.ca Fri Feb 14 22:42:44 2014 From: ijmorlan at uwaterloo.ca (Isaac Morland) Date: Fri, 14 Feb 2014 16:42:44 -0500 (EST) Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> Message-ID: On Fri, 14 Feb 2014, Chris Barker wrote: > This is actually a pretty common case -- the question here is: Is this > really None? or does None have a special meaning. It sounds like this is a > wrapper for a?PostgreSQL?range object, in which case None isn't really > right, there must be a +-infinity concept there -- how does postgress handle > it? (is this a generic range, or a datetime range? I'm not up on it). > If this means what I think it does, None is really a hack, both because of > al lteh specia case code required for sorting, comparing, etc, but also > because it's missing information -- the two ends of the range should really > have different meanings. Postgres range types are a separate kind of type which can be created using CREATE TYPE: http://www.postgresql.org/docs/current/interactive/sql-createtype.html Several handy range types are built in, although not as many as one might expect. Postgres ranges can be made infinite by specifying NULL as a bound. There isn't a separate special value for this. This has somewhat strange effects when the base type of the range has its own "infinity" value: => select upper_inf (daterange (current_date, 'infinity')); upper_inf ----------- f (1 row) => select isfinite ('infinity'::date); isfinite ---------- f (1 row) => select upper_inf (daterange (current_date, null)); upper_inf ----------- t (1 row) In other words, a range with non-NULL bounds is finite according to the range functions even if the bound is the infinity value of the base type. Isaac Morland CSCF Web Guru DC 2619, x36650 WWW Software Specialist From tjreedy at udel.edu Fri Feb 14 23:10:36 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 14 Feb 2014 17:10:36 -0500 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> Message-ID: The idea of top and bottom objects, by whatever name, has be proposed, discussed, and rejected on python-ideas list (which is where this discussion really belongs if continued). On 2/14/2014 4:41 PM, Chris Angelico wrote: >> (though it could get a bit tricky -- what would AlwaysGreater > float('inf") >> evaluate to? >> > > It'd be true. AlwaysGreater is greater than infinity. It is greater > than float("nan"). It is greater than the entire concept of floating > point. It is greater than everything imaginable... except for a nice > MLT - a mutton, lettuce, and tomato sandwich... oh. > > Where things get tricky is when you compare two AlwaysGreater objects. > Or even compare one against itself. It has to be greater. equal, and less than itself. Or not, depending on exactly how the methods are defined and coded. Which, as I remember, is part of why the idea of a *generic* class was rejected. Real cases are better served by custom classes that meet the custom need. -- Terry Jan Reedy From ncoghlan at gmail.com Sat Feb 15 01:27:28 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 15 Feb 2014 10:27:28 +1000 Subject: [Python-Dev] Function to handle None in sort operation, was Re: python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> Message-ID: On 14 February 2014 22:01, Peter Otten <__peter__ at web.de> wrote: > M.-A. Lemburg wrote: > >> IMO, it was a mistake to have None return a TypeError in >> comparisons, since it makes many typical data operations >> fail, e.g. >> >> Python2: >>>>> l = [1,2,None,4,5,None,6] >>>>> l.sort() >>>>> l >> [None, None, 1, 2, 4, 5, 6] >> >> Python3: >>>>> l = [1,2,None,4,5,None,6] >>>>> l.sort() >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: unorderable types: NoneType() < int() > > While it is trivial to fix > >>>> sorted([1,2,None,4,5,None,6], > ... key=lambda x: (x is None, x)) > [1, 2, 4, 5, 6, None, None] > > maybe the key should be given a name like functools.none_first/none_last in > order to offer a standard approach? Yeah, I was thinking something similar. I posted an RFE if anyone wants to run with the idea for Python 3.5: http://bugs.python.org/issue20630 (it's also the kind of thing that is quite amenable to inclusion in libraries like six and future for Python 2/3 compatible code bases - in Python 2, it can just be a pass through, while in Python 3 it can actually alter the sort operation to allow None values). At a broaded scale, this thread made me realise it may be worth defining a __key__ protocol so custom comparisons are easier to implement correctly: http://bugs.python.org/issue20632 That would need a PEP, but currently, there are lots of things you need to deal with around hashability, comparison with other types, etc when attempting to define a custom ordering. A new __key__ protocol could provide sensible default behaviour for all of those, such that you only needed to define the other methods if you wanted to override the defaults for some reason. (As noted in the issue, this is also amenable to being implemented as a third party module prior to proposing it as a language feature) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Feb 15 01:33:37 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 15 Feb 2014 10:33:37 +1000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> Message-ID: On 15 February 2014 08:10, Terry Reedy wrote: > The idea of top and bottom objects, by whatever name, has be proposed, > discussed, and rejected on python-ideas list (which is where this discussion > really belongs if continued). A fair point, but my recollection of those discussions is that we completely missed the use case of being able to sensibly map SQL NULL handling in ORDER BY clauses to the sorting of Python 3 containers. Having a concrete use case with widespread implications makes a big difference :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Sat Feb 15 07:03:59 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 15 Feb 2014 15:03:59 +0900 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <52FDFF33.40703@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> Message-ID: <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> M.-A. Lemburg writes: > IMO, it was a mistake to have None return a TypeError in > comparisons, since it makes many typical data operations > fail, e.g. I don't understand this statement. The theory is that they *should* fail. The example of sort is a good one. Sometimes you want missing values to be collected at the beginning of a list, sometimes at the end. Sometimes you want them treated as top elements, sometimes as bottom. And sometimes it is a real error for missing values to be present. Not to mention that sometimes the programmer simply hasn't thought about the appropriate policy. I don't think Python should silently impose a policy in that case, especially given that the programmer may have experience with any of the above treatments in other contexts. From nitinagarwal3006 at gmail.com Sat Feb 15 06:23:51 2014 From: nitinagarwal3006 at gmail.com (Nitin Agarwal) Date: Sat, 15 Feb 2014 10:23:51 +0500 Subject: [Python-Dev] Would like to contribute to Core Python Comunity on the Project Idea Message-ID: Hi, Here, its Nitin Agarwal, an open source software developer and enthusiast. Currently, I am pursuing Computer Science and Engineering at International Institute of Information Technology, INDIA. I had an experience working with the open source organisations and development of open source software. You can have a look at my github profile.[1] My first priority programming languages include GNU C/C++ and Python. I have been using these languages for the Coding, Programming challenges, Application development and contributing to open source software. Apart from this, I am also aware of version control systems git/github, SVN, bazaar and other web technologies like Javascript, Jinja templates, HTML, CSS too. I am ambititous person eager to learn other technologies when in need while working. You can have a look at my Wikipedia profile.[2] My interests include Programming Languages, open source software development, networking concepts and system administration tasks. While looking at the Project Ideas proposed by the mentors and developers of the Python Community, I found the Email Projects working to work for support of RFC 6532 in the email package and support of RFC 6531 in the smtplib and/or smtpd. I am willing to work on this project ideas as a prospective student in the upcoming Google Summer of Code 2014. I would like to discuss with the mentors more about the project in detail and would like to start off contributing to the project early enough so as to make a strong proposal for Gsoc 2014. *Nitin Agarwal* [1] https://github.com/NitinAgarwal [2] https://en.wikipedia.org/wiki/User:Nitinagarwal3006 [3] IRC handle : nitinagarwal3006 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jessica.mckellar at gmail.com Sat Feb 15 13:52:55 2014 From: jessica.mckellar at gmail.com (Jessica McKellar) Date: Sat, 15 Feb 2014 07:52:55 -0500 Subject: [Python-Dev] Would like to contribute to Core Python Comunity on the Project Idea In-Reply-To: References: Message-ID: Hi Nitin, Thanks for your interest in CPython for Google Summer of Code. We are still getting our ducks in a row, and mentor organizations haven't been finalized yet (the official mentor list will go out on February 24th). Can you check back in then? In the meantime, I recommend: 1. Joining http://pythonmentors.com/ 2. Reading through the developers guide: http://docs.python.org/devguide/ 3. Picking and working on a ticket to get used to the workflow: http://bugs.python.org/ Regards, -Jessica On Sat, Feb 15, 2014 at 12:23 AM, Nitin Agarwal wrote: > Hi, > > Here, its Nitin Agarwal, an open source software developer and enthusiast. > Currently, I am pursuing Computer Science and Engineering at International > Institute of Information Technology, INDIA. I had an experience working > with > the open source organisations and development of open source software. > > > You can have a look at my github profile.[1] My first priority > programming languages > include GNU C/C++ and Python. I have been using these languages for the > Coding, > Programming challenges, Application development and contributing to open > source > software. Apart from this, I am also aware of version control systems > git/github, SVN, > bazaar and other web technologies like Javascript, Jinja templates, HTML, > CSS too. > I am ambititous person eager to learn other technologies when in need > while working. > You can have a look at my Wikipedia profile.[2] > My interests include Programming Languages, open source software > development, > networking concepts and system administration tasks. > > > While looking at the Project Ideas proposed by the mentors and developers > of the Python > Community, I found the Email Projects working to work for support of RFC > 6532 in the > email package and support of RFC 6531 in the smtplib and/or smtpd. > > I am willing to work on this project ideas as a prospective student in the > upcoming > Google Summer of Code 2014. I would like to discuss with the mentors more > about > the project in detail and would like to start off contributing to the > project early enough > so as to make a strong proposal for Gsoc 2014. > > > > *Nitin Agarwal* > > > [1] https://github.com/NitinAgarwal > [2] https://en.wikipedia.org/wiki/User:Nitinagarwal3006 > [3] IRC handle : nitinagarwal3006 > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jessica.mckellar%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nitinagarwal3006 at gmail.com Sat Feb 15 16:06:43 2014 From: nitinagarwal3006 at gmail.com (Nitin Agarwal) Date: Sat, 15 Feb 2014 20:06:43 +0500 Subject: [Python-Dev] Would like to contribute to Core Python Comunity on the Project Idea In-Reply-To: References: Message-ID: Thanks for your reply. I have joined on the pythonmentors.com and gone through the developer guide. Yes, I am going through the present bugs to resolve some of them. Could you Please reply as soon as you get the mentors allocated for the Possible email projects being proposed for Gsoc 2014. It will help me discuss the project idea with the mentor so as to make a good proposal and get a great work done in the 3 months of upcoming Summer of Code. *Nitin Agarwal* On Sat, Feb 15, 2014 at 5:52 PM, Jessica McKellar < jessica.mckellar at gmail.com> wrote: > Hi Nitin, > > Thanks for your interest in CPython for Google Summer of Code. > > We are still getting our ducks in a row, and mentor organizations haven't > been finalized yet (the official mentor list will go out on February 24th). > Can you check back in then? > > In the meantime, I recommend: > > 1. Joining http://pythonmentors.com/ > 2. Reading through the developers guide: http://docs.python.org/devguide/ > 3. Picking and working on a ticket to get used to the workflow: > http://bugs.python.org/ > > Regards, > -Jessica > > > > > > On Sat, Feb 15, 2014 at 12:23 AM, Nitin Agarwal < > nitinagarwal3006 at gmail.com> wrote: > >> Hi, >> >> Here, its Nitin Agarwal, an open source software developer and >> enthusiast. >> Currently, I am pursuing Computer Science and Engineering at International >> Institute of Information Technology, INDIA. I had an experience working >> with >> the open source organisations and development of open source software. >> >> >> You can have a look at my github profile.[1] My first priority >> programming languages >> include GNU C/C++ and Python. I have been using these languages for the >> Coding, >> Programming challenges, Application development and contributing to open >> source >> software. Apart from this, I am also aware of version control systems >> git/github, SVN, >> bazaar and other web technologies like Javascript, Jinja templates, HTML, >> CSS too. >> I am ambititous person eager to learn other technologies when in need >> while working. >> You can have a look at my Wikipedia profile.[2] >> My interests include Programming Languages, open source software >> development, >> networking concepts and system administration tasks. >> >> >> While looking at the Project Ideas proposed by the mentors and developers >> of the Python >> Community, I found the Email Projects working to work for support of RFC >> 6532 in the >> email package and support of RFC 6531 in the smtplib and/or smtpd. >> >> I am willing to work on this project ideas as a prospective student in >> the upcoming >> Google Summer of Code 2014. I would like to discuss with the mentors more >> about >> the project in detail and would like to start off contributing to the >> project early enough >> so as to make a strong proposal for Gsoc 2014. >> >> >> >> *Nitin Agarwal* >> >> >> [1] https://github.com/NitinAgarwal >> [2] https://en.wikipedia.org/wiki/User:Nitinagarwal3006 >> [3] IRC handle : nitinagarwal3006 >> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/jessica.mckellar%40gmail.com >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat Feb 15 17:52:51 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 15 Feb 2014 18:52:51 +0200 Subject: [Python-Dev] Add Py_REPLACE and Py_XREPLACE macros In-Reply-To: References: Message-ID: 29.01.14 20:24, Serhiy Storchaka ???????(??): > The Py_CLEAR macros is used as safe alternative for following unsafe > idiomatic code: > > Py_XDECREF(ptr); > ptr = NULL; > > But other unsafe idiomatic code is widely used in the sources: > > Py_XDECREF(ptr); > ptr = new_value; > > Every occurrence of such code is potential bug for same reasons as for > Py_CLEAR. > > It was offered [1] to introduce new macros Py_REPLACE and Py_XREPLACE > for safe replace with Py_DECREF and Py_XDECREF respectively. > Automatically generated patch contains about 50 replaces [2]. > > [1] http://bugs.python.org/issue16447 > [2] http://bugs.python.org/issue20440 There are objections to these patches. Raymond against backporting the patch unless some known bugs are being fixed [1]. But it fixes at least one bug that caused a crash. And I suspect that there may be other bugs, just we still have no reproducers. Even if we don't know how to reproduce the bug, the current code looks obviously wrong. Also porting the patch will make the sync easier. Note that the patches were automatically generated, which reduces the possibility of errors. I just slightly corrected formatting, remove unused variables and fixed one error. Martin's objections are that the macros do add to the learning curve and his expects that Py_REPLACE(op, op2) does an INCREF on op2, but it does not [2]. Antoine's original Py_(X)SETREF macros did INCREF and seems this was one of their flaw, because in most cases INCREF is not needed. Alternative names Py_(X)ASSIGN were suggested to break connotation of an INCREF [3]. As for learning curve, I think that it would be better to have standard macros for code that can be written (and often are written) incorrectly. And even already correct code can be written with these macros in more short and clear way [4]. Greg shared Martin's expectation and suggested to revive this thread. [1] http://bugs.python.org/issue20440#msg209701 [2] http://bugs.python.org/issue20440#msg209894 [3] http://bugs.python.org/issue20440#msg210447 [4] http://bugs.python.org/issue3081#msg102645 From storchaka at gmail.com Sat Feb 15 19:12:33 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 15 Feb 2014 20:12:33 +0200 Subject: [Python-Dev] Using more specific methods in Python unit tests Message-ID: Many Python tests were written a very long time before the unittest, using simple asserts. Then, when they have been ported to the unittest, asserts were replaced with the assert_ method and then with assertTrue. The unittest has a number of other methods to check for and report failure, from assertEqual, to more specific assertIs, assertIn, assertIsInstance, etc, added in 2.7. New methods provide better reporting in case of failure. I wrote a large patch which modifies the tests to use more specific methods [1]. Because it is too large, it was divided into many smaller patches, and separate issues were opened for them. At the moment the major part of the original patch has already been committed. Many thanks to Ezio for making a review for the majority of the issues. Some changes have been made by other people in unrelated issues. Although Raymond approved a patch for test_bigmem [2], his expressed the insistent recommendation not to do this. So I stop committing new reviewed patches. Terry recommended to discuss this in Python-Dev. What are your thoughts? [1] http://bugs.python.org/issue16510 [2] http://bugs.python.org/issue20547 From storchaka at gmail.com Sat Feb 15 20:01:36 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 15 Feb 2014 21:01:36 +0200 Subject: [Python-Dev] Pickling of Enums Message-ID: How Enum items should be pickled, by value or by name? I think that Enum will be used to collect system-depending constants, so the value of AddressFamily.AF_UNIX can be 1 on one platform and 2 on other. If pickle enums by value, then pickled AddressFamily.AF_INET on on platform can be unpickled as AddressFamily.AF_UNIX on other platform. This looks weird and contrary to the nature of enums. From benjamin at python.org Sat Feb 15 20:24:10 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 15 Feb 2014 11:24:10 -0800 Subject: [Python-Dev] Using more specific methods in Python unit tests In-Reply-To: References: Message-ID: <1392492250.26338.83831085.39A5ED08@webmail.messagingengine.com> On Sat, Feb 15, 2014, at 10:12 AM, Serhiy Storchaka wrote: > Many Python tests were written a very long time before the unittest, > using simple asserts. Then, when they have been ported to the unittest, > asserts were replaced with the assert_ method and then with assertTrue. > The unittest has a number of other methods to check for and report > failure, from assertEqual, to more specific assertIs, assertIn, > assertIsInstance, etc, added in 2.7. New methods provide better > reporting in case of failure. > > I wrote a large patch which modifies the tests to use more specific > methods [1]. Because it is too large, it was divided into many smaller > patches, and separate issues were opened for them. At the moment the > major part of the original patch has already been committed. Many thanks > to Ezio for making a review for the majority of the issues. Some changes > have been made by other people in unrelated issues. > > Although Raymond approved a patch for test_bigmem [2], his expressed the > insistent recommendation not to do this. So I stop committing new > reviewed patches. Terry recommended to discuss this in Python-Dev. What > are your thoughts? I tend to agree with Raymond. I think such changes are very welcome when the module or tests are otherwise being changed, but on their on constitute unnecessary churn. From solipsis at pitrou.net Sat Feb 15 20:37:46 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 15 Feb 2014 20:37:46 +0100 Subject: [Python-Dev] Pickling of Enums References: Message-ID: <20140215203746.66e951e1@fsol> On Sat, 15 Feb 2014 21:01:36 +0200 Serhiy Storchaka wrote: > How Enum items should be pickled, by value or by name? > > I think that Enum will be used to collect system-depending constants, so > the value of AddressFamily.AF_UNIX can be 1 on one platform and 2 on > other. If pickle enums by value, then pickled AddressFamily.AF_INET on > on platform can be unpickled as AddressFamily.AF_UNIX on other platform. > This looks weird and contrary to the nature of enums. I agree with you, they should be pickled by name. An enum is a kind of global in this regard. (but of course, before AF_UNIX was an enum it was pickled by value) Regards Antoine. From tjreedy at udel.edu Sat Feb 15 20:55:31 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 15 Feb 2014 14:55:31 -0500 Subject: [Python-Dev] Using more specific methods in Python unit tests In-Reply-To: References: Message-ID: On 2/15/2014 1:12 PM, Serhiy Storchaka wrote: > Many Python tests were written a very long time before the unittest, > using simple asserts. Then, when they have been ported to the unittest, > asserts were replaced with the assert_ method and then with assertTrue. > The unittest has a number of other methods to check for and report > failure, from assertEqual, to more specific assertIs, assertIn, > assertIsInstance, etc, added in 2.7. New methods provide better > reporting in case of failure. Failure of assertTrue reports 'False is not true'. > I wrote a large patch which modifies the tests to use more specific > methods [1]. Because it is too large, it was divided into many smaller > patches, and separate issues were opened for them. At the moment the > major part of the original patch has already been committed. Many thanks > to Ezio for making a review for the majority of the issues. Some changes > have been made by other people in unrelated issues. > > Although Raymond approved a patch for test_bigmem [2], his expressed the > insistent recommendation not to do this. So I stop committing new > reviewed patches. Terry recommended to discuss this in Python-Dev. What > are your thoughts? > > [1] http://bugs.python.org/issue16510 > [2] http://bugs.python.org/issue20547 After thinking about Raymond's objections and checking http://docs.python.org/3/library/unittest.html#test-cases and noting Ezio's explicit approval and others tacit approval (by lack of objection), I think you should continue and finish. The reasons for not making making style changes in public stdlib modules are that we want people to import and use them, have promised stability absent notice otherwise, and because it is reasonably possible to make unintended semantic changes that might pass unnoticed by incomplete tests. The case is different on all counts for test.text_xyz modules. They are private application modules, not to be imported (except to run them), and subject to change without notice. Nearly all the changes proposed are from assertTrue to various specialized assertXs with documented equivalences to assertTrue. I presume you made the 1-to-1 replacements with a script using capturing regexes, with a check for false replacements such as might occasionally happen due to an operator being inside a string. If we are ever to make the replacements, doing so mechanically and in bulk is easier and safer than doing them one at a time by hand. This includes reviewing. -- Terry Jan Reedy From p.f.moore at gmail.com Sat Feb 15 23:19:15 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 15 Feb 2014 22:19:15 +0000 Subject: [Python-Dev] Possible major bug with zipimport on Windows in Python 3.3.4 In-Reply-To: References: <84841BB3-CAE7-44DE-A854-A6428DEC743D@stufft.io> Message-ID: On 13 February 2014 20:58, Paul Moore wrote: > On 13 February 2014 20:54, Donald Stufft wrote: >> >> On Feb 13, 2014, at 3:53 PM, Paul Moore wrote: >> >>> Can someone please take a look at http://bugs.python.org/issue20621 >>> for me? It appears that there is a significant problem with zipimport >>> in 3.3.4. It's causing breakage in virtualenv but I've isolated the >>> issue to what I think is Python itself. >>> >>> Basically, until this bug is fixed, virtualenv won't work on Windows >>> with Python 3.3.4, as far as I can tell. >> >> Does it affect 3.4? > > Just checked that. No, it's not an issue in 3.4rc0. Nor is it in > 3.3.3, so it's a regression in the 3.3 series only. OK, this appears to be a serious regression that breaks virtualenv. Steve Dower suggested a workaround on the tracker, but I don't think that can be made to work in the case of virtualenv (for reasons not worth explaining now, ask me if you want the details though). Given where we are, what is the way forward here? I'm not entirely sure what I should be saying to the user who reported the bug in virtualenv. I think I'll recommend downgrading to 3.3.3 for now, but it would be good to be able to give an idea of what the longer term solution is likely to be. Thanks Paul From nad at acm.org Sun Feb 16 00:20:28 2014 From: nad at acm.org (Ned Deily) Date: Sat, 15 Feb 2014 15:20:28 -0800 Subject: [Python-Dev] Using more specific methods in Python unit tests References: <1392492250.26338.83831085.39A5ED08@webmail.messagingengine.com> Message-ID: In article <1392492250.26338.83831085.39A5ED08 at webmail.messagingengine.com>, Benjamin Peterson wrote: > On Sat, Feb 15, 2014, at 10:12 AM, Serhiy Storchaka wrote: > > Although Raymond approved a patch for test_bigmem [2], his expressed the > > insistent recommendation not to do this. So I stop committing new > > reviewed patches. Terry recommended to discuss this in Python-Dev. What > > are your thoughts? > I tend to agree with Raymond. I think such changes are very welcome when > the module or tests are otherwise being changed, but on their on > constitute unnecessary churn. +1 Integrity of the test suite and minimizing code churn top any benefits of more specific messages on failures. The expectation is that most tests will never fail so their changed messages will never be seen. For the rare cases when a test does fail, quite often the test was written in a way that will require examination of the code to understand exactly what the test case was intending to test and why it failed. Having a more specific exception message wouldn't help for many tests without further modifications; the key point is to know that the test failed. -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Sun Feb 16 01:05:54 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Feb 2014 10:05:54 +1000 Subject: [Python-Dev] Add Py_REPLACE and Py_XREPLACE macros In-Reply-To: References: Message-ID: On 16 February 2014 02:52, Serhiy Storchaka wrote: > There are objections to these patches. Raymond against backporting the patch > unless some known bugs are being fixed [1]. But it fixes at least one bug > that caused a crash. And I suspect that there may be other bugs, just we > still have no reproducers. Even if we don't know how to reproduce the bug, > the current code looks obviously wrong. Also porting the patch will make the > sync easier. Note that the patches were automatically generated, which > reduces the possibility of errors. I just slightly corrected formatting, > remove unused variables and fixed one error. It's also likely than many of these crashes could only be reproduced through incorrect usage of the C API. Py_CLEAR/Py_XCLEAR was relatively simple, as there's only one pointer involved. As Martin noted, Py_REPLACE is more complex, as there's the question of whether or not the refcount for the RHS gets incremented. Rather than using a completely different name, perhaps we can leverage "Py_CLEAR" to make it more obvious that this operation is "like Py_CLEAR and for the same reason, just setting the pointer to something other than NULL". For example: Py_CLEAR_AND_SET Py_XCLEAR_AND_SET Such that Py_CLEAR and Py_XCLEAR are equivalent to: Py_CLEAR_AND_SET(target, NULL); Py_XCLEAR_AND_SET(target, NULL); (Note that existing occurrences of SET in C API macros like PyList_SET_ITEM and PyTuple_SET_ITEM also assume that any required reference count adjustments are handled externally). While the name does suggest the macro will expand to: Py_CLEAR(target); target = value; which isn't quite accurate, it's close enough that people should still be able to understand what the operation does. Explicitly pointing out in that docs that Py_CLEAR(target) is functionally equivalent to Py_CLEAR_AND_SET(target, NULL) should help correct the misapprehension. On the question of updating 3.4+ only vs also updating 3.3, I'm inclined towards fixing it systematically in all currently supported branches, as it's a low risk mechanical change. On the other hand, we still have other crash bugs with known reproducers (in Lib/test/crashers), so I'm also OK with leaving 3.3 alone. (Assuming we can agree on a name, I definitely think it's worth asking Larry for permission to make the late C API addition for 3.4, though) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 16 01:22:04 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 16 Feb 2014 10:22:04 +1000 Subject: [Python-Dev] Using more specific methods in Python unit tests In-Reply-To: References: <1392492250.26338.83831085.39A5ED08@webmail.messagingengine.com> Message-ID: On 16 February 2014 09:20, Ned Deily wrote: > In article > <1392492250.26338.83831085.39A5ED08 at webmail.messagingengine.com>, > Benjamin Peterson wrote: >> On Sat, Feb 15, 2014, at 10:12 AM, Serhiy Storchaka wrote: >> > Although Raymond approved a patch for test_bigmem [2], his expressed the >> > insistent recommendation not to do this. So I stop committing new >> > reviewed patches. Terry recommended to discuss this in Python-Dev. What >> > are your thoughts? >> I tend to agree with Raymond. I think such changes are very welcome when >> the module or tests are otherwise being changed, but on their on >> constitute unnecessary churn. > > +1 > > Integrity of the test suite and minimizing code churn top any benefits > of more specific messages on failures. The expectation is that most > tests will never fail so their changed messages will never be seen. For > the rare cases when a test does fail, quite often the test was written > in a way that will require examination of the code to understand exactly > what the test case was intending to test and why it failed. Having a > more specific exception message wouldn't help for many tests without > further modifications; the key point is to know that the test failed. Right, there are a few key problems with large scale style changes to the test suite: 1. The worst case scenario is where we subtly change a test so that it is no longer testing what it is supposed to be testing, allowing the future introduction of an undetected regression. This isn't particularly *likely*, but a serious problem if it happens. 2. If there are pending patches for that module that include new tests, then the style change may cause the patches to no longer apply cleanly, require rework of bug fix and new feature patches to accommodate the style change. 3. Merges between branches may become more complicated (for reasons similar to 2), unless the style change is also applied to the maintenance branches (which is problematic due to 1). The practical benefits of this kind of change in the test suite are also highly dubious, because they *only help if the test fails at some point in the future*. At that point, whoever caused the test to fail will switch into debugging mode, and a couple of relevant points apply: * the cause of the problem may be immediately obvious anyway, since it's likely in whatever code they just changed * if they need more information, then they can refactor the failing test to use richer (and perhaps additional) assertions as part of the debugging exercise Now, debugging a failing test isn't a skill most programming courses teach, so it may be worth our while to actually provide some pointers on doing so effectively in the devguide (including a note about checking that the failing test is using rich assertions and potentially updating it if it isn't), but the general policy against large scale changes to take advantage of new features still applies to the test suite just as much as it does to the rest of the standard library. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From g.brandl at gmx.net Sun Feb 16 09:19:33 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 16 Feb 2014 09:19:33 +0100 Subject: [Python-Dev] Possible major bug with zipimport on Windows in Python 3.3.4 In-Reply-To: References: <84841BB3-CAE7-44DE-A854-A6428DEC743D@stufft.io> Message-ID: Am 15.02.2014 23:19, schrieb Paul Moore: > On 13 February 2014 20:58, Paul Moore wrote: >> On 13 February 2014 20:54, Donald Stufft wrote: >>> >>> On Feb 13, 2014, at 3:53 PM, Paul Moore wrote: >>> >>>> Can someone please take a look at http://bugs.python.org/issue20621 >>>> for me? It appears that there is a significant problem with zipimport >>>> in 3.3.4. It's causing breakage in virtualenv but I've isolated the >>>> issue to what I think is Python itself. >>>> >>>> Basically, until this bug is fixed, virtualenv won't work on Windows >>>> with Python 3.3.4, as far as I can tell. >>> >>> Does it affect 3.4? >> >> Just checked that. No, it's not an issue in 3.4rc0. Nor is it in >> 3.3.3, so it's a regression in the 3.3 series only. > > OK, this appears to be a serious regression that breaks virtualenv. > Steve Dower suggested a workaround on the tracker, but I don't think > that can be made to work in the case of virtualenv (for reasons not > worth explaining now, ask me if you want the details though). > > Given where we are, what is the way forward here? I'm not entirely > sure what I should be saying to the user who reported the bug in > virtualenv. I think I'll recommend downgrading to 3.3.3 for now, but > it would be good to be able to give an idea of what the longer term > solution is likely to be. As soon as a patch has been provided and tested, I will make a schedule for 3.3.5 including the fix. Until then, using 3.3.3 is probably the best solution. Georg From ezio.melotti at gmail.com Sun Feb 16 09:40:48 2014 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Sun, 16 Feb 2014 10:40:48 +0200 Subject: [Python-Dev] CLA link from bugs.python.org In-Reply-To: References: Message-ID: Hi, On Sun, May 5, 2013 at 7:45 AM, Ezio Melotti wrote: > Hi, > > On Sun, May 5, 2013 at 4:23 AM, Tim Delaney wrote: >> It appears there's no obvious link from bugs.python.org to the contributor >> agreement - you need to go via the unintuitive link Foundation -> >> Contribution Forms (and from what I've read, you're prompted when you add a >> patch to the tracker). >> >> I'd suggest that if the "Contributor Form Received" field is "No" in user >> details, there be a link to http://www.python.org/psf/contrib/. >> > > See http://psf.upfronthosting.co.za/roundup/meta/issue461. > This is now done: users who submit(ted) patches without having signed the contributor agreement will get a note in tracker with the link and a short explanation. (Sorry it took me so long to get this fixed.) Best Regards, Ezio Melotti > Best Regards, > Ezio Melotti > >> Tim Delaney >> From g.brandl at gmx.net Sun Feb 16 10:06:34 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 16 Feb 2014 10:06:34 +0100 Subject: [Python-Dev] CLA link from bugs.python.org In-Reply-To: References: Message-ID: Am 16.02.2014 09:40, schrieb Ezio Melotti: > Hi, > > On Sun, May 5, 2013 at 7:45 AM, Ezio Melotti wrote: >> Hi, >> >> On Sun, May 5, 2013 at 4:23 AM, Tim Delaney wrote: >>> It appears there's no obvious link from bugs.python.org to the contributor >>> agreement - you need to go via the unintuitive link Foundation -> >>> Contribution Forms (and from what I've read, you're prompted when you add a >>> patch to the tracker). >>> >>> I'd suggest that if the "Contributor Form Received" field is "No" in user >>> details, there be a link to http://www.python.org/psf/contrib/. >>> >> >> See http://psf.upfronthosting.co.za/roundup/meta/issue461. >> > > This is now done: users who submit(ted) patches without having signed > the contributor agreement will get a note in tracker with the link and > a short explanation. > (Sorry it took me so long to get this fixed.) Thanks, that is a great improvement. (Although I don't think I like the red background color... ) Georg From ja.py at farowl.co.uk Sun Feb 16 11:23:39 2014 From: ja.py at farowl.co.uk (Jeff Allen) Date: Sun, 16 Feb 2014 10:23:39 +0000 Subject: [Python-Dev] Using more specific methods in Python unit tests In-Reply-To: References: <1392492250.26338.83831085.39A5ED08@webmail.messagingengine.com> Message-ID: <530091AB.2010905@farowl.co.uk> On 16/02/2014 00:22, Nick Coghlan wrote: > On 16 February 2014 09:20, Ned Deily wrote: >> In article >> <1392492250.26338.83831085.39A5ED08 at webmail.messagingengine.com>, >> Benjamin Peterson wrote: >>> On Sat, Feb 15, 2014, at 10:12 AM, Serhiy Storchaka wrote: >>>> Although Raymond approved a patch for test_bigmem [2], his expressed the >>>> insistent recommendation not to do this. So I stop committing new >>>> reviewed patches. Terry recommended to discuss this in Python-Dev. What >>>> are your thoughts? >>> I tend to agree with Raymond. I think such changes are very welcome when >>> the module or tests are otherwise being changed, but on their on >>> constitute unnecessary churn. >>> > Right, there are a few key problems with large scale style changes to > the test suite: > > 1. The worst case scenario is where we subtly change a test so that it > is no longer testing what it is supposed to be testing, allowing the > future introduction of an undetected regression. This isn't > particularly *likely*, but a serious problem if it happens. > > 2. If there are pending patches for that module that include new > tests, then the style change may cause the patches to no longer apply > cleanly, require rework of bug fix and new feature patches to > accommodate the style change. > > 3. Merges between branches may become more complicated (for reasons > similar to 2), unless the style change is also applied to the > maintenance branches (which is problematic due to 1). I spend a *lot* of time working with the Python test suite on behalf of Jython, so I appreciate the care CPython puts into its testing. To a large extent, tests written for CPython drive Jython development: I suspect I work with a lot more failing tests than anyone here. Where we have a custom test, I often update them from in the latest CPython tests. Often a test failure is not caused by code I just wrote, but by adding a CPython test or removing a "skip if Jython", and not having written anything yet. While the irrefutable "False is not true" always raises a smile, I'd welcome something more informative. It's a more than a "style" issue. What Nick says above is also not false, as general guidance, but taken as an iron rule seem to argue against concurrent development at all. Don't we manage this change pretty well already? I see little risk of problems 1-3 in the actual proposal, as the changes themselves are 99% of the "drop-in replacement" type: - self.assertTrue(isinstance(x, int)) + self.assertIsInstance(x, int) I found few places, on a quick scan, that risked changing the meaning: they introduce an if-statement, or refactor the expression -- I don't mean they're actually wrong. The point about breaking Serhiy's patch into independent parts will help manage with merging and this risk. The tests are not library code, but their other use is as an example of good practice in unit testing. I pretty much get my idea of Python's test facilities from this work. It was a while before I realised more expressive methods were available. Jeff Jeff Allen From solipsis at pitrou.net Sun Feb 16 11:39:30 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 16 Feb 2014 11:39:30 +0100 Subject: [Python-Dev] Using more specific methods in Python unit tests References: Message-ID: <20140216113930.71629ad5@fsol> On Sat, 15 Feb 2014 20:12:33 +0200 Serhiy Storchaka wrote: > > I wrote a large patch which modifies the tests to use more specific > methods [1]. Because it is too large, it was divided into many smaller > patches, and separate issues were opened for them. At the moment the > major part of the original patch has already been committed. Many thanks > to Ezio for making a review for the majority of the issues. Some changes > have been made by other people in unrelated issues. > > Although Raymond approved a patch for test_bigmem [2], his expressed the > insistent recommendation not to do this. So I stop committing new > reviewed patches. Terry recommended to discuss this in Python-Dev. What > are your thoughts? When it comes specifically to test_bigmem, it is important for error messages to be informative, because the failures may be hard (if not enough RAM) or very long to diagnose on a developer's machine. So +1 to changing test_bigmem. As for the rest of the test suite, I find the "assertSpecific" form more readable that "assertTrue(... with some operator)". But I may be in a minority here :-) As for the "code churn" argument, I find that a much less important concern for the test suite than for the rest of the code. Regards Antoine. From p.f.moore at gmail.com Sun Feb 16 12:21:22 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 16 Feb 2014 11:21:22 +0000 Subject: [Python-Dev] Possible major bug with zipimport on Windows in Python 3.3.4 In-Reply-To: References: <84841BB3-CAE7-44DE-A854-A6428DEC743D@stufft.io> Message-ID: On 16 February 2014 08:19, Georg Brandl wrote: > As soon as a patch has been provided and tested, I will make a schedule > for 3.3.5 including the fix. Until then, using 3.3.3 is probably the > best solution. Fantastic - thanks for that. Paul From dickinsm at gmail.com Sun Feb 16 12:40:37 2014 From: dickinsm at gmail.com (Mark Dickinson) Date: Sun, 16 Feb 2014 11:40:37 +0000 Subject: [Python-Dev] Using more specific methods in Python unit tests In-Reply-To: References: <1392492250.26338.83831085.39A5ED08@webmail.messagingengine.com> Message-ID: On Sun, Feb 16, 2014 at 12:22 AM, Nick Coghlan wrote: > The practical benefits of this kind of change in the test suite are > also highly dubious, because they *only help if the test fails at some > point in the future*. At that point, whoever caused the test to fail > will switch into debugging mode, and a couple of relevant points > apply: > One place where those points don't apply so cleanly is when the test failure is coming from continuous integration and can't easily be reproduced locally (e.g., because there's a problem on a platform you don't have access to, or because it's some kind of threading-related intermittent failure that's exacerbated by the timing conditions on a particular machine). In those situations, an informative error message can easily save significant debugging time. Count me as +1 on the test updates, provided they're done carefully. (And all those I've looked at from Serhiy do indeed look careful.) -- Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sun Feb 16 13:38:36 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 16 Feb 2014 13:38:36 +0100 Subject: [Python-Dev] Using more specific methods in Python unit tests In-Reply-To: <530091AB.2010905@farowl.co.uk> References: <1392492250.26338.83831085.39A5ED08@webmail.messagingengine.com> <530091AB.2010905@farowl.co.uk> Message-ID: Jeff Allen, 16.02.2014 11:23: > I spend a *lot* of time working with the Python test suite on behalf of > Jython, so I appreciate the care CPython puts into its testing. To a large > extent, tests written for CPython drive Jython development: I suspect I > work with a lot more failing tests than anyone here. Careful with such a bold statement. ;) > What Nick says above is also not false, as general guidance, but taken as > an iron rule seem to argue against concurrent development at all. Don't we > manage this change pretty well already? I see little risk of problems 1-3 > in the actual proposal, as the changes themselves are 99% of the "drop-in > replacement" type: > > - self.assertTrue(isinstance(x, int)) > + self.assertIsInstance(x, int) While I generally second Nick's objections to this, I also agree that the kind of change above is such an obvious and straight forward improvement (since unittest doesn't have py.test's assert analysis) that I'm +1 on applying them. I've been annoyed more than once by a test failure in CPython's test suite (when compiled with Cython) that required me to look up and read the test source and rerun it locally in order to actually understand what was happening. Seeing a more informative error message right on our CI server would certainly help, if only to get a quicker idea if this failure is worth taking a closer look at. Stefan From sky.kok at speaklikeaking.com Sun Feb 16 14:53:10 2014 From: sky.kok at speaklikeaking.com (Vajrasky Kok) Date: Sun, 16 Feb 2014 21:53:10 +0800 Subject: [Python-Dev] The desired behaviour for resolve() when the path doesn't exist In-Reply-To: References: <20140107212841.6ec5acc3@fsol> Message-ID: On Wed, Jan 8, 2014 at 4:45 AM, Serhiy Storchaka wrote: > --canonicalize is not strict. --canonicalize-existing is most strict and > --canonicalize-missing is least strict. When you have a function which have > non-strict behavior (--canonicalize), you can implement a wrapper with > strict behavior (--canonicalize-existing), but not vice verse. > Sorry, only now that I have time to look into this. So what we are going to do before implementing the behaviour for resolve(strict=False) is to change the behaviour of resolve(strict=True) from --canonicalize-existing to --canonicalize? Is there any time left because we are in RC1 already? Should we postpone it to 3.5? But then, we'll have backward compatibility problem. From ezio.melotti at gmail.com Sun Feb 16 15:26:01 2014 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Sun, 16 Feb 2014 16:26:01 +0200 Subject: [Python-Dev] CLA link from bugs.python.org In-Reply-To: References: Message-ID: On Sun, Feb 16, 2014 at 11:06 AM, Georg Brandl wrote: > Am 16.02.2014 09:40, schrieb Ezio Melotti: >> Hi, >> >> On Sun, May 5, 2013 at 7:45 AM, Ezio Melotti wrote: >>> Hi, >>> >>> On Sun, May 5, 2013 at 4:23 AM, Tim Delaney wrote: >>>> It appears there's no obvious link from bugs.python.org to the contributor >>>> agreement - you need to go via the unintuitive link Foundation -> >>>> Contribution Forms (and from what I've read, you're prompted when you add a >>>> patch to the tracker). >>>> >>>> I'd suggest that if the "Contributor Form Received" field is "No" in user >>>> details, there be a link to http://www.python.org/psf/contrib/. >>>> >>> >>> See http://psf.upfronthosting.co.za/roundup/meta/issue461. >>> >> >> This is now done: users who submit(ted) patches without having signed >> the contributor agreement will get a note in tracker with the link and >> a short explanation. >> (Sorry it took me so long to get this fixed.) > > Thanks, that is a great improvement. (Although I don't think I like the > red background color... ) Agreed, that's why I timemachined it gray: http://hg.python.org/tracker/python-dev/rev/4cdbeb1c74c6#l2.11 > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ezio.melotti%40gmail.com From amk at amk.ca Sun Feb 16 15:52:21 2014 From: amk at amk.ca (A.M. Kuchling) Date: Sun, 16 Feb 2014 09:52:21 -0500 Subject: [Python-Dev] pootle.python.org is down Message-ID: <20140216145221.GA93416@datlandrewk.home> I came across http://bugs.python.org/issue13663, which is about a pootle.python.org installation. http://pootle.python.org/ currently returns a 500. Are we still using Pootle, or should I just close #13663? (Maybe the installation got broken in the move to OSL and then forgotten?) --amk From benjamin at python.org Sun Feb 16 16:32:17 2014 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 16 Feb 2014 07:32:17 -0800 Subject: [Python-Dev] pootle.python.org is down In-Reply-To: <20140216145221.GA93416@datlandrewk.home> References: <20140216145221.GA93416@datlandrewk.home> Message-ID: <1392564737.16160.84036429.4D02CEA8@webmail.messagingengine.com> On Sun, Feb 16, 2014, at 06:52 AM, A.M. Kuchling wrote: > I came across http://bugs.python.org/issue13663, which is about a > pootle.python.org installation. http://pootle.python.org/ currently > returns a 500. Are we still using Pootle, or should I just close #13663? > (Maybe the installation got broken in the move to OSL and then > forgotten?) Per the comments in that bug (esp from Martin), I think we should just remove pootle.python.org for good. From g.brandl at gmx.net Sun Feb 16 17:19:26 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 16 Feb 2014 17:19:26 +0100 Subject: [Python-Dev] pootle.python.org is down In-Reply-To: <1392564737.16160.84036429.4D02CEA8@webmail.messagingengine.com> References: <20140216145221.GA93416@datlandrewk.home> <1392564737.16160.84036429.4D02CEA8@webmail.messagingengine.com> Message-ID: Am 16.02.2014 16:32, schrieb Benjamin Peterson: > On Sun, Feb 16, 2014, at 06:52 AM, A.M. Kuchling wrote: >> I came across http://bugs.python.org/issue13663, which is about a >> pootle.python.org installation. http://pootle.python.org/ currently >> returns a 500. Are we still using Pootle, or should I just close #13663? >> (Maybe the installation got broken in the move to OSL and then >> forgotten?) > > Per the comments in that bug (esp from Martin), I think we should just > remove pootle.python.org for good. For now. Georg From larry at hastings.org Sun Feb 16 19:31:21 2014 From: larry at hastings.org (Larry Hastings) Date: Sun, 16 Feb 2014 10:31:21 -0800 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches Message-ID: <530103F9.8070005@hastings.org> Let's begin with a status update of The Great Argument Clinic Conversion Derby. In retrospect, the Derby was way too ambitious. Once it started I was quickly overwhelmed. Even doing nothing but Derby work, all day every day for two straight weeks, I couldn't keep up with all the bug fixes, feature requests, correspondence, and documentation updates it demanded. There was no way I could simultaneously review patches too. As a result: there is, still, an enormous backlog of Derby patches that need reviewing. Few of the Derby patches got integrated before we reached rc1. The underlying theory of the Derby was that it would be a purely mechanical process. It would be a simple matter of converting the existing parsing code into its Argument Clinic equivalent, resulting solely in code churn. And, indeed, a significant portion of the Derby patches are exactly that. But the conversion process peered into a lot of dusty corners, and raised a lot of questions, and as a result it was a much more complicated and time-consuming process than I anticipated. So here we are in the "release candidate" period for 3.4, and we still have all these unmerged Derby patches. And it's simply too late in the release cycle to merge them for 3.4.0. Here's how I propose we move forward. 1) We merge the Derby patch for the builtins module into 3.4, simply because it will "demo well". If someone wants to play with signatures on builtins, I think it's likely they'll try something like "len". Obviously this wouldn't be permitted to change the semantics of argument parsing in any way--this would be a "code churn" patch only. (In case you're wondering, Nick did the conversion of the builtins module, and naturally I will be reviewing it.) 2) We change all Clinic conversions in 3.4 so they write the generated code to a separate file--in Clinic parlance, change them so they 'use the "file" destination'. Going forward this will be the preferred way to check in Argument Clinic changes into Python. These first two are the only changes resulting from the Derby that I will accept between now and 3.4.0 final, and I expect to have them in for 3.4.0rc2. Continuing from there: 3) We hold off on merging the rest of the Derby patches until after 3.4.0 final ships, then we merge them into the 3.4 maintenance branch so they go into 3.4.1. We use the time between now and then to get the patches totally, totally perfect. Again, these patches will not be permitted to change the parsing semantics of the functions so converted. I expect to do these checkins in a private branch, and land the bulk of it immediately upon the opening of the 3.4 maintenance branch. 4) We accelerate the schedule for 3.4.1 slightly, so we can get these new signatures into the hands of users sooner. Specifically, I propose we ship 3.4.1 two months after 3.4.0. I figure we would release 3.4.1 rc1 on Sunday May 4th, and 3.4.1 final on Sunday May 18th. 5) Any proposed changes in Derby patches that change the semantics of a builtin may only be checked into default for 3.5, after 3.4.0 ships. I'm very sorry that many people contributed to the Derby expecting their patches to go in to 3.4. This is my fault, for severely miscalculating how the Derby would play out. And I feel awful about it. But I'm convinced the best thing for Python is to hold off on merging until after 3.4.0 ships. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sun Feb 16 20:52:04 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 16 Feb 2014 21:52:04 +0200 Subject: [Python-Dev] Possible major bug with zipimport on Windows in Python 3.3.4 In-Reply-To: References: <84841BB3-CAE7-44DE-A854-A6428DEC743D@stufft.io> Message-ID: 16.02.14 10:19, Georg Brandl ???????(??): > As soon as a patch has been provided and tested, I will make a schedule > for 3.3.5 including the fix. Until then, using 3.3.3 is probably the > best solution. Then could you please include the fix for #20538 (this bug allows easily crash Python e.g. by special HTTP request or e-mail). And I think it is very important to port the fix for #19619. Simple patch for #17671 fixes other crash (less important, because can be triggered only by programming error). From storchaka at gmail.com Sun Feb 16 21:10:40 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 16 Feb 2014 22:10:40 +0200 Subject: [Python-Dev] Add Py_REPLACE and Py_XREPLACE macros In-Reply-To: References: Message-ID: 16.02.14 02:05, Nick Coghlan ???????(??): > It's also likely than many of these crashes could only be reproduced > through incorrect usage of the C API. Rather through queer or malicious usage of Python classes with strange code in __del__. > For example: > > Py_CLEAR_AND_SET > Py_XCLEAR_AND_SET > > Such that Py_CLEAR and Py_XCLEAR are equivalent to: There is no Py_XCLEAR. Py_CLEAR itself checks its argument for NULL (as Py_XDECREF and Py_XINCREF). And these names looks too cumbersome to me. > While the name does suggest the macro will expand to: > > Py_CLEAR(target); > target = value; > > which isn't quite accurate, it's close enough that people should still > be able to understand what the operation does. This is not just inaccurate, this is wrong. Py_REPLACE first assigns new value and then DECREF old value. So it rather can be named as Py_SET_AND_DECREF, but of course this name looks ugly and confusing. From tjreedy at udel.edu Sun Feb 16 21:14:58 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 16 Feb 2014 15:14:58 -0500 Subject: [Python-Dev] Possible major bug with zipimport on Windows in Python 3.3.4 In-Reply-To: References: <84841BB3-CAE7-44DE-A854-A6428DEC743D@stufft.io> Message-ID: On 2/16/2014 2:52 PM, Serhiy Storchaka wrote: > 16.02.14 10:19, Georg Brandl ???????(??): >> As soon as a patch has been provided and tested, I will make a schedule >> for 3.3.5 including the fix. Until then, using 3.3.3 is probably the >> best solution. > > Then could you please include the fix for #20538 (this bug allows easily > crash Python e.g. by special HTTP request or e-mail). And I think it is > very important to port the fix for #19619. Simple patch for #17671 fixes > other crash (less important, because can be triggered only by > programming error). A 3.3.5 should have all the bugfixes committed before the candidate release. -- Terry Jan Reedy From storchaka at gmail.com Sun Feb 16 21:36:58 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 16 Feb 2014 22:36:58 +0200 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: <530103F9.8070005@hastings.org> References: <530103F9.8070005@hastings.org> Message-ID: 16.02.14 20:31, Larry Hastings ???????(??): > So here we are in the "release candidate" period for 3.4, and we still > have all these unmerged Derby patches. And it's simply too late in the > release cycle to merge them for 3.4.0. > > Here's how I propose we move forward. Your plan LGTM (except one question below). > 4) We accelerate the schedule for 3.4.1 slightly, so we can get these > new signatures into the hands of users sooner. Specifically, I propose > we ship 3.4.1 two months after 3.4.0. I figure we would release 3.4.1 > rc1 on Sunday May 4th, and 3.4.1 final on Sunday May 18th. How this will affect the schedule for 3.3.x? Will it accelerately switched to security-only fix mode? From larry at hastings.org Sun Feb 16 22:27:54 2014 From: larry at hastings.org (Larry Hastings) Date: Sun, 16 Feb 2014 13:27:54 -0800 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: References: <530103F9.8070005@hastings.org> Message-ID: <53012D5A.5070606@hastings.org> On 02/16/2014 12:36 PM, Serhiy Storchaka wrote: > 16.02.14 20:31, Larry Hastings ???????(??): >> 4) We accelerate the schedule for 3.4.1 slightly, so we can get these >> new signatures into the hands of users sooner. Specifically, I propose >> we ship 3.4.1 two months after 3.4.0. I figure we would release 3.4.1 >> rc1 on Sunday May 4th, and 3.4.1 final on Sunday May 18th. > > How this will affect the schedule for 3.3.x? Will it accelerately > switched to security-only fix mode? That would really be a question for Georg Brandl, the 3.3 release manager. However I don't know why 3.4.1 would have any effect on 3.3, regardless of when it was released. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Feb 16 22:51:34 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 16 Feb 2014 22:51:34 +0100 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches References: <530103F9.8070005@hastings.org> Message-ID: <20140216225134.3b72cec2@fsol> On Sun, 16 Feb 2014 10:31:21 -0800 Larry Hastings wrote: > > Here's how I propose we move forward. > > 1) We merge the Derby patch for the builtins module into 3.4, simply > because it will "demo well". This still brings potential unstability during the rc phase, so I'd prefer this patch to wait for 3.4.1. Quoting Wikipedia: "A release candidate (RC) is a beta version with potential to be a final product, which is ready to release unless significant bugs emerge." Regards Antoine. From g.brandl at gmx.net Sun Feb 16 23:02:55 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 16 Feb 2014 23:02:55 +0100 Subject: [Python-Dev] Possible major bug with zipimport on Windows in Python 3.3.4 In-Reply-To: References: <84841BB3-CAE7-44DE-A854-A6428DEC743D@stufft.io> Message-ID: Am 16.02.2014 21:14, schrieb Terry Reedy: > On 2/16/2014 2:52 PM, Serhiy Storchaka wrote: >> 16.02.14 10:19, Georg Brandl ???????(??): >>> As soon as a patch has been provided and tested, I will make a schedule >>> for 3.3.5 including the fix. Until then, using 3.3.3 is probably the >>> best solution. >> >> Then could you please include the fix for #20538 (this bug allows easily >> crash Python e.g. by special HTTP request or e-mail). And I think it is >> very important to port the fix for #19619. Simple patch for #17671 fixes >> other crash (less important, because can be triggered only by >> programming error). > > A 3.3.5 should have all the bugfixes committed before the candidate release. Exactly. If these are all committed by the rc (only #19619 is missing AFAICT), they will go in. The 3.3.5 rc will be released next weekend, Feb 22th, and the final (if nothing else comes up) on Mar 1st. cheers, Georg From g.brandl at gmx.net Sun Feb 16 23:08:10 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 16 Feb 2014 23:08:10 +0100 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: <53012D5A.5070606@hastings.org> References: <530103F9.8070005@hastings.org> <53012D5A.5070606@hastings.org> Message-ID: Am 16.02.2014 22:27, schrieb Larry Hastings: > On 02/16/2014 12:36 PM, Serhiy Storchaka wrote: >> 16.02.14 20:31, Larry Hastings ???????(??): >>> 4) We accelerate the schedule for 3.4.1 slightly, so we can get these >>> new signatures into the hands of users sooner. Specifically, I propose >>> we ship 3.4.1 two months after 3.4.0. I figure we would release 3.4.1 >>> rc1 on Sunday May 4th, and 3.4.1 final on Sunday May 18th. >> >> How this will affect the schedule for 3.3.x? Will it accelerately switched to >> security-only fix mode? > > That would really be a question for Georg Brandl, the 3.3 release manager. > However I don't know why 3.4.1 would have any effect on 3.3, regardless of when > it was released. There's no effect; the final 3.3.x point release (hopefully 3.3.6) will be released after 3.4.0 final, with the branch going into security maintenance mode afterwards. Currently I'd be fine with doing 3.3.6 in parallel with 3.4.1, but that is not a requirement and I don't want to finalize that schedule yet. cheers, Georg From greg.ewing at canterbury.ac.nz Sun Feb 16 23:34:17 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 17 Feb 2014 11:34:17 +1300 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: <530103F9.8070005@hastings.org> References: <530103F9.8070005@hastings.org> Message-ID: <53013CE9.9070207@canterbury.ac.nz> Larry Hastings wrote: > 3) We hold off on merging the rest of the Derby patches until after > 3.4.0 final ships, then we merge them into the 3.4 maintenance branch so > they go into 3.4.1. But wouldn't that be introducing a new feature into a maintenance release? (I.e. some functions that didn't have introspectable signatures before would gain them.) -- Greg From nad at acm.org Sun Feb 16 23:46:20 2014 From: nad at acm.org (Ned Deily) Date: Sun, 16 Feb 2014 14:46:20 -0800 Subject: [Python-Dev] cpython: Doc: do not rely on checked-out Sphinx toolchain from svn.python.org anymore References: <3fRhkW2rDjz7LmY@mail.python.org> Message-ID: In article <3fRhkW2rDjz7LmY at mail.python.org>, georg.brandl wrote: > http://hg.python.org/cpython/rev/eef7899ea7ab > changeset: 89212:eef7899ea7ab > user: Georg Brandl > date: Sun Feb 16 09:46:36 2014 +0100 > summary: > Doc: do not rely on checked-out Sphinx toolchain from svn.python.org > anymore > > Nowadays it is likely that people will have Sphinx installed, > and if not, they will know how to install it. > > This also simplifies life a lot for distribution packagers, > who typically do not want the doc build process to connect > to external web resources. This checkin breaks the OS X installer builds, see for instance: http://buildbot.python.org/all/builders/bolen-dmg-3.x/builds/513 I've opened Issue20644 to address updating the OS X installer build process and buildbot to deal with this change. I imagine it also affects the Windows installer build. This change should not be cherry-picked into 3.4.x until these issues are addressed. -- Ned Deily, nad at acm.org From g.brandl at gmx.net Mon Feb 17 00:10:29 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 17 Feb 2014 00:10:29 +0100 Subject: [Python-Dev] cpython: Doc: do not rely on checked-out Sphinx toolchain from svn.python.org anymore In-Reply-To: References: <3fRhkW2rDjz7LmY@mail.python.org> Message-ID: Am 16.02.2014 23:46, schrieb Ned Deily: > In article <3fRhkW2rDjz7LmY at mail.python.org>, > georg.brandl wrote: > >> http://hg.python.org/cpython/rev/eef7899ea7ab >> changeset: 89212:eef7899ea7ab >> user: Georg Brandl >> date: Sun Feb 16 09:46:36 2014 +0100 >> summary: >> Doc: do not rely on checked-out Sphinx toolchain from svn.python.org >> anymore >> >> Nowadays it is likely that people will have Sphinx installed, >> and if not, they will know how to install it. >> >> This also simplifies life a lot for distribution packagers, >> who typically do not want the doc build process to connect >> to external web resources. > > This checkin breaks the OS X installer builds, see for instance: > > http://buildbot.python.org/all/builders/bolen-dmg-3.x/builds/513 > > I've opened Issue20644 to address updating the OS X installer build > process and buildbot to deal with this change. I imagine it also > affects the Windows installer build. This change should not be > cherry-picked into 3.4.x until these issues are addressed. Thanks, I hope it will be easy to fix. Georg From ncoghlan at gmail.com Mon Feb 17 00:13:09 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 17 Feb 2014 09:13:09 +1000 Subject: [Python-Dev] pootle.python.org is down In-Reply-To: References: <20140216145221.GA93416@datlandrewk.home> <1392564737.16160.84036429.4D02CEA8@webmail.messagingengine.com> Message-ID: On 17 Feb 2014 02:20, "Georg Brandl" wrote: > > Am 16.02.2014 16:32, schrieb Benjamin Peterson: > > On Sun, Feb 16, 2014, at 06:52 AM, A.M. Kuchling wrote: > >> I came across http://bugs.python.org/issue13663, which is about a > >> pootle.python.org installation. http://pootle.python.org/ currently > >> returns a 500. Are we still using Pootle, or should I just close #13663? > >> (Maybe the installation got broken in the move to OSL and then > >> forgotten?) > > > > Per the comments in that bug (esp from Martin), I think we should just > > remove pootle.python.org for good. > > For now. We should ideally figure out another way to provide support for docs translations, though. I already have a slot at the language summit to talk about how we manage docs development in general, so if anyone has info on the current status of docs translation efforts, I'd be happy to bring that up as well. Cheers, Nick. > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Mon Feb 17 00:25:52 2014 From: larry at hastings.org (Larry Hastings) Date: Sun, 16 Feb 2014 15:25:52 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final Message-ID: <53014900.7090908@hastings.org> Right now we're in the "release candidate" phase of 3.4. 3.4.0 rc1 has been released, and the next release will be rc2. You might think that anything you check in to the "default" branch in Python trunk will go into 3.4.0 rc2, and after that ships, checkins would go into 3.4.0 final. Ho ho ho! That's not true! Instead, anything checked in to "default" between my last revision for "rc1" (e64ae8b82672) and 3.4.0 final will by default go into 3.4.1. Only fixes that I cherry-pick into my local branch will go into 3.4.0 rc2 and final. And my local branch will remain private until 3.4.0 final ships! If you have a Terribly Important Fix That Must Go Into 3.4.0 rc2 or final, please go to the issue tracker and create a new issue with the following attributes: The title should start with "3.4 cherry-pick: " followed by the revision id and a short summary example: "3.4 cherry-pick: b328f8ccbccf __getnewargs__ fix" The version should be "Python 3.4" The assignee should be "larry" The priority should be "release blocker" The comment should *also* contain the revision id (the tracker will turn it into a link) I'm also working on automatically publishing the merged/unmerged revision status to a web page. You can see a mock-up here: http://www.midwinter.com/~larry/3.4.merge.status.html The page is marked "beta" because it doesn't have real data yet--I'm still experimenting with my automation, so I haven't created the real 3.4 local branch yet. Again, just to be crystal-clear: the revisions marked "merged" on that page are just experiments, they aren't actually merged for 3.4. Once I'm ready for real merging, I'll remove the beta warning. (By the way: on that page, clicking on a revision takes you to the revision web page. Clicking on the first line of the comment expands it to show the complete comment.) Please use your best judgment before asking that a revision be cherry-picked into 3.4.0. Our goal in the release candidate phase is to stabilize Python, and to do that we must stop changing it. Only important interface changes, new features, or bugfixes should be checked in now, and preferably they should be low-risk. Cheers, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 17 00:27:40 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 17 Feb 2014 09:27:40 +1000 Subject: [Python-Dev] Add Py_REPLACE and Py_XREPLACE macros In-Reply-To: References: Message-ID: On 17 Feb 2014 06:12, "Serhiy Storchaka" wrote: > > 16.02.14 02:05, Nick Coghlan ???????(??): > >> It's also likely than many of these crashes could only be reproduced >> through incorrect usage of the C API. > > > Rather through queer or malicious usage of Python classes with strange code in __del__. This change doesn't fix any of the known crashers in Lib/test/crashers, though - I applied the patch locally and checked. >> For example: >> >> Py_CLEAR_AND_SET >> Py_XCLEAR_AND_SET >> >> Such that Py_CLEAR and Py_XCLEAR are equivalent to: > > > There is no Py_XCLEAR. Py_CLEAR itself checks its argument for NULL (as Py_XDECREF and Py_XINCREF). And these names looks too cumbersome to me. > > >> While the name does suggest the macro will expand to: >> >> Py_CLEAR(target); >> target = value; >> >> which isn't quite accurate, it's close enough that people should still >> be able to understand what the operation does. > > > This is not just inaccurate, this is wrong. Py_REPLACE first assigns new value and then DECREF old value. The point is that people already know what Py_CLEAR does. This operation is like Py_CLEAR (the old reference is only removed *after* the pointer has been updated), except that the value it is being replaced with can be something other than NULL. If the replacement value *is* NULL, then the new operation is *exactly* equivalent to Py_CLEAR. Operations that do related things should ideally have related names. The point of my deliberately erroneous expansion is that it's an error a reader can make while still correctly understanding the *logic* of the code, even though they're missing a subtlety of the mechanics. > So it rather can be named as Py_SET_AND_DECREF, but of course this name looks ugly and confusing. An explicit name like Py_SET_AND_DECREF would also be reasonable. It's substantially less confusing than Py_REPLACE, as it is less ambiguous about whether or not the refcount on the new value is adjusted. Cheers, Nick. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 17 00:43:13 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 17 Feb 2014 09:43:13 +1000 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: <53013CE9.9070207@canterbury.ac.nz> References: <530103F9.8070005@hastings.org> <53013CE9.9070207@canterbury.ac.nz> Message-ID: On 17 Feb 2014 08:36, "Greg Ewing" wrote: > > Larry Hastings wrote: > >> 3) We hold off on merging the rest of the Derby patches until after 3.4.0 final ships, then we merge them into the 3.4 maintenance branch so they go into 3.4.1. > > > But wouldn't that be introducing a new feature into a > maintenance release? (I.e. some functions that didn't > have introspectable signatures before would gain them.) >From a compatibility point of view, 3.4.0 will already force introspection users and tool developers to cope with the fact that some, but not all, builtin and extension types provide valid signature data. Additional clinic conversions that don't alter semantics then just move additional callables into the "supports programmatic introspection" category. It's certainly in a grey area, but "What's in the best interest of end users?" pushes me in the direction of counting clinic conversions that don't change semantics as bug fixes - they get improved introspection support sooner, and it shouldn't make life any harder for tool developers because all of the adjustments for 3.4 will be to the associated functional changes in the inspect module. The key thing is to make sure to postpone any changes that impact *semantics* (like adding keyword argument support). Larry's plan sounds OK to me, although at this point I'd be happier leaving even the builtin conversion until 3.4.1 (especially if we decide to publish that not long after PyCon US). Cheers, Nick. > > -- > Greg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Feb 17 00:45:15 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 16 Feb 2014 23:45:15 +0000 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <53014900.7090908@hastings.org> References: <53014900.7090908@hastings.org> Message-ID: is significant enough to be resulting in a 3.3.5 release - can you make sure the 3.4 fix goes in? I'm not sure how to find the revision number that contains the fix to follow the process you outline above, so I'm just mentioning it here & on the issue to make sure it's not missed... Paul On 16 February 2014 23:25, Larry Hastings wrote: > > > Right now we're in the "release candidate" phase of 3.4. 3.4.0 rc1 has been > released, and the next release will be rc2. > > You might think that anything you check in to the "default" branch in Python > trunk will go into 3.4.0 rc2, and after that ships, checkins would go into > 3.4.0 final. Ho ho ho! That's not true! Instead, anything checked in to > "default" between my last revision for "rc1" (e64ae8b82672) and 3.4.0 final > will by default go into 3.4.1. Only fixes that I cherry-pick into my local > branch will go into 3.4.0 rc2 and final. And my local branch will remain > private until 3.4.0 final ships! > > If you have a Terribly Important Fix That Must Go Into 3.4.0 rc2 or final, > please go to the issue tracker and create a new issue with the following > attributes: > > The title should start with "3.4 cherry-pick: " followed by the revision id > and a short summary > example: "3.4 cherry-pick: b328f8ccbccf __getnewargs__ fix" > The version should be "Python 3.4" > The assignee should be "larry" > The priority should be "release blocker" > The comment should *also* contain the revision id (the tracker will turn it > into a link) > > I'm also working on automatically publishing the merged/unmerged revision > status to a web page. You can see a mock-up here: > > http://www.midwinter.com/~larry/3.4.merge.status.html > > The page is marked "beta" because it doesn't have real data yet--I'm still > experimenting with my automation, so I haven't created the real 3.4 local > branch yet. Again, just to be crystal-clear: the revisions marked "merged" > on that page are just experiments, they aren't actually merged for 3.4. > Once I'm ready for real merging, I'll remove the beta warning. > > (By the way: on that page, clicking on a revision takes you to the revision > web page. Clicking on the first line of the comment expands it to show the > complete comment.) > > > Please use your best judgment before asking that a revision be cherry-picked > into 3.4.0. Our goal in the release candidate phase is to stabilize Python, > and to do that we must stop changing it. Only important interface changes, > new features, or bugfixes should be checked in now, and preferably they > should be low-risk. > > Cheers, > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/p.f.moore%40gmail.com > From victor.stinner at gmail.com Mon Feb 17 01:03:07 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 17 Feb 2014 01:03:07 +0100 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: <530103F9.8070005@hastings.org> References: <530103F9.8070005@hastings.org> Message-ID: Hi, The PEP 436 is still a draft and not mentionned in Python 3.4 changelog. The PEP proposes to add a DSL, not to modify all modules implemented in C. I think that it should be marked as Final and mentionned in the changelog. http://www.python.org/dev/peps/pep-0436/ 2014-02-16 19:31 GMT+01:00 Larry Hastings : > Here's how I propose we move forward. > > 1) We merge the Derby patch for the builtins module into 3.4, simply because > it will "demo well". Where is the issue to implement this feature? Anyway, I expect a huge patch which is non-trivial and so very risky for such a very important module :-/ It's too late IMO. Victor From ethan at stoneleaf.us Mon Feb 17 00:51:54 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 16 Feb 2014 15:51:54 -0800 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: <530103F9.8070005@hastings.org> References: <530103F9.8070005@hastings.org> Message-ID: <53014F1A.5070007@stoneleaf.us> On 02/16/2014 10:31 AM, Larry Hastings wrote: > > I'm very sorry that many people contributed to the Derby expecting > their patches to go in to 3.4. This is my fault, for severely > miscalculating how the Derby would play out. And I feel awful > about it. Don't worry too much. It was a great effort, and much got done that will be integrated soon. > But I'm convinced the best thing for Python is to hold off on > merging until after 3.4.0 ships. If 3.4.1 will be released so soon, just hold off on all the Derby changes until then. -- ~Ethan~ From songofacandy at gmail.com Mon Feb 17 02:33:13 2014 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 17 Feb 2014 10:33:13 +0900 Subject: [Python-Dev] pootle.python.org is down In-Reply-To: References: <20140216145221.GA93416@datlandrewk.home> <1392564737.16160.84036429.4D02CEA8@webmail.messagingengine.com> Message-ID: FYI, Japanese translation project is now uses Transifex to translate Py3k document. https://www.transifex.com/projects/p/python-33-ja/ http://docs.python.jp/3/ On Mon, Feb 17, 2014 at 8:13 AM, Nick Coghlan wrote: > > On 17 Feb 2014 02:20, "Georg Brandl" wrote: >> >> Am 16.02.2014 16:32, schrieb Benjamin Peterson: >> > On Sun, Feb 16, 2014, at 06:52 AM, A.M. Kuchling wrote: >> >> I came across http://bugs.python.org/issue13663, which is about a >> >> pootle.python.org installation. http://pootle.python.org/ currently >> >> returns a 500. Are we still using Pootle, or should I just close >> >> #13663? >> >> (Maybe the installation got broken in the move to OSL and then >> >> forgotten?) >> > >> > Per the comments in that bug (esp from Martin), I think we should just >> > remove pootle.python.org for good. >> >> For now. > > We should ideally figure out another way to provide support for docs > translations, though. I already have a slot at the language summit to talk > about how we manage docs development in general, so if anyone has info on > the current status of docs translation efforts, I'd be happy to bring that > up as well. > > Cheers, > Nick. > >> >> Georg >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com > -- INADA Naoki From greg at krypto.org Mon Feb 17 04:09:50 2014 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 16 Feb 2014 19:09:50 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> Message-ID: For 3.4.0rc2 the commit to merge from issue20621 is 52ab9e1ff46a. On Sun, Feb 16, 2014 at 3:45 PM, Paul Moore wrote: > is significant enough to be > resulting in a 3.3.5 release - can you make sure the 3.4 fix goes in? > I'm not sure how to find the revision number that contains the fix to > follow the process you outline above, so I'm just mentioning it here & > on the issue to make sure it's not missed... > > Paul > > On 16 February 2014 23:25, Larry Hastings wrote: > > > > > > Right now we're in the "release candidate" phase of 3.4. 3.4.0 rc1 has > been > > released, and the next release will be rc2. > > > > You might think that anything you check in to the "default" branch in > Python > > trunk will go into 3.4.0 rc2, and after that ships, checkins would go > into > > 3.4.0 final. Ho ho ho! That's not true! Instead, anything checked in > to > > "default" between my last revision for "rc1" (e64ae8b82672) and 3.4.0 > final > > will by default go into 3.4.1. Only fixes that I cherry-pick into my > local > > branch will go into 3.4.0 rc2 and final. And my local branch will remain > > private until 3.4.0 final ships! > > > > If you have a Terribly Important Fix That Must Go Into 3.4.0 rc2 or > final, > > please go to the issue tracker and create a new issue with the following > > attributes: > > > > The title should start with "3.4 cherry-pick: " followed by the revision > id > > and a short summary > > example: "3.4 cherry-pick: b328f8ccbccf __getnewargs__ fix" > > The version should be "Python 3.4" > > The assignee should be "larry" > > The priority should be "release blocker" > > The comment should *also* contain the revision id (the tracker will turn > it > > into a link) > > > > I'm also working on automatically publishing the merged/unmerged revision > > status to a web page. You can see a mock-up here: > > > > http://www.midwinter.com/~larry/3.4.merge.status.html > > > > The page is marked "beta" because it doesn't have real data yet--I'm > still > > experimenting with my automation, so I haven't created the real 3.4 local > > branch yet. Again, just to be crystal-clear: the revisions marked > "merged" > > on that page are just experiments, they aren't actually merged for 3.4. > > Once I'm ready for real merging, I'll remove the beta warning. > > > > (By the way: on that page, clicking on a revision takes you to the > revision > > web page. Clicking on the first line of the comment expands it to show > the > > complete comment.) > > > > > > Please use your best judgment before asking that a revision be > cherry-picked > > into 3.4.0. Our goal in the release candidate phase is to stabilize > Python, > > and to do that we must stop changing it. Only important interface > changes, > > new features, or bugfixes should be checked in now, and preferably they > > should be low-risk. > > > > Cheers, > > > > > > /arry > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/p.f.moore%40gmail.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Mon Feb 17 04:13:46 2014 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 16 Feb 2014 19:13:46 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> Message-ID: http://bugs.python.org/issue20651 filed to track this as larry requested. On Sun, Feb 16, 2014 at 7:09 PM, Gregory P. Smith wrote: > For 3.4.0rc2 the commit to merge from issue20621 is 52ab9e1ff46a. > > > On Sun, Feb 16, 2014 at 3:45 PM, Paul Moore wrote: > >> is significant enough to be >> resulting in a 3.3.5 release - can you make sure the 3.4 fix goes in? >> I'm not sure how to find the revision number that contains the fix to >> follow the process you outline above, so I'm just mentioning it here & >> on the issue to make sure it's not missed... >> >> Paul >> >> On 16 February 2014 23:25, Larry Hastings wrote: >> > >> > >> > Right now we're in the "release candidate" phase of 3.4. 3.4.0 rc1 has >> been >> > released, and the next release will be rc2. >> > >> > You might think that anything you check in to the "default" branch in >> Python >> > trunk will go into 3.4.0 rc2, and after that ships, checkins would go >> into >> > 3.4.0 final. Ho ho ho! That's not true! Instead, anything checked in >> to >> > "default" between my last revision for "rc1" (e64ae8b82672) and 3.4.0 >> final >> > will by default go into 3.4.1. Only fixes that I cherry-pick into my >> local >> > branch will go into 3.4.0 rc2 and final. And my local branch will >> remain >> > private until 3.4.0 final ships! >> > >> > If you have a Terribly Important Fix That Must Go Into 3.4.0 rc2 or >> final, >> > please go to the issue tracker and create a new issue with the following >> > attributes: >> > >> > The title should start with "3.4 cherry-pick: " followed by the >> revision id >> > and a short summary >> > example: "3.4 cherry-pick: b328f8ccbccf __getnewargs__ fix" >> > The version should be "Python 3.4" >> > The assignee should be "larry" >> > The priority should be "release blocker" >> > The comment should *also* contain the revision id (the tracker will >> turn it >> > into a link) >> > >> > I'm also working on automatically publishing the merged/unmerged >> revision >> > status to a web page. You can see a mock-up here: >> > >> > http://www.midwinter.com/~larry/3.4.merge.status.html >> > >> > The page is marked "beta" because it doesn't have real data yet--I'm >> still >> > experimenting with my automation, so I haven't created the real 3.4 >> local >> > branch yet. Again, just to be crystal-clear: the revisions marked >> "merged" >> > on that page are just experiments, they aren't actually merged for 3.4. >> > Once I'm ready for real merging, I'll remove the beta warning. >> > >> > (By the way: on that page, clicking on a revision takes you to the >> revision >> > web page. Clicking on the first line of the comment expands it to show >> the >> > complete comment.) >> > >> > >> > Please use your best judgment before asking that a revision be >> cherry-picked >> > into 3.4.0. Our goal in the release candidate phase is to stabilize >> Python, >> > and to do that we must stop changing it. Only important interface >> changes, >> > new features, or bugfixes should be checked in now, and preferably they >> > should be low-risk. >> > >> > Cheers, >> > >> > >> > /arry >> > >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > >> https://mail.python.org/mailman/options/python-dev/p.f.moore%40gmail.com >> > >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Feb 17 12:14:05 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 17 Feb 2014 12:14:05 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <5301EEFD.9020502@egenix.com> On 15.02.2014 07:03, Stephen J. Turnbull wrote: > M.-A. Lemburg writes: > > > IMO, it was a mistake to have None return a TypeError in > > comparisons, since it makes many typical data operations > > fail, e.g. > > I don't understand this statement. The theory is that they *should* > fail. > > The example of sort is a good one. Sometimes you want missing values > to be collected at the beginning of a list, sometimes at the end. > Sometimes you want them treated as top elements, sometimes as bottom. > And sometimes it is a real error for missing values to be present. > Not to mention that sometimes the programmer simply hasn't thought > about the appropriate policy. I don't think Python should silently > impose a policy in that case, especially given that the programmer may > have experience with any of the above treatments in other contexts. None is special in Python and has always (and intentionally) sorted before any other object. In data processing and elsewhere in Python programming, it's used to signal: no value available. Python 3 breaks this notion by always raising an exception when using None in an ordered comparison, making it pretty much useless for the above purpose. Yes, there are ways around this, but none of them are intuitive. Here's a particularly nasty case: >>> l = [(1, None), (2, None)] >>> l.sort() >>> l [(1, None), (2, None)] >>> l = [(1, None), (2, None), (3, 4)] >>> l.sort() >>> l [(1, None), (2, None), (3, 4)] >>> l = [(1, None), (2, None), (3, 4), (2, 3)] >>> l.sort() Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: int() < NoneType() -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 17 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From gjcarneiro at gmail.com Mon Feb 17 12:23:37 2014 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Mon, 17 Feb 2014 11:23:37 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5301EEFD.9020502@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> Message-ID: On 17 February 2014 11:14, M.-A. Lemburg wrote: > On 15.02.2014 07:03, Stephen J. Turnbull wrote: > > M.-A. Lemburg writes: > > > > > IMO, it was a mistake to have None return a TypeError in > > > comparisons, since it makes many typical data operations > > > fail, e.g. > > > > I don't understand this statement. The theory is that they *should* > > fail. > > > > The example of sort is a good one. Sometimes you want missing values > > to be collected at the beginning of a list, sometimes at the end. > > Sometimes you want them treated as top elements, sometimes as bottom. > > And sometimes it is a real error for missing values to be present. > > Not to mention that sometimes the programmer simply hasn't thought > > about the appropriate policy. I don't think Python should silently > > impose a policy in that case, especially given that the programmer may > > have experience with any of the above treatments in other contexts. > > None is special in Python and has always (and intentionally) sorted > before any other object. In data processing and elsewhere in Python > programming, it's used to signal: no value available. > > Python 3 breaks this notion by always raising an exception when > using None in an ordered comparison, making it pretty much useless > for the above purpose. > > Yes, there are ways around this, but none of them are intuitive. > > Here's a particularly nasty case: > > >>> l = [(1, None), (2, None)] > >>> l.sort() > >>> l > [(1, None), (2, None)] > > >>> l = [(1, None), (2, None), (3, 4)] > >>> l.sort() > >>> l > [(1, None), (2, None), (3, 4)] > > >>> l = [(1, None), (2, None), (3, 4), (2, 3)] > >>> l.sort() > Traceback (most recent call last): > File "", line 1, in > TypeError: unorderable types: int() < NoneType() > > Maybe Python 3 should have a couple of None-like objects that compare the way you want: AlwaysComparesLess and AlwaysComparesGreater, but with better names (maybe 'PlusInfinity' and 'MinusInfinity'?). Just leave None alone, please. > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 17 2014) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com > -- Gustavo J. A. M. Carneiro Gambit Research LLC "The universe is always one step beyond logic." -- Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Feb 17 12:43:25 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 17 Feb 2014 12:43:25 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> Message-ID: <5301F5DD.9050101@egenix.com> On 17.02.2014 12:23, Gustavo Carneiro wrote: > On 17 February 2014 11:14, M.-A. Lemburg wrote: > >> On 15.02.2014 07:03, Stephen J. Turnbull wrote: >>> M.-A. Lemburg writes: >>> >>> > IMO, it was a mistake to have None return a TypeError in >>> > comparisons, since it makes many typical data operations >>> > fail, e.g. >>> >>> I don't understand this statement. The theory is that they *should* >>> fail. >>> >>> The example of sort is a good one. Sometimes you want missing values >>> to be collected at the beginning of a list, sometimes at the end. >>> Sometimes you want them treated as top elements, sometimes as bottom. >>> And sometimes it is a real error for missing values to be present. >>> Not to mention that sometimes the programmer simply hasn't thought >>> about the appropriate policy. I don't think Python should silently >>> impose a policy in that case, especially given that the programmer may >>> have experience with any of the above treatments in other contexts. >> >> None is special in Python and has always (and intentionally) sorted >> before any other object. In data processing and elsewhere in Python >> programming, it's used to signal: no value available. >> >> Python 3 breaks this notion by always raising an exception when >> using None in an ordered comparison, making it pretty much useless >> for the above purpose. >> >> Yes, there are ways around this, but none of them are intuitive. >> >> Here's a particularly nasty case: >> >>>>> l = [(1, None), (2, None)] >>>>> l.sort() >>>>> l >> [(1, None), (2, None)] >> >>>>> l = [(1, None), (2, None), (3, 4)] >>>>> l.sort() >>>>> l >> [(1, None), (2, None), (3, 4)] >> >>>>> l = [(1, None), (2, None), (3, 4), (2, 3)] >>>>> l.sort() >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: unorderable types: int() < NoneType() >> >> > Maybe Python 3 should have a couple of None-like objects that compare the > way you want: AlwaysComparesLess and AlwaysComparesGreater, but with better > names (maybe 'PlusInfinity' and 'MinusInfinity'?). Just leave None alone, > please. This doesn't only apply to numeric comparisons. In Python 2 you can compare None with any kind of object and it always sorts first, based on the intuition that nothing is less than anything :-) FWIW, I don't think we need to invent a new name for it, just add an appropriate tp_richcompare slot to the PyNoneType or readd the special case to Object/object.c. This would also aid in porting existing Python 2 code to Python 3. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 17 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From storchaka at gmail.com Mon Feb 17 12:47:19 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 17 Feb 2014 13:47:19 +0200 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5301EEFD.9020502@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> Message-ID: 17.02.14 13:14, M.-A. Lemburg ???????(??): > Here's a particularly nasty case: > >>>> l = [(1, None), (2, None)] >>>> l.sort() >>>> l > [(1, None), (2, None)] > >>>> l = [(1, None), (2, None), (3, 4)] >>>> l.sort() >>>> l > [(1, None), (2, None), (3, 4)] > >>>> l = [(1, None), (2, None), (3, 4), (2, 3)] >>>> l.sort() > Traceback (most recent call last): > File "", line 1, in > TypeError: unorderable types: int() < NoneType() If you replace None to another value that cannot be compared with int (e.g. string), you will got the same nasty case. From gjcarneiro at gmail.com Mon Feb 17 12:49:52 2014 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Mon, 17 Feb 2014 11:49:52 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5301F5DD.9050101@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> Message-ID: On 17 February 2014 11:43, M.-A. Lemburg wrote: > On 17.02.2014 12:23, Gustavo Carneiro wrote: > > On 17 February 2014 11:14, M.-A. Lemburg wrote: > > > >> On 15.02.2014 07:03, Stephen J. Turnbull wrote: > >>> M.-A. Lemburg writes: > >>> > >>> > IMO, it was a mistake to have None return a TypeError in > >>> > comparisons, since it makes many typical data operations > >>> > fail, e.g. > >>> > >>> I don't understand this statement. The theory is that they *should* > >>> fail. > >>> > >>> The example of sort is a good one. Sometimes you want missing values > >>> to be collected at the beginning of a list, sometimes at the end. > >>> Sometimes you want them treated as top elements, sometimes as bottom. > >>> And sometimes it is a real error for missing values to be present. > >>> Not to mention that sometimes the programmer simply hasn't thought > >>> about the appropriate policy. I don't think Python should silently > >>> impose a policy in that case, especially given that the programmer may > >>> have experience with any of the above treatments in other contexts. > >> > >> None is special in Python and has always (and intentionally) sorted > >> before any other object. In data processing and elsewhere in Python > >> programming, it's used to signal: no value available. > >> > >> Python 3 breaks this notion by always raising an exception when > >> using None in an ordered comparison, making it pretty much useless > >> for the above purpose. > >> > >> Yes, there are ways around this, but none of them are intuitive. > >> > >> Here's a particularly nasty case: > >> > >>>>> l = [(1, None), (2, None)] > >>>>> l.sort() > >>>>> l > >> [(1, None), (2, None)] > >> > >>>>> l = [(1, None), (2, None), (3, 4)] > >>>>> l.sort() > >>>>> l > >> [(1, None), (2, None), (3, 4)] > >> > >>>>> l = [(1, None), (2, None), (3, 4), (2, 3)] > >>>>> l.sort() > >> Traceback (most recent call last): > >> File "", line 1, in > >> TypeError: unorderable types: int() < NoneType() > >> > >> > > Maybe Python 3 should have a couple of None-like objects that compare the > > way you want: AlwaysComparesLess and AlwaysComparesGreater, but with > better > > names (maybe 'PlusInfinity' and 'MinusInfinity'?). Just leave None > alone, > > please. > > This doesn't only apply to numeric comparisons. In Python 2 you > can compare None with any kind of object and it always sorts first, > based on the intuition that nothing is less than anything :-) > > I still think that relying on your intuition is not the right way for Python. Refuse the temptation to guess. > FWIW, I don't think we need to invent a new name for it, just add > an appropriate tp_richcompare slot to the PyNoneType or readd the > special case to Object/object.c. This would also aid in porting > existing Python 2 code to Python 3. > Based on your comment, SortsFirst and SortsLast sound like good names ;-) These would be "universal sortable objects", that could be compared to any other type. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 17 2014) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ > -- Gustavo J. A. M. Carneiro Gambit Research LLC "The universe is always one step beyond logic." -- Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Feb 17 13:02:13 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 17 Feb 2014 12:02:13 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> Message-ID: On 17 February 2014 11:49, Gustavo Carneiro wrote: > >> >> FWIW, I don't think we need to invent a new name for it, just add >> an appropriate tp_richcompare slot to the PyNoneType or readd the >> special case to Object/object.c. This would also aid in porting >> existing Python 2 code to Python 3. > > > Based on your comment, SortsFirst and SortsLast sound like good names ;-) > > These would be "universal sortable objects", that could be compared to any > other type. I think that having both is over-engineered. For the use cases I know of, all that is wanted is *one* object that doesn't raise a TypeError when compared with any other object, and compares predictably (always before or always after, doesn't matter which). Whether that object is None, or whether it should be a new singleton, is not obvious. The advantage of None is that it's Python 2 compatible (which aids porting as noted). The advantage of a new object is that not all uses of None need universal sortability, and indeed in some cases universal sortability will hide bugs. Paul From mal at python.org Mon Feb 17 12:56:31 2014 From: mal at python.org (M.-A. Lemburg) Date: Mon, 17 Feb 2014 12:56:31 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> Message-ID: <5301F8EF.3000402@python.org> On 17.02.2014 12:47, Serhiy Storchaka wrote: > 17.02.14 13:14, M.-A. Lemburg ???????(??): >> Here's a particularly nasty case: >> >>>>> l = [(1, None), (2, None)] >>>>> l.sort() >>>>> l >> [(1, None), (2, None)] >> >>>>> l = [(1, None), (2, None), (3, 4)] >>>>> l.sort() >>>>> l >> [(1, None), (2, None), (3, 4)] >> >>>>> l = [(1, None), (2, None), (3, 4), (2, 3)] >>>>> l.sort() >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: unorderable types: int() < NoneType() > > If you replace None to another value that cannot be compared with int (e.g. string), you will got > the same nasty case. Yes, but that's not the point. Unlike strings or other mixed types that you cannot compare, None is used as placeholder in data processing as special value to mean "no value available". You intentionally use such values in programming. It's not a bug to have None in a data list or as value of a variable. -- Marc-Andre Lemburg Director Python Software Foundation http://www.python.org/psf/ From mal at python.org Mon Feb 17 13:11:47 2014 From: mal at python.org (M.-A. Lemburg) Date: Mon, 17 Feb 2014 13:11:47 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> Message-ID: <5301FC83.6080404@python.org> On 17.02.2014 12:49, Gustavo Carneiro wrote: > On 17 February 2014 11:43, M.-A. Lemburg wrote: > >> On 17.02.2014 12:23, Gustavo Carneiro wrote: >>> On 17 February 2014 11:14, M.-A. Lemburg wrote: >>> >>>> On 15.02.2014 07:03, Stephen J. Turnbull wrote: >>>>> M.-A. Lemburg writes: >>>>> >>>>> > IMO, it was a mistake to have None return a TypeError in >>>>> > comparisons, since it makes many typical data operations >>>>> > fail, e.g. >>>>> >>>>> I don't understand this statement. The theory is that they *should* >>>>> fail. >>>>> >>>>> The example of sort is a good one. Sometimes you want missing values >>>>> to be collected at the beginning of a list, sometimes at the end. >>>>> Sometimes you want them treated as top elements, sometimes as bottom. >>>>> And sometimes it is a real error for missing values to be present. >>>>> Not to mention that sometimes the programmer simply hasn't thought >>>>> about the appropriate policy. I don't think Python should silently >>>>> impose a policy in that case, especially given that the programmer may >>>>> have experience with any of the above treatments in other contexts. >>>> >>>> None is special in Python and has always (and intentionally) sorted >>>> before any other object. In data processing and elsewhere in Python >>>> programming, it's used to signal: no value available. >>>> >>>> Python 3 breaks this notion by always raising an exception when >>>> using None in an ordered comparison, making it pretty much useless >>>> for the above purpose. >>>> >>>> Yes, there are ways around this, but none of them are intuitive. >>>> >>>> Here's a particularly nasty case: >>>> >>>>>>> l = [(1, None), (2, None)] >>>>>>> l.sort() >>>>>>> l >>>> [(1, None), (2, None)] >>>> >>>>>>> l = [(1, None), (2, None), (3, 4)] >>>>>>> l.sort() >>>>>>> l >>>> [(1, None), (2, None), (3, 4)] >>>> >>>>>>> l = [(1, None), (2, None), (3, 4), (2, 3)] >>>>>>> l.sort() >>>> Traceback (most recent call last): >>>> File "", line 1, in >>>> TypeError: unorderable types: int() < NoneType() >>>> >>>> >>> Maybe Python 3 should have a couple of None-like objects that compare the >>> way you want: AlwaysComparesLess and AlwaysComparesGreater, but with >> better >>> names (maybe 'PlusInfinity' and 'MinusInfinity'?). Just leave None >> alone, >>> please. >> >> This doesn't only apply to numeric comparisons. In Python 2 you >> can compare None with any kind of object and it always sorts first, >> based on the intuition that nothing is less than anything :-) >> > I still think that relying on your intuition is not the right way for > Python. Refuse the temptation to guess. Hmm, are you sure ? There should be one-- and preferably only one --obvious way to do it. and then Although that way may not be obvious at first unless you're Dutch. which directly relates to: changeset: 16123:f997ded4e219 branch: legacy-trunk user: Guido van Rossum date: Mon Jan 22 19:28:09 2001 +0000 summary: New special case in comparisons: None is smaller than any other object :-) >> FWIW, I don't think we need to invent a new name for it, just add >> an appropriate tp_richcompare slot to the PyNoneType or readd the >> special case to Object/object.c. This would also aid in porting >> existing Python 2 code to Python 3. > > Based on your comment, SortsFirst and SortsLast sound like good names ;-) > > These would be "universal sortable objects", that could be compared to any > other type. Of course, it's easy to add a new type for this, but a lot of Python 2 code relies on None behaving this way, esp. code that reads data from databases, since None is the Python mapping for SQL NULL. -- Marc-Andre Lemburg Director Python Software Foundation http://www.python.org/psf/ From ncoghlan at gmail.com Mon Feb 17 13:12:48 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 17 Feb 2014 22:12:48 +1000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5301EEFD.9020502@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> Message-ID: On 17 Feb 2014 21:15, "M.-A. Lemburg" wrote: > > On 15.02.2014 07:03, Stephen J. Turnbull wrote: > > M.-A. Lemburg writes: > > > > > IMO, it was a mistake to have None return a TypeError in > > > comparisons, since it makes many typical data operations > > > fail, e.g. > > > > I don't understand this statement. The theory is that they *should* > > fail. > > > > The example of sort is a good one. Sometimes you want missing values > > to be collected at the beginning of a list, sometimes at the end. > > Sometimes you want them treated as top elements, sometimes as bottom. > > And sometimes it is a real error for missing values to be present. > > Not to mention that sometimes the programmer simply hasn't thought > > about the appropriate policy. I don't think Python should silently > > impose a policy in that case, especially given that the programmer may > > have experience with any of the above treatments in other contexts. > > None is special in Python and has always (and intentionally) sorted > before any other object. In data processing and elsewhere in Python > programming, it's used to signal: no value available. This is the first I've ever heard of that sorting behaviour being an intentional feature, rather than just an artefact of Python 2 allowing arbitrarily ordered comparisons between different types. Can you point me to the relevant entry in the Python 2 language reference? Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon Feb 17 13:19:49 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 17 Feb 2014 14:19:49 +0200 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5301F8EF.3000402@python.org> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F8EF.3000402@python.org> Message-ID: 17.02.14 13:56, M.-A. Lemburg ???????(??): > Yes, but that's not the point. Unlike strings or other mixed types that > you cannot compare, None is used as placeholder in data processing as > special value to mean "no value available". Isn't float('nan') such placeholder? > You intentionally use such values in programming. It's not a bug to > have None in a data list or as value of a variable. You can't have None in array('f'), you can't add or multiply by None. Relation operators don't looks an exception here. Applying sorted() to a list which contains numbers and Nones makes as much sense as applying sum() to it. From mal at egenix.com Mon Feb 17 13:25:42 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 17 Feb 2014 13:25:42 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> Message-ID: <5301FFC6.6030500@egenix.com> On 17.02.2014 13:12, Nick Coghlan wrote: > On 17 Feb 2014 21:15, "M.-A. Lemburg" wrote: >> >> On 15.02.2014 07:03, Stephen J. Turnbull wrote: >>> M.-A. Lemburg writes: >>> >>> > IMO, it was a mistake to have None return a TypeError in >>> > comparisons, since it makes many typical data operations >>> > fail, e.g. >>> >>> I don't understand this statement. The theory is that they *should* >>> fail. >>> >>> The example of sort is a good one. Sometimes you want missing values >>> to be collected at the beginning of a list, sometimes at the end. >>> Sometimes you want them treated as top elements, sometimes as bottom. >>> And sometimes it is a real error for missing values to be present. >>> Not to mention that sometimes the programmer simply hasn't thought >>> about the appropriate policy. I don't think Python should silently >>> impose a policy in that case, especially given that the programmer may >>> have experience with any of the above treatments in other contexts. >> >> None is special in Python and has always (and intentionally) sorted >> before any other object. In data processing and elsewhere in Python >> programming, it's used to signal: no value available. > > This is the first I've ever heard of that sorting behaviour being an > intentional feature, rather than just an artefact of Python 2 allowing > arbitrarily ordered comparisons between different types. Can you point me > to the relevant entry in the Python 2 language reference? This is not documented anywhere in the language spec, AFAIK. It is documented in the code (Python 2.7; Object/object.c): default_3way_compare(PyObject *v, PyObject *w) ... /* None is smaller than anything */ if (v == Py_None) return -1; if (w == Py_None) return 1; Note that it's not important whether None is smaller or greater than any other object. The important aspect is that it's sorting order is consistent and doesn't raise a TypeError when doing an ordered comparison with other objects. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 17 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Mon Feb 17 13:30:25 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 17 Feb 2014 13:30:25 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F8EF.3000402@python.org> Message-ID: <530200E1.8080301@egenix.com> On 17.02.2014 13:19, Serhiy Storchaka wrote: > 17.02.14 13:56, M.-A. Lemburg ???????(??): >> Yes, but that's not the point. Unlike strings or other mixed types that >> you cannot compare, None is used as placeholder in data processing as >> special value to mean "no value available". > > Isn't float('nan') such placeholder? You can easily construct other such placeholders, but None was intended for this purpose: http://docs.python.org/2.7/c-api/none.html?highlight=none#Py_None """ The Python None object, denoting lack of value. ... """ >> You intentionally use such values in programming. It's not a bug to >> have None in a data list or as value of a variable. > > You can't have None in array('f'), you can't add or multiply by None. Relation operators don't looks > an exception here. Applying sorted() to a list which contains numbers and Nones makes as much sense > as applying sum() to it. Of course, you cannot apply any operations with None - it doesn't have a value -, but you can compare it to other objects and it provides a consistent behavior in Python 2. Python 3 is missing such an object. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 17 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From gjcarneiro at gmail.com Mon Feb 17 14:29:36 2014 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Mon, 17 Feb 2014 13:29:36 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <530200E1.8080301@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F8EF.3000402@python.org> <530200E1.8080301@egenix.com> Message-ID: On 17 February 2014 12:30, M.-A. Lemburg wrote: > On 17.02.2014 13:19, Serhiy Storchaka wrote: > > 17.02.14 13:56, M.-A. Lemburg ???????(??): > >> Yes, but that's not the point. Unlike strings or other mixed types that > >> you cannot compare, None is used as placeholder in data processing as > >> special value to mean "no value available". > > > > Isn't float('nan') such placeholder? > > You can easily construct other such placeholders, but None was intended > for this purpose: > > http://docs.python.org/2.7/c-api/none.html?highlight=none#Py_None > > """ > The Python None object, denoting lack of value. ... > """ > > >> You intentionally use such values in programming. It's not a bug to > >> have None in a data list or as value of a variable. > > > > You can't have None in array('f'), you can't add or multiply by None. > Relation operators don't looks > > an exception here. Applying sorted() to a list which contains numbers > and Nones makes as much sense > > as applying sum() to it. > > Of course, you cannot apply any operations with None - it doesn't > have a value -, but you can compare it to other objects and it provides > a consistent behavior in Python 2. Python 3 is missing such an object. > I agree with you that Python 3 could use such an object. Just don't make it the default. Leave None as it is. Also I agree that my previous naming suggestions are bad. What we need is only one (or two) additional object whose main semantic meaning is still "no value", but which also adds a meaning of "comparable". Then it's a matter of choosing a good name for it, with lots of bikeshedding involved. Just lets not change None only because we're too lazy to discuss a proper alternative name. Also this use case is not _that_ common, so it's ok if it has a slightly longer name than None. Also think of the implications of changing None at this point. It would allow us to write programs that work Python >= 3.5 and Python <= 2.7, but fail mysteriously in all other versions in between. What a mess that would be... -- Gustavo J. A. M. Carneiro Gambit Research LLC "The universe is always one step beyond logic." -- Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Mon Feb 17 15:30:39 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 17 Feb 2014 15:30:39 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F8EF.3000402@python.org> <530200E1.8080301@egenix.com> Message-ID: <53021D0F.6030409@egenix.com> On 17.02.2014 14:29, Gustavo Carneiro wrote: > On 17 February 2014 12:30, M.-A. Lemburg wrote: > >> On 17.02.2014 13:19, Serhiy Storchaka wrote: >>> 17.02.14 13:56, M.-A. Lemburg ???????(??): >>>> Yes, but that's not the point. Unlike strings or other mixed types that >>>> you cannot compare, None is used as placeholder in data processing as >>>> special value to mean "no value available". >>> >>> Isn't float('nan') such placeholder? >> >> You can easily construct other such placeholders, but None was intended >> for this purpose: >> >> http://docs.python.org/2.7/c-api/none.html?highlight=none#Py_None >> >> """ >> The Python None object, denoting lack of value. ... >> """ >> >>>> You intentionally use such values in programming. It's not a bug to >>>> have None in a data list or as value of a variable. >>> >>> You can't have None in array('f'), you can't add or multiply by None. >> Relation operators don't looks >>> an exception here. Applying sorted() to a list which contains numbers >> and Nones makes as much sense >>> as applying sum() to it. >> >> Of course, you cannot apply any operations with None - it doesn't >> have a value -, but you can compare it to other objects and it provides >> a consistent behavior in Python 2. Python 3 is missing such an object. >> > > I agree with you that Python 3 could use such an object. Just don't make > it the default. Leave None as it is. > > Also I agree that my previous naming suggestions are bad. What we need is > only one (or two) additional object whose main semantic meaning is still > "no value", but which also adds a meaning of "comparable". Then it's a > matter of choosing a good name for it, with lots of bikeshedding involved. > Just lets not change None only because we're too lazy to discuss a proper > alternative name. Also this use case is not _that_ common, so it's ok if > it has a slightly longer name than None. > > Also think of the implications of changing None at this point. It would > allow us to write programs that work Python >= 3.5 and Python <= 2.7, but > fail mysteriously in all other versions in between. What a mess that would > be... Yes, that's unfortunately true. If you regard this as bug in the Python 3 series, we could still fix it for 3.3 and 3.4, though. After all, we'd just be removing an unwanted exception and not make already working Python 3 applications fail. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 17 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From rosuav at gmail.com Mon Feb 17 15:39:51 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 18 Feb 2014 01:39:51 +1100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <53021D0F.6030409@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F8EF.3000402@python.org> <530200E1.8080301@egenix.com> <53021D0F.6030409@egenix.com> Message-ID: On Tue, Feb 18, 2014 at 1:30 AM, M.-A. Lemburg wrote: >> >> Also think of the implications of changing None at this point. It would >> allow us to write programs that work Python >= 3.5 and Python <= 2.7, but >> fail mysteriously in all other versions in between. What a mess that would >> be... > > Yes, that's unfortunately true. I don't know that that's in itself that much of a problem. (BTW, I wouldn't call it "<= 2.7"; it'd be all 2.x, it's not like some of the 2.7.y versions aren't included.) There are already barriers to supporting 2.7 and 3.1/3.2, like the u"asdf" notation for Unicode literals. Making it easier to support 2.x and 3.x from the same codebase is an improvement that has been done and will be done more. It's not such a mysterious failure; it's just that you support Python 2.5+ and 3.4+ (or whatever the specific versions are). ChrisA From jon+python-dev at unequivocal.co.uk Mon Feb 17 15:38:11 2014 From: jon+python-dev at unequivocal.co.uk (Jon Ribbens) Date: Mon, 17 Feb 2014 14:38:11 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5301F5DD.9050101@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> Message-ID: <20140217143811.GX4415@snowy.squish.net> On Mon, Feb 17, 2014 at 12:43:25PM +0100, M.-A. Lemburg wrote: > This doesn't only apply to numeric comparisons. In Python 2 you > can compare None with any kind of object and it always sorts first, No you can't. See http://bugs.python.org/issue1673405 . According to Tim Peters, the "None is less than everything" rule never existed. From mal at egenix.com Mon Feb 17 16:22:54 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 17 Feb 2014 16:22:54 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <20140217143811.GX4415@snowy.squish.net> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <20140217143811.GX4415@snowy.squish.net> Message-ID: <5302294E.6040805@egenix.com> On 17.02.2014 15:38, Jon Ribbens wrote: > On Mon, Feb 17, 2014 at 12:43:25PM +0100, M.-A. Lemburg wrote: >> This doesn't only apply to numeric comparisons. In Python 2 you >> can compare None with any kind of object and it always sorts first, > > No you can't. See http://bugs.python.org/issue1673405 . > > According to Tim Peters, the "None is less than everything" rule > never existed. Well, then Tim probably didn't read the code in object.c :-) Seriously, the datetime module types were the first types to experiment with the new mixed type operation mechanism added at the time: http://www.python.org/dev/peps/pep-0207/ http://www.python.org/dev/peps/pep-0208/ Objects implementing the rich comparison slot can indeed override these defaults and also raise exceptions in comparison operations, so you're right: None comparisons can still be made to raise exceptions, even in Python 2.7. Still, None comparisons work just fine for most types in Python 2.x and people have written code assuming that it works for many years. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 17 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From tjreedy at udel.edu Mon Feb 17 16:29:14 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 Feb 2014 10:29:14 -0500 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5301FFC6.6030500@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301FFC6.6030500@egenix.com> Message-ID: On 2/17/2014 7:25 AM, M.-A. Lemburg wrote: > On 17.02.2014 13:12, Nick Coghlan wrote: >> On 17 Feb 2014 21:15, "M.-A. Lemburg" wrote: >>> >>> On 15.02.2014 07:03, Stephen J. Turnbull wrote: >>>> M.-A. Lemburg writes: >>>> >>>> > IMO, it was a mistake to have None return a TypeError in >>>> > comparisons, since it makes many typical data operations >>>> > fail, e.g. >>>> >>>> I don't understand this statement. The theory is that they *should* >>>> fail. >>>> >>>> The example of sort is a good one. Sometimes you want missing values >>>> to be collected at the beginning of a list, sometimes at the end. >>>> Sometimes you want them treated as top elements, sometimes as bottom. >>>> And sometimes it is a real error for missing values to be present. >>>> Not to mention that sometimes the programmer simply hasn't thought >>>> about the appropriate policy. I don't think Python should silently >>>> impose a policy in that case, especially given that the programmer may >>>> have experience with any of the above treatments in other contexts. >>> >>> None is special in Python and has always (and intentionally) sorted >>> before any other object. In data processing and elsewhere in Python >>> programming, it's used to signal: no value available. >> >> This is the first I've ever heard of that sorting behaviour being an >> intentional feature, rather than just an artefact of Python 2 allowing >> arbitrarily ordered comparisons between different types. Can you point me >> to the relevant entry in the Python 2 language reference? > > This is not documented anywhere in the language spec, AFAIK. http://docs.python.org/2/reference/expressions.html#not-in "The operators <, >, ==, >=, <=, and != compare the values of two objects. The objects need not have the same type. If both are numbers, they are converted to a common type. Otherwise, objects of different types always compare unequal, and are ordered consistently but arbitrarily." http://docs.python.org/2/library/stdtypes.html#comparisons "Objects of different types, except different numeric types and different string types, never compare equal; such objects are ordered consistently but arbitrarily" It goes on to note the exception for complex numbers, but none for None. It continues "CPython implementation detail: Objects of different types except numbers are ordered by their type names;" Again, there is no exception noted for None, although, since the type name of None is 'NoneType', its behavior does not match that doc note. I believe that CPython implementation detail was some of the arbitrary orderings changed in CPython between 1.3 and 2.7. I do not know whether other implementations all mimicked CPython or not, but the reference manual does not require that. > It is documented in the code (Python 2.7; Object/object.c): > > default_3way_compare(PyObject *v, PyObject *w) This is the 'final fallback' for comparisons, not the first thing tried. > ... > /* None is smaller than anything */ Reading CPython C code is not supposed to be a requirement for programming in Python. If that comment were true, I would regard it as only documenting a version of CPython, not the language standard. But it is not even true in 2.7.6. But it is not even true. >>> class Bottom(object): # get same results below without 'object' def __lt__(self, other): return True # the following two results are consistent and # contradict the claim that 'None is smaller than anything' >>> Bottom() < None True >>> cmp(Bottom(), None) -1 # the following two results are not consistent with the # definition of cmp, so 1 of the 2 is buggy >>> None < Bottom() True >>> cmp(None, Bottom()) 1 It appears that 'anything' needs to be qualified as something like 'anything that does not itself handle comparison with None or is not given a chance to do so in a particular comparison expression'. > if (v == Py_None) > return -1; > if (w == Py_None) > return 1; > Note that it's not important whether None is smaller or greater > than any other object. The important aspect is that it's sorting > order is consistent and doesn't raise a TypeError when doing an > ordered comparison with other objects. I can agree that it might have been nice if None had been defined as a universal bottom object, but it has not been so defined and it did not act as such. To make it (or anything else) a true bottom object would require a change in the way comparisons are evaluated. Comparisons with 'bottom' would have to be detected and special-cased at the beginning of the code, not at the end as a fallback. -- Terry Jan Reedy From larry at hastings.org Mon Feb 17 16:33:26 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 17 Feb 2014 07:33:26 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> Message-ID: <53022BC6.50902@hastings.org> On 02/16/2014 03:45 PM, Paul Moore wrote: > is significant enough to be > resulting in a 3.3.5 release - can you make sure the 3.4 fix goes in? > I'm not sure how to find the revision number that contains the fix to > follow the process you outline above, so I'm just mentioning it here & > on the issue to make sure it's not missed... If it has an issue number like that, then generally Roundup Robot will notify the issue when something is checked in. And those comments will have the revision id in them. Also, you can scan the output of "hg log" to find the revision numbers. The first line of each stanza looks like changeset: 89231:75a12cf63f20 Here, "75a12cf63f20" is the revision id. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Feb 17 17:03:39 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 17 Feb 2014 16:03:39 +0000 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <53022BC6.50902@hastings.org> References: <53014900.7090908@hastings.org> <53022BC6.50902@hastings.org> Message-ID: Thanks, I see. Greg already filed a tracking issue, so it's covered anyway. On 17 February 2014 15:33, Larry Hastings wrote: > On 02/16/2014 03:45 PM, Paul Moore wrote: > > is significant enough to be > resulting in a 3.3.5 release - can you make sure the 3.4 fix goes in? > I'm not sure how to find the revision number that contains the fix to > follow the process you outline above, so I'm just mentioning it here & > on the issue to make sure it's not missed... > > > If it has an issue number like that, then generally Roundup Robot will > notify the issue when something is checked in. And those comments will have > the revision id in them. > > Also, you can scan the output of "hg log" to find the revision numbers. The > first line of each stanza looks like > > changeset: 89231:75a12cf63f20 > > Here, "75a12cf63f20" is the revision id. > > > /arry From larry at hastings.org Mon Feb 17 18:36:17 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 17 Feb 2014 09:36:17 -0800 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: References: <530103F9.8070005@hastings.org> Message-ID: <53024891.903@hastings.org> On 02/16/2014 04:03 PM, Victor Stinner wrote: > Hi, > > The PEP 436 is still a draft and not mentionned in Python 3.4 > changelog. The PEP proposes to add a DSL, not to modify all modules > implemented in C. I think that it should be marked as Final and > mentionned in the changelog. > http://www.python.org/dev/peps/pep-0436/ I need to clean it up and submit it for final inspection. This is on my to-do list for before 3.4.0 final. > 2014-02-16 19:31 GMT+01:00 Larry Hastings : >> 1) We merge the Derby patch for the builtins module into 3.4, simply because >> it will "demo well". > Where is the issue to implement this feature? Anyway, I expect a huge > patch which is non-trivial and so very risky for such a very important > module :-/ It's too late IMO. http://bugs.python.org/issue20184 The point is that it *is* trivial and not particularly risky. Doing this right means that we execute essentially the same statements in the same order, merely split across two functions, and with a slightly different docstring. Literally the only detectable runtime difference that should exist after applying the patch is that the functions now publish docstrings. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Feb 17 18:59:44 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 Feb 2014 12:59:44 -0500 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5302294E.6040805@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <20140217143811.GX4415@snowy.squish.net> <5302294E.6040805@egenix.com> Message-ID: On 2/17/2014 10:22 AM, M.-A. Lemburg wrote: > On 17.02.2014 15:38, Jon Ribbens wrote: >> On Mon, Feb 17, 2014 at 12:43:25PM +0100, M.-A. Lemburg wrote: >>> This doesn't only apply to numeric comparisons. In Python 2 you >>> can compare None with any kind of object and it always sorts first, >> >> No you can't. See http://bugs.python.org/issue1673405 . >> >> According to Tim Peters, the "None is less than everything" rule >> never existed. Tim is correct. Copying from my other response (posted after you wrote this) >>> class Bottom(object): # get same results below without 'object' def __lt__(self, other): return True # the following two results are consistent and # contradict the claim that 'None is smaller than anything' >>> Bottom() < None True >>> cmp(Bottom(), None) -1 # the following two results are not consistent with the # definition of cmp, so 1 of the 2 is buggy >>> None < Bottom() True >>> cmp(None, Bottom()) 1 > Well, then Tim probably didn't read the code in object.c :-) I did, as I suspect Time has also. Function default_3way_compare is a 'final fallback'. The comment within, besides being a code comment and not the doc, is wrong unless 'anything' is qualified. -- Terry Jan Reedy From storchaka at gmail.com Mon Feb 17 19:18:24 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 17 Feb 2014 20:18:24 +0200 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5301FC83.6080404@python.org> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> Message-ID: 17.02.14 14:11, M.-A. Lemburg ???????(??): > Of course, it's easy to add a new type for this, but a lot of Python 2 > code relies on None behaving this way, esp. code that reads data from > databases, since None is the Python mapping for SQL NULL. At the same time a lot of Python 2 relies on the assumption that any two values are comparable. I suppose that I see a code that try to sort a heterogeneous list more often than a code which want to sort a list containing only numbers and Nones. From storchaka at gmail.com Mon Feb 17 19:23:33 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 17 Feb 2014 20:23:33 +0200 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> Message-ID: 17.02.14 20:18, Serhiy Storchaka ???????(??): > 17.02.14 14:11, M.-A. Lemburg ???????(??): >> Of course, it's easy to add a new type for this, but a lot of Python 2 >> code relies on None behaving this way, esp. code that reads data from >> databases, since None is the Python mapping for SQL NULL. > > At the same time a lot of Python 2 relies on the assumption that any two > values are comparable. I suppose that I see a code that try to sort a > heterogeneous list more often than a code which want to sort a list > containing only numbers and Nones. Sorry, I press "Send" too early. If we consider the idea to make None universally comparable, we should consider the idea to make all other values universally comparable too. And consider again the reasons which caused these changes in Python 3. From tjreedy at udel.edu Mon Feb 17 19:50:20 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 Feb 2014 13:50:20 -0500 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <20140217143811.GX4415@snowy.squish.net> <5302294E.6040805@egenix.com> Message-ID: On 2/17/2014 12:59 PM, Terry Reedy wrote: > On 2/17/2014 10:22 AM, M.-A. Lemburg wrote: >> On 17.02.2014 15:38, Jon Ribbens wrote: >>> On Mon, Feb 17, 2014 at 12:43:25PM +0100, M.-A. Lemburg wrote: >>>> This doesn't only apply to numeric comparisons. In Python 2 you >>>> can compare None with any kind of object and it always sorts first, >>> >>> No you can't. See http://bugs.python.org/issue1673405 . This issue was about the fact that datetimes do not compare with None, which means that even in 2.x, putting None in a list of datetimes (to mean no datetime) means that the list cannot be sorted. >>> According to Tim Peters, the "None is less than everything" rule >>> never existed. > > Tim is correct. Copying from my other response (posted after you wrote > this) > > >>> class Bottom(object): # get same results below without 'object' > def __lt__(self, other): > return True > > # the following two results are consistent and > # contradict the claim that 'None is smaller than anything' > >>> Bottom() < None > True > >>> cmp(Bottom(), None) > -1 > > # the following two results are not consistent with the > # definition of cmp, so 1 of the 2 is buggy > >>> None < Bottom() > True > >>> cmp(None, Bottom()) > 1 > >> Well, then Tim probably didn't read the code in object.c :-) > > I did, as I suspect Time has also. Function default_3way_compare is a > 'final fallback'. The comment within, besides being a code comment and > not the doc, is wrong unless 'anything' is qualified. FWIW: do_cmp first calls try_rich_to_3way_compare and if that fails it calls try_3way_compare and if that fails it calls default_3way_compare. Somewhat confusingly, try_rich_to_3way_compare first calls try_3way_compare and if that fails, it calls default_3way_compare. So the backup calls in do_cmp are redundant, as they will have already failed in try_rich_to_3way_compare. The special casing of None in default_3way_compare was added by Guido in rev. 16123 on 2001 Jan 22. Before that, None was compared by the typename, as still specified in the docs. Tim was correct when he wrote "For older types, the result of inequality comparison with None isn't defined by the language, and the outcome does vary across CPython releases." -- Terry Jan Reedy From tjreedy at udel.edu Mon Feb 17 20:00:44 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 Feb 2014 14:00:44 -0500 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> Message-ID: On 2/17/2014 1:18 PM, Serhiy Storchaka wrote: > 17.02.14 14:11, M.-A. Lemburg ???????(??): >> Of course, it's easy to add a new type for this, but a lot of Python 2 >> code relies on None behaving this way, esp. code that reads data from >> databases, since None is the Python mapping for SQL NULL. > > At the same time a lot of Python 2 relies on the assumption that any two > values are comparable. That assumption was very intentionally broken and abandoned by Guido when complex numbers were added over a decade ago. It was further abandoned, intentionally, when datetimes were added. I think it was decided then to finish the job in 3.0. I suppose that I see a code that try to sort a > heterogeneous list more often than a code which want to sort a list > containing only numbers and Nones. -- Terry Jan Reedy From tim.peters at gmail.com Mon Feb 17 21:12:47 2014 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 17 Feb 2014 14:12:47 -0600 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> Message-ID: The behavior of None in comparisons is intentional in Python 3. You can agitate to change it, but it will probably die when Guido gets wind of it ;-) The catch-all mixed-type comparison rules in Pythons 1 and 2 were only intended to be "arbitrary but consistent". Of course each specific release picked some specific scheme, but these schemes weren't documented, and intentionally not. For "a long time" CPython's default for mixed-typed comparisons was to compare the names of the types (as strings). And for just as long, nobody had noticed that this _wasn't_ always "consistent". For example, 3L < 4 must be true. But [] < 3L was also true (because "list" < "long"), and 4 < [] was true too (because "int" < "list"). So, in all, [] < 3L < 4 < [] was true, implying [] < []. Part of fixing that was removing some cases of "compare objects of different types by comparing the type name strings". Guido & I were both in the office at the time, and one said to the other: "what about None? Comparing that to other types via comparing the string 'None' doesn't make much sense." "Ya, OK ... how about changing None to - by default - comparing 'less than' objects of other types?" "Don't see why not - sure." "OK, done!". No more than 2 minutes of thought went into it. There was no intent to cater to any real use case here - the only intent was to pick some arbitrary-but-consistent rule that didn't suck _quite_ as badly as pretending None was the string "None" ;-) Guido wanted to drop all the "arbitrary but consistent" mixed-type comparison crud for Python 3. Nothing special about None in that. As already noted, the various `datetime` types were the first to experiment with implementing full blown Python3-ish mixed-type comparison rules. After the first two times that caught actual bugs in code using the new types, there was no turning back. It's not so much that Python 3 finished the job as that Python 2 started it ;-) much-ado-about-nonething-ly y'rs - tim From mal at egenix.com Mon Feb 17 21:53:32 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 17 Feb 2014 21:53:32 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> Message-ID: <530276CC.5040408@egenix.com> On 17.02.2014 21:12, Tim Peters wrote: > [...] > > Guido wanted to drop all the "arbitrary but consistent" mixed-type > comparison crud for Python 3. Nothing special about None in that. As > already noted, the various `datetime` types were the first to > experiment with implementing full blown Python3-ish mixed-type > comparison rules. After the first two times that caught actual bugs > in code using the new types, there was no turning back. It's not so > much that Python 3 finished the job as that Python 2 started it ;-) Well, I guess it depends on how you look at it. None worked as "compares less than all other objects" simply due to the fact that None is a singleton and doesn't implement the comparison slots (which for all objects not implementing rich comparisons, meant that the fallback code triggered in Python 2). In Python 3 the special casing was dropped and because None still doesn't implement the comparison slot (tp_richcompare this time), it doesn't support ordering comparisons anymore. Now, the choice to have None compare less than all other objects may have been arbitrary, but IMO it was a good, consistent and useful choice. So why not bring it back and perhaps this time in a way that actually does work consistently for all Python objects by implementing the tp_richcompare slot on PyNoneType objects and documenting it ?! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 17 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Mon Feb 17 23:16:58 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 18 Feb 2014 08:16:58 +1000 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: <53024891.903@hastings.org> References: <530103F9.8070005@hastings.org> <53024891.903@hastings.org> Message-ID: On 18 Feb 2014 03:37, "Larry Hastings" wrote: > > > On 02/16/2014 04:03 PM, Victor Stinner wrote: >> >> Hi, >> >> The PEP 436 is still a draft and not mentionned in Python 3.4 >> changelog. The PEP proposes to add a DSL, not to modify all modules >> implemented in C. I think that it should be marked as Final and >> mentionned in the changelog. >> http://www.python.org/dev/peps/pep-0436/ > > > I need to clean it up and submit it for final inspection. This is on my to-do list for before 3.4.0 final. > > >> 2014-02-16 19:31 GMT+01:00 Larry Hastings : >>> >>> 1) We merge the Derby patch for the builtins module into 3.4, simply because >>> it will "demo well". >> >> Where is the issue to implement this feature? Anyway, I expect a huge >> patch which is non-trivial and so very risky for such a very important >> module :-/ It's too late IMO. > > > http://bugs.python.org/issue20184 > > The point is that it *is* trivial and not particularly risky. Doing this right means that we execute essentially the same statements in the same order, merely split across two functions, and with a slightly different docstring. Literally the only detectable runtime difference that should exist after applying the patch is that the functions now publish docstrings. (Last "docstrings" should be "introspectable signatures") The ones which weren't trivial, either won't be changed at all until 3.5 (with a comment in the existing patch explaining why not), or else have had their automated generation disabled after the initial conversion so I could move them back closer to their existing behaviour. I also discovered that gdb depends on the *exact* signature of the id() C API, but Argument Clinic was able to handle that. That said, I still recommend leaving this particular change until 3.4.1 at this stage of the release cycle - it's a nice-to-have, not a must have, and I think there are more important things to focus on right now (like handling any identified regressions relative to 3.3, since the release candidates see a higher level of third party testing than any of the earlier pre-releases). The current patch also can't be applied as is - it needs to be regenerated to account for the signature format change and to move the argument parsing helpers out to a separate file. Something I think we *should* do (and I'm willing to take care of) is update the What's New to say explicitly that adding programmatic introspection support to a function implemented in C is now considered a bug fix (as long as it can be done without changing semantics and the inspect module can handle it), and so more builtin and extension module APIs will start supporting that over the course of the 3.4.x series. I'll also find some already converted examples to include in the What's New description (relying on the converted builtin slots if I have to, but I suspect some better examples are already available, even without the builtins converted). Cheers, Nick. > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 17 23:25:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 18 Feb 2014 08:25:27 +1000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5301FFC6.6030500@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301FFC6.6030500@egenix.com> Message-ID: On 17 Feb 2014 22:25, "M.-A. Lemburg" wrote: > > On 17.02.2014 13:12, Nick Coghlan wrote: > > On 17 Feb 2014 21:15, "M.-A. Lemburg" wrote: > >> > >> On 15.02.2014 07:03, Stephen J. Turnbull wrote: > >>> M.-A. Lemburg writes: > >>> > >>> > IMO, it was a mistake to have None return a TypeError in > >>> > comparisons, since it makes many typical data operations > >>> > fail, e.g. > >>> > >>> I don't understand this statement. The theory is that they *should* > >>> fail. > >>> > >>> The example of sort is a good one. Sometimes you want missing values > >>> to be collected at the beginning of a list, sometimes at the end. > >>> Sometimes you want them treated as top elements, sometimes as bottom. > >>> And sometimes it is a real error for missing values to be present. > >>> Not to mention that sometimes the programmer simply hasn't thought > >>> about the appropriate policy. I don't think Python should silently > >>> impose a policy in that case, especially given that the programmer may > >>> have experience with any of the above treatments in other contexts. > >> > >> None is special in Python and has always (and intentionally) sorted > >> before any other object. In data processing and elsewhere in Python > >> programming, it's used to signal: no value available. > > > > This is the first I've ever heard of that sorting behaviour being an > > intentional feature, rather than just an artefact of Python 2 allowing > > arbitrarily ordered comparisons between different types. Can you point me > > to the relevant entry in the Python 2 language reference? > > This is not documented anywhere in the language spec, AFAIK. It > is documented in the code (Python 2.7; Object/object.c): > > default_3way_compare(PyObject *v, PyObject *w) > ... > /* None is smaller than anything */ > if (v == Py_None) > return -1; > if (w == Py_None) > return 1; > > Note that it's not important whether None is smaller or greater > than any other object. The important aspect is that it's sorting > order is consistent and doesn't raise a TypeError when doing an > ordered comparison with other objects. Thanks, that's enough to persuade me that it is a good idea to restore that behaviour (and make it an official part of the language spec) as part of the "eliminate remaining barriers to migration from Python 2" effort in 3.5. It will mean that SQL sorting involving NULL values maps cleanly again, and it's easy enough to use any() to check that container doesn't contain None. Cheers, Nick. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 17 2014) > >>> Python Projects, Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 > > ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 17 23:38:48 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 18 Feb 2014 08:38:48 +1000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301FFC6.6030500@egenix.com> Message-ID: On 18 Feb 2014 08:25, "Nick Coghlan" wrote: > > > On 17 Feb 2014 22:25, "M.-A. Lemburg" wrote: > > > > On 17.02.2014 13:12, Nick Coghlan wrote: > > > On 17 Feb 2014 21:15, "M.-A. Lemburg" wrote: > > >> > > >> None is special in Python and has always (and intentionally) sorted > > >> before any other object. In data processing and elsewhere in Python > > >> programming, it's used to signal: no value available. > > > > > > This is the first I've ever heard of that sorting behaviour being an > > > intentional feature, rather than just an artefact of Python 2 allowing > > > arbitrarily ordered comparisons between different types. Can you point me > > > to the relevant entry in the Python 2 language reference? > > > > This is not documented anywhere in the language spec, AFAIK. It > > is documented in the code (Python 2.7; Object/object.c): > > > > default_3way_compare(PyObject *v, PyObject *w) > > ... > > /* None is smaller than anything */ > > if (v == Py_None) > > return -1; > > if (w == Py_None) > > return 1; > > > > Note that it's not important whether None is smaller or greater > > than any other object. The important aspect is that it's sorting > > order is consistent and doesn't raise a TypeError when doing an > > ordered comparison with other objects. > > Thanks, that's enough to persuade me that it is a good idea to restore that behaviour (and make it an official part of the language spec) as part of the "eliminate remaining barriers to migration from Python 2" effort in 3.5. > > It will mean that SQL sorting involving NULL values maps cleanly again, and it's easy enough to use any() to check that container doesn't contain None. Note that I sent this before reading Tim's reply regarding the origin of the Python 2 sorting behaviour, and Terry's point that the relevant Python 2 code lived in the legacy 3 way comparison fallback that was deliberately removed from 3.0. MAL's perspective does still at least mean I am only -0 rather than -1 on making None always sorts first rather than throwing an exception, but my preferred solution remains to provide backporting-friendly support for handling of None values in comparisons (and particularly sorting operations): http://bugs.python.org/issue20630 Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Feb 18 00:12:32 2014 From: barry at python.org (Barry Warsaw) Date: Mon, 17 Feb 2014 18:12:32 -0500 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301FFC6.6030500@egenix.com> Message-ID: <20140217181232.1e185c55@anarchist.wooz.org> On Feb 18, 2014, at 08:25 AM, Nick Coghlan wrote: >Thanks, that's enough to persuade me that it is a good idea to restore that >behaviour (and make it an official part of the language spec) as part of >the "eliminate remaining barriers to migration from Python 2" effort in 3.5. At the very least, it's an idea worth PEP-ifying. -Barry From victor.stinner at gmail.com Tue Feb 18 00:20:03 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Feb 2014 00:20:03 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <53014900.7090908@hastings.org> References: <53014900.7090908@hastings.org> Message-ID: 2014-02-17 0:25 GMT+01:00 Larry Hastings : > You might think that anything you check in to the "default" branch in Python > trunk will go into 3.4.0 rc2, and after that ships, checkins would go into > 3.4.0 final. Ho ho ho! That's not true! Instead, anything checked in to > "default" between my last revision for "rc1" (e64ae8b82672) and 3.4.0 final > will by default go into 3.4.1. Only fixes that I cherry-pick into my local > branch will go into 3.4.0 rc2 and final. And my local branch will remain > private until 3.4.0 final ships! Does it mean that the default branch is open for changes which target 3.4.1? Victor From tjreedy at udel.edu Tue Feb 18 00:25:01 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 Feb 2014 18:25:01 -0500 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301FFC6.6030500@egenix.com> Message-ID: <53029A4D.30702@udel.edu> On 2/17/2014 5:25 PM, Nick Coghlan wrote: > > On 17 Feb 2014 22:25, "M.-A. Lemburg" > wrote: > > default_3way_compare(PyObject *v, PyObject *w) > > ... > > /* None is smaller than anything */ Unless it is not, as with datetimes, perhaps other classes written similarly, and some user class instances. > > Note that it's not important whether None is smaller or greater > > than any other object. The important aspect is that it's sorting > > order is consistent and doesn't raise a TypeError when doing an > > ordered comparison with other objects. > > Thanks, that's enough to persuade me that it is a good idea to restore > that behaviour Would you restore the actual sometimes inconsistent 2.x behavior or implement something new -- what M-A claims but never was? I doubt the former would be trivial since it was part of the now deleted cmp and 3way machinery. To make None a true bottom object, the rich comparison methods would have to special-case None as either argument before looking at the __rc__ special methods of either. Regardless of how implemented, such a change would break user code that defines a Bottom class and depends on Bottom() < None == True. Given the python-ideas discussions about adding a top or bottom class, I expect that such classes are in use. So a deprecation period would be needed, pushing the change off to 3.7. It is also possible that someone took the ending of cross-type comparisons seriously and is depending on the TypeError. while True: x = f(args) try: if x > 10: a(x) else: b(x) except TypeError: # f did not return a number break -- Terry Jan Reedy From larry at hastings.org Tue Feb 18 00:33:18 2014 From: larry at hastings.org (Larry Hastings) Date: Mon, 17 Feb 2014 15:33:18 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> Message-ID: <53029C3E.8030905@hastings.org> On 02/17/2014 03:20 PM, Victor Stinner wrote: > 2014-02-17 0:25 GMT+01:00 Larry Hastings : >> You might think that anything you check in to the "default" branch in Python >> trunk will go into 3.4.0 rc2, and after that ships, checkins would go into >> 3.4.0 final. Ho ho ho! That's not true! Instead, anything checked in to >> "default" between my last revision for "rc1" (e64ae8b82672) and 3.4.0 final >> will by default go into 3.4.1. Only fixes that I cherry-pick into my local >> branch will go into 3.4.0 rc2 and final. And my local branch will remain >> private until 3.4.0 final ships! > Does it mean that the default branch is open for changes which target 3.4.1? Basically, yes. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Tue Feb 18 05:25:28 2014 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 17 Feb 2014 22:25:28 -0600 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <530276CC.5040408@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <530276CC.5040408@egenix.com> Message-ID: [M.-A. Lemburg] > ... > None worked as "compares less than all other objects" simply due > to the fact that None is a singleton and doesn't implement the > comparison slots (which for all objects not implementing rich > comparisons, meant that the fallback code triggered in Python 2). And the fallback code - which you've already found - went on to special-case the snot out of None to make "less than" work. It's not like it was "a natural" outcome: it was forced. > In Python 3 the special casing was dropped and because None > still doesn't implement the comparison slot (tp_richcompare this time), > it doesn't support ordering comparisons anymore. Which makes mixed-type non-numeric default comparisons involving None work the same way all other mixed-type non-numeric default comparisons work in Python 3. This was the result of a deliberate design decision about how comparison _should_ work in Python 3, not an accidental consequence of the None type not defining tp_richcompare. IOW, this is about design, not about implementation. The differing implementations you see are consequences of the differing designs. Staring at the implementations is irrelevant to the design decisions. > Now, the choice to have None compare less than all other objects > may have been arbitrary, but IMO it was a good, consistent and > useful choice. Possibly useful for some apps, sure. Not for my apps. For example, when I initialize an object attribute to None in Python 3, I _expect_ I'll get an exception if I try to use that attribute in most contexts. It makes no more sense to ask whether that attribute is, say, less than 3, than it does to add 3 to it. The exception is most useful then. More often than not, it was annoying to me that Python 2 _didn't_ whine about trying to compare None. > So why not bring it back A huge barrier is (or should be) that Python 3 is over 5 years old now. Fiddling with the semantics of basic builtin types was possible - and even welcome - 6 years ago. Now they really shouldn't be touched in the absence of a critical bug, or a wholly backward-compatible (with Python 3) addition. > and perhaps this time in a way that actually does work consistently for > all Python objects by implementing the tp_richcompare slot on > PyNoneType objects and documenting it ?! Something to suggest for Python 4, in which case I'll only be -0 ;-) From tjreedy at udel.edu Tue Feb 18 05:36:43 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 Feb 2014 23:36:43 -0500 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> Message-ID: On 2/17/2014 6:20 PM, Victor Stinner wrote: > 2014-02-17 0:25 GMT+01:00 Larry Hastings : >> You might think that anything you check in to the "default" branch in Python >> trunk will go into 3.4.0 rc2, and after that ships, checkins would go into >> 3.4.0 final. Ho ho ho! That's not true! Instead, anything checked in to >> "default" between my last revision for "rc1" (e64ae8b82672) and 3.4.0 final >> will by default go into 3.4.1. Only fixes that I cherry-pick into my local >> branch will go into 3.4.0 rc2 and final. And my local branch will remain >> private until 3.4.0 final ships! > > Does it mean that the default branch is open for changes which target 3.4.1? Presumably. That is what I started doing today. However, the top section of 3.4 News says 3.4.0rc2 rather than 3.4.1. It should be relabeled and a 3.4.0rc2 sections with items actually added to 3.4.0c2 (and those items should not be in the 3.4.1 section. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Tue Feb 18 06:11:18 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Feb 2014 18:11:18 +1300 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> Message-ID: <5302EB76.1050809@canterbury.ac.nz> Tim Peters wrote: > Guido wanted to drop all the "arbitrary but consistent" mixed-type > comparison crud for Python 3. Nobody is asking for a return to the arbitrary-but- [in]consistent mess of Python 2, only to bring back *one* special case, i.e. None comparing less than everything else. I think there is a reasonable argument to be made in favour of that. Like it or not, None does have a special place in Python as the one obvious way to represent a null or missing value, and often one wants to sort a collection of objects having keys that can take on null values. Refusing to make that easy seems like allowing purity to beat practicality. -- Greg From tim.peters at gmail.com Tue Feb 18 06:29:13 2014 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 17 Feb 2014 23:29:13 -0600 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5302EB76.1050809@canterbury.ac.nz> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> Message-ID: [Tim] >> Guido wanted to drop all the "arbitrary but consistent" mixed-type >> comparison crud for Python 3. [Greg Ewing] > Nobody is asking for a return to the arbitrary-but- > [in]consistent mess of Python 2, only to bring > back *one* special case, i.e. None comparing less > than everything else. Of course. My point was that dropping None comparisons wasn't specifically aimed at None comparisons. > I think there is a reasonable argument to be made > in favour of that. There may have been 6 years ago, but Python 3 isn't a thought experiment anymore. It was first released over 5 years ago, and has its own compatibility (with Python 3, not Python 2) constraints now. > Like it or not, None does have > a special place in Python as the one obvious way > to represent a null or missing value, and often > one wants to sort a collection of objects having > keys that can take on null values. Perhaps that's often true of your code, but it's never been true of mine. > Refusing to make that easy seems like allowing purity to beat > practicality. Since I've never had a need for it, I can't say for sure whether or not it's easy - but the .sort() interface is rich enough that I almost always find even truly complex key transformations "easy enough". Perhaps a real, concrete use case would help. I have a hard time imagining why I'd _want_ to sort a list of objects with "null or missing" keys, instead of filtering such garbage out of the list first and sorting what remains. But even with a compelling use case, I'd still think it's years too late to change this in Python 3. From zachary.ware at gmail.com Tue Feb 18 06:19:15 2014 From: zachary.ware at gmail.com (Zachary Ware) Date: Mon, 17 Feb 2014 21:19:15 -0800 Subject: [Python-Dev] [Python-checkins] cpython: Close #20656: Fix select.select() on OpenBSD 64-bit In-Reply-To: <3fSjmY758Hz7LjS@mail.python.org> References: <3fSjmY758Hz7LjS@mail.python.org> Message-ID: On Mon, Feb 17, 2014 at 6:36 PM, victor.stinner wrote: > http://hg.python.org/cpython/rev/79ccf36b0fd0 > changeset: 89239:79ccf36b0fd0 > user: Victor Stinner > date: Tue Feb 18 01:35:40 2014 +0100 > summary: > Close #20656: Fix select.select() on OpenBSD 64-bit > > files: > Modules/selectmodule.c | 22 ++++++++++++---------- > 1 files changed, 12 insertions(+), 10 deletions(-) This changeset caused a compile warning on 32-bit Windows: ..\Modules\selectmodule.c(238): warning C4244: '=' : conversion from 'time_t' to 'long', possible loss of data [P:\ath\to\cpython\PCbuild\select.vcxproj] > diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c > --- a/Modules/selectmodule.c > +++ b/Modules/selectmodule.c > @@ -212,11 +212,18 @@ > return NULL; > } > else { > -#ifdef MS_WINDOWS > + /* On OpenBSD 5.4, timeval.tv_sec is a long. > + * Example: long is 64-bit, whereas time_t is 32-bit. */ > time_t sec; > - if (_PyTime_ObjectToTimeval(tout, &sec, &tv.tv_usec, > + /* On OS X 64-bit, timeval.tv_usec is an int (and thus still 4 > + bytes as required), but no longer defined by a long. */ > + long usec; > + if (_PyTime_ObjectToTimeval(tout, &sec, &usec, > _PyTime_ROUND_UP) == -1) > return NULL; > +#ifdef MS_WINDOWS > + /* On Windows, timeval.tv_sec is a long (32 bit), > + * whereas time_t can be 64-bit. */ > assert(sizeof(tv.tv_sec) == sizeof(long)); > #if SIZEOF_TIME_T > SIZEOF_LONG > if (sec > LONG_MAX) { > @@ -225,16 +232,11 @@ > return NULL; > } > #endif > - tv.tv_sec = (long)sec; > #else > - /* 64-bit OS X has struct timeval.tv_usec as an int (and thus still 4 > - bytes as required), but no longer defined by a long. */ > - long tv_usec; > - if (_PyTime_ObjectToTimeval(tout, &tv.tv_sec, &tv_usec, > - _PyTime_ROUND_UP) == -1) > - return NULL; > - tv.tv_usec = tv_usec; > + assert(sizeof(tv.tv_sec) >= sizeof(sec)); > #endif > + tv.tv_sec = sec; This is the offending line. I'm not sure how best to fix it, so I'm just pointing it out :) -- Zach From greg.ewing at canterbury.ac.nz Tue Feb 18 06:32:19 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Feb 2014 18:32:19 +1300 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <53029A4D.30702@udel.edu> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301FFC6.6030500@egenix.com> <53029A4D.30702@udel.edu> Message-ID: <5302F063.3060603@canterbury.ac.nz> Terry Reedy wrote: > To make None a true bottom object, the rich comparison > methods would have to special-case None as either argument before > looking at the __rc__ special methods of either. I don't think it would be necessary to go that far. It would be sufficient to put the special case *after* giving both operations a chance to handle the operation via the type slots. Well-behaved objects generally wouldn't go out of their way to defeat that, but they could do so if necessary, e.g. to create a Bottom() that compares less than None. Although once None becomes usable as a bottom in most cases, there shouldn't be much need for people to introduce their own bottoms. -- Greg From greg.ewing at canterbury.ac.nz Tue Feb 18 08:35:52 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Feb 2014 20:35:52 +1300 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> Message-ID: <53030D58.4000307@canterbury.ac.nz> Tim Peters wrote: > > [Greg Ewing] > >>often >>one wants to sort a collection of objects having >>keys that can take on null values. > > Perhaps that's often true of your code, but it's never been true of mine. It's fairly common in accounting circles. I have a collection of invoices, each of which can have a due date specified, but doesn't have to. I want to sort the invoices by due date. It's not particularly important whether the missing dates sort first or last, but it *is* important that the sort *not blow up*. Dates seem to be a particularly irksome case. Here's one suggestion from StackOverflow for dealing with the problem: import datetime mindate = datetime.date(datetime.MINYEAR, 1, 1) def getaccountingdate(x): return x['accountingdate'] or mindate results = sorted(data, key=getaccountingdate) That's a ridiculous amount of boilerplate for doing something that ought to be straightforward. If you don't want to touch comparison in general, how about an option to sort()? results = sorted(invoices, key=attrgetter('duedate'), none='first') > I have a hard time > imagining why I'd _want_ to sort a list of objects with "null or > missing" keys, instead of filtering such garbage out of the list first A piece of data is *not* garbage just because an optional part of it is not specified. -- Greg From p.f.moore at gmail.com Tue Feb 18 09:10:17 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 18 Feb 2014 08:10:17 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <53030D58.4000307@canterbury.ac.nz> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <53030D58.4000307@canterbury.ac.nz> Message-ID: On 18 February 2014 07:35, Greg Ewing wrote: > If you don't want to touch comparison in general, > how about an option to sort()? > > results = sorted(invoices, key=attrgetter('duedate'), none='first') Or alternatively, a "default on None" function - Oracle SQL calls this nvl, so I will too: def nvl(x, dflt): return dflt if x is None else x results = sorted(invoices, key=lambda x: nvl(x.duedate, datetime(MINYEAR,1,1)) Admittedly the key function is starting to get complex, but I find that key functions often do - they are the one big area where Python uses a lot of functional-style constructs. The existence of itemgetter/attrgetter are a testament to this fact. PS isn't this python-ideas territory by now? Paul From victor.stinner at gmail.com Tue Feb 18 09:32:07 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 18 Feb 2014 09:32:07 +0100 Subject: [Python-Dev] [Python-checkins] cpython: Close #20656: Fix select.select() on OpenBSD 64-bit In-Reply-To: References: <3fSjmY758Hz7LjS@mail.python.org> Message-ID: Hi, 2014-02-18 6:19 GMT+01:00 Zachary Ware : > On Mon, Feb 17, 2014 at 6:36 PM, victor.stinner > wrote: >> http://hg.python.org/cpython/rev/79ccf36b0fd0 >> changeset: 89239:79ccf36b0fd0 >> user: Victor Stinner >> date: Tue Feb 18 01:35:40 2014 +0100 >> summary: >> Close #20656: Fix select.select() on OpenBSD 64-bit >> >> files: >> Modules/selectmodule.c | 22 ++++++++++++---------- >> 1 files changed, 12 insertions(+), 10 deletions(-) > > This changeset caused a compile warning on 32-bit Windows: > > ..\Modules\selectmodule.c(238): warning C4244: '=' : conversion from > 'time_t' to 'long', possible loss of data > [P:\ath\to\cpython\PCbuild\select.vcxproj] Ah yes, I didn't remember why I wrote the downcast initially. You're right, it is still neeed (on 64-bit Windows). I restored it: --- user: Victor Stinner date: Tue Feb 18 09:30:33 2014 +0100 files: Modules/selectmodule.c description: Issue #20656: Restore explicit downcast in select_select(). Cast from time_t (64 bit) to long (32 bit). It should fix a compiler warning. --- Victor From regebro at gmail.com Tue Feb 18 09:31:21 2014 From: regebro at gmail.com (Lennart Regebro) Date: Tue, 18 Feb 2014 09:31:21 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <53030D58.4000307@canterbury.ac.nz> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <53030D58.4000307@canterbury.ac.nz> Message-ID: On Tue, Feb 18, 2014 at 8:35 AM, Greg Ewing wrote: > If you don't want to touch comparison in general, > how about an option to sort()? > > results = sorted(invoices, key=attrgetter('duedate'), none='first') I think this is a much better option. //Lennart From g.brandl at gmx.net Tue Feb 18 09:39:58 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 18 Feb 2014 09:39:58 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <53030D58.4000307@canterbury.ac.nz> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <53030D58.4000307@canterbury.ac.nz> Message-ID: Am 18.02.2014 08:35, schrieb Greg Ewing: > Tim Peters wrote: >> >> [Greg Ewing] >> >>>often >>>one wants to sort a collection of objects having >>>keys that can take on null values. >> >> Perhaps that's often true of your code, but it's never been true of mine. > > It's fairly common in accounting circles. I have a > collection of invoices, each of which can have a due > date specified, but doesn't have to. I want to sort > the invoices by due date. It's not particularly > important whether the missing dates sort first or > last, but it *is* important that the sort *not blow > up*. > > Dates seem to be a particularly irksome case. Here's > one suggestion from StackOverflow for dealing with > the problem: > > import datetime > mindate = datetime.date(datetime.MINYEAR, 1, 1) > > def getaccountingdate(x): > return x['accountingdate'] or mindate > > results = sorted(data, key=getaccountingdate) > > That's a ridiculous amount of boilerplate for doing > something that ought to be straightforward. Seeing how you need a key function in any case for this sort to work, it's only the "or mindate" added, which I can't recognize as "ridiculous amount of boilerplate". Much more so since you can put the key function in a shared module and import it from there anywhere you need it. Georg From mal at egenix.com Tue Feb 18 11:08:44 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 18 Feb 2014 11:08:44 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <530276CC.5040408@egenix.com> Message-ID: <5303312C.9020201@egenix.com> On 18.02.2014 05:25, Tim Peters wrote: > [M.-A. Lemburg] >> Now, the choice to have None compare less than all other objects >> may have been arbitrary, but IMO it was a good, consistent and >> useful choice. > > Possibly useful for some apps, sure. Not for my apps. For example, > when I initialize an object attribute to None in Python 3, I _expect_ > I'll get an exception if I try to use that attribute in most contexts. > It makes no more sense to ask whether that attribute is, say, less > than 3, than it does to add 3 to it. The exception is most useful > then. More often than not, it was annoying to me that Python 2 > _didn't_ whine about trying to compare None. Yes, I see your point, but please also consider that None is used in database applications as natural mapping for SQL NULL and in other data processing applications to mean "no value". This is not garbage data, it's just missing information for a particular field and you can still happily process such data without having the information for these fields as values (indeed in some cases, the information whether a field has a value or not is important, e.g. for data validations and to cross check entries). You do still want to be able to sort such data, but as it stands, Python 3 doesn't allow you to, without adding an extra layer of protection against None values deep inside the structures you're sorting. >> So why not bring it back > > A huge barrier is (or should be) that Python 3 is over 5 years old > now. Fiddling with the semantics of basic builtin types was possible > - and even welcome - 6 years ago. Now they really shouldn't be > touched in the absence of a critical bug, or a wholly > backward-compatible (with Python 3) addition. > >> and perhaps this time in a way that actually does work consistently for >> all Python objects by implementing the tp_richcompare slot on >> PyNoneType objects and documenting it ?! > > Something to suggest for Python 4, in which case I'll only be -0 ;-) Well, like I said: we'd be making something work again that has worked before and only remove a failure case. The barrier for entry should be lower in such a case. This is similar to bringing back the u"" literals. Those used to cause a SyntaxError and now they no longer do. Even better: we have a chance to properly document the behavior this time around and make it consistent all the way. The alternative would be adding a new singleton to mean mostly the same thing as None, but having the property of comparing less than all other objects and then recommend its use in the DB-API for Python 3 applications... which would break a lot of existing DB-API based apps of course... which is the reason why I'm advocating for changing None instead :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 18 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-02-12: Released mxODBC.Connect 2.0.4 ... http://egenix.com/go53 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From storchaka at gmail.com Tue Feb 18 14:48:20 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 18 Feb 2014 15:48:20 +0200 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <53030D58.4000307@canterbury.ac.nz> Message-ID: 18.02.14 10:10, Paul Moore ???????(??): > Or alternatively, a "default on None" function - Oracle SQL calls this > nvl, so I will too: > > def nvl(x, dflt): > return dflt if x is None else x > > results = sorted(invoices, key=lambda x: nvl(x.duedate, datetime(MINYEAR,1,1)) Or, as was proposed above: results = sorted(invoices, key=lambda x: (x.duedate is not None, x.duedate)) From python at mrabarnett.plus.com Tue Feb 18 15:11:38 2014 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 18 Feb 2014 14:11:38 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <53030D58.4000307@canterbury.ac.nz> Message-ID: <53036A1A.3080901@mrabarnett.plus.com> On 2014-02-18 13:48, Serhiy Storchaka wrote: > 18.02.14 10:10, Paul Moore ???????(??): >> Or alternatively, a "default on None" function - Oracle SQL calls this >> nvl, so I will too: >> >> def nvl(x, dflt): >> return dflt if x is None else x >> >> results = sorted(invoices, key=lambda x: nvl(x.duedate, datetime(MINYEAR,1,1)) > > Or, as was proposed above: > > results = sorted(invoices, > key=lambda x: (x.duedate is not None, x.duedate)) > That makes me wonder. Why is: None < None unorderable and not False but: (None, ) < (None, ) orderable? From robert.kern at gmail.com Tue Feb 18 15:29:10 2014 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 18 Feb 2014 14:29:10 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <53036A1A.3080901@mrabarnett.plus.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <53030D58.4000307@canterbury.ac.nz> <53036A1A.3080901@mrabarnett.plus.com> Message-ID: On 2014-02-18 14:11, MRAB wrote: > On 2014-02-18 13:48, Serhiy Storchaka wrote: >> 18.02.14 10:10, Paul Moore ???????(??): >>> Or alternatively, a "default on None" function - Oracle SQL calls this >>> nvl, so I will too: >>> >>> def nvl(x, dflt): >>> return dflt if x is None else x >>> >>> results = sorted(invoices, key=lambda x: nvl(x.duedate, datetime(MINYEAR,1,1)) >> >> Or, as was proposed above: >> >> results = sorted(invoices, >> key=lambda x: (x.duedate is not None, x.duedate)) >> > That makes me wonder. > > Why is: > > None < None > > unorderable and not False but: > > (None, ) < (None, ) > > orderable? tuple's rich comparison uses PyObject_RichCompareBool(x, y, Py_EQ) to find the first pair of items that is unequal. Then it will test the order of any remaining elements. http://hg.python.org/cpython/file/79e5bb0d9b8e/Objects/tupleobject.c#l591 PyObject_RichCompareBool(x, y, Py_EQ) treats identical objects as equal. http://hg.python.org/cpython/file/79e5bb0d9b8e/Objects/object.c#l716 -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From tjreedy at udel.edu Tue Feb 18 16:45:52 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Feb 2014 10:45:52 -0500 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5302EB76.1050809@canterbury.ac.nz> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> Message-ID: On 2/18/2014 12:11 AM, Greg Ewing wrote: > Nobody is asking for a return to the arbitrary-but- > [in]consistent mess of Python 2, only to bring > back *one* special case, i.e. None comparing less > than everything else. For a < None, that is only the fallback rule if a does not handle the comparison. The result is a mess, including a possible inconsistency between direct comparison and cmp. See my previous posts. 'Bringing back' what was or an improved version would be a semantic change that could break code and would require a two-version deprecation period. -- Terry Jan Reedy From tjreedy at udel.edu Tue Feb 18 16:53:33 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Feb 2014 10:53:33 -0500 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <53030D58.4000307@canterbury.ac.nz> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <53030D58.4000307@canterbury.ac.nz> Message-ID: On 2/18/2014 2:35 AM, Greg Ewing wrote: > results = sorted(invoices, key=attrgetter('duedate'), none='first') I think this is the best idea on the thread. As a pure enhancement, it could be added in 3.5. The only tricky part of the implementation is maintaining stability of the sort. The obvious implementation of swapping Nones with objects at one end would break that. Instead, a scan listing the positions of Nones should be followed by a series of block moves of objects (pointers) between the Nones. It would still be O(n). -- Terry Jan Reedy From brett at python.org Tue Feb 18 16:57:23 2014 From: brett at python.org (Brett Cannon) Date: Tue, 18 Feb 2014 10:57:23 -0500 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5303312C.9020201@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <530276CC.5040408@egenix.com> <5303312C.9020201@egenix.com> Message-ID: I'll reply inline, but tl;dr: I'm with Tim on this. On Tue, Feb 18, 2014 at 5:08 AM, M.-A. Lemburg wrote: > On 18.02.2014 05:25, Tim Peters wrote: > > [M.-A. Lemburg] > >> Now, the choice to have None compare less than all other objects > >> may have been arbitrary, but IMO it was a good, consistent and > >> useful choice. > > > > Possibly useful for some apps, sure. Not for my apps. For example, > > when I initialize an object attribute to None in Python 3, I _expect_ > > I'll get an exception if I try to use that attribute in most contexts. > > It makes no more sense to ask whether that attribute is, say, less > > than 3, than it does to add 3 to it. The exception is most useful > > then. More often than not, it was annoying to me that Python 2 > > _didn't_ whine about trying to compare None. > > Yes, I see your point, but please also consider that None is > used in database applications as natural mapping for SQL NULL > and in other data processing applications to mean "no value". > Fine, but not everything uses a database. =) > > This is not garbage data, it's just missing information for > a particular field and you can still happily process such data > without having the information for these fields as values > (indeed in some cases, the information whether a field has a > value or not is important, e.g. for data validations and to cross > check entries). > > You do still want to be able to sort such data, but as it stands, > Python 3 doesn't allow you to, without adding an extra layer > of protection against None values deep inside the structures > you're sorting. > Which in my opinion is fine as the boilerplate is typically minimal. No one has said any solution is going to take even 10 lines of code to work around this. > > >> So why not bring it back > > > > A huge barrier is (or should be) that Python 3 is over 5 years old > > now. Fiddling with the semantics of basic builtin types was possible > > - and even welcome - 6 years ago. Now they really shouldn't be > > touched in the absence of a critical bug, or a wholly > > backward-compatible (with Python 3) addition. > > > >> and perhaps this time in a way that actually does work consistently for > >> all Python objects by implementing the tp_richcompare slot on > >> PyNoneType objects and documenting it ?! > > > > Something to suggest for Python 4, in which case I'll only be -0 ;-) > > Well, like I said: we'd be making something work again that has > worked before and only remove a failure case. The barrier for > entry should be lower in such a case. > But this is still a semantic change. Python 3 users could very well be relying on the fact that comparing against None raises an exception as a way to detect when erroneous data has leaked into some data structure. What is being suggested is a semantic change to a core data type within a major release. We waited until Python 3 to make comparisons against disparate types illegal for a reason. > > This is similar to bringing back the u"" literals. Those used > to cause a SyntaxError and now they no longer do. > I don't think they are similar at all. Allowing the u prefix on strings basically made something that was a typo or accidental hold-over from Python 2 just not throw an error. There was no semantic shift in what u meant in Python 3.2 vs. 3.3 which could break some code. Basically we just made some syntactic fluff not upset the parser which in no way changed the semantic meaning of the string literal it was attached to. Making None sort a specific way is beyond just adding some syntactic fluff. > > Even better: we have a chance to properly document the behavior > this time around and make it consistent all the way. > > The alternative would be adding a new singleton to mean mostly > the same thing as None, but having the property of comparing > less than all other objects and then recommend its use in the > DB-API for Python 3 applications... which would break a lot > of existing DB-API based apps of course... which is the reason > why I'm advocating for changing None instead :-) > Or DB-API stuff needs to be updated to use some None-that-sorts singleton if they want to use None to represent NULL but sort in a specific way. As Tim has said, 3.0 has been out for five years, so we are talking about breaking code for this over what can be viewed as a minor inconvenience for DB code. The current situation is not insurmountable in any way for those that want None to sort. I should also mention I view None as representing nothing, which includes not sharing a type with anything, which is why it can't be compared against any other type in Python 3. But this suggestion of having None sort as the lowest thing no matter what in way says None is implicitly every type when it comes to sorting which breaks that mental model of None representing the void of information by being void of value, but everything when it comes to its type for comparison purposes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Feb 18 16:59:21 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Feb 2014 10:59:21 -0500 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5302F063.3060603@canterbury.ac.nz> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301FFC6.6030500@egenix.com> <53029A4D.30702@udel.edu> <5302F063.3060603@canterbury.ac.nz> Message-ID: On 2/18/2014 12:32 AM, Greg Ewing wrote: > Terry Reedy wrote: >> To make None a true bottom object, the rich comparison methods would >> have to special-case None as either argument before looking at the >> __rc__ special methods of either. > > I don't think it would be necessary to go that far. It is if you want None to be 'a true bottom object'. If you think something less is sufficient, or even desirable, then yes, ... > It would be sufficient to put the special case *after* > giving both operations a chance to handle the operation > via the type slots. That would more or less reproduce the 2.x situation, except that a conflict between < and cmp would not be possible. > Well-behaved objects generally wouldn't go out of their > way to defeat that, but they could do so if necessary, > e.g. to create a Bottom() that compares less than None. > Although once None becomes usable as a bottom in most > cases, there shouldn't be much need for people to > introduce their own bottoms. -- Terry Jan Reedy From breamoreboy at yahoo.co.uk Tue Feb 18 17:10:52 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Tue, 18 Feb 2014 16:10:52 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> Message-ID: On 18/02/2014 15:45, Terry Reedy wrote: > On 2/18/2014 12:11 AM, Greg Ewing wrote: > >> Nobody is asking for a return to the arbitrary-but- >> [in]consistent mess of Python 2, only to bring >> back *one* special case, i.e. None comparing less >> than everything else. > > For a < None, that is only the fallback rule if a does not handle the > comparison. The result is a mess, including a possible inconsistency > between direct comparison and cmp. See my previous posts. > > 'Bringing back' what was or an improved version would be a semantic > change that could break code and would require a two-version deprecation > period. > Sorry if this has already been suggested, but why not introduce a new singleton to make the database people happier if not happy? To avoid confusion call it dbnull? A reasonable compromise or complete cobblers? :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From oscar.j.benjamin at gmail.com Tue Feb 18 17:12:24 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 18 Feb 2014 16:12:24 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <53030D58.4000307@canterbury.ac.nz> Message-ID: On 18 February 2014 15:53, Terry Reedy wrote: > On 2/18/2014 2:35 AM, Greg Ewing wrote: > >> results = sorted(invoices, key=attrgetter('duedate'), none='first') > > I think this is the best idea on the thread. As a pure enhancement, it could > be added in 3.5. The only tricky part of the implementation is maintaining > stability of the sort. The obvious implementation of swapping Nones with > objects at one end would break that. Instead, a scan listing the positions > of Nones should be followed by a series of block moves of objects (pointers) > between the Nones. It would still be O(n). This only works if the list entry is a simple None. If the None is embedded in e.g. a tuple then it would fail: >>> (1, 2, 3) < (1, 2, None) Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: int() < NoneType() Oscar From ethan at stoneleaf.us Tue Feb 18 18:20:15 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 18 Feb 2014 09:20:15 -0800 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5303312C.9020201@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <530276CC.5040408@egenix.com> <5303312C.9020201@egenix.com> Message-ID: <5303964F.5030203@stoneleaf.us> On 02/18/2014 02:08 AM, M.-A. Lemburg wrote: > > This is not garbage data, it's just missing information for > a particular field [...] I think the basic problem is that None was over-used (or even mis-used?) to the point where 3.0 had to make a choice to bring consistency back to the language. It seems to me what we really need is either a Null singleton to represent a data point with no known value but that can still be sorted, or have every data type able to represent a no-value state (datetime, I'm looking at you!). -- ~Ethan~ From ethan at stoneleaf.us Tue Feb 18 18:11:35 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 18 Feb 2014 09:11:35 -0800 Subject: [Python-Dev] Pickling of Enums In-Reply-To: References: Message-ID: <53039447.70705@stoneleaf.us> On 02/15/2014 11:01 AM, Serhiy Storchaka wrote: > How Enum items should be pickled, by value or by name? > > I think that Enum will be used to collect system-depending constants, so the value of AddressFamily.AF_UNIX can be 1 on > one platform and 2 on other. If pickle enums by value, then pickled AddressFamily.AF_INET on on platform can be > unpickled as AddressFamily.AF_UNIX on other platform. This looks weird and contrary to the nature of enums. There is one more wrinkle to pickling by name (it's actually still there in pickle by value, just more obvious in pickle by name) -- aliases. It seems to me the most common scenario to having a name represent different values on different systems is when on system A they are different, but on system B they are the same: System A: class SystemEnum(Enum): value1 = 1 value2 = 2 System B: class SystemEnum(Enum): value1 = 1 value2 = 1 If you're on system B there is no way to pickle (by name or value) value2 such that we get value2 back on system A. The only way I know of to make that work would be to dispense with identity comparison, use the normal == comparison, and have aliases actually be separate objects (we could still use singletons, but it would be one per name instead of the current one per value, and it would also be an implementation detail). Thoughts? -- ~Ethan~ From guido at python.org Tue Feb 18 18:47:43 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Feb 2014 09:47:43 -0800 Subject: [Python-Dev] Pickling of Enums In-Reply-To: <53039447.70705@stoneleaf.us> References: <53039447.70705@stoneleaf.us> Message-ID: I'm confused. Hasn't this all been decided by the PEP long ago? On Tue, Feb 18, 2014 at 9:11 AM, Ethan Furman wrote: > On 02/15/2014 11:01 AM, Serhiy Storchaka wrote: > >> How Enum items should be pickled, by value or by name? >> >> I think that Enum will be used to collect system-depending constants, so >> the value of AddressFamily.AF_UNIX can be 1 on >> one platform and 2 on other. If pickle enums by value, then pickled >> AddressFamily.AF_INET on on platform can be >> unpickled as AddressFamily.AF_UNIX on other platform. This looks weird >> and contrary to the nature of enums. >> > > There is one more wrinkle to pickling by name (it's actually still there > in pickle by value, just more obvious in pickle by name) -- aliases. It > seems to me the most common scenario to having a name represent different > values on different systems is when on system A they are different, but on > system B they are the same: > > System A: > > class SystemEnum(Enum): > value1 = 1 > value2 = 2 > > System B: > > class SystemEnum(Enum): > value1 = 1 > value2 = 1 > > If you're on system B there is no way to pickle (by name or value) value2 > such that we get value2 back on system A. The only way I know of to make > that work would be to dispense with identity comparison, use the normal == > comparison, and have aliases actually be separate objects (we could still > use singletons, but it would be one per name instead of the current one per > value, and it would also be an implementation detail). > > Thoughts? > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Feb 18 19:01:42 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 18 Feb 2014 10:01:42 -0800 Subject: [Python-Dev] Pickling of Enums In-Reply-To: References: <53039447.70705@stoneleaf.us> Message-ID: <5303A006.6010201@stoneleaf.us> On 02/18/2014 09:47 AM, Guido van Rossum wrote: > > I'm confused. Hasn't this all been decided by the PEP long ago? The PEP only mentions pickling briefly, as in "the normal rules apply". How pickling occurs is an implementation detail, and it turns out that pickling by name is more robust. Serhiy, as part of his argument for using the _name_ instead of the _value_ for pickling, brought up the point that different systems could have different values for the same name. If true in practice (and I believe it is) this raises the issue of aliases, which currently *cannot* be pickled by name because there is no distinct object for the alias. If you ask for Color['alias_for_red'] you'll get Color.red instead. Using identity comparison was part of the PEP. I guess the question is which is more important? Identity comparison or this (probably) rare use-case? If we stick with identity I'm not aware of any work-around for pickling enum members that are aliases on one system, but distinct on another. I've been talking about pickling specifically, but this applies to any serialization method. -- ~Ethan~ From solipsis at pitrou.net Tue Feb 18 19:05:52 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 18 Feb 2014 19:05:52 +0100 Subject: [Python-Dev] Pickling of Enums References: <53039447.70705@stoneleaf.us> <5303A006.6010201@stoneleaf.us> Message-ID: <20140218190552.2fe9e7b4@fsol> On Tue, 18 Feb 2014 10:01:42 -0800 Ethan Furman wrote: > > I guess the question is which is more important? Identity comparison or this (probably) rare use-case? If we stick > with identity I'm not aware of any work-around for pickling enum members that are aliases on one system, but distinct on > another. I don't think identity comparison is important. Enum values are supposed to act like values, not full-blown objects. OTOH, the "pickled aliases may end up different on other systems" issue is sufficiently fringy that we may simply paper over it. Regards Antoine. From guido at python.org Tue Feb 18 19:05:51 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Feb 2014 10:05:51 -0800 Subject: [Python-Dev] Pickling of Enums In-Reply-To: <5303A006.6010201@stoneleaf.us> References: <53039447.70705@stoneleaf.us> <5303A006.6010201@stoneleaf.us> Message-ID: Hm. But there's an implementation that has made it unscathed through several betas and an RC. AFAICT that beta pickles enums by value. And I happen to think that that is the better choice (but I don't have time to explain this gut feeling until after 3.4 has been released). On Tue, Feb 18, 2014 at 10:01 AM, Ethan Furman wrote: > On 02/18/2014 09:47 AM, Guido van Rossum wrote: > >> >> I'm confused. Hasn't this all been decided by the PEP long ago? >> > > The PEP only mentions pickling briefly, as in "the normal rules apply". > How pickling occurs is an implementation detail, and it turns out that > pickling by name is more robust. > > Serhiy, as part of his argument for using the _name_ instead of the > _value_ for pickling, brought up the point that different systems could > have different values for the same name. If true in practice (and I > believe it is) this raises the issue of aliases, which currently *cannot* > be pickled by name because there is no distinct object for the alias. If > you ask for Color['alias_for_red'] you'll get Color.red instead. > > Using identity comparison was part of the PEP. > > I guess the question is which is more important? Identity comparison or > this (probably) rare use-case? If we stick with identity I'm not aware of > any work-around for pickling enum members that are aliases on one system, > but distinct on another. > > I've been talking about pickling specifically, but this applies to any > serialization method. > > > -- > ~Ethan~ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Feb 18 19:21:01 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 18 Feb 2014 20:21:01 +0200 Subject: [Python-Dev] Pickling of Enums In-Reply-To: <53039447.70705@stoneleaf.us> References: <53039447.70705@stoneleaf.us> Message-ID: 18.02.14 19:11, Ethan Furman ???????(??): > There is one more wrinkle to pickling by name (it's actually still there > in pickle by value, just more obvious in pickle by name) -- aliases. It > seems to me the most common scenario to having a name represent > different values on different systems is when on system A they are > different, but on system B they are the same: > > System A: > > class SystemEnum(Enum): > value1 = 1 > value2 = 2 > > System B: > > class SystemEnum(Enum): > value1 = 1 > value2 = 1 > > If you're on system B there is no way to pickle (by name or value) > value2 such that we get value2 back on system A. The only way I know of > to make that work would be to dispense with identity comparison, use the > normal == comparison, and have aliases actually be separate objects (we > could still use singletons, but it would be one per name instead of the > current one per value, and it would also be an implementation detail). > > Thoughts? There are aliases and aliases. If there are modern name and deprecated name, then it should be one object referred by different names on all systems. If there are different entities with accidentally equal values, then they should be different objects. From ethan at stoneleaf.us Tue Feb 18 19:16:17 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 18 Feb 2014 10:16:17 -0800 Subject: [Python-Dev] Pickling of Enums In-Reply-To: References: <53039447.70705@stoneleaf.us> <5303A006.6010201@stoneleaf.us> Message-ID: <5303A371.7080109@stoneleaf.us> On 02/18/2014 10:05 AM, Guido van Rossum wrote: > Hm. But there's an implementation that has made it unscathed through several betas and an RC. AFAICT that beta pickles > enums by value. And I happen to think that that is the better choice (but I don't have time to explain this gut feeling > until after 3.4 has been released). This conversation wasn't in the PEP, but as I recall we decided to go with value instead of name for json because the receiving end may not be running Python. Is having json do it one way and pickle another a problem? -- ~Ethan~ From storchaka at gmail.com Tue Feb 18 19:40:02 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 18 Feb 2014 20:40:02 +0200 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5303964F.5030203@stoneleaf.us> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <530276CC.5040408@egenix.com> <5303312C.9020201@egenix.com> <5303964F.5030203@stoneleaf.us> Message-ID: 18.02.14 19:20, Ethan Furman ???????(??): > It seems to me what we really need is either a Null singleton to > represent a data point with no known value but that can still be sorted, > or have every data type able to represent a no-value state (datetime, > I'm looking at you!). A Never singleton? From guido at python.org Tue Feb 18 20:20:44 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Feb 2014 11:20:44 -0800 Subject: [Python-Dev] Pickling of Enums In-Reply-To: <5303A371.7080109@stoneleaf.us> References: <53039447.70705@stoneleaf.us> <5303A006.6010201@stoneleaf.us> <5303A371.7080109@stoneleaf.us> Message-ID: I'm confused. AFAICT enums are pickled by value too. What am I missing? Are we confused about terminology or about behavior? (I'm just guessing that the pickling happens by value because I don't see the string AF_INET.) $ python3 Python 3.4.0rc1+ (default:2ba583191550, Feb 11 2014, 16:05:24) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import socket, pickle, json, pickletools >>> socket.AF_INET >>> pickle.dumps(socket.AF_INET) b'\x80\x03csocket\nAddressFamily\nq\x00K\x02\x85q\x01Rq\x02.' >>> json.dumps(socket.AF_INET) '2' >>> pickletools.dis(pickle.dumps(socket.AF_INET)) 0: \x80 PROTO 3 2: c GLOBAL 'socket AddressFamily' 24: q BINPUT 0 26: K BININT1 2 28: \x85 TUPLE1 29: q BINPUT 1 31: R REDUCE 32: q BINPUT 2 34: . STOP highest protocol among opcodes = 2 >>> On Tue, Feb 18, 2014 at 10:16 AM, Ethan Furman wrote: > On 02/18/2014 10:05 AM, Guido van Rossum wrote: > >> Hm. But there's an implementation that has made it unscathed through >> several betas and an RC. AFAICT that beta pickles >> enums by value. And I happen to think that that is the better choice (but >> I don't have time to explain this gut feeling >> until after 3.4 has been released). >> > > This conversation wasn't in the PEP, but as I recall we decided to go with > value instead of name for json because the receiving end may not be running > Python. > > Is having json do it one way and pickle another a problem? > > -- > ~Ethan~ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Feb 18 20:37:15 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Feb 2014 11:37:15 -0800 Subject: [Python-Dev] Pickling of Enums In-Reply-To: <5303B3E5.50202@stoneleaf.us> References: <53039447.70705@stoneleaf.us> <5303A006.6010201@stoneleaf.us> <5303A371.7080109@stoneleaf.us> <5303B3E5.50202@stoneleaf.us> Message-ID: Well, I'm against that. On Tue, Feb 18, 2014 at 11:26 AM, Ethan Furman wrote: > On 02/18/2014 11:20 AM, Guido van Rossum wrote: > >> >> I'm confused. AFAICT enums are pickled by value too. What am I missing? >> Are we confused about terminology or about >> behavior? (I'm just guessing that the pickling happens by value because I >> don't see the string AF_INET.) >> > > There's an open issue [1] to switch to pickling by name. > > -- > ~Ethan~ > > [1] http://bugs.python.org/issue20653 > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Feb 18 20:46:27 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 18 Feb 2014 21:46:27 +0200 Subject: [Python-Dev] Pickling of Enums In-Reply-To: <5303A371.7080109@stoneleaf.us> References: <53039447.70705@stoneleaf.us> <5303A006.6010201@stoneleaf.us> <5303A371.7080109@stoneleaf.us> Message-ID: 18.02.14 20:16, Ethan Furman ???????(??): > This conversation wasn't in the PEP, but as I recall we decided to go > with value instead of name for json because the receiving end may not be > running Python. > > Is having json do it one way and pickle another a problem? We decided to go with value instead of name for JSON because JSON doesn't support enums, but supports integers and strings, and because enums are comparable with they values, but not with they names. >>> json.loads(json.dumps(socket.AF_INET)) == socket.AF_INET True We simply had no other choice. From ethan at stoneleaf.us Tue Feb 18 20:26:29 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 18 Feb 2014 11:26:29 -0800 Subject: [Python-Dev] Pickling of Enums In-Reply-To: References: <53039447.70705@stoneleaf.us> <5303A006.6010201@stoneleaf.us> <5303A371.7080109@stoneleaf.us> Message-ID: <5303B3E5.50202@stoneleaf.us> On 02/18/2014 11:20 AM, Guido van Rossum wrote: > > I'm confused. AFAICT enums are pickled by value too. What am I missing? Are we confused about terminology or about > behavior? (I'm just guessing that the pickling happens by value because I don't see the string AF_INET.) There's an open issue [1] to switch to pickling by name. -- ~Ethan~ [1] http://bugs.python.org/issue20653 From storchaka at gmail.com Tue Feb 18 20:53:23 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 18 Feb 2014 21:53:23 +0200 Subject: [Python-Dev] Pickling of Enums In-Reply-To: References: <53039447.70705@stoneleaf.us> <5303A006.6010201@stoneleaf.us> <5303A371.7080109@stoneleaf.us> Message-ID: 18.02.14 21:20, Guido van Rossum ???????(??): > I'm confused. AFAICT enums are pickled by value too. What am I missing? > Are we confused about terminology or about behavior? (I'm just guessing > that the pickling happens by value because I don't see the string AF_INET.) Pickling was not even working two weeks ago. [1] [1] http://bugs.python.org/issue20534 From guido at python.org Tue Feb 18 20:55:58 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Feb 2014 11:55:58 -0800 Subject: [Python-Dev] Pickling of Enums In-Reply-To: References: <53039447.70705@stoneleaf.us> <5303A006.6010201@stoneleaf.us> <5303A371.7080109@stoneleaf.us> Message-ID: Well, I still think it should be done by value. On Tue, Feb 18, 2014 at 11:53 AM, Serhiy Storchaka wrote: > 18.02.14 21:20, Guido van Rossum ???????(??): > > I'm confused. AFAICT enums are pickled by value too. What am I missing? >> Are we confused about terminology or about behavior? (I'm just guessing >> that the pickling happens by value because I don't see the string >> AF_INET.) >> > > Pickling was not even working two weeks ago. [1] > > [1] http://bugs.python.org/issue20534 > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Tue Feb 18 21:03:08 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 18 Feb 2014 12:03:08 -0800 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <530276CC.5040408@egenix.com> <5303312C.9020201@egenix.com> Message-ID: <5303BC7C.1020905@g.nevcal.com> On 2/18/2014 7:57 AM, Brett Cannon wrote: > > Yes, I see your point, but please also consider that None is > used in database applications as natural mapping for SQL NULL > and in other data processing applications to mean "no value". > > > Fine, but not everything uses a database. =) Python None and SQL NULL share some properties, which makes it seem like it is a natural mapping. However, they also have some differing properties, which means that really it isn't a natural mapping. A big mistake in most Python/SQL marriages is the failure to implement an SQL NULL class that fully matches SQL semantics. Of course it is easier to map SQL NULL to Python None and then complain about the semantic differences. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lele at metapensiero.it Tue Feb 18 21:48:06 2014 From: lele at metapensiero.it (Lele Gaifax) Date: Tue, 18 Feb 2014 21:48:06 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <530276CC.5040408@egenix.com> <5303312C.9020201@egenix.com> Message-ID: <87wqgsf8qh.fsf@nautilus.nautilus> Brett Cannon writes: >> Yes, I see your point, but please also consider that None is >> used in database applications as natural mapping for SQL NULL >> and in other data processing applications to mean "no value". >> > > Fine, but not everything uses a database. =) Right, and even if everything did, as already said not all databases share the same default ordering rules, or have the same syntax to impose one specific direction. But more to the point, no database I know let you say "WHERE somecolumn < 1" and expect that when "somecolumn IS NULL" the condition is true... Paradoxically, if Python goes back to give a meaning to "None < 1", it would have an even more different semantic than most database engines out there. ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From tjreedy at udel.edu Tue Feb 18 22:09:28 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Feb 2014 16:09:28 -0500 Subject: [Python-Dev] Treating tokenize.Untokenizer as private Message-ID: I am working through the multiple bugs afflicting tokenize.untokenize, which is described in the tokenize doc and has an even longer docstring. While the function could be implemented as one 70-line function, it happens to be implemented as a 4-line wrapper for a completely undocumented (Untokenizer class with 4 methods. (It is unmentioned in the doc and there are currently no docstrings.) I view the class as a private implementation detail and would like to treat it as such, and perhaps even rename it _Untokenizer to make that clear. The issue arises in #9974. It appears that a fix may require the addition of an instance attribute or .add_whitespace parameter. If there is objection to treating the whole class as private, I would at least like to treat add_whitespace as the private helper that it is. There is no reason to call it directly except for testing. Otherwise, it could just as well have been left inline at the one call site. -- Terry Jan Reedy From rosuav at gmail.com Tue Feb 18 22:11:40 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 19 Feb 2014 08:11:40 +1100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> Message-ID: On Wed, Feb 19, 2014 at 3:10 AM, Mark Lawrence wrote: > Sorry if this has already been suggested, but why not introduce a new > singleton to make the database people happier if not happy? To avoid > confusion call it dbnull? A reasonable compromise or complete cobblers? :) That would be a major change to the DB API. Possibly the best solution, though. Start off by having the default be to return None (as now) and have a flag that can be set "please return sys.dbnull instead" (does dbnull belong in sys?), and let that settle for a few years. Recommend that all applications explicitly set the flag, either to True or to False. Then eventually, with the full deprecation warnings, change the default to True. (Or maybe make it an error to not set it.) Then, after another long round of deprecations, drop the None behaviour from the spec altogether. But even in its early steps, that could solve the problem. Anyone who wants to sort a list of tuples that came from the database can simply set the flag (assuming their back-end supports that flag) and happily use dbnull. ChrisA From greg.ewing at canterbury.ac.nz Tue Feb 18 22:17:00 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Feb 2014 10:17:00 +1300 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <53030D58.4000307@canterbury.ac.nz> Message-ID: <5303CDCC.3040604@canterbury.ac.nz> Georg Brandl wrote: > Seeing how you need a key function in any case for this sort to work, > it's only the "or mindate" added, which I can't recognize as "ridiculous > amount of boilerplate". Well, I don't much like having to construct a key function in the first place for something as common as sorting on an attribute. Also, in the particular case of dates, there's the annoying fact that the datetime module doesn't provide any kind of null object that can be compared with datetimes, so you have to make a fake one yourself. All of these small annoyances add up to what is, for me, a fairly big annoyance. What I'd *really* like to be able to write is: sort(invoices, keyattr = 'duedate', none = 'first') -- Greg From p.f.moore at gmail.com Tue Feb 18 22:24:40 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 18 Feb 2014 21:24:40 +0000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5303CDCC.3040604@canterbury.ac.nz> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <53030D58.4000307@canterbury.ac.nz> <5303CDCC.3040604@canterbury.ac.nz> Message-ID: On 18 February 2014 21:17, Greg Ewing wrote: > What I'd *really* like to be able to write is: > > sort(invoices, keyattr = 'duedate', none = 'first') Which is of course a pretty trivial utility function to write. But I understand your point - these "trivial helpers" add up over time into a bit of a burden. But fixing that (if anyone feels it's worth doing so) can be handled many ways, and changing the semantics of None is far from the simplest or most obvious. Pul From greg.ewing at canterbury.ac.nz Tue Feb 18 22:27:15 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Feb 2014 10:27:15 +1300 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5303312C.9020201@egenix.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <530276CC.5040408@egenix.com> <5303312C.9020201@egenix.com> Message-ID: <5303D033.7070505@canterbury.ac.nz> M.-A. Lemburg wrote: > The alternative would be adding a new singleton to mean mostly > the same thing as None, but having the property of comparing > less than all other objects and then recommend its use in the > DB-API for Python 3 applications... Which I think would be a *really bad* idea, because there would then no longer be One Obvious Way to represent and process null values. E.g. you would no longer be able to write 'value is None' and trust it to work on all kinds of null value you're likely to get. -- Greg From ethan at stoneleaf.us Tue Feb 18 21:48:33 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 18 Feb 2014 12:48:33 -0800 Subject: [Python-Dev] Pickling of Enums In-Reply-To: References: <53039447.70705@stoneleaf.us> <5303A006.6010201@stoneleaf.us> <5303A371.7080109@stoneleaf.us> <5303B3E5.50202@stoneleaf.us> Message-ID: <5303C721.6050207@stoneleaf.us> On 02/18/2014 11:37 AM, Guido van Rossum wrote: > > Well, I'm against that. Given the lack of a tidal wave of support for the idea, I'll let it die with that. Still, many thanks to Serhiy for greatly improving the way pickling is implemented for Enums, even using values. -- ~Ethan~ From ethan at stoneleaf.us Tue Feb 18 21:55:26 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 18 Feb 2014 12:55:26 -0800 Subject: [Python-Dev] Pickling of Enums In-Reply-To: References: <53039447.70705@stoneleaf.us> <5303A006.6010201@stoneleaf.us> <5303A371.7080109@stoneleaf.us> Message-ID: <5303C8BE.5010609@stoneleaf.us> On 02/18/2014 11:53 AM, Serhiy Storchaka wrote: > 18.02.14 21:20, Guido van Rossum ???????(??): >> I'm confused. AFAICT enums are pickled by value too. What am I missing? >> Are we confused about terminology or about behavior? (I'm just guessing >> that the pickling happens by value because I don't see the string AF_INET.) > > Pickling was not even working two weeks ago. [1] For the record, pickling worked just fine for protocols 2 and 3, and 4 didn't exist at the time. -- ~Ethan~ From ncoghlan at gmail.com Tue Feb 18 23:27:03 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Feb 2014 08:27:03 +1000 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <53030D58.4000307@canterbury.ac.nz> <5303CDCC.3040604@canterbury.ac.nz> Message-ID: On 19 Feb 2014 07:26, "Paul Moore" wrote: > > On 18 February 2014 21:17, Greg Ewing wrote: > > What I'd *really* like to be able to write is: > > > > sort(invoices, keyattr = 'duedate', none = 'first') > > Which is of course a pretty trivial utility function to write. But I > understand your point - these "trivial helpers" add up over time into > a bit of a burden. But fixing that (if anyone feels it's worth doing > so) can be handled many ways, and changing the semantics of None is > far from the simplest or most obvious. So perhaps the real answer is for someone to write a sorting helper module and put it on PyPI. Things like sort_byattr (implicitly uses attrgetter), sort_byitem (implicitly uses itemgetter), a flag argument defaulting to "none_first=True", SortsLow and SortsHigh singletons, etc. Those don't need to be builtins, but collating them into a clean helper API would not only allow that module to be used directly, but also act as a reference for anyone wanting to implement the behaviour themselves. (I know, I know, we're way into python-ideas territory - it's just that so much if the thread *was* on topic for python-dev, it seems relatively pointless to split this iteration of the discussion. The *next* thread about it should definitely be on python-ideas, though) Cheers, Nick. > > Pul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Tue Feb 18 23:34:41 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 18 Feb 2014 14:34:41 -0800 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5303D033.7070505@canterbury.ac.nz> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <530276CC.5040408@egenix.com> <5303312C.9020201@egenix.com> <5303D033.7070505@canterbury.ac.nz> Message-ID: <5303E001.8010507@g.nevcal.com> On 2/18/2014 1:27 PM, Greg Ewing wrote: > M.-A. Lemburg wrote: >> The alternative would be adding a new singleton to mean mostly >> the same thing as None, but having the property of comparing >> less than all other objects and then recommend its use in the >> DB-API for Python 3 applications... > > Which I think would be a *really bad* idea, because > there would then no longer be One Obvious Way to > represent and process null values. > > E.g. you would no longer be able to write > 'value is None' and trust it to work on all kinds > of null value you're likely to get. Of course you couldn't... value is None would never work full Null values, only for None, like it should. -------------- next part -------------- An HTML attachment was scrubbed... URL: From doko at ubuntu.com Wed Feb 19 00:38:15 2014 From: doko at ubuntu.com (Matthias Klose) Date: Wed, 19 Feb 2014 00:38:15 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <53014900.7090908@hastings.org> References: <53014900.7090908@hastings.org> Message-ID: <5303EEE7.8000101@ubuntu.com> Am 17.02.2014 00:25, schrieb Larry Hastings: > And my local branch will remain private until 3.4.0 final ships! sorry, but this is so wrong. Is there *any* reason why to keep this branch private? Matthias From larry at hastings.org Wed Feb 19 00:46:05 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 18 Feb 2014 15:46:05 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <5303EEE7.8000101@ubuntu.com> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> Message-ID: <5303F0BD.6080203@hastings.org> On 02/18/2014 03:38 PM, Matthias Klose wrote: > Am 17.02.2014 00:25, schrieb Larry Hastings: >> And my local branch will remain private until 3.4.0 final ships! > sorry, but this is so wrong. Is there *any* reason why to keep this branch private? Yes. It ensures that nobody can check something into it against my wishes. Also, in the event that I cherry-pick revisions out-of-order, it allows me to rebase, making merging easier. Is there *any* reason to make this branch public before 3.4.0 final? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Feb 19 00:54:32 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 19 Feb 2014 00:54:32 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <5303F0BD.6080203@hastings.org> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <5303F0BD.6080203@hastings.org> Message-ID: 2014-02-19 0:46 GMT+01:00 Larry Hastings : > Is there *any* reason to make this branch public before 3.4.0 final? I'm a little bit worried by the fact that buildbots will not test it. Cherry-picking many patches is complex. It's safe if you have a very short list of changes. Would it be insane to use default as the next RC2? It looks like there are too many changes between RC1 and RC2. Another release candidate is maybe needed. Victor From doko at ubuntu.com Wed Feb 19 00:56:53 2014 From: doko at ubuntu.com (Matthias Klose) Date: Wed, 19 Feb 2014 00:56:53 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <5303F0BD.6080203@hastings.org> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <5303F0BD.6080203@hastings.org> Message-ID: <5303F345.9060108@ubuntu.com> Am 19.02.2014 00:46, schrieb Larry Hastings: > On 02/18/2014 03:38 PM, Matthias Klose wrote: >> Am 17.02.2014 00:25, schrieb Larry Hastings: >>> And my local branch will remain private until 3.4.0 final ships! >> sorry, but this is so wrong. Is there *any* reason why to keep this branch >> private? > > Yes. It ensures that nobody can check something into it against my wishes. > Also, in the event that I cherry-pick revisions out-of-order, it allows me to > rebase, making merging easier. > > Is there *any* reason to make this branch public before 3.4.0 final? - Python is an open source project. Why do we need to hide development for a month or more? - Not even four eyes looking at the code seems to be odd. You can make mistakes too. This seems to be a social or a technical problem. I assume making this branch available read-only would address your concerns? Does hg allow this? And if not, why not create this branch in the upstream repository, and tell people not to commit to it? Why shouldn't such a social restriction work? Seems to work well for other projects. Matthias From victor.stinner at gmail.com Wed Feb 19 00:58:28 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 19 Feb 2014 00:58:28 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <53014900.7090908@hastings.org> References: <53014900.7090908@hastings.org> Message-ID: 2014-02-17 0:25 GMT+01:00 Larry Hastings : > You might think that anything you check in to the "default" branch in Python > trunk will go into 3.4.0 rc2, and after that ships, checkins would go into > 3.4.0 final. Ho ho ho! That's not true! Instead, anything checked in to > "default" between my last revision for "rc1" (e64ae8b82672) and 3.4.0 final > will by default go into 3.4.1. Only fixes that I cherry-pick into my local > branch will go into 3.4.0 rc2 and final. And my local branch will remain > private until 3.4.0 final ships! I'm a little bit lost with the Misc/NEWS file. The current title is still "What's New in Python 3.4.0 release candidate 2?". Should we start a new "Python 3.4.1" section? Victor From larry at hastings.org Wed Feb 19 01:05:01 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 18 Feb 2014 16:05:01 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <5303F345.9060108@ubuntu.com> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <5303F0BD.6080203@hastings.org> <5303F345.9060108@ubuntu.com> Message-ID: <5303F52D.4090108@hastings.org> On 02/18/2014 03:56 PM, Matthias Klose wrote: > Am 19.02.2014 00:46, schrieb Larry Hastings: >> On 02/18/2014 03:38 PM, Matthias Klose wrote: >>> Am 17.02.2014 00:25, schrieb Larry Hastings: >>>> And my local branch will remain private until 3.4.0 final ships! >>> sorry, but this is so wrong. Is there *any* reason why to keep this branch >>> private? >> Yes. It ensures that nobody can check something into it against my wishes. >> Also, in the event that I cherry-pick revisions out-of-order, it allows me to >> rebase, making merging easier. >> >> Is there *any* reason to make this branch public before 3.4.0 final? > - Python is an open source project. Why do we need to hide > development for a month or more? > > - Not even four eyes looking at the code seems to be odd. You > can make mistakes too. > > This seems to be a social or a technical problem. I assume making this branch > available read-only would address your concerns? Does hg allow this? And if > not, why not create this branch in the upstream repository, and tell people not > to commit to it? Why shouldn't such a social restriction work? Seems to work > well for other projects. When you are release manager for Python, you may institute this policy if you like. Right now, I have enough to do as it is. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed Feb 19 01:07:38 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 18 Feb 2014 16:07:38 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <5303F0BD.6080203@hastings.org> Message-ID: <5303F5CA.3010001@hastings.org> On 02/18/2014 03:54 PM, Victor Stinner wrote: > 2014-02-19 0:46 GMT+01:00 Larry Hastings : >> Is there *any* reason to make this branch public before 3.4.0 final? > I'm a little bit worried by the fact that buildbots will not test it. "fact"? http://docs.python.org/devguide/buildbots.html#custom-builders > Would it be insane to use default as the next RC2? Yes. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From doko at ubuntu.com Wed Feb 19 01:19:31 2014 From: doko at ubuntu.com (Matthias Klose) Date: Wed, 19 Feb 2014 01:19:31 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <5303F52D.4090108@hastings.org> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <5303F0BD.6080203@hastings.org> <5303F345.9060108@ubuntu.com> <5303F52D.4090108@hastings.org> Message-ID: <5303F893.4030909@ubuntu.com> Am 19.02.2014 01:05, schrieb Larry Hastings: > On 02/18/2014 03:56 PM, Matthias Klose wrote: >> Am 19.02.2014 00:46, schrieb Larry Hastings: >>> On 02/18/2014 03:38 PM, Matthias Klose wrote: >>>> Am 17.02.2014 00:25, schrieb Larry Hastings: >>>>> And my local branch will remain private until 3.4.0 final ships! >>>> sorry, but this is so wrong. Is there *any* reason why to keep this branch >>>> private? >>> Yes. It ensures that nobody can check something into it against my wishes. >>> Also, in the event that I cherry-pick revisions out-of-order, it allows me to >>> rebase, making merging easier. >>> >>> Is there *any* reason to make this branch public before 3.4.0 final? >> - Python is an open source project. Why do we need to hide >> development for a month or more? >> >> - Not even four eyes looking at the code seems to be odd. You >> can make mistakes too. >> >> This seems to be a social or a technical problem. I assume making this branch >> available read-only would address your concerns? Does hg allow this? And if >> not, why not create this branch in the upstream repository, and tell people not >> to commit to it? Why shouldn't such a social restriction work? Seems to work >> well for other projects. > > When you are release manager for Python, you may institute this policy if you > like. Right now, I have enough to do as it is. is it too much to ask for a public mirror / tarball / something of this branch? This seems to be a minor effort compared to the clinic work that went into 3.4. Matthias From larry at hastings.org Wed Feb 19 01:31:56 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 18 Feb 2014 16:31:56 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <5303F893.4030909@ubuntu.com> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <5303F0BD.6080203@hastings.org> <5303F345.9060108@ubuntu.com> <5303F52D.4090108@hastings.org> <5303F893.4030909@ubuntu.com> Message-ID: <5303FB7C.5010807@hastings.org> On 02/18/2014 04:19 PM, Matthias Klose wrote: > Am 19.02.2014 01:05, schrieb Larry Hastings: >> On 02/18/2014 03:56 PM, Matthias Klose wrote: >>> Am 19.02.2014 00:46, schrieb Larry Hastings: >>>> On 02/18/2014 03:38 PM, Matthias Klose wrote: >>>>> Am 17.02.2014 00:25, schrieb Larry Hastings: >>>>>> And my local branch will remain private until 3.4.0 final ships! >>>>> sorry, but this is so wrong. Is there *any* reason why to keep this branch >>>>> private? >>>> Yes. It ensures that nobody can check something into it against my wishes. >>>> Also, in the event that I cherry-pick revisions out-of-order, it allows me to >>>> rebase, making merging easier. >>>> >>>> Is there *any* reason to make this branch public before 3.4.0 final? >>> - Python is an open source project. Why do we need to hide >>> development for a month or more? >>> >>> - Not even four eyes looking at the code seems to be odd. You >>> can make mistakes too. >>> >>> This seems to be a social or a technical problem. I assume making this branch >>> available read-only would address your concerns? Does hg allow this? And if >>> not, why not create this branch in the upstream repository, and tell people not >>> to commit to it? Why shouldn't such a social restriction work? Seems to work >>> well for other projects. >> When you are release manager for Python, you may institute this policy if you >> like. Right now, I have enough to do as it is. > is it too much to ask for a public mirror / tarball / something of this branch? > This seems to be a minor effort compared to the clinic work that went into 3.4. Why do you need this? What is your use case? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Feb 19 00:54:23 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 18 Feb 2014 18:54:23 -0500 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <5303EEE7.8000101@ubuntu.com> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> Message-ID: <20140218185423.256a3e4d@anarchist.wooz.org> On Feb 19, 2014, at 12:38 AM, Matthias Klose wrote: >Am 17.02.2014 00:25, schrieb Larry Hastings: >> And my local branch will remain private until 3.4.0 final ships! > >sorry, but this is so wrong. Is there *any* reason why to keep this branch >private? IMO, no. read-only for !larry sure, but not private. -Barry From guido at python.org Wed Feb 19 03:46:16 2014 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Feb 2014 18:46:16 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <20140218185423.256a3e4d@anarchist.wooz.org> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: I do think there's one legitimate concern -- someone might pull a diff from Larry's branch and then accidentally push it back to the public repo, and then Larry would be in trouble if he was planning to rebase that diff. (The joys of DVCS -- we never had this problem in the cvs or svn days...) Publishing tarballs would prevent this and still let people test the exact code Larry is assembling on their favorite obscure platform. On Tue, Feb 18, 2014 at 3:54 PM, Barry Warsaw wrote: > On Feb 19, 2014, at 12:38 AM, Matthias Klose wrote: > > >Am 17.02.2014 00:25, schrieb Larry Hastings: > >> And my local branch will remain private until 3.4.0 final ships! > > > >sorry, but this is so wrong. Is there *any* reason why to keep this branch > >private? > > IMO, no. read-only for !larry sure, but not private. > > -Barry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed Feb 19 04:09:02 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 18 Feb 2014 21:09:02 -0600 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: Sounds like you aren't exactly a DVCS fan... On Tue, Feb 18, 2014 at 8:46 PM, Guido van Rossum wrote: > I do think there's one legitimate concern -- someone might pull a diff > from Larry's branch and then accidentally push it back to the public repo, > and then Larry would be in trouble if he was planning to rebase that diff. > (The joys of DVCS -- we never had this problem in the cvs or svn days...) > > Publishing tarballs would prevent this and still let people test the exact > code Larry is assembling on their favorite obscure platform. > > > On Tue, Feb 18, 2014 at 3:54 PM, Barry Warsaw wrote: > >> On Feb 19, 2014, at 12:38 AM, Matthias Klose wrote: >> >> >Am 17.02.2014 00:25, schrieb Larry Hastings: >> >> And my local branch will remain private until 3.4.0 final ships! >> > >> >sorry, but this is so wrong. Is there *any* reason why to keep this >> branch >> >private? >> >> IMO, no. read-only for !larry sure, but not private. >> >> -Barry >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed Feb 19 05:03:31 2014 From: larry at hastings.org (Larry Hastings) Date: Tue, 18 Feb 2014 20:03:31 -0800 Subject: [Python-Dev] Preview of 3.4 rc2 (so far) is up Message-ID: <53042D13.2010002@hastings.org> The URL has changed slightly. Please go here: http://midwinter.com/~larry/3.4.status/ You'll notice two things: * a "merge.status.html" file, which shows you the list of revisions that I've cherry-picked after rc1. * a tarball containing the resulting source tree. As I cherry-pick more revisions, I'll add new tarballs and update the merge status. For the record, I've passed over only two requested cherry-pick revisions so far: http://bugs.python.org/issue20646 select and kqueue round the timeout aways from zero http://bugs.python.org/issue20679 improve Enum subclass behavior I haven't rejected them, I just want more review. If you'd like to see these changes get cherry-picked for 3.4.0 rc2 (and final) please review them or convince someone else to contribute a review. Only thirty cherry-picked revisions so far. Gosh, you're making my life easy, guys, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Feb 19 06:22:10 2014 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 18 Feb 2014 21:22:10 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <20140218185423.256a3e4d@anarchist.wooz.org> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: <1392787330.9476.85115673.7E6CBD5E@webmail.messagingengine.com> On Tue, Feb 18, 2014, at 03:54 PM, Barry Warsaw wrote: > On Feb 19, 2014, at 12:38 AM, Matthias Klose wrote: > > >Am 17.02.2014 00:25, schrieb Larry Hastings: > >> And my local branch will remain private until 3.4.0 final ships! > > > >sorry, but this is so wrong. Is there *any* reason why to keep this branch > >private? > > IMO, no. read-only for !larry sure, but not private. I've always published my releasing repo on hg.p.o as releasing/2.7.X. I wasn't aware people actually used it; it's mostly a nice backup for when I rm rf * things in anger... :) From g.brandl at gmx.net Wed Feb 19 10:39:05 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 19 Feb 2014 10:39:05 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <5303F5CA.3010001@hastings.org> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <5303F0BD.6080203@hastings.org> <5303F5CA.3010001@hastings.org> Message-ID: Am 19.02.2014 01:07, schrieb Larry Hastings: > On 02/18/2014 03:54 PM, Victor Stinner wrote: >> 2014-02-19 0:46 GMT+01:00 Larry Hastings : >>> Is there *any* reason to make this branch public before 3.4.0 final? >> I'm a little bit worried by the fact that buildbots will not test it. > > "fact"? > > http://docs.python.org/devguide/buildbots.html#custom-builders And this works without a public repo on hg.python.org how? Georg From g.brandl at gmx.net Wed Feb 19 10:40:17 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 19 Feb 2014 10:40:17 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: Am 19.02.2014 03:46, schrieb Guido van Rossum: > I do think there's one legitimate concern -- someone might pull a diff from > Larry's branch and then accidentally push it back to the public repo, and then > Larry would be in trouble if he was planning to rebase that diff. (The joys of > DVCS -- we never had this problem in the cvs or svn days...) Pushes to hg.python.org repos are fully reversible. If that is Larry's concern he can even put it on bitbucket, where only he can push by default. Georg From g.brandl at gmx.net Wed Feb 19 10:42:44 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 19 Feb 2014 10:42:44 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <20140218185423.256a3e4d@anarchist.wooz.org> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: Am 19.02.2014 00:54, schrieb Barry Warsaw: > On Feb 19, 2014, at 12:38 AM, Matthias Klose wrote: > >>Am 17.02.2014 00:25, schrieb Larry Hastings: >>> And my local branch will remain private until 3.4.0 final ships! >> >>sorry, but this is so wrong. Is there *any* reason why to keep this branch >>private? > > IMO, no. read-only for !larry sure, but not private. I emphatically agree. There is no need at all for secrecy, or paranoia. And it is very understandable that vendors (or even "just" our binary building experts) want to make as many tests with what will be RC2 and then final as they can, to catch possible issues before release. Georg From ncoghlan at gmail.com Wed Feb 19 11:04:50 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Feb 2014 20:04:50 +1000 Subject: [Python-Dev] Preview of 3.4 rc2 (so far) is up In-Reply-To: <53042D13.2010002@hastings.org> References: <53042D13.2010002@hastings.org> Message-ID: On 19 Feb 2014 14:05, "Larry Hastings" wrote: > > > > The URL has changed slightly. Please go here: >> >> http://midwinter.com/~larry/3.4.status/ > > You'll notice two things: > a "merge.status.html" file, which shows you the list of revisions that I've cherry-picked after rc1. > a tarball containing the resulting source tree. > As I cherry-pick more revisions, I'll add new tarballs and update the merge status. > > > For the record, I've passed over only two requested cherry-pick revisions so far: >> >> http://bugs.python.org/issue20646 >> select and kqueue round the timeout aways from zero >> >> http://bugs.python.org/issue20679 >> improve Enum subclass behavior > > I haven't rejected them, I just want more review. If you'd like to see these changes get cherry-picked for 3.4.0 rc2 (and final) please review them or convince someone else to contribute a review. > > > Only thirty cherry-picked revisions so far. Gosh, you're making my life easy, guys, Larry, you announced your preferred release candidate management process too late to rely on it entirely - you should still audit all the deferred blockers and release blockers flagged for 3.4, and ask for an update on their status, with a pointer to the archived python-dev post describing how to request that the change be included in 3.4.0 rather than being left to 3.4.1. I know at least I have been setting those on the assumption things would work the same as they have in previous releases, since you hadn't said anything prior to rc1 about doing things differently. Regards, Nick. > > > /arry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Wed Feb 19 11:09:00 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 19 Feb 2014 11:09:00 +0100 Subject: [Python-Dev] Preview of 3.4 rc2 (so far) is up In-Reply-To: References: <53042D13.2010002@hastings.org> Message-ID: Am 19.02.2014 11:04, schrieb Nick Coghlan: > > On 19 Feb 2014 14:05, "Larry Hastings" > wrote: >> >> >> >> The URL has changed slightly. Please go here: >>> >>> http://midwinter.com/~larry/3.4.status/ >> >> You'll notice two things: >> a "merge.status.html" file, which shows you the list of revisions that I've > cherry-picked after rc1. >> a tarball containing the resulting source tree. >> As I cherry-pick more revisions, I'll add new tarballs and update the merge > status. >> >> >> For the record, I've passed over only two requested cherry-pick revisions so far: >>> >>> http://bugs.python.org/issue20646 >>> select and kqueue round the timeout aways from zero >>> >>> http://bugs.python.org/issue20679 >>> improve Enum subclass behavior >> >> I haven't rejected them, I just want more review. If you'd like to see these > changes get cherry-picked for 3.4.0 rc2 (and final) please review them or > convince someone else to contribute a review. >> >> >> Only thirty cherry-picked revisions so far. Gosh, you're making my life easy, > guys, > > Larry, you announced your preferred release candidate management process too > late to rely on it entirely - you should still audit all the deferred blockers > and release blockers flagged for 3.4, and ask for an update on their status, > with a pointer to the archived python-dev post describing how to request that > the change be included in 3.4.0 rather than being left to 3.4.1. I know at least > I have been setting those on the assumption things would work the same as they > have in previous releases, since you hadn't said anything prior to rc1 about > doing things differently. To be fair this isn't really different from 3.3.0, just that I didn't require a specific format for issues and went through all changes manually. Georg From regebro at gmail.com Wed Feb 19 11:53:46 2014 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 19 Feb 2014 11:53:46 +0100 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> Message-ID: On Tue, Feb 18, 2014 at 5:10 PM, Mark Lawrence wrote: > Sorry if this has already been suggested, but why not introduce a new > singleton to make the database people happier if not happy? To avoid > confusion call it dbnull? A reasonable compromise or complete cobblers? :) I think this is possible already, for the database people. The problem is that it will not pass the is None test, which at the very least is not backwards compatible with how they have used it before. From ncoghlan at gmail.com Wed Feb 19 12:47:30 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Feb 2014 21:47:30 +1000 Subject: [Python-Dev] Preview of 3.4 rc2 (so far) is up In-Reply-To: References: <53042D13.2010002@hastings.org> Message-ID: On 19 February 2014 20:09, Georg Brandl wrote: > Am 19.02.2014 11:04, schrieb Nick Coghlan: >> >> On 19 Feb 2014 14:05, "Larry Hastings" > > wrote: >>> >>> The URL has changed slightly. Please go here: >>>> >>>> http://midwinter.com/~larry/3.4.status/ >>> >>> You'll notice two things: >>> a "merge.status.html" file, which shows you the list of revisions that I've >> cherry-picked after rc1. >>> a tarball containing the resulting source tree. >>> As I cherry-pick more revisions, I'll add new tarballs and update the merge >> status. >>> >>> >>> For the record, I've passed over only two requested cherry-pick revisions so far: >>>> >>>> http://bugs.python.org/issue20646 >>>> select and kqueue round the timeout aways from zero >>>> >>>> http://bugs.python.org/issue20679 >>>> improve Enum subclass behavior >>> >>> I haven't rejected them, I just want more review. If you'd like to see these >> changes get cherry-picked for 3.4.0 rc2 (and final) please review them or >> convince someone else to contribute a review. >>> >>> >>> Only thirty cherry-picked revisions so far. Gosh, you're making my life easy, >> guys, >> >> Larry, you announced your preferred release candidate management process too >> late to rely on it entirely - you should still audit all the deferred blockers >> and release blockers flagged for 3.4, and ask for an update on their status, >> with a pointer to the archived python-dev post describing how to request that >> the change be included in 3.4.0 rather than being left to 3.4.1. I know at least >> I have been setting those on the assumption things would work the same as they >> have in previous releases, since you hadn't said anything prior to rc1 about >> doing things differently. > > To be fair this isn't really different from 3.3.0, just that I didn't require a > specific format for issues and went through all changes manually. That's the part that worries me - if Larry is assuming the post to python-dev is enough to get people to change their behaviour at short notice, I'm concerned things that should be release blockers won't end up blocking doing so. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Wed Feb 19 13:13:40 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Feb 2014 13:13:40 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: <20140219131340.0922b063@fsol> On Tue, 18 Feb 2014 18:54:23 -0500 Barry Warsaw wrote: > On Feb 19, 2014, at 12:38 AM, Matthias Klose wrote: > > >Am 17.02.2014 00:25, schrieb Larry Hastings: > >> And my local branch will remain private until 3.4.0 final ships! > > > >sorry, but this is so wrong. Is there *any* reason why to keep this branch > >private? > > IMO, no. read-only for !larry sure, but not private. Agreed too. Python isn't developed in private. Regards Antoine. From solipsis at pitrou.net Wed Feb 19 13:20:13 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Feb 2014 13:20:13 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: <20140219132013.157be496@fsol> On Tue, 18 Feb 2014 18:46:16 -0800 Guido van Rossum wrote: > I do think there's one legitimate concern -- someone might pull a diff from > Larry's branch and then accidentally push it back to the public repo, and > then Larry would be in trouble if he was planning to rebase that diff. (The > joys of DVCS -- we never had this problem in the cvs or svn days...) I don't think I understand the concern. Why is this different from any other mistake someone may make when pushing code? Also "rebase" is only really ok on private repos, as soon as something is published you should use "merge". Regards Antoine. From solipsis at pitrou.net Wed Feb 19 13:42:10 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 19 Feb 2014 13:42:10 +0100 Subject: [Python-Dev] Preview of 3.4 rc2 (so far) is up References: <53042D13.2010002@hastings.org> Message-ID: <20140219134210.002b418f@fsol> On Tue, 18 Feb 2014 20:03:31 -0800 Larry Hastings wrote: > > Only thirty cherry-picked revisions so far. Gosh, you're making my life > easy, guys, That's a large number of cherry-picked revisions. How many are actually release-critical? Regards Antoine. From hrvoje.niksic at avl.com Wed Feb 19 13:50:51 2014 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Wed, 19 Feb 2014 13:50:51 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <20140219132013.157be496@fsol> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <20140219132013.157be496@fsol> Message-ID: <5304A8AB.6020303@avl.com> On 02/19/2014 01:20 PM, Antoine Pitrou wrote: > On Tue, 18 Feb 2014 18:46:16 -0800 > Guido van Rossum wrote: >> I do think there's one legitimate concern -- someone might pull a diff from >> Larry's branch and then accidentally push it back to the public repo, and >> then Larry would be in trouble if he was planning to rebase that diff. (The >> joys of DVCS -- we never had this problem in the cvs or svn days...) > > I don't think I understand the concern. Why is this different from any > other mistake someone may make when pushing code? > Also "rebase" is only really ok on private repos, as soon as something > is published you should use "merge". If the branch were private, pushing to it would not count as "publishing", but would still provide the benefit of having a redundant server-side backup of the data. Being able to rebase without fuss is a possible legitimate reason to keep the branch private, which Guido provided in response to Matthias's question: >sorry, but this is so wrong. Is there *any* reason why to keep >this branch private? From yselivanov.ml at gmail.com Wed Feb 19 16:20:24 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 19 Feb 2014 10:20:24 -0500 Subject: [Python-Dev] Preview of 3.4 rc2 (so far) is up In-Reply-To: <20140219134210.002b418f@fsol> References: <53042D13.2010002@hastings.org> <20140219134210.002b418f@fsol> Message-ID: <5304CBB8.2080205@gmail.com> About 21 of those are related to asyncio. On 2/19/2014, 7:42 AM, Antoine Pitrou wrote: > On Tue, 18 Feb 2014 20:03:31 -0800 > Larry Hastings wrote: >> Only thirty cherry-picked revisions so far. Gosh, you're making my life >> easy, guys, > That's a large number of cherry-picked revisions. How many are actually > release-critical? > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com From guido at python.org Wed Feb 19 16:50:16 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Feb 2014 07:50:16 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: On Wed, Feb 19, 2014 at 1:42 AM, Georg Brandl wrote: > Am 19.02.2014 00:54, schrieb Barry Warsaw: > > On Feb 19, 2014, at 12:38 AM, Matthias Klose wrote: > > > >>Am 17.02.2014 00:25, schrieb Larry Hastings: > >>> And my local branch will remain private until 3.4.0 final ships! > >> > >>sorry, but this is so wrong. Is there *any* reason why to keep this > branch > >>private? > > > > IMO, no. read-only for !larry sure, but not private. > > I emphatically agree. There is no need at all for secrecy, or paranoia. > > And it is very understandable that vendors (or even "just" our binary > building experts) want to make as many tests with what will be RC2 and > then final as they can, to catch possible issues before release. > That's why it's RC2 and not 3.4final, right? Once Larry says it's baked, everyone *will* have a chance to test it. What value is a preview of the preview really going to add? Give Larry some trust and freedom to do things in the way that makes him comfortable. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Feb 19 17:16:02 2014 From: barry at python.org (Barry Warsaw) Date: Wed, 19 Feb 2014 11:16:02 -0500 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: <20140219111602.5786b892@anarchist.wooz.org> On Feb 19, 2014, at 07:50 AM, Guido van Rossum wrote: >That's why it's RC2 and not 3.4final, right? Once Larry says it's baked, >everyone *will* have a chance to test it. What value is a preview of the >preview really going to add? Give Larry some trust and freedom to do things >in the way that makes him comfortable. I totally agree that Larry should be given fairly wide discretion. He's also feeling out his first big release and deserves some slack. However, I think Matthias wants read access to the release repo because he's *also* cherry picking patches into Ubuntu's archive. We're already seeing some problems, which we want to investigate, and Matthias has also performed archive-wide test rebuilds to find Python 3.4 incompatibilities in 3rd party libraries, most of which we'd like to fix (e.g. main packages, if its not possible to get to everything in universe). Matthias just switched the default for Ubuntu 14.04 to Python 3.4 by default, so this is a great test bed to find problems. Cheers, -Barry From guido at python.org Wed Feb 19 18:13:09 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Feb 2014 09:13:09 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <20140219131340.0922b063@fsol> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <20140219131340.0922b063@fsol> Message-ID: On Wed, Feb 19, 2014 at 4:13 AM, Antoine Pitrou wrote: > Agreed too. Python isn't developed in private. That's a ridiculous accusation, bordering on malicious. Larry isn't "developing Python in private". He is simply working on something that he'll release when he feels comfortable releasing it. > I don't think I understand the concern. Why is this different from any > other mistake someone may make when pushing code? Maybe because 1000s of people are apparently ready to micro-manage and criticize Larry's work and waiting for him to screw up? Why are you trying to tell Larry how to use his tools? Larry *volunteered* to be the release manager and got widespread support when he did. If you don't trust him, go fucking fork the release yourself. > Also "rebase" is only really ok on private repos, as soon as something > is published you should use "merge". And that's exactly why Larry is holding off pushing. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Feb 19 18:20:02 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Feb 2014 09:20:02 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <20140219111602.5786b892@anarchist.wooz.org> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <20140219111602.5786b892@anarchist.wooz.org> Message-ID: On Wed, Feb 19, 2014 at 8:16 AM, Barry Warsaw wrote: > On Feb 19, 2014, at 07:50 AM, Guido van Rossum wrote: > > >That's why it's RC2 and not 3.4final, right? Once Larry says it's baked, > >everyone *will* have a chance to test it. What value is a preview of the > >preview really going to add? Give Larry some trust and freedom to do > things > >in the way that makes him comfortable. > > I totally agree that Larry should be given fairly wide discretion. He's > also > feeling out his first big release and deserves some slack. > Thanks for the support! However, I think Matthias wants read access to the release repo because he's > *also* cherry picking patches into Ubuntu's archive. We're already seeing > some problems, which we want to investigate, and Matthias has also > performed > archive-wide test rebuilds to find Python 3.4 incompatibilities in 3rd > party > libraries, most of which we'd like to fix (e.g. main packages, if its not > possible to get to everything in universe). > Again, this is what RC2 is for (and RC1, for that matter; apart from 20+ asyncio patches there really isn't much of a difference between RC1 and RC2). Larry may legitimately feel uncomfortable with what he's got on his local drive and prefer to tweak some things before telling people "go ahead test with this" -- the difference is that if he was working on new *code*, he could just not commit his work-in-progress, but since here he is assembling the final sequence of *revisions*, he prefers just not to push. (Georg alluded to the fact that you can undo changes in a public repo after they've been pushed, but I suspect he's referring to hg backout, which creates extra revisions, rather than a remote version of hg strip, which would go against the philosophy of DVCS. Either way, Larry's use of Hg is a totally legitimate workflow.) > Matthias just switched the default for Ubuntu 14.04 to Python 3.4 by > default, > so this is a great test bed to find problems. > And that's great, of course. But what is really gained by giving Larry trouble over a few days' worth of delay, at most? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Wed Feb 19 19:00:43 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 19 Feb 2014 19:00:43 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: Am 19.02.2014 16:50, schrieb Guido van Rossum: > On Wed, Feb 19, 2014 at 1:42 AM, Georg Brandl > wrote: > > Am 19.02.2014 00:54, schrieb Barry Warsaw: > > On Feb 19, 2014, at 12:38 AM, Matthias Klose wrote: > > > >>Am 17.02.2014 00:25, schrieb Larry Hastings: > >>> And my local branch will remain private until 3.4.0 final ships! > >> > >>sorry, but this is so wrong. Is there *any* reason why to keep this branch > >>private? > > > > IMO, no. read-only for !larry sure, but not private. > > I emphatically agree. There is no need at all for secrecy, or paranoia. > > And it is very understandable that vendors (or even "just" our binary > building experts) want to make as many tests with what will be RC2 and > then final as they can, to catch possible issues before release. > > > That's why it's RC2 and not 3.4final, right? Once Larry says it's baked, > everyone *will* have a chance to test it. What value is a preview of the preview > really going to add? Ned told me just a few days ago that he does regular pre-tag builds of the OSX installers, and as for the Debian/Ubuntu side Barry can say more. The thing with the RCs is that as long as you add patches during the RC phase (which is more or less unavoidable if the schedule is to be kept), the state of the release branch can only profit from more eyes. > Give Larry some trust and freedom to do things in the way that makes him > comfortable. I have no doubts that Larry will make 3.4 the best Python yet :) So far he has discussed most of his procedures with us, so I don't see a reason not to weigh in here. Georg From g.brandl at gmx.net Wed Feb 19 19:09:07 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 19 Feb 2014 19:09:07 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: Am 19.02.2014 19:00, schrieb Georg Brandl: >> Give Larry some trust and freedom to do things in the way that makes him >> comfortable. > > I have no doubts that Larry will make 3.4 the best Python yet :) So far he > has discussed most of his procedures with us, so I don't see a reason not to > weigh in here. NB, "us" being the previous release managers. Georg From g.brandl at gmx.net Wed Feb 19 19:17:47 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 19 Feb 2014 19:17:47 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: Am 19.02.2014 19:00, schrieb Georg Brandl: > Am 19.02.2014 16:50, schrieb Guido van Rossum: >> On Wed, Feb 19, 2014 at 1:42 AM, Georg Brandl > > wrote: >> >> Am 19.02.2014 00:54, schrieb Barry Warsaw: >> > On Feb 19, 2014, at 12:38 AM, Matthias Klose wrote: >> > >> >>Am 17.02.2014 00:25, schrieb Larry Hastings: >> >>> And my local branch will remain private until 3.4.0 final ships! >> >> >> >>sorry, but this is so wrong. Is there *any* reason why to keep this branch >> >>private? >> > >> > IMO, no. read-only for !larry sure, but not private. >> >> I emphatically agree. There is no need at all for secrecy, or paranoia. >> >> And it is very understandable that vendors (or even "just" our binary >> building experts) want to make as many tests with what will be RC2 and >> then final as they can, to catch possible issues before release. >> >> >> That's why it's RC2 and not 3.4final, right? Once Larry says it's baked, >> everyone *will* have a chance to test it. What value is a preview of the preview >> really going to add? > > Ned told me just a few days ago that he does regular pre-tag builds of the OSX > installers, and as for the Debian/Ubuntu side Barry can say more. The thing > with the RCs is that as long as you add patches during the RC phase (which is > more or less unavoidable if the schedule is to be kept), the state of the > release branch can only profit from more eyes. There's even some helpful people on #python-dev (like Arfrever from Gentoo) who frequently do post-push reviews, catching embarrassing bugs before they can sneak their way into a release (thank you Arfrever!). OK, that's my reasoning, I'm going to fucking shut up now. Georg From v+python at g.nevcal.com Wed Feb 19 20:23:48 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 19 Feb 2014 11:23:48 -0800 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> Message-ID: <530504C4.6000400@g.nevcal.com> On 2/19/2014 2:53 AM, Lennart Regebro wrote: > On Tue, Feb 18, 2014 at 5:10 PM, Mark Lawrence wrote: >> Sorry if this has already been suggested, but why not introduce a new >> singleton to make the database people happier if not happy? To avoid >> confusion call it dbnull? A reasonable compromise or complete cobblers? :) > I think this is possible already, for the database people. The problem > is that it will not pass the is None test, which at the very least is > not backwards compatible with how they have used it before. The new singleton will be called something else, likely with Null in the name, so that's what I'll call it here... Null. So when switching from None to Null, you must also switch from "is None" to "is Null". Of course it is not backwards compatible... but once all the "database related None usage" is switched to "Null usage" it should work the same as before, but with proper (for some database's definition of proper) semantics. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Feb 19 22:18:50 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Feb 2014 07:18:50 +1000 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: On 20 Feb 2014 04:18, "Georg Brandl" wrote: > > Am 19.02.2014 19:00, schrieb Georg Brandl: > > Am 19.02.2014 16:50, schrieb Guido van Rossum: > >> On Wed, Feb 19, 2014 at 1:42 AM, Georg Brandl >> > wrote: > >> > >> Am 19.02.2014 00:54, schrieb Barry Warsaw: > >> > On Feb 19, 2014, at 12:38 AM, Matthias Klose wrote: > >> > > >> >>Am 17.02.2014 00:25, schrieb Larry Hastings: > >> >>> And my local branch will remain private until 3.4.0 final ships! > >> >> > >> >>sorry, but this is so wrong. Is there *any* reason why to keep this branch > >> >>private? > >> > > >> > IMO, no. read-only for !larry sure, but not private. > >> > >> I emphatically agree. There is no need at all for secrecy, or paranoia. > >> > >> And it is very understandable that vendors (or even "just" our binary > >> building experts) want to make as many tests with what will be RC2 and > >> then final as they can, to catch possible issues before release. > >> > >> > >> That's why it's RC2 and not 3.4final, right? Once Larry says it's baked, > >> everyone *will* have a chance to test it. What value is a preview of the preview > >> really going to add? > > > > Ned told me just a few days ago that he does regular pre-tag builds of the OSX > > installers, and as for the Debian/Ubuntu side Barry can say more. The thing > > with the RCs is that as long as you add patches during the RC phase (which is > > more or less unavoidable if the schedule is to be kept), the state of the > > release branch can only profit from more eyes. > > There's even some helpful people on #python-dev (like Arfrever from Gentoo) who > frequently do post-push reviews, catching embarrassing bugs before they can > sneak their way into a release (thank you Arfrever!). > > OK, that's my reasoning, I'm going to fucking shut up now. I suspect everyone is also highly aware of the fact that there are some ambitious changes in 3.4, the release of rc1 is bringing the usual wave of additional third party testing that has picked up some interesting regressions and usability issues (e.g. the Alembic test suite found a fun one in the inspect module, while the pip installation doesn't currently play nice with UAC on Windows), and the Ubuntu 14.04 deadline restricts our ability to add a 3rd rc. That brings with it a strong desire to parallelise things as much as possible, and read only access to the upcoming release helps greatly in knowing which regressions have already been addressed, allowing third party testers to more easily focus on any remaining issues. A "user beware, this may be rebased without warning" clone would be fine for that purpose, and I suspect in most cases just running rc2 -> final with such a clone available (preserving Larry's current workflow until rc2) would be sufficient to address most concerns. Regards, Nick. > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Feb 20 00:25:54 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Feb 2014 10:25:54 +1100 Subject: [Python-Dev] PEPs don't redirect on python.org Message-ID: Apologies if this is misdirected! I notice the switch to the new python.org web site has happened; but now PEPs are simply 404: http://www.python.org/dev/peps/pep-0008/ However, trimming the URL offers a redirect: http://www.python.org/dev/peps/ -> http://legacy.python.org/dev/peps/ from which the content can be found. Can a blanket redirect rule be put in that makes all the PEPs at least go to /dev/peps/ ? ChrisA From rosuav at gmail.com Thu Feb 20 00:28:52 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 20 Feb 2014 10:28:52 +1100 Subject: [Python-Dev] PEPs don't redirect on python.org In-Reply-To: References: Message-ID: On Thu, Feb 20, 2014 at 10:25 AM, Chris Angelico wrote: > I notice the switch to the new python.org web site has happened; but > now PEPs are simply 404: > > http://www.python.org/dev/peps/pep-0008/ > > However, trimming the URL offers a redirect: > > http://www.python.org/dev/peps/ > > -> > > http://legacy.python.org/dev/peps/ Oh! Must have been a transitional period. The PEPs now redirect too. Thanks, whoever did that! Makes the linking much easier. :) ChrisA From larry at hastings.org Thu Feb 20 01:31:45 2014 From: larry at hastings.org (Larry Hastings) Date: Wed, 19 Feb 2014 16:31:45 -0800 Subject: [Python-Dev] Preview of 3.4 rc2 (so far) is up In-Reply-To: References: <53042D13.2010002@hastings.org> Message-ID: <53054CF1.3070400@hastings.org> On 02/19/2014 02:04 AM, Nick Coghlan wrote: > > > On 19 Feb 2014 14:05, "Larry Hastings" > wrote: > > > > > > > > The URL has changed slightly. Please go here: > >> > >> http://midwinter.com/~larry/3.4.status/ > > > > > You'll notice two things: > > a "merge.status.html" file, which shows you the list of revisions > that I've cherry-picked after rc1. > > a tarball containing the resulting source tree. > > As I cherry-pick more revisions, I'll add new tarballs and update > the merge status. > > > > > > For the record, I've passed over only two requested cherry-pick > revisions so far: > >> > >> http://bugs.python.org/issue20646 > >> select and kqueue round the timeout aways from zero > >> > >> http://bugs.python.org/issue20679 > >> improve Enum subclass behavior > > > > I haven't rejected them, I just want more review. If you'd like to > see these changes get cherry-picked for 3.4.0 rc2 (and final) please > review them or convince someone else to contribute a review. > > > > > > Only thirty cherry-picked revisions so far. Gosh, you're making my > life easy, guys, > > Larry, you announced your preferred release candidate management > process too late to rely on it entirely - you should still audit all > the deferred blockers and release blockers flagged for 3.4, and ask > for an update on their status, with a pointer to the archived > python-dev post describing how to request that the change be included > in 3.4.0 rather than being left to 3.4.1. I know at least I have been > setting those on the assumption things would work the same as they > have in previous releases, since you hadn't said anything prior to rc1 > about doing things differently. > The release is still about a month away. And yes I still plan to go through the release blockers. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Thu Feb 20 01:42:57 2014 From: larry at hastings.org (Larry Hastings) Date: Wed, 19 Feb 2014 16:42:57 -0800 Subject: [Python-Dev] Preview of 3.4 rc2 (so far) is up In-Reply-To: <5304CBB8.2080205@gmail.com> References: <53042D13.2010002@hastings.org> <20140219134210.002b418f@fsol> <5304CBB8.2080205@gmail.com> Message-ID: <53054F91.8090305@hastings.org> On 02/19/2014 07:20 AM, Yury Selivanov wrote: > About 21 of those are related to asyncio. > > On 2/19/2014, 7:42 AM, Antoine Pitrou wrote: >> On Tue, 18 Feb 2014 20:03:31 -0800 >> Larry Hastings wrote: >>> Only thirty cherry-picked revisions so far. Gosh, you're making my >>> life >>> easy, guys, >> That's a large number of cherry-picked revisions. How many are actually >> release-critical? >> >> Regards >> >> Antoine. Yes, I'm allowing in basically all the asyncio changes, because a) there's no installed base, and b) Guido is himself very heavily involved in those changes. It's my opinion that asyncio is going to get a lot of scrutiny after 3.4.0 ships, so even though it's marked provisional it's important to get it right. And I don't have enough domain knowledge to be able to pick and choose their changes. So I'm relying on Victor / Guido / Yury, merging all their changes, and hoping for the best. I really am hoping it'll settle down soon, though, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Feb 20 01:55:26 2014 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Feb 2014 16:55:26 -0800 Subject: [Python-Dev] Preview of 3.4 rc2 (so far) is up In-Reply-To: <53054F91.8090305@hastings.org> References: <53042D13.2010002@hastings.org> <20140219134210.002b418f@fsol> <5304CBB8.2080205@gmail.com> <53054F91.8090305@hastings.org> Message-ID: On Wed, Feb 19, 2014 at 4:42 PM, Larry Hastings wrote: > Yes, I'm allowing in basically all the asyncio changes, because a) > there's no installed base, and b) Guido is himself very heavily involved in > those changes. It's my opinion that asyncio is going to get a lot of > scrutiny after 3.4.0 ships, so even though it's marked provisional it's > important to get it right. And I don't have enough domain knowledge to be > able to pick and choose their changes. So I'm relying on Victor / Guido / > Yury, merging all their changes, and hoping for the best. > > I really am hoping it'll settle down soon, though, > I am actively clamping down on these now. Yuri's and Victor's youthful enthusiasm is adorable. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Feb 20 02:01:52 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Feb 2014 11:01:52 +1000 Subject: [Python-Dev] Preview of 3.4 rc2 (so far) is up In-Reply-To: <53054CF1.3070400@hastings.org> References: <53042D13.2010002@hastings.org> <53054CF1.3070400@hastings.org> Message-ID: On 20 February 2014 10:31, Larry Hastings wrote: > On 02/19/2014 02:04 AM, Nick Coghlan wrote: > > The release is still about a month away. And yes I still plan to go through > the release blockers. Thanks. I confess I had missed that rc2 -> final was already 3 weeks rather than the 2 weeks between rc1 and rc2, so my mental timeline was off. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Thu Feb 20 02:24:16 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 20 Feb 2014 10:24:16 +0900 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > I suspect everyone is also highly aware of the fact that there are > some ambitious changes in 3.4, Which is an argument for longer beta and RC periods than usual, or maybe some of the ambition should have been restrained. It's not necessarily a reason why more eyes help (see below). > the release of rc1 is bringing the usual wave of additional third > party testing that has picked up some interesting regressions and > usability issues (e.g. the Alembic test suite found a fun one in > the inspect module, while the pip installation doesn't currently > play nice with UAC on Windows), and the Ubuntu 14.04 deadline > restricts our ability to add a 3rd rc. OK, but that means we're screwed regardless. We've decided to release what we've got on a specific timeline, and a few extra days of testing is going to make a marginal difference on average. Remember that under time pressure in bugfixing, the average programmer introduces a new bug that gets through to a product every ten lines. OK, so we're[1] 100X better than average, and I suppose for some subset 1000X better. Still that means several new bugs, and some of them may be doozies. > That brings with it a strong desire to parallelise things as much > as possible, and read only access to the upcoming release helps > greatly in knowing which regressions have already been addressed, > allowing third party testers to more easily focus on any remaining > issues. Sure, but it *doesn't* help in knowing which ones are *correctly* addressed. These *are* ambitious changes; some of the remaining bugs may be very deep. The obvious fixes may do more harm than good. Ie, "more eyes" is (a) mostly a fallacy (as Heinlein put it, the wisdom of a group is less than or equal to the maximum of the wisdom of the members) and (b) in any case the "more eyes" effect is diluted if people are deliberately looking at different parts of the code. > A "user beware, this may be rebased without warning" clone would be > fine for that purpose, and I suspect in most cases just running rc2 > -> final with such a clone available (preserving Larry's current > workflow until rc2) would be sufficient to address most concerns. Larry's already providing tarballs as I understand it. The argument that a "read-only, no cherrypicking by committers" repo is nothing but a better tarball is valid, but as I say, AFAICS the expected gain is pretty marginal. The conflict here is not Larry's process, it's the decision to make an ambitious release on a short time schedule. I sympathize with Ubuntu to some extent -- they have a business to run, after all. But should Ubuntu desires be distorting a volunteer RE's process? Was Larry told that commercial interests should be respected in designing his process? Footnotes: [1] FVO "we" not containing "me". You'll notice I'm not submitting patches. From sky.kok at speaklikeaking.com Thu Feb 20 02:52:27 2014 From: sky.kok at speaklikeaking.com (Vajrasky Kok) Date: Thu, 20 Feb 2014 09:52:27 +0800 Subject: [Python-Dev] A misredirected ticket link in hg.python.org/cpython Message-ID: Go to any commit link in hg.python.org/cpython, for example http://hg.python.org/cpython/rev/d11ca14c9a61. You have this commit message: Issue #6815: os.path.expandvars() now supports non-ASCII Unicode environment variables names and values. [#6815] The [#6815] link is going to http://bugs.python.org/6815. If you follow that link, it redirects to http://legacy.python.org/sf/ and you get message: You did not provide a report number. The link should be http://bugs.python.org/issue6815. Cheers, Vajrasky Kok From barry at python.org Thu Feb 20 03:23:55 2014 From: barry at python.org (Barry Warsaw) Date: Wed, 19 Feb 2014 21:23:55 -0500 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140219212355.5a47f90f@anarchist.wooz.org> On Feb 20, 2014, at 10:24 AM, Stephen J. Turnbull wrote: >But should Ubuntu desires be distorting a volunteer RE's process? Ubuntu 14.04 final freeze is April 10[1], so I think that's the drop dead date for getting 3.4 final into Ubuntu 14.04. Matthias may correct me, but I think if we can hit that date (well, maybe a day or two early to be safe) we're good. Missing that date probably isn't catastrophic, especially if there are few to no changes between the last Python 3.4 rc before our final freeze, and Python 3.4 final. What it means is that 14.04 won't ship with the final Python 3.4 and we'll have to do a "stable release update" after 14.04 to catch up to the Python 3.4 final release. It does mean that many Ubuntu users won't see it until 14.04.1, whenever that is, if at all. But if the only difference is a version string and sys.version_info, then I don't think that's so bad. And ultimately we'll have to do that anyway to get the LTS users Python 3.4.1, 3.4.2, and so on. Two notes: Matthias just enabled Python 3.4 as the default Python 3, and there's no going back. Also, we have aligned the Python release schedules with external schedules before, most notably Apple a couple of times. I think it's reasonable to do so *if* we can do it without sacrificing the quality of Python 3.4's final release. Cheers, -Barry [1] https://wiki.ubuntu.com/TrustyTahr/ReleaseSchedule From ethan at stoneleaf.us Thu Feb 20 03:25:33 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 19 Feb 2014 18:25:33 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <5305679D.10503@stoneleaf.us> On 02/19/2014 05:24 PM, Stephen J. Turnbull wrote: > > The conflict here is not Larry's > process, it's the decision to make an ambitious release on a short > time schedule. I sympathize with Ubuntu to some extent -- they have a > business to run, after all. But should Ubuntu desires be distorting a > volunteer RE's process? Was Larry told that commercial interests > should be respected in designing his process? When talk of slipping the final date again was discussed, Guido basically said no. -- ~Ethan~ From ncoghlan at gmail.com Thu Feb 20 03:52:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Feb 2014 12:52:05 +1000 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <5305679D.10503@stoneleaf.us> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> <5305679D.10503@stoneleaf.us> Message-ID: On 20 Feb 2014 12:26, "Ethan Furman" wrote: > > On 02/19/2014 05:24 PM, Stephen J. Turnbull wrote: >> >> >> The conflict here is not Larry's >> process, it's the decision to make an ambitious release on a short >> time schedule. I sympathize with Ubuntu to some extent -- they have a >> business to run, after all. But should Ubuntu desires be distorting a >> volunteer RE's process? Was Larry told that commercial interests >> should be respected in designing his process? > > > When talk of slipping the final date again was discussed, Guido basically said no. Guido said no more to the additional Argument Clinic related changes, rather than to an extra rc in general. Note that I made my comment before Larry pointed out the existing schedule was a week longer than I thought, and Barry clarified that there *is* still room for a third rc if Larry decides that would be appropriate. Cheers, Nick. > > -- > ~Ethan~ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Feb 20 04:11:41 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Feb 2014 16:11:41 +1300 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <530504C4.6000400@g.nevcal.com> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <530504C4.6000400@g.nevcal.com> Message-ID: <5305726D.9040901@canterbury.ac.nz> On 20/02/14 08:23, Glenn Linderman wrote: > Of course it is not backwards compatible... but once all the "database related > None usage" is switched to "Null usage" it should work the same as before, My problem with this is that there is no clear distinction between "database-related None usage" and None usage in general. Data retrieved from a database is often passed to other code that has nothing to do with databases, and data received from other code is inserted into databases. Somewhere in between someone is going to have to be careful to translate back and forth between Nones and Nulls. This is a recipe for chaos. -- Greg From larry at hastings.org Thu Feb 20 04:20:12 2014 From: larry at hastings.org (Larry Hastings) Date: Wed, 19 Feb 2014 19:20:12 -0800 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <5305746C.7000309@hastings.org> On 02/19/2014 05:24 PM, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > A "user beware, this may be rebased without warning" clone would be > > fine for that purpose, and I suspect in most cases just running rc2 > > -> final with such a clone available (preserving Larry's current > > workflow until rc2) would be sufficient to address most concerns. > > Larry's already providing tarballs as I understand it. Yep. Well, just "tarball" so far ;-) As for a "user beware" clone: I worry about providing anything that looks/tastes/smells like a repo. Someone could still inadvertently push those revisions back to trunk, and then we'd have a real mess on our hands. Publishing tarballs drops the possibility down to about zero. > The conflict here is not Larry's > process, it's the decision to make an ambitious release on a short > time schedule. I sympathize with Ubuntu to some extent -- they have a > business to run, after all. But should Ubuntu desires be distorting a > volunteer RE's process? Was Larry told that commercial interests > should be respected in designing his process? I haven't seen anything that makes me think we're in trouble. Every release has its bumps; that's what the rc period is for. I remind you we're still a month away. I grant you asyncio is still evolving surprisingly rapidly for an rc. But it doesn't have an installed base yet, and it's provisional anyway, so it's not making me anxious. Worst case, we issue a 3.4.1 on a very accelerated schedule. But it doesn't seem like it'll be necessary. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Thu Feb 20 05:41:39 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 20 Feb 2014 13:41:39 +0900 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <5305746C.7000309@hastings.org> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> <5305746C.7000309@hastings.org> Message-ID: <874n3uwg3g.fsf@uwakimon.sk.tsukuba.ac.jp> Larry Hastings writes: > Someone could still inadvertently push those revisions back to > trunk, and then we'd have a real mess on our hands. Publishing > tarballs drops the possibility down to about zero. Note: I see no reason for you to change your process for the 3.4.0 release. I just want to record my comments in this thread for future reference. I don't think any of the above is true. First, although "possibility of inadvertant push" as written is obviously correct, I don't think the implication that the *probability* of an *inadvertant* push is any higher is correct (somebody else -- Antoine? Guido? -- already pointed this out). The reasoning is that if somebody clones from a mirror of your release branch, they will have to make a deliberate decision to push to trunk, or a deliberate decision to merge or cherrypick from your branch into a branch destined for trunk, to cause a problem. That's actually safer than if they apply a patch from the tracker by hand, since in the case of a patch there will be no metadata to indicate that the conflict was caused by concurrent cherrypicks, and if they're sufficiently incautious, the actual change data may be different. That is messy. Second, what "real mess"? "VCS means never having to say 'Oh, shit!'" If such a thing happens, you just take your branch, do an "ours" merge with trunk obsoleting the trunk, and then cherrypick everything appropriate from obs-trunk. Tedious, yes. Somewhat error-prone, I suppose. Keep the buildbots very busy for a while, for sure. Real mess? IMHO, no. The fact that the repair procedure uses only "proper merges" (vs. rebase) means that rebase confusion can't propagate silently, nor can committed changes (in anybody's branch) be inadvertantly lost. (There are better strategies than the above, which I'll be happy to discuss if and when necessary. And for tedium reduction, there are features like git's filter-branch, which a Mercurial dev assures me can be done with hg too.) Third, tarballs don't drop the possibility to zero. We know that patch reviewers have those patches in local workspaces. In some cases you can diff a file from the tarball and get the patch you want (mostly, as mentioned above) and apply that to your feature branch. In the case of asyncio, some such path to a duplicate cherrypick seems quite probable (compared with "near zero", anyway). > I haven't seen anything that makes me think we're in trouble. Your evaluation is plenty good enough for me. But the actual information that your assessment is based on is almost private to you, and that's the reason other folks want access to a repo. By "almost private" I mean that the manipulation of revision information that DVCSes make so convenient is unavailable to them. From stephen at xemacs.org Thu Feb 20 06:08:44 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 20 Feb 2014 14:08:44 +0900 Subject: [Python-Dev] python 3 niggle: None < 1 raises TypeError In-Reply-To: <5305726D.9040901@canterbury.ac.nz> References: <52FDCE0D.5090402@simplistix.co.uk> <20140214110202.3ce2257a@fsol> <20140214112034.78c9a02e@fsol> <52FDFF33.40703@egenix.com> <87zjlsx67k.fsf@uwakimon.sk.tsukuba.ac.jp> <5301EEFD.9020502@egenix.com> <5301F5DD.9050101@egenix.com> <5301FC83.6080404@python.org> <5302EB76.1050809@canterbury.ac.nz> <530504C4.6000400@g.nevcal.com> <5305726D.9040901@canterbury.ac.nz> Message-ID: <8738jeweub.fsf@uwakimon.sk.tsukuba.ac.jp> Greg Ewing writes: > Data retrieved from a database is often passed to other code > that has nothing to do with databases, and data received from > other code is inserted into databases. Somewhere in between > someone is going to have to be careful to translate back and > forth between Nones and Nulls. Sure. The suggestion is to assign responsibility for such careful translation to implementers of the DB API. > This is a recipe for chaos. If it is, then chaos is already upon us because you can't be careful even if you want to. I don't know if fixing it is worth the work and confusion involved in the fixing process, but fixing it will conserve (and maybe reduce) chaos, not create it. From nad at acm.org Thu Feb 20 08:03:44 2014 From: nad at acm.org (Ned Deily) Date: Wed, 19 Feb 2014 23:03:44 -0800 Subject: [Python-Dev] A misredirected ticket link in hg.python.org/cpython References: Message-ID: In article , Vajrasky Kok wrote: > Go to any commit link in hg.python.org/cpython, for example > http://hg.python.org/cpython/rev/d11ca14c9a61. > > You have this commit message: > > Issue #6815: os.path.expandvars() now supports non-ASCII Unicode > environment variables names and values. [#6815] > > The [#6815] link is going to http://bugs.python.org/6815. If you > follow that link, it redirects to http://legacy.python.org/sf/ and you > get message: You did not provide a report number. The link should be > http://bugs.python.org/issue6815. Thanks! I've reported the problem to Noah on the infrastructure team. -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Thu Feb 20 08:09:52 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Feb 2014 17:09:52 +1000 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: <294D476D-2A33-445E-BBCC-68B01DFE77CE@mac.com> References: <530103F9.8070005@hastings.org> <53013CE9.9070207@canterbury.ac.nz> <294D476D-2A33-445E-BBCC-68B01DFE77CE@mac.com> Message-ID: On 20 February 2014 16:42, Ronald Oussoren wrote: > > On 17 Feb 2014, at 00:43, Nick Coghlan wrote: > >> >> On 17 Feb 2014 08:36, "Greg Ewing" wrote: >> > >> > Larry Hastings wrote: >> > >> >> 3) We hold off on merging the rest of the Derby patches until after 3.4.0 final ships, then we merge them into the 3.4 maintenance branch so they go into 3.4.1. >> > >> > >> > But wouldn't that be introducing a new feature into a >> > maintenance release? (I.e. some functions that didn't >> > have introspectable signatures before would gain them.) >> >> From a compatibility point of view, 3.4.0 will already force introspection users and tool developers to cope with the fact that some, but not all, builtin and extension types provide valid signature data. Additional clinic conversions that don't alter semantics then just move additional callables into the "supports programmatic introspection" category. >> >> It's certainly in a grey area, but "What's in the best interest of end users?" pushes me in the direction of counting clinic conversions that don't change semantics as bug fixes - they get improved introspection support sooner, and it shouldn't make life any harder for tool developers because all of the adjustments for 3.4 will be to the associated functional changes in the inspect module. >> >> The key thing is to make sure to postpone any changes that impact *semantics* (like adding keyword argument support). > > But there is a semantic change: some functions without a signature in 3.4.0 would have a signature in 3.4.1. That's unlikely to affect user code much because AFAIK signatures aren't used a lot yet, but it is a semantic change non the less :-) Heh, you must have managed to miss all the Argument Clinic debates - "semantics" there is shorthand for "the semantics of the call" :) It turns out there are some current C signatures where we either need to slightly change the semantics of the API or else add new features to the inspect module before we can represent them properly at the Python layer. So, for the life of Python 3.4, those are off limits for conversion, and we'll sort them out as part of PEP 457 for 3.5. However, there are plenty of others where the signature *can* be adequately represented in 3.4, they just aren't (yet). So the approach Larry has taken is to declare that "could expose valid signature data, but doesn't" counts as a bug fix for Python 3.4 maintenance release purposes. We'll make sure the porting section of the What's New guide addresses that explicitly for the benefit of anyone porting introspection tools to Python 3.4 (having all of the argument introspection in the inspect module be based on inspect.signature and various enhancements to inspect.signature itself has significantly increased the number of callables that now support introspection). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ronaldoussoren at mac.com Thu Feb 20 08:24:39 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 20 Feb 2014 08:24:39 +0100 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: References: <530103F9.8070005@hastings.org> <53013CE9.9070207@canterbury.ac.nz> <294D476D-2A33-445E-BBCC-68B01DFE77CE@mac.com> Message-ID: <418EC6BD-D206-4A60-ABE8-2553F9BF1515@mac.com> On 20 Feb 2014, at 08:09, Nick Coghlan wrote: > On 20 February 2014 16:42, Ronald Oussoren wrote: >> >> On 17 Feb 2014, at 00:43, Nick Coghlan wrote: >> >>> >>> On 17 Feb 2014 08:36, "Greg Ewing" wrote: >>>> >>>> Larry Hastings wrote: >>>> >>>>> 3) We hold off on merging the rest of the Derby patches until after 3.4.0 final ships, then we merge them into the 3.4 maintenance branch so they go into 3.4.1. >>>> >>>> >>>> But wouldn't that be introducing a new feature into a >>>> maintenance release? (I.e. some functions that didn't >>>> have introspectable signatures before would gain them.) >>> >>> From a compatibility point of view, 3.4.0 will already force introspection users and tool developers to cope with the fact that some, but not all, builtin and extension types provide valid signature data. Additional clinic conversions that don't alter semantics then just move additional callables into the "supports programmatic introspection" category. >>> >>> It's certainly in a grey area, but "What's in the best interest of end users?" pushes me in the direction of counting clinic conversions that don't change semantics as bug fixes - they get improved introspection support sooner, and it shouldn't make life any harder for tool developers because all of the adjustments for 3.4 will be to the associated functional changes in the inspect module. >>> >>> The key thing is to make sure to postpone any changes that impact *semantics* (like adding keyword argument support). >> >> But there is a semantic change: some functions without a signature in 3.4.0 would have a signature in 3.4.1. That's unlikely to affect user code much because AFAIK signatures aren't used a lot yet, but it is a semantic change non the less :-) > > Heh, you must have managed to miss all the Argument Clinic debates - > "semantics" there is shorthand for "the semantics of the call" :) I skipped most of that discussion, due to the sheer volume and limited time to add something meaningful to that discussion. > > It turns out there are some current C signatures where we either need > to slightly change the semantics of the API or else add new features > to the inspect module before we can represent them properly at the > Python layer. So, for the life of Python 3.4, those are off limits for > conversion, and we'll sort them out as part of PEP 457 for 3.5. That much I noticed, and IIRC it was noticed fairly early on in Argument Clinic?s history that there are C functions that have an API that cannot easily be represented in a pure Python function (other than using ?*args, **kw? and manually parsing the argument list). I totally agree that changing the signature of functions should wait for 3.5, but that?s the easy bit. > > However, there are plenty of others where the signature *can* be > adequately represented in 3.4, they just aren't (yet). So the approach > Larry has taken is to declare that "could expose valid signature data, > but doesn't" counts as a bug fix for Python 3.4 maintenance release > purposes. We'll make sure the porting section of the What's New guide > addresses that explicitly for the benefit of anyone porting > introspection tools to Python 3.4 (having all of the argument > introspection in the inspect module be based on inspect.signature and > various enhancements to inspect.signature itself has significantly > increased the number of callables that now support introspection). I can live with that, but don?t really agree that exposing new signature data is a bug fix. But maybe I?m just too conservative :-) To end on a positive not, I do like signature objects, and have added support for them to PyObjC to enrich its introspection capabilities. Ronald > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Thu Feb 20 07:42:18 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 20 Feb 2014 07:42:18 +0100 Subject: [Python-Dev] Python 3.4: What to do about the Derby patches In-Reply-To: References: <530103F9.8070005@hastings.org> <53013CE9.9070207@canterbury.ac.nz> Message-ID: <294D476D-2A33-445E-BBCC-68B01DFE77CE@mac.com> On 17 Feb 2014, at 00:43, Nick Coghlan wrote: > > On 17 Feb 2014 08:36, "Greg Ewing" wrote: > > > > Larry Hastings wrote: > > > >> 3) We hold off on merging the rest of the Derby patches until after 3.4.0 final ships, then we merge them into the 3.4 maintenance branch so they go into 3.4.1. > > > > > > But wouldn't that be introducing a new feature into a > > maintenance release? (I.e. some functions that didn't > > have introspectable signatures before would gain them.) > > From a compatibility point of view, 3.4.0 will already force introspection users and tool developers to cope with the fact that some, but not all, builtin and extension types provide valid signature data. Additional clinic conversions that don't alter semantics then just move additional callables into the "supports programmatic introspection" category. > > It's certainly in a grey area, but "What's in the best interest of end users?" pushes me in the direction of counting clinic conversions that don't change semantics as bug fixes - they get improved introspection support sooner, and it shouldn't make life any harder for tool developers because all of the adjustments for 3.4 will be to the associated functional changes in the inspect module. > > The key thing is to make sure to postpone any changes that impact *semantics* (like adding keyword argument support). But there is a semantic change: some functions without a signature in 3.4.0 would have a signature in 3.4.1. That?s unlikely to affect user code much because AFAIK signatures aren?t used a lot yet, but it is a semantic change non the less :-) Ronald From solipsis at pitrou.net Thu Feb 20 12:53:09 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 20 Feb 2014 12:53:09 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140220125309.282e79fb@fsol> On Thu, 20 Feb 2014 10:24:16 +0900 "Stephen J. Turnbull" wrote: > > The argument that a "read-only, no cherrypicking by committers" repo > is nothing but a better tarball is valid, but as I say, AFAICS the > expected gain is pretty marginal. The conflict here is not Larry's > process, it's the decision to make an ambitious release on a short > time schedule. I don't really buy the "short time schedule" argument. The delay between 3.3 and 3.4 is ~18 months as usual. Regards Antoine. From solipsis at pitrou.net Thu Feb 20 12:54:26 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 20 Feb 2014 12:54:26 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> <5305746C.7000309@hastings.org> Message-ID: <20140220125426.023e3ee5@fsol> On Wed, 19 Feb 2014 19:20:12 -0800 Larry Hastings wrote: > > As for a "user beware" clone: I worry about providing anything that > looks/tastes/smells like a repo. Someone could still inadvertently push > those revisions back to trunk, and then we'd have a real mess on our > hands. If you're using a separate named branch for your release work, then the hg.python.org hooks will forbid pushing them back to the main repo. Regards Antoine. From jimjjewett at gmail.com Thu Feb 20 16:38:55 2014 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 20 Feb 2014 10:38:55 -0500 Subject: [Python-Dev] Extent of post-rc churn Message-ID: http://midwinter.com/~larry/3.4.status/merge.status.html lists enough changes that it sounds more like a bugfix release than "just a few last tweaks after the rc". It would probably help if the what's-new-in-rc2 explicitly mentioned that asyncio is new and provisional with 3.4, and listed its changes in a separate subsection, so that the "final tweaks to something I might already be using" section would be less intimidating. -jJ From martin at v.loewis.de Thu Feb 20 16:38:04 2014 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 20 Feb 2014 16:38:04 +0100 Subject: [Python-Dev] A misredirected ticket link in hg.python.org/cpython In-Reply-To: References: Message-ID: <20140220163804.Horde.tdjLozgDJyb1peHROIQCoQ1@webmail.df.eu> Quoting Vajrasky Kok : > Go to any commit link in hg.python.org/cpython, for example > http://hg.python.org/cpython/rev/d11ca14c9a61. > > You have this commit message: > > Issue #6815: os.path.expandvars() now supports non-ASCII Unicode > environment variables names and values. [#6815] > > The [#6815] link is going to http://bugs.python.org/6815. If you > follow that link, it redirects to http://legacy.python.org/sf/ and you > get message: You did not provide a report number. The link should be > http://bugs.python.org/issue6815. The bug really is on www.python.org, which (now) redirects /sf/6815 to http://legacy.python.org/sf/, i.e. it drops the issue number there. Regards, Martin From jimjjewett at gmail.com Thu Feb 20 17:18:18 2014 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 20 Feb 2014 11:18:18 -0500 Subject: [Python-Dev] DB-API v2.1 or v3 [inspired by: python 3 niggle: None < 1 raises TypeError] Message-ID: I personally regret that sorting isn't safe, but that ship has sailed. There is practicality benefit in making None compare to everything, just as C and Java do with null pointers -- but it is too late to do by default. Adding a keyword to sorted might be nice -- but then shouldn't it also be added to other sorts, and maybe max/min? It might just be trading one sort of mess for another. What *can* reasonably be changed is the DB-API. Why not just specify that the DB type objects themselves should handle comparison to None? http://www.python.org/dev/peps/pep-0249/#type-objects-and-constructors -jJ From larry at hastings.org Thu Feb 20 22:00:59 2014 From: larry at hastings.org (Larry Hastings) Date: Thu, 20 Feb 2014 13:00:59 -0800 Subject: [Python-Dev] Second preview of 3.4.0rc2 is up Message-ID: <53066D0B.6020206@hastings.org> This time I was a lot more careful about Misc/NEWS items. The graft process likes to link in improper changes. So every time I have a graft merge collision, I now recheck what the revision I'm merging has done, and I only keep Misc/NEWS entries that are relevant. I just realized that I based the 3.4 branch on the wrong revision. I had used the last revision before I started tagging 3.4.0rc1 ( 6343bdbb7085) when I should be using the last revision in 3.4.0rc1 before merging back into trunk ( e64ae8b82672). So I get to do it all over from scratch, *again*, for the next attempt. Hooray! You can find the current status and a fresh tarball here: http://midwinter.com/~larry/3.4.status/ Good luck, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Feb 20 22:15:39 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 20 Feb 2014 22:15:39 +0100 Subject: [Python-Dev] Second preview of 3.4.0rc2 is up In-Reply-To: <53066D0B.6020206@hastings.org> References: <53066D0B.6020206@hastings.org> Message-ID: Hi, Thanks Larry for being our release manager. How can we help you? Sorry for giving you too much work with asyncio changes :-) Python 3.4 is the largest release in term of new features since Python 3. To give you an overview of new features, 8 new modules were added between Python 2.7 and 3.3. As many modules have been added between Python 3.3 and 3.4 as between Python 2.7 and Python 3.3! Python 3.4 has 7 new modules: http://techs.enovance.com/6521/openstack_python3 In Python 3.4: 14 PEP have been accepted and implemented. And I didn't count Larry's PEP which still a draft (but its status should be final IMO). Victor From doko at ubuntu.com Thu Feb 20 23:37:08 2014 From: doko at ubuntu.com (Matthias Klose) Date: Thu, 20 Feb 2014 23:37:08 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> Message-ID: <53068394.2030006@ubuntu.com> Am 19.02.2014 22:18, schrieb Nick Coghlan: > and the Ubuntu 14.04 deadline restricts our ability to add a 3rd rc. well, I think it would be wrong to restrict that for only that reason. I did object to delay the release cycle a second time for completing a feature. If the release has to be delayed for QA reasons, that's a good reason. Having a Debian background myself ;) I'm fine to ship with a rc3 too, and update it post-release, after wading through distribution bureaucracy ... but please don't use this as an example of a distribution influencing an upstream. There are "better" examples for this :-/ Matthias From ncoghlan at gmail.com Fri Feb 21 00:40:43 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Feb 2014 09:40:43 +1000 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <53068394.2030006@ubuntu.com> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <53068394.2030006@ubuntu.com> Message-ID: On 21 Feb 2014 08:38, "Matthias Klose" wrote: > > Am 19.02.2014 22:18, schrieb Nick Coghlan: > > and the Ubuntu 14.04 deadline restricts our ability to add a 3rd rc. > > well, I think it would be wrong to restrict that for only that reason. I did > object to delay the release cycle a second time for completing a feature. If > the release has to be delayed for QA reasons, that's a good reason. Having a > Debian background myself ;) > > I'm fine to ship with a rc3 too, and update it post-release, after wading > through distribution bureaucracy ... but please don't use this as an example of > a distribution influencing an upstream. There are "better" examples for this :-/ Thanks for the clarification. That said, supporting someone else's external deadline can definitely help with resisting the urge to allow "just one more little adjustment" :) Cheers, Nick. > > Matthias > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 21 04:15:59 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Feb 2014 14:15:59 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions Message-ID: PEP: 463 Title: Exception-catching expressions Version: $Revision$ Last-Modified: $Date$ Author: Chris Angelico Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Feb-2014 Python-Version: 3.5 Post-History: 16-Feb-2014, 21-Feb-2014 Abstract ======== Just as PEP 308 introduced a means of value-based conditions in an expression, this system allows exception-based conditions to be used as part of an expression. Motivation ========== A number of functions and methods have parameters which will cause them to return a specified value instead of raising an exception. The current system is ad-hoc and inconsistent, and requires that each function be individually written to have this functionality; not all support this. * dict.get(key, default) - second positional argument in place of KeyError * next(iter, default) - second positional argument in place of StopIteration * list.pop() - no way to return a default * seq[index] - no way to handle a bounds error * min(sequence, default=default) - keyword argument in place of ValueError * sum(sequence, start=default) - slightly different but can do the same job * statistics.mean(data) - no way to handle an empty iterator Rationale ========= The current system requires that a function author predict the need for a default, and implement support for it. If this is not done, a full try/except block is needed. Since try/except is a statement, it is impossible to catch exceptions in the middle of an expression. Just as if/else does for conditionals and lambda does for function definitions, so does this allow exception catching in an expression context. This provides a clean and consistent way for a function to provide a default: it simply raises an appropriate exception, and the caller catches it. With some situations, an LBYL technique can be used (checking if some sequence has enough length before indexing into it, for instance). This is not safe in all cases, but as it is often convenient, programmers will be tempted to sacrifice the safety of EAFP in favour of the notational brevity of LBYL. Additionally, some LBYL techniques (eg involving getattr with three arguments) warp the code into looking like literal strings rather than attribute lookup, which can impact readability. A convenient EAFP notation solves all of this. There's no convenient way to write a helper function to do this; the nearest is something ugly using either lambda:: def except_(expression, exception_list, default): try: return expression() except exception_list: return default() value = except_(lambda: 1/x, ZeroDivisionError, lambda: float("nan")) which is clunky, and unable to handle multiple exception clauses; or eval:: def except_(expression, exception_list, default): try: return eval(expression, globals_of_caller(), locals_of_caller()) except exception_list as exc: l = locals_of_caller().copy() l['exc'] = exc return eval(default, globals_of_caller(), l) def globals_of_caller(): return sys._getframe(2).f_globals def locals_of_caller(): return sys._getframe(2).f_locals value = except_("""1/x""",ZeroDivisionError,""" "Can't divide by zero" """) which is even clunkier, and relies on implementation-dependent hacks. (Writing globals_of_caller() and locals_of_caller() for interpreters other than CPython is left as an exercise for the reader.) Raymond Hettinger `expresses`__ a desire for such a consistent API. Something similar has been `requested`__ `multiple`__ `times`__ in the past. __ https://mail.python.org/pipermail/python-ideas/2014-February/025443.html __ https://mail.python.org/pipermail/python-ideas/2013-March/019760.html __ https://mail.python.org/pipermail/python-ideas/2009-August/005441.html __ https://mail.python.org/pipermail/python-ideas/2008-August/001801.html Proposal ======== Just as the 'or' operator and the three part 'if-else' expression give short circuiting methods of catching a falsy value and replacing it, this syntax gives a short-circuiting method of catching an exception and replacing it. This currently works:: lst = [1, 2, None, 3] value = lst[2] or "No value" The proposal adds this:: lst = [1, 2] value = lst[2] except IndexError: "No value" Specifically, the syntax proposed is:: expr except exception_list: default where expr, exception_list, and default are all expressions. First, expr is evaluated. If no exception is raised, its value is the value of the overall expression. If any exception is raised, exception_list is evaluated, and should result in either a type or a tuple, just as with the statement form of try/except. Any matching exception will result in the corresponding default expression being evaluated and becoming the value of the expression. As with the statement form of try/except, non-matching exceptions will propagate upward. Note that the current proposal does not allow the exception object to be captured. Where this is needed, the statement form must be used. (See below for discussion and elaboration on this.) This ternary operator would be between lambda and if/else in precedence. Consider this example of a two-level cache:: for key in sequence: x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key))) # do something with x This cannot be rewritten as:: x = lvl1.get(key, lvl2.get(key, f(key))) which, despite being shorter, defeats the purpose of the cache, as it must calculate a default value to pass to get(). The .get() version calculates backwards; the exception-testing version calculates forwards, as would be expected. The nearest useful equivalent would be:: x = lvl1.get(key) or lvl2.get(key) or f(key) which depends on the values being nonzero, as well as depending on the cache object supporting this functionality. Alternative Proposals ===================== Discussion on python-ideas brought up the following syntax suggestions:: value = expr except default if Exception [as e] value = expr except default for Exception [as e] value = expr except default from Exception [as e] value = expr except Exception [as e] return default value = expr except (Exception [as e]: default) value = expr except Exception [as e] try default value = expr except Exception [as e] continue with default value = default except Exception [as e] else expr value = try expr except Exception [as e]: default value = expr except default # Catches anything value = expr except(Exception) default # Catches only the named type(s) value = default if expr raise Exception value = expr or else default if Exception value = expr except Exception [as e] -> default value = expr except Exception [as e] pass default It has also been suggested that a new keyword be created, rather than reusing an existing one. Such proposals fall into the same structure as the last form, but with a different keyword in place of 'pass'. Suggestions include 'then', 'when', and 'use'. Also, in the context of the "default if expr raise Exception" proposal, it was suggested that a new keyword "raises" be used. All forms involving the 'as' capturing clause have been deferred from this proposal in the interests of simplicity, but are preserved in the table above as an accurate record of suggestions. Open Issues =========== Parentheses around the entire expression ---------------------------------------- Generator expressions require parentheses, unless they would be strictly redundant. Ambiguities with except expressions could be resolved in the same way, forcing nested except-in-except trees to be correctly parenthesized and requiring that the outer expression be clearly delineated. `Steven D'Aprano elaborates on the issue.`__ __ https://mail.python.org/pipermail/python-ideas/2014-February/025647.html Example usage ============= For each example, an approximately-equivalent statement form is given, to show how the expression will be parsed. These are not always strictly equivalent, but will accomplish the same purpose. It is NOT safe for the interpreter to translate one into the other. A number of these examples are taken directly from the Python standard library, with file names and line numbers correct as of early Feb 2014. Many of these patterns are extremely common. Retrieve an argument, defaulting to None:: cond = args[1] except IndexError: None # Lib/pdb.py:803: try: cond = args[1] except IndexError: cond = None Fetch information from the system if available:: pwd = os.getcwd() except OSError: None # Lib/tkinter/filedialog.py:210: try: pwd = os.getcwd() except OSError: pwd = None Attempt a translation, falling back on the original:: e.widget = self._nametowidget(W) except KeyError: W # Lib/tkinter/__init__.py:1222: try: e.widget = self._nametowidget(W) except KeyError: e.widget = W Read from an iterator, continuing with blank lines once it's exhausted:: line = readline() except StopIteration: '' # Lib/lib2to3/pgen2/tokenize.py:370: try: line = readline() except StopIteration: line = '' Retrieve platform-specific information (note the DRY improvement); this particular example could be taken further, turning a series of separate assignments into a single large dict initialization:: # sys.abiflags may not be defined on all platforms. _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: '' # Lib/sysconfig.py:529: try: _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: # sys.abiflags may not be defined on all platforms. _CONFIG_VARS['abiflags'] = '' Retrieve an indexed item, defaulting to None (similar to dict.get):: def getNamedItem(self, name): return self._attrs[name] except KeyError: None # Lib/xml/dom/minidom.py:573: def getNamedItem(self, name): try: return self._attrs[name] except KeyError: return None Translate numbers to names, falling back on the numbers:: g = grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: tarinfo.uid # Lib/tarfile.py:2198: try: g = grp.getgrnam(tarinfo.gname)[2] except KeyError: g = tarinfo.gid try: u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: u = tarinfo.uid Perform some lengthy calculations in EAFP mode, handling division by zero as a sort of sticky NaN:: value = calculate(x) except ZeroDivisionError: float("nan") try: value = calculate(x) except ZeroDivisionError: value = float("nan") Calculate the mean of a series of numbers, falling back on zero:: value = statistics.mean(lst) except statistics.StatisticsError: 0 try: value = statistics.mean(lst) except statistics.StatisticsError: value = 0 Retrieving a message from either a cache or the internet, with auth check:: logging.info("Message shown to user: %s",((cache[k] except LookupError: (backend.read(k) except OSError: 'Resource not available') ) if check_permission(k) else 'Access denied' ) except BaseException: "This is like a bare except clause") try: if check_permission(k): try: _ = cache[k] except LookupError: try: _ = backend.read(k) except OSError: _ = 'Resource not available' else: _ = 'Access denied' except BaseException: _ = "This is like a bare except clause" logging.info("Message shown to user: %s", _) Looking up objects in a sparse list of overrides:: (overrides[x] or default except IndexError: default).ping() try: (overrides[x] or default).ping() except IndexError: default.ping() Narrowing of exception-catching scope ------------------------------------- The following examples, taken directly from Python's standard library, demonstrate how the scope of the try/except can be conveniently narrowed. To do this with the statement form of try/except would require a temporary variable, but it's far cleaner as an expression. Lib/ipaddress.py:343:: try: ips.append(ip.ip) except AttributeError: ips.append(ip.network_address) Becomes:: ips.append(ip.ip except AttributeError: ip.network_address) The expression form is nearly equivalent to this:: try: _ = ip.ip except AttributeError: _ = ip.network_address ips.append(_) Lib/tempfile.py:130:: try: dirlist.append(_os.getcwd()) except (AttributeError, OSError): dirlist.append(_os.curdir) Becomes:: dirlist.append(_os.getcwd() except (AttributeError, OSError): _os.curdir) Lib/asyncore.py:264:: try: status.append('%s:%d' % self.addr) except TypeError: status.append(repr(self.addr)) Becomes:: status.append('%s:%d' % self.addr except TypeError: repr(self.addr)) Comparisons with other languages ================================ (With thanks to Andrew Barnert for compiling this section.) `Ruby's`__ "begin?rescue?rescue?else?ensure?end" is an expression (potentially with statements inside it). It has the equivalent of an "as" clause, and the equivalent of bare except. And it uses no punctuation or keyword between the bare except/exception class/exception class with as clause and the value. (And yes, it's ambiguous unless you understand Ruby's statement/expression rules.) __ http://www.skorks.com/2009/09/ruby-exceptions-and-exception-handling/ :: x = begin computation() rescue MyException => e default(e) end; x = begin computation() rescue MyException default() end; x = begin computation() rescue default() end; x = begin computation() rescue MyException default() rescue OtherException other() end; In terms of this PEP:: x = computation() except MyException as e default(e) x = computation() except MyException default(e) x = computation() except default(e) x = computation() except MyException default() except OtherException other() `Erlang`__ has a try expression that looks like this:: __ http://erlang.org/doc/reference_manual/expressions.html#id79284 x = try computation() catch MyException:e -> default(e) end; x = try computation() catch MyException:e -> default(e); OtherException:e -> other(e) end; The class and "as" name are mandatory, but you can use "_" for either. There's also an optional "when" guard on each, and a "throw" clause that you can catch, which I won't get into. To handle multiple exceptions, you just separate the clauses with semicolons, which I guess would map to commas in Python. So:: x = try computation() except MyException as e -> default(e) x = try computation() except MyException as e -> default(e), OtherException as e->other_default(e) Erlang also has a "catch" expression, which, despite using the same keyword, is completely different, and you don't want to know about it. The ML family has two different ways of dealing with this, "handle" and "try"; the difference between the two is that "try" pattern-matches the exception, which gives you the effect of multiple except clauses and as clauses. In either form, the handler clause is punctuated by "=>" in some dialects, "->" in others. To avoid confusion, I'll write the function calls in Python style. Here's `SML's`__ "handle":: __ http://www.cs.cmu.edu/~rwh/introsml/core/exceptions.htm let x = computation() handle MyException => default();; Here's `OCaml's`__ "try":: __ http://www2.lib.uchicago.edu/keith/ocaml-class/exceptions.html let x = try computation() with MyException explanation -> default(explanation);; let x = try computation() with MyException(e) -> default(e) | MyOtherException() -> other_default() | (e) -> fallback(e);; In terms of this PEP, these would be something like:: x = computation() except MyException => default() x = try computation() except MyException e -> default() x = (try computation() except MyException as e -> default(e) except MyOtherException -> other_default() except BaseException as e -> fallback(e)) Many ML-inspired but not-directly-related languages from academia mix things up, usually using more keywords and fewer symbols. So, the `Oz`__ would map to Python as:: __ http://mozart.github.io/mozart-v1/doc-1.4.0/tutorial/node5.html x = try computation() catch MyException as e then default(e) Many Lisp-derived languages, like `Clojure,`__ implement try/catch as special forms (if you don't know what that means, think function-like macros), so you write, effectively:: __ http://clojure.org/special_forms#Special%20Forms--(try%20expr*%20catch-clause*%20finally-clause?) try(computation(), catch(MyException, explanation, default(explanation))) try(computation(), catch(MyException, explanation, default(explanation)), catch(MyOtherException, explanation, other_default(explanation))) In Common Lisp, this is done with a slightly clunkier `"handler-case" macro,`__ but the basic idea is the same. __ http://clhs.lisp.se/Body/m_hand_1.htm The Lisp style is, surprisingly, used by some languages that don't have macros, like Lua, where `xpcall`__ takes functions. Writing lambdas Python-style instead of Lua-style:: __ http://www.gammon.com.au/scripts/doc.php?lua=xpcall x = xpcall(lambda: expression(), lambda e: default(e)) This actually returns (true, expression()) or (false, default(e)), but I think we can ignore that part. Haskell is actually similar to Lua here (except that it's all done with monads, of course):: x = do catch(lambda: expression(), lambda e: default(e)) You can write a pattern matching expression within the function to decide what to do with it; catching and re-raising exceptions you don't want is cheap enough to be idiomatic. But Haskell infixing makes this nicer:: x = do expression() `catch` lambda: default() x = do expression() `catch` lambda e: default(e) And that makes the parallel between the lambda colon and the except colon in the proposal much more obvious:: x = expression() except Exception: default() x = expression() except Exception as e: default(e) `Tcl`__ has the other half of Lua's xpcall; catch is a function which returns true if an exception was caught, false otherwise, and you get the value out in other ways. And it's all built around the the implicit quote-and-exec that everything in Tcl is based on, making it even harder to describe in Python terms than Lisp macros, but something like:: __ http://wiki.tcl.tk/902 if {[ catch("computation()") "explanation"]} { default(explanation) } `Smalltalk`__ is also somewhat hard to map to Python. The basic version would be:: __ http://smalltalk.gnu.org/wiki/exceptions x := computation() on:MyException do:default() ? but that's basically Smalltalk's passing-arguments-with-colons syntax, not its exception-handling syntax. Deferred sub-proposals ====================== Multiple except clauses ----------------------- An examination of use-cases shows that this is not needed as often as it would be with the statement form, and as its syntax is a point on which consensus has not been reached, the entire feature is deferred. In order to ensure compatibility with future versions, ensure that any consecutive except operators are parenthesized to guarantee the interpretation you expect. Multiple 'except' keywords can be used, and they will all catch exceptions raised in the original expression (only):: # Will catch any of the listed exceptions thrown by expr; # any exception thrown by a default expression will propagate. value = (expr except Exception1 [as e]: default1 except Exception2 [as e]: default2 # ... except ExceptionN [as e]: defaultN ) Using parentheses to force an alternative interpretation works as expected:: # Will catch an Exception2 thrown by either expr or default1 value = ( (expr except Exception1: default1) except Exception2: default2 ) # Will catch an Exception2 thrown by default1 only value = (expr except Exception1: (default1 except Exception2: default2) ) This last form is confusing and should be discouraged by PEP 8, but it is syntactically legal: you can put any sort of expression inside a ternary-except; ternary-except is an expression; therefore you can put a ternary-except inside a ternary-except. Open question: Where there are multiple except clauses, should they be separated by commas? It may be easier for the parser, that way:: value = (expr except Exception1 [as e]: default1, except Exception2 [as e]: default2, # ... except ExceptionN [as e]: defaultN, ) with an optional comma after the last, as per tuple rules. Downside: Omitting the comma would be syntactically valid, and would have almost identical semantics, but would nest the entire preceding expression in its exception catching rig - a matching exception raised in the default clause would be caught by the subsequent except clause. As this difference is so subtle, it runs the risk of being a major bug magnet. As a mitigation of this risk, this form:: value = expr except Exception1: default1 except Exception2: default2 could be syntactically forbidden, and parentheses required if the programmer actually wants that behaviour:: value = (expr except Exception1: default1) except Exception2: default2 This would prevent the accidental omission of a comma from changing the expression's meaning. Capturing the exception object ------------------------------ In a try/except block, the use of 'as' to capture the exception object creates a local name binding, and implicitly deletes that binding in a finally clause. As 'finally' is not a part of this proposal (see below), this makes it tricky to describe; also, this use of 'as' gives a way to create a name binding in an expression context. Should the default clause have an inner scope in which the name exists, shadowing anything of the same name elsewhere? Should it behave the same way the statement try/except does, and unbind the name? Should it bind the name and leave it bound? (Almost certainly not; this behaviour was changed in Python 3 for good reason.) Additionally, this syntax would allow a convenient way to capture exceptions in interactive Python; returned values are captured by "_", but exceptions currently are not. This could be spelled: >>> expr except Exception as e: e (The inner scope idea is tempting, but currently CPython handles list comprehensions with a nested function call, as this is considered easier. It may be of value to simplify both comprehensions and except expressions, but that is a completely separate proposal to this PEP; alternatively, it may be better to stick with what's known to work. `Nick Coghlan elaborates.`__) __ https://mail.python.org/pipermail/python-ideas/2014-February/025702.html An examination of the Python standard library shows that, while the use of 'as' is fairly common (occurring in roughly one except clause in five), it is extremely *uncommon* in the cases which could logically be converted into the expression form. Its few uses can simply be left unchanged. Consequently, in the interests of simplicity, the 'as' clause is not included in this proposal. A subsequent Python version can add this without breaking any existing code, as 'as' is already a keyword. One example where this could possibly be useful is Lib/imaplib.py:568:: try: typ, dat = self._simple_command('LOGOUT') except: typ, dat = 'NO', ['%s: %s' % sys.exc_info()[:2]] This could become:: typ, dat = (self._simple_command('LOGOUT') except BaseException as e: ('NO', '%s: %s' % (type(e), e))) Or perhaps some other variation. This is hardly the most compelling use-case, but an intelligent look at this code could tidy it up significantly. In the absence of further examples showing any need of the exception object, I have opted to defer indefinitely the recommendation. Rejected sub-proposals ====================== finally clause -------------- The statement form try... finally or try... except... finally has no logical corresponding expression form. Therefore the finally keyword is not a part of this proposal, in any way. Bare except having different meaning ------------------------------------ With several of the proposed syntaxes, omitting the exception type name would be easy and concise, and would be tempting. For convenience's sake, it might be advantageous to have a bare 'except' clause mean something more useful than "except BaseException". Proposals included having it catch Exception, or some specific set of "common exceptions" (subclasses of a new type called ExpressionError), or have it look for a tuple named ExpressionError in the current scope, with a built-in default such as (ValueError, UnicodeError, AttributeError, EOFError, IOError, OSError, LookupError, NameError, ZeroDivisionError). All of these were rejected, for severa reasons. * First and foremost, consistency with the statement form of try/except would be broken. Just as a list comprehension or ternary if expression can be explained by "breaking it out" into its vertical statement form, an expression-except should be able to be explained by a relatively mechanical translation into a near-equivalent statement. Any form of syntax common to both should therefore have the same semantics in each, and above all should not have the subtle difference of catching more in one than the other, as it will tend to attract unnoticed bugs. * Secondly, the set of appropriate exceptions to catch would itself be a huge point of contention. It would be impossible to predict exactly which exceptions would "make sense" to be caught; why bless some of them with convenient syntax and not others? * And finally (this partly because the recommendation was that a bare except should be actively encouraged, once it was reduced to a "reasonable" set of exceptions), any situation where you catch an exception you don't expect to catch is an unnecessary bug magnet. Consequently, the use of a bare 'except' is down to two possibilities: either it is syntactically forbidden in the expression form, or it is permitted with the exact same semantics as in the statement form (namely, that it catch BaseException and be unable to capture it with 'as'). Bare except clauses ------------------- PEP 8 rightly advises against the use of a bare 'except'. While it is syntactically legal in a statement, and for backward compatibility must remain so, there is little value in encouraging its use. In an expression except clause, "except:" is a SyntaxError; use the equivalent long-hand form "except BaseException:" instead. A future version of Python MAY choose to reinstate this, which can be done without breaking compatibility. Parentheses around the except clauses ------------------------------------- Should it be legal to parenthesize the except clauses, separately from the expression that could raise? Example:: value = expr ( except Exception1 [as e]: default1 except Exception2 [as e]: default2 # ... except ExceptionN [as e]: defaultN ) This is more compelling when one or both of the deferred sub-proposals of multiple except clauses and/or exception capturing is included. In their absence, the parentheses would be thus:: value = expr except ExceptionType: default value = expr (except ExceptionType: default) The advantage is minimal, and the potential to confuse a reader into thinking the except clause is separate from the expression, or into thinking this is a function call, makes this non-compelling. The expression can, of course, be parenthesized if desired, as can the default:: value = (expr) except ExceptionType: (default) Short-hand for "except: pass" ----------------------------- The following was been suggested as a similar short-hand, though not technically an expression:: statement except Exception: pass try: statement except Exception: pass For instance, a common use-case is attempting the removal of a file:: os.unlink(some_file) except OSError: pass There is an equivalent already in Python 3.4, however, in contextlib:: from contextlib import suppress with suppress(OSError): os.unlink(some_file) As this is already a single line (or two with a break after the colon), there is little need of new syntax and a confusion of statement vs expression to achieve this. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From ethan at stoneleaf.us Fri Feb 21 05:54:03 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 20 Feb 2014 20:54:03 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <5306DBEB.1050804@stoneleaf.us> On 02/20/2014 07:15 PM, Chris Angelico wrote: > PEP: 463 > Title: Exception-catching expressions > Version: $Revision$ > Last-Modified: $Date$ > Author: Chris Angelico > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 15-Feb-2014 > Python-Version: 3.5 > Post-History: 16-Feb-2014, 21-Feb-2014 Nice work with the PEP, Chris! +1 to general idea. (I'll vote on the syntax when the bikeshedding begins ;). -- ~Ethan~ From rosuav at gmail.com Fri Feb 21 06:00:38 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Feb 2014 16:00:38 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5306DBEB.1050804@stoneleaf.us> References: <5306DBEB.1050804@stoneleaf.us> Message-ID: On Fri, Feb 21, 2014 at 3:54 PM, Ethan Furman wrote: > (I'll vote on the syntax when the bikeshedding begins ;). Go get the keys to the time machine! Bikeshedding begins a week ago. ChrisA From ethan at stoneleaf.us Fri Feb 21 05:58:15 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 20 Feb 2014 20:58:15 -0800 Subject: [Python-Dev] What is 'default' point to now? Message-ID: <5306DCE7.6000207@stoneleaf.us> Now that Larry is working on the 3.4.0 branch away from default, what is default pointing to? 3.4.1 or 3.5? -- ~Ethan~ From ethan at stoneleaf.us Fri Feb 21 06:26:25 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 20 Feb 2014 21:26:25 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <5306DBEB.1050804@stoneleaf.us> Message-ID: <5306E381.8020505@stoneleaf.us> On 02/20/2014 09:00 PM, Chris Angelico wrote: > On Fri, Feb 21, 2014 at 3:54 PM, Ethan Furman wrote: >> (I'll vote on the syntax when the bikeshedding begins ;). > > Go get the keys to the time machine! Bikeshedding begins a week ago. Oh, no, not on PyDev it hasn't! Get ready for round 2! -- ~Ethan~ From tjreedy at udel.edu Fri Feb 21 06:56:36 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 21 Feb 2014 00:56:36 -0500 Subject: [Python-Dev] What is 'default' point to now? In-Reply-To: <5306DCE7.6000207@stoneleaf.us> References: <5306DCE7.6000207@stoneleaf.us> Message-ID: On 2/20/2014 11:58 PM, Ethan Furman wrote: > Now that Larry is working on the 3.4.0 branch away from default, what is > default pointing to? 3.4.1 or 3.5? Until a 3.4 branch is split off, default is effectively 3.4.1, which means bugfixes only. -- Terry Jan Reedy From anju55tiwari at gmail.com Fri Feb 21 08:06:05 2014 From: anju55tiwari at gmail.com (anju Tiwari) Date: Fri, 21 Feb 2014 12:36:05 +0530 Subject: [Python-Dev] rpm needs python Message-ID: Hi all, I have two version of python 2.4 and 2.7. By default python version is 2.4 . I want to install need to install some rpm which needs python 2.7 interpreter. how can I enable 2.7 interpreter for only those packages which are requiring python 2.7, I don't want to change my default python version(2.4). -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Fri Feb 21 09:12:58 2014 From: larry at hastings.org (Larry Hastings) Date: Fri, 21 Feb 2014 00:12:58 -0800 Subject: [Python-Dev] Third preview of 3.4.0rc2 is up Message-ID: <53070A8A.8080101@hastings.org> We're now up to 66 cherry-picks. 75%+ of those are asyncio changes. Hopefully I won't have to redo the cherry-picking from scratch again, I can just pile more picks on top of the ones I've got. As before you'll find the results here: http://midwinter.com/~larry/3.4.status/ //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Feb 21 09:48:02 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 21 Feb 2014 03:48:02 -0500 Subject: [Python-Dev] rpm needs python In-Reply-To: References: Message-ID: On 2/21/2014 2:06 AM, anju Tiwari wrote: > I have two version of python 2.4 and 2.7. > By default python version is 2.4 . I want to install need to install > some rpm > which needs python 2.7 interpreter. how can I enable 2.7 interpreter for > only those > packages which are requiring python 2.7, I don?t want to change my > default python version(2.4). Ask usage questions like this on python-list or similar user forums. pydev is for development *of* python, not *with* python. -- Terry Jan Reedy From ethan at stoneleaf.us Fri Feb 21 09:36:09 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Feb 2014 00:36:09 -0800 Subject: [Python-Dev] cherry pick a change to Enum Message-ID: <53070FF9.9050907@stoneleaf.us> Greetings, all! Larry asked me to bring this to PyDev, so here it is. The issues involved are: http://bugs.python.org/issue20534 -> already in RC1 http://bugs.python.org/issue20653 -> hoping to get cherry-picked Background ========== When I put Enum together, my knowledge of pickle and its protocols was dreadfully weak. As a result, I didn't make the best choice in implementing it [1]. Just before RC1 was cut, Serhiy discovered a problem with pickling Enums at protocol four, and with his help I fixed that (this is the part that is already in RC1). This fix consisted primarily of adding a __reduce_ex__ to the Enum class, and also checking for all the pickle protocols in a mixed-in type [2], but I left __getnewargs__ in. Not only that, and worse, I wasn't checking to see if the new subclass had its own __getnewargs__ or __reduce_ex__ before I invalidated the class for pickling [3]. This means that even though a subclass could have its own __reduce_ex__ to make pickling/unpickling possible, it will be ignored. Problem ======= All the previous changes to disallow something have been for strategic reasons (the usual ones being either consistency, or "it doesn't make sense for an enum member/class to behave that way"), but this is effectively disallowing a user to specify how their enums should be pickled, and I am unaware of any reason why this should be. Discussion ========== Is there a valid reason to not allow a user to modify how their enums are pickled? If not, should we put the change in RC2 / Final? The main reason I'm pushing for this is that Enum is still new, but after 3.4.0 is cut we then have to deal with backwards compatibility issues. Thanks for your time and thoughts. -- ~Ethan~ [1] I chose __getnewargs__, with the result that originally Enum didn't support pickle protocols 0 or 1, and when 4 was added later it had trouble with some types of Enum subclasses (tuples, to be exact). [2] Those protocol methods being __reduce__, __reduce_ex__, __getnewargs__, and __getnewargs_ex__. [3] This logic error has been there from the beginning, it did not sneak in in the last patch before RC1. From nad at acm.org Fri Feb 21 10:30:04 2014 From: nad at acm.org (Ned Deily) Date: Fri, 21 Feb 2014 01:30:04 -0800 Subject: [Python-Dev] Third preview of 3.4.0rc2 is up References: <53070A8A.8080101@hastings.org> Message-ID: In article <53070A8A.8080101 at hastings.org>, Larry Hastings wrote: > As before you'll find the results here: > > http://midwinter.com/~larry/3.4.status/ Status says that: eef7899ea7ab Doc: do not rely on checked-out Sphinx toolchain from svn.python.org anymore is unmerged, which is what Georg and I agreed upon in Issue20661. Yet, in the current python.3.4.2014.02.21.00.07.42.tgz tarball, that change appears to be present and, as such, causes installer builds to fail. -- Ned Deily, nad at acm.org From andreas.roehler at online.de Fri Feb 21 10:38:23 2014 From: andreas.roehler at online.de (=?ISO-8859-1?Q?Andreas_R=F6hler?=) Date: Fri, 21 Feb 2014 10:38:23 +0100 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <53071E8F.2000105@online.de> Am 20.02.2014 02:24, schrieb Stephen J. Turnbull: [ ... ] > Sure, but it *doesn't* help in knowing which ones are *correctly* > addressed. These *are* ambitious changes; some of the remaining bugs > may be very deep. The obvious fixes may do more harm than good. Ie, > "more eyes" is (a) mostly a fallacy (as Heinlein put it, the wisdom of > a group is less than or equal to the maximum of the wisdom of the > members) and (b) in any case the "more eyes" effect is diluted if > people are deliberately looking at different parts of the code. > All depends from the way, a group is organized, composed etc. What might stiffle results in a group for example is hierarchy, resp. a fear to get chastised by the leader. After all I'm not in position to make a guess here - but rather would expect much higher results by groups-coop than from most gifted single person among. From larry at hastings.org Fri Feb 21 11:06:07 2014 From: larry at hastings.org (Larry Hastings) Date: Fri, 21 Feb 2014 02:06:07 -0800 Subject: [Python-Dev] Third preview of 3.4.0rc2 is up In-Reply-To: References: <53070A8A.8080101@hastings.org> Message-ID: <5307250F.3040109@hastings.org> On 02/21/2014 01:30 AM, Ned Deily wrote: > In article <53070A8A.8080101 at hastings.org>, > Larry Hastings wrote: >> As before you'll find the results here: >> >> http://midwinter.com/~larry/3.4.status/ > Status says that: > > eef7899ea7ab Doc: do not rely on checked-out Sphinx toolchain from > svn.python.org anymore > > is unmerged, which is what Georg and I agreed upon in Issue20661. Yet, > in the current python.3.4.2014.02.21.00.07.42.tgz tarball, that change > appears to be present and, as such, causes installer builds to fail. > Whoopsie! My local branch is actually correct. But! I effectively did this in my automation: % hg clone 3.4 python-{time} % cd python-{time} % rm -rf .hg* .bzr* .git* % cd .. % tar cvfz python-{time}.tgz python-{time} Can you spot the error? That's right: when you clone, the clone always starts out in "default". So all the tarballs I've published so far have been wrong. ARGH. Sorry! I've added a "hg branch 3.4" in a judicious spot in my automation. The resulting tarball doesn't have the sphinx toolchain change. Look for a new tarball in a few minutes, once "make test" finishes, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Feb 21 11:29:51 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Feb 2014 11:29:51 +0100 Subject: [Python-Dev] cherry pick a change to Enum References: <53070FF9.9050907@stoneleaf.us> Message-ID: <20140221112951.67d3f212@fsol> On Fri, 21 Feb 2014 00:36:09 -0800 Ethan Furman wrote: > > Discussion > ========== > > Is there a valid reason to not allow a user to modify how their enums are pickled? > > If not, should we put the change in RC2 / Final? The main reason I'm pushing for this is that Enum is still new, but > after 3.4.0 is cut we then have to deal with backwards compatibility issues. I think this should be cherry-picked. (but see my comment on the issue) Regards Antoine. From arfrever.fta at gmail.com Fri Feb 21 11:30:20 2014 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Fri, 21 Feb 2014 11:30:20 +0100 Subject: [Python-Dev] Third preview of 3.4.0rc2 is up In-Reply-To: <5307250F.3040109@hastings.org> References: <53070A8A.8080101@hastings.org> <5307250F.3040109@hastings.org> Message-ID: <201402211130.22453.Arfrever.FTA@gmail.com> 2014-02-21 11:06 Larry Hastings napisa?(a): > On 02/21/2014 01:30 AM, Ned Deily wrote: > > In article <53070A8A.8080101 at hastings.org>, > > Larry Hastings wrote: > >> As before you'll find the results here: > >> > >> http://midwinter.com/~larry/3.4.status/ > > Status says that: > > > > eef7899ea7ab Doc: do not rely on checked-out Sphinx toolchain from > > svn.python.org anymore > > > > is unmerged, which is what Georg and I agreed upon in Issue20661. Yet, > > in the current python.3.4.2014.02.21.00.07.42.tgz tarball, that change > > appears to be present and, as such, causes installer builds to fail. > > > > Whoopsie! My local branch is actually correct. But! I effectively did > this in my automation: > > % hg clone 3.4 python-{time} > % cd python-{time} > % rm -rf .hg* .bzr* .git* > % cd .. > % tar cvfz python-{time}.tgz python-{time} > > Can you spot the error? That's right: when you clone, the clone always > starts out in "default". So all the tarballs I've published so far have > been wrong. ARGH. Sorry! > > I've added a "hg branch 3.4" in a judicious spot in my automation. You can use 'hg clone -u 3.4 3.4 python-{time}'. $ hg clone --help ... To check out a particular version, use -u/--update, or -U/--noupdate to create a clone with no working directory. ... -u --updaterev REV revision, tag or branch to check out -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From solipsis at pitrou.net Fri Feb 21 12:17:53 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Feb 2014 12:17:53 +0100 Subject: [Python-Dev] Third preview of 3.4.0rc2 is up References: <53070A8A.8080101@hastings.org> <5307250F.3040109@hastings.org> Message-ID: <20140221121753.584c4a4e@fsol> On Fri, 21 Feb 2014 02:06:07 -0800 Larry Hastings wrote: > > Whoopsie! My local branch is actually correct. But! I effectively did > this in my automation: > > % hg clone 3.4 python-{time} > % cd python-{time} > % rm -rf .hg* .bzr* .git* > % cd .. > % tar cvfz python-{time}.tgz python-{time} You could use "hg archive". Regards Antoine. From ncoghlan at gmail.com Fri Feb 21 12:35:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Feb 2014 21:35:20 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 21 February 2014 13:15, Chris Angelico wrote: > PEP: 463 > Title: Exception-catching expressions > Version: $Revision$ > Last-Modified: $Date$ > Author: Chris Angelico > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 15-Feb-2014 > Python-Version: 3.5 > Post-History: 16-Feb-2014, 21-Feb-2014 > > > Abstract > ======== > > Just as PEP 308 introduced a means of value-based conditions in an > expression, this system allows exception-based conditions to be used > as part of an expression. Great work on this Chris - this is one of the best researched and justified Python syntax proposals I've seen :) > Open Issues > =========== > > Parentheses around the entire expression > ---------------------------------------- > > Generator expressions require parentheses, unless they would be > strictly redundant. Ambiguities with except expressions could be > resolved in the same way, forcing nested except-in-except trees to be > correctly parenthesized and requiring that the outer expression be > clearly delineated. `Steven D'Aprano elaborates on the issue.`__ > > __ https://mail.python.org/pipermail/python-ideas/2014-February/025647.html I'd like to make the case that the PEP should adopt this as its default position. My rationale is mainly that if we start by requiring the parentheses, it's pretty straightforward to take that requirement away in specific cases later, as well as making it easier to introduce multiple except clauses if that ever seems necessary. However, if we start without the required parentheses, that's it - we can't introduce a requirement for parentheses later if we decide the bare form is too confusing in too many contexts, and there's plenty of potential for such confusion. In addition to the odd interactions with other uses of the colon as a marker in the syntax, including suite headers, lambdas and function annotations, omitting the parentheses makes it harder to decide which behaviour was intended in ambiguous cases, while the explicit parentheses would force the writer to be clear which one they meant. Consider: x = get_value() except NotFound: foo is not None There are two plausible interpretations of that: x = (get_value() except NotFound: foo) is not None x = get_value() except NotFound: (foo is not None) With the proposed precedence in the PEP (which I agree with), the latter is the correct interpretation, but that's not at all obvious to the reader - they would have to "just know" that's the way it works. By contrast, if the parentheses are required, then the spelling would have to be one of the following to be legal: x = (get_value() except NotFound: foo is not None) x = (get_value() except NotFound: foo) is not None Which means the ":" and the closing ")" nicely bracket the fallback value in both cases and make the author's intent relatively clear. The required parentheses also help in the cases where there is a nearby colon with a different meaning: if check() except Exception: False: ... if (check() except Exception: False): ... lambda x: calculate(x) except Exception: None lambda x: (calculate(x) except Exception: None) def f(a: "OS dependent" = os_defaults[os.name] except KeyError: None): pass def f(a: "OS dependent" = (os_defaults[os.name] except KeyError: None)): pass Rather than making people consider "do I need the parentheses in this case or not?", adopting the genexp rule makes it simple: yes, you need them, because the compiler will complain if you leave them out. > Retrieving a message from either a cache or the internet, with auth > check:: > > logging.info("Message shown to user: %s",((cache[k] > except LookupError: > (backend.read(k) except OSError: 'Resource not available') > ) > if check_permission(k) else 'Access denied' > ) except BaseException: "This is like a bare except clause") I don't think taking it all the way to one expression shows the new construct in the best light. Keeping this as multiple statements assigning to a temporary variable improves the readability quite a bit: if not check_permission(k): msg = 'Access denied' else: msg = (cache[k] except LookupError: None) if msg is None: msg = (backend.read(k) except OSError: 'Resource not available') logging.info("Message shown to user: %s", msg) I would also move the "bare except clause" equivalent out to a separate example. Remember, you're trying to convince people to *like* the PEP, not scare them away with the consequences of what happens when people try to jam too much application logic into a single statement. While we're admittedly giving people another tool to help them win obfuscated Python contests, we don't have to *encourage* them :) > try: > if check_permission(k): > try: > _ = cache[k] > except LookupError: > try: > _ = backend.read(k) > except OSError: > _ = 'Resource not available' > else: > _ = 'Access denied' > except BaseException: > _ = "This is like a bare except clause" > logging.info("Message shown to user: %s", _) A real variable name like "msg" would also be appropriate in the expanded form of this particular example. > Deferred sub-proposals > ====================== > > Capturing the exception object > ------------------------------ > > An examination of the Python standard library shows that, while the use > of 'as' is fairly common (occurring in roughly one except clause in five), > it is extremely *uncommon* in the cases which could logically be converted > into the expression form. Its few uses can simply be left unchanged. > Consequently, in the interests of simplicity, the 'as' clause is not > included in this proposal. A subsequent Python version can add this without > breaking any existing code, as 'as' is already a keyword. We can't defer this one - if we don't implement it now, we should reject it as a future addition. The reason we can't defer it is subtle, but relatively easy to demonstrate with a list comprehension: >>> i = 1 >>> [i for __ in range(1)] [1] >>> class C: ... j = 2 ... [i for __ in range(1)] ... [j for __ in range(1)] ... Traceback (most recent call last): File "", line 1, in File "", line 4, in C File "", line 4, in NameError: global name 'j' is not defined The problem is that "" scope that is visible in the traceback - because it's a true closure, it's treated the same as any other function defined inside a class body, and the subexpressions can't see the class level variables. An except expression that allowed capturing of the exception would need to behave the same way in order to be consistent, but if we don't allow capturing, then we can dispense with the closure entirely. However, if we do that, we can't decide to add capturing later, because that would mean adding the closure, which would be potentially backwards incompatible with usage at class scopes. And if we only add the closure if the exception is captured, then the behaviour of the other subexpressions will depend on whether the exception is captured or not, and that's just messy. So I think it makes more sense to reject this subproposal outright - it makes the change more complex and make the handling of the common case worse, for the sake of something that will almost never be an appropriate thing to do. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Feb 21 12:42:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Feb 2014 21:42:18 +1000 Subject: [Python-Dev] cherry pick a change to Enum In-Reply-To: <20140221112951.67d3f212@fsol> References: <53070FF9.9050907@stoneleaf.us> <20140221112951.67d3f212@fsol> Message-ID: On 21 February 2014 20:29, Antoine Pitrou wrote: > On Fri, 21 Feb 2014 00:36:09 -0800 > Ethan Furman wrote: >> >> Discussion >> ========== >> >> Is there a valid reason to not allow a user to modify how their enums are pickled? >> >> If not, should we put the change in RC2 / Final? The main reason I'm pushing for this is that Enum is still new, but >> after 3.4.0 is cut we then have to deal with backwards compatibility issues. > > I think this should be cherry-picked. > (but see my comment on the issue) That course of action makes sense to me, too. While we ideally want to catch design bugs during beta, the fact is that we sometime won't notice them until the release candidate phase, and it makes sense to fix them before the final release. However, it also makes sense to discuss the corner cases here before deciding how to proceed, and if we uncover any more after rc2, such a change would probably be a trigger for a 3rd release candidate (although that would ultimately be Larry's call as release manager). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Fri Feb 21 13:37:00 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 21 Feb 2014 12:37:00 +0000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 21 February 2014 11:35, Nick Coghlan wrote: >> Just as PEP 308 introduced a means of value-based conditions in an >> expression, this system allows exception-based conditions to be used >> as part of an expression. > > Great work on this Chris - this is one of the best researched and > justified Python syntax proposals I've seen :) Agreed - given the number of differing opinions on python-ideas, it's particularly impressive how well the debate was conducted too. Paul From rosuav at gmail.com Fri Feb 21 13:42:50 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Feb 2014 23:42:50 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Fri, Feb 21, 2014 at 10:35 PM, Nick Coghlan wrote: > On 21 February 2014 13:15, Chris Angelico wrote: >> PEP: 463 >> Title: Exception-catching expressions > Great work on this Chris - this is one of the best researched and > justified Python syntax proposals I've seen :) It is? Wow... I'm not sure what that says about other syntax proposals. This is one week's python-ideas discussion plus one little script doing analysis on the standard library. Hardly PhD level research :) >> Parentheses around the entire expression >> ---------------------------------------- >> >> Generator expressions require parentheses, unless they would be >> strictly redundant. Ambiguities with except expressions could be >> resolved in the same way, forcing nested except-in-except trees to be >> correctly parenthesized and requiring that the outer expression be >> clearly delineated. `Steven D'Aprano elaborates on the issue.`__ >> >> __ https://mail.python.org/pipermail/python-ideas/2014-February/025647.html > > I'd like to make the case that the PEP should adopt this as its > default position. My rationale is mainly that if we start by requiring > the parentheses, it's pretty straightforward to take that requirement > away in specific cases later, as well as making it easier to introduce > multiple except clauses if that ever seems necessary. > > However, if we start without the required parentheses, that's it - we > can't introduce a requirement for parentheses later if we decide the > bare form is too confusing in too many contexts, and there's plenty of > potential for such confusion. If once the parens are made mandatory, they'll most likely stay mandatory forever - I can't imagine there being any strong impetus to change the language to make them optional. > The required parentheses also help in the cases where there is a > nearby colon with a different meaning: > > if check() except Exception: False: > ... > if (check() except Exception: False): People can already write: if (x if y else z): without the parens, and it works. Readability suffers when the same keyword is used twice (here "if" rather than the colon, but same difference), yet the parens are considered optional. Python is a language that, by and large, lacks syntactic salt; style guides are free to stipulate more, but the language doesn't make demands. I would strongly *recommend* using parens in all the cases you've shown, especially lambda: > lambda x: calculate(x) except Exception: None > lambda x: (calculate(x) except Exception: None) as it would otherwise depend on operator precedence; but mandating them feels to me like demanding readability. > def f(a: "OS dependent" = os_defaults[os.name] except KeyError: None): pass > def f(a: "OS dependent" = (os_defaults[os.name] except KeyError: None)): pass Ehh, that one's a mess. I'd be looking at breaking out the default: default = os_defaults[os.name] except KeyError: None def f(a: "OS dependent" = default): pass with possibly some better name than 'default'. The one-liner is almost 80 characters long without indentation and with very short names. But if it's okay to wrap it, that would work without the parens: def f( a: "OS dependent" = os_defaults[os.name] except KeyError: None, another_arg ........., more, args ......, ): pass Clarity is maintained by judicious placement of newlines just as much as by parentheses. > Rather than making people consider "do I need the parentheses in this > case or not?", adopting the genexp rule makes it simple: yes, you need > them, because the compiler will complain if you leave them out. Yes, that is a reasonable line of argument. On the other hand, there's no demand for parens when you mix and and or: x or y and z I'd wager more than half of Python programmers would be unable to say for sure which would be evaluated first. The compiler could have been written to reject this (by placing and and or at the same precedence and having no associativity - I'm not sure if the current lexer in CPython can do that, but it's certainly not conceptually inconceivable), but a decision was made to make this legal. >> Retrieving a message from either a cache or the internet, with auth >> check:: >> >> logging.info("Message shown to user: %s",((cache[k] >> except LookupError: >> (backend.read(k) except OSError: 'Resource not available') >> ) >> if check_permission(k) else 'Access denied' >> ) except BaseException: "This is like a bare except clause") > > I don't think taking it all the way to one expression shows the new > construct in the best light. Keeping this as multiple statements > assigning to a temporary variable improves the readability quite a > bit: Yeah, good point. I tried to strike a balance between simple and complex examples, but it's hard to judge. > I would also move the "bare except clause" equivalent out to a > separate example. Remember, you're trying to convince people to *like* > the PEP, not scare them away with the consequences of what happens > when people try to jam too much application logic into a single > statement. While we're admittedly giving people another tool to help > them win obfuscated Python contests, we don't have to *encourage* them > :) Heh. That example was written when an actual bare except clause was part of the proposal. I'll drop that entire example; it's contrived, and we have a number of concrete examples now. >> Capturing the exception object >> ------------------------------ >> >> An examination of the Python standard library shows that, while the use >> of 'as' is fairly common (occurring in roughly one except clause in five), >> it is extremely *uncommon* in the cases which could logically be converted >> into the expression form. Its few uses can simply be left unchanged. >> Consequently, in the interests of simplicity, the 'as' clause is not >> included in this proposal. A subsequent Python version can add this without >> breaking any existing code, as 'as' is already a keyword. > > We can't defer this one - if we don't implement it now, we should > reject it as a future addition. The reason we can't defer it is > [ chomped for brevity ] > > So I think it makes more sense to reject this subproposal outright - > it makes the change more complex and make the handling of the common > case worse, for the sake of something that will almost never be an > appropriate thing to do. Interesting. That's an aspect I never thought of. Suppose Python were to add support for anonymous sub-scopes. This could be applied to every current use of 'as', as well as comprehensions: spam = "Original spam" try: 1/0 except Exception as spam: assert isinstance(spam, Exception) with open(fn) as spam: assert hasattr(spam, "read") [spam for spam in [1,2,3]] assert spam == "Original spam" It'd be a backward-incompatible change, but it's more likely to be what people expect. The general assumption of "with ... as ..." is that the thing should be used inside the block, and should be finished with when you exit the block, so having the name valid only inside the block does make sense. That's a completely separate proposal. But suppose that were to happen, and to not require a closure. It would then make good sense to be able to capture an exception inside an expression - and it could be done without breaking anything. So, if it is to be rejected, I'd say it's on the technical grounds that it would require a closure in CPython, and that a closure is incompatible with the current proposal. Does that sound right? (It's also not a huge loss, since it's almost unused. But it's an incompatibility between statement and expression form.) ChrisA From rosuav at gmail.com Fri Feb 21 13:46:02 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 21 Feb 2014 23:46:02 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Fri, Feb 21, 2014 at 11:37 PM, Paul Moore wrote: > On 21 February 2014 11:35, Nick Coghlan wrote: >>> Just as PEP 308 introduced a means of value-based conditions in an >>> expression, this system allows exception-based conditions to be used >>> as part of an expression. >> >> Great work on this Chris - this is one of the best researched and >> justified Python syntax proposals I've seen :) > > Agreed - given the number of differing opinions on python-ideas, it's > particularly impressive how well the debate was conducted too. On that subject, I'd like to thank everyone involved in the python-ideas discussion, and particularly Steven D'Aprano, Rob Cliffe, and Andrew Barnert; it was an amazingly productive thread, getting to nearly four hundred emails before seriously meandering. And even then, it mostly just started looping back on itself, which isn't surprising given that it was so long - anyone not majorly invested in the topic won't have read every single post. ChrisA From ncoghlan at gmail.com Fri Feb 21 14:20:59 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Feb 2014 23:20:59 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 21 February 2014 22:42, Chris Angelico wrote: > It'd be a backward-incompatible change, but it's more likely to be > what people expect. The general assumption of "with ... as ..." is > that the thing should be used inside the block, and should be finished > with when you exit the block, so having the name valid only inside the > block does make sense. > > That's a completely separate proposal. But suppose that were to > happen, and to not require a closure. It would then make good sense to > be able to capture an exception inside an expression - and it could be > done without breaking anything. > > So, if it is to be rejected, I'd say it's on the technical grounds > that it would require a closure in CPython, and that a closure is > incompatible with the current proposal. Does that sound right? > > (It's also not a huge loss, since it's almost unused. But it's an > incompatibility between statement and expression form.) It's probably OK to leave it in the deferred section and just note the difficulty of implementing it in a backwards compatible way, since we're *not* going to be introducing a closure. Python 3 except clauses are already a bit weird anyway, since they do an implicit delete at the end, but only if the except clause is actually triggered: >>> e = "Hello" >>> try: ... 1/0 ... except ZeroDivisionError as e: ... pass ... >>> e Traceback (most recent call last): File "", line 1, in NameError: name 'e' is not defined >>> e = "Hello" >>> try: ... pass ... except Exception as e: ... pass ... >>> e 'Hello' Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From nad at acm.org Fri Feb 21 14:31:05 2014 From: nad at acm.org (Ned Deily) Date: Fri, 21 Feb 2014 05:31:05 -0800 Subject: [Python-Dev] Third preview of 3.4.0rc2 is up References: <53070A8A.8080101@hastings.org> <5307250F.3040109@hastings.org> Message-ID: In article <5307250F.3040109 at hastings.org>, Larry Hastings wrote: > Whoopsie! My local branch is actually correct. But! I effectively did > this in my automation: > > % hg clone 3.4 python-{time} > % cd python-{time} > % rm -rf .hg* .bzr* .git* > % cd .. > % tar cvfz python-{time}.tgz python-{time} > > Can you spot the error? That's right: when you clone, the clone always > starts out in "default". So all the tarballs I've published so far have > been wrong. ARGH. Sorry! > > I've added a "hg branch 3.4" in a judicious spot in my automation. The > resulting tarball doesn't have the sphinx toolchain change. > > Look for a new tarball in a few minutes, once "make test" finishes, Much better - thanks! -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Fri Feb 21 14:44:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Feb 2014 23:44:18 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 21 February 2014 22:42, Chris Angelico wrote: > On Fri, Feb 21, 2014 at 10:35 PM, Nick Coghlan wrote: >> On 21 February 2014 13:15, Chris Angelico wrote: >>> PEP: 463 >>> Title: Exception-catching expressions >> Great work on this Chris - this is one of the best researched and >> justified Python syntax proposals I've seen :) > > It is? Wow... I'm not sure what that says about other syntax > proposals. This is one week's python-ideas discussion plus one little > script doing analysis on the standard library. Hardly PhD level > research :) Right, it just takes someone willing to put in the time to actually put a concrete proposal together, read all the discussions, attempt to summarise them into a coherent form and go looking for specific examples to help make their case. I think a large part of my pleased reaction is the fact that several of the python-ideas regulars (including me) have a habit of responding to new syntax proposals with a list of things to do to make a good PEP (especially Raymond's "search the stdlib for code that would be improved" criterion), and it's quite a novelty to have someone take that advice and put together a compelling argument - the more typical reaction is for the poster to decide that a PEP sounds like too much work and drop the idea (or else to realise that they can't actually provide the compelling use cases requested). As you have discovered, creating a PEP really isn't that arduous, so long as you have enough spare time to keep up with the discussions, and there actually are compelling examples of possible improvements readily available :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Fri Feb 21 14:52:05 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Feb 2014 14:52:05 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions References: Message-ID: <20140221145205.0b8b670d@fsol> On Fri, 21 Feb 2014 14:15:59 +1100 Chris Angelico wrote: > > A number of functions and methods have parameters which will cause > them to return a specified value instead of raising an exception. The > current system is ad-hoc and inconsistent, and requires that each > function be individually written to have this functionality; not all > support this. While I sympathize with the motivation, I really don't like the end result: > lst = [1, 2] > value = lst[2] except IndexError: "No value" is too much of a break from the usual stylistic conventions, and looks like several statements were stuffed on a single line. In other words, the gain in concision is counterbalanced by a loss in readability, a compromise which doesn't fit in Python's overall design principles. (compare with "with", which solves actual readability issues due to the distance between the "try" and the "finally" clause, and also promotes better resource management) So -0.5 from me. Regards Antoine. From ncoghlan at gmail.com Fri Feb 21 14:53:46 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Feb 2014 23:53:46 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 21 February 2014 22:42, Chris Angelico wrote: > People can already write: > > if (x if y else z): > > without the parens, and it works. Readability suffers when the same > keyword is used twice (here "if" rather than the colon, but same > difference), yet the parens are considered optional. Python is a > language that, by and large, lacks syntactic salt; style guides are > free to stipulate more, but the language doesn't make demands. I would > strongly *recommend* using parens in all the cases you've shown, > especially lambda: > >> lambda x: calculate(x) except Exception: None >> lambda x: (calculate(x) except Exception: None) > > as it would otherwise depend on operator precedence; but mandating > them feels to me like demanding readability. Right, that's why my main motivation for this suggestion is the one relating to keeping future options open. If the parentheses are optional, than adding multiple except clauses latter isn't possible, since this would already be valid, but mean something different: expr except Exception1: default1 except Exception2: default2 The deferral currently has this snippet: """In order to ensure compatibility with future versions, ensure that any consecutive except operators are parenthesized to guarantee the interpretation you expect.""" That's not a reasonable expectation - either the parentheses have to be mandatory as part of the deferral, or else multiple except clause support needs to be listed as rejected rather than deferred. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From eliben at gmail.com Fri Feb 21 15:03:12 2014 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 21 Feb 2014 06:03:12 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Thu, Feb 20, 2014 at 7:15 PM, Chris Angelico wrote: > PEP: 463 > Title: Exception-catching expressions > Version: $Revision$ > Last-Modified: $Date$ > Author: Chris Angelico > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 15-Feb-2014 > Python-Version: 3.5 > Post-History: 16-Feb-2014, 21-Feb-2014 > > > Abstract > ======== > > Just as PEP 308 introduced a means of value-based conditions in an > expression, this system allows exception-based conditions to be used > as part of an expression. > Chris, while I also commend you for the comprehensive PEP, I'm -1 on the proposal, for two main reasons: 1. Many proposals suggest new syntax to gain some succinctness. Each has to be judged for its own merits, and in this case IMHO the cons eclipse the pros. I don't think this will save a lot of code in a typical well-structured program - maybe a few lines out of hundreds. On the other hand, it adds yet another syntax to remember and understand, which is not the Pythonic way. 2. Worse, this idea subverts exceptions to control flow, which is not only un-Pythonic but also against the accepted practices of programming in general. Here, the comparison to PEP 308 is misguided. PEP 308, whatever syntax it adds, still remains within the domain of normal control flow. PEP 463, OTOH, makes it deliberately easy to make exceptions part of non-exceptional code, encouraging very bad programming practices. Eli > > > Motivation > ========== > > A number of functions and methods have parameters which will cause > them to return a specified value instead of raising an exception. The > current system is ad-hoc and inconsistent, and requires that each > function be individually written to have this functionality; not all > support this. > > * dict.get(key, default) - second positional argument in place of > KeyError > > * next(iter, default) - second positional argument in place of > StopIteration > > * list.pop() - no way to return a default > > * seq[index] - no way to handle a bounds error > > * min(sequence, default=default) - keyword argument in place of > ValueError > > * sum(sequence, start=default) - slightly different but can do the > same job > > * statistics.mean(data) - no way to handle an empty iterator > > > Rationale > ========= > > The current system requires that a function author predict the need > for a default, and implement support for it. If this is not done, a > full try/except block is needed. > > Since try/except is a statement, it is impossible to catch exceptions > in the middle of an expression. Just as if/else does for conditionals > and lambda does for function definitions, so does this allow exception > catching in an expression context. > > This provides a clean and consistent way for a function to provide a > default: it simply raises an appropriate exception, and the caller > catches it. > > With some situations, an LBYL technique can be used (checking if some > sequence has enough length before indexing into it, for instance). This is > not safe in all cases, but as it is often convenient, programmers will be > tempted to sacrifice the safety of EAFP in favour of the notational brevity > of LBYL. Additionally, some LBYL techniques (eg involving getattr with > three arguments) warp the code into looking like literal strings rather > than attribute lookup, which can impact readability. A convenient EAFP > notation solves all of this. > > There's no convenient way to write a helper function to do this; the > nearest is something ugly using either lambda:: > > def except_(expression, exception_list, default): > try: > return expression() > except exception_list: > return default() > value = except_(lambda: 1/x, ZeroDivisionError, lambda: float("nan")) > > which is clunky, and unable to handle multiple exception clauses; or > eval:: > > def except_(expression, exception_list, default): > try: > return eval(expression, globals_of_caller(), > locals_of_caller()) > except exception_list as exc: > l = locals_of_caller().copy() > l['exc'] = exc > return eval(default, globals_of_caller(), l) > > def globals_of_caller(): > return sys._getframe(2).f_globals > > def locals_of_caller(): > return sys._getframe(2).f_locals > > value = except_("""1/x""",ZeroDivisionError,""" "Can't divide by zero" > """) > > which is even clunkier, and relies on implementation-dependent hacks. > (Writing globals_of_caller() and locals_of_caller() for interpreters > other than CPython is left as an exercise for the reader.) > > Raymond Hettinger `expresses`__ a desire for such a consistent > API. Something similar has been `requested`__ `multiple`__ `times`__ > in the past. > > __ > https://mail.python.org/pipermail/python-ideas/2014-February/025443.html > __ https://mail.python.org/pipermail/python-ideas/2013-March/019760.html > __ https://mail.python.org/pipermail/python-ideas/2009-August/005441.html > __ https://mail.python.org/pipermail/python-ideas/2008-August/001801.html > > > Proposal > ======== > > Just as the 'or' operator and the three part 'if-else' expression give > short circuiting methods of catching a falsy value and replacing it, > this syntax gives a short-circuiting method of catching an exception > and replacing it. > > This currently works:: > > lst = [1, 2, None, 3] > value = lst[2] or "No value" > > The proposal adds this:: > > lst = [1, 2] > value = lst[2] except IndexError: "No value" > > Specifically, the syntax proposed is:: > > expr except exception_list: default > > where expr, exception_list, and default are all expressions. First, > expr is evaluated. If no exception is raised, its value is the value > of the overall expression. If any exception is raised, exception_list > is evaluated, and should result in either a type or a tuple, just as > with the statement form of try/except. Any matching exception will > result in the corresponding default expression being evaluated and > becoming the value of the expression. As with the statement form of > try/except, non-matching exceptions will propagate upward. > > Note that the current proposal does not allow the exception object to > be captured. Where this is needed, the statement form must be used. > (See below for discussion and elaboration on this.) > > This ternary operator would be between lambda and if/else in > precedence. > > Consider this example of a two-level cache:: > for key in sequence: > x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: > f(key))) > # do something with x > > This cannot be rewritten as:: > x = lvl1.get(key, lvl2.get(key, f(key))) > > which, despite being shorter, defeats the purpose of the cache, as it must > calculate a default value to pass to get(). The .get() version calculates > backwards; the exception-testing version calculates forwards, as would be > expected. The nearest useful equivalent would be:: > x = lvl1.get(key) or lvl2.get(key) or f(key) > which depends on the values being nonzero, as well as depending on the > cache > object supporting this functionality. > > > Alternative Proposals > ===================== > > Discussion on python-ideas brought up the following syntax suggestions:: > > value = expr except default if Exception [as e] > value = expr except default for Exception [as e] > value = expr except default from Exception [as e] > value = expr except Exception [as e] return default > value = expr except (Exception [as e]: default) > value = expr except Exception [as e] try default > value = expr except Exception [as e] continue with default > value = default except Exception [as e] else expr > value = try expr except Exception [as e]: default > value = expr except default # Catches anything > value = expr except(Exception) default # Catches only the named type(s) > value = default if expr raise Exception > value = expr or else default if Exception > value = expr except Exception [as e] -> default > value = expr except Exception [as e] pass default > > It has also been suggested that a new keyword be created, rather than > reusing an existing one. Such proposals fall into the same structure > as the last form, but with a different keyword in place of 'pass'. > Suggestions include 'then', 'when', and 'use'. Also, in the context of > the "default if expr raise Exception" proposal, it was suggested that a > new keyword "raises" be used. > > All forms involving the 'as' capturing clause have been deferred from > this proposal in the interests of simplicity, but are preserved in the > table above as an accurate record of suggestions. > > > Open Issues > =========== > > Parentheses around the entire expression > ---------------------------------------- > > Generator expressions require parentheses, unless they would be > strictly redundant. Ambiguities with except expressions could be > resolved in the same way, forcing nested except-in-except trees to be > correctly parenthesized and requiring that the outer expression be > clearly delineated. `Steven D'Aprano elaborates on the issue.`__ > > __ > https://mail.python.org/pipermail/python-ideas/2014-February/025647.html > > > Example usage > ============= > > For each example, an approximately-equivalent statement form is given, > to show how the expression will be parsed. These are not always > strictly equivalent, but will accomplish the same purpose. It is NOT > safe for the interpreter to translate one into the other. > > A number of these examples are taken directly from the Python standard > library, with file names and line numbers correct as of early Feb 2014. > Many of these patterns are extremely common. > > Retrieve an argument, defaulting to None:: > cond = args[1] except IndexError: None > > # Lib/pdb.py:803: > try: > cond = args[1] > except IndexError: > cond = None > > Fetch information from the system if available:: > pwd = os.getcwd() except OSError: None > > # Lib/tkinter/filedialog.py:210: > try: > pwd = os.getcwd() > except OSError: > pwd = None > > Attempt a translation, falling back on the original:: > e.widget = self._nametowidget(W) except KeyError: W > > # Lib/tkinter/__init__.py:1222: > try: > e.widget = self._nametowidget(W) > except KeyError: > e.widget = W > > Read from an iterator, continuing with blank lines once it's > exhausted:: > line = readline() except StopIteration: '' > > # Lib/lib2to3/pgen2/tokenize.py:370: > try: > line = readline() > except StopIteration: > line = '' > > Retrieve platform-specific information (note the DRY improvement); > this particular example could be taken further, turning a series of > separate assignments into a single large dict initialization:: > # sys.abiflags may not be defined on all platforms. > _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: '' > > # Lib/sysconfig.py:529: > try: > _CONFIG_VARS['abiflags'] = sys.abiflags > except AttributeError: > # sys.abiflags may not be defined on all platforms. > _CONFIG_VARS['abiflags'] = '' > > Retrieve an indexed item, defaulting to None (similar to dict.get):: > def getNamedItem(self, name): > return self._attrs[name] except KeyError: None > > # Lib/xml/dom/minidom.py:573: > def getNamedItem(self, name): > try: > return self._attrs[name] > except KeyError: > return None > > > Translate numbers to names, falling back on the numbers:: > g = grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid > u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: tarinfo.uid > > # Lib/tarfile.py:2198: > try: > g = grp.getgrnam(tarinfo.gname)[2] > except KeyError: > g = tarinfo.gid > try: > u = pwd.getpwnam(tarinfo.uname)[2] > except KeyError: > u = tarinfo.uid > > Perform some lengthy calculations in EAFP mode, handling division by > zero as a sort of sticky NaN:: > > value = calculate(x) except ZeroDivisionError: float("nan") > > try: > value = calculate(x) > except ZeroDivisionError: > value = float("nan") > > Calculate the mean of a series of numbers, falling back on zero:: > > value = statistics.mean(lst) except statistics.StatisticsError: 0 > > try: > value = statistics.mean(lst) > except statistics.StatisticsError: > value = 0 > > Retrieving a message from either a cache or the internet, with auth > check:: > > logging.info("Message shown to user: %s",((cache[k] > except LookupError: > (backend.read(k) except OSError: 'Resource not available') > ) > if check_permission(k) else 'Access denied' > ) except BaseException: "This is like a bare except clause") > > try: > if check_permission(k): > try: > _ = cache[k] > except LookupError: > try: > _ = backend.read(k) > except OSError: > _ = 'Resource not available' > else: > _ = 'Access denied' > except BaseException: > _ = "This is like a bare except clause" > logging.info("Message shown to user: %s", _) > > Looking up objects in a sparse list of overrides:: > > (overrides[x] or default except IndexError: default).ping() > > try: > (overrides[x] or default).ping() > except IndexError: > default.ping() > > > Narrowing of exception-catching scope > ------------------------------------- > > The following examples, taken directly from Python's standard library, > demonstrate how the scope of the try/except can be conveniently narrowed. > To do this with the statement form of try/except would require a temporary > variable, but it's far cleaner as an expression. > > Lib/ipaddress.py:343:: > try: > ips.append(ip.ip) > except AttributeError: > ips.append(ip.network_address) > Becomes:: > ips.append(ip.ip except AttributeError: ip.network_address) > The expression form is nearly equivalent to this:: > try: > _ = ip.ip > except AttributeError: > _ = ip.network_address > ips.append(_) > > Lib/tempfile.py:130:: > try: > dirlist.append(_os.getcwd()) > except (AttributeError, OSError): > dirlist.append(_os.curdir) > Becomes:: > dirlist.append(_os.getcwd() except (AttributeError, OSError): > _os.curdir) > > Lib/asyncore.py:264:: > try: > status.append('%s:%d' % self.addr) > except TypeError: > status.append(repr(self.addr)) > Becomes:: > status.append('%s:%d' % self.addr except TypeError: > repr(self.addr)) > > > Comparisons with other languages > ================================ > > (With thanks to Andrew Barnert for compiling this section.) > > `Ruby's`__ "begin...rescue...rescue...else...ensure...end" is an expression > (potentially with statements inside it). It has the equivalent of an "as" > clause, and the equivalent of bare except. And it uses no punctuation or > keyword between the bare except/exception class/exception class with as > clause and the value. (And yes, it's ambiguous unless you understand > Ruby's statement/expression rules.) > > __ http://www.skorks.com/2009/09/ruby-exceptions-and-exception-handling/ > > :: > > x = begin computation() rescue MyException => e default(e) end; > x = begin computation() rescue MyException default() end; > x = begin computation() rescue default() end; > x = begin computation() rescue MyException default() rescue > OtherException other() end; > > In terms of this PEP:: > > x = computation() except MyException as e default(e) > x = computation() except MyException default(e) > x = computation() except default(e) > x = computation() except MyException default() except OtherException > other() > > `Erlang`__ has a try expression that looks like this:: > > __ http://erlang.org/doc/reference_manual/expressions.html#id79284 > > x = try computation() catch MyException:e -> default(e) end; > x = try computation() catch MyException:e -> default(e); > OtherException:e -> other(e) end; > > The class and "as" name are mandatory, but you can use "_" for either. > There's also an optional "when" guard on each, and a "throw" clause that > you can catch, which I won't get into. To handle multiple exceptions, > you just separate the clauses with semicolons, which I guess would map > to commas in Python. So:: > > x = try computation() except MyException as e -> default(e) > x = try computation() except MyException as e -> default(e), > OtherException as e->other_default(e) > > Erlang also has a "catch" expression, which, despite using the same > keyword, > is completely different, and you don't want to know about it. > > > The ML family has two different ways of dealing with this, "handle" and > "try"; the difference between the two is that "try" pattern-matches the > exception, which gives you the effect of multiple except clauses and as > clauses. In either form, the handler clause is punctuated by "=>" in > some dialects, "->" in others. > > To avoid confusion, I'll write the function calls in Python style. > > Here's `SML's`__ "handle":: > __ http://www.cs.cmu.edu/~rwh/introsml/core/exceptions.htm > > let x = computation() handle MyException => default();; > > Here's `OCaml's`__ "try":: > __ http://www2.lib.uchicago.edu/keith/ocaml-class/exceptions.html > > let x = try computation() with MyException explanation -> > default(explanation);; > > let x = try computation() with > > MyException(e) -> default(e) > | MyOtherException() -> other_default() > | (e) -> fallback(e);; > > In terms of this PEP, these would be something like:: > > x = computation() except MyException => default() > x = try computation() except MyException e -> default() > x = (try computation() > except MyException as e -> default(e) > except MyOtherException -> other_default() > except BaseException as e -> fallback(e)) > > Many ML-inspired but not-directly-related languages from academia mix > things > up, usually using more keywords and fewer symbols. So, the `Oz`__ would map > to Python as:: > __ http://mozart.github.io/mozart-v1/doc-1.4.0/tutorial/node5.html > > x = try computation() catch MyException as e then default(e) > > > Many Lisp-derived languages, like `Clojure,`__ implement try/catch as > special > forms (if you don't know what that means, think function-like macros), so > you > write, effectively:: > __ > http://clojure.org/special_forms#Special%20Forms--(try%20expr*%20catch-clause*%20finally-clause > ?) > > try(computation(), catch(MyException, explanation, > default(explanation))) > > try(computation(), > catch(MyException, explanation, default(explanation)), > catch(MyOtherException, explanation, other_default(explanation))) > > In Common Lisp, this is done with a slightly clunkier `"handler-case" > macro,`__ > but the basic idea is the same. > > __ http://clhs.lisp.se/Body/m_hand_1.htm > > > The Lisp style is, surprisingly, used by some languages that don't have > macros, like Lua, where `xpcall`__ takes functions. Writing lambdas > Python-style instead of Lua-style:: > __ http://www.gammon.com.au/scripts/doc.php?lua=xpcall > > x = xpcall(lambda: expression(), lambda e: default(e)) > > This actually returns (true, expression()) or (false, default(e)), but > I think we can ignore that part. > > > Haskell is actually similar to Lua here (except that it's all done > with monads, of course):: > > x = do catch(lambda: expression(), lambda e: default(e)) > > You can write a pattern matching expression within the function to decide > what to do with it; catching and re-raising exceptions you don't want is > cheap enough to be idiomatic. > > But Haskell infixing makes this nicer:: > > x = do expression() `catch` lambda: default() > x = do expression() `catch` lambda e: default(e) > > And that makes the parallel between the lambda colon and the except > colon in the proposal much more obvious:: > > > x = expression() except Exception: default() > x = expression() except Exception as e: default(e) > > > `Tcl`__ has the other half of Lua's xpcall; catch is a function which > returns > true if an exception was caught, false otherwise, and you get the value out > in other ways. And it's all built around the the implicit quote-and-exec > that everything in Tcl is based on, making it even harder to describe in > Python terms than Lisp macros, but something like:: > __ http://wiki.tcl.tk/902 > > if {[ catch("computation()") "explanation"]} { default(explanation) } > > > `Smalltalk`__ is also somewhat hard to map to Python. The basic version > would be:: > __ http://smalltalk.gnu.org/wiki/exceptions > > x := computation() on:MyException do:default() > > ... but that's basically Smalltalk's passing-arguments-with-colons > syntax, not its exception-handling syntax. > > > Deferred sub-proposals > ====================== > > Multiple except clauses > ----------------------- > > An examination of use-cases shows that this is not needed as often as > it would be with the statement form, and as its syntax is a point on > which consensus has not been reached, the entire feature is deferred. > > In order to ensure compatibility with future versions, ensure that any > consecutive except operators are parenthesized to guarantee the > interpretation you expect. > > Multiple 'except' keywords can be used, and they will all catch > exceptions raised in the original expression (only):: > > # Will catch any of the listed exceptions thrown by expr; > # any exception thrown by a default expression will propagate. > value = (expr > except Exception1 [as e]: default1 > except Exception2 [as e]: default2 > # ... except ExceptionN [as e]: defaultN > ) > > Using parentheses to force an alternative interpretation works as > expected:: > > # Will catch an Exception2 thrown by either expr or default1 > value = ( > (expr except Exception1: default1) > except Exception2: default2 > ) > # Will catch an Exception2 thrown by default1 only > value = (expr except Exception1: > (default1 except Exception2: default2) > ) > > This last form is confusing and should be discouraged by PEP 8, but it > is syntactically legal: you can put any sort of expression inside a > ternary-except; ternary-except is an expression; therefore you can put > a ternary-except inside a ternary-except. > > Open question: Where there are multiple except clauses, should they be > separated by commas? It may be easier for the parser, that way:: > > value = (expr > except Exception1 [as e]: default1, > except Exception2 [as e]: default2, > # ... except ExceptionN [as e]: defaultN, > ) > > with an optional comma after the last, as per tuple rules. Downside: > Omitting the comma would be syntactically valid, and would have almost > identical semantics, but would nest the entire preceding expression in > its exception catching rig - a matching exception raised in the > default clause would be caught by the subsequent except clause. As > this difference is so subtle, it runs the risk of being a major bug > magnet. > > As a mitigation of this risk, this form:: > > value = expr except Exception1: default1 except Exception2: default2 > > could be syntactically forbidden, and parentheses required if the > programmer actually wants that behaviour:: > > value = (expr except Exception1: default1) except Exception2: default2 > > This would prevent the accidental omission of a comma from changing > the expression's meaning. > > > Capturing the exception object > ------------------------------ > > In a try/except block, the use of 'as' to capture the exception object > creates a local name binding, and implicitly deletes that binding in a > finally clause. As 'finally' is not a part of this proposal (see > below), this makes it tricky to describe; also, this use of 'as' gives > a way to create a name binding in an expression context. Should the > default clause have an inner scope in which the name exists, shadowing > anything of the same name elsewhere? Should it behave the same way the > statement try/except does, and unbind the name? Should it bind the > name and leave it bound? (Almost certainly not; this behaviour was > changed in Python 3 for good reason.) > > Additionally, this syntax would allow a convenient way to capture > exceptions in interactive Python; returned values are captured by "_", > but exceptions currently are not. This could be spelled: > > >>> expr except Exception as e: e > > (The inner scope idea is tempting, but currently CPython handles list > comprehensions with a nested function call, as this is considered > easier. It may be of value to simplify both comprehensions and except > expressions, but that is a completely separate proposal to this PEP; > alternatively, it may be better to stick with what's known to > work. `Nick Coghlan elaborates.`__) > > __ > https://mail.python.org/pipermail/python-ideas/2014-February/025702.html > > An examination of the Python standard library shows that, while the use > of 'as' is fairly common (occurring in roughly one except clause in five), > it is extremely *uncommon* in the cases which could logically be converted > into the expression form. Its few uses can simply be left unchanged. > Consequently, in the interests of simplicity, the 'as' clause is not > included in this proposal. A subsequent Python version can add this > without > breaking any existing code, as 'as' is already a keyword. > > One example where this could possibly be useful is Lib/imaplib.py:568:: > try: typ, dat = self._simple_command('LOGOUT') > except: typ, dat = 'NO', ['%s: %s' % sys.exc_info()[:2]] > This could become:: > typ, dat = (self._simple_command('LOGOUT') > except BaseException as e: ('NO', '%s: %s' % (type(e), e))) > Or perhaps some other variation. This is hardly the most compelling > use-case, > but an intelligent look at this code could tidy it up significantly. In > the > absence of further examples showing any need of the exception object, I > have > opted to defer indefinitely the recommendation. > > > Rejected sub-proposals > ====================== > > finally clause > -------------- > The statement form try... finally or try... except... finally has no > logical corresponding expression form. Therefore the finally keyword > is not a part of this proposal, in any way. > > > Bare except having different meaning > ------------------------------------ > > With several of the proposed syntaxes, omitting the exception type name > would be easy and concise, and would be tempting. For convenience's sake, > it might be advantageous to have a bare 'except' clause mean something > more useful than "except BaseException". Proposals included having it > catch Exception, or some specific set of "common exceptions" (subclasses > of a new type called ExpressionError), or have it look for a tuple named > ExpressionError in the current scope, with a built-in default such as > (ValueError, UnicodeError, AttributeError, EOFError, IOError, OSError, > LookupError, NameError, ZeroDivisionError). All of these were rejected, > for severa reasons. > > * First and foremost, consistency with the statement form of try/except > would be broken. Just as a list comprehension or ternary if expression > can be explained by "breaking it out" into its vertical statement form, > an expression-except should be able to be explained by a relatively > mechanical translation into a near-equivalent statement. Any form of > syntax common to both should therefore have the same semantics in each, > and above all should not have the subtle difference of catching more in > one than the other, as it will tend to attract unnoticed bugs. > > * Secondly, the set of appropriate exceptions to catch would itself be > a huge point of contention. It would be impossible to predict exactly > which exceptions would "make sense" to be caught; why bless some of them > with convenient syntax and not others? > > * And finally (this partly because the recommendation was that a bare > except should be actively encouraged, once it was reduced to a "reasonable" > set of exceptions), any situation where you catch an exception you don't > expect to catch is an unnecessary bug magnet. > > Consequently, the use of a bare 'except' is down to two possibilities: > either it is syntactically forbidden in the expression form, or it is > permitted with the exact same semantics as in the statement form (namely, > that it catch BaseException and be unable to capture it with 'as'). > > > Bare except clauses > ------------------- > > PEP 8 rightly advises against the use of a bare 'except'. While it is > syntactically legal in a statement, and for backward compatibility must > remain so, there is little value in encouraging its use. In an expression > except clause, "except:" is a SyntaxError; use the equivalent long-hand > form "except BaseException:" instead. A future version of Python MAY choose > to reinstate this, which can be done without breaking compatibility. > > > Parentheses around the except clauses > ------------------------------------- > > Should it be legal to parenthesize the except clauses, separately from > the expression that could raise? Example:: > > value = expr ( > except Exception1 [as e]: default1 > except Exception2 [as e]: default2 > # ... except ExceptionN [as e]: defaultN > ) > > This is more compelling when one or both of the deferred sub-proposals > of multiple except clauses and/or exception capturing is included. In > their absence, the parentheses would be thus:: > value = expr except ExceptionType: default > value = expr (except ExceptionType: default) > > The advantage is minimal, and the potential to confuse a reader into > thinking the except clause is separate from the expression, or into > thinking > this is a function call, makes this non-compelling. The expression can, of > course, be parenthesized if desired, as can the default:: > value = (expr) except ExceptionType: (default) > > > Short-hand for "except: pass" > ----------------------------- > > The following was been suggested as a similar > short-hand, though not technically an expression:: > > statement except Exception: pass > > try: > statement > except Exception: > pass > > For instance, a common use-case is attempting the removal of a file:: > os.unlink(some_file) except OSError: pass > > There is an equivalent already in Python 3.4, however, in contextlib:: > from contextlib import suppress > with suppress(OSError): os.unlink(some_file) > > As this is already a single line (or two with a break after the colon), > there is little need of new syntax and a confusion of statement vs > expression to achieve this. > > > Copyright > ========= > > This document has been placed in the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Fri Feb 21 15:22:34 2014 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 21 Feb 2014 14:22:34 +0000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <5307612A.3050607@btinternet.com> My 2 cents worth: On 21/02/2014 12:42, Chris Angelico wrote: > On Fri, Feb 21, 2014 at 10:35 PM, Nick Coghlan wrote: >> Great work on this Chris - this is one of the best researched and >> justified Python syntax proposals I've seen :) Hear, hear! ("Praise from the praiseworthy is praise indeed" - Tolkien.) Small point: in one of your examples you give a plug for the PEP "note the DRY improvement". I would suggest that similarly perhaps in your Lib/tarfile.py:2198 example you point out the increase in readability due to the 2 lines lining up in your Lib/ipaddress.py:343 example you point out that the new form is probably an improvement (i.e. probably closer to the author's intention) as it will NOT catch an AttributeError evaluating "ips.append" (this would matter e.g. if "append" were mis-spelt). YOU are clearly aware of this but it would often escape the casual reader. >>> Retrieving a message from either a cache or the internet, with auth >>> check:: >>> >>> logging.info("Message shown to user: %s",((cache[k] >>> except LookupError: >>> (backend.read(k) except OSError: 'Resource not available') >>> ) >>> if check_permission(k) else 'Access denied' >>> ) except BaseException: "This is like a bare except clause") >> I don't think taking it all the way to one expression shows the new >> construct in the best light. Keeping this as multiple statements >> assigning to a temporary variable improves the readability quite a >> bit: I agree, it looks scary (too many branches in one statement). The shorter the example, the more convincing. > Yeah, good point. I tried to strike a balance between simple and > complex examples, but it's hard to judge. > > >>> Capturing the exception object >>> ------------------------------ >>> >>> An examination of the Python standard library shows that, while the use >>> of 'as' is fairly common (occurring in roughly one except clause in five), >>> it is extremely *uncommon* in the cases which could logically be converted >>> into the expression form. Its few uses can simply be left unchanged. >>> Consequently, in the interests of simplicity, the 'as' clause is not >>> included in this proposal. A subsequent Python version can add this without >>> breaking any existing code, as 'as' is already a keyword. >> We can't defer this one - if we don't implement it now, we should >> reject it as a future addition. The reason we can't defer it is >> [ chomped for brevity ] >> >> So I think it makes more sense to reject this subproposal outright - >> it makes the change more complex and make the handling of the common >> case worse, for the sake of something that will almost never be an >> appropriate thing to do. Thanks for looking into this Nick. I confess I don't entirely understand the technical argument (my understanding breaks down at "the subexpressions can't see the class level variables", but I don't want to waste anybody's time expecting an explanation, I can always look into it myself) but I accept that there is a good reason for disallowing 'as' and its usage would be rare, so I will (slightly regretfully) wave it goodbye. > > > So, if it is to be rejected, I'd say it's on the technical grounds > that it would require a closure in CPython, and that a closure is > incompatible with the current proposal. Does that sound right? It does to me FWIW. > > (It's also not a huge loss, since it's almost unused. But it's an > incompatibility between statement and expression form.) > > ChrisA Assuming the PEP is accepted (not a trivial assumption!) I hope that at some stage allowing multiple except clauses (to trap different exceptions raised by the original expression) will be reinstated. But I have to accept that it would be rarely used. To my mind the real unsung hero of this story is the existing "try ... except ... finally" syntax and its flexibility. In my code base there were a few hundred of these, but just over a dozen where the new syntax would be appropriate. (Hope I'm not starting to argue against the PEP!) Rob Cliffe > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rob.cliffe%40btinternet.com > > > ----- > No virus found in this message. > Checked by AVG - www.avg.com > Version: 2012.0.2247 / Virus Database: 3705/6611 - Release Date: 02/20/14 > > From ncoghlan at gmail.com Fri Feb 21 15:28:01 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Feb 2014 00:28:01 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 22 February 2014 00:03, Eli Bendersky wrote: > On Thu, Feb 20, 2014 at 7:15 PM, Chris Angelico wrote: >> >> PEP: 463 >> Title: Exception-catching expressions >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Chris Angelico >> Status: Draft >> Type: Standards Track >> Content-Type: text/x-rst >> Created: 15-Feb-2014 >> Python-Version: 3.5 >> Post-History: 16-Feb-2014, 21-Feb-2014 >> >> >> Abstract >> ======== >> >> Just as PEP 308 introduced a means of value-based conditions in an >> expression, this system allows exception-based conditions to be used >> as part of an expression. > > > > Chris, while I also commend you for the comprehensive PEP, I'm -1 on the > proposal, for two main reasons: > > 1. Many proposals suggest new syntax to gain some succinctness. Each has to > be judged for its own merits, and in this case IMHO the cons eclipse the > pros. I don't think this will save a lot of code in a typical > well-structured program - maybe a few lines out of hundreds. On the other > hand, it adds yet another syntax to remember and understand, which is not > the Pythonic way. > > 2. Worse, this idea subverts exceptions to control flow, which is not only > un-Pythonic but also against the accepted practices of programming in > general. Here, the comparison to PEP 308 is misguided. PEP 308, whatever > syntax it adds, still remains within the domain of normal control flow. PEP > 463, OTOH, makes it deliberately easy to make exceptions part of > non-exceptional code, encouraging very bad programming practices. Neither of these objections addresses the problems with the status quo, though: - the status quo encourages overbroad exception handling (as illustrated by examples in the PEP) - the status quo encourages an ad hoc approach to hiding exception handling inside functions PEP 308 was accepted primarily because the and/or hack was a bug magnet. The status quo for exception handling is both a bug magnet (due to overbroad exception handlers), and a cause of creeping complexity in API design (as more and more APIs, both in the standard library and elsewhere, acquire ways to request implicit exception handling). That's why the comparison to PEP 308 is appropriate: it's less about making the language better directly, and more about deciding the consequences of not having it are no longer acceptable. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Fri Feb 21 15:34:34 2014 From: brett at python.org (Brett Cannon) Date: Fri, 21 Feb 2014 09:34:34 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140221145205.0b8b670d@fsol> References: <20140221145205.0b8b670d@fsol> Message-ID: On Fri, Feb 21, 2014 at 8:52 AM, Antoine Pitrou wrote: > On Fri, 21 Feb 2014 14:15:59 +1100 > Chris Angelico wrote: > > > > A number of functions and methods have parameters which will cause > > them to return a specified value instead of raising an exception. The > > current system is ad-hoc and inconsistent, and requires that each > > function be individually written to have this functionality; not all > > support this. > > While I sympathize with the motivation, I really don't like the end > result: > > > lst = [1, 2] > > value = lst[2] except IndexError: "No value" > > is too much of a break from the usual stylistic conventions, and looks > like several statements were stuffed on a single line. > > In other words, the gain in concision is counterbalanced by a loss in > readability, a compromise which doesn't fit in Python's overall design > principles. > > (compare with "with", which solves actual readability issues due to the > distance between the "try" and the "finally" clause, and also promotes > better resource management) > While I like the general concept, I agree that it looks too much like a crunched statement; the use of the colon is a non-starter for me. I'm sure I'm not the only one whose brain has been trained to view a colon in Python to mean "statement", period. This goes against that syntactic practice and just doesn't work for me. I'm -1 with the current syntax, but it can go into the + range if a better syntax can be chosen. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Fri Feb 21 15:37:25 2014 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 21 Feb 2014 06:37:25 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Fri, Feb 21, 2014 at 6:28 AM, Nick Coghlan wrote: > On 22 February 2014 00:03, Eli Bendersky wrote: > > On Thu, Feb 20, 2014 at 7:15 PM, Chris Angelico > wrote: > >> > >> PEP: 463 > >> Title: Exception-catching expressions > >> Version: $Revision$ > >> Last-Modified: $Date$ > >> Author: Chris Angelico > >> Status: Draft > >> Type: Standards Track > >> Content-Type: text/x-rst > >> Created: 15-Feb-2014 > >> Python-Version: 3.5 > >> Post-History: 16-Feb-2014, 21-Feb-2014 > >> > >> > >> Abstract > >> ======== > >> > >> Just as PEP 308 introduced a means of value-based conditions in an > >> expression, this system allows exception-based conditions to be used > >> as part of an expression. > > > > > > > > Chris, while I also commend you for the comprehensive PEP, I'm -1 on the > > proposal, for two main reasons: > > > > 1. Many proposals suggest new syntax to gain some succinctness. Each has > to > > be judged for its own merits, and in this case IMHO the cons eclipse the > > pros. I don't think this will save a lot of code in a typical > > well-structured program - maybe a few lines out of hundreds. On the other > > hand, it adds yet another syntax to remember and understand, which is not > > the Pythonic way. > > > > 2. Worse, this idea subverts exceptions to control flow, which is not > only > > un-Pythonic but also against the accepted practices of programming in > > general. Here, the comparison to PEP 308 is misguided. PEP 308, whatever > > syntax it adds, still remains within the domain of normal control flow. > PEP > > 463, OTOH, makes it deliberately easy to make exceptions part of > > non-exceptional code, encouraging very bad programming practices. > > Neither of these objections addresses the problems with the status quo, > though: > > - the status quo encourages overbroad exception handling (as > illustrated by examples in the PEP) > - the status quo encourages an ad hoc approach to hiding exception > handling inside functions > I think the PEP, and your reply, focuses too much on one single "status quo" situation, which is the dict.get-like usage. However, the PEP does not propose a narrow solution - it proposes a significant change in the way expressions may be written and exceptions may be caught, and thus opens a can of worms. Even if the status quo will be made better by it, and even this I'm not sure about (*), many many other possibilities of bad code open up. (*) IMHO it's better to hide these exceptions inside well defined functions -- because dict.get tells me it's the normal code flow, not the exceptional code flow. On the other hand, the syntax proposed herein tells me - yeah it's the exceptional code flow, but let me merge it into the normal code flow for you. This goes against anything I understand about how exceptions should and should not be used. Eli > > PEP 308 was accepted primarily because the and/or hack was a bug > magnet. The status quo for exception handling is both a bug magnet > (due to overbroad exception handlers), and a cause of creeping > complexity in API design (as more and more APIs, both in the standard > library and elsewhere, acquire ways to request implicit exception > handling). > > That's why the comparison to PEP 308 is appropriate: it's less about > making the language better directly, and more about deciding the > consequences of not having it are no longer acceptable. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Feb 21 15:42:15 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Feb 2014 00:42:15 +1000 Subject: [Python-Dev] Tangent on class level scoping rules (was Re: PEP 463: Exception-catching expressions) Message-ID: On 22 February 2014 00:22, Rob Cliffe wrote: > Thanks for looking into this Nick. > I confess I don't entirely understand the technical argument (my > understanding breaks down at "the subexpressions can't see the class level > variables", but I don't want to waste anybody's time expecting an > explanation, I can always look into it myself) It's a relatively arcane scoping rule that only matters if you have non-trivial logic at class scope. The vast majority of Python programmers will never have to care, because they do the typical thing and their class bodies consist almost entirely of function definitions and relatively simple assignment statements. However, it's something that expression level name binding proposals need to take into account if they want to avoid side effects on the containing scope. Python's scoping rules are deliberately designed so that methods *can't* see class attributes directly, they have to access them via their first argument or the magic __class__ variable (Py3 only for the latter). For example: >>> class C: ... attr = "Hello" ... def m(self): ... print(self.attr) ... print(attr) ... >>> C().m() Hello Traceback (most recent call last): File "", line 1, in File "", line 5, in m NameError: global name 'attr' is not defined The trick that makes it relevant here is that the "hidden" iteration variable in comprehensions is implemented by creating an implicit closure and then calling it. From the point of view of the scoping rules, that implicit closure behaves the same way as any other method definition, so most of the subexpressions can't see the class variables. The sole exception is the outermost iterator expression, because that is evaluated in the containing scope and then passed to the closure as an argument (this is all based on the way generator expressions work, and those *have* to be a closure, because they need to run in their own completely independent frame). As Chris later noted, you likely *could* still implement expression local name binding for an except expression without a full closure, it would just be rather difficult. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From barry at python.org Fri Feb 21 15:44:41 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 21 Feb 2014 09:44:41 -0500 Subject: [Python-Dev] cherry pick a change to Enum In-Reply-To: <53070FF9.9050907@stoneleaf.us> References: <53070FF9.9050907@stoneleaf.us> Message-ID: <20140221094441.02891b90@anarchist.wooz.org> On Feb 21, 2014, at 12:36 AM, Ethan Furman wrote: >If not, should we put the change in RC2 / Final? The main reason I'm pushing >for this is that Enum is still new, but after 3.4.0 is cut we then have to >deal with backwards compatibility issues. I concur this should be cherry picked. Let's not hobble this API for all eternity if it can be fixed before the release. Thanks for working so hard to make enums great in Python 3.4. -Barry From brett at python.org Fri Feb 21 15:39:42 2014 From: brett at python.org (Brett Cannon) Date: Fri, 21 Feb 2014 09:39:42 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Fri, Feb 21, 2014 at 9:03 AM, Eli Bendersky wrote: > On Thu, Feb 20, 2014 at 7:15 PM, Chris Angelico wrote: > >> PEP: 463 >> Title: Exception-catching expressions >> Version: $Revision$ >> Last-Modified: $Date$ >> Author: Chris Angelico >> Status: Draft >> Type: Standards Track >> Content-Type: text/x-rst >> Created: 15-Feb-2014 >> Python-Version: 3.5 >> Post-History: 16-Feb-2014, 21-Feb-2014 >> >> >> Abstract >> ======== >> >> Just as PEP 308 introduced a means of value-based conditions in an >> expression, this system allows exception-based conditions to be used >> as part of an expression. >> > > > Chris, while I also commend you for the comprehensive PEP, I'm -1 on the > proposal, for two main reasons: > > 1. Many proposals suggest new syntax to gain some succinctness. Each has > to be judged for its own merits, and in this case IMHO the cons eclipse the > pros. I don't think this will save a lot of code in a typical > well-structured program - maybe a few lines out of hundreds. On the other > hand, it adds yet another syntax to remember and understand, which is not > the Pythonic way. > > 2. Worse, this idea subverts exceptions to control flow, which is not only > un-Pythonic but also against the accepted practices of programming in > general. Here, the comparison to PEP 308 is misguided. PEP 308, whatever > syntax it adds, still remains within the domain of normal control flow. PEP > 463, OTOH, makes it deliberately easy to make exceptions part of > non-exceptional code, encouraging very bad programming practices. > > But we subsumed using exception for control flow long ago. StopIteration is the most blatant, but when we added dict.get(), getattr() with a default value, iter() with its default, etc. we made it very clear that while exceptions typically represent exceptional situations, they can also be used as an easy signal of something which we expect you to handle and easily recover from (and returning None or some other singleton doesn't make sense), and yet without explicit API support it takes a heavy-handed statement to deal with. -Brett > Eli > > > > >> >> >> Motivation >> ========== >> >> A number of functions and methods have parameters which will cause >> them to return a specified value instead of raising an exception. The >> current system is ad-hoc and inconsistent, and requires that each >> function be individually written to have this functionality; not all >> support this. >> >> * dict.get(key, default) - second positional argument in place of >> KeyError >> >> * next(iter, default) - second positional argument in place of >> StopIteration >> >> * list.pop() - no way to return a default >> >> * seq[index] - no way to handle a bounds error >> >> * min(sequence, default=default) - keyword argument in place of >> ValueError >> >> * sum(sequence, start=default) - slightly different but can do the >> same job >> >> * statistics.mean(data) - no way to handle an empty iterator >> >> >> Rationale >> ========= >> >> The current system requires that a function author predict the need >> for a default, and implement support for it. If this is not done, a >> full try/except block is needed. >> >> Since try/except is a statement, it is impossible to catch exceptions >> in the middle of an expression. Just as if/else does for conditionals >> and lambda does for function definitions, so does this allow exception >> catching in an expression context. >> >> This provides a clean and consistent way for a function to provide a >> default: it simply raises an appropriate exception, and the caller >> catches it. >> >> With some situations, an LBYL technique can be used (checking if some >> sequence has enough length before indexing into it, for instance). This is >> not safe in all cases, but as it is often convenient, programmers will be >> tempted to sacrifice the safety of EAFP in favour of the notational >> brevity >> of LBYL. Additionally, some LBYL techniques (eg involving getattr with >> three arguments) warp the code into looking like literal strings rather >> than attribute lookup, which can impact readability. A convenient EAFP >> notation solves all of this. >> >> There's no convenient way to write a helper function to do this; the >> nearest is something ugly using either lambda:: >> >> def except_(expression, exception_list, default): >> try: >> return expression() >> except exception_list: >> return default() >> value = except_(lambda: 1/x, ZeroDivisionError, lambda: float("nan")) >> >> which is clunky, and unable to handle multiple exception clauses; or >> eval:: >> >> def except_(expression, exception_list, default): >> try: >> return eval(expression, globals_of_caller(), >> locals_of_caller()) >> except exception_list as exc: >> l = locals_of_caller().copy() >> l['exc'] = exc >> return eval(default, globals_of_caller(), l) >> >> def globals_of_caller(): >> return sys._getframe(2).f_globals >> >> def locals_of_caller(): >> return sys._getframe(2).f_locals >> >> value = except_("""1/x""",ZeroDivisionError,""" "Can't divide by >> zero" """) >> >> which is even clunkier, and relies on implementation-dependent hacks. >> (Writing globals_of_caller() and locals_of_caller() for interpreters >> other than CPython is left as an exercise for the reader.) >> >> Raymond Hettinger `expresses`__ a desire for such a consistent >> API. Something similar has been `requested`__ `multiple`__ `times`__ >> in the past. >> >> __ >> https://mail.python.org/pipermail/python-ideas/2014-February/025443.html >> __ https://mail.python.org/pipermail/python-ideas/2013-March/019760.html >> __ https://mail.python.org/pipermail/python-ideas/2009-August/005441.html >> __ https://mail.python.org/pipermail/python-ideas/2008-August/001801.html >> >> >> Proposal >> ======== >> >> Just as the 'or' operator and the three part 'if-else' expression give >> short circuiting methods of catching a falsy value and replacing it, >> this syntax gives a short-circuiting method of catching an exception >> and replacing it. >> >> This currently works:: >> >> lst = [1, 2, None, 3] >> value = lst[2] or "No value" >> >> The proposal adds this:: >> >> lst = [1, 2] >> value = lst[2] except IndexError: "No value" >> >> Specifically, the syntax proposed is:: >> >> expr except exception_list: default >> >> where expr, exception_list, and default are all expressions. First, >> expr is evaluated. If no exception is raised, its value is the value >> of the overall expression. If any exception is raised, exception_list >> is evaluated, and should result in either a type or a tuple, just as >> with the statement form of try/except. Any matching exception will >> result in the corresponding default expression being evaluated and >> becoming the value of the expression. As with the statement form of >> try/except, non-matching exceptions will propagate upward. >> >> Note that the current proposal does not allow the exception object to >> be captured. Where this is needed, the statement form must be used. >> (See below for discussion and elaboration on this.) >> >> This ternary operator would be between lambda and if/else in >> precedence. >> >> Consider this example of a two-level cache:: >> for key in sequence: >> x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: >> f(key))) >> # do something with x >> >> This cannot be rewritten as:: >> x = lvl1.get(key, lvl2.get(key, f(key))) >> >> which, despite being shorter, defeats the purpose of the cache, as it must >> calculate a default value to pass to get(). The .get() version calculates >> backwards; the exception-testing version calculates forwards, as would be >> expected. The nearest useful equivalent would be:: >> x = lvl1.get(key) or lvl2.get(key) or f(key) >> which depends on the values being nonzero, as well as depending on the >> cache >> object supporting this functionality. >> >> >> Alternative Proposals >> ===================== >> >> Discussion on python-ideas brought up the following syntax suggestions:: >> >> value = expr except default if Exception [as e] >> value = expr except default for Exception [as e] >> value = expr except default from Exception [as e] >> value = expr except Exception [as e] return default >> value = expr except (Exception [as e]: default) >> value = expr except Exception [as e] try default >> value = expr except Exception [as e] continue with default >> value = default except Exception [as e] else expr >> value = try expr except Exception [as e]: default >> value = expr except default # Catches anything >> value = expr except(Exception) default # Catches only the named >> type(s) >> value = default if expr raise Exception >> value = expr or else default if Exception >> value = expr except Exception [as e] -> default >> value = expr except Exception [as e] pass default >> >> It has also been suggested that a new keyword be created, rather than >> reusing an existing one. Such proposals fall into the same structure >> as the last form, but with a different keyword in place of 'pass'. >> Suggestions include 'then', 'when', and 'use'. Also, in the context of >> the "default if expr raise Exception" proposal, it was suggested that a >> new keyword "raises" be used. >> >> All forms involving the 'as' capturing clause have been deferred from >> this proposal in the interests of simplicity, but are preserved in the >> table above as an accurate record of suggestions. >> >> >> Open Issues >> =========== >> >> Parentheses around the entire expression >> ---------------------------------------- >> >> Generator expressions require parentheses, unless they would be >> strictly redundant. Ambiguities with except expressions could be >> resolved in the same way, forcing nested except-in-except trees to be >> correctly parenthesized and requiring that the outer expression be >> clearly delineated. `Steven D'Aprano elaborates on the issue.`__ >> >> __ >> https://mail.python.org/pipermail/python-ideas/2014-February/025647.html >> >> >> Example usage >> ============= >> >> For each example, an approximately-equivalent statement form is given, >> to show how the expression will be parsed. These are not always >> strictly equivalent, but will accomplish the same purpose. It is NOT >> safe for the interpreter to translate one into the other. >> >> A number of these examples are taken directly from the Python standard >> library, with file names and line numbers correct as of early Feb 2014. >> Many of these patterns are extremely common. >> >> Retrieve an argument, defaulting to None:: >> cond = args[1] except IndexError: None >> >> # Lib/pdb.py:803: >> try: >> cond = args[1] >> except IndexError: >> cond = None >> >> Fetch information from the system if available:: >> pwd = os.getcwd() except OSError: None >> >> # Lib/tkinter/filedialog.py:210: >> try: >> pwd = os.getcwd() >> except OSError: >> pwd = None >> >> Attempt a translation, falling back on the original:: >> e.widget = self._nametowidget(W) except KeyError: W >> >> # Lib/tkinter/__init__.py:1222: >> try: >> e.widget = self._nametowidget(W) >> except KeyError: >> e.widget = W >> >> Read from an iterator, continuing with blank lines once it's >> exhausted:: >> line = readline() except StopIteration: '' >> >> # Lib/lib2to3/pgen2/tokenize.py:370: >> try: >> line = readline() >> except StopIteration: >> line = '' >> >> Retrieve platform-specific information (note the DRY improvement); >> this particular example could be taken further, turning a series of >> separate assignments into a single large dict initialization:: >> # sys.abiflags may not be defined on all platforms. >> _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: '' >> >> # Lib/sysconfig.py:529: >> try: >> _CONFIG_VARS['abiflags'] = sys.abiflags >> except AttributeError: >> # sys.abiflags may not be defined on all platforms. >> _CONFIG_VARS['abiflags'] = '' >> >> Retrieve an indexed item, defaulting to None (similar to dict.get):: >> def getNamedItem(self, name): >> return self._attrs[name] except KeyError: None >> >> # Lib/xml/dom/minidom.py:573: >> def getNamedItem(self, name): >> try: >> return self._attrs[name] >> except KeyError: >> return None >> >> >> Translate numbers to names, falling back on the numbers:: >> g = grp.getgrnam(tarinfo.gname)[2] except KeyError: >> tarinfo.gid >> u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: >> tarinfo.uid >> >> # Lib/tarfile.py:2198: >> try: >> g = grp.getgrnam(tarinfo.gname)[2] >> except KeyError: >> g = tarinfo.gid >> try: >> u = pwd.getpwnam(tarinfo.uname)[2] >> except KeyError: >> u = tarinfo.uid >> >> Perform some lengthy calculations in EAFP mode, handling division by >> zero as a sort of sticky NaN:: >> >> value = calculate(x) except ZeroDivisionError: float("nan") >> >> try: >> value = calculate(x) >> except ZeroDivisionError: >> value = float("nan") >> >> Calculate the mean of a series of numbers, falling back on zero:: >> >> value = statistics.mean(lst) except statistics.StatisticsError: 0 >> >> try: >> value = statistics.mean(lst) >> except statistics.StatisticsError: >> value = 0 >> >> Retrieving a message from either a cache or the internet, with auth >> check:: >> >> logging.info("Message shown to user: %s",((cache[k] >> except LookupError: >> (backend.read(k) except OSError: 'Resource not available') >> ) >> if check_permission(k) else 'Access denied' >> ) except BaseException: "This is like a bare except clause") >> >> try: >> if check_permission(k): >> try: >> _ = cache[k] >> except LookupError: >> try: >> _ = backend.read(k) >> except OSError: >> _ = 'Resource not available' >> else: >> _ = 'Access denied' >> except BaseException: >> _ = "This is like a bare except clause" >> logging.info("Message shown to user: %s", _) >> >> Looking up objects in a sparse list of overrides:: >> >> (overrides[x] or default except IndexError: default).ping() >> >> try: >> (overrides[x] or default).ping() >> except IndexError: >> default.ping() >> >> >> Narrowing of exception-catching scope >> ------------------------------------- >> >> The following examples, taken directly from Python's standard library, >> demonstrate how the scope of the try/except can be conveniently narrowed. >> To do this with the statement form of try/except would require a temporary >> variable, but it's far cleaner as an expression. >> >> Lib/ipaddress.py:343:: >> try: >> ips.append(ip.ip) >> except AttributeError: >> ips.append(ip.network_address) >> Becomes:: >> ips.append(ip.ip except AttributeError: ip.network_address) >> The expression form is nearly equivalent to this:: >> try: >> _ = ip.ip >> except AttributeError: >> _ = ip.network_address >> ips.append(_) >> >> Lib/tempfile.py:130:: >> try: >> dirlist.append(_os.getcwd()) >> except (AttributeError, OSError): >> dirlist.append(_os.curdir) >> Becomes:: >> dirlist.append(_os.getcwd() except (AttributeError, OSError): >> _os.curdir) >> >> Lib/asyncore.py:264:: >> try: >> status.append('%s:%d' % self.addr) >> except TypeError: >> status.append(repr(self.addr)) >> Becomes:: >> status.append('%s:%d' % self.addr except TypeError: >> repr(self.addr)) >> >> >> Comparisons with other languages >> ================================ >> >> (With thanks to Andrew Barnert for compiling this section.) >> >> `Ruby's`__ "begin?rescue?rescue?else?ensure?end" is an expression >> (potentially with statements inside it). It has the equivalent of an "as" >> clause, and the equivalent of bare except. And it uses no punctuation or >> keyword between the bare except/exception class/exception class with as >> clause and the value. (And yes, it's ambiguous unless you understand >> Ruby's statement/expression rules.) >> >> __ http://www.skorks.com/2009/09/ruby-exceptions-and-exception-handling/ >> >> :: >> >> x = begin computation() rescue MyException => e default(e) end; >> x = begin computation() rescue MyException default() end; >> x = begin computation() rescue default() end; >> x = begin computation() rescue MyException default() rescue >> OtherException other() end; >> >> In terms of this PEP:: >> >> x = computation() except MyException as e default(e) >> x = computation() except MyException default(e) >> x = computation() except default(e) >> x = computation() except MyException default() except OtherException >> other() >> >> `Erlang`__ has a try expression that looks like this:: >> >> __ http://erlang.org/doc/reference_manual/expressions.html#id79284 >> >> x = try computation() catch MyException:e -> default(e) end; >> x = try computation() catch MyException:e -> default(e); >> OtherException:e -> other(e) end; >> >> The class and "as" name are mandatory, but you can use "_" for either. >> There's also an optional "when" guard on each, and a "throw" clause that >> you can catch, which I won't get into. To handle multiple exceptions, >> you just separate the clauses with semicolons, which I guess would map >> to commas in Python. So:: >> >> x = try computation() except MyException as e -> default(e) >> x = try computation() except MyException as e -> default(e), >> OtherException as e->other_default(e) >> >> Erlang also has a "catch" expression, which, despite using the same >> keyword, >> is completely different, and you don't want to know about it. >> >> >> The ML family has two different ways of dealing with this, "handle" and >> "try"; the difference between the two is that "try" pattern-matches the >> exception, which gives you the effect of multiple except clauses and as >> clauses. In either form, the handler clause is punctuated by "=>" in >> some dialects, "->" in others. >> >> To avoid confusion, I'll write the function calls in Python style. >> >> Here's `SML's`__ "handle":: >> __ http://www.cs.cmu.edu/~rwh/introsml/core/exceptions.htm >> >> let x = computation() handle MyException => default();; >> >> Here's `OCaml's`__ "try":: >> __ http://www2.lib.uchicago.edu/keith/ocaml-class/exceptions.html >> >> let x = try computation() with MyException explanation -> >> default(explanation);; >> >> let x = try computation() with >> >> MyException(e) -> default(e) >> | MyOtherException() -> other_default() >> | (e) -> fallback(e);; >> >> In terms of this PEP, these would be something like:: >> >> x = computation() except MyException => default() >> x = try computation() except MyException e -> default() >> x = (try computation() >> except MyException as e -> default(e) >> except MyOtherException -> other_default() >> except BaseException as e -> fallback(e)) >> >> Many ML-inspired but not-directly-related languages from academia mix >> things >> up, usually using more keywords and fewer symbols. So, the `Oz`__ would >> map >> to Python as:: >> __ http://mozart.github.io/mozart-v1/doc-1.4.0/tutorial/node5.html >> >> x = try computation() catch MyException as e then default(e) >> >> >> Many Lisp-derived languages, like `Clojure,`__ implement try/catch as >> special >> forms (if you don't know what that means, think function-like macros), so >> you >> write, effectively:: >> __ >> http://clojure.org/special_forms#Special%20Forms--(try%20expr*%20catch-clause*%20finally-clause >> ?) >> >> try(computation(), catch(MyException, explanation, >> default(explanation))) >> >> try(computation(), >> catch(MyException, explanation, default(explanation)), >> catch(MyOtherException, explanation, other_default(explanation))) >> >> In Common Lisp, this is done with a slightly clunkier `"handler-case" >> macro,`__ >> but the basic idea is the same. >> >> __ http://clhs.lisp.se/Body/m_hand_1.htm >> >> >> The Lisp style is, surprisingly, used by some languages that don't have >> macros, like Lua, where `xpcall`__ takes functions. Writing lambdas >> Python-style instead of Lua-style:: >> __ http://www.gammon.com.au/scripts/doc.php?lua=xpcall >> >> x = xpcall(lambda: expression(), lambda e: default(e)) >> >> This actually returns (true, expression()) or (false, default(e)), but >> I think we can ignore that part. >> >> >> Haskell is actually similar to Lua here (except that it's all done >> with monads, of course):: >> >> x = do catch(lambda: expression(), lambda e: default(e)) >> >> You can write a pattern matching expression within the function to decide >> what to do with it; catching and re-raising exceptions you don't want is >> cheap enough to be idiomatic. >> >> But Haskell infixing makes this nicer:: >> >> x = do expression() `catch` lambda: default() >> x = do expression() `catch` lambda e: default(e) >> >> And that makes the parallel between the lambda colon and the except >> colon in the proposal much more obvious:: >> >> >> x = expression() except Exception: default() >> x = expression() except Exception as e: default(e) >> >> >> `Tcl`__ has the other half of Lua's xpcall; catch is a function which >> returns >> true if an exception was caught, false otherwise, and you get the value >> out >> in other ways. And it's all built around the the implicit quote-and-exec >> that everything in Tcl is based on, making it even harder to describe in >> Python terms than Lisp macros, but something like:: >> __ http://wiki.tcl.tk/902 >> >> if {[ catch("computation()") "explanation"]} { default(explanation) } >> >> >> `Smalltalk`__ is also somewhat hard to map to Python. The basic version >> would be:: >> __ http://smalltalk.gnu.org/wiki/exceptions >> >> x := computation() on:MyException do:default() >> >> ? but that's basically Smalltalk's passing-arguments-with-colons >> syntax, not its exception-handling syntax. >> >> >> Deferred sub-proposals >> ====================== >> >> Multiple except clauses >> ----------------------- >> >> An examination of use-cases shows that this is not needed as often as >> it would be with the statement form, and as its syntax is a point on >> which consensus has not been reached, the entire feature is deferred. >> >> In order to ensure compatibility with future versions, ensure that any >> consecutive except operators are parenthesized to guarantee the >> interpretation you expect. >> >> Multiple 'except' keywords can be used, and they will all catch >> exceptions raised in the original expression (only):: >> >> # Will catch any of the listed exceptions thrown by expr; >> # any exception thrown by a default expression will propagate. >> value = (expr >> except Exception1 [as e]: default1 >> except Exception2 [as e]: default2 >> # ... except ExceptionN [as e]: defaultN >> ) >> >> Using parentheses to force an alternative interpretation works as >> expected:: >> >> # Will catch an Exception2 thrown by either expr or default1 >> value = ( >> (expr except Exception1: default1) >> except Exception2: default2 >> ) >> # Will catch an Exception2 thrown by default1 only >> value = (expr except Exception1: >> (default1 except Exception2: default2) >> ) >> >> This last form is confusing and should be discouraged by PEP 8, but it >> is syntactically legal: you can put any sort of expression inside a >> ternary-except; ternary-except is an expression; therefore you can put >> a ternary-except inside a ternary-except. >> >> Open question: Where there are multiple except clauses, should they be >> separated by commas? It may be easier for the parser, that way:: >> >> value = (expr >> except Exception1 [as e]: default1, >> except Exception2 [as e]: default2, >> # ... except ExceptionN [as e]: defaultN, >> ) >> >> with an optional comma after the last, as per tuple rules. Downside: >> Omitting the comma would be syntactically valid, and would have almost >> identical semantics, but would nest the entire preceding expression in >> its exception catching rig - a matching exception raised in the >> default clause would be caught by the subsequent except clause. As >> this difference is so subtle, it runs the risk of being a major bug >> magnet. >> >> As a mitigation of this risk, this form:: >> >> value = expr except Exception1: default1 except Exception2: default2 >> >> could be syntactically forbidden, and parentheses required if the >> programmer actually wants that behaviour:: >> >> value = (expr except Exception1: default1) except Exception2: default2 >> >> This would prevent the accidental omission of a comma from changing >> the expression's meaning. >> >> >> Capturing the exception object >> ------------------------------ >> >> In a try/except block, the use of 'as' to capture the exception object >> creates a local name binding, and implicitly deletes that binding in a >> finally clause. As 'finally' is not a part of this proposal (see >> below), this makes it tricky to describe; also, this use of 'as' gives >> a way to create a name binding in an expression context. Should the >> default clause have an inner scope in which the name exists, shadowing >> anything of the same name elsewhere? Should it behave the same way the >> statement try/except does, and unbind the name? Should it bind the >> name and leave it bound? (Almost certainly not; this behaviour was >> changed in Python 3 for good reason.) >> >> Additionally, this syntax would allow a convenient way to capture >> exceptions in interactive Python; returned values are captured by "_", >> but exceptions currently are not. This could be spelled: >> >> >>> expr except Exception as e: e >> >> (The inner scope idea is tempting, but currently CPython handles list >> comprehensions with a nested function call, as this is considered >> easier. It may be of value to simplify both comprehensions and except >> expressions, but that is a completely separate proposal to this PEP; >> alternatively, it may be better to stick with what's known to >> work. `Nick Coghlan elaborates.`__) >> >> __ >> https://mail.python.org/pipermail/python-ideas/2014-February/025702.html >> >> An examination of the Python standard library shows that, while the use >> of 'as' is fairly common (occurring in roughly one except clause in five), >> it is extremely *uncommon* in the cases which could logically be converted >> into the expression form. Its few uses can simply be left unchanged. >> Consequently, in the interests of simplicity, the 'as' clause is not >> included in this proposal. A subsequent Python version can add this >> without >> breaking any existing code, as 'as' is already a keyword. >> >> One example where this could possibly be useful is Lib/imaplib.py:568:: >> try: typ, dat = self._simple_command('LOGOUT') >> except: typ, dat = 'NO', ['%s: %s' % sys.exc_info()[:2]] >> This could become:: >> typ, dat = (self._simple_command('LOGOUT') >> except BaseException as e: ('NO', '%s: %s' % (type(e), e))) >> Or perhaps some other variation. This is hardly the most compelling >> use-case, >> but an intelligent look at this code could tidy it up significantly. In >> the >> absence of further examples showing any need of the exception object, I >> have >> opted to defer indefinitely the recommendation. >> >> >> Rejected sub-proposals >> ====================== >> >> finally clause >> -------------- >> The statement form try... finally or try... except... finally has no >> logical corresponding expression form. Therefore the finally keyword >> is not a part of this proposal, in any way. >> >> >> Bare except having different meaning >> ------------------------------------ >> >> With several of the proposed syntaxes, omitting the exception type name >> would be easy and concise, and would be tempting. For convenience's sake, >> it might be advantageous to have a bare 'except' clause mean something >> more useful than "except BaseException". Proposals included having it >> catch Exception, or some specific set of "common exceptions" (subclasses >> of a new type called ExpressionError), or have it look for a tuple named >> ExpressionError in the current scope, with a built-in default such as >> (ValueError, UnicodeError, AttributeError, EOFError, IOError, OSError, >> LookupError, NameError, ZeroDivisionError). All of these were rejected, >> for severa reasons. >> >> * First and foremost, consistency with the statement form of try/except >> would be broken. Just as a list comprehension or ternary if expression >> can be explained by "breaking it out" into its vertical statement form, >> an expression-except should be able to be explained by a relatively >> mechanical translation into a near-equivalent statement. Any form of >> syntax common to both should therefore have the same semantics in each, >> and above all should not have the subtle difference of catching more in >> one than the other, as it will tend to attract unnoticed bugs. >> >> * Secondly, the set of appropriate exceptions to catch would itself be >> a huge point of contention. It would be impossible to predict exactly >> which exceptions would "make sense" to be caught; why bless some of them >> with convenient syntax and not others? >> >> * And finally (this partly because the recommendation was that a bare >> except should be actively encouraged, once it was reduced to a >> "reasonable" >> set of exceptions), any situation where you catch an exception you don't >> expect to catch is an unnecessary bug magnet. >> >> Consequently, the use of a bare 'except' is down to two possibilities: >> either it is syntactically forbidden in the expression form, or it is >> permitted with the exact same semantics as in the statement form (namely, >> that it catch BaseException and be unable to capture it with 'as'). >> >> >> Bare except clauses >> ------------------- >> >> PEP 8 rightly advises against the use of a bare 'except'. While it is >> syntactically legal in a statement, and for backward compatibility must >> remain so, there is little value in encouraging its use. In an expression >> except clause, "except:" is a SyntaxError; use the equivalent long-hand >> form "except BaseException:" instead. A future version of Python MAY >> choose >> to reinstate this, which can be done without breaking compatibility. >> >> >> Parentheses around the except clauses >> ------------------------------------- >> >> Should it be legal to parenthesize the except clauses, separately from >> the expression that could raise? Example:: >> >> value = expr ( >> except Exception1 [as e]: default1 >> except Exception2 [as e]: default2 >> # ... except ExceptionN [as e]: defaultN >> ) >> >> This is more compelling when one or both of the deferred sub-proposals >> of multiple except clauses and/or exception capturing is included. In >> their absence, the parentheses would be thus:: >> value = expr except ExceptionType: default >> value = expr (except ExceptionType: default) >> >> The advantage is minimal, and the potential to confuse a reader into >> thinking the except clause is separate from the expression, or into >> thinking >> this is a function call, makes this non-compelling. The expression can, >> of >> course, be parenthesized if desired, as can the default:: >> value = (expr) except ExceptionType: (default) >> >> >> Short-hand for "except: pass" >> ----------------------------- >> >> The following was been suggested as a similar >> short-hand, though not technically an expression:: >> >> statement except Exception: pass >> >> try: >> statement >> except Exception: >> pass >> >> For instance, a common use-case is attempting the removal of a file:: >> os.unlink(some_file) except OSError: pass >> >> There is an equivalent already in Python 3.4, however, in contextlib:: >> from contextlib import suppress >> with suppress(OSError): os.unlink(some_file) >> >> As this is already a single line (or two with a break after the colon), >> there is little need of new syntax and a confusion of statement vs >> expression to achieve this. >> >> >> Copyright >> ========= >> >> This document has been placed in the public domain. >> >> >> .. >> Local Variables: >> mode: indented-text >> indent-tabs-mode: nil >> sentence-end-double-space: t >> fill-column: 70 >> coding: utf-8 >> End: >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Fri Feb 21 15:44:45 2014 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 21 Feb 2014 06:44:45 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Fri, Feb 21, 2014 at 6:39 AM, Brett Cannon wrote: > > > > On Fri, Feb 21, 2014 at 9:03 AM, Eli Bendersky wrote: > >> On Thu, Feb 20, 2014 at 7:15 PM, Chris Angelico wrote: >> >>> PEP: 463 >>> Title: Exception-catching expressions >>> Version: $Revision$ >>> Last-Modified: $Date$ >>> Author: Chris Angelico >>> Status: Draft >>> Type: Standards Track >>> Content-Type: text/x-rst >>> Created: 15-Feb-2014 >>> Python-Version: 3.5 >>> Post-History: 16-Feb-2014, 21-Feb-2014 >>> >>> >>> Abstract >>> ======== >>> >>> Just as PEP 308 introduced a means of value-based conditions in an >>> expression, this system allows exception-based conditions to be used >>> as part of an expression. >>> >> >> >> Chris, while I also commend you for the comprehensive PEP, I'm -1 on the >> proposal, for two main reasons: >> >> 1. Many proposals suggest new syntax to gain some succinctness. Each has >> to be judged for its own merits, and in this case IMHO the cons eclipse the >> pros. I don't think this will save a lot of code in a typical >> well-structured program - maybe a few lines out of hundreds. On the other >> hand, it adds yet another syntax to remember and understand, which is not >> the Pythonic way. >> >> 2. Worse, this idea subverts exceptions to control flow, which is not >> only un-Pythonic but also against the accepted practices of programming in >> general. Here, the comparison to PEP 308 is misguided. PEP 308, whatever >> syntax it adds, still remains within the domain of normal control flow. PEP >> 463, OTOH, makes it deliberately easy to make exceptions part of >> non-exceptional code, encouraging very bad programming practices. >> >> > But we subsumed using exception for control flow long ago. > StopIteration is the most blatant, but when we added dict.get(), getattr() > with a default value, iter() with its default, etc. we made it very clear > that while exceptions typically represent exceptional situations, they can > also be used as an easy signal of something which we expect you to handle > and easily recover from (and returning None or some other singleton doesn't > make sense), and yet without explicit API support it takes a heavy-handed > statement to deal with. > > True, but at least you still have to explicitly try...except... which takes a toll on the code so isn't taken lightly. Adding except into expressions, I fear, will proliferate this usage much more. Eli > -Brett > > > >> Eli >> >> >> >> >>> >>> >>> Motivation >>> ========== >>> >>> A number of functions and methods have parameters which will cause >>> them to return a specified value instead of raising an exception. The >>> current system is ad-hoc and inconsistent, and requires that each >>> function be individually written to have this functionality; not all >>> support this. >>> >>> * dict.get(key, default) - second positional argument in place of >>> KeyError >>> >>> * next(iter, default) - second positional argument in place of >>> StopIteration >>> >>> * list.pop() - no way to return a default >>> >>> * seq[index] - no way to handle a bounds error >>> >>> * min(sequence, default=default) - keyword argument in place of >>> ValueError >>> >>> * sum(sequence, start=default) - slightly different but can do the >>> same job >>> >>> * statistics.mean(data) - no way to handle an empty iterator >>> >>> >>> Rationale >>> ========= >>> >>> The current system requires that a function author predict the need >>> for a default, and implement support for it. If this is not done, a >>> full try/except block is needed. >>> >>> Since try/except is a statement, it is impossible to catch exceptions >>> in the middle of an expression. Just as if/else does for conditionals >>> and lambda does for function definitions, so does this allow exception >>> catching in an expression context. >>> >>> This provides a clean and consistent way for a function to provide a >>> default: it simply raises an appropriate exception, and the caller >>> catches it. >>> >>> With some situations, an LBYL technique can be used (checking if some >>> sequence has enough length before indexing into it, for instance). This >>> is >>> not safe in all cases, but as it is often convenient, programmers will be >>> tempted to sacrifice the safety of EAFP in favour of the notational >>> brevity >>> of LBYL. Additionally, some LBYL techniques (eg involving getattr with >>> three arguments) warp the code into looking like literal strings rather >>> than attribute lookup, which can impact readability. A convenient EAFP >>> notation solves all of this. >>> >>> There's no convenient way to write a helper function to do this; the >>> nearest is something ugly using either lambda:: >>> >>> def except_(expression, exception_list, default): >>> try: >>> return expression() >>> except exception_list: >>> return default() >>> value = except_(lambda: 1/x, ZeroDivisionError, lambda: float("nan")) >>> >>> which is clunky, and unable to handle multiple exception clauses; or >>> eval:: >>> >>> def except_(expression, exception_list, default): >>> try: >>> return eval(expression, globals_of_caller(), >>> locals_of_caller()) >>> except exception_list as exc: >>> l = locals_of_caller().copy() >>> l['exc'] = exc >>> return eval(default, globals_of_caller(), l) >>> >>> def globals_of_caller(): >>> return sys._getframe(2).f_globals >>> >>> def locals_of_caller(): >>> return sys._getframe(2).f_locals >>> >>> value = except_("""1/x""",ZeroDivisionError,""" "Can't divide by >>> zero" """) >>> >>> which is even clunkier, and relies on implementation-dependent hacks. >>> (Writing globals_of_caller() and locals_of_caller() for interpreters >>> other than CPython is left as an exercise for the reader.) >>> >>> Raymond Hettinger `expresses`__ a desire for such a consistent >>> API. Something similar has been `requested`__ `multiple`__ `times`__ >>> in the past. >>> >>> __ >>> https://mail.python.org/pipermail/python-ideas/2014-February/025443.html >>> __ https://mail.python.org/pipermail/python-ideas/2013-March/019760.html >>> __ >>> https://mail.python.org/pipermail/python-ideas/2009-August/005441.html >>> __ >>> https://mail.python.org/pipermail/python-ideas/2008-August/001801.html >>> >>> >>> Proposal >>> ======== >>> >>> Just as the 'or' operator and the three part 'if-else' expression give >>> short circuiting methods of catching a falsy value and replacing it, >>> this syntax gives a short-circuiting method of catching an exception >>> and replacing it. >>> >>> This currently works:: >>> >>> lst = [1, 2, None, 3] >>> value = lst[2] or "No value" >>> >>> The proposal adds this:: >>> >>> lst = [1, 2] >>> value = lst[2] except IndexError: "No value" >>> >>> Specifically, the syntax proposed is:: >>> >>> expr except exception_list: default >>> >>> where expr, exception_list, and default are all expressions. First, >>> expr is evaluated. If no exception is raised, its value is the value >>> of the overall expression. If any exception is raised, exception_list >>> is evaluated, and should result in either a type or a tuple, just as >>> with the statement form of try/except. Any matching exception will >>> result in the corresponding default expression being evaluated and >>> becoming the value of the expression. As with the statement form of >>> try/except, non-matching exceptions will propagate upward. >>> >>> Note that the current proposal does not allow the exception object to >>> be captured. Where this is needed, the statement form must be used. >>> (See below for discussion and elaboration on this.) >>> >>> This ternary operator would be between lambda and if/else in >>> precedence. >>> >>> Consider this example of a two-level cache:: >>> for key in sequence: >>> x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: >>> f(key))) >>> # do something with x >>> >>> This cannot be rewritten as:: >>> x = lvl1.get(key, lvl2.get(key, f(key))) >>> >>> which, despite being shorter, defeats the purpose of the cache, as it >>> must >>> calculate a default value to pass to get(). The .get() version calculates >>> backwards; the exception-testing version calculates forwards, as would be >>> expected. The nearest useful equivalent would be:: >>> x = lvl1.get(key) or lvl2.get(key) or f(key) >>> which depends on the values being nonzero, as well as depending on the >>> cache >>> object supporting this functionality. >>> >>> >>> Alternative Proposals >>> ===================== >>> >>> Discussion on python-ideas brought up the following syntax suggestions:: >>> >>> value = expr except default if Exception [as e] >>> value = expr except default for Exception [as e] >>> value = expr except default from Exception [as e] >>> value = expr except Exception [as e] return default >>> value = expr except (Exception [as e]: default) >>> value = expr except Exception [as e] try default >>> value = expr except Exception [as e] continue with default >>> value = default except Exception [as e] else expr >>> value = try expr except Exception [as e]: default >>> value = expr except default # Catches anything >>> value = expr except(Exception) default # Catches only the named >>> type(s) >>> value = default if expr raise Exception >>> value = expr or else default if Exception >>> value = expr except Exception [as e] -> default >>> value = expr except Exception [as e] pass default >>> >>> It has also been suggested that a new keyword be created, rather than >>> reusing an existing one. Such proposals fall into the same structure >>> as the last form, but with a different keyword in place of 'pass'. >>> Suggestions include 'then', 'when', and 'use'. Also, in the context of >>> the "default if expr raise Exception" proposal, it was suggested that a >>> new keyword "raises" be used. >>> >>> All forms involving the 'as' capturing clause have been deferred from >>> this proposal in the interests of simplicity, but are preserved in the >>> table above as an accurate record of suggestions. >>> >>> >>> Open Issues >>> =========== >>> >>> Parentheses around the entire expression >>> ---------------------------------------- >>> >>> Generator expressions require parentheses, unless they would be >>> strictly redundant. Ambiguities with except expressions could be >>> resolved in the same way, forcing nested except-in-except trees to be >>> correctly parenthesized and requiring that the outer expression be >>> clearly delineated. `Steven D'Aprano elaborates on the issue.`__ >>> >>> __ >>> https://mail.python.org/pipermail/python-ideas/2014-February/025647.html >>> >>> >>> Example usage >>> ============= >>> >>> For each example, an approximately-equivalent statement form is given, >>> to show how the expression will be parsed. These are not always >>> strictly equivalent, but will accomplish the same purpose. It is NOT >>> safe for the interpreter to translate one into the other. >>> >>> A number of these examples are taken directly from the Python standard >>> library, with file names and line numbers correct as of early Feb 2014. >>> Many of these patterns are extremely common. >>> >>> Retrieve an argument, defaulting to None:: >>> cond = args[1] except IndexError: None >>> >>> # Lib/pdb.py:803: >>> try: >>> cond = args[1] >>> except IndexError: >>> cond = None >>> >>> Fetch information from the system if available:: >>> pwd = os.getcwd() except OSError: None >>> >>> # Lib/tkinter/filedialog.py:210: >>> try: >>> pwd = os.getcwd() >>> except OSError: >>> pwd = None >>> >>> Attempt a translation, falling back on the original:: >>> e.widget = self._nametowidget(W) except KeyError: W >>> >>> # Lib/tkinter/__init__.py:1222: >>> try: >>> e.widget = self._nametowidget(W) >>> except KeyError: >>> e.widget = W >>> >>> Read from an iterator, continuing with blank lines once it's >>> exhausted:: >>> line = readline() except StopIteration: '' >>> >>> # Lib/lib2to3/pgen2/tokenize.py:370: >>> try: >>> line = readline() >>> except StopIteration: >>> line = '' >>> >>> Retrieve platform-specific information (note the DRY improvement); >>> this particular example could be taken further, turning a series of >>> separate assignments into a single large dict initialization:: >>> # sys.abiflags may not be defined on all platforms. >>> _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: '' >>> >>> # Lib/sysconfig.py:529: >>> try: >>> _CONFIG_VARS['abiflags'] = sys.abiflags >>> except AttributeError: >>> # sys.abiflags may not be defined on all platforms. >>> _CONFIG_VARS['abiflags'] = '' >>> >>> Retrieve an indexed item, defaulting to None (similar to dict.get):: >>> def getNamedItem(self, name): >>> return self._attrs[name] except KeyError: None >>> >>> # Lib/xml/dom/minidom.py:573: >>> def getNamedItem(self, name): >>> try: >>> return self._attrs[name] >>> except KeyError: >>> return None >>> >>> >>> Translate numbers to names, falling back on the numbers:: >>> g = grp.getgrnam(tarinfo.gname)[2] except KeyError: >>> tarinfo.gid >>> u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: >>> tarinfo.uid >>> >>> # Lib/tarfile.py:2198: >>> try: >>> g = grp.getgrnam(tarinfo.gname)[2] >>> except KeyError: >>> g = tarinfo.gid >>> try: >>> u = pwd.getpwnam(tarinfo.uname)[2] >>> except KeyError: >>> u = tarinfo.uid >>> >>> Perform some lengthy calculations in EAFP mode, handling division by >>> zero as a sort of sticky NaN:: >>> >>> value = calculate(x) except ZeroDivisionError: float("nan") >>> >>> try: >>> value = calculate(x) >>> except ZeroDivisionError: >>> value = float("nan") >>> >>> Calculate the mean of a series of numbers, falling back on zero:: >>> >>> value = statistics.mean(lst) except statistics.StatisticsError: 0 >>> >>> try: >>> value = statistics.mean(lst) >>> except statistics.StatisticsError: >>> value = 0 >>> >>> Retrieving a message from either a cache or the internet, with auth >>> check:: >>> >>> logging.info("Message shown to user: %s",((cache[k] >>> except LookupError: >>> (backend.read(k) except OSError: 'Resource not available') >>> ) >>> if check_permission(k) else 'Access denied' >>> ) except BaseException: "This is like a bare except clause") >>> >>> try: >>> if check_permission(k): >>> try: >>> _ = cache[k] >>> except LookupError: >>> try: >>> _ = backend.read(k) >>> except OSError: >>> _ = 'Resource not available' >>> else: >>> _ = 'Access denied' >>> except BaseException: >>> _ = "This is like a bare except clause" >>> logging.info("Message shown to user: %s", _) >>> >>> Looking up objects in a sparse list of overrides:: >>> >>> (overrides[x] or default except IndexError: default).ping() >>> >>> try: >>> (overrides[x] or default).ping() >>> except IndexError: >>> default.ping() >>> >>> >>> Narrowing of exception-catching scope >>> ------------------------------------- >>> >>> The following examples, taken directly from Python's standard library, >>> demonstrate how the scope of the try/except can be conveniently narrowed. >>> To do this with the statement form of try/except would require a >>> temporary >>> variable, but it's far cleaner as an expression. >>> >>> Lib/ipaddress.py:343:: >>> try: >>> ips.append(ip.ip) >>> except AttributeError: >>> ips.append(ip.network_address) >>> Becomes:: >>> ips.append(ip.ip except AttributeError: ip.network_address) >>> The expression form is nearly equivalent to this:: >>> try: >>> _ = ip.ip >>> except AttributeError: >>> _ = ip.network_address >>> ips.append(_) >>> >>> Lib/tempfile.py:130:: >>> try: >>> dirlist.append(_os.getcwd()) >>> except (AttributeError, OSError): >>> dirlist.append(_os.curdir) >>> Becomes:: >>> dirlist.append(_os.getcwd() except (AttributeError, OSError): >>> _os.curdir) >>> >>> Lib/asyncore.py:264:: >>> try: >>> status.append('%s:%d' % self.addr) >>> except TypeError: >>> status.append(repr(self.addr)) >>> Becomes:: >>> status.append('%s:%d' % self.addr except TypeError: >>> repr(self.addr)) >>> >>> >>> Comparisons with other languages >>> ================================ >>> >>> (With thanks to Andrew Barnert for compiling this section.) >>> >>> `Ruby's`__ "begin...rescue...rescue...else...ensure...end" is an expression >>> (potentially with statements inside it). It has the equivalent of an >>> "as" >>> clause, and the equivalent of bare except. And it uses no punctuation or >>> keyword between the bare except/exception class/exception class with as >>> clause and the value. (And yes, it's ambiguous unless you understand >>> Ruby's statement/expression rules.) >>> >>> __ http://www.skorks.com/2009/09/ruby-exceptions-and-exception-handling/ >>> >>> :: >>> >>> x = begin computation() rescue MyException => e default(e) end; >>> x = begin computation() rescue MyException default() end; >>> x = begin computation() rescue default() end; >>> x = begin computation() rescue MyException default() rescue >>> OtherException other() end; >>> >>> In terms of this PEP:: >>> >>> x = computation() except MyException as e default(e) >>> x = computation() except MyException default(e) >>> x = computation() except default(e) >>> x = computation() except MyException default() except OtherException >>> other() >>> >>> `Erlang`__ has a try expression that looks like this:: >>> >>> __ http://erlang.org/doc/reference_manual/expressions.html#id79284 >>> >>> x = try computation() catch MyException:e -> default(e) end; >>> x = try computation() catch MyException:e -> default(e); >>> OtherException:e -> other(e) end; >>> >>> The class and "as" name are mandatory, but you can use "_" for either. >>> There's also an optional "when" guard on each, and a "throw" clause that >>> you can catch, which I won't get into. To handle multiple exceptions, >>> you just separate the clauses with semicolons, which I guess would map >>> to commas in Python. So:: >>> >>> x = try computation() except MyException as e -> default(e) >>> x = try computation() except MyException as e -> default(e), >>> OtherException as e->other_default(e) >>> >>> Erlang also has a "catch" expression, which, despite using the same >>> keyword, >>> is completely different, and you don't want to know about it. >>> >>> >>> The ML family has two different ways of dealing with this, "handle" and >>> "try"; the difference between the two is that "try" pattern-matches the >>> exception, which gives you the effect of multiple except clauses and as >>> clauses. In either form, the handler clause is punctuated by "=>" in >>> some dialects, "->" in others. >>> >>> To avoid confusion, I'll write the function calls in Python style. >>> >>> Here's `SML's`__ "handle":: >>> __ http://www.cs.cmu.edu/~rwh/introsml/core/exceptions.htm >>> >>> let x = computation() handle MyException => default();; >>> >>> Here's `OCaml's`__ "try":: >>> __ http://www2.lib.uchicago.edu/keith/ocaml-class/exceptions.html >>> >>> let x = try computation() with MyException explanation -> >>> default(explanation);; >>> >>> let x = try computation() with >>> >>> MyException(e) -> default(e) >>> | MyOtherException() -> other_default() >>> | (e) -> fallback(e);; >>> >>> In terms of this PEP, these would be something like:: >>> >>> x = computation() except MyException => default() >>> x = try computation() except MyException e -> default() >>> x = (try computation() >>> except MyException as e -> default(e) >>> except MyOtherException -> other_default() >>> except BaseException as e -> fallback(e)) >>> >>> Many ML-inspired but not-directly-related languages from academia mix >>> things >>> up, usually using more keywords and fewer symbols. So, the `Oz`__ would >>> map >>> to Python as:: >>> __ http://mozart.github.io/mozart-v1/doc-1.4.0/tutorial/node5.html >>> >>> x = try computation() catch MyException as e then default(e) >>> >>> >>> Many Lisp-derived languages, like `Clojure,`__ implement try/catch as >>> special >>> forms (if you don't know what that means, think function-like macros), >>> so you >>> write, effectively:: >>> __ >>> http://clojure.org/special_forms#Special%20Forms--(try%20expr*%20catch-clause*%20finally-clause >>> ?) >>> >>> try(computation(), catch(MyException, explanation, >>> default(explanation))) >>> >>> try(computation(), >>> catch(MyException, explanation, default(explanation)), >>> catch(MyOtherException, explanation, other_default(explanation))) >>> >>> In Common Lisp, this is done with a slightly clunkier `"handler-case" >>> macro,`__ >>> but the basic idea is the same. >>> >>> __ http://clhs.lisp.se/Body/m_hand_1.htm >>> >>> >>> The Lisp style is, surprisingly, used by some languages that don't have >>> macros, like Lua, where `xpcall`__ takes functions. Writing lambdas >>> Python-style instead of Lua-style:: >>> __ http://www.gammon.com.au/scripts/doc.php?lua=xpcall >>> >>> x = xpcall(lambda: expression(), lambda e: default(e)) >>> >>> This actually returns (true, expression()) or (false, default(e)), but >>> I think we can ignore that part. >>> >>> >>> Haskell is actually similar to Lua here (except that it's all done >>> with monads, of course):: >>> >>> x = do catch(lambda: expression(), lambda e: default(e)) >>> >>> You can write a pattern matching expression within the function to decide >>> what to do with it; catching and re-raising exceptions you don't want is >>> cheap enough to be idiomatic. >>> >>> But Haskell infixing makes this nicer:: >>> >>> x = do expression() `catch` lambda: default() >>> x = do expression() `catch` lambda e: default(e) >>> >>> And that makes the parallel between the lambda colon and the except >>> colon in the proposal much more obvious:: >>> >>> >>> x = expression() except Exception: default() >>> x = expression() except Exception as e: default(e) >>> >>> >>> `Tcl`__ has the other half of Lua's xpcall; catch is a function which >>> returns >>> true if an exception was caught, false otherwise, and you get the value >>> out >>> in other ways. And it's all built around the the implicit quote-and-exec >>> that everything in Tcl is based on, making it even harder to describe in >>> Python terms than Lisp macros, but something like:: >>> __ http://wiki.tcl.tk/902 >>> >>> if {[ catch("computation()") "explanation"]} { default(explanation) } >>> >>> >>> `Smalltalk`__ is also somewhat hard to map to Python. The basic version >>> would be:: >>> __ http://smalltalk.gnu.org/wiki/exceptions >>> >>> x := computation() on:MyException do:default() >>> >>> ... but that's basically Smalltalk's passing-arguments-with-colons >>> syntax, not its exception-handling syntax. >>> >>> >>> Deferred sub-proposals >>> ====================== >>> >>> Multiple except clauses >>> ----------------------- >>> >>> An examination of use-cases shows that this is not needed as often as >>> it would be with the statement form, and as its syntax is a point on >>> which consensus has not been reached, the entire feature is deferred. >>> >>> In order to ensure compatibility with future versions, ensure that any >>> consecutive except operators are parenthesized to guarantee the >>> interpretation you expect. >>> >>> Multiple 'except' keywords can be used, and they will all catch >>> exceptions raised in the original expression (only):: >>> >>> # Will catch any of the listed exceptions thrown by expr; >>> # any exception thrown by a default expression will propagate. >>> value = (expr >>> except Exception1 [as e]: default1 >>> except Exception2 [as e]: default2 >>> # ... except ExceptionN [as e]: defaultN >>> ) >>> >>> Using parentheses to force an alternative interpretation works as >>> expected:: >>> >>> # Will catch an Exception2 thrown by either expr or default1 >>> value = ( >>> (expr except Exception1: default1) >>> except Exception2: default2 >>> ) >>> # Will catch an Exception2 thrown by default1 only >>> value = (expr except Exception1: >>> (default1 except Exception2: default2) >>> ) >>> >>> This last form is confusing and should be discouraged by PEP 8, but it >>> is syntactically legal: you can put any sort of expression inside a >>> ternary-except; ternary-except is an expression; therefore you can put >>> a ternary-except inside a ternary-except. >>> >>> Open question: Where there are multiple except clauses, should they be >>> separated by commas? It may be easier for the parser, that way:: >>> >>> value = (expr >>> except Exception1 [as e]: default1, >>> except Exception2 [as e]: default2, >>> # ... except ExceptionN [as e]: defaultN, >>> ) >>> >>> with an optional comma after the last, as per tuple rules. Downside: >>> Omitting the comma would be syntactically valid, and would have almost >>> identical semantics, but would nest the entire preceding expression in >>> its exception catching rig - a matching exception raised in the >>> default clause would be caught by the subsequent except clause. As >>> this difference is so subtle, it runs the risk of being a major bug >>> magnet. >>> >>> As a mitigation of this risk, this form:: >>> >>> value = expr except Exception1: default1 except Exception2: default2 >>> >>> could be syntactically forbidden, and parentheses required if the >>> programmer actually wants that behaviour:: >>> >>> value = (expr except Exception1: default1) except Exception2: >>> default2 >>> >>> This would prevent the accidental omission of a comma from changing >>> the expression's meaning. >>> >>> >>> Capturing the exception object >>> ------------------------------ >>> >>> In a try/except block, the use of 'as' to capture the exception object >>> creates a local name binding, and implicitly deletes that binding in a >>> finally clause. As 'finally' is not a part of this proposal (see >>> below), this makes it tricky to describe; also, this use of 'as' gives >>> a way to create a name binding in an expression context. Should the >>> default clause have an inner scope in which the name exists, shadowing >>> anything of the same name elsewhere? Should it behave the same way the >>> statement try/except does, and unbind the name? Should it bind the >>> name and leave it bound? (Almost certainly not; this behaviour was >>> changed in Python 3 for good reason.) >>> >>> Additionally, this syntax would allow a convenient way to capture >>> exceptions in interactive Python; returned values are captured by "_", >>> but exceptions currently are not. This could be spelled: >>> >>> >>> expr except Exception as e: e >>> >>> (The inner scope idea is tempting, but currently CPython handles list >>> comprehensions with a nested function call, as this is considered >>> easier. It may be of value to simplify both comprehensions and except >>> expressions, but that is a completely separate proposal to this PEP; >>> alternatively, it may be better to stick with what's known to >>> work. `Nick Coghlan elaborates.`__) >>> >>> __ >>> https://mail.python.org/pipermail/python-ideas/2014-February/025702.html >>> >>> An examination of the Python standard library shows that, while the use >>> of 'as' is fairly common (occurring in roughly one except clause in >>> five), >>> it is extremely *uncommon* in the cases which could logically be >>> converted >>> into the expression form. Its few uses can simply be left unchanged. >>> Consequently, in the interests of simplicity, the 'as' clause is not >>> included in this proposal. A subsequent Python version can add this >>> without >>> breaking any existing code, as 'as' is already a keyword. >>> >>> One example where this could possibly be useful is Lib/imaplib.py:568:: >>> try: typ, dat = self._simple_command('LOGOUT') >>> except: typ, dat = 'NO', ['%s: %s' % sys.exc_info()[:2]] >>> This could become:: >>> typ, dat = (self._simple_command('LOGOUT') >>> except BaseException as e: ('NO', '%s: %s' % (type(e), e))) >>> Or perhaps some other variation. This is hardly the most compelling >>> use-case, >>> but an intelligent look at this code could tidy it up significantly. In >>> the >>> absence of further examples showing any need of the exception object, I >>> have >>> opted to defer indefinitely the recommendation. >>> >>> >>> Rejected sub-proposals >>> ====================== >>> >>> finally clause >>> -------------- >>> The statement form try... finally or try... except... finally has no >>> logical corresponding expression form. Therefore the finally keyword >>> is not a part of this proposal, in any way. >>> >>> >>> Bare except having different meaning >>> ------------------------------------ >>> >>> With several of the proposed syntaxes, omitting the exception type name >>> would be easy and concise, and would be tempting. For convenience's sake, >>> it might be advantageous to have a bare 'except' clause mean something >>> more useful than "except BaseException". Proposals included having it >>> catch Exception, or some specific set of "common exceptions" (subclasses >>> of a new type called ExpressionError), or have it look for a tuple named >>> ExpressionError in the current scope, with a built-in default such as >>> (ValueError, UnicodeError, AttributeError, EOFError, IOError, OSError, >>> LookupError, NameError, ZeroDivisionError). All of these were rejected, >>> for severa reasons. >>> >>> * First and foremost, consistency with the statement form of try/except >>> would be broken. Just as a list comprehension or ternary if expression >>> can be explained by "breaking it out" into its vertical statement form, >>> an expression-except should be able to be explained by a relatively >>> mechanical translation into a near-equivalent statement. Any form of >>> syntax common to both should therefore have the same semantics in each, >>> and above all should not have the subtle difference of catching more in >>> one than the other, as it will tend to attract unnoticed bugs. >>> >>> * Secondly, the set of appropriate exceptions to catch would itself be >>> a huge point of contention. It would be impossible to predict exactly >>> which exceptions would "make sense" to be caught; why bless some of them >>> with convenient syntax and not others? >>> >>> * And finally (this partly because the recommendation was that a bare >>> except should be actively encouraged, once it was reduced to a >>> "reasonable" >>> set of exceptions), any situation where you catch an exception you don't >>> expect to catch is an unnecessary bug magnet. >>> >>> Consequently, the use of a bare 'except' is down to two possibilities: >>> either it is syntactically forbidden in the expression form, or it is >>> permitted with the exact same semantics as in the statement form (namely, >>> that it catch BaseException and be unable to capture it with 'as'). >>> >>> >>> Bare except clauses >>> ------------------- >>> >>> PEP 8 rightly advises against the use of a bare 'except'. While it is >>> syntactically legal in a statement, and for backward compatibility must >>> remain so, there is little value in encouraging its use. In an expression >>> except clause, "except:" is a SyntaxError; use the equivalent long-hand >>> form "except BaseException:" instead. A future version of Python MAY >>> choose >>> to reinstate this, which can be done without breaking compatibility. >>> >>> >>> Parentheses around the except clauses >>> ------------------------------------- >>> >>> Should it be legal to parenthesize the except clauses, separately from >>> the expression that could raise? Example:: >>> >>> value = expr ( >>> except Exception1 [as e]: default1 >>> except Exception2 [as e]: default2 >>> # ... except ExceptionN [as e]: defaultN >>> ) >>> >>> This is more compelling when one or both of the deferred sub-proposals >>> of multiple except clauses and/or exception capturing is included. In >>> their absence, the parentheses would be thus:: >>> value = expr except ExceptionType: default >>> value = expr (except ExceptionType: default) >>> >>> The advantage is minimal, and the potential to confuse a reader into >>> thinking the except clause is separate from the expression, or into >>> thinking >>> this is a function call, makes this non-compelling. The expression can, >>> of >>> course, be parenthesized if desired, as can the default:: >>> value = (expr) except ExceptionType: (default) >>> >>> >>> Short-hand for "except: pass" >>> ----------------------------- >>> >>> The following was been suggested as a similar >>> short-hand, though not technically an expression:: >>> >>> statement except Exception: pass >>> >>> try: >>> statement >>> except Exception: >>> pass >>> >>> For instance, a common use-case is attempting the removal of a file:: >>> os.unlink(some_file) except OSError: pass >>> >>> There is an equivalent already in Python 3.4, however, in contextlib:: >>> from contextlib import suppress >>> with suppress(OSError): os.unlink(some_file) >>> >>> As this is already a single line (or two with a break after the colon), >>> there is little need of new syntax and a confusion of statement vs >>> expression to achieve this. >>> >>> >>> Copyright >>> ========= >>> >>> This document has been placed in the public domain. >>> >>> >>> .. >>> Local Variables: >>> mode: indented-text >>> indent-tabs-mode: nil >>> sentence-end-double-space: t >>> fill-column: 70 >>> coding: utf-8 >>> End: >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com >>> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Feb 21 15:46:45 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Feb 2014 00:46:45 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 22 February 2014 00:37, Eli Bendersky wrote: > This goes against anything I understand about how exceptions should and > should not be used. I think you're thinking of a language that isn't Python - we use exceptions for control flow all over the place (it's how hasattr() is *defined*, for example). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Feb 21 15:50:02 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Feb 2014 00:50:02 +1000 Subject: [Python-Dev] Tangent on class level scoping rules (was Re: PEP 463: Exception-catching expressions) Message-ID: Sorry folks, that was meant to go to python-ideas, not python-dev. I've been so used to the PEP 463 threads being on python-ideas, I missed that this was the python-dev one :) Cheers, Nick. From eliben at gmail.com Fri Feb 21 15:49:55 2014 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 21 Feb 2014 06:49:55 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Fri, Feb 21, 2014 at 6:46 AM, Nick Coghlan wrote: > On 22 February 2014 00:37, Eli Bendersky wrote: > > This goes against anything I understand about how exceptions should and > > should not be used. > > I think you're thinking of a language that isn't Python - we use > exceptions for control flow all over the place (it's how hasattr() is > *defined*, for example). > No, it is Python I'm thinking about. As I mentioned in the reply to Brett's message, I see a difference between allowing exceptions on expression level and statement level. The latter is less likely to be abused. Once you add exceptions into expressions, all bets are off. For instance, it is sometime non-trivial to know which exceptions some function may throw. When you write a try...raise statement, you think hard about covering all the bases. In an expression you're unlikely to, which opens up a lot of possibilities for bugs. Again, please stop focusing just on the list[index] case -- the proposal adds a new significant feature to the language that can (and thus will) be used in a variety of scenarios. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Feb 21 15:50:26 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Feb 2014 15:50:26 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions References: Message-ID: <20140221155026.14d39fa0@fsol> On Sat, 22 Feb 2014 00:28:01 +1000 Nick Coghlan wrote: > > Neither of these objections addresses the problems with the status quo, though: > > - the status quo encourages overbroad exception handling (as > illustrated by examples in the PEP) I don't get this. Using the proper exception class in a "try...except" suite is no more bothersome than using the proper exception class in this newly-proposed construct. Regards Antoine. From ncoghlan at gmail.com Fri Feb 21 15:57:37 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Feb 2014 00:57:37 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 22 February 2014 00:44, Eli Bendersky wrote: > True, but at least you still have to explicitly try...except... which takes > a toll on the code so isn't taken lightly. Adding except into expressions, I > fear, will proliferate this usage much more. The same fears were raised regarding conditional expressions, and they'll be dealt with the same way: style guides and code review. That's also why the examples part of the PEP is so important, though: this is about providing a tool to *improve* readability in the cases where it applies, without significantly increasing the cognitive burden of the language. One example not mentioned in the PEP is that "text.find(substr)" would become just a shorthand for "text.index(substr) except ValueError: -1", but the latter has the benefit that you can substitute "None" instead, which avoids the trap where "-1" is a valid subscript for the string (but doesn't give you the answer you want). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Fri Feb 21 15:59:15 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 01:59:15 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Sat, Feb 22, 2014 at 12:53 AM, Nick Coghlan wrote: > On 21 February 2014 22:42, Chris Angelico wrote: >> People can already write: >> >> if (x if y else z): >> >> without the parens, and it works. Readability suffers when the same >> keyword is used twice (here "if" rather than the colon, but same >> difference), yet the parens are considered optional. Python is a >> language that, by and large, lacks syntactic salt; style guides are >> free to stipulate more, but the language doesn't make demands. I would >> strongly *recommend* using parens in all the cases you've shown, >> especially lambda: >> >>> lambda x: calculate(x) except Exception: None >>> lambda x: (calculate(x) except Exception: None) >> >> as it would otherwise depend on operator precedence; but mandating >> them feels to me like demanding readability. > > Right, that's why my main motivation for this suggestion is the one > relating to keeping future options open. If the parentheses are > optional, than adding multiple except clauses latter isn't possible, > since this would already be valid, but mean something different: > > expr except Exception1: default1 except Exception2: default2 > > The deferral currently has this snippet: > > """In order to ensure compatibility with future versions, ensure that > any consecutive except operators are parenthesized to guarantee the > interpretation you expect.""" > > That's not a reasonable expectation - either the parentheses have to > be mandatory as part of the deferral, or else multiple except clause > support needs to be listed as rejected rather than deferred. I've spent the better part of the last hour debating this in my head. It's basically a question of simplicity versus future flexibility: either keep the syntax clean and deny the multiple-except-clause option, or mandate the parens and permit it. The first option has, in my own head, the stronger case - this is designed for simplicity, and it wouldn't be that big a deal to completely reject multiple except clauses and simply require that the From rosuav at gmail.com Fri Feb 21 16:03:07 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 02:03:07 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Sat, Feb 22, 2014 at 1:59 AM, Chris Angelico wrote: > I've spent the better part of the last hour debating this in my head. > It's basically a question of simplicity versus future flexibility: > either keep the syntax clean and deny the multiple-except-clause > option, or mandate the parens and permit it. The first option has, in > my own head, the stronger case - this is designed for simplicity, and > it wouldn't be that big a deal to completely reject multiple except > clauses and simply require that the Oops, hit the wrong key and sent that half-written. ... and simply require that the statement form be used. But the whelming opinion of python-dev seems to be in favour of the parens anyway, and since they give us the possibility of future expansion effectively for free, I've gone that way. Parens are now required; the syntax is: value = (expr except Exception: default) and, as per genexp rules, redundant parens can be omitted: print(lst[i] except IndexError: "Out of bounds") ChrisA From ncoghlan at gmail.com Fri Feb 21 16:04:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Feb 2014 01:04:05 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 22 February 2014 00:59, Chris Angelico wrote: > On Sat, Feb 22, 2014 at 12:53 AM, Nick Coghlan wrote: >> The deferral currently has this snippet: >> >> """In order to ensure compatibility with future versions, ensure that >> any consecutive except operators are parenthesized to guarantee the >> interpretation you expect.""" >> >> That's not a reasonable expectation - either the parentheses have to >> be mandatory as part of the deferral, or else multiple except clause >> support needs to be listed as rejected rather than deferred. > > I've spent the better part of the last hour debating this in my head. > It's basically a question of simplicity versus future flexibility: > either keep the syntax clean and deny the multiple-except-clause > option, or mandate the parens and permit it. The first option has, in > my own head, the stronger case - this is designed for simplicity, and > it wouldn't be that big a deal to completely reject multiple except > clauses and simply require that the Yep, moving multiple exceptions to the "Rejected subproposals" section would work for me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Feb 21 16:02:41 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Feb 2014 01:02:41 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140221155026.14d39fa0@fsol> References: <20140221155026.14d39fa0@fsol> Message-ID: On 22 February 2014 00:50, Antoine Pitrou wrote: > On Sat, 22 Feb 2014 00:28:01 +1000 > Nick Coghlan wrote: >> >> Neither of these objections addresses the problems with the status quo, though: >> >> - the status quo encourages overbroad exception handling (as >> illustrated by examples in the PEP) > > I don't get this. Using the proper exception class in a "try...except" > suite is no more bothersome than using the proper exception class in > this newly-proposed construct. Not overbroad in the sense of catching too many different kinds of exception, overbroad in the sense of covering too much code in the try block. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Fri Feb 21 16:12:04 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 02:12:04 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140221155026.14d39fa0@fsol> References: <20140221155026.14d39fa0@fsol> Message-ID: On Sat, Feb 22, 2014 at 1:50 AM, Antoine Pitrou wrote: > On Sat, 22 Feb 2014 00:28:01 +1000 > Nick Coghlan wrote: >> >> Neither of these objections addresses the problems with the status quo, though: >> >> - the status quo encourages overbroad exception handling (as >> illustrated by examples in the PEP) > > I don't get this. Using the proper exception class in a "try...except" > suite is no more bothersome than using the proper exception class in > this newly-proposed construct. Overbroad exception handling comes in two ways. One is simply catching Exception or BaseException when a narrower class would be better, and that's not addressed by this PEP (except insofar as it does not have a bare "except:" syntax, and so it forces you to at least be explicit about catching BaseException). The other is this: try: f(x[i]) except IndexError: f(default) Translating that into this form: f(x[i] except IndexError: default) means that an IndexError thrown inside f() will not be caught. While it is, of course, possible to write that currently: try: arg = x[i] except IndexError: arg = default f(arg) it's that much less readable, and as evidenced by several examples in the standard library (some of which are in the PEP, and a few more can be found in my associated collection [1]), it's common to short-cut this. By providing a clean and convenient syntax for catching an exception in just one small expression, we encourage the narrowing of the catch front. ChrisA [1] https://github.com/Rosuav/ExceptExpr/blob/master/examples.py From brett at python.org Fri Feb 21 16:19:25 2014 From: brett at python.org (Brett Cannon) Date: Fri, 21 Feb 2014 10:19:25 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Fri, Feb 21, 2014 at 9:49 AM, Eli Bendersky wrote: > On Fri, Feb 21, 2014 at 6:46 AM, Nick Coghlan wrote: > >> On 22 February 2014 00:37, Eli Bendersky wrote: >> > This goes against anything I understand about how exceptions should and >> > should not be used. >> >> I think you're thinking of a language that isn't Python - we use >> exceptions for control flow all over the place (it's how hasattr() is >> *defined*, for example). >> > > No, it is Python I'm thinking about. As I mentioned in the reply to > Brett's message, I see a difference between allowing exceptions on > expression level and statement level. The latter is less likely to be > abused. Once you add exceptions into expressions, all bets are off. > > For instance, it is sometime non-trivial to know which exceptions some > function may throw. When you write a try...raise statement, you think hard > about covering all the bases. In an expression you're unlikely to, which > opens up a lot of possibilities for bugs. Again, please stop focusing just > on the list[index] case -- the proposal adds a new significant feature to > the language that can (and thus will) be used in a variety of scenarios. > I understand you are arguing that a try expression will lead to people just doing `something() except Exception: None` or whatever and that people will simply get lazy and not think about what they are doing with their exceptions. Unfortunately they already are; that shipped sailed when we didn't eliminate bare except clauses in Python 3 (hopefully we can change that in Python 4). I personally don't think that making people type out a try statement is going to lead to that much thought compared to an expression. I'm willing to bet most IDEs have a code snippet for creating a try statement so people are already not using the extra typing of a full-blown statement with at least two clauses as a way to stop and think about what they are doing. I'm totally fine restricting this proposal to not having any concept of exception catching or finally clause: it just replaces the simplest try/except clause possible (while requiring an exception be specified). That takes care of the common control flow use case of exceptions while requiring more thought for more complex cases. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 21 16:27:13 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 02:27:13 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Sat, Feb 22, 2014 at 1:03 AM, Eli Bendersky wrote: > Chris, while I also commend you for the comprehensive PEP, I'm -1 on the > proposal, for two main reasons: > > 1. Many proposals suggest new syntax to gain some succinctness. Each has to > be judged for its own merits, and in this case IMHO the cons eclipse the > pros. I don't think this will save a lot of code in a typical > well-structured program - maybe a few lines out of hundreds. On the other > hand, it adds yet another syntax to remember and understand, which is not > the Pythonic way. It doesn't need to save a huge number of lines. Just like lambda and the if/else expression, it's there as a tool - if it makes your code easier to read, it's a good too, and if it makes it harder, then don't use it. Yes, it's more to learn, but so is the proliferation of ad-hoc alternatives, several of which are listed in the 'Motivation' section at the top. > 2. Worse, this idea subverts exceptions to control flow, which is not only > un-Pythonic but also against the accepted practices of programming in > general. Here, the comparison to PEP 308 is misguided. PEP 308, whatever > syntax it adds, still remains within the domain of normal control flow. PEP > 463, OTOH, makes it deliberately easy to make exceptions part of > non-exceptional code, encouraging very bad programming practices. Again, I point to the current alternatives, including: * dict[key] -> dict.get(key, default) * min(sequence) -> min(sequence, default=default), as of 3.4 Both offer a way to either get an exception back or use a (usually constant) default value. In each case, the author of the class/function had to cater to the fact that some callers might want it to not raise an exception. The alternative is to always use the full try/except block, which leads to questions like "How can I keep going after an exception?" with code like this: try: spam = d["spam"] ham = d["ham"] eggs = d["eggs"] sausage = d["sausage"] except KeyError: thing_that_failed = "" The dict class offers a way to avoid the exception here, by showing that it's non-exceptional: spam = d.get("spam","") ham = d.get("ham","") eggs = d.get("eggs","") sausage = d.get("sausage","") But not everything does. Writing that with just exception handling looks like this: try: spam = d["spam"] except KeyError: span = "" try: ham = d["ham"] except KeyError: ham = "" try: eggs = d["eggs"] except KeyError: eggs = "" try: sausage = d["sausage"] except KeyError: sausage = "" with extreme likelihood of an error - do you see what I got wrong there? With inline exception handling, d could be a custom class that simply defines [...] to raise KeyError on unknowns, and the code can be written thus: spam = d["spam"] except KeyError: "" ham = d["ham"] except KeyError: "" eggs = d["eggs"] except KeyError: "" sausage = d["sausage"] except KeyError: "" It's still a bit repetitive, but that's hard to avoid. And it puts the exception handling at the exact point where it stops being exceptional and starts being normal - exactly as the try/except statement should. ChrisA From rosuav at gmail.com Fri Feb 21 16:41:21 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 02:41:21 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307612A.3050607@btinternet.com> References: <5307612A.3050607@btinternet.com> Message-ID: On Sat, Feb 22, 2014 at 1:22 AM, Rob Cliffe wrote: > Small point: in one of your examples you give a plug for the PEP "note the > DRY improvement". > I would suggest that similarly > perhaps in your Lib/tarfile.py:2198 example you point out the increase > in readability due to the 2 lines lining up > in your Lib/ipaddress.py:343 example you point out that the new form is > probably an improvement (i.e. probably closer to the author's intention) as > it will NOT catch an AttributeError evaluating "ips.append" (this would > matter e.g. if "append" were mis-spelt). YOU are clearly aware of this but > it would often escape the casual reader. Sure. Added a paragraph down the bottom of the section explaining the benefit of the narrowed scope. ChrisA From rosuav at gmail.com Fri Feb 21 16:52:59 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 02:52:59 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> Message-ID: On Sat, Feb 22, 2014 at 1:34 AM, Brett Cannon wrote: > While I like the general concept, I agree that it looks too much like a > crunched statement; the use of the colon is a non-starter for me. I'm sure > I'm not the only one whose brain has been trained to view a colon in Python > to mean "statement", period. This goes against that syntactic practice and > just doesn't work for me. > > I'm -1 with the current syntax, but it can go into the + range if a better > syntax can be chosen. We bikeshedded that extensively on -ideas. The four best options are: value = (expr except Exception: default) value = (expr except Exception -> default) value = (expr except Exception pass default) value = (expr except Exception then default) Note that the last option involves the creation of a new keyword. Would any of the others feel better to you? ChrisA From rosuav at gmail.com Fri Feb 21 17:13:58 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 03:13:58 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Sat, Feb 22, 2014 at 2:19 AM, Brett Cannon wrote: > I understand you are arguing that a try expression will lead to people just > doing `something() except Exception: None` or whatever and that people will > simply get lazy and not think about what they are doing with their > exceptions. Unfortunately they already are; that shipped sailed when we > didn't eliminate bare except clauses in Python 3 (hopefully we can change > that in Python 4). That's already come up a few times. I wrote a script to try to find potential conversion targets, and technically it's correct to show this up: Doc/tools/rstlint.py:73: try: compile(code, fn, 'exec') except SyntaxError as err: yield err.lineno, 'not compilable: %s' % err Yes, technically you could treat that as an expression. Well, okay. Current version of the proposal doesn't include 'as', so you can't do that. Let's grab the next one. Lib/asyncio/events.py:40: try: self._callback(*self._args) except Exception: logger.exception('Exception in callback %s %r', self._callback, self._args) That could, per PEP 463, become: (self._callback(*self._args) except Exception: logger.exception('Exception in callback %s %r', self._callback, self._args)) But that would be abuse of the syntax. It's just like converting this: for s in list_of_strings: print(s) into an expression: [print(s) for s in list_of_strings] list(map(print,list_of_strings)) Neither of them is a good way to write code, yet that's not the fault of either map() or the list comp. > I'm totally fine restricting this proposal to not having any concept of > exception catching or finally clause: it just replaces the simplest > try/except clause possible (while requiring an exception be specified). That > takes care of the common control flow use case of exceptions while requiring > more thought for more complex cases. The finally clause was never in the proposal. I added it as an open issue early on, just to see if it gathered any interest, but the weight of opinion agreed with my initial feeling: it's utterly useless to the exception syntax. Same goes for an else clause. Currently, the two complexities not possible in the expression form are capturing the exception with 'as' (for technical reasons) and stringing multiple except clauses on (for complexity reasons). In the standard library, counting up all the cases that could plausibly be converted to expression form, very very few use either feature. So this is a case of "do just the really simple option and catch 98.5% of the actual usage". ChrisA From solipsis at pitrou.net Fri Feb 21 17:18:30 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Feb 2014 17:18:30 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions References: <20140221155026.14d39fa0@fsol> Message-ID: <20140221171830.6a729839@fsol> On Sat, 22 Feb 2014 02:12:04 +1100 Chris Angelico wrote: > > Overbroad exception handling comes in two ways. One is simply catching > Exception or BaseException when a narrower class would be better, and > that's not addressed by this PEP (except insofar as it does not have a > bare "except:" syntax, and so it forces you to at least be explicit > about catching BaseException). The other is this: > > try: > f(x[i]) > except IndexError: > f(default) > > Translating that into this form: > > f(x[i] except IndexError: default) > > means that an IndexError thrown inside f() will not be caught. Thank you and Nick for the explanation. This is a good point. I'm still put off by the syntax, though, just like Brett. Regards Antoine. From ericsnowcurrently at gmail.com Fri Feb 21 17:40:45 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 21 Feb 2014 09:40:45 -0700 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: [re: "as" clause] On Fri, Feb 21, 2014 at 6:20 AM, Nick Coghlan wrote: > It's probably OK to leave it in the deferred section and just note the > difficulty of implementing it in a backwards compatible way, since > we're *not* going to be introducing a closure. Agreed. -eric From ericsnowcurrently at gmail.com Fri Feb 21 17:46:39 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 21 Feb 2014 09:46:39 -0700 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221155026.14d39fa0@fsol> Message-ID: On Fri, Feb 21, 2014 at 8:12 AM, Chris Angelico wrote: > On Sat, Feb 22, 2014 at 1:50 AM, Antoine Pitrou wrote: >> On Sat, 22 Feb 2014 00:28:01 +1000 >> Nick Coghlan wrote: >>> >>> Neither of these objections addresses the problems with the status quo, though: >>> >>> - the status quo encourages overbroad exception handling (as >>> illustrated by examples in the PEP) >> >> I don't get this. Using the proper exception class in a "try...except" >> suite is no more bothersome than using the proper exception class in >> this newly-proposed construct. > > Overbroad exception handling comes in two ways. One is simply catching > Exception or BaseException when a narrower class would be better, and > that's not addressed by this PEP (except insofar as it does not have a > bare "except:" syntax, and so it forces you to at least be explicit > about catching BaseException). The other is this: > > try: > f(x[i]) > except IndexError: > f(default) > > Translating that into this form: > > f(x[i] except IndexError: default) > > means that an IndexError thrown inside f() will not be caught. While > it is, of course, possible to write that currently: > > try: > arg = x[i] > except IndexError: > arg = default > f(arg) > > it's that much less readable, and as evidenced by several examples in > the standard library (some of which are in the PEP, and a few more can > be found in my associated collection [1]), it's common to short-cut > this. By providing a clean and convenient syntax for catching an > exception in just one small expression, we encourage the narrowing of > the catch front. Be sure to capture in the PEP (within reason) a summary of concerns and rebuttals/acquiescence. Eli's, Brett's, and Antoine's concerns likely reflect what others are thinking as well. The PEP and its result will be better for recording such matters. In fact, that's a key goal of PEPs. :) -eric From ericsnowcurrently at gmail.com Fri Feb 21 17:52:53 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 21 Feb 2014 09:52:53 -0700 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Fri, Feb 21, 2014 at 8:19 AM, Brett Cannon wrote: > I understand you are arguing that a try expression will lead to people just > doing `something() except Exception: None` or whatever and that people will > simply get lazy and not think about what they are doing with their > exceptions. Exactly. > Unfortunately they already are; that shipped sailed when we > didn't eliminate bare except clauses in Python 3 (hopefully we can change > that in Python 4). We should have a page somewhere tracking a py4k "wishlist". :) > > I personally don't think that making people type out a try statement is > going to lead to that much thought compared to an expression. I'm willing to > bet most IDEs have a code snippet for creating a try statement so people are > already not using the extra typing of a full-blown statement with at least > two clauses as a way to stop and think about what they are doing. > > I'm totally fine restricting this proposal to not having any concept of > exception catching or finally clause: it just replaces the simplest > try/except clause possible (while requiring an exception be specified). That > takes care of the common control flow use case of exceptions while requiring > more thought for more complex cases. +1 -eric From ericsnowcurrently at gmail.com Fri Feb 21 17:57:16 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 21 Feb 2014 09:57:16 -0700 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Fri, Feb 21, 2014 at 8:27 AM, Chris Angelico wrote: > It doesn't need to save a huge number of lines. Just like lambda and > the if/else expression, it's there as a tool - if it makes your code > easier to read, it's a good too, and if it makes it harder, then don't > use it. And as long as the tool/syntax is obviously linked to an existing one (and does not change semantics), people really won't need to learn anything new. So it wouldn't be adding much to the keep-it-in-my-head size of the language. -eric From status at bugs.python.org Fri Feb 21 18:07:54 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 21 Feb 2014 18:07:54 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140221170754.8AEDF56915@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-02-14 - 2014-02-21) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4558 (+34) closed 27937 (+62) total 32495 (+96) Open issues with patches: 2059 Issues opened (60) ================== #10510: distutils upload/register should use CRLF in HTTP requests http://bugs.python.org/issue10510 reopened by eric.araujo #20628: csv.DictReader http://bugs.python.org/issue20628 opened by rogererens #20629: Python ctypes BigEndianStructure bitfield assignment misbehavi http://bugs.python.org/issue20629 opened by Alan.Ning #20630: Add sorting helpers for collections containing None values http://bugs.python.org/issue20630 opened by ncoghlan #20631: python readline module crashing on NULL access http://bugs.python.org/issue20631 opened by tfiala #20632: Define a new __key__ protocol http://bugs.python.org/issue20632 opened by ncoghlan #20633: SystemError: Parent module 'multiprocessing' not loaded, canno http://bugs.python.org/issue20633 opened by Amr.Ali #20635: Fix the grid geometry manager and add tests for geometry manag http://bugs.python.org/issue20635 opened by serhiy.storchaka #20636: Better repr for tkinter widgets http://bugs.python.org/issue20636 opened by serhiy.storchaka #20637: Support key-sharing dictionaries in subclasses http://bugs.python.org/issue20637 opened by pingebretson #20639: pathlib.PurePath.with_suffix() does not allow removing the suf http://bugs.python.org/issue20639 opened by july #20640: Adds idle test for configHelpSourceEdit http://bugs.python.org/issue20640 opened by sahutd #20641: Python installer needs elevated rights to install pip http://bugs.python.org/issue20641 opened by poke #20642: Enhance deepcopy-ing for tuples http://bugs.python.org/issue20642 opened by Claudiu.Popa #20643: Strange dot in documentation (after generator.close) http://bugs.python.org/issue20643 opened by rutsky #20644: OS X installer build broken by changes to documentation build http://bugs.python.org/issue20644 opened by ned.deily #20646: 3.4 cherry-pick: 180e4b678003 select and kqueue round the time http://bugs.python.org/issue20646 opened by haypo #20648: 3.4 cherry-pick: multiple changesets for asyncio http://bugs.python.org/issue20648 opened by haypo #20650: asyncio.BaseEventLoop.run_in_executor docs have awkward wordin http://bugs.python.org/issue20650 opened by brett.cannon #20653: Pickle enums by name http://bugs.python.org/issue20653 opened by serhiy.storchaka #20657: OpenBSD: Merge patches http://bugs.python.org/issue20657 opened by haypo #20658: os.environ.clear() fails with empty keys (posix.unsetenv) http://bugs.python.org/issue20658 opened by blueyed #20659: Want to make a class method a property by combining decorators http://bugs.python.org/issue20659 opened by the.mulhern #20660: Starting a second multiprocessing.Manager causes INCREF on all http://bugs.python.org/issue20660 opened by mythsmith #20662: Pydoc doesn't escape parameter defaults in html http://bugs.python.org/issue20662 opened by serhiy.storchaka #20663: Introduce exception argument to iter http://bugs.python.org/issue20663 opened by cool-RR #20664: _findLib_crle and _get_soname broken on latest SunOS 5.11 http://bugs.python.org/issue20664 opened by rygorde4 #20668: Remove dependency on tests.txt when running test_asyncio suite http://bugs.python.org/issue20668 opened by vajrasky #20669: OpenBSD: socket.recvmsg tests fail with OSError: [Errno 40] Me http://bugs.python.org/issue20669 opened by haypo #20671: test_create_at_shutdown_with_encoding() of test_io hangs on "S http://bugs.python.org/issue20671 opened by haypo #20674: Update comments in dictobject.c http://bugs.python.org/issue20674 opened by rhettinger #20677: Minor typo in enum docs http://bugs.python.org/issue20677 opened by flashk #20678: re does not allow back references in {} matching operator http://bugs.python.org/issue20678 opened by hardkrash #20679: 3.4 cherry-pick: 587fd4b91120 improve Enum subclass behavior http://bugs.python.org/issue20679 opened by ethan.furman #20680: Pickle socket enums by names http://bugs.python.org/issue20680 opened by serhiy.storchaka #20683: [doc] Ease comprehension of section 9.2 of docs for Python 2 a http://bugs.python.org/issue20683 opened by krichter #20686: Confusing statement http://bugs.python.org/issue20686 opened by Daniel.U..Thibault #20687: Change in expectedFailure breaks testtools http://bugs.python.org/issue20687 opened by barry #20689: socket.AddressFamily is absent in html pydoc http://bugs.python.org/issue20689 opened by serhiy.storchaka #20690: IDLE Indents convert to spaces and then throws error http://bugs.python.org/issue20690 opened by Justin.Barker #20691: inspect.signature: Consider exposing 'follow_wrapper_chains' o http://bugs.python.org/issue20691 opened by yselivanov #20692: Tutorial section 9.4 and FAQ: how to call a method on an int http://bugs.python.org/issue20692 opened by Jon.Shemitz #20693: Sidebar scrolls down 2x as fast as page content http://bugs.python.org/issue20693 opened by nicktimko #20699: Behavior of ZipFile with file-like object and BufferedWriter. http://bugs.python.org/issue20699 opened by Henning.von.Bargen #20701: warning in compileall.rst http://bugs.python.org/issue20701 opened by pitrou #20702: warning in cmdline.rst http://bugs.python.org/issue20702 opened by pitrou #20703: RuntimeError caused by lazy imports in pdb http://bugs.python.org/issue20703 opened by xdegaye #20705: distutils.extension.Extension with empty 'sources' list http://bugs.python.org/issue20705 opened by remram #20708: commands has no "RANDOM" environment? http://bugs.python.org/issue20708 opened by addict2tux #20709: os.utime(path_to_directory): wrong documentation for Windows. http://bugs.python.org/issue20709 opened by jgehrcke #20712: Make inspect agnostic about whether functions are implemented http://bugs.python.org/issue20712 opened by larry #20714: Allow for ]]> in CDATA in minidom http://bugs.python.org/issue20714 opened by arturcz #20715: 3.4 cherry-pick: 2000c27ebe80 inspect: Fix getfullargspec to s http://bugs.python.org/issue20715 opened by yselivanov #20717: test_4_daemon_threads() fails randomly on "x86 Windows Server http://bugs.python.org/issue20717 opened by haypo #20718: OpenBSD/AIX: tests passing a file descriptor with sendmsg/recv http://bugs.python.org/issue20718 opened by haypo #20719: test_robotparser failure on "SPARC Solaris 10 (cc%2C 64b) [SB] http://bugs.python.org/issue20719 opened by haypo #20720: test_create_server() of test_asyncio failure on "x86 Windows S http://bugs.python.org/issue20720 opened by haypo #20721: 3.4 cherry-pick: 005d0678f93c Update pip to 1.5.4 http://bugs.python.org/issue20721 opened by dstufft #20722: newline is (partially) independent of universal newlines; need http://bugs.python.org/issue20722 opened by Devin Jeanpierre #20723: Make test (Python 3.3.4) http://bugs.python.org/issue20723 opened by datienzalopez Most recent 15 issues with no replies (15) ========================================== #20721: 3.4 cherry-pick: 005d0678f93c Update pip to 1.5.4 http://bugs.python.org/issue20721 #20720: test_create_server() of test_asyncio failure on "x86 Windows S http://bugs.python.org/issue20720 #20715: 3.4 cherry-pick: 2000c27ebe80 inspect: Fix getfullargspec to s http://bugs.python.org/issue20715 #20709: os.utime(path_to_directory): wrong documentation for Windows. http://bugs.python.org/issue20709 #20705: distutils.extension.Extension with empty 'sources' list http://bugs.python.org/issue20705 #20703: RuntimeError caused by lazy imports in pdb http://bugs.python.org/issue20703 #20702: warning in cmdline.rst http://bugs.python.org/issue20702 #20701: warning in compileall.rst http://bugs.python.org/issue20701 #20699: Behavior of ZipFile with file-like object and BufferedWriter. http://bugs.python.org/issue20699 #20691: inspect.signature: Consider exposing 'follow_wrapper_chains' o http://bugs.python.org/issue20691 #20690: IDLE Indents convert to spaces and then throws error http://bugs.python.org/issue20690 #20689: socket.AddressFamily is absent in html pydoc http://bugs.python.org/issue20689 #20683: [doc] Ease comprehension of section 9.2 of docs for Python 2 a http://bugs.python.org/issue20683 #20680: Pickle socket enums by names http://bugs.python.org/issue20680 #20674: Update comments in dictobject.c http://bugs.python.org/issue20674 Most recent 15 issues waiting for review (15) ============================================= #20714: Allow for ]]> in CDATA in minidom http://bugs.python.org/issue20714 #20705: distutils.extension.Extension with empty 'sources' list http://bugs.python.org/issue20705 #20680: Pickle socket enums by names http://bugs.python.org/issue20680 #20677: Minor typo in enum docs http://bugs.python.org/issue20677 #20669: OpenBSD: socket.recvmsg tests fail with OSError: [Errno 40] Me http://bugs.python.org/issue20669 #20668: Remove dependency on tests.txt when running test_asyncio suite http://bugs.python.org/issue20668 #20658: os.environ.clear() fails with empty keys (posix.unsetenv) http://bugs.python.org/issue20658 #20653: Pickle enums by name http://bugs.python.org/issue20653 #20642: Enhance deepcopy-ing for tuples http://bugs.python.org/issue20642 #20640: Adds idle test for configHelpSourceEdit http://bugs.python.org/issue20640 #20639: pathlib.PurePath.with_suffix() does not allow removing the suf http://bugs.python.org/issue20639 #20637: Support key-sharing dictionaries in subclasses http://bugs.python.org/issue20637 #20636: Better repr for tkinter widgets http://bugs.python.org/issue20636 #20635: Fix the grid geometry manager and add tests for geometry manag http://bugs.python.org/issue20635 #20631: python readline module crashing on NULL access http://bugs.python.org/issue20631 Top 10 most discussed issues (10) ================================= #20648: 3.4 cherry-pick: multiple changesets for asyncio http://bugs.python.org/issue20648 50 msgs #20653: Pickle enums by name http://bugs.python.org/issue20653 27 msgs #20628: csv.DictReader http://bugs.python.org/issue20628 17 msgs #19748: test_time failures on AIX http://bugs.python.org/issue19748 12 msgs #20658: os.environ.clear() fails with empty keys (posix.unsetenv) http://bugs.python.org/issue20658 11 msgs #20679: 3.4 cherry-pick: 587fd4b91120 improve Enum subclass behavior http://bugs.python.org/issue20679 10 msgs #19081: zipimport behaves badly when the zip file changes while the pr http://bugs.python.org/issue19081 9 msgs #20608: 'SyntaxError: invalid token' is unfriendly http://bugs.python.org/issue20608 9 msgs #20570: Bundle pip 1.5.3 in Python 3.4rc2 http://bugs.python.org/issue20570 7 msgs #20671: test_create_at_shutdown_with_encoding() of test_io hangs on "S http://bugs.python.org/issue20671 7 msgs Issues closed (61) ================== #9009: Improve quality of Python/dtoa.c http://bugs.python.org/issue9009 closed by mark.dickinson #12211: Better document math.copysign behavior. http://bugs.python.org/issue12211 closed by akuchling #13663: pootle.python.org is outdated. http://bugs.python.org/issue13663 closed by benjamin.peterson #13720: argparse print_help() fails if COLUMNS is set to a low value http://bugs.python.org/issue13720 closed by serhiy.storchaka #16224: tokenize.untokenize() misbehaves when moved to "compatiblity m http://bugs.python.org/issue16224 closed by terry.reedy #19769: test_venv: test_with_pip() failure on "AMD64 Windows Server 20 http://bugs.python.org/issue19769 closed by haypo #19890: Typo in multiprocessing docs http://bugs.python.org/issue19890 closed by ezio.melotti #20080: Unused variable in Lib/sqlite3/test/factory.py http://bugs.python.org/issue20080 closed by python-dev #20221: #define hypot _hypot conflicts with existing definition http://bugs.python.org/issue20221 closed by zach.ware #20237: Ambiguous sentence in document of xml package. http://bugs.python.org/issue20237 closed by akuchling #20241: Bad reference to RFC in document of ipaddress? http://bugs.python.org/issue20241 closed by berker.peksag #20497: Unclear word in socket document. http://bugs.python.org/issue20497 closed by python-dev #20507: TypeError from str.join has no message http://bugs.python.org/issue20507 closed by python-dev #20510: Test cases in test_sys don't match the comments http://bugs.python.org/issue20510 closed by zach.ware #20511: asyncio: StreamReader should use bytearray for its internal bu http://bugs.python.org/issue20511 closed by haypo #20564: locks cannot be interrupted on OpenBSD http://bugs.python.org/issue20564 closed by haypo #20569: IDLE : Add clipboard history feature http://bugs.python.org/issue20569 closed by sahutd #20573: "built-in repr()" function link on the repr module documentati http://bugs.python.org/issue20573 closed by python-dev #20609: Always running kill_python breaks building x64 on 32-bit Windo http://bugs.python.org/issue20609 closed by zach.ware #20610: Documentation with two slashes in URL links to v2.6.5c2 http://bugs.python.org/issue20610 closed by benjamin.peterson #20616: Add tracemalloc.Traceback.format() method http://bugs.python.org/issue20616 closed by haypo #20621: Issue with zipimport in 3.3.4 and 3.4.0rc1 http://bugs.python.org/issue20621 closed by gregory.p.smith #20622: Python3.3 venv pip fails to run if path contains spaces http://bugs.python.org/issue20622 closed by eric.araujo #20625: Argument names in __annotations__ are not mangled for function http://bugs.python.org/issue20625 closed by yselivanov #20634: IDLE test : Typo in readme file http://bugs.python.org/issue20634 closed by ezio.melotti #20638: KeyError building the Python html doc with sphinx http://bugs.python.org/issue20638 closed by georg.brandl #20645: 3.4 cherry-pick: 1166b3321012 Revert changes of issue #19466 ( http://bugs.python.org/issue20645 closed by larry #20647: 3.4 cherry-pick: d50a95fab002 add tracemalloc.Traceback.format http://bugs.python.org/issue20647 closed by larry #20649: Minor grammatical mistake for asyncio dev docs http://bugs.python.org/issue20649 closed by python-dev #20651: 3.4 cherry-pick: 52ab9e1ff46a rolls back bad zipimport changes http://bugs.python.org/issue20651 closed by larry #20652: Example in asyncio task gives resource warning http://bugs.python.org/issue20652 closed by python-dev #20654: Pydoc (and help) fails with socket.AddressFamily http://bugs.python.org/issue20654 closed by serhiy.storchaka #20655: test_subprocess is not executed in python -m test.test_asyncio http://bugs.python.org/issue20655 closed by haypo #20656: OpenBSD: timeval.tv_sec type is long, not time_t http://bugs.python.org/issue20656 closed by python-dev #20661: 3.4 cherry-pick: eef7899ea7ab use system doc toolchain instead http://bugs.python.org/issue20661 closed by georg.brandl #20665: 3.4 cherry-pick: 400a8e4599d9 rename posix_close to avoid name http://bugs.python.org/issue20665 closed by larry #20666: asyncio/OpenBSD: make concurrent.futures dependency optional http://bugs.python.org/issue20666 closed by haypo #20667: asyncio: KqueueEventLoopTests.test_read_pty_output() of test_a http://bugs.python.org/issue20667 closed by haypo #20670: 3.4 cherry-pick: 2dd4922c9371 set line and column numbers for http://bugs.python.org/issue20670 closed by larry #20672: test_list() of test_tarfile fails on http://bugs.python.org/issue20672 closed by serhiy.storchaka #20673: asyncio: Add support for UNIX Domain Sockets. http://bugs.python.org/issue20673 closed by yselivanov #20675: 3.4 cherry-pick: 69d13cdc4eeb fix 64-bit build on 32-bit Windo http://bugs.python.org/issue20675 closed by larry #20676: 3.4 cherry-pick: a63327162063 Mangle __parameters in __annotat http://bugs.python.org/issue20676 closed by larry #20681: asyncio: New error handling API http://bugs.python.org/issue20681 closed by yselivanov #20682: test_create_ssl_unix_connection() of test_asyncio failed on "x http://bugs.python.org/issue20682 closed by python-dev #20684: inspect.getfullargspec (etc) incorrectly follows __wrapped__ c http://bugs.python.org/issue20684 closed by yselivanov #20685: 3.4 cherry pick: 9f76adbac8b7 better handling of missing SSL/T http://bugs.python.org/issue20685 closed by larry #20688: 3.4 cherry-pick: 65fb95578f94 inspect: Fix getfullargspec() to http://bugs.python.org/issue20688 closed by larry #20694: asyncio.docs: Document subprocess_exec and subprocess_shell http://bugs.python.org/issue20694 closed by yselivanov #20695: test_urllibnet.urlretrieveNetworkTests fails due to new python http://bugs.python.org/issue20695 closed by benjamin.peterson #20696: asyncio.docs: Document StreamReader an other streams module AP http://bugs.python.org/issue20696 closed by yselivanov #20697: 3.4 cherry-pick: 0399e842073a http://bugs.python.org/issue20697 closed by larry #20698: 3.4 cherry-pick: b328f8ccbccf pickle shouldn't look up dunder http://bugs.python.org/issue20698 closed by larry #20700: Docs sidebar scrolls strangely in 3.4 docs http://bugs.python.org/issue20700 closed by zach.ware #20704: asyncio: Implement debug API http://bugs.python.org/issue20704 closed by yselivanov #20706: asyncio.docs: Fix wording in 'this method returns a coroutine http://bugs.python.org/issue20706 closed by yselivanov #20707: 3.4 cherry-pick: bf413a97f1a9 fix conflicting hypot definition http://bugs.python.org/issue20707 closed by larry #20710: Make pydoc consistent about bound methods http://bugs.python.org/issue20710 closed by larry #20711: inspect.getfullargspec does not correctly work with builtin mo http://bugs.python.org/issue20711 closed by yselivanov #20713: 3.4 cherry-pick: d57df3f72715 Upgrade pip to 1.5.3 http://bugs.python.org/issue20713 closed by larry #20716: 3.4 cherry-pick: 5e73bb72662e pydoc shouldn't display "self" f http://bugs.python.org/issue20716 closed by larry From rosuav at gmail.com Fri Feb 21 18:18:54 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 04:18:54 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221155026.14d39fa0@fsol> Message-ID: On Sat, Feb 22, 2014 at 3:46 AM, Eric Snow wrote: > Be sure to capture in the PEP (within reason) a summary of concerns > and rebuttals/acquiescence. Eli's, Brett's, and Antoine's concerns > likely reflect what others are thinking as well. The PEP and its > result will be better for recording such matters. In fact, that's a > key goal of PEPs. :) I now have a paragraph summarizing the "narrowing of scope" benefit, though not worded as a rebuttal of a concern. Eli's concern is mainly that this can be abused, which everything can; should I include something about that? Brett's issue is with the colon. I'll write up a paragraph or two of reasoning as to why that was selected, possibly tomorrow, but the main reason is simply that nobody came up with a compelling argument in favour of any other. Hardly the most epic tale :) ChrisA From solipsis at pitrou.net Fri Feb 21 18:32:50 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 21 Feb 2014 18:32:50 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions References: <20140221145205.0b8b670d@fsol> Message-ID: <20140221183250.2b44f860@fsol> On Sat, 22 Feb 2014 02:52:59 +1100 Chris Angelico wrote: > On Sat, Feb 22, 2014 at 1:34 AM, Brett Cannon wrote: > > While I like the general concept, I agree that it looks too much like a > > crunched statement; the use of the colon is a non-starter for me. I'm sure > > I'm not the only one whose brain has been trained to view a colon in Python > > to mean "statement", period. This goes against that syntactic practice and > > just doesn't work for me. > > > > I'm -1 with the current syntax, but it can go into the + range if a better > > syntax can be chosen. > > We bikeshedded that extensively on -ideas. The four best options are: > > value = (expr except Exception: default) -0.5 > value = (expr except Exception -> default) -0.5 > value = (expr except Exception pass default) -1 (looks terribly weird) > value = (expr except Exception then default) +0.5 But I'm aware it requires reserving "then" as a keyword, which might need a prior SyntaxWarning. Regards Antoine. From rosuav at gmail.com Fri Feb 21 18:36:25 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 04:36:25 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140221183250.2b44f860@fsol> References: <20140221145205.0b8b670d@fsol> <20140221183250.2b44f860@fsol> Message-ID: On Sat, Feb 22, 2014 at 4:32 AM, Antoine Pitrou wrote: >> value = (expr except Exception then default) > > +0.5 > But I'm aware it requires reserving "then" as a keyword, which might > need a prior SyntaxWarning. There are no instances of "then" used as a name in the Python stdlib, I already checked. If anyone has a good-sized codebase to check, you're welcome to use the same analysis script - it's an AST parser/walker, so it finds actual use as a name, not in a comment or a string or anything. https://github.com/Rosuav/ExceptExpr/blob/master/find_except_expr.py ChrisA From guido at python.org Fri Feb 21 18:37:29 2014 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Feb 2014 09:37:29 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140221183250.2b44f860@fsol> References: <20140221145205.0b8b670d@fsol> <20140221183250.2b44f860@fsol> Message-ID: On Fri, Feb 21, 2014 at 9:32 AM, Antoine Pitrou wrote: > On Sat, 22 Feb 2014 02:52:59 +1100 > Chris Angelico wrote: > > > On Sat, Feb 22, 2014 at 1:34 AM, Brett Cannon wrote: > > > While I like the general concept, I agree that it looks too much like a > > > crunched statement; the use of the colon is a non-starter for me. I'm > sure > > > I'm not the only one whose brain has been trained to view a colon in > Python > > > to mean "statement", period. This goes against that syntactic practice > and > > > just doesn't work for me. > > > > > > I'm -1 with the current syntax, but it can go into the + range if a > better > > > syntax can be chosen. > > > > We bikeshedded that extensively on -ideas. The four best options are: > > > > value = (expr except Exception: default) > > -0.5 > > > value = (expr except Exception -> default) > > -0.5 > > > value = (expr except Exception pass default) > > -1 (looks terribly weird) > > > value = (expr except Exception then default) > > +0.5 > But I'm aware it requires reserving "then" as a keyword, which might > need a prior SyntaxWarning. > I'm put off by the ':' syntax myself (it looks to me as if someone forgot a newline somewhere) but 'then' feels even weirder (it's been hard-coded in my brain as meaning the first branch of an 'if'). I am going to sleep on this. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Feb 21 19:19:19 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Feb 2014 10:19:19 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <530798A7.7040007@stoneleaf.us> On 02/20/2014 07:15 PM, Chris Angelico wrote: > PEP: 463 > Title: Exception-catching expressions > > [snip] > > Deferred sub-proposals > ====================== > > Multiple except clauses > ----------------------- > > An examination of use-cases shows that this is not needed as often as > it would be with the statement form, and as its syntax is a point on > which consensus has not been reached, the entire feature is deferred. > > In order to ensure compatibility with future versions, ensure that any > consecutive except operators are parenthesized to guarantee the > interpretation you expect. > > Multiple 'except' keywords can be used, and they will all catch > exceptions raised in the original expression (only):: > > # Will catch any of the listed exceptions thrown by expr; > # any exception thrown by a default expression will propagate. > value = (expr > except Exception1 [as e]: default1 > except Exception2 [as e]: default2 > # ... except ExceptionN [as e]: defaultN > ) -1 If one wants to catch multiple exceptions from the same original operation, use the statement block. Keep the exception expression syntax simple and chained, akin to the if..else syntax: value = expr1 except exc1 expr2 except exc2 expr3 ... So each following except is only catching exceptions from the last expression (in other words, exc1 only from expr1, exc2 only from expr2, etc.) Keep it simple. If one wants the power of the statement form, use the statement syntax. -- ~Ethan~ From yselivanov.ml at gmail.com Fri Feb 21 20:04:45 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 21 Feb 2014 14:04:45 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <5307A34D.6050101@gmail.com> Thank you for writing this PEP, Chris. I'm impressed by the quality of this PEP, and how you handled the discussion on python-ideas. I initially liked this idea, however, after reading the PEP in detail, my vote is: -1 on the current syntax; -1 on the whole idea. On 2/20/2014, 10:15 PM, Chris Angelico wrote: > [snip] > * dict.get(key, default) - second positional argument in place of > KeyError > > * next(iter, default) - second positional argument in place of > StopIteration > > * list.pop() - no way to return a default We can fix that in 3.5. > * seq[index] - no way to handle a bounds error We can add 'list.get(index, default)' method, similar to 'Mapping.get'. It's far more easier than introducing new syntax. > * min(sequence, default=default) - keyword argument in place of > ValueError > > * sum(sequence, start=default) - slightly different but can do the > same job 'sum' is not a very frequently used builtin. I think the Motivation section is pretty weak. Inconvenience of dict[] raising KeyError was solved by introducing the dict.get() method. And I think that dct.get('a', 'b') is 1000 times better than dct['a'] except KeyError: 'b' I don't want to see this (or any other syntax) used by anyone. I also searched how many 'except IndexError' are in the standard library code. Around 60. That's a rather low number, that can justify adding 'list.get' but not advocate a new syntax. Moreover, I think that explicit handling of IndexError is rather ugly and error prone, using len() is usually reads better. > [snip] > > Consider this example of a two-level cache:: > for key in sequence: > x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key))) > # do something with x I'm sorry, it took me a minute to understand what your example is doing. I would rather see two try..except blocks than this. > > Retrieve an argument, defaulting to None:: > cond = args[1] except IndexError: None > > # Lib/pdb.py:803: > try: > cond = args[1] > except IndexError: > cond = None cond = None if (len(args) < 2) else args[1] > Attempt a translation, falling back on the original:: > e.widget = self._nametowidget(W) except KeyError: W > > # Lib/tkinter/__init__.py:1222: > try: > e.widget = self._nametowidget(W) > except KeyError: > e.widget = W I'm not sure this is a good example either. I presume '_nametowidget' is some function, that might raise a KeyError because of a bug in its implementation, or to signify that there is no widget 'W'. Your new syntax just helps to work with this error prone api. > > Read from an iterator, continuing with blank lines once it's > exhausted:: > line = readline() except StopIteration: '' > > # Lib/lib2to3/pgen2/tokenize.py:370: > try: > line = readline() > except StopIteration: > line = '' Handling StopIteration exception is more common in standard library than IndexError (although not much more), but again, not all of that code is suitable for your syntax. I'd say about 30%, which is about 20-30 spots (correct me if I'm wrong). > > Retrieve platform-specific information (note the DRY improvement); > this particular example could be taken further, turning a series of > separate assignments into a single large dict initialization:: > # sys.abiflags may not be defined on all platforms. > _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: '' > > # Lib/sysconfig.py:529: > try: > _CONFIG_VARS['abiflags'] = sys.abiflags > except AttributeError: > # sys.abiflags may not be defined on all platforms. > _CONFIG_VARS['abiflags'] = '' Ugly. _CONFIG_VARS['abiflags'] = getattr(sys, 'abiflags', '') Much more readable. > Retrieve an indexed item, defaulting to None (similar to dict.get):: > def getNamedItem(self, name): > return self._attrs[name] except KeyError: None > > # Lib/xml/dom/minidom.py:573: > def getNamedItem(self, name): > try: > return self._attrs[name] > except KeyError: > return None _attrs there is a dict (or at least it's something that quaks like a dict, and has [] and keys()), so return self._attrs.get(name) > Translate numbers to names, falling back on the numbers:: > g = grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid > u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: tarinfo.uid > > # Lib/tarfile.py:2198: > try: > g = grp.getgrnam(tarinfo.gname)[2] > except KeyError: > g = tarinfo.gid > try: > u = pwd.getpwnam(tarinfo.uname)[2] > except KeyError: > u = tarinfo.uid This one is a valid example, but totally unparseable by humans. Moreover, it promotes a bad pattern, as you mask KeyErrors in 'grp.getgrnam(tarinfo.gname)' call. > > Perform some lengthy calculations in EAFP mode, handling division by > zero as a sort of sticky NaN:: > > value = calculate(x) except ZeroDivisionError: float("nan") > > try: > value = calculate(x) > except ZeroDivisionError: > value = float("nan") > > Calculate the mean of a series of numbers, falling back on zero:: > > value = statistics.mean(lst) except statistics.StatisticsError: 0 > > try: > value = statistics.mean(lst) > except statistics.StatisticsError: > value = 0 I think all of the above more readable with try statement. > > Retrieving a message from either a cache or the internet, with auth > check:: > > logging.info("Message shown to user: %s",((cache[k] > except LookupError: > (backend.read(k) except OSError: 'Resource not available') > ) > if check_permission(k) else 'Access denied' > ) except BaseException: "This is like a bare except clause") > > try: > if check_permission(k): > try: > _ = cache[k] > except LookupError: > try: > _ = backend.read(k) > except OSError: > _ = 'Resource not available' > else: > _ = 'Access denied' > except BaseException: > _ = "This is like a bare except clause" > logging.info("Message shown to user: %s", _) If you replace '_' with a 'msg' (why did you use '_'??) then try statements are *much* more readable. > [snip] > > Lib/ipaddress.py:343:: > try: > ips.append(ip.ip) > except AttributeError: > ips.append(ip.network_address) > Becomes:: > ips.append(ip.ip except AttributeError: ip.network_address) or it may become: ips.append(getattr(ip, 'ip', ip.network_address)) or address = getattr(ip, 'ip', ip.network_address) ips.append(address) --- All in all, your proposal scares me. I doesn't make python code readable, it doesn't solve the problem of overbroad exceptions handling (you have couple examples of overbroad handling in your PEP examples section). Yes, some examples look neat. But your syntax is much easier to abuse, than 'if..else' expression, and if people start abusing it, Python will simply loose it's readability advantage. Yury From timothy.c.delaney at gmail.com Fri Feb 21 21:33:20 2014 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Sat, 22 Feb 2014 07:33:20 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On 22 February 2014 02:03, Chris Angelico wrote: > Oops, hit the wrong key and sent that half-written. > > ... and simply require that the statement form be used. But the > whelming opinion of python-dev seems to be in favour of the parens > anyway, and since they give us the possibility of future expansion > effectively for free, I've gone that way. Parens are now required; the > syntax is: > > value = (expr except Exception: default) > Let me add my congratulations on a fine PEP. I think it's much more readable with the parens (doesn't look as much like there's a missing newline). I'd also strongly argue for permanently disallowing multiple exceptions - as you said, this is intended to be a simple, readable syntax. Even with the parens, I share the bad gut feeling others are having with the colon in the syntax. I also don't think "then" is a good fit (even if it didn't add a keyword). Unfortunately, I can't come up with anything better ... Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Feb 21 22:55:54 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 21 Feb 2014 14:55:54 -0700 Subject: [Python-Dev] Tangent on class level scoping rules (was Re: PEP 463: Exception-catching expressions) In-Reply-To: References: Message-ID: On Fri, Feb 21, 2014 at 7:42 AM, Nick Coghlan wrote: > It's a relatively arcane scoping rule that only matters if you have > non-trivial logic at class scope. The vast majority of Python > programmers will never have to care, because they do the typical thing > and their class bodies consist almost entirely of function definitions > and relatively simple assignment statements. That is definitely an esoteric corner. It's only really bitten me when I was doing stuff with nested classes [1] and mined too deeply. Here's a simple example: class Spam: class Ham: A = None B = None class Eggs: class Bacon(Ham): A = 3 If I recall correctly, Larry Hastings ran into something similar a while back. -eric [1] The nested classes were used just for easy-to-read namespaces, effectively hijacking the class definition syntax with no intention of actually using the class as a type. From greg.ewing at canterbury.ac.nz Fri Feb 21 23:06:22 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Feb 2014 11:06:22 +1300 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <5307CDDE.8090106@canterbury.ac.nz> Nick Coghlan wrote: > On 21 February 2014 13:15, Chris Angelico wrote: > >>Generator expressions require parentheses, unless they would be >>strictly redundant. Ambiguities with except expressions could be >>resolved in the same way, forcing nested except-in-except trees to be >>correctly parenthesized > > I'd like to make the case that the PEP should adopt this as its > default position. I generally agree, but I'd like to point out that this doesn't necessarily mean making the parenthesizing rules as strict as they are for generator expressions. The starting point for genexps is that the parens are part of the syntax, the same way that square brackets are part of the syntax of a list comprehension; we only allow them to be omitted in very special circumstances. On the other hand, I don't think there's any harm in allowing an except expression to stand on its own when there is no risk of ambiguity, e.g. foo = things[i] except IndexError: None should be allowed, just as we allow x = a if b else c and don't require x = (a if b else c) -- Greg From eric at trueblade.com Fri Feb 21 23:26:54 2014 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 21 Feb 2014 17:26:54 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307CDDE.8090106@canterbury.ac.nz> References: <5307CDDE.8090106@canterbury.ac.nz> Message-ID: <5307D2AE.8030608@trueblade.com> On 2/21/2014 5:06 PM, Greg Ewing wrote: > Nick Coghlan wrote: >> On 21 February 2014 13:15, Chris Angelico wrote: >> >>> Generator expressions require parentheses, unless they would be >>> strictly redundant. Ambiguities with except expressions could be >>> resolved in the same way, forcing nested except-in-except trees to be >>> correctly parenthesized >> >> I'd like to make the case that the PEP should adopt this as its >> default position. > > I generally agree, but I'd like to point out that this > doesn't necessarily mean making the parenthesizing rules as > strict as they are for generator expressions. > > The starting point for genexps is that the parens are part of > the syntax, the same way that square brackets are part of > the syntax of a list comprehension; we only allow them to > be omitted in very special circumstances. > > On the other hand, I don't think there's any harm in allowing > an except expression to stand on its own when there is no > risk of ambiguity, e.g. > > foo = things[i] except IndexError: None I agree that it would be a shame to disallow this simple usage. I'd like to leave this like any other expression: add parens if they add clarity or if they are required for the precedence you'd prefer. Personally, I'd probably always use parens unless it was a simple assignment (as above), or an argument to a function call. But I think it's a style issue. Eric. > should be allowed, just as we allow > > x = a if b else c > > and don't require > > x = (a if b else c) > From greg.ewing at canterbury.ac.nz Sat Feb 22 00:29:59 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Feb 2014 12:29:59 +1300 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140221145205.0b8b670d@fsol> References: <20140221145205.0b8b670d@fsol> Message-ID: <5307E177.3080602@canterbury.ac.nz> Antoine Pitrou wrote: >> lst = [1, 2] >> value = lst[2] except IndexError: "No value" > > the gain in concision is counterbalanced by a loss in > readability, This version might be more readable: value = lst[2] except "No value" if IndexError since it puts the normal and exceptional values next to each other, and relegates the exception type (which is of much less interest) to a trailing aside. -- Greg From ethan at stoneleaf.us Sat Feb 22 00:36:02 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Feb 2014 15:36:02 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307D2AE.8030608@trueblade.com> References: <5307CDDE.8090106@canterbury.ac.nz> <5307D2AE.8030608@trueblade.com> Message-ID: <5307E2E2.3070007@stoneleaf.us> On 02/21/2014 02:26 PM, Eric V. Smith wrote: > On 2/21/2014 5:06 PM, Greg Ewing wrote: >>> On 21 February 2014 13:15, Chris Angelico wrote: >>> >>>> Generator expressions require parentheses, unless they would be >>>> strictly redundant. Ambiguities with except expressions could be >>>> resolved in the same way, forcing nested except-in-except trees to be >>>> correctly parenthesized There would be no ambiguity if only nested excepts are allowed. If one wants to catch multiple exceptions from one expression, /and do something different for each one/, use the statement form as it's going to be clearer. For example: try: value = 1/x except ZeroDivisionError: try: value = 1/default['denominator'] except KeyError: value = NaN is much cleaner as: value = 1/x except ZeroDivisionError: 1/default['denominator'] except KeyError: NaN However, this: try: result = Parse(some_stuff) except MissingOperator: result = ... except InvalidOperand: result = ... except SomethingElse: result = ... would not benefit from being condensed into a single expression -- ~Ethan~ From greg.ewing at canterbury.ac.nz Sat Feb 22 00:57:28 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Feb 2014 12:57:28 +1300 Subject: [Python-Dev] Tangent on class level scoping rules (was Re: PEP 463: Exception-catching expressions) In-Reply-To: References: Message-ID: <5307E7E8.6060001@canterbury.ac.nz> Nick Coghlan wrote: > As Chris later noted, you likely *could* still implement expression > local name binding for an except expression without a full closure, it > would just be rather difficult. I'm still not convinced it would be all *that* difficult. Seems to me it would be semantically equivalent to renaming the inner variable and adding a finally clause to unbind it. Is there something I'm missing? -- Greg From greg.ewing at canterbury.ac.nz Sat Feb 22 01:15:13 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Feb 2014 13:15:13 +1300 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <5307EC11.90407@canterbury.ac.nz> Eli Bendersky wrote: > For instance, it is sometime non-trivial to know which exceptions some > function may throw. When you write a try...raise statement, you think > hard about covering all the bases. In an expression you're unlikely to, Speak for yourself. I don't think I would put any less thought into which exception I caught with an except expression as I would for an except statement. In fact, an except expression may even make it easier to catch exceptions in an appropriately targeted way. For example, a pattern frequently encountered is: result = computation(int(arg)) and you want to guard against arg not being a well-formed int. It's tempting to do this: try: result = computation(int(arg)) except ValueError: abort("Invalid int") But that's bad, because the try clause encompasses too much. Doing it properly requires splitting up the expression: try: i = int(arg) except: abort("Invalid int") else: result = computation(i) With an except expression, it could be written: result = computation(int(arg) except ValueError: abort("Invalid int")) -- Greg From ethan at stoneleaf.us Sat Feb 22 00:38:01 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Feb 2014 15:38:01 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307E177.3080602@canterbury.ac.nz> References: <20140221145205.0b8b670d@fsol> <5307E177.3080602@canterbury.ac.nz> Message-ID: <5307E359.6080603@stoneleaf.us> On 02/21/2014 03:29 PM, Greg Ewing wrote: > Antoine Pitrou wrote: > >>> lst = [1, 2] >>> value = lst[2] except IndexError: "No value" >> >> the gain in concision is counterbalanced by a loss in >> readability, > > This version might be more readable: > > value = lst[2] except "No value" if IndexError It does read nicely, and is fine for the single, non-nested, case (which is probably the vast majority), but how would it handle nested exceptions? -- ~Ethan~ From timothy.c.delaney at gmail.com Sat Feb 22 01:56:08 2014 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Sat, 22 Feb 2014 11:56:08 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307E177.3080602@canterbury.ac.nz> References: <20140221145205.0b8b670d@fsol> <5307E177.3080602@canterbury.ac.nz> Message-ID: On 22 February 2014 10:29, Greg Ewing wrote: > Antoine Pitrou wrote: > > lst = [1, 2] >>> value = lst[2] except IndexError: "No value" >>> >> >> the gain in concision is counterbalanced by a loss in >> readability, >> > > This version might be more readable: > > value = lst[2] except "No value" if IndexError > +1 - it's readable, clear, and only uses existing keywords. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From zuo at chopin.edu.pl Sat Feb 22 02:06:08 2014 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Sat, 22 Feb 2014 02:06:08 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <20140221183250.2b44f860@fsol> Message-ID: <52fb1d300a923efdc57920d4ef8f3a35@chopin.edu.pl> 21.02.2014 18:37, Guido van Rossum wrote: > I'm put off by the ':' syntax myself (it looks to me as if someone > forgot a newline somewhere) As I mentioned at python-ideas I believe that parens neutralize, at least to some extent, that unfortunate statement-ish flavor of the colon. This one has some statement-like smell: msg = seq[i] except IndexError: "nothing" But this looks better, I believe: msg = (seq[i] except IndexError: "nothing") Or even (still being my favorite): msg = seq[i] except (IndexError: "nothing") Cheers. *j From greg.ewing at canterbury.ac.nz Sat Feb 22 02:12:51 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Feb 2014 14:12:51 +1300 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <5307F993.7000106@canterbury.ac.nz> Chris Angelico wrote: > it wouldn't be that big a deal to completely reject multiple except > clauses and simply require that the (rest of original post truncated) Oh, no! The PSU has gotten wind of this proposal and doesn't like it! -- Greg From v+python at g.nevcal.com Sat Feb 22 02:10:50 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 21 Feb 2014 17:10:50 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307E177.3080602@canterbury.ac.nz> References: <20140221145205.0b8b670d@fsol> <5307E177.3080602@canterbury.ac.nz> Message-ID: <5307F91A.6030304@g.nevcal.com> On 2/21/2014 3:29 PM, Greg Ewing wrote: > Antoine Pitrou wrote: > >>> lst = [1, 2] >>> value = lst[2] except IndexError: "No value" >> >> the gain in concision is counterbalanced by a loss in >> readability, > > This version might be more readable: > > value = lst[2] except "No value" if IndexError > > since it puts the normal and exceptional values next > to each other, and relegates the exception type (which > is of much less interest) to a trailing aside. Ternary if teaches us that the normal and alternate values should be on either end, so the present proposal corresponds to that; ternary if also teaches us that nesting affects the right value, in the absence of parentheses, this proposal seems to correspond, although requiring parentheses or not is presently a topic of discussion in a different subthread. On the other hand, "if" and "except" are opposites in another way: "if" is followed by an expression that, if true, means the left value is used, and if false, means the right value is used. "Except" is followed by an exception that, if thrown, means the right value is used, and if not thrown, means the left value is used... but the word "except" is opposite in meaning to "if", so this actually fits nicely with English definitions. Here's a challenge: There has been a big thread about None versus (SQL) Null. Show how an except: expression can help the DB API more easily convert from using None to using a new Null singleton, and you'll have a winner :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Feb 22 01:42:36 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Feb 2014 16:42:36 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307A34D.6050101@gmail.com> References: <5307A34D.6050101@gmail.com> Message-ID: <5307F27C.5010207@stoneleaf.us> On 02/21/2014 11:04 AM, Yury Selivanov wrote: > On 2/20/2014, 10:15 PM, Chris Angelico wrote: >> >> * list.pop() - no way to return a default > > We can fix that in 3.5. How many are you going to "fix"? How are you going to "fix" the routines you don't control? >> * seq[index] - no way to handle a bounds error > > We can add 'list.get(index, default)' method, similar to > 'Mapping.get'. It's far more easier than introducing new > syntax. When I have to keep writing the same code over and over and aver again, I find a better way to do the job. In this case, an exception expression does quite nicely. > I also searched how many 'except IndexError' are in > the standard library code. Around 60. That's a rather > low number, that can justify adding 'list.get' but not > advocate a new syntax. And roughly 200 of KeyError, another couple hundred of ValueError... This is not just about better handling of [missing] default values, but of better exception handling. This PEP adds the ability to use a scalpel instead of a sledge hammer. -- ~Ethan~ From steve at pearwood.info Sat Feb 22 02:21:31 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 22 Feb 2014 12:21:31 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307EC11.90407@canterbury.ac.nz> References: <5307EC11.90407@canterbury.ac.nz> Message-ID: <20140222012131.GA3684@ando> On Sat, Feb 22, 2014 at 01:15:13PM +1300, Greg Ewing wrote: > Eli Bendersky wrote: > >For instance, it is sometime non-trivial to know which exceptions some > >function may throw. When you write a try...raise statement, you think > >hard about covering all the bases. In an expression you're unlikely to, > > Speak for yourself. I don't think I would put any less > thought into which exception I caught with an except > expression as I would for an except statement. +1 With the possibly exception of messing about in the interactive interpreter, "catch as much as possible" is an anti-pattern. If you're not trying to catch as little as possible, you're doing it wrong. > In fact, an except expression may even make it easier > to catch exceptions in an appropriately targeted way. > For example, a pattern frequently encountered is: > > result = computation(int(arg)) > > and you want to guard against arg not being a > well-formed int. It's tempting to do this: > > try: > result = computation(int(arg)) > except ValueError: > abort("Invalid int") > > But that's bad, because the try clause encompasses > too much. Doing it properly requires splitting up the > expression: > > try: > i = int(arg) > except: > abort("Invalid int") > else: > result = computation(i) > > With an except expression, it could be written: > > result = computation(int(arg) > except ValueError: abort("Invalid int")) Nice example! Except I'd lay the code out a bit better to emphasise which part is being guarded: result = computation( int(arg) except ValueError: abort("Invalid int") ) Actually, not quite so nice as I first thought, since you're relying on the side-effects of abort() rather than returning a value. But apart from that quibble, the ability to guard *small* subcalculations without needing a visually heavy try...block is a good use-case. -- Steven From greg.ewing at canterbury.ac.nz Sat Feb 22 02:41:20 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Feb 2014 14:41:20 +1300 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307E359.6080603@stoneleaf.us> References: <20140221145205.0b8b670d@fsol> <5307E177.3080602@canterbury.ac.nz> <5307E359.6080603@stoneleaf.us> Message-ID: <53080040.8060803@canterbury.ac.nz> Ethan Furman wrote: > On 02/21/2014 03:29 PM, Greg Ewing wrote: > >> value = lst[2] except "No value" if IndexError > > It does read nicely, and is fine for the single, non-nested, case (which > is probably the vast majority), but how would it handle nested exceptions? Hmmm, probably not very well, unless we define a except b if E1 except c if E2 to mean a except (b except c if E2) if E1 If E1 == E2, that could perhaps be abbreviated to a except b except c if E Or we could just decide that the nested case is going to be so rare it's not worth worrying about. -- Greg From yselivanov.ml at gmail.com Sat Feb 22 02:48:01 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 21 Feb 2014 20:48:01 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307F27C.5010207@stoneleaf.us> References: <5307A34D.6050101@gmail.com> <5307F27C.5010207@stoneleaf.us> Message-ID: <530801D1.1000000@gmail.com> On 2/21/2014, 7:42 PM, Ethan Furman wrote: > On 02/21/2014 11:04 AM, Yury Selivanov wrote: >> On 2/20/2014, 10:15 PM, Chris Angelico wrote: >>> >>> * list.pop() - no way to return a default >> >> We can fix that in 3.5. > > How many are you going to "fix"? How are you going to "fix" the > routines you don't control? This new syntax won't magically fix all the code either. But it may let people write code like this: # I'm sorry, I really can't read this. logging.info("Message shown to user: %s",((cache[k] except LookupError: (backend.read(k) except OSError: 'Resource not available') ) if check_permission(k) else 'Access denied' ) except BaseException: "This is like a bare except clause") or this: # We happily mask exceptions from getgrnam g = grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid And this particular case or list.pop method, let's be honest, can be fixed ;) > >>> * seq[index] - no way to handle a bounds error >> >> We can add 'list.get(index, default)' method, similar to >> 'Mapping.get'. It's far more easier than introducing new >> syntax. > > When I have to keep writing the same code over and over and aver > again, I find a better way to do the job. In this case, an exception > expression does quite nicely. I can't believe you find list[42] except IndexError: 'spam' to be better than list.get(42, 'spam') If IndexError is such a big deal, I think we can add this function to list and collections.abc.Sequence. Authors of libraries will follow. > >> I also searched how many 'except IndexError' are in >> the standard library code. Around 60. That's a rather >> low number, that can justify adding 'list.get' but not >> advocate a new syntax. > > And roughly 200 of KeyError, another couple hundred of ValueError... Many KeyErrors can be fixed with a proper use of '.get()', or 'in' operator. Many AttributeErrors can be fixed with use of getattr() or hasattr(), or by designing a better API. Many ValueErrors... wait, there are no ValueError examples in the PEP, so I won't comment. > > This is not just about better handling of [missing] default values, > but of better exception handling. This PEP adds the ability to use a > scalpel instead of a sledge hammer. I'd say it's another sledge-hammer, but a tad smaller and a cuter one (to some people). But it's still exception handling, still more code than a function call. Yury From ethan at stoneleaf.us Sat Feb 22 02:28:30 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Feb 2014 17:28:30 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222012131.GA3684@ando> References: <5307EC11.90407@canterbury.ac.nz> <20140222012131.GA3684@ando> Message-ID: <5307FD3E.7010402@stoneleaf.us> On 02/21/2014 05:21 PM, Steven D'Aprano wrote: > On Sat, Feb 22, 2014 at 01:15:13PM +1300, Greg Ewing wrote: >> >> With an except expression, it could be written: >> >> result = computation(int(arg) >> except ValueError: abort("Invalid int")) > > Nice example! Except I'd lay the code out a bit better to emphasise > which part is being guarded: > > result = computation( > int(arg) except ValueError: abort("Invalid int") > ) > > Actually, not quite so nice as I first thought, since you're relying on > the side-effects of abort() rather than returning a value. Think of it rather as transforming the exception to be more useful. -- ~Ethan~ From steve at pearwood.info Sat Feb 22 02:52:04 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 22 Feb 2014 12:52:04 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307F27C.5010207@stoneleaf.us> References: <5307A34D.6050101@gmail.com> <5307F27C.5010207@stoneleaf.us> Message-ID: <20140222015204.GD3684@ando> On Fri, Feb 21, 2014 at 04:42:36PM -0800, Ethan Furman wrote: > This is not just about better handling of [missing] default values, but of > better exception handling. This PEP adds the ability to use a scalpel > instead of a sledge hammer. Beautifully said! -- Steven From greg.ewing at canterbury.ac.nz Sat Feb 22 02:55:16 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Feb 2014 14:55:16 +1300 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222012131.GA3684@ando> References: <5307EC11.90407@canterbury.ac.nz> <20140222012131.GA3684@ando> Message-ID: <53080384.3080808@canterbury.ac.nz> Steven D'Aprano wrote: > result = computation( > int(arg) except ValueError: abort("Invalid int") > ) > > Actually, not quite so nice as I first thought, since you're relying on > the side-effects of abort() rather than returning a value. Yeah, while I was writing that I wondered whether you should be allowed to write int(arg) except ValueError: raise UserError("Invalid int") That looks heretical, because 'raise' can't in any way be interpreted as a value-returning expression. But you can achieve the same result using a function that always raises and exception, so forbidding it on those grounds would be pointless. And writing it that way at least makes it obvious that it *does* always raise an exception, in the same way that try: i = int(arg) except ValueError: raise UserError("Invalid int") else: result = computation(i) makes it obvious that control can't fall off the end of the except branch. -- Greg From v+python at g.nevcal.com Sat Feb 22 03:08:22 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 21 Feb 2014 18:08:22 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <52fb1d300a923efdc57920d4ef8f3a35@chopin.edu.pl> References: <20140221145205.0b8b670d@fsol> <20140221183250.2b44f860@fsol> <52fb1d300a923efdc57920d4ef8f3a35@chopin.edu.pl> Message-ID: <53080696.3050000@g.nevcal.com> On 2/21/2014 5:06 PM, Jan Kaliszewski wrote: > Or even (still being my favorite): > > msg = seq[i] except (IndexError: "nothing") This syntax actually has a benefit: the parenthesized syntax after except could become a list, to allow handling different exceptions from the tried expression with different results: msg = seq[dictionary[i]] except (IndexError: "nothing", KeyError: "serious problems") And still allows nesting: msg = seq[i] except (IndexError: dictionary[i] except (KeyError: "no fallback data for %s" % i)) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Sat Feb 22 03:08:26 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 21 Feb 2014 19:08:26 -0700 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <52fb1d300a923efdc57920d4ef8f3a35@chopin.edu.pl> References: <20140221145205.0b8b670d@fsol> <20140221183250.2b44f860@fsol> <52fb1d300a923efdc57920d4ef8f3a35@chopin.edu.pl> Message-ID: On Fri, Feb 21, 2014 at 6:06 PM, Jan Kaliszewski wrote: > 21.02.2014 18:37, Guido van Rossum wrote: > >> I'm put off by the ':' syntax myself (it looks to me as if someone >> forgot a newline somewhere) > > > As I mentioned at python-ideas I believe that parens neutralize, > at least to some extent, that unfortunate statement-ish flavor > of the colon. > > This one has some statement-like smell: > > msg = seq[i] except IndexError: "nothing" > > But this looks better, I believe: > > msg = (seq[i] except IndexError: "nothing") Agreed. The same holds true with {'a': 5} and function annotations (e.g. "def f(a: 5)"). > Or even (still being my favorite): > > msg = seq[i] except (IndexError: "nothing") +1 -eric From navneet35371 at gmail.com Sat Feb 22 03:00:48 2014 From: navneet35371 at gmail.com (NAVNEET SUMAN) Date: Sat, 22 Feb 2014 07:30:48 +0530 Subject: [Python-Dev] GSOC 2014 Message-ID: Hi, This is my first year in gsoc. I have been working with python and django from quite a time. One of the Gsoc proposed ideas drew my attention and i would surely like to work on that. I would like to work for creating a extension for idle to integrate PEP8, what are the prerequisites . I have been studying the PEP8 code past a week. This might be useful for a quick start to this project. Please guide. -- Thanks and Regards, Navneet suman, IIIT-Allahabad +91-9305612151 -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Sat Feb 22 03:07:17 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 22 Feb 2014 03:07:17 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: Hi, 2014-02-21 4:15 GMT+01:00 Chris Angelico : > PEP: 463 > Title: Exception-catching expressions Nice PEP. Good luck, it's really hard to modify the language. Be prepared to get many alternatives, criticisms, and suggestions. Good luck to handle them :-) Here is mine. I like the simple case "expr1 except Exception: expr2", but it is used twice like "expr1 except Exception: (expr2 except Exception: expr3)". > * list.pop() - no way to return a default I never used this once, I didn't know that it exists :) > * min(sequence, default=default) - keyword argument in place of > ValueError > * sum(sequence, start=default) - slightly different but can do the > same job You should remove the sum() from your list, the start parameter is unrelated. It is used to control input and output types for example. It's not related to an exception. > The proposal adds this:: > > lst = [1, 2] > value = lst[2] except IndexError: "No value" The PEP looks nice on such simple example... > Consider this example of a two-level cache:: > for key in sequence: > x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key))) > # do something with x ... but I don't like when it is used to build complex expressions. At the first read, I'm unable to understand this long expression. At the second read, I'm still unable to see which instruction will be executed first: lvl1[key] or lvl2[key]? The advantage of the current syntax is that the control flow is obvious, from the top to the bottom: # start try: x = lvl1[key] # first instruction except KeyError: try: x = lvl2[key] except KeyError: x = f(key) # latest instruction # end Thanks to Python indentation, it's easier to see the control flow. After having read a lot of code last 10 years, I now try to follow this rule: one instruction per line. For example, I don't like "if ((res = func()) == NULL)" in the C language, I always split it into two lines. A drawback of writing more than one instruction is that it's hard to debug instruction per instruction (in pdb/gdb). Think for example of "if ((res = func()) == NULL)": if you execute step by step, how do you know which instruction is currently executed? You should maybe write your example differently: x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key))) It looks like the classic try/except syntax. > # sys.abiflags may not be defined on all platforms. > _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: '' > > # Lib/sysconfig.py:529: > try: > _CONFIG_VARS['abiflags'] = sys.abiflags > except AttributeError: > # sys.abiflags may not be defined on all platforms. > _CONFIG_VARS['abiflags'] = '' getattr(sys, 'abiflags', '') can be used here. > Retrieve an indexed item, defaulting to None (similar to dict.get):: > def getNamedItem(self, name): > return self._attrs[name] except KeyError: None > > # Lib/xml/dom/minidom.py:573: > def getNamedItem(self, name): > try: > return self._attrs[name] > except KeyError: > return None Hum, I don't understand this one. name is an index or a key? If it's a key and self._attrs is a dict, you can use self._attrs.get(name). > Translate numbers to names, falling back on the numbers:: > g = grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid > u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: tarinfo.uid > > # Lib/tarfile.py:2198: > try: > g = grp.getgrnam(tarinfo.gname)[2] > except KeyError: > g = tarinfo.gid > try: > u = pwd.getpwnam(tarinfo.uname)[2] > except KeyError: > u = tarinfo.uid Note (for myself): the KeyError comes from the function call, not from entry[2]. > Retrieving a message from either a cache or the internet, with auth > check:: > > logging.info("Message shown to user: %s",((cache[k] > except LookupError: > (backend.read(k) except OSError: 'Resource not available') > ) > if check_permission(k) else 'Access denied' > ) except BaseException: "This is like a bare except clause") Oh, again: I prefer the try/except syntax (above), it's more easy to see the control flow and understand the code. Too many instruction per lines, the code is too "dense". It looks like cache[k] is the first executed instruction, but it's wrong: check_permission(k) is executed before. So I prefer this version: > try: > if check_permission(k): > try: > _ = cache[k] > except LookupError: > try: > _ = backend.read(k) > except OSError: > _ = 'Resource not available' > else: > _ = 'Access denied' > except BaseException: > _ = "This is like a bare except clause" > logging.info("Message shown to user: %s", _) (...) > Looking up objects in a sparse list of overrides:: > > (overrides[x] or default except IndexError: default).ping() > > try: > (overrides[x] or default).ping() > except IndexError: > default.ping() The try/except version should use a variable and call the ping() method outside the try/except block, to ensure that the IndexError comes from overrides[x], not from the ping() method. Victor From ericsnowcurrently at gmail.com Sat Feb 22 03:28:01 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 21 Feb 2014 19:28:01 -0700 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <530801D1.1000000@gmail.com> References: <5307A34D.6050101@gmail.com> <5307F27C.5010207@stoneleaf.us> <530801D1.1000000@gmail.com> Message-ID: On Fri, Feb 21, 2014 at 6:48 PM, Yury Selivanov wrote: > This new syntax won't magically fix all the code either. > But it may let people write code like this: > > # I'm sorry, I really can't read this. > > logging.info("Message shown to user: %s",((cache[k] > except LookupError: > (backend.read(k) except OSError: 'Resource not available') > ) > if check_permission(k) else 'Access denied' > ) except BaseException: "This is like a bare except clause") You could argue the same thing about people chaining ternary conditional expressions (because it would be about as hard to read). Ditto for lambda. These expression forms (and expressions in general) are useful for brief snippets of code. If someone wants to write expressions that span multiple lines like that, then that's their problem. I don't see how the PEP 463 syntax will make it more likely that someone will abuse expression syntax like that. > > or this: > > # We happily mask exceptions from getgrnam > > g = grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid Either you'd get this anyway: try: g = grp.getgrnam(tarinfo.gname)[2] except KeyError: g = tarinfo.gid or your fallback value will be evaluated prematurely: g = grp.getgrnam(tarinfo.gname).get(2, tarinfo.gid) So to get this right: _info = grp.getgrnam(tarinfo.gname) try: g = _info[2] except KeyError: g = tarinfo.gid vs. _info = grp.getgrnam(tarinfo.gname) g = _info[2] except KeyError: tarinfo.gid > I can't believe you find > > list[42] except IndexError: 'spam' > > to be better than > > list.get(42, 'spam') What about: list[42] except IndexError: something_expensive() or: list[42] except IndexError: something_lazy_really_we_want_to_put_off() -eric From ericsnowcurrently at gmail.com Sat Feb 22 03:49:20 2014 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 21 Feb 2014 19:49:20 -0700 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Fri, Feb 21, 2014 at 7:07 PM, Victor Stinner wrote: >> Consider this example of a two-level cache:: >> for key in sequence: >> x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key))) >> # do something with x > > ... but I don't like when it is used to build complex expressions. This is true of any expression syntax, not just this proposal--though some expression syntax is more apt to be abused than others: you won't see many multiline int literals! ;) > > At the first read, I'm unable to understand this long expression. At > the second read, I'm still unable to see which instruction will be > executed first: lvl1[key] or lvl2[key]? > > The advantage of the current syntax is that the control flow is > obvious, from the top to the bottom: > > # start > try: > x = lvl1[key] # first instruction > except KeyError: > try: > x = lvl2[key] > except KeyError: > x = f(key) # latest instruction > # end > > Thanks to Python indentation, it's easier to see the control flow. > > After having read a lot of code last 10 years, I now try to follow > this rule: one instruction per line. +1 > > For example, I don't like "if ((res = func()) == NULL)" in the C > language, I always split it into two lines. A drawback of writing more > than one instruction is that it's hard to debug instruction per > instruction (in pdb/gdb). Think for example of "if ((res = func()) == > NULL)": if you execute step by step, how do you know which instruction > is currently executed? So true. > > You should maybe write your example differently: > > x = (lvl1[key] > except KeyError: (lvl2[key] > except KeyError: f(key))) > > It looks like the classic try/except syntax. +1 -eric From jessica.mckellar at gmail.com Sat Feb 22 04:09:57 2014 From: jessica.mckellar at gmail.com (Jessica McKellar) Date: Fri, 21 Feb 2014 22:09:57 -0500 Subject: [Python-Dev] GSOC 2014 In-Reply-To: References: Message-ID: Hi Navneet, > This is my first year in gsoc. I have been working with python and django > from quite a time. One of the Gsoc proposed ideas drew my attention and i > would surely like to work on that. > I would like to work for creating a extension for idle to integrate > PEP8, what are the prerequisites . I have been studying the PEP8 code past > a week. This might be useful for a quick start to this project. Please > guide. Thanks for your interest in CPython for Google Summer of Code! Mentor organizations haven't been selected yet (the official mentor list will go out on February 24th), and we are still finalizing our project list. In the meantime, I recommend: 1. Joining http://pythonmentors.com/ 2. Reading through the developers guide: http://docs.python.org/devguide/ 3. Picking and working on a ticket to get used to the workflow: http://bugs.python.org/ In particular, it will be important to have contributed some patches to CPython before you apply, so I recommend focusing on that for the next couple of weeks. If you have questions while finding bugs and preparing patches, please don't hesitate to ask on the pythonmentors.com mailing list. Regards, -Jessica From rosuav at gmail.com Sat Feb 22 04:42:30 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 14:42:30 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307A34D.6050101@gmail.com> References: <5307A34D.6050101@gmail.com> Message-ID: On Sat, Feb 22, 2014 at 6:04 AM, Yury Selivanov wrote: >> * seq[index] - no way to handle a bounds error > > We can add 'list.get(index, default)' method, similar to > 'Mapping.get'. It's far more easier than introducing new > syntax. That fixes it for the list. Square brackets notation works for any sequence. Are you proposing adding magic to the language so that any class that defines a __getitem__ automatically has a .get() with these semantics? Or, conversely, that defining __getitem__ requires that you also explicitly define get()? > Inconvenience of dict[] raising KeyError was solved by > introducing the dict.get() method. And I think that > > dct.get('a', 'b') > > is 1000 times better than > > dct['a'] except KeyError: 'b' > > I don't want to see this (or any other syntax) used by > anyone. Every separate method has to be written. That's code that has to be tested, etc. Also, when you go searching for usage of something, you have to cope with the fact that it can be spelled two different ways. This is more noticeable with attributes: print(spam.eggs) # versus print(getattr(spam,"eggs","(no eggs)") The second one definitely doesn't look like it's retrieving spam.eggs, either in plain text search or in an AST walk. I would like to see getattr used primarily where the second argument isn't a literal, and an exception-catching rig used when a default is wanted; that keeps the "x.y" form predictable. > Moreover, I think that explicit handling of IndexError is > rather ugly and error prone, using len() is usually > reads better. >> >> Retrieve an argument, defaulting to None:: >> cond = args[1] except IndexError: None >> >> # Lib/pdb.py:803: >> try: >> cond = args[1] >> except IndexError: >> cond = None > > > cond = None if (len(args) < 2) else args[1] There's a distinct difference between those: one is LBYL and the other is EAFP. Maybe it won't matter with retrieving arguments, but it will if you're trying to pop from a queue in a multi-threaded program. >> Attempt a translation, falling back on the original:: >> e.widget = self._nametowidget(W) except KeyError: W >> >> # Lib/tkinter/__init__.py:1222: >> try: >> e.widget = self._nametowidget(W) >> except KeyError: >> e.widget = W > > I'm not sure this is a good example either. > I presume '_nametowidget' is some function, > that might raise a KeyError because of a bug in > its implementation, or to signify that there is > no widget 'W'. Your new syntax just helps to work > with this error prone api. I don't know the details; this is exactly what you can see in tkinter at the file and line I point to. Maybe some of these are highlighting other problems to be solved, I don't know, but certainly there will be cases where the API is exactly like this. >> Read from an iterator, continuing with blank lines once it's >> exhausted:: >> line = readline() except StopIteration: '' >> >> # Lib/lib2to3/pgen2/tokenize.py:370: >> try: >> line = readline() >> except StopIteration: >> line = '' > > Handling StopIteration exception is more common in standard > library than IndexError (although not much more), but again, > not all of that code is suitable for your syntax. I'd say > about 30%, which is about 20-30 spots (correct me if I'm > wrong). I haven't counted them up, but it wouldn't be hard to. Probably not terribly many cases of this in the stdlib, but a reasonable few. >> Retrieve platform-specific information (note the DRY improvement); >> this particular example could be taken further, turning a series of >> separate assignments into a single large dict initialization:: >> # sys.abiflags may not be defined on all platforms. >> _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: '' >> >> # Lib/sysconfig.py:529: >> try: >> _CONFIG_VARS['abiflags'] = sys.abiflags >> except AttributeError: >> # sys.abiflags may not be defined on all platforms. >> _CONFIG_VARS['abiflags'] = '' > > Ugly. > _CONFIG_VARS['abiflags'] = getattr(sys, 'abiflags', '') > Much more readable. Go ahead and make that change, if you prefer it. That's exactly how it really is currently - the try/except block. Downside is as I mentioned above: it no longer looks like "sys.abiflags", and won't come up when you search for that. >> Retrieve an indexed item, defaulting to None (similar to dict.get):: >> def getNamedItem(self, name): >> return self._attrs[name] except KeyError: None >> >> # Lib/xml/dom/minidom.py:573: >> def getNamedItem(self, name): >> try: >> return self._attrs[name] >> except KeyError: >> return None > > _attrs there is a dict (or at least it's something that quaks > like a dict, and has [] and keys()), so > > return self._attrs.get(name) To what extent does it have to quack like a dict? In this particular example, I traced through a few levels of "where did _attrs come from", and got bogged down. Does "quacks like a dict" have to include a .get() method? >> Translate numbers to names, falling back on the numbers:: >> g = grp.getgrnam(tarinfo.gname)[2] except KeyError: >> tarinfo.gid >> u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: >> tarinfo.uid >> >> # Lib/tarfile.py:2198: >> try: >> g = grp.getgrnam(tarinfo.gname)[2] >> except KeyError: >> g = tarinfo.gid >> try: >> u = pwd.getpwnam(tarinfo.uname)[2] >> except KeyError: >> u = tarinfo.uid > > This one is a valid example, but totally unparseable by > humans. Moreover, it promotes a bad pattern, as you > mask KeyErrors in 'grp.getgrnam(tarinfo.gname)' call. My translation masks nothing that the original didn't mask. The KeyError will come from the function call; it would be IndexError if the function returns a too-short tuple, and that one's allowed to bubble up. >>> import pwd >>> pwd.getpwnam("rosuav") pwd.struct_passwd(pw_name='rosuav', pw_passwd='x', pw_uid=1000, pw_gid=1000, pw_gecos='Chris Angelico,,,', pw_dir='/home/rosuav', pw_shell='/bin/bash') >>> pwd.getpwnam("spam") Traceback (most recent call last): File "", line 1, in KeyError: 'getpwnam(): name not found: spam' (Note that it's possible for 'import pwd' to fail, in which case pwd is set to None early in the script. But this particular bit of code checks "if pwd" before continuing anyway, so we don't expect AttributeError here.) >> Calculate the mean of a series of numbers, falling back on zero:: >> >> value = statistics.mean(lst) except statistics.StatisticsError: 0 >> >> try: >> value = statistics.mean(lst) >> except statistics.StatisticsError: >> value = 0 > > I think all of the above more readable with try statement. Readability is a matter of personal preference, to some extent. I find it clearer to use the shorter form, for the same reason as I'd use this: def query(prompt, default): return input("%s [%s]: "%(prompt, default)) or default I wouldn't use long-hand: def query(prompt, default): s = input("%s [%s]: "%(prompt, default)) if not s: s = default return s It's easier to see that it's calling something, and defaulting to something else. >> Retrieving a message from either a cache or the internet, with auth >> check:: >> >> logging.info("Message shown to user: %s",((cache[k] >> except LookupError: >> (backend.read(k) except OSError: 'Resource not available') >> ) >> if check_permission(k) else 'Access denied' >> ) except BaseException: "This is like a bare except clause") >> >> try: >> if check_permission(k): >> try: >> _ = cache[k] >> except LookupError: >> try: >> _ = backend.read(k) >> except OSError: >> _ = 'Resource not available' >> else: >> _ = 'Access denied' >> except BaseException: >> _ = "This is like a bare except clause" >> logging.info("Message shown to user: %s", _) > > > If you replace '_' with a 'msg' (why did you use '_'??) > then try statements are *much* more readable. I've removed that example. The reason for using _ was because I wanted it to have the "feel" of still being an expression, where nothing's named. But it's not a very helpful example anyway; part of the confusion comes from the if/else in the middle, which completely wrecks evaluation order expectations. >> Lib/ipaddress.py:343:: >> try: >> ips.append(ip.ip) >> except AttributeError: >> ips.append(ip.network_address) >> Becomes:: >> ips.append(ip.ip except AttributeError: ip.network_address) > > or it may become: > > ips.append(getattr(ip, 'ip', ip.network_address)) > > or > > address = getattr(ip, 'ip', ip.network_address) > ips.append(address) There's a subtle difference here that makes that not equivalent. With the original try/except statement, evaluation proceeds thus: 1) Attempt to look up ip.ip. If that succeeds, call ips.append(). 2) If AttributeError is not thrown in #1, done. Otherwise, proceed. 3) Attempt to look up ip.network_address. If that succeeds, call ips.append. 4) Any exception raised will propagate. This means that, if neither ip nor network_address is defined, an AttributeError will come up, but that if ip is, network_address won't even be looked at. My version narrows the scope slightly, but is functionally similar. 1) Attempt to look up ip.ip. 2) If AttributeError is thrown in #1, attempt to look up ip.network_address. 3) If either #1 or #2 succeeds, call ips.append. 4) Any exception raised anywhere else will propagate. Your version, however, inverts the evaluation order: 1) Attempt to look up ip.network_address 2) If AttributeError is thrown in #1, propagate it up and stop evaluating. 3) Retrieve ip.ip, defaulting to the looked-up network address. 4) Pass that to ips.append(). It's probably not safe to use 'or' here, but if you can be sure ip.ip will never be blank, you could get lazy evaluation this way: ips.append(getattr(ip, 'ip', '') or ip.network_address) But at this point, the clarity advantage over the except clause is diminishing, plus it conflates AttributeError and ''. > Yes, some examples look neat. But your syntax is much easier > to abuse, than 'if..else' expression, and if people start > abusing it, Python will simply loose it's readability > advantage. If people start abusing it, style guides can tell them off. Unlike the if/else operator, evaluation of "expr except Exception: default" happens in strict left-to-right order, so in that sense it's _less_ confusing. I'll be adding a paragraph to the PEP shortly explaining that. ChrisA From rosuav at gmail.com Sat Feb 22 04:46:34 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 14:46:34 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307CDDE.8090106@canterbury.ac.nz> References: <5307CDDE.8090106@canterbury.ac.nz> Message-ID: On Sat, Feb 22, 2014 at 9:06 AM, Greg Ewing wrote: > Nick Coghlan wrote: >> >> On 21 February 2014 13:15, Chris Angelico wrote: >> >>> Generator expressions require parentheses, unless they would be >>> strictly redundant. Ambiguities with except expressions could be >>> resolved in the same way, forcing nested except-in-except trees to be >>> correctly parenthesized >> >> >> I'd like to make the case that the PEP should adopt this as its >> default position. > > > I generally agree, but I'd like to point out that this > doesn't necessarily mean making the parenthesizing rules as > strict as they are for generator expressions. > > The starting point for genexps is that the parens are part of > the syntax, the same way that square brackets are part of > the syntax of a list comprehension; we only allow them to > be omitted in very special circumstances. > > On the other hand, I don't think there's any harm in allowing > an except expression to stand on its own when there is no > risk of ambiguity, e.g. > > foo = things[i] except IndexError: None > > should be allowed, just as we allow > > x = a if b else c > > and don't require > > x = (a if b else c) I'm inclined to agree with you. I fought against the mandated parens for a long time. Would be happy to un-mandate them, as long as there's no way for there to be ambiguity. Is CPython able to make an operator non-associative? That is, to allow these: value = expr except Exception: default value = (expr except Exception: default) except Exception: default value = expr except Exception: (default except Exception: default) but not this: value = expr except Exception: default except Exception: default ? By effectively demanding parens any time two except-expressions meet, we could leave the option open to read multiple clauses off a single base exception. If that can be done, and if people's need for clarity can be satisfied, and if the rules aren't too complicated, I'd be happy to go back to "parens are optional". ChrisA From rosuav at gmail.com Sat Feb 22 04:51:57 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 14:51:57 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307E177.3080602@canterbury.ac.nz> References: <20140221145205.0b8b670d@fsol> <5307E177.3080602@canterbury.ac.nz> Message-ID: On Sat, Feb 22, 2014 at 10:29 AM, Greg Ewing wrote: > Antoine Pitrou wrote: > >>> lst = [1, 2] >>> value = lst[2] except IndexError: "No value" >> >> >> the gain in concision is counterbalanced by a loss in >> readability, > > > This version might be more readable: > > value = lst[2] except "No value" if IndexError > > since it puts the normal and exceptional values next > to each other, and relegates the exception type (which > is of much less interest) to a trailing aside. I'll expand on this in the PEP shortly, but there are two downsides to this. 1) Everywhere else in Python, "if" is followed by a boolean, and "except" is followed by an exception list. As a boolean, IndexError is always going to be true, which will confuse a human; and "No value" doesn't look like a modern exception at all. 2) Order of evaluation puts the main expression first, then the exception list, and lastly the default. Putting them in another order in the code is confusing. With ternary if/else, this is justified, but it's usually something to avoid. 3) Erm, among the downsides are such diverse elements... ahem. The part after the except clause is a full expression, and could plausibly have the if/else ternary operator. Starting with some expression, then if, then another expression, means you have to look for an else before you can be sure you're reading it correctly. Of course, any sane programmer who actually puts an if/else inside an except/if should parenthesize, but when you read someone else's code, you have to be prepared for the more normal sort of programmer. ChrisA From rosuav at gmail.com Sat Feb 22 04:57:28 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 14:57:28 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307F91A.6030304@g.nevcal.com> References: <20140221145205.0b8b670d@fsol> <5307E177.3080602@canterbury.ac.nz> <5307F91A.6030304@g.nevcal.com> Message-ID: On Sat, Feb 22, 2014 at 12:10 PM, Glenn Linderman wrote: > Here's a challenge: There has been a big thread about None versus (SQL) > Null. Show how an except: expression can help the DB API more easily convert > from using None to using a new Null singleton, and you'll have a winner :) Heh! I'm not entirely sure how that would work, as I've only skimmed the DB API thread, but I understand part of it is to do with sorting. I'm not sure how you could embed an except-expression into that without either changing the sort function or potentially doing part of the sort twice: lst = sorted(stuff) except TypeError: sorted(stuff, key=keyfunc) at which point you may as well go straight to the key= form from the start. ChrisA From stephen at xemacs.org Sat Feb 22 04:57:48 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 22 Feb 2014 12:57:48 +0900 Subject: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final In-Reply-To: <20140220125309.282e79fb@fsol> References: <53014900.7090908@hastings.org> <5303EEE7.8000101@ubuntu.com> <20140218185423.256a3e4d@anarchist.wooz.org> <87a9dmwp8f.fsf@uwakimon.sk.tsukuba.ac.jp> <20140220125309.282e79fb@fsol> Message-ID: <87y513vlxf.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > On Thu, 20 Feb 2014 10:24:16 +0900 > "Stephen J. Turnbull" wrote: > > > > The argument that a "read-only, no cherrypicking by committers" repo > > is nothing but a better tarball is valid, but as I say, AFAICS the > > expected gain is pretty marginal. The conflict here is not Larry's > > process, it's the decision to make an ambitious release on a short > > time schedule. > > I don't really buy the "short time schedule" argument. The delay > between 3.3 and 3.4 is ~18 months as usual. "Short" here isn't relative to calendar time nor to the development cycle length. It's relative to the time needed to properly beta and RC an "ambitious" set of changes, with more than the usual number of patches during RC AFAIK, and the time allowed for that testing. I'm not saying it isn't enough time, but I certainly think that more time or a less ambitious release would make folks feel more comfortable, and less apt to suggest changes in a release process at this late date. My point is that I don't think that changing Larry's process would make a big difference to our ability to test the release candidates, Ubuntu deadlines not withstanding. From rosuav at gmail.com Sat Feb 22 05:01:46 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 15:01:46 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <53080384.3080808@canterbury.ac.nz> References: <5307EC11.90407@canterbury.ac.nz> <20140222012131.GA3684@ando> <53080384.3080808@canterbury.ac.nz> Message-ID: On Sat, Feb 22, 2014 at 12:55 PM, Greg Ewing wrote: > Steven D'Aprano wrote: > >> result = computation( >> int(arg) except ValueError: abort("Invalid int") >> ) >> >> Actually, not quite so nice as I first thought, since you're relying on >> the side-effects of abort() rather than returning a value. > > > Yeah, while I was writing that I wondered whether > you should be allowed to write > > int(arg) except ValueError: raise UserError("Invalid int") > > That looks heretical, because 'raise' can't in any > way be interpreted as a value-returning expression. > But you can achieve the same result using a function > that always raises and exception, so forbidding it > on those grounds would be pointless. def throw(exc): raise exc int(arg) except ValueError: throw(UserError("Invalid int")) Tiny helper function and then it doesn't need any special syntax. It's not so much that PEP 463 forbids this, as that it doesn't explicitly permit it. The 'default' part of the syntax is an expression, raise is not an expression, ergo it's not permitted. But if you think this is sufficiently common (translate an exception on the way through), show some code and justify its addition - or just write up a separate proposal for "raise X" to become an expression, same as "yield X" is. ChrisA From rosuav at gmail.com Sat Feb 22 05:04:35 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 15:04:35 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Sat, Feb 22, 2014 at 1:07 PM, Victor Stinner wrote: > At the first read, I'm unable to understand this long expression. At > the second read, I'm still unable to see which instruction will be > executed first: lvl1[key] or lvl2[key]? > > The advantage of the current syntax is that the control flow is > obvious, from the top to the bottom: > > # start > try: > x = lvl1[key] # first instruction > except KeyError: > try: > x = lvl2[key] > except KeyError: > x = f(key) # latest instruction > # end That's why I'm strongly in favour of syntax variants that have evaluation order be equally obvious: left to right. Its notation may be uglier, but C's ternary operator does get this right, where Python's executes from the inside out. It's not a big deal when most of it is constants, but it can help a lot when the expressions nest. ChrisA From rosuav at gmail.com Sat Feb 22 05:22:44 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 15:22:44 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Sat, Feb 22, 2014 at 3:04 PM, Chris Angelico wrote: > On Sat, Feb 22, 2014 at 1:07 PM, Victor Stinner > wrote: >> At the first read, I'm unable to understand this long expression. At >> the second read, I'm still unable to see which instruction will be >> executed first: lvl1[key] or lvl2[key]? >> >> The advantage of the current syntax is that the control flow is >> obvious, from the top to the bottom: >> >> # start >> try: >> x = lvl1[key] # first instruction >> except KeyError: >> try: >> x = lvl2[key] >> except KeyError: >> x = f(key) # latest instruction >> # end > > That's why I'm strongly in favour of syntax variants that have > evaluation order be equally obvious: left to right. Its notation may > be uglier, but C's ternary operator does get this right, where > Python's executes from the inside out. It's not a big deal when most > of it is constants, but it can help a lot when the expressions nest. I've added a couple of paragraphs to my draft PEP: https://raw.github.com/Rosuav/ExceptExpr/master/pep-0463.txt If someone could please commit that version to the official repo? Or I can submit a diff against the peps repo if that would be easier. ChrisA From yselivanov.ml at gmail.com Sat Feb 22 05:25:48 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 21 Feb 2014 23:25:48 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <5307A34D.6050101@gmail.com> Message-ID: <530826CC.4020708@gmail.com> On 2/21/2014, 10:42 PM, Chris Angelico wrote: > On Sat, Feb 22, 2014 at 6:04 AM, Yury Selivanov wrote: >>> * seq[index] - no way to handle a bounds error >> We can add 'list.get(index, default)' method, similar to >> 'Mapping.get'. It's far more easier than introducing new >> syntax. > That fixes it for the list. Square brackets notation works for any > sequence. Are you proposing adding magic to the language so that any > class that defines a __getitem__ automatically has a .get() with these > semantics? Or, conversely, that defining __getitem__ requires that you > also explicitly define get()? No, I proposed to consider adding 'list.get()' and 'collections.abc.Sequence.get()' (or a better name), if catching IndexError is such a popular pattern. We can also fix stdlib. Authors of the python libraries/ frameworks will likely follow. No magic in the language, no requirements on __getitem__, obviously. >> Inconvenience of dict[] raising KeyError was solved by >> introducing the dict.get() method. And I think that >> >> dct.get('a', 'b') >> >> is 1000 times better than >> >> dct['a'] except KeyError: 'b' >> >> I don't want to see this (or any other syntax) used by >> anyone. > Every separate method has to be written. That's code that has to be > tested, etc. Also, when you go searching for usage of something, you > have to cope with the fact that it can be spelled two different ways. > This is more noticeable with attributes: > > print(spam.eggs) > # versus > print(getattr(spam,"eggs","(no eggs)") > > The second one definitely doesn't look like it's retrieving spam.eggs, > either in plain text search or in an AST walk. And that's not a common thing to use AttributeError/getattr at all. It is common in frameworks, yes, but not that much in client code. > I would like to see > getattr used primarily where the second argument isn't a literal, and > an exception-catching rig used when a default is wanted; that keeps > the "x.y" form predictable. I understand. But I still think that using getattr is better than 'a.b except AttributeError: c' >> Moreover, I think that explicit handling of IndexError is >> rather ugly and error prone, using len() is usually >> reads better. >>> Retrieve an argument, defaulting to None:: >>> cond = args[1] except IndexError: None >>> >>> # Lib/pdb.py:803: >>> try: >>> cond = args[1] >>> except IndexError: >>> cond = None >> >> cond = None if (len(args) < 2) else args[1] > There's a distinct difference between those: one is LBYL and the other > is EAFP. Maybe it won't matter with retrieving arguments, but it will > if you're trying to pop from a queue in a multi-threaded program. Using a method that modifies its underlying object deep in expression is also questionable, in terms of readability and maintainability of the code. I find try..except statement more favourable here. >>> Attempt a translation, falling back on the original:: >>> e.widget = self._nametowidget(W) except KeyError: W >>> >>> # Lib/tkinter/__init__.py:1222: >>> try: >>> e.widget = self._nametowidget(W) >>> except KeyError: >>> e.widget = W >> I'm not sure this is a good example either. >> I presume '_nametowidget' is some function, >> that might raise a KeyError because of a bug in >> its implementation, or to signify that there is >> no widget 'W'. Your new syntax just helps to work >> with this error prone api. > I don't know the details; this is exactly what you can see in tkinter > at the file and line I point to. Maybe some of these are highlighting > other problems to be solved, I don't know, but certainly there will be > cases where the API is exactly like this. > >>> Read from an iterator, continuing with blank lines once it's >>> exhausted:: >>> line = readline() except StopIteration: '' >>> >>> # Lib/lib2to3/pgen2/tokenize.py:370: >>> try: >>> line = readline() >>> except StopIteration: >>> line = '' >> Handling StopIteration exception is more common in standard >> library than IndexError (although not much more), but again, >> not all of that code is suitable for your syntax. I'd say >> about 30%, which is about 20-30 spots (correct me if I'm >> wrong). > I haven't counted them up, but it wouldn't be hard to. Probably not > terribly many cases of this in the stdlib, but a reasonable few. Just having a "reasonable few" use cases in huge stdlib isn't a warrant for new syntax. > >>> Retrieve platform-specific information (note the DRY improvement); >>> this particular example could be taken further, turning a series of >>> separate assignments into a single large dict initialization:: >>> # sys.abiflags may not be defined on all platforms. >>> _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: '' >>> >>> # Lib/sysconfig.py:529: >>> try: >>> _CONFIG_VARS['abiflags'] = sys.abiflags >>> except AttributeError: >>> # sys.abiflags may not be defined on all platforms. >>> _CONFIG_VARS['abiflags'] = '' >> Ugly. >> _CONFIG_VARS['abiflags'] = getattr(sys, 'abiflags', '') >> Much more readable. > Go ahead and make that change, if you prefer it. That's exactly how it > really is currently - the try/except block. Downside is as I mentioned > above: it no longer looks like "sys.abiflags", and won't come up when > you search for that. I'd search for 'abiflags' since it's not a common name ;) But again, I get your point here. >>> Retrieve an indexed item, defaulting to None (similar to dict.get):: >>> def getNamedItem(self, name): >>> return self._attrs[name] except KeyError: None >>> >>> # Lib/xml/dom/minidom.py:573: >>> def getNamedItem(self, name): >>> try: >>> return self._attrs[name] >>> except KeyError: >>> return None >> _attrs there is a dict (or at least it's something that quaks >> like a dict, and has [] and keys()), so >> >> return self._attrs.get(name) > To what extent does it have to quack like a dict? In this particular > example, I traced through a few levels of "where did _attrs come > from", and got bogged down. Does "quacks like a dict" have to include > a .get() method? The point is that whoever wrote that code knew what is _attrs. Likely it's a dict, since they use '.keys()' on it. And likely, that person just prefers try..except instead of using '.get()'. And I just want to say, that this particular example isn't a good use case for the new syntax you propose. >>> Translate numbers to names, falling back on the numbers:: >>> g = grp.getgrnam(tarinfo.gname)[2] except KeyError: >>> tarinfo.gid >>> u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: >>> tarinfo.uid >>> >>> # Lib/tarfile.py:2198: >>> try: >>> g = grp.getgrnam(tarinfo.gname)[2] >>> except KeyError: >>> g = tarinfo.gid >>> try: >>> u = pwd.getpwnam(tarinfo.uname)[2] >>> except KeyError: >>> u = tarinfo.uid >> This one is a valid example, but totally unparseable by >> humans. Moreover, it promotes a bad pattern, as you >> mask KeyErrors in 'grp.getgrnam(tarinfo.gname)' call. > My translation masks nothing that the original didn't mask. The > KeyError will come from the function call; it would be IndexError if > the function returns a too-short tuple, and that one's allowed to > bubble up. Right > >>>> import pwd >>>> pwd.getpwnam("rosuav") > pwd.struct_passwd(pw_name='rosuav', pw_passwd='x', pw_uid=1000, > pw_gid=1000, pw_gecos='Chris Angelico,,,', pw_dir='/home/rosuav', > pw_shell='/bin/bash') >>>> pwd.getpwnam("spam") > Traceback (most recent call last): > File "", line 1, in > KeyError: 'getpwnam(): name not found: spam' > > (Note that it's possible for 'import pwd' to fail, in which case pwd > is set to None early in the script. But this particular bit of code > checks "if pwd" before continuing anyway, so we don't expect > AttributeError here.) > >>> Calculate the mean of a series of numbers, falling back on zero:: >>> >>> value = statistics.mean(lst) except statistics.StatisticsError: 0 >>> >>> try: >>> value = statistics.mean(lst) >>> except statistics.StatisticsError: >>> value = 0 >> I think all of the above more readable with try statement. > Readability is a matter of personal preference, to some extent. I find > it clearer to use the shorter form, for the same reason as I'd use > this: > > def query(prompt, default): > return input("%s [%s]: "%(prompt, default)) or default > > I wouldn't use long-hand: > > def query(prompt, default): > s = input("%s [%s]: "%(prompt, default)) > if not s: s = default > return s > > It's easier to see that it's calling something, and defaulting to > something else. > >>> Retrieving a message from either a cache or the internet, with auth >>> check:: >>> >>> logging.info("Message shown to user: %s",((cache[k] >>> except LookupError: >>> (backend.read(k) except OSError: 'Resource not available') >>> ) >>> if check_permission(k) else 'Access denied' >>> ) except BaseException: "This is like a bare except clause") >>> >>> try: >>> if check_permission(k): >>> try: >>> _ = cache[k] >>> except LookupError: >>> try: >>> _ = backend.read(k) >>> except OSError: >>> _ = 'Resource not available' >>> else: >>> _ = 'Access denied' >>> except BaseException: >>> _ = "This is like a bare except clause" >>> logging.info("Message shown to user: %s", _) >> >> If you replace '_' with a 'msg' (why did you use '_'??) >> then try statements are *much* more readable. > I've removed that example. The reason for using _ was because I wanted > it to have the "feel" of still being an expression, where nothing's > named. But it's not a very helpful example anyway; part of the > confusion comes from the if/else in the middle, which completely > wrecks evaluation order expectations. I think that this example is better to be kept ;) But up to you. While each PEP want's to be Final, it's still good to see how people can abuse it. And this example is excellent in that regard. >>> Lib/ipaddress.py:343:: >>> try: >>> ips.append(ip.ip) >>> except AttributeError: >>> ips.append(ip.network_address) >>> Becomes:: >>> ips.append(ip.ip except AttributeError: ip.network_address) >> or it may become: >> >> ips.append(getattr(ip, 'ip', ip.network_address)) >> >> or >> >> address = getattr(ip, 'ip', ip.network_address) >> ips.append(address) > There's a subtle difference here that makes that not equivalent. With > the original try/except statement, evaluation proceeds thus: > > 1) Attempt to look up ip.ip. If that succeeds, call ips.append(). > 2) If AttributeError is not thrown in #1, done. Otherwise, proceed. > 3) Attempt to look up ip.network_address. If that succeeds, call ips.append. > 4) Any exception raised will propagate. > > This means that, if neither ip nor network_address is defined, an > AttributeError will come up, but that if ip is, network_address won't > even be looked at. > > My version narrows the scope slightly, but is functionally similar. > > 1) Attempt to look up ip.ip. > 2) If AttributeError is thrown in #1, attempt to look up ip.network_address. > 3) If either #1 or #2 succeeds, call ips.append. > 4) Any exception raised anywhere else will propagate. > > Your version, however, inverts the evaluation order: > > 1) Attempt to look up ip.network_address > 2) If AttributeError is thrown in #1, propagate it up and stop evaluating. > 3) Retrieve ip.ip, defaulting to the looked-up network address. > 4) Pass that to ips.append(). > > It's probably not safe to use 'or' here, but if you can be sure ip.ip > will never be blank, you could get lazy evaluation this way: > > ips.append(getattr(ip, 'ip', '') or ip.network_address) > > But at this point, the clarity advantage over the except clause is > diminishing, plus it conflates AttributeError and ''. Yes, conditional evaluation a good part of the new syntax (but again, this *particular* example doesn't show it; better to use some ORM maybe, where each __getattr__ is potentially a query). >> Yes, some examples look neat. But your syntax is much easier >> to abuse, than 'if..else' expression, and if people start >> abusing it, Python will simply loose it's readability >> advantage. > If people start abusing it, style guides can tell them off. Unlike the > if/else operator, evaluation of "expr except Exception: default" > happens in strict left-to-right order, so in that sense it's _less_ > confusing. I'll be adding a paragraph to the PEP shortly explaining > that. > There is one more thing: "There should be one-- and preferably only one --obvious way to do it." -- so far it's true for all python language constructs. They all are solving a real pain point. Maybe it's just me, but I fail to see the real pain point this proposal solves. To conclude: 1. I'm still not convinced that the new syntax is really necessary. 2. You should probably invest some time in finding better examples for the PEP. Examples, that will focus on its strengths, and show the real need. 3. Syntax. Using ':' in it makes it look bad. I know there are many other alternatives, and 400+ emails on python-ideas, but still, we couldn't find a viable alternative. Perhaps, we need to look for it some more time. My 2 cents. Yury From yselivanov.ml at gmail.com Sat Feb 22 05:29:36 2014 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Fri, 21 Feb 2014 23:29:36 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <530827B0.5070108@gmail.com> On 2/21/2014, 11:22 PM, Chris Angelico wrote: > I've added a couple of paragraphs to my draft PEP: > > https://raw.github.com/Rosuav/ExceptExpr/master/pep-0463.txt > > If someone could please commit that version to the official repo? Or I > can submit a diff against the peps repo if that would be easier. I've committed the new version. Yury From ethan at stoneleaf.us Sat Feb 22 05:08:42 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 21 Feb 2014 20:08:42 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <5307CDDE.8090106@canterbury.ac.nz> Message-ID: <530822CA.6030609@stoneleaf.us> On 02/21/2014 07:46 PM, Chris Angelico wrote: > > but not this: > > value = expr except Exception: default except Exception: default This should be the way it works. Nothing is gained in readability by turning a try with multiple except statements into an expression. -- ~Ethan~ From steve at pearwood.info Sat Feb 22 06:01:16 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 22 Feb 2014 16:01:16 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307A34D.6050101@gmail.com> References: <5307A34D.6050101@gmail.com> Message-ID: <20140222050114.GE3684@ando> On Fri, Feb 21, 2014 at 02:04:45PM -0500, Yury Selivanov wrote: > Inconvenience of dict[] raising KeyError was solved by > introducing the dict.get() method. And I think that > > dct.get('a', 'b') > > is 1000 times better than > > dct['a'] except KeyError: 'b' I don't think it is better. I think that if we had good exception-catching syntax, we wouldn't need a get method. "get" methods do not solve the problem, they are shift it. Every class that defines [] look-ups also needs a get method. We can write dict.get(key, default), but we can't write list.get(index, default). If we "fix" list, we then find that tuple.get(index, default) fails. So we "fix" tuple, and then discover that str.get(index, default) fails. If Python were Ruby, then all sequences could inherit from one Sequence class that defines: def get(self, index, default=None): try: return self[index] except IndexError: return default and similarly for mappings, with KeyError instead of IndexError. But this is Python, not Ruby, and there is no requirement that sequences must inherit from the same parent class. They just have to provide the same duck-typed interface. Having added a get method to every mapping and sequence, you then call list.pop() and discover that it too needs a default. And so on, forever. We're forever chasing the next method and function, adding default parameters to everything in sight. The latest example is max and min. Then there are functions or methods which come in pairs, like str.find and str.index. From time to time people ask for a version of list.index that returns a default value instead of raising. Should we give them one? I don't think Python is a better language by adding complexity into the libraries and built-ins in this way. (And let's not forget that while methods and functions can have default parameters, expressions cannot. So there is a whole class of potential operations that can never be given a default, because they aren't a function call.) The essence of programming is combining primitive constructs to make more complicated ones, not in the proliferation of special-purpose methods or functions. "get" is less primitive than normal [] lookup, and it exists for a special purpose: to avoid needing to break apart what ought is a single conceptual expression into a multi-line statement. Consider a single conceptual chunk of code, the expression process(mydict[key]). In the past, if you wanted to use a default value if the key didn't exist, the only alternative was to break that expression up into two statements, regardless of whether you used LBYL or EAFP idioms: if key in mydict: tmp = mydict[key] else: tmp = default result = process(tmp) try: tmp = mydict[key] except KeyError: tmp = default result = process(tmp) Consequently, to express a single expression *as a single expression*, we needed an expression that can do the same thing, and in the past that had to be a method: result = process(mydict.get(key, default)) But now we have a LBYL expression form: result = process(mydict[key] if key in mydict else default) and are suggesting a EAFP form: result = process(mydict[key] except KeyError: default) If Python had this syntax 20 years ago, would dicts have grown a get method? I doubt it very much. People would be told to wrap it in an except expression. > >Consider this example of a two-level cache:: > > for key in sequence: > > x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: > > f(key))) > > # do something with x > > I'm sorry, it took me a minute to understand what your > example is doing. I would rather see two try..except blocks > than this. It helps to break it up more appropriately. (Also less cryptic names.) I can see PEP 8 will need some changes if this syntax is approved. for key in sequence: x = (cache1[key] except KeyError: (cache2[key] except KeyError: f(key))) # Do something with x This proposal is not about saving lines of code, and if examples are written jammed into a single line or two, and that hurts readability, they should be formatted better. It is about expressing the EAFP idiom in an expression, without giving up the useful property that the "except" clause is only evaluated when needed, not in advance. (Chris, I think that ought to go in the motivation section of the PEP.) -- Steven From rosuav at gmail.com Sat Feb 22 06:34:40 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 16:34:40 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222050114.GE3684@ando> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> Message-ID: On Sat, Feb 22, 2014 at 4:01 PM, Steven D'Aprano wrote: > (Chris, I think that ought to go in the motivation section of the PEP.) Added to my draft, and here's the peps diff: diff -r c52a2ae3d98e pep-0463.txt --- a/pep-0463.txt Fri Feb 21 23:27:51 2014 -0500 +++ b/pep-0463.txt Sat Feb 22 16:33:37 2014 +1100 @@ -43,6 +43,34 @@ * statistics.mean(data) - no way to handle an empty iterator +Had this facility existed early in Python's history, there would have been +no need to create dict.get() and related methods; the one obvious way to +handle an absent key would be to respond to the exception. One method is +written which signal the absence in one way, and one consistent technique +is used to respond to the absence. Instead, we have dict.get(), and as of +Python 3.4, we also have min(... default=default), and myriad others. We +have a LBYL syntax for testing inside an expression, but there is currently +no EAFP notation; compare the following:: + + # LBYL: + if key in dic: + process(dic[key]) + else: + process(None) + # As an expression: + process(dic[key] if key in dic else None) + + # EAFP: + try: + process(dic[key]) + except KeyError: + process(None) + # As an expression: + process(dic[key] except KeyError: None) + +Python generally recommends the EAFP policy, but must then proliferate +utility functions like dic.get(key,None) to enable this. + Rationale ========= From stephen at xemacs.org Sat Feb 22 07:57:02 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 22 Feb 2014 15:57:02 +0900 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <530822CA.6030609@stoneleaf.us> References: <5307CDDE.8090106@canterbury.ac.nz> <530822CA.6030609@stoneleaf.us> Message-ID: <87sirbvdmp.fsf@uwakimon.sk.tsukuba.ac.jp> Ethan Furman writes: > On 02/21/2014 07:46 PM, Chris Angelico wrote: > > > > but not this: > > > > value = expr except Exception: default except Exception: default > > This should be the way it works. Nothing is gained in readability > by turning a try with multiple except statements into an > expression. Examples have been given several times. In general, if 'expr' is a function call, it may well have a couple of different ways to fail which imply different default values. interpolable = func(key) except TypeError: "not a string: %s" % key \ except KeyError: "no such key: %s" % key print("Some message that refers to '%s' % interpolable") versus try: interpolable = func(key) except TypeError: interpolable = "not a string: %s" % key except KeyError: interpolable = "no such key: %s" % key print("Some message that refers to '%s' % interpolable") I think the latter begs to be written as the former. From v+python at g.nevcal.com Sat Feb 22 07:51:09 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 21 Feb 2014 22:51:09 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <5307E177.3080602@canterbury.ac.nz> <5307F91A.6030304@g.nevcal.com> Message-ID: <530848DD.6000903@g.nevcal.com> On 2/21/2014 7:57 PM, Chris Angelico wrote: > On Sat, Feb 22, 2014 at 12:10 PM, Glenn Linderman wrote: >> Here's a challenge: There has been a big thread about None versus (SQL) >> Null. Show how an except: expression can help the DB API more easily convert >> from using None to using a new Null singleton, and you'll have a winner :) > Heh! I'm not entirely sure how that would work, as I've only skimmed > the DB API thread, but I understand part of it is to do with sorting. > I'm not sure how you could embed an except-expression into that > without either changing the sort function or potentially doing part of > the sort twice: > > lst = sorted(stuff) except TypeError: sorted(stuff, key=keyfunc) > > at which point you may as well go straight to the key= form from the start. Yes, the "challenge" was sort of tongue-in-cheek... it was the latest heavily opinionated thread with no conclusion... and if a particular feature could help arrive at a good solution I think a lot of people would be happy! And probably a lot would still think it wasn't a good solution! It does help acceptance of a new feature to describe and demonstrate a solution to a compelling problem; Stephen's recent explanation of LBYL and EAFP expressions helped make it more compelling to me, it sounded like you added some of that to the PEP, which is good. Some folks didn't find the PEP examples compelling, it seems. The sorting is a smoke-screen. The real problem is that None and Null are not the same thing, and so internally the DB Api should invent and use Null, and translate to None (optionally) at the interfaces, for backwards compatibility until their users can update their software to use Null also. The implementation of Null can handle the comparison/sorting issue... there are a fixed number of types for SQL, so it isn't that hard. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat Feb 22 08:12:27 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 22 Feb 2014 16:12:27 +0900 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222050114.GE3684@ando> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> Message-ID: <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > On Fri, Feb 21, 2014 at 02:04:45PM -0500, Yury Selivanov wrote: > > > Inconvenience of dict[] raising KeyError was solved by > > introducing the dict.get() method. And I think that > > > > dct.get('a', 'b') > > > > is 1000 times better than > > > > dct['a'] except KeyError: 'b' Aside from the "no need for 'get'" argument, I like the looks of the latter better, because dct.get('a', 'b') could be doing anything, while the syntax in the expression is defined by the language and says exactly what it's doing, even if read in English (obviously you have to be a Python programmer to understand that, but you don't need to be Dutch). > I don't think it is better. I think that if we had good > exception-catching syntax, we wouldn't need a get method. > > "get" methods do not solve the problem, they are shift it. Note in support: I originally thought that "get" methods would be more efficient, but since Nick pointed out that "haveattr" is implemented by catching the exception (Yikes! LBYL implemented by using EAFP!), I assume that get methods also are (explicitly or implicitly) implemented that way. I'm not a huge fan of the proposal (in my own code there aren't a lot of potential uses, and I think the colon syntax is a little ugly), but the "eliminates need for .get()" argument persuades me that the colon syntax is better than nothing. From larry at hastings.org Sat Feb 22 08:42:57 2014 From: larry at hastings.org (Larry Hastings) Date: Sat, 22 Feb 2014 01:42:57 -0600 Subject: [Python-Dev] One more cherry-pick request for 3.4.0 that I'd like a little public debate on Message-ID: <53085501.7070406@hastings.org> Victor has asked me to cherry-pick 180e4b678003: http://bugs.python.org/issue20320 (original issue) http://hg.python.org/cpython/rev/180e4b678003/ (checkin into trunk) http://bugs.python.org/issue20646 (cherry-pick request) This revision changes the rounding behavior of fractional-second timeouts for select.select and select.kqueue. I don't have enough context to judge whether or not this is bad enough to warrant cherry-picking, and the discussion on the issue didn't seem to come to a firm consensus. Can I get some opinions on this? FWIW, I'm going to cut the next preview in a few minutes, and this won't be in it, but I don't actually tag until most of a day from now. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sat Feb 22 09:22:57 2014 From: larry at hastings.org (Larry Hastings) Date: Sat, 22 Feb 2014 02:22:57 -0600 Subject: [Python-Dev] Fourth (final?) preview of rc2 is up Message-ID: <53085E61.1030200@hastings.org> Only three new revisions this time. Victor must have taken the day off! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Feb 22 09:06:42 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 22 Feb 2014 00:06:42 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <87sirbvdmp.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5307CDDE.8090106@canterbury.ac.nz> <530822CA.6030609@stoneleaf.us> <87sirbvdmp.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <53085A92.8070000@stoneleaf.us> On 02/21/2014 10:57 PM, Stephen J. Turnbull wrote: > Ethan Furman writes: >> On 02/21/2014 07:46 PM, Chris Angelico wrote: >>> >>> but not this: >>> >>> value = expr except Exception: default except Exception: default >> >> This should be the way it works. Nothing is gained in readability >> by turning a try with multiple except statements into an >> expression. > > Examples have been given several times. In general, if 'expr' is a > function call, it may well have a couple of different ways to fail > which imply different default values. > > interpolable = func(key) except TypeError: "not a string: %s" % key \ > except KeyError: "no such key: %s" % key > print("Some message that refers to '%s' % interpolable") > > versus > > try: > interpolable = func(key) > except TypeError: > interpolable = "not a string: %s" % key > except KeyError: > interpolable = "no such key: %s" % key > print("Some message that refers to '%s' % interpolable") > > I think the latter begs to be written as the former. Okay, that's the best example of that style I've seen so far (sorry, Chris, if something similar was in the PEP and I missed it). I will yield the point that something is gained -- still, I think it is a small something compared to converting a nested except statement into an expression, and if only allowing one or the other makes the whole thing simpler I vote for the nested excepts to be converted, not the already easily read multiple excepts. -- ~Ethan~ From solipsis at pitrou.net Sat Feb 22 10:12:48 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Feb 2014 10:12:48 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions References: <5307CDDE.8090106@canterbury.ac.nz> <530822CA.6030609@stoneleaf.us> <87sirbvdmp.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140222101248.46d76ff4@fsol> On Sat, 22 Feb 2014 15:57:02 +0900 "Stephen J. Turnbull" wrote: > > try: > interpolable = func(key) > except TypeError: > interpolable = "not a string: %s" % key > except KeyError: > interpolable = "no such key: %s" % key > print("Some message that refers to '%s' % interpolable") I think that's a rare enough case, though (compared to the other idioms showcased in the PEP), that we needn't care about it. Regards Antoine. From solipsis at pitrou.net Sat Feb 22 10:13:17 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Feb 2014 10:13:17 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <20140221183250.2b44f860@fsol> Message-ID: <20140222101317.058e10c2@fsol> On Fri, 21 Feb 2014 09:37:29 -0800 Guido van Rossum wrote: > I'm put off by the ':' syntax myself (it looks to me as if someone forgot a > newline somewhere) but 'then' feels even weirder (it's been hard-coded in > my brain as meaning the first branch of an 'if'). Would 'else' work rather than 'then'? Regards Antoine. From solipsis at pitrou.net Sat Feb 22 10:20:51 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Feb 2014 10:20:51 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20140222102051.16b65fda@fsol> On Sat, 22 Feb 2014 16:12:27 +0900 "Stephen J. Turnbull" wrote: > > Note in support: I originally thought that "get" methods would be more > efficient, but since Nick pointed out that "haveattr" is implemented > by catching the exception (Yikes! LBYL implemented by using EAFP!), I > assume that get methods also are (explicitly or implicitly) > implemented that way. Well, the only way to know that a key (or attribute) exists is to do the lookup. What else would you suggest? And, yes, EAFP can avoid race conditions and the like (besides being more efficient with non-trivial keys). Regards Antoine. From solipsis at pitrou.net Sat Feb 22 10:23:54 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Feb 2014 10:23:54 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions References: Message-ID: <20140222102354.3d49ec85@fsol> On Fri, 21 Feb 2014 19:49:20 -0700 Eric Snow wrote: > On Fri, Feb 21, 2014 at 7:07 PM, Victor Stinner > wrote: > >> Consider this example of a two-level cache:: > >> for key in sequence: > >> x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key))) > >> # do something with x > > > > ... but I don't like when it is used to build complex expressions. > > This is true of any expression syntax, not just this proposal--though > some expression syntax is more apt to be abused than others: you won't > see many multiline int literals! ;) But this is precisely why Python has refrained from making everything an expression. For example, why assignements are not expressions. Regards Antoine. From rosuav at gmail.com Sat Feb 22 10:29:27 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 20:29:27 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222102051.16b65fda@fsol> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> Message-ID: On Sat, Feb 22, 2014 at 8:20 PM, Antoine Pitrou wrote: > On Sat, 22 Feb 2014 16:12:27 +0900 > "Stephen J. Turnbull" wrote: >> >> Note in support: I originally thought that "get" methods would be more >> efficient, but since Nick pointed out that "haveattr" is implemented >> by catching the exception (Yikes! LBYL implemented by using EAFP!), I >> assume that get methods also are (explicitly or implicitly) >> implemented that way. > > Well, the only way to know that a key (or attribute) exists is to do > the lookup. What else would you suggest? > > And, yes, EAFP can avoid race conditions and the like (besides being > more efficient with non-trivial keys). Which means that, fundamentally, EAFP is the way to do it. So if PEP 463 expressions had existed from the beginning, hasattr() probably wouldn't have been written - people would just use an except-expression instead. ChrisA From solipsis at pitrou.net Sat Feb 22 10:27:11 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Feb 2014 10:27:11 +0100 Subject: [Python-Dev] One more cherry-pick request for 3.4.0 that I'd like a little public debate on References: <53085501.7070406@hastings.org> Message-ID: <20140222102711.0a492256@fsol> On Sat, 22 Feb 2014 01:42:57 -0600 Larry Hastings wrote: > > Victor has asked me to cherry-pick 180e4b678003: > > http://bugs.python.org/issue20320 (original issue) > http://hg.python.org/cpython/rev/180e4b678003/ (checkin into trunk) > http://bugs.python.org/issue20646 (cherry-pick request) > > This revision changes the rounding behavior of fractional-second > timeouts for select.select and select.kqueue. I don't have enough > context to judge whether or not this is bad enough to warrant > cherry-picking, and the discussion on the issue didn't seem to come to a > firm consensus. > > Can I get some opinions on this? Well, it's certainly not rc-critical. It improves a bit a fringe behaviour that's unlikely to be noticeable by anyone in the real world. If you look at http://bugs.python.org/issue20320, Charles-Fran?ois there explains why it's a minor issue. Regards Antoine. From rosuav at gmail.com Sat Feb 22 10:45:30 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 20:45:30 +1100 Subject: [Python-Dev] Tangent on class level scoping rules (was Re: PEP 463: Exception-catching expressions) In-Reply-To: <5307E7E8.6060001@canterbury.ac.nz> References: <5307E7E8.6060001@canterbury.ac.nz> Message-ID: On Sat, Feb 22, 2014 at 10:57 AM, Greg Ewing wrote: > Nick Coghlan wrote: >> >> As Chris later noted, you likely *could* still implement expression >> local name binding for an except expression without a full closure, it >> would just be rather difficult. > > > I'm still not convinced it would be all *that* difficult. > Seems to me it would be semantically equivalent to > renaming the inner variable and adding a finally clause > to unbind it. Is there something I'm missing? An inner scope should shadow rather than unbinding. Ideally: spam = "Initial spam" try: 1/0 except Exception as spam: assert isinstance(spam, Exception) assert [spam for spam in ["List Comp"]][0] == "List Comp" with open("test","w") as spam: assert hasattr(spam,"write") assert (lambda spam: spam)("Function") == "Function" assert spam == "Initial spam" Currently, the list comp and lambda work that way. The exception will unbind the name, not changing the scope at all but preventing refloop after exit. The 'with... as' works in the same scope and leaves the name bound, which is a bit surprising in some cases (you usually end up with a closed file object, or whatever it be, lurking around). A clean inner-scope concept would solve all of this. The lambda would still be a closure, because you can pass that around. All the others would be inner scopes - shadowing cleanly and then disappearing. ChrisA From solipsis at pitrou.net Sat Feb 22 10:58:29 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Feb 2014 10:58:29 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> Message-ID: <20140222105829.376aab18@fsol> On Sat, 22 Feb 2014 20:29:27 +1100 Chris Angelico wrote: > > Which means that, fundamentally, EAFP is the way to do it. So if PEP > 463 expressions had existed from the beginning, hasattr() probably > wouldn't have been written - people would just use an > except-expression instead. Really? hasattr() is much easier to write than the corresponding except-expression. Regards Antoine. From thomas at python.org Sat Feb 22 11:08:57 2014 From: thomas at python.org (Thomas Wouters) Date: Sat, 22 Feb 2014 02:08:57 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <5307CDDE.8090106@canterbury.ac.nz> Message-ID: On Fri, Feb 21, 2014 at 7:46 PM, Chris Angelico wrote: > On Sat, Feb 22, 2014 at 9:06 AM, Greg Ewing > wrote: > > Nick Coghlan wrote: > >> > >> On 21 February 2014 13:15, Chris Angelico wrote: > >> > >>> Generator expressions require parentheses, unless they would be > >>> strictly redundant. Ambiguities with except expressions could be > >>> resolved in the same way, forcing nested except-in-except trees to be > >>> correctly parenthesized > >> > >> > >> I'd like to make the case that the PEP should adopt this as its > >> default position. > > > > > > I generally agree, but I'd like to point out that this > > doesn't necessarily mean making the parenthesizing rules as > > strict as they are for generator expressions. > > > > The starting point for genexps is that the parens are part of > > the syntax, the same way that square brackets are part of > > the syntax of a list comprehension; we only allow them to > > be omitted in very special circumstances. > > > > On the other hand, I don't think there's any harm in allowing > > an except expression to stand on its own when there is no > > risk of ambiguity, e.g. > > > > foo = things[i] except IndexError: None > > > > should be allowed, just as we allow > > > > x = a if b else c > > > > and don't require > > > > x = (a if b else c) > > I'm inclined to agree with you. I fought against the mandated parens > for a long time. Would be happy to un-mandate them, as long as there's > no way for there to be ambiguity. Is CPython able to make an operator > non-associative? That is, to allow these: > > value = expr except Exception: default > value = (expr except Exception: default) except Exception: default > value = expr except Exception: (default except Exception: default) > > but not this: > > value = expr except Exception: default except Exception: default > > ? Yes, although how to do it depends on what else 'default' and 'expr' can be. The simple answer is to make 'expr' and 'default' be subsets of expressions that don't include the 'except' expression themselves. Just like how in 'A if C else B', 'A' and 'C' do not include the 'if' syntax, but 'B' does: you need parentheses in '(a if c1 else b) if c2 else d' and 'a if (c1 if c3 else c2) else b' but not in 'a if c1 else b if c2 else d'. However, the question becomes what to do with 'lambda' and 'if-else'. Which of these should be valid (and mean what it means with the parentheses, which is how the PEP currently suggests it should be; ones without parenthesized alternatives are unambiguous, as far as I can tell.) 1. make_adder(x) except TypeError: lambda y: y + 0 2. lambda x: x + 0 except TypeError: 1 -> lambda x: (x + 0 except TypeError: 1) 3. A if f(A) except TypeError: C else B 4. f(A1) except TypeError: A2 if C else B -> f(A1) except TypeError: (A2 if C else B) 5. f(A1) except TypeError if C else ValueError: f(A2) 6. A if C else f(B) except TypeError: B -> (A if C else f(B)) except TypeError: B 7. f(A) except E1(A) except E2(A): E2: E1 -> f(A) except (E1(A) except E2(A): E2): E1 8. f(A) or f(B) except TypeError: f(C) -> (f(A) or f(B)) except TypeError: f(C) 9. f(A) except TypeError: f(B) or f(C) -> f(A) except TypeError: (f(B) or f(C)) 10. A == f(A) except TypeError: B -> (A == f(A)) except TypeError: B 11. f(A) except TypeError: A == B -> f(A) except TypeError: (A == B) 12. A + B except TypeError: C -> (A + B) except TypeError: C 13. f(A) except TypeError: A + B -> f(A) except TypeError: (A + B) #6 in particular and to some extent #4, #8 and #10 look wrong to me, so if they'd be allowed without parentheses perhaps the precedence of if-expr and except should be more nuanced. ('between lambda and ifexpr' is not really a sensible way to describe precedence anyway, since despite the insistence of the reference manual, the precedence of 'lambda' and 'ifexpr' is at best 'mixed': the grammar actually tries to match ifexpr first, but it works the way we all expect because 'lambda' is not an infix operator.) It's easy to disallow most of these (allowing only their parenthesized versions), the question is where to draw the line :) Some of these require a bit more work in the grammar, duplicating some of the grammar nodes, but it shouldn't be too bad. However, disallowing any infix operators in the 'default' expression means it'll still be valid, just mean something different; #9, for example, would be parsed as '(f(A) except TypeError: f(B)) or f(C)' if the 'default' expression couldn't be an or-expression. Disallowing it in the 'expr' expression wouldn't have that effect without parentheses. (FWIW, I have a working patch without tests that allows all of these, I'll upload it tonight so people can play with it. Oh, and FWIW, currently I'm +0 on the idea, -0 on the specific syntax.) -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Feb 22 11:09:07 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 21:09:07 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222105829.376aab18@fsol> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> Message-ID: On Sat, Feb 22, 2014 at 8:58 PM, Antoine Pitrou wrote: > On Sat, 22 Feb 2014 20:29:27 +1100 > Chris Angelico wrote: >> >> Which means that, fundamentally, EAFP is the way to do it. So if PEP >> 463 expressions had existed from the beginning, hasattr() probably >> wouldn't have been written - people would just use an >> except-expression instead. > > Really? hasattr() is much easier to write than the corresponding > except-expression. But would it be sufficiently easier to justify the creation of a built-in? Imagine this were the other way around: we have except-expressions, but we don't have hasattr. Now someone comes onto python-ideas and says, "Wouldn't it be nice if we had a hasattr() function to tell us whether something has an attribute or not". Considering that hasattr can be easily implemented using an except-expression, it would be unlikely to be considered worthy of a built-in. ChrisA From solipsis at pitrou.net Sat Feb 22 11:17:56 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Feb 2014 11:17:56 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> Message-ID: <20140222111756.17bb8a4c@fsol> On Sat, 22 Feb 2014 21:09:07 +1100 Chris Angelico wrote: > On Sat, Feb 22, 2014 at 8:58 PM, Antoine Pitrou wrote: > > On Sat, 22 Feb 2014 20:29:27 +1100 > > Chris Angelico wrote: > >> > >> Which means that, fundamentally, EAFP is the way to do it. So if PEP > >> 463 expressions had existed from the beginning, hasattr() probably > >> wouldn't have been written - people would just use an > >> except-expression instead. > > > > Really? hasattr() is much easier to write than the corresponding > > except-expression. > > But would it be sufficiently easier to justify the creation of a > built-in? Well, can you propose the corresponding except-expression? Regards Antoine. From turnbull at sk.tsukuba.ac.jp Sat Feb 22 11:29:41 2014 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sat, 22 Feb 2014 19:29:41 +0900 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222102051.16b65fda@fsol> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> Message-ID: <87ppmfv3sa.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > Well, the only way to know that a key (or attribute) exists is to do > the lookup. What else would you suggest? Do the lookup at the C level (or whatever the implementation language is) and generate no exception, of course. That's what would make it possibly more efficient. From solipsis at pitrou.net Sat Feb 22 11:35:39 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Feb 2014 11:35:39 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <87ppmfv3sa.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <87ppmfv3sa.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1393065339.2290.4.camel@fsol> On sam., 2014-02-22 at 19:29 +0900, Stephen J. Turnbull wrote: > Antoine Pitrou writes: > > > Well, the only way to know that a key (or attribute) exists is to do > > the lookup. What else would you suggest? > > Do the lookup at the C level (or whatever the implementation language > is) and generate no exception, of course. That's what would make it > possibly more efficient. Let's see: - hasattr() does the lookup at the C level, and silences the AttributeError - dict.get() does the lookup at the C level, and doesn't generate an exception So apart from the minor inefficiency of generating and silencing the AttributeError, those functions already do what you suggest. Regards Antoine. From greg.ewing at canterbury.ac.nz Sat Feb 22 12:01:52 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 23 Feb 2014 00:01:52 +1300 Subject: [Python-Dev] Tangent on class level scoping rules (was Re: PEP 463: Exception-catching expressions) In-Reply-To: References: <5307E7E8.6060001@canterbury.ac.nz> Message-ID: <530883A0.5050309@canterbury.ac.nz> Chris Angelico wrote: > On Sat, Feb 22, 2014 at 10:57 AM, Greg Ewing > wrote: > >>I'm still not convinced it would be all *that* difficult. >>Seems to me it would be semantically equivalent to >>renaming the inner variable and adding a finally clause >>to unbind it. Is there something I'm missing? > > An inner scope should shadow rather than unbinding. It would. The name being unbound would be the renamed inner one, not the one being shadowed. -- Greg From rosuav at gmail.com Sat Feb 22 12:13:58 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 22:13:58 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222111756.17bb8a4c@fsol> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> <20140222111756.17bb8a4c@fsol> Message-ID: On Sat, Feb 22, 2014 at 9:17 PM, Antoine Pitrou wrote: > On Sat, 22 Feb 2014 21:09:07 +1100 > Chris Angelico wrote: >> On Sat, Feb 22, 2014 at 8:58 PM, Antoine Pitrou wrote: >> > On Sat, 22 Feb 2014 20:29:27 +1100 >> > Chris Angelico wrote: >> >> >> >> Which means that, fundamentally, EAFP is the way to do it. So if PEP >> >> 463 expressions had existed from the beginning, hasattr() probably >> >> wouldn't have been written - people would just use an >> >> except-expression instead. >> > >> > Really? hasattr() is much easier to write than the corresponding >> > except-expression. >> >> But would it be sufficiently easier to justify the creation of a >> built-in? > > Well, can you propose the corresponding except-expression? It's hard to do hasattr itself without something messy - the best I can come up with is this: hasattr(x,"y") <-> (x.y or True except AttributeError: False) but the bulk of uses of it are immediately before attempting to use the attribute. Many require multiple statements, so they'd be better done as a full try/except: cpython/python-gdb.py:1392: if not hasattr(self._gdbframe, 'select'): print ('Unable to select frame: ' 'this build of gdb does not expose a gdb.Frame.select method') return False self._gdbframe.select() return True becomes try: self._gdbframe.select() return True except AttributeError: print ('Unable to select frame: ' 'this build of gdb does not expose a gdb.Frame.select method') return False but others are clearly expressions in disguise: Lib/aifc.py:882: if hasattr(f, 'mode'): mode = f.mode else: mode = 'rb' becomes mode = (f.mode except AttributeError: 'rb') (In fact, I'm adding that one to the PEP's examples section.) Lib/cgi.py:145: if hasattr(fp,'encoding'): encoding = fp.encoding else: encoding = 'latin-1' becomes encoding = (fp.encoding except AttributeError: 'latin-1') Some could be done either way. If hasattr didn't exist, then this: Lib/argparse.py:597: if hasattr(params[name], '__name__'): params[name] = params[name].__name__ could be written instead as: params[name] = (params[name].__name__ except AttributeError: params[name]) which is similar length and doesn't require a built-in. Some are fairly clearly asking to be done as try/except, irrespective of this PEP: Lib/decimal.py:449: if hasattr(threading.current_thread(), '__decimal_context__'): del threading.current_thread().__decimal_context__ becomes try: del threading.current_thread().__decimal_context__ except AttributeError: pass (also ibid:476) Some are a bit of a mixture. Lib/dis.py:40: if hasattr(x, '__func__'): # Method x = x.__func__ if hasattr(x, '__code__'): # Function x = x.__code__ if hasattr(x, '__dict__'): # Class or module ... lots more code ... Could be done as try/except; first part could be done cleanly as an expression. This one's not quite as clean, but if hasattr didn't exist, this could be done either of two ways: Lib/fileinput.py:342: if hasattr(os, 'O_BINARY'): mode |= os.O_BINARY As an expression: mode |= (os.O_BINARY except AttributeError: 0) Or as a statement: try: mode |= os.O_BINARY except AttributeError: pass This one definitely would want to be changed, and is also going in the PEP: Lib/inspect.py:1350: return sys._getframe(1) if hasattr(sys, "_getframe") else None becomes return (sys._getframe(1) except AttributeError: None) Lib/ntpath.py:558: # Win9x family and earlier have no Unicode filename support. supports_unicode_filenames = (hasattr(sys, "getwindowsversion") and sys.getwindowsversion()[3] >= 2) becomes supports_unicode_filenames = (sys.getwindowsversion()[3] >= 2 except AttributeError: False) Another ternary-if LBYL that could become an expression-except EAFP: Lib/pdb.py:745: globs = self.curframe.f_globals if hasattr(self, 'curframe') else None becomes globs = (self.curframe.f_globals except AttributeError: None) although that will return None if self.curframe has no f_globals. Another nice easy one: Lib/pickletools.py:2227: if hasattr(data, "tell"): getpos = data.tell else: getpos = lambda: None becomes getpos = (data.tell except AttributeError: lambda: None) I could continue this theme, but behold, as Rose Maybud said, I have said enough. There are definitely cases where a local hasattr function could be useful, but if the code were already written to use try/except or an except-expression, there aren't many that would justify the creation of a builtin. ChrisA From rosuav at gmail.com Sat Feb 22 12:17:37 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 22:17:37 +1100 Subject: [Python-Dev] Tangent on class level scoping rules (was Re: PEP 463: Exception-catching expressions) In-Reply-To: <530883A0.5050309@canterbury.ac.nz> References: <5307E7E8.6060001@canterbury.ac.nz> <530883A0.5050309@canterbury.ac.nz> Message-ID: On Sat, Feb 22, 2014 at 10:01 PM, Greg Ewing wrote: > Chris Angelico wrote: >> >> On Sat, Feb 22, 2014 at 10:57 AM, Greg Ewing >> wrote: >> >>> I'm still not convinced it would be all *that* difficult. >>> Seems to me it would be semantically equivalent to >>> renaming the inner variable and adding a finally clause >>> to unbind it. Is there something I'm missing? >> >> >> An inner scope should shadow rather than unbinding. > > > It would. The name being unbound would be the renamed > inner one, not the one being shadowed. If the whole inner scope is disappearing, then there's no need to unbind in a finally clause. The current behaviour of try/except is exactly what you're describing, with no subscope: Python 3.4.0rc1+ (default:9f76adbac8b7, Feb 15 2014, 20:19:30) [GCC 4.7.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> e = 2.71828 >>> try: 1/0 ... except ZeroDivisionError as e: pass ... >>> e Traceback (most recent call last): File "", line 1, in NameError: name 'e' is not defined It's been unbound from the parent scope. It's not shadowed, it's actually overwritten. ChrisA From solipsis at pitrou.net Sat Feb 22 12:27:17 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Feb 2014 12:27:17 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> <20140222111756.17bb8a4c@fsol> Message-ID: <20140222122717.6b7c9785@fsol> On Sat, 22 Feb 2014 22:13:58 +1100 Chris Angelico wrote: > > > > Well, can you propose the corresponding except-expression? > > It's hard to do hasattr itself without something messy - the best I > can come up with is this: > > hasattr(x,"y") <-> (x.y or True except AttributeError: False) But it's not the same. hasattr() returns a boolean, not an arbitrary value. > try: > self._gdbframe.select() > return True > except AttributeError: > print ('Unable to select frame: ' > 'this build of gdb does not expose a > gdb.Frame.select method') > return False But then you can't distinguish between a missing "select" method and an AttributeError raised by self._gdbframe.select() itself. > but others are clearly expressions in disguise: > > Lib/aifc.py:882: > if hasattr(f, 'mode'): > mode = f.mode > else: > mode = 'rb' > becomes > mode = (f.mode except AttributeError: 'rb') Not significantly less wordy. Note you can already write: mode = getattr(f, 'mode', 'rb') which is more concise. > (In fact, I'm adding that one to the PEP's examples section.) > > Lib/cgi.py:145: > if hasattr(fp,'encoding'): > encoding = fp.encoding > else: > encoding = 'latin-1' > becomes > encoding = (fp.encoding except AttributeError: 'latin-1') Same here: encoding = getattr(fp, 'encoding', 'latin-1') > Some could be done either way. If hasattr didn't exist, then this: > > Lib/argparse.py:597: > if hasattr(params[name], '__name__'): > params[name] = params[name].__name__ > > could be written instead as: > params[name] = (params[name].__name__ > except AttributeError: params[name]) This makes a useless assignment if the attribute doesn't exist; it also spans a single expression over two lines instead of having two simple statements. It's definitely worse IMO. > Some are fairly clearly asking to be done as try/except, irrespective > of this PEP: > > Lib/decimal.py:449: > if hasattr(threading.current_thread(), '__decimal_context__'): > del threading.current_thread().__decimal_context__ > becomes > try: del threading.current_thread().__decimal_context__ > except AttributeError: pass May hide a bug if threading.current_thread doesn't exist. > Lib/inspect.py:1350: > return sys._getframe(1) if hasattr(sys, "_getframe") else None > becomes > return (sys._getframe(1) except AttributeError: None) May hide a bug if sys._getframe(1) itself raises AttributeError. (etc.) > I could continue this theme, but behold, as Rose Maybud said, I have > said enough. Yeah, none of these examples makes a convincing case that hasattr() is useless, IMO. Regards Antoine. From rosuav at gmail.com Sat Feb 22 12:44:06 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 22:44:06 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222122717.6b7c9785@fsol> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> <20140222111756.17bb8a4c@fsol> <20140222122717.6b7c9785@fsol> Message-ID: On Sat, Feb 22, 2014 at 10:27 PM, Antoine Pitrou wrote: > Yeah, none of these examples makes a convincing case that hasattr() is > useless, IMO. I'm not trying to make the case that it's useless. I'm trying to show that, if it didn't exist, all of these would be written some other way, and the case for its creation would not be terribly strong. It's definitely of value (and as you've shown in some of those cases, its proper use can narrow the exception-catching scope - a good thing), but not enough to be worth blessing with a built-in function. ChrisA From stephen at xemacs.org Sat Feb 22 12:54:22 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 22 Feb 2014 20:54:22 +0900 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <1393065339.2290.4.camel@fsol> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <87ppmfv3sa.fsf@uwakimon.sk.tsukuba.ac.jp> <1393065339.2290.4.camel@fsol> Message-ID: <87ob1zuzv5.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > On sam., 2014-02-22 at 19:29 +0900, Stephen J. Turnbull wrote: > > Antoine Pitrou writes: > > > > > Well, the only way to know that a key (or attribute) exists is to do > > > the lookup. What else would you suggest? > > > > Do the lookup at the C level (or whatever the implementation language > > is) and generate no exception, of course. That's what would make it > > possibly more efficient. > > Let's see: > - hasattr() does the lookup at the C level, and silences the > AttributeError > - dict.get() does the lookup at the C level, and doesn't generate an > exception > > So apart from the minor inefficiency of generating and silencing the > AttributeError, those functions already do what you suggest. But that's precisely the inefficiency I'm referring to. From solipsis at pitrou.net Sat Feb 22 12:59:01 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Feb 2014 12:59:01 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <87ob1zuzv5.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <87ppmfv3sa.fsf@uwakimon.sk.tsukuba.ac.jp> <1393065339.2290.4.camel@fsol> <87ob1zuzv5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1393070341.2290.7.camel@fsol> On sam., 2014-02-22 at 20:54 +0900, Stephen J. Turnbull wrote: > Antoine Pitrou writes: > > On sam., 2014-02-22 at 19:29 +0900, Stephen J. Turnbull wrote: > > > Antoine Pitrou writes: > > > > > > > Well, the only way to know that a key (or attribute) exists is to do > > > > the lookup. What else would you suggest? > > > > > > Do the lookup at the C level (or whatever the implementation language > > > is) and generate no exception, of course. That's what would make it > > > possibly more efficient. > > > > Let's see: > > - hasattr() does the lookup at the C level, and silences the > > AttributeError > > - dict.get() does the lookup at the C level, and doesn't generate an > > exception > > > > So apart from the minor inefficiency of generating and silencing the > > AttributeError, those functions already do what you suggest. > > But that's precisely the inefficiency I'm referring to. Sure, but complaining about inefficiencies without asserting their significance is not very useful. Regards Antoine. From stephen at xemacs.org Sat Feb 22 13:14:05 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 22 Feb 2014 21:14:05 +0900 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222122717.6b7c9785@fsol> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> <20140222111756.17bb8a4c@fsol> <20140222122717.6b7c9785@fsol> Message-ID: <87lhx3uyya.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > On Sat, 22 Feb 2014 22:13:58 +1100 > Chris Angelico wrote: > > hasattr(x,"y") <-> (x.y or True except AttributeError: False) > But it's not the same. hasattr() returns a boolean, not an arbitrary > value. I think he meant hasattr(x,"y") <-> (x.y and True except AttributeError: False) From eric at trueblade.com Sat Feb 22 13:22:17 2014 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 22 Feb 2014 07:22:17 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222122717.6b7c9785@fsol> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> <20140222111756.17bb8a4c@fsol> <20140222122717.6b7c9785@fsol> Message-ID: <53089679.6000307@trueblade.com> On 2/22/2014 6:27 AM, Antoine Pitrou wrote: > On Sat, 22 Feb 2014 22:13:58 +1100 > Chris Angelico wrote: >> Lib/inspect.py:1350: >> return sys._getframe(1) if hasattr(sys, "_getframe") else None >> becomes >> return (sys._getframe(1) except AttributeError: None) > > May hide a bug if sys._getframe(1) itself raises AttributeError. return (sys._getframe except AttributeError: lambda i: None)(1) Assuming I have the syntax correct, and the bindings work this way, of course. From rosuav at gmail.com Sat Feb 22 13:49:37 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 22 Feb 2014 23:49:37 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <87lhx3uyya.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> <20140222111756.17bb8a4c@fsol> <20140222122717.6b7c9785@fsol> <87lhx3uyya.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Sat, Feb 22, 2014 at 11:14 PM, Stephen J. Turnbull wrote: > Antoine Pitrou writes: > > On Sat, 22 Feb 2014 22:13:58 +1100 > > Chris Angelico wrote: > > > > hasattr(x,"y") <-> (x.y or True except AttributeError: False) > > > But it's not the same. hasattr() returns a boolean, not an arbitrary > > value. > > I think he meant > > hasattr(x,"y") <-> (x.y and True except AttributeError: False) No, I meant 'or' to ensure that an attribute holding a false value doesn't come up false. But if you really want a boolean, just wrap it up in bool(). My main point, though, was that most usage of hasattr is probing just before something gets used - something like this: if hasattr(obj, "attr"): blah blah obj.attr else: maybe use a default or maybe do nothing Some cases could become except-expressions; others could become try/except statements. In each case, the loss would be small. I'm not saying hasattr should be removed, just that it wouldn't have a strong case if it didn't already exist. ChrisA From ncoghlan at gmail.com Sat Feb 22 15:42:06 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Feb 2014 00:42:06 +1000 Subject: [Python-Dev] Tangent on class level scoping rules (was Re: PEP 463: Exception-catching expressions) In-Reply-To: <5307E7E8.6060001@canterbury.ac.nz> References: <5307E7E8.6060001@canterbury.ac.nz> Message-ID: On 22 Feb 2014 09:59, "Greg Ewing" wrote: > > Nick Coghlan wrote: >> >> As Chris later noted, you likely *could* still implement expression >> local name binding for an except expression without a full closure, it >> would just be rather difficult. > > > I'm still not convinced it would be all *that* difficult. > Seems to me it would be semantically equivalent to > renaming the inner variable and adding a finally clause > to unbind it. Is there something I'm missing? Dealing with references from nested closures is the hard part. It's not impossible to solve, but would require introducing a new kind of scope not previously seen in Python, which is a rather dubious suggestion when the existing closure semantics can typically do the job. However, we're getting off topic for python-dev. Cheers, Nick. > > -- > Greg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Feb 22 16:29:15 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Feb 2014 01:29:15 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <87lhx3uyya.fsf@uwakimon.sk.tsukuba.ac.jp> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> <20140222111756.17bb8a4c@fsol> <20140222122717.6b7c9785@fsol> <87lhx3uyya.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 22 Feb 2014 22:15, "Stephen J. Turnbull" wrote: > > Antoine Pitrou writes: > > On Sat, 22 Feb 2014 22:13:58 +1100 > > Chris Angelico wrote: > > > > hasattr(x,"y") <-> (x.y or True except AttributeError: False) > > > But it's not the same. hasattr() returns a boolean, not an arbitrary > > value. > > I think he meant > > hasattr(x,"y") <-> (x.y and True except AttributeError: False) With PEP 463, the explicit equivalent of hasattr() would be something like : hasattr(x,"y") <-> (bool(x.y) or True except AttributeError: False) The version Chris came up with was close, but as Antoine noted, didn't ensure the result was always exactly True or False. The translation isn't simple because we don't allow an "else" clause on the except expression (and I agree with this limitation), so the first expression needs to be one that will *evaluate* x.y, but ensure the result of the expression is True if no exception is thrown. However, as Chris noted in his reply, there are still cases where using hasattr makes more sense, so the fact it *can* be written as an except expression instead is a curiosity rather than anything with deep practical implications. Cheers, Nick. P.S. The fully general variant of "else" emulation under PEP 463: ((bool(EXPR) and False) or NO_EXC_RESULT except EXC: EXC_RESULT) Note: never actually use this, it's effectively unreadable ;) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From navneet35371 at gmail.com Sat Feb 22 07:56:53 2014 From: navneet35371 at gmail.com (NAVNEET SUMAN) Date: Sat, 22 Feb 2014 12:26:53 +0530 Subject: [Python-Dev] GSOC 2014 In-Reply-To: References: Message-ID: Thank you for your recommendations Jessica. I am looking forward to work with python this gsoc. On 22 February 2014 08:39, Jessica McKellar wrote: > Hi Navneet, > > > This is my first year in gsoc. I have been working with python and > django > > from quite a time. One of the Gsoc proposed ideas drew my attention and i > > would surely like to work on that. > > I would like to work for creating a extension for idle to integrate > > PEP8, what are the prerequisites . I have been studying the PEP8 code > past > > a week. This might be useful for a quick start to this project. Please > > guide. > > Thanks for your interest in CPython for Google Summer of Code! > > Mentor organizations haven't been selected yet (the official mentor > list will go out on February 24th), and we are still finalizing our > project list. > > In the meantime, I recommend: > > 1. Joining http://pythonmentors.com/ > 2. Reading through the developers guide: http://docs.python.org/devguide/ > 3. Picking and working on a ticket to get used to the workflow: > http://bugs.python.org/ > > In particular, it will be important to have contributed some patches > to CPython before you apply, so I recommend focusing on that for the > next couple of weeks. If you have questions while finding bugs and > preparing patches, please don't hesitate to ask on the > pythonmentors.com mailing list. > > Regards, > -Jessica > -- Thanks and Regards, Navneet suman, IIIT-Allahabad +91-9305612151 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Feb 22 17:30:42 2014 From: brett at python.org (Brett Cannon) Date: Sat, 22 Feb 2014 11:30:42 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> <20140222111756.17bb8a4c@fsol> <20140222122717.6b7c9785@fsol> <87lhx3uyya.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Fri, Feb 21, 2014 at 8:41 PM, Greg Ewing wrote: > Ethan Furman wrote: > >> On 02/21/2014 03:29 PM, Greg Ewing wrote: >> >> value = lst[2] except "No value" if IndexError >>> >> >> It does read nicely, and is fine for the single, non-nested, case (which >> is probably the vast majority), but how would it handle nested exceptions? >> > > Hmmm, probably not very well, unless we define > > a except b if E1 except c if E2 > > to mean > > a except (b except c if E2) if E1 > > If E1 == E2, that could perhaps be abbreviated to > > a except b except c if E > > Or we could just decide that the nested case is going > to be so rare it's not worth worrying about. +1 on not caring. Keep the expression for simple, obvious cases of a single exception type and fall back to the statement for fancier use (just like listcomps). The focus should be ease of expressiveness for a common pattern, not trying to convert tons of try/except statements into an expression just because we can. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Feb 22 17:36:51 2014 From: brett at python.org (Brett Cannon) Date: Sat, 22 Feb 2014 11:36:51 -0500 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <20140222101317.058e10c2@fsol> References: <20140221145205.0b8b670d@fsol> <20140221183250.2b44f860@fsol> <20140222101317.058e10c2@fsol> Message-ID: On Sat, Feb 22, 2014 at 4:13 AM, Antoine Pitrou wrote: > On Fri, 21 Feb 2014 09:37:29 -0800 > Guido van Rossum wrote: > > I'm put off by the ':' syntax myself (it looks to me as if someone > forgot a > > newline somewhere) but 'then' feels even weirder (it's been hard-coded in > > my brain as meaning the first branch of an 'if'). > > Would 'else' work rather than 'then'? > thing = stuff['key'] except KeyError else None That reads to me like the exception was silenced and only if there is no exception the None is returned, just like an 'else' clause on a 'try' statement. I personally don't mind the 'then' as my brain has been hard-coded to mean "the first branch of a statement" so it's looser than being explicitly associated with 'if' but with any multi-clause statement. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ja.py at farowl.co.uk Sat Feb 22 21:28:01 2014 From: ja.py at farowl.co.uk (Jeff Allen) Date: Sat, 22 Feb 2014 20:28:01 +0000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <20140221183250.2b44f860@fsol> <20140222101317.058e10c2@fsol> Message-ID: <53090851.8000500@farowl.co.uk> On 22/02/2014 16:36, Brett Cannon wrote: > > On Sat, Feb 22, 2014 at 4:13 AM, Antoine Pitrou > wrote: > > On Fri, 21 Feb 2014 09:37:29 -0800 > Guido van Rossum > wrote: > > I'm put off by the ':' syntax myself (it looks to me as if > someone forgot a > > newline somewhere) but 'then' feels even weirder (it's been > hard-coded in > > my brain as meaning the first branch of an 'if'). > > Would 'else' work rather than 'then'? > > > thing = stuff['key'] except KeyError else None > > That reads to me like the exception was silenced and only if there is > no exception the None is returned, just like an 'else' clause on a > 'try' statement. > > I personally don't mind the 'then' as my brain has been hard-coded to > mean "the first branch of a statement" so it's looser than being > explicitly associated with 'if' but with any multi-clause statement. > I read *except* as 'except if', and *:* as 'then' (often), so the main proposal reads naturally to me. I'm surprised to find others don't also, as that's the (only?) pronunciation that makes the familiar if-else and try-except constructs approximate English. Isn't adding a new keyword (*then*) likely to be a big deal? There is the odd example of its use as an identifier, just in our test code: http://hg.python.org/cpython/file/0695e465affe/Lib/test/test_epoll.py#l168 http://hg.python.org/cpython/file/0695e465affe/Lib/test/test_xmlrpc.py#l310 Jeff Allen -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Sat Feb 22 22:26:32 2014 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Sun, 23 Feb 2014 08:26:32 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> <20140222111756.17bb8a4c@fsol> <20140222122717.6b7c9785@fsol> <87lhx3uyya.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 23 February 2014 02:29, Nick Coghlan wrote: > On 22 Feb 2014 22:15, "Stephen J. Turnbull" wrote: > > Antoine Pitrou writes: > > > Chris Angelico wrote: > > > > hasattr(x,"y") <-> (x.y or True except AttributeError: False) > > > But it's not the same. hasattr() returns a boolean, not an arbitrary > > > value. > > I think he meant > > hasattr(x,"y") <-> (x.y and True except AttributeError: False) > > With PEP 463, the explicit equivalent of hasattr() would be something like > : > > hasattr(x,"y") <-> (bool(x.y) or True except AttributeError: False) > That would work, but I think I'd prefer: hasattr(x,"y") <-> bool(x.y or True except AttributeError: False) Makes it clearer IMO that the entire expression will always return a boolean. If exception expressions already existed in the language, I would think there would be a strong argument for a library function hasattr(), but probably not a builtin. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From Nikolaus at rath.org Sat Feb 22 21:29:23 2014 From: Nikolaus at rath.org (Nikolaus Rath) Date: Sat, 22 Feb 2014 12:29:23 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <87sirbvdmp.fsf@uwakimon.sk.tsukuba.ac.jp> (Stephen J. Turnbull's message of "Sat, 22 Feb 2014 15:57:02 +0900") References: <5307CDDE.8090106@canterbury.ac.nz> <530822CA.6030609@stoneleaf.us> <87sirbvdmp.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87bnxy99i4.fsf@vostro.rath.org> "Stephen J. Turnbull" writes: > Ethan Furman writes: > > On 02/21/2014 07:46 PM, Chris Angelico wrote: > > > > > > but not this: > > > > > > value = expr except Exception: default except Exception: default > > > > This should be the way it works. Nothing is gained in readability > > by turning a try with multiple except statements into an > > expression. > > Examples have been given several times. In general, if 'expr' is a > function call, it may well have a couple of different ways to fail > which imply different default values. > > interpolable = func(key) except TypeError: "not a string: %s" % key \ > except KeyError: "no such key: %s" % key > print("Some message that refers to '%s' % interpolable") > > versus > > try: > interpolable = func(key) > except TypeError: > interpolable = "not a string: %s" % key > except KeyError: > interpolable = "no such key: %s" % key > print("Some message that refers to '%s' % interpolable") I think the following suggestion from elsewhere in the thread would look even better in this case: interpolable = func(key) except (TypeError: "not a string: %s" % key, KeyError: "no such key: %s" % key) print("Some message that refers to '%s' % interpolable") It does not require the backslash, it is shorter, and it can still be chained: interpolable = func(key) except (TypeError: "not a string: %s" % key, KeyError: defaults[key] except (KeyError: "no such key: %s" % key)) print("Some message that refers to '%s' % interpolable") Best, -Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ?Time flies like an arrow, fruit flies like a Banana.? From stephen at xemacs.org Sun Feb 23 00:52:20 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 23 Feb 2014 08:52:20 +0900 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <1393070341.2290.7.camel@fsol> References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <87ppmfv3sa.fsf@uwakimon.sk.tsukuba.ac.jp> <1393065339.2290.4.camel@fsol> <87ob1zuzv5.fsf@uwakimon.sk.tsukuba.ac.jp> <1393070341.2290.7.camel@fsol> Message-ID: <87ios6vh6z.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > Sure, but complaining about inefficiencies without asserting their > significance is not very useful. Since you completely missed the point of my post, I'll explain. I was in no way complaining about inefficiencies. My point was precisely the opposite: to the extent that most 'get' methods would be implemented in Python, even the minor inefficiency of creating and suppressing a useless exception can't be avoided with the LBYL of a get method, because haveattr itself is implemented by generating and suppressing an exception. I know that it's not a big deal[1], but it did help swing me to positive on this PEP. Footnotes: [1] If it were, somebody would have reimplemented haveattr to avoid generating an exception. From thomas at python.org Sun Feb 23 01:00:59 2014 From: thomas at python.org (Thomas Wouters) Date: Sat, 22 Feb 2014 16:00:59 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <5307CDDE.8090106@canterbury.ac.nz> Message-ID: On Sat, Feb 22, 2014 at 2:08 AM, Thomas Wouters wrote: > > > (FWIW, I have a working patch without tests that allows all of these, I'll > upload it tonight so people can play with it. Oh, and FWIW, currently I'm > +0 on the idea, -0 on the specific syntax.) > http://bugs.python.org/issue20739 is the patch. -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Feb 23 01:09:36 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Feb 2014 11:09:36 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <5307CDDE.8090106@canterbury.ac.nz> Message-ID: On Sun, Feb 23, 2014 at 11:00 AM, Thomas Wouters wrote: > On Sat, Feb 22, 2014 at 2:08 AM, Thomas Wouters wrote: >> >> (FWIW, I have a working patch without tests that allows all of these, I'll >> upload it tonight so people can play with it. Oh, and FWIW, currently I'm +0 >> on the idea, -0 on the specific syntax.) > > > http://bugs.python.org/issue20739 is the patch. Thanks! You make a comment about precedence. When I wrote that up, it was basically just "that seems about right"; whether it's equal to lambda, equal to if/else, above both, below both, or in between, is free to be tweaked according to what makes sense. Nobody has to date discussed the exact precedence order, so feel free to tweak it for the benefit of implementation. ChrisA From python at mrabarnett.plus.com Sun Feb 23 02:11:45 2014 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 23 Feb 2014 01:11:45 +0000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <5307CDDE.8090106@canterbury.ac.nz> Message-ID: <53094AD1.9090805@mrabarnett.plus.com> On 2014-02-23 00:09, Chris Angelico wrote: > On Sun, Feb 23, 2014 at 11:00 AM, Thomas Wouters > wrote: >> On Sat, Feb 22, 2014 at 2:08 AM, Thomas Wouters >> wrote: >>> >>> (FWIW, I have a working patch without tests that allows all of >>> these, I'll upload it tonight so people can play with it. Oh, and >>> FWIW, currently I'm +0 on the idea, -0 on the specific syntax.) >> >> >> http://bugs.python.org/issue20739 is the patch. > > Thanks! > > You make a comment about precedence. When I wrote that up, it was > basically just "that seems about right"; whether it's equal to > lambda, equal to if/else, above both, below both, or in between, is > free to be tweaked according to what makes sense. Nobody has to date > discussed the exact precedence order, so feel free to tweak it for > the benefit of implementation. > My feeling is that catching exceptions should have a lower precedence than the other parts of an expression, but higher than comma, so: A if C else B except E: D would be parsed as: (A if C else B) except E: D I think that's because it's kind of replacing: try: _ = expr except E: _ = D with the try..except enclosing the expression. From ncoghlan at gmail.com Sun Feb 23 02:29:16 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Feb 2014 11:29:16 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <53094AD1.9090805@mrabarnett.plus.com> References: <5307CDDE.8090106@canterbury.ac.nz> <53094AD1.9090805@mrabarnett.plus.com> Message-ID: On 23 February 2014 11:11, MRAB wrote: > On 2014-02-23 00:09, Chris Angelico wrote: >> On Sun, Feb 23, 2014 at 11:00 AM, Thomas Wouters >> wrote: >>> On Sat, Feb 22, 2014 at 2:08 AM, Thomas Wouters >>> wrote: >>>> >>>> (FWIW, I have a working patch without tests that allows all of >>>> these, I'll upload it tonight so people can play with it. Oh, and >>>> FWIW, currently I'm +0 on the idea, -0 on the specific syntax.) >>> >>> >>> http://bugs.python.org/issue20739 is the patch. >> >> Thanks! >> >> You make a comment about precedence. When I wrote that up, it was >> basically just "that seems about right"; whether it's equal to >> lambda, equal to if/else, above both, below both, or in between, is >> free to be tweaked according to what makes sense. Nobody has to date >> discussed the exact precedence order, so feel free to tweak it for >> the benefit of implementation. >> > My feeling is that catching exceptions should have a lower precedence > than the other parts of an expression, but higher than comma, so: > > A if C else B except E: D > > would be parsed as: > > (A if C else B) except E: D > > I think that's because it's kind of replacing: > > try: > _ = expr > except E: > _ = D > > with the try..except enclosing the expression. Note that mandatory parentheses means we can duck the precedence question entirely, which I count as another point in favour of requiring them :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Sun Feb 23 02:56:50 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 22 Feb 2014 17:56:50 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 Message-ID: <53095562.8010304@stoneleaf.us> Greetings, all! I think I'm about ready to ask for pronouncement for this PEP, but I would like opinions on the Open Questions question so I can close it. :) Please let me know if anything else needs tweaking. ------------------------------------------------------ PEP: 461 Title: Adding % formatting to bytes and bytearray Version: $Revision$ Last-Modified: $Date$ Author: Ethan Furman Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2014-01-13 Python-Version: 3.5 Post-History: 2014-01-14, 2014-01-15, 2014-01-17, 2014-02-22 Resolution: Abstract ======== This PEP proposes adding % formatting operations similar to Python 2's ``str`` type to ``bytes`` and ``bytearray`` [1]_ [2]_. Rationale ========= While interpolation is usually thought of as a string operation, there are cases where interpolation on ``bytes`` or ``bytearrays`` make sense, and the work needed to make up for this missing functionality detracts from the overall readability of the code. Motivation ========== With Python 3 and the split between ``str`` and ``bytes``, one small but important area of programming became slightly more difficult, and much more painful -- wire format protocols [3]_. This area of programming is characterized by a mixture of binary data and ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in writing new wire format code, and in porting Python 2 wire format code. Overriding Principles ===================== In order to avoid the problems of auto-conversion and Unicode exceptions that could plague Python 2 code, :class:`str` objects will not be supported as interpolation values [4]_ [5]_. Proposed semantics for ``bytes`` and ``bytearray`` formatting ======================================= %-interpolation --------------- All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, ``%g``, etc.) will be supported, and will work as they do for str, including the padding, justification and other related modifiers. Example:: >>> b'%4x' % 10 b' a' >>> '%#4x' % 10 ' 0xa' >>> '%04X' % 10 '000A' ``%c`` will insert a single byte, either from an ``int`` in range(256), or from a ``bytes`` argument of length 1, not from a ``str``. Example: >>> b'%c' % 48 b'0' >>> b'%c' % b'a' b'a' ``%s`` is restricted in what it will accept:: - input type supports ``Py_buffer`` [6]_? use it to collect the necessary bytes - input type is something else? use its ``__bytes__`` method [7]_ ; if there isn't one, raise a ``TypeError`` Examples: >>> b'%s' % b'abc' b'abc' >>> b'%s' % 3.14 Traceback (most recent call last): ... TypeError: 3.14 has no __bytes__ method, use a numeric code instead >>> b'%s' % 'hello world!' Traceback (most recent call last): ... TypeError: 'hello world' has no __bytes__ method, perhaps you need to encode it? .. note:: Because the ``str`` type does not have a ``__bytes__`` method, attempts to directly use ``'a string'`` as a bytes interpolation value will raise an exception. To use strings they must be encoded or otherwise transformed into a ``bytes`` sequence:: 'a string'.encode('latin-1') ``%a`` will call :func:``ascii()`` on the interpolated value's :func:``repr()``. This is intended as a debugging aid, rather than something that should be used in production. Non-ascii values will be encoded to either ``\xnn`` or ``\unnnn`` representation. Unsupported codes ----------------- ``%r`` (which calls ``__repr__`` and returns a :class:`str`) is not supported. Proposed variations =================== It was suggested to let ``%s`` accept numbers, but since numbers have their own format codes this idea was discarded. It has been proposed to automatically use ``.encode('ascii','strict')`` for ``str`` arguments to ``%s``. - Rejected as this would lead to intermittent failures. Better to have the operation always fail so the trouble-spot can be correctly fixed. It has been proposed to have ``%s`` return the ascii-encoded repr when the value is a ``str`` (b'%s' % 'abc' --> b"'abc'"). - Rejected as this would lead to hard to debug failures far from the problem site. Better to have the operation always fail so the trouble-spot can be easily fixed. Originally this PEP also proposed adding format-style formatting, but it was decided that format and its related machinery were all strictly text (aka ``str``) based, and it was dropped. Various new special methods were proposed, such as ``__ascii__``, ``__format_bytes__``, etc.; such methods are not needed at this time, but can be visited again later if real-world use shows deficiencies with this solution. Open Questions ============== It has been suggested to use ``%b`` for bytes as well as ``%s``. - Pro: clearly says 'this is bytes'; should be used for new code. - Con: does not exist in Python 2.x, so we would have two ways of doing the same thing, ``%s`` and ``%b``, with no difference between them. Footnotes ========= .. [1] http://docs.python.org/2/library/stdtypes.html#string-formatting .. [2] neither string.Template, format, nor str.format are under consideration .. [3] https://mail.python.org/pipermail/python-dev/2014-January/131518.html .. [4] to use a str object in a bytes interpolation, encode it first .. [5] %c is not an exception as neither of its possible arguments are str .. [6] http://docs.python.org/3/c-api/buffer.html examples: ``memoryview``, ``array.array``, ``bytearray``, ``bytes`` .. [7] http://docs.python.org/3/reference/datamodel.html#object.__bytes__ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From ethan at stoneleaf.us Sun Feb 23 03:30:03 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 22 Feb 2014 18:30:03 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <53095562.8010304@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> Message-ID: <53095D2B.8060905@stoneleaf.us> Sorry, found a couple more comments in a different thread. Here's what I added: +Objections +========== + +The objections raised against this PEP were mainly variations on two themes:: + + - the ``bytes`` and ``bytearray`` types are for pure binary data, with no + assumptions about encodings + - offering %-interpolation that assumes an ASCII encoding will be an + attractive nuisance and lead us back to the problems of the Python 2 + ``str``/``unicode`` text model + +As was seen during the discussion, ``bytes`` and ``bytearray`` are also used +for mixed binary data and ASCII-compatible segments: file formats such as +``dbf`` and ``pdf``, network protocols such as ``ftp`` and ``email``, etc. + +``bytes`` and ``bytearray`` already have several methods which assume an ASCII +compatible encoding. ``upper()``, ``isalpha()``, and ``expandtabs()`` to name +just a few. %-interpolation, with its very restricted mini-language, will not +be any more of a nuisance than the already existing methdods. + + From rosuav at gmail.com Sun Feb 23 04:07:48 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 23 Feb 2014 14:07:48 +1100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <53095562.8010304@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> Message-ID: On Sun, Feb 23, 2014 at 12:56 PM, Ethan Furman wrote: > Open Questions > ============== > > It has been suggested to use ``%b`` for bytes as well as ``%s``. > > - Pro: clearly says 'this is bytes'; should be used for new code. > > - Con: does not exist in Python 2.x, so we would have two ways of doing > the > same thing, ``%s`` and ``%b``, with no difference between them. The fact that the format string is bytes says 'this is bytes'. Also the fact that you're explicitly encoding any strings used. I'm -1 on having %b as a redundant duplicate of %s. ChrisA From breamoreboy at yahoo.co.uk Sun Feb 23 04:29:17 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Sun, 23 Feb 2014 03:29:17 +0000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <53095D2B.8060905@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> <53095D2B.8060905@stoneleaf.us> Message-ID: On 23/02/2014 02:30, Ethan Furman wrote: > +be any more of a nuisance than the already existing methdods. Typo "methdods". -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From cs at zip.com.au Sun Feb 23 04:47:17 2014 From: cs at zip.com.au (Cameron Simpson) Date: Sun, 23 Feb 2014 14:47:17 +1100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <53095562.8010304@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> Message-ID: <20140223034717.GA94757@cskk.homeip.net> On 22Feb2014 17:56, Ethan Furman wrote: > Please let me know if anything else needs tweaking. > [...] > This area of programming is characterized by a mixture of binary data and > ASCII compatible segments of text (aka ASCII-encoded text). > [...] > %-interpolation > > All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, > ``%g``, etc.) will be supported, and will work as they do for str, including > the padding, justification and other related modifiers. I would like a single sentence here clarifying that the formatting of numeric values uses an ASCII encoding. It might be inferred from the earlier context, but I do not think it can be deduced and therefore I think it should be said outright. All the other formatting codes are quite explicit about how their arguments transform into bytes, but the numeric codes just quietly assume ASCII. The PEP should be blatant. Otherwise I think the PEP is clear and reasonable. Cheers, -- Cameron Simpson ASCII n s. [from the greek] Those people who, at certain times of the year, have no shadow at noon; such are the inhabitatants of the torrid zone. - 1837 copy of Johnson's Dictionary From ethan at stoneleaf.us Sun Feb 23 05:45:11 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 22 Feb 2014 20:45:11 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: <53095562.8010304@stoneleaf.us> <53095D2B.8060905@stoneleaf.us> Message-ID: <53097CD7.2040007@stoneleaf.us> On 02/22/2014 07:29 PM, Mark Lawrence wrote: > On 23/02/2014 02:30, Ethan Furman wrote: > >> +be any more of a nuisance than the already existing methdods. > > Typo "methdods". Thanks, fixed. -- ~Ethan~ From ethan at stoneleaf.us Sun Feb 23 05:48:04 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 22 Feb 2014 20:48:04 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <20140223034717.GA94757@cskk.homeip.net> References: <53095562.8010304@stoneleaf.us> <20140223034717.GA94757@cskk.homeip.net> Message-ID: <53097D84.2090201@stoneleaf.us> On 02/22/2014 07:47 PM, Cameron Simpson wrote: > On 22Feb2014 17:56, Ethan Furman wrote: >> Please let me know if anything else needs tweaking. >> [...] >> This area of programming is characterized by a mixture of binary data and >> ASCII compatible segments of text (aka ASCII-encoded text). >> [...] >> %-interpolation >> >> All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, >> ``%g``, etc.) will be supported, and will work as they do for str, including >> the padding, justification and other related modifiers. > > I would like a single sentence here clarifying that the formatting > of numeric values uses an ASCII encoding. How's this? All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, ``%g``, etc.) will be supported, and will work as they do for str, including the padding, justification and other related modifiers. The only difference will be that the results from these codes will be ASCII-encoded bytes, not unicode. -- ~Ethan~ From ncoghlan at gmail.com Sun Feb 23 07:31:33 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Feb 2014 16:31:33 +1000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <20140223034717.GA94757@cskk.homeip.net> References: <53095562.8010304@stoneleaf.us> <20140223034717.GA94757@cskk.homeip.net> Message-ID: On 23 February 2014 13:47, Cameron Simpson wrote: > On 22Feb2014 17:56, Ethan Furman wrote: >> Please let me know if anything else needs tweaking. >> [...] >> This area of programming is characterized by a mixture of binary data and >> ASCII compatible segments of text (aka ASCII-encoded text). >> [...] >> %-interpolation >> >> All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, >> ``%g``, etc.) will be supported, and will work as they do for str, including >> the padding, justification and other related modifiers. > > I would like a single sentence here clarifying that the formatting > of numeric values uses an ASCII encoding. > > It might be inferred from the earlier context, but I do not think > it can be deduced and therefore I think it should be said outright. > All the other formatting codes are quite explicit about how their > arguments transform into bytes, but the numeric codes just quietly > assume ASCII. The PEP should be blatant. Specifically, I believe the PEP should state that, for the numeric codes: b"%x" % val is equivalent to: b"%s" % (("%x" % val).encode("ascii")) The rationale for including them is the unreadability of the latter form :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 23 07:45:19 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Feb 2014 16:45:19 +1000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <53095562.8010304@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> Message-ID: Thanks Ethan, this mostly looks excellent. On 23 February 2014 11:56, Ethan Furman wrote: > ``%a`` will call :func:``ascii()`` on the interpolated value's > :func:``repr()``. > This is intended as a debugging aid, rather than something that should be > used > in production. Non-ascii values will be encoded to either ``\xnn`` or > ``\unnnn`` > representation. Is this really what is intended? It seems to me that what is needed is to just call ascii(), which is inherently based on repr(): >>> ascii(1) '1' >>> ascii("1") "'1'" >>> ascii(b"1") "b'1'" The current wording in the PEP effectively suggests invoking repr() twice, which is just weird: >>> ascii(repr(1)) "'1'" >>> ascii(repr("1")) '"\'1\'"' >>> ascii(repr(b"1")) '"b\'1\'"' And inconsistent with the meaning of %a in text interpolation: >>> ("%a" % 1).encode("ascii") b'1' > Open Questions > ============== > > It has been suggested to use ``%b`` for bytes as well as ``%s``. > > - Pro: clearly says 'this is bytes'; should be used for new code. > > - Con: does not exist in Python 2.x, so we would have two ways of doing > the > same thing, ``%s`` and ``%b``, with no difference between them. Another con is that using %b this way would be inconsistent with the "b" numeric format code that requests binary output: >>> format(2, "b") '10' >>> format(2, "#b") '0b10' So -1 for "%b" from me on both TOOWTDI and consistency grounds. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From larry at hastings.org Sun Feb 23 09:11:04 2014 From: larry at hastings.org (Larry Hastings) Date: Sun, 23 Feb 2014 02:11:04 -0600 Subject: [Python-Dev] One more cherry-pick request for 3.4.0 that I'd like a little public debate on In-Reply-To: <20140222102711.0a492256@fsol> References: <53085501.7070406@hastings.org> <20140222102711.0a492256@fsol> Message-ID: <5309AD18.7030206@hastings.org> On 02/22/2014 03:27 AM, Antoine Pitrou wrote: > On Sat, 22 Feb 2014 01:42:57 -0600 > Larry Hastings wrote: >> Victor has asked me to cherry-pick 180e4b678003: >> >> http://bugs.python.org/issue20320 (original issue) >> http://hg.python.org/cpython/rev/180e4b678003/ (checkin into trunk) >> http://bugs.python.org/issue20646 (cherry-pick request) >> >> This revision changes the rounding behavior of fractional-second >> timeouts for select.select and select.kqueue. I don't have enough >> context to judge whether or not this is bad enough to warrant >> cherry-picking, and the discussion on the issue didn't seem to come to a >> firm consensus. >> >> Can I get some opinions on this? > Well, it's certainly not rc-critical. It improves a bit a fringe > behaviour that's unlikely to be noticeable by anyone in the real world. > > If you look at http://bugs.python.org/issue20320, Charles-Fran?ois > there explains why it's a minor issue. Yes, I read the reply by Charles-Fran?ois. I was interested in seeing more opinions. However, nobody seems to be stepping up with any, so I'll assume for now it can be left unpicked. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Feb 23 12:31:54 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Feb 2014 12:31:54 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 References: <53095562.8010304@stoneleaf.us> <20140223034717.GA94757@cskk.homeip.net> <53097D84.2090201@stoneleaf.us> Message-ID: <20140223123154.58e83df5@fsol> On Sat, 22 Feb 2014 20:48:04 -0800 Ethan Furman wrote: > On 02/22/2014 07:47 PM, Cameron Simpson wrote: > > On 22Feb2014 17:56, Ethan Furman wrote: > >> Please let me know if anything else needs tweaking. > >> [...] > >> This area of programming is characterized by a mixture of binary data and > >> ASCII compatible segments of text (aka ASCII-encoded text). > >> [...] > >> %-interpolation > >> > >> All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, > >> ``%g``, etc.) will be supported, and will work as they do for str, including > >> the padding, justification and other related modifiers. > > > > I would like a single sentence here clarifying that the formatting > > of numeric values uses an ASCII encoding. > > How's this? > > All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, > ``%g``, etc.) will be supported, and will work as they do for str, including > the padding, justification and other related modifiers. The only difference > will be that the results from these codes will be ASCII-encoded bytes, not > unicode. You can't encode bytes, so it should be "ASCII-encoded text" ;-) Regards Antoine. From solipsis at pitrou.net Sun Feb 23 12:33:52 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Feb 2014 12:33:52 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 References: <53095562.8010304@stoneleaf.us> Message-ID: <20140223123352.5aa94595@fsol> On Sat, 22 Feb 2014 17:56:50 -0800 Ethan Furman wrote: > > ``%a`` will call :func:``ascii()`` on the interpolated value's :func:``repr()``. > This is intended as a debugging aid, rather than something that should be used > in production. Non-ascii values will be encoded to either ``\xnn`` or ``\unnnn`` > representation. Why is "%a" here? I don't remember: was this discussed before? "Intended as a debugging aid" sounds like a weak justification to me. Regards Antoine. From victor.stinner at gmail.com Sun Feb 23 12:30:25 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 23 Feb 2014 12:30:25 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <53095562.8010304@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> Message-ID: Hi, First, this is a warning in reST syntax: System Message: WARNING/2 (pep-0461.txt, line 53) > This area of programming is characterized by a mixture of binary data and > ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a > restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in > writing new wire format code, and in porting Python 2 wire format code. You may give some examples here: HTTP (Latin1 headers, binary body), SMTP, FTP, etc. > All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, > ``%g``, etc.) will be supported, and will work as they do for str, including > the padding, justification and other related modifiers. IMO you should give the exhaustive list here and we should only support one formatter for integers: %d. Python 2 supports "%d", "%u" and "%i" with "%u" marked as obsolete. Python 3.5 should not reintroduce obsolete formatters. If you want to use the same code base for Python 2.6, 2.7 and 3.5: modify your code to only use %d. Same rule apply for 2to3 tool: modify your source code to be compatible with Python 3. Please also mention all flags: #, +, -, '0', ' '. > ``%c`` will insert a single byte, either from an ``int`` in range(256), or > from > a ``bytes`` argument of length 1, not from a ``str``. I'm not sure that supporting bytes argument of 1 byte is useful, but it should not be hard to implement and may be convinient. > ``%s`` is restricted in what it will accept:: > > - input type supports ``Py_buffer`` [6]_? > use it to collect the necessary bytes > > - input type is something else? > use its ``__bytes__`` method [7]_ ; if there isn't one, raise a > ``TypeError`` Hum, you may mention that bytes(n: int) creates a bytes string of n null bytes, but b'%s' % 123 will raise an error because int.__bytes__() is not defined. Just to be more explicit. > ``%a`` will call :func:``ascii()`` on the interpolated value's > :func:``repr()``. > This is intended as a debugging aid, rather than something that should be > used > in production. Non-ascii values will be encoded to either ``\xnn`` or > ``\unnnn`` > representation. (You forgot "/Uhhhhhhhh" representation (it's an antislah, but I don't see the key on my Mac keyboard?).) What is the use case of this *new* formatter? How do you use it? print(b'%a" % 123) may emit a BytesWarning and may lead to bugs. IMO %a should be restricted for str%args. > It has been suggested to use ``%b`` for bytes as well as ``%s``. PyArg_ParseTuple() uses %y format for the exact bytes type. > > - Pro: clearly says 'this is bytes'; should be used for new code. > > - Con: does not exist in Python 2.x, so we would have two ways of doing > the > same thing, ``%s`` and ``%b``, with no difference between them. IMO it's useless, b'%s' % bytes just work fine in Python 2 and Python 3. -- I would like to help you to implement the PEP. IMO we should share as much code as possible with PyUnicodeObject. Something using the stringlib and maybe a new PyBytesWriter API which would have an API close to PyUnicodeWriter API. We should also try to share code between PyBytes_Format() and PyBytes_FromFormat(). Victor From cs at zip.com.au Sun Feb 23 12:40:52 2014 From: cs at zip.com.au (Cameron Simpson) Date: Sun, 23 Feb 2014 22:40:52 +1100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: Message-ID: <20140223114052.GA59190@cskk.homeip.net> On 23Feb2014 16:31, Nick Coghlan wrote: > On 23 February 2014 13:47, Cameron Simpson wrote: > > On 22Feb2014 17:56, Ethan Furman wrote: > >> Please let me know if anything else needs tweaking. > >> [...] > >> This area of programming is characterized by a mixture of binary data and > >> ASCII compatible segments of text (aka ASCII-encoded text). > >> [...] > >> %-interpolation > >> > >> All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, > >> ``%g``, etc.) will be supported, and will work as they do for str, including > >> the padding, justification and other related modifiers. > > > > I would like a single sentence here clarifying that the formatting > > of numeric values uses an ASCII encoding. > > > > It might be inferred from the earlier context, but I do not think > > it can be deduced and therefore I think it should be said outright. > > All the other formatting codes are quite explicit about how their > > arguments transform into bytes, but the numeric codes just quietly > > assume ASCII. The PEP should be blatant. > > Specifically, I believe the PEP should state that, for the numeric codes: > > b"%x" % val > > is equivalent to: > > b"%s" % (("%x" % val).encode("ascii")) > > The rationale for including them is the unreadability of the latter form :) Hmm. Isn't: ("%x" % val).encode("ascii") sufficient here? I still think that the term ASCII should appear in the prose, rather than forcing the reader to decode the above. Example, shoehorning off Ethan's response: The substituted bytes will be an ASCII encoding of the corresponding str formatting codes. Specificaly, for any numeric formatting code "%x": b"%x" % val is equivalent to: ("%x" % val).encode("ascii") That ticks my wishes and includes Nick's explicit algorithmic expression of the process. Cheers, -- Cameron Simpson Me, I'm looking for obituaries. Lately a gratifyingly large number of my most odious near-contemporaries are achieving their long-deserved quietus. Not enough, and not always the right ones, but their time will come. Peeve: I may not live to see them dead. - Lee Rudolph, rudolph at cis.umassd.edu From cs at zip.com.au Sun Feb 23 13:05:24 2014 From: cs at zip.com.au (Cameron Simpson) Date: Sun, 23 Feb 2014 23:05:24 +1100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: Message-ID: <20140223120524.GA71227@cskk.homeip.net> On 23Feb2014 12:30, Victor Stinner wrote: > > All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, > > ``%g``, etc.) will be supported, and will work as they do for str, including > > the padding, justification and other related modifiers. > > IMO you should give the exhaustive list here and we should only > support one formatter for integers: %d. Python 2 supports "%d", "%u" > and "%i" with "%u" marked as obsolete. Python 3.5 should not > reintroduce obsolete formatters. If you want to use the same code base > for Python 2.6, 2.7 and 3.5: modify your code to only use %d. Same > rule apply for 2to3 tool: modify your source code to be compatible > with Python 3. > Please also mention all flags: #, +, -, '0', ' '. Is this really necessary? Can't one just refer the the str %-formatting section of the doco? By section and title to make it easy to find. I think this should just refer the reader to the str %-formatting doco for the numeric codes and their meanings, along with the flags. Otherwise the PEP will get unreadable, to no value that I can see. If we include Nick's equivalent code example, there is no ambiguity or vagueness. I'm against restricting to just %d for int too; if the current Python supports others (eg %o, %x) for str, so should this PEP for bytes. > > ``%c`` will insert a single byte, either from an ``int`` in range(256), or > > from > > a ``bytes`` argument of length 1, not from a ``str``. > > I'm not sure that supporting bytes argument of 1 byte is useful, but > it should not be hard to implement and may be convinient. I'm +0.5 for a bytes argument of length 1; while bytes are arrays of small ints, just as str has no distinct "char" type a bytes has no distinct byte type. With a string we commonly use s str of length 1 to denote a single character in isolation; the same programming idioms will get you a bytes of length 1 in situations when you mean a byte. > (You forgot "/Uhhhhhhhh" representation (it's an antislah, but I don't > see the key on my Mac keyboard?).) My Mac has one above the "return" key. Um, non-English locale? Curious. Cheers, -- Cameron Simpson 16 October. I also asked Anthea how many mature oaks she thought it would have taken to build a top-of-the-line ship in Nelson's day. She guessed ten. The astonishing answer (from Brewer's) is about 3,500 - 900 acres of oak forest. She said, "I wonder what we're doing now that's as wasteful as that". I said it's still called Defence. - Brian Eno, _A Year With Swollen Appendices_ From Nikolaus at rath.org Sun Feb 23 07:50:11 2014 From: Nikolaus at rath.org (Nikolaus Rath) Date: Sat, 22 Feb 2014 22:50:11 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <53095562.8010304@stoneleaf.us> (Ethan Furman's message of "Sat, 22 Feb 2014 17:56:50 -0800") References: <53095562.8010304@stoneleaf.us> Message-ID: <87zjliuxuk.fsf@vostro.rath.org> Ethan Furman writes: > Example:: > > >>> b'%4x' % 10 > b' a' > > >>> '%#4x' % 10 > ' 0xa' > > >>> '%04X' % 10 > '000A' Shouldn't the second two examples also be bytes, ie. b'%#4x' instead of '%#4x'? Best, -Nikolaus -- Encrypted emails preferred. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C ?Time flies like an arrow, fruit flies like a Banana.? From ijmorlan at uwaterloo.ca Sun Feb 23 14:21:25 2014 From: ijmorlan at uwaterloo.ca (Isaac Morland) Date: Sun, 23 Feb 2014 08:21:25 -0500 (EST) Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <5307CDDE.8090106@canterbury.ac.nz> <53094AD1.9090805@mrabarnett.plus.com> Message-ID: On Sun, 23 Feb 2014, Nick Coghlan wrote: > Note that mandatory parentheses means we can duck the precedence > question entirely, which I count as another point in favour of > requiring them :) Careful, if you take that too far then Python 4 will have to be Scheme. ;-) Isaac Morland CSCF Web Guru DC 2619, x36650 WWW Software Specialist From georg at python.org Sun Feb 23 17:25:21 2014 From: georg at python.org (Georg Brandl) Date: Sun, 23 Feb 2014 17:25:21 +0100 Subject: [Python-Dev] [RELEASED] Python 3.3.5 release candidate 1 Message-ID: <530A20F1.3030400@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I'm happy to announce the release of Python 3.3.5, release candidate 1. Python 3.3.5 includes a fix for a regression in zipimport in 3.3.4 (see http://bugs.python.org/issue20621) and a few other bugs. Python 3.3 includes a range of improvements of the 3.x series, as well as easier porting between 2.x and 3.x. In total, almost 500 API items are new or improved in Python 3.3. For a more extensive list of changes in the 3.3 series, see http://docs.python.org/3.3/whatsnew/3.3.html To download Python 3.3.5 visit: http://www.python.org/download/releases/3.3.5/ This is a preview release, please report any bugs to http://bugs.python.org/ The final release is scheduled one week from now. Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.3's contributors) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlMKIPEACgkQN9GcIYhpnLCjXACfQwbC/eD/lhKAZ+XCwTwYPVWj GMwAnjWkbdk7hqsKoh12EiagpGApEPSA =2BCx -----END PGP SIGNATURE----- From stefan_ml at behnel.de Sun Feb 23 19:51:43 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 23 Feb 2014 19:51:43 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: Chris Angelico, 21.02.2014 04:15: > Just as PEP 308 introduced a means of value-based conditions in an > expression, this system allows exception-based conditions to be used > as part of an expression. > [...] > This currently works:: > > lst = [1, 2, None, 3] > value = lst[2] or "No value" > > The proposal adds this:: > > lst = [1, 2] > value = lst[2] except IndexError: "No value" I see a risk of interfering with in-place assignment operators, e.g. x /= y except ZeroDivisionError: 1 might not do what one could expect, because (as I assume) it would behave differently from x = x / y except ZeroDivisionError: 1 I think that falls under the "overly broad exception handling" issue. If you want to include the assignment, you'll have to spell out the try-except block yourself. I find the difference in the two behaviours very unfortunate, though. This also reduces the scope of applicability somewhat. Cython has typed assignments, so a straight forward idea would be to handle TypeErrors in assignments like this: cdef str s s = x except TypeError: str(x) However, I guess that would similarly counter the idea of exception handling in an *expression*, and the correct and non-ambiguous way to do this would be to spell out the try-except block. Summing it up, my impression is that it helps some use cases but leaves others more ambiguous/unclear/unfortunate, which makes me lean towards rejecting it. Stefan From stefan_ml at behnel.de Sun Feb 23 20:20:52 2014 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 23 Feb 2014 20:20:52 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: Stefan Behnel, 23.02.2014 19:51: > Cython has typed > assignments, so a straight forward idea would be to handle TypeErrors in > assignments like this: > > cdef str s > s = x except TypeError: str(x) Similar code in Python would be this: from array import array x = array('i', [1,2,3]) value = "123" x[0] = value except TypeError: int(value) > However, I guess that would similarly counter the idea of exception > handling in an *expression*, and the correct and non-ambiguous way to do > this would be to spell out the try-except block. Stefan From thomas at python.org Sun Feb 23 20:26:21 2014 From: thomas at python.org (Thomas Wouters) Date: Sun, 23 Feb 2014 11:26:21 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Feb 23, 2014 7:52 PM, "Stefan Behnel" wrote: > > Chris Angelico, 21.02.2014 04:15: > > Just as PEP 308 introduced a means of value-based conditions in an > > expression, this system allows exception-based conditions to be used > > as part of an expression. > > [...] > > This currently works:: > > > > lst = [1, 2, None, 3] > > value = lst[2] or "No value" > > > > The proposal adds this:: > > > > lst = [1, 2] > > value = lst[2] except IndexError: "No value" > > I see a risk of interfering with in-place assignment operators, e.g. > > x /= y except ZeroDivisionError: 1 > > might not do what one could expect, because (as I assume) it would behave > differently from > > x = x / y except ZeroDivisionError: 1 Yes. Augmented assignment is still assignment, so a statement. The only way to parse that is as x /= (y except ZeroDivisionError: 1) and it'd be equivalent to x = x / (y except ZeroDivisionError: 1) (If the parentheses are mandatory that makes it easier to spot the difference.) > > I think that falls under the "overly broad exception handling" issue. If > you want to include the assignment, you'll have to spell out the try-except > block yourself. I find the difference in the two behaviours very > unfortunate, though. > > This also reduces the scope of applicability somewhat. Cython has typed > assignments, so a straight forward idea would be to handle TypeErrors in > assignments like this: > > cdef str s > s = x except TypeError: str(x) > > However, I guess that would similarly counter the idea of exception > handling in an *expression*, and the correct and non-ambiguous way to do > this would be to spell out the try-except block. > > Summing it up, my impression is that it helps some use cases but leaves > others more ambiguous/unclear/unfortunate, which makes me lean towards > rejecting it. > > Stefan > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/thomas%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Feb 23 20:44:53 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Feb 2014 11:44:53 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <20140223123154.58e83df5@fsol> References: <53095562.8010304@stoneleaf.us> <20140223034717.GA94757@cskk.homeip.net> <53097D84.2090201@stoneleaf.us> <20140223123154.58e83df5@fsol> Message-ID: <530A4FB5.7020507@stoneleaf.us> On 02/23/2014 03:31 AM, Antoine Pitrou wrote: > On Sat, 22 Feb 2014 20:48:04 -0800 Ethan Furman wrote: >> >> All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, >> ``%g``, etc.) will be supported, and will work as they do for str, including >> the padding, justification and other related modifiers. The only difference >> will be that the results from these codes will be ASCII-encoded bytes, not >> unicode. > > You can't encode bytes, so it should be "ASCII-encoded text" ;-) Good point, thanks. -- ~Ethan~ From ethan at stoneleaf.us Sun Feb 23 20:47:51 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Feb 2014 11:47:51 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <87zjliuxuk.fsf@vostro.rath.org> References: <53095562.8010304@stoneleaf.us> <87zjliuxuk.fsf@vostro.rath.org> Message-ID: <530A5067.9070701@stoneleaf.us> On 02/22/2014 10:50 PM, Nikolaus Rath wrote: > Ethan Furman writes: >> Example:: >> >> >>> b'%4x' % 10 >> b' a' >> >> >>> '%#4x' % 10 >> ' 0xa' >> >> >>> '%04X' % 10 >> '000A' > > Shouldn't the second two examples also be bytes, ie. b'%#4x' instead of > '%#4x'? Yup, thanks. -- ~Ethan~ From ethan at stoneleaf.us Sun Feb 23 21:42:59 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Feb 2014 12:42:59 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <20140223123352.5aa94595@fsol> References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> Message-ID: <530A5D53.7070108@stoneleaf.us> On 02/23/2014 03:33 AM, Antoine Pitrou wrote: > On Sat, 22 Feb 2014 17:56:50 -0800 > Ethan Furman wrote: >> >> ``%a`` will call :func:``ascii()`` on the interpolated value's :func:``repr()``. >> This is intended as a debugging aid, rather than something that should be used >> in production. Non-ascii values will be encoded to either ``\xnn`` or ``\unnnn`` >> representation. > > Why is "%a" here? I don't remember: was this discussed before? > "Intended as a debugging aid" sounds like a weak justification to me. https://mail.python.org/pipermail/python-dev/2014-January/131808.html The idea being if we offer %a, folks won't be tempted to abuse __bytes__. -- ~Ethan~ From ethan at stoneleaf.us Sun Feb 23 22:25:45 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Feb 2014 13:25:45 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: <53095562.8010304@stoneleaf.us> Message-ID: <530A6759.4030809@stoneleaf.us> On 02/23/2014 03:30 AM, Victor Stinner wrote: > > First, this is a warning in reST syntax: > > System Message: WARNING/2 (pep-0461.txt, line 53) Yup, fixed that. >> This area of programming is characterized by a mixture of binary data and >> ASCII compatible segments of text (aka ASCII-encoded text). Bringing back a >> restricted %-interpolation for ``bytes`` and ``bytearray`` will aid both in >> writing new wire format code, and in porting Python 2 wire format code. > > You may give some examples here: HTTP (Latin1 headers, binary body), > SMTP, FTP, etc. > >> All the numeric formatting codes (such as ``%x``, ``%o``, ``%e``, ``%f``, >> ``%g``, etc.) will be supported, and will work as they do for str, including >> the padding, justification and other related modifiers. > > IMO you should give the exhaustive list here and we should only > support one formatter for integers: %d. Python 2 supports "%d", "%u" > and "%i" with "%u" marked as obsolete. Python 3.5 should not > reintroduce obsolete formatters. If you want to use the same code base > for Python 2.6, 2.7 and 3.5: modify your code to only use %d. Same > rule apply for 2to3 tool: modify your source code to be compatible > with Python 3. A link is provided to the exhaustive list. Including it verbatim here detracts from the overall readablity. I agree that having only one decimal format code would be nice, or even two if the second one did something different, and that three seems completely over the top -- unfortunately, Python 3.4 still supports all three (%d, %i, and %u). Not supporting two of them would just lead to frustration. There is also no reason to exclude %o nor %x and making the programmer reach for oct() and hex(). We're trying to simplify %-interpolation, not garner exclamations of "What were they thinking?!?" ;) >> ``%s`` is restricted in what it will accept:: >> >> - input type supports ``Py_buffer`` [6]_? >> use it to collect the necessary bytes >> >> - input type is something else? >> use its ``__bytes__`` method [7]_ ; if there isn't one, raise a >> ``TypeError`` > > Hum, you may mention that bytes(n: int) creates a bytes string of n > null bytes, but b'%s' % 123 will raise an error because > int.__bytes__() is not defined. Just to be more explicit. I added a line stating that %s does not accept numbers, but I'm not sure how bytes(n: int) is relevant? >> ``%a`` will call :func:``ascii()`` on the interpolated value's >> :func:``repr()``. >> This is intended as a debugging aid, rather than something that should be >> used >> in production. Non-ascii values will be encoded to either ``\xnn`` or >> ``\unnnn`` >> representation. > > (You forgot "/Uhhhhhhhh" representation (it's an antislah, but I don't > see the key on my Mac keyboard?).) Hard to forget what you don't know. ;) Will ascii() ever emit an antislash representation? > What is the use case of this *new* formatter? How do you use it? An aid to debugging -- need to see what's what at that moment? Toss it into %a. It is not intended for production code, but is included to hopefully circumvent the inappropriate use of __bytes__ methods on classes. > print(b'%a" % 123) may emit a BytesWarning and may lead to bugs. Why would it emit a BytesWarning? > I would like to help you to implement the PEP. IMO we should share as > much code as possible with PyUnicodeObject. Something using the > stringlib and maybe a new PyBytesWriter API which would have an API > close to PyUnicodeWriter API. We should also try to share code between > PyBytes_Format() and PyBytes_FromFormat(). Thanks. I'll holler when I get that far. :) -- ~Ethan~ From solipsis at pitrou.net Sun Feb 23 22:37:45 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Feb 2014 22:37:45 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> <530A5D53.7070108@stoneleaf.us> Message-ID: <20140223223745.231a5b5e@fsol> On Sun, 23 Feb 2014 12:42:59 -0800 Ethan Furman wrote: > On 02/23/2014 03:33 AM, Antoine Pitrou wrote: > > On Sat, 22 Feb 2014 17:56:50 -0800 > > Ethan Furman wrote: > >> > >> ``%a`` will call :func:``ascii()`` on the interpolated value's :func:``repr()``. > >> This is intended as a debugging aid, rather than something that should be used > >> in production. Non-ascii values will be encoded to either ``\xnn`` or ``\unnnn`` > >> representation. > > > > Why is "%a" here? I don't remember: was this discussed before? > > "Intended as a debugging aid" sounds like a weak justification to me. > > https://mail.python.org/pipermail/python-dev/2014-January/131808.html > > The idea being if we offer %a, folks won't be tempted to abuse __bytes__. Which folks are we talking about? This sounds gratuitous. Also, I don't understand what debugging is supposed to be in the context of bytes formatting. You print debugging output to a text stream, not a bytes stream. And you certainly *don't* print debugging output into a wire protocol. Regards Antoine. From ethan at stoneleaf.us Sun Feb 23 21:51:57 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 23 Feb 2014 12:51:57 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <530A5F6D.5020000@stoneleaf.us> On 02/23/2014 11:26 AM, Thomas Wouters wrote: > > On Feb 23, 2014 7:52 PM, "Stefan Behnel" > wrote: >> >> Chris Angelico, 21.02.2014 04:15: >> > Just as PEP 308 introduced a means of value-based conditions in an >> > expression, this system allows exception-based conditions to be used >> > as part of an expression. >> > [...] >> > This currently works:: >> > >> > lst = [1, 2, None, 3] >> > value = lst[2] or "No value" >> > >> > The proposal adds this:: >> > >> > lst = [1, 2] >> > value = lst[2] except IndexError: "No value" >> >> I see a risk of interfering with in-place assignment operators, e.g. >> >> x /= y except ZeroDivisionError: 1 >> >> might not do what one could expect, because (as I assume) it would behave >> differently from >> >> x = x / y except ZeroDivisionError: 1 > > Yes. Augmented assignment is still assignment, so a statement. The only way to parse that is as > > x /= (y except ZeroDivisionError: 1) Well, that is certainly not what I would have expected. I can also see how parentheses can help, but I still would like them optional. -- ~Ethan~ From rosuav at gmail.com Sun Feb 23 22:55:32 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 24 Feb 2014 08:55:32 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: On Mon, Feb 24, 2014 at 6:26 AM, Thomas Wouters wrote: >> I see a risk of interfering with in-place assignment operators, e.g. >> >> x /= y except ZeroDivisionError: 1 >> >> might not do what one could expect, because (as I assume) it would behave >> differently from >> >> x = x / y except ZeroDivisionError: 1 > > Yes. Augmented assignment is still assignment, so a statement. The only way > to parse that is as > > x /= (y except ZeroDivisionError: 1) > > and it'd be equivalent to > > x = x / (y except ZeroDivisionError: 1) > > (If the parentheses are mandatory that makes it easier to spot the > difference.) > >> >> I think that falls under the "overly broad exception handling" issue. If >> you want to include the assignment, you'll have to spell out the >> try-except >> block yourself. I find the difference in the two behaviours very >> unfortunate, though. Thomas's analysis is correct. It's not overly broad; in fact, what you have is an overly _narrow_ exception handler, catching a ZeroDivisionError from the evaluation of y only. ChrisA From victor.stinner at gmail.com Sun Feb 23 22:56:27 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Sun, 23 Feb 2014 22:56:27 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <530A6759.4030809@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> <530A6759.4030809@stoneleaf.us> Message-ID: > > >>> >> (You forgot "/Uhhhhhhhh" representation (it's an antislah, but I don't >> see the key on my Mac keyboard?).) >> > > Hard to forget what you don't know. ;) Will ascii() ever emit an > antislash representation? Try ascii(chr(0x1fffff)). What is the use case of this *new* formatter? How do you use it? >> > > An aid to debugging -- need to see what's what at that moment? Toss it > into %a. It is not intended for production code, but is included to > hopefully circumvent the inappropriate use of __bytes__ methods on classes. How do you plan to use this output? Write it into a socket or a file? When I debug, I use print & logging which both expect text string. So I think that b'%a' is useless. > print(b'%a" % 123) may emit a BytesWarning and may lead to bugs. >> > > Why would it emit a BytesWarning? Because print expects a text string, and print(bytes) does an implicit conversion to Unicode. Try: python -bb -c "print(b'hello')". Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Sun Feb 23 23:14:55 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 23 Feb 2014 14:14:55 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <20140223223745.231a5b5e@fsol> References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> <530A5D53.7070108@stoneleaf.us> <20140223223745.231a5b5e@fsol> Message-ID: <530A72DF.8080702@g.nevcal.com> On 2/23/2014 1:37 PM, Antoine Pitrou wrote: > And you certainly*don't* print debugging output into a wire protocol. Web server applications do, so they can be displayed in the browser. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Feb 23 23:25:41 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 23 Feb 2014 23:25:41 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> <530A5D53.7070108@stoneleaf.us> <20140223223745.231a5b5e@fsol> <530A72DF.8080702@g.nevcal.com> Message-ID: <20140223232541.7029fe78@fsol> On Sun, 23 Feb 2014 14:14:55 -0800 Glenn Linderman wrote: > On 2/23/2014 1:37 PM, Antoine Pitrou wrote: > > And you certainly*don't* print debugging output into a wire protocol. > > Web server applications do, so they can be displayed in the browser. They may embed debugging information into some HTML code, which then will be sent over the wire. However, usually they don't print debugging output directly into HTTP. Regards Antoine. From ncoghlan at gmail.com Sun Feb 23 23:54:08 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Feb 2014 08:54:08 +1000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <20140223223745.231a5b5e@fsol> References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> <530A5D53.7070108@stoneleaf.us> <20140223223745.231a5b5e@fsol> Message-ID: On 24 Feb 2014 07:39, "Antoine Pitrou" wrote: > > On Sun, 23 Feb 2014 12:42:59 -0800 > Ethan Furman wrote: > > On 02/23/2014 03:33 AM, Antoine Pitrou wrote: > > > On Sat, 22 Feb 2014 17:56:50 -0800 > > > Ethan Furman wrote: > > >> > > >> ``%a`` will call :func:``ascii()`` on the interpolated value's :func:``repr()``. > > >> This is intended as a debugging aid, rather than something that should be used > > >> in production. Non-ascii values will be encoded to either ``\xnn`` or ``\unnnn`` > > >> representation. > > > > > > Why is "%a" here? I don't remember: was this discussed before? > > > "Intended as a debugging aid" sounds like a weak justification to me. > > > > https://mail.python.org/pipermail/python-dev/2014-January/131808.html > > > > The idea being if we offer %a, folks won't be tempted to abuse __bytes__. > > Which folks are we talking about? This sounds gratuitous. It's a harm containment tactic, based on the assumption people *will* want to include the output of ascii() in binary protocols containing ASCII segments, regardless of whether or not we consider their reasons for doing so to be particularly good. If %a exists, then the path of least resistance to doing this only affects the format string, and it can handle arbitrary types (except bytes under -b and -bb). By contrast, if %a doesn't exist, then it becomes more attractive to use %s in the format string and define an ASCII assuming __bytes__ implementation on a custom type. That latter scenario is substantially more problematic, since __bytes__ implementations assuming ASCII compatibility is categorically wrong, while embedding an ASCII representation in a binary protocol that includes ASCII compatible segments is merely a bit strange. Cheers, Nick. > > Also, I don't understand what debugging is supposed to be in the > context of bytes formatting. You print debugging output to a text > stream, not a bytes stream. And you certainly *don't* print debugging > output into a wire protocol. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at zip.com.au Sun Feb 23 23:56:39 2014 From: cs at zip.com.au (Cameron Simpson) Date: Mon, 24 Feb 2014 09:56:39 +1100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: Message-ID: <20140223225639.GA82232@cskk.homeip.net> On 23Feb2014 22:56, Victor Stinner wrote: > > An aid to debugging -- need to see what's what at that moment? Toss it > > into %a. It is not intended for production code, but is included to > > hopefully circumvent the inappropriate use of __bytes__ methods on classes. > > How do you plan to use this output? Write it into a socket or a file? > When I debug, I use print & logging which both expect text string. So I > think that b'%a' is useless. The case from the email thread, which I support at +0.5 or maybe only +0.1, is printing to a binary log. The classic example that comes to mind is syslog packets. I agree %a invites data mangling. One would hope it doesn't see use in wire protocols, only in debugging scenarios. Regrettably, syslog is such a binary logging protocol, purportedly for "text". Cheers, -- Cameron Simpson We had the experience, but missed the meaning. - T.S. Eliot From solipsis at pitrou.net Mon Feb 24 00:04:33 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 24 Feb 2014 00:04:33 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> <530A5D53.7070108@stoneleaf.us> <20140223223745.231a5b5e@fsol> Message-ID: <20140224000433.23c15db1@fsol> On Mon, 24 Feb 2014 08:54:08 +1000 Nick Coghlan wrote: > > > The idea being if we offer %a, folks won't be tempted to abuse > __bytes__. > > > > Which folks are we talking about? This sounds gratuitous. > > It's a harm containment tactic, based on the assumption people *will* want > to include the output of ascii() in binary protocols containing ASCII > segments But why would they? ascii() doesn't do what they want, since it's repr()-like, not str()-like. It seems your assumption is wrong. > By contrast, if %a doesn't exist, then it becomes more attractive to use %s > in the format string and define an ASCII assuming __bytes__ implementation > on a custom type. Uh... Few Python programmers would actually think of writing a __bytes__ method just to enable bytes interpolation for their custom types. However, adding "%a" as a supported interpolation format just makes things confusing for *everyone*. Regards Antoine. From eric at trueblade.com Mon Feb 24 00:14:11 2014 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 23 Feb 2014 18:14:11 -0500 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <530A6759.4030809@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> <530A6759.4030809@stoneleaf.us> Message-ID: <530A80C3.9060008@trueblade.com> On 2/23/2014 4:25 PM, Ethan Furman wrote: > I agree that having only one decimal format code would be nice, or even > two if the second one did something different, and that three seems > completely over the top -- unfortunately, Python 3.4 still supports all > three (%d, %i, and %u). Not supporting two of them would just lead to > frustration. There is also no reason to exclude %o nor %x and making > the programmer reach for oct() and hex(). We're trying to simplify > %-interpolation, not garner exclamations of "What were they > thinking?!?" ;) There are things that can be done with %o and %x that cannot be done with oct() and hex(), or at least cannot be done without a terrific amount of byte munging. For example: >>> '%#.4x' % 42 '0x002a' Not sure you'd ever need to do that in a wire protocol, but it's possible. Since one of the motivators of this feature is to make porting easier, I'd suggest fully supporting the numeric codes that are supported in 2.7. I do have some sympathy for the "change your code to a common 2.x-3.x subset" position. But since 2.7's -3 flag won't (and can't) warn you when you're doing something with %-formatting that's not support in 3.x, I think the user-friendliest approach is to support all of the numeric codes as completely as possible. Eric. From v+python at g.nevcal.com Mon Feb 24 00:29:02 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 23 Feb 2014 15:29:02 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <20140223232541.7029fe78@fsol> References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> <530A5D53.7070108@stoneleaf.us> <20140223223745.231a5b5e@fsol> <530A72DF.8080702@g.nevcal.com> <20140223232541.7029fe78@fsol> Message-ID: <530A843E.9050006@g.nevcal.com> On 2/23/2014 2:25 PM, Antoine Pitrou wrote: > On Sun, 23 Feb 2014 14:14:55 -0800 > Glenn Linderman wrote: >> On 2/23/2014 1:37 PM, Antoine Pitrou wrote: >>> And you certainly*don't* print debugging output into a wire protocol. >> Web server applications do, so they can be displayed in the browser. > They may embed debugging information into some HTML code, which then > will be sent over the wire. However, usually they don't print debugging > output directly into HTTP. The HTML is sent over the wire via HTTP... that's pretty directly in the wire protocol... the HTTP headers are immediately followed by the HTML, and when the document is being generated on the fly, it may also be being encoded on the fly. I've seen it done, although I can't confirm or deny the "usually" claim you have made. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zuo at chopin.edu.pl Mon Feb 24 01:36:46 2014 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Mon, 24 Feb 2014 01:36:46 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: Message-ID: <6f9ca1119a4f971c2cc12bc5f87aa232@chopin.edu.pl> 23.02.2014 19:51, Stefan Behnel wrote: > I see a risk of interfering with in-place assignment operators, e.g. > > x /= y except ZeroDivisionError: 1 > > might not do what one could expect, because (as I assume) it would > behave > differently from > > x = x / y except ZeroDivisionError: 1 [snip] Please note that: x /= y if y else 0 also behaves differently from x = x / y if y else 0 Anyway, enclosing in parens would make that expicit and clear. Cheers. *j From rosuav at gmail.com Mon Feb 24 01:50:09 2014 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 24 Feb 2014 11:50:09 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <530A5F6D.5020000@stoneleaf.us> References: <530A5F6D.5020000@stoneleaf.us> Message-ID: On Mon, Feb 24, 2014 at 7:51 AM, Ethan Furman wrote: >> Yes. Augmented assignment is still assignment, so a statement. The only >> way to parse that is as >> >> x /= (y except ZeroDivisionError: 1) > > > Well, that is certainly not what I would have expected. I can see that you'd want to have that go back and redo the division with a second argument of 1, which'd look like this in statement form: try: x /= y except ZeroDivisionError: x /= 1 But, just like the decried error suppression technique, the second half of this is something that should instead be written "pass". At very least, I'd say that an except-expression where one or other of its forms is better spelled "pass" is code smell, and at worst, I'd say it's a hint that the expression form might not even be what you think it is - as in this case. Remember, this is a scope-narrowing. Where previously you had to try/except entire statements, now you can try/except just one part of something. That means you won't catch errors in the actual assignment - which is usually a good thing - but it does affect augmented assignment. My recommendation: Just use try... except pass. I'm not trying to supplant the statement form :) ChrisA From stephen at xemacs.org Mon Feb 24 03:30:45 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 24 Feb 2014 11:30:45 +0900 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <530A843E.9050006@g.nevcal.com> References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> <530A5D53.7070108@stoneleaf.us> <20140223223745.231a5b5e@fsol> <530A72DF.8080702@g.nevcal.com> <20140223232541.7029fe78@fsol> <530A843E.9050006@g.nevcal.com> Message-ID: <874n3putre.fsf@uwakimon.sk.tsukuba.ac.jp> Glenn Linderman writes: > On 2/23/2014 2:25 PM, Antoine Pitrou wrote: >> On Sun, 23 Feb 2014 14:14:55 -0800 Glenn Linderman wrote: >>> On 2/23/2014 1:37 PM, Antoine Pitrou wrote: >>>> And you certainly*don't* print debugging output into a wire protocol. >>> Web server applications do, so they can be displayed in the browser. >> They may embed debugging information into some HTML code, which >> then will be sent over the wire. However, usually they don't >> print debugging output directly into HTTP. > The HTML is sent over the wire via HTTP... that's pretty directly > in the wire protocol... Not in the relevant sense. In a modern web framework, the HTML will typically be in internal text encoding because the framework can't know what the programmer/web developer/user will be using. So there's no need at all for PEP 461 here: you're going to be using str, and then running it through .encode() anyway. > the HTTP headers are immediately followed by the HTML, and when the > document is being generated on the fly, it may also be being > encoded on the fly. I've seen it done, although I can't confirm or > deny the "usually" claim you have made. I'm sure you've seen it done. Is it worth providing special support for it? I don't think so, and Nick's "we don't want people writing __bytes__ methods" argument sounds suspiciously like a child-proof cap to me. If people really wanna do that, let them use ascii(). From ncoghlan at gmail.com Mon Feb 24 03:45:42 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Feb 2014 12:45:42 +1000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <20140223225639.GA82232@cskk.homeip.net> References: <20140223225639.GA82232@cskk.homeip.net> Message-ID: On 24 February 2014 08:56, Cameron Simpson wrote: > On 23Feb2014 22:56, Victor Stinner wrote: >> > An aid to debugging -- need to see what's what at that moment? Toss it >> > into %a. It is not intended for production code, but is included to >> > hopefully circumvent the inappropriate use of __bytes__ methods on classes. >> >> How do you plan to use this output? Write it into a socket or a file? >> When I debug, I use print & logging which both expect text string. So I >> think that b'%a' is useless. > > The case from the email thread, which I support at +0.5 or maybe > only +0.1, is printing to a binary log. The classic example that > comes to mind is syslog packets. We actually hit a bug related to that in Beaker not that long ago - we were interpolating (Python 2) 8-bit strings directly into the syslog data, and it corrupted the log message when one of those strings contained a NULL value. Would leaving %a out destroy the utility of the PEP? No. Is leaving it in useful? I think so, yes, as it provides OOWTD interpolation of pure ASCII representations into binary formats that contain ASCII compatible segments, and it's directly analogous to the handling of the numeric formatting codes with (b"%a" % obj) being a shorthand for (b"%s" % ("%a" % obj).encode("ascii")). (Note that invoking repr() or ascii() on a bytes instance is perfectly legal, even under -b and -bb - it's only str() that triggers a warning or error) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From larry at hastings.org Mon Feb 24 05:01:20 2014 From: larry at hastings.org (Larry Hastings) Date: Sun, 23 Feb 2014 22:01:20 -0600 Subject: [Python-Dev] [RELEASED] Python 3.4.0 release candidate 2 is now available Message-ID: <530AC410.4020600@hastings.org> On behalf of the Python development team, I'm delighted to announce the second and final release candidate of Python 3.4. This is a preview release, and its use is not recommended for production settings. Python 3.4 includes a range of improvements of the 3.x series, including hundreds of small improvements and bug fixes. Major new features and changes in the 3.4 release series include: * PEP 428, a "pathlib" module providing object-oriented filesystem paths * PEP 435, a standardized "enum" module * PEP 436, a build enhancement that will help generate introspection information for builtins * PEP 442, improved semantics for object finalization * PEP 443, adding single-dispatch generic functions to the standard library * PEP 445, a new C API for implementing custom memory allocators * PEP 446, changing file descriptors to not be inherited by default in subprocesses * PEP 450, a new "statistics" module * PEP 451, standardizing module metadata for Python's module import system * PEP 453, a bundled installer for the *pip* package manager * PEP 454, a new "tracemalloc" module for tracing Python memory allocations * PEP 456, a new hash algorithm for Python strings and binary data * PEP 3154, a new and improved protocol for pickled objects * PEP 3156, a new "asyncio" module, a new framework for asynchronous I/O Python 3.4 is now in "feature freeze", meaning that no new features will be added. The final release is projected for mid-March 2014. The python.org web site has recently been updated to something completely new, and I'm having some difficulty updating it. For now I've made Python 3.4.0rc2 available on the legacy web site: http://legacy.python.org/download/releases/3.4.0/ Once I can update the new web site, Python 3.4.0rc2 will be available here: http://python.org/download/releases/ (I'm not sure what the final URL will be, but you'll see it listed on that page.) Please consider trying Python 3.4.0rc2 with your code and reporting any new issues you notice to: http://bugs.python.org/ Enjoy! -- Larry Hastings, Release Manager larry at hastings.org (on behalf of the entire python-dev team and 3.4's contributors) -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Feb 24 10:14:38 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 24 Feb 2014 10:14:38 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: <20140223225639.GA82232@cskk.homeip.net> Message-ID: 2014-02-24 3:45 GMT+01:00 Nick Coghlan : > Would leaving %a out destroy the utility of the PEP? Usually, debug code is not even commited. So writing b'var=%s' % ascii(var).encode() is not hard. Or maybe: b'var=%s' % repr(var).encode('ascii', 'backslashreplace') which is the same but longer :-) Victor From ethan at stoneleaf.us Mon Feb 24 18:15:29 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 24 Feb 2014 09:15:29 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> <530A5D53.7070108@stoneleaf.us> <20140223223745.231a5b5e@fsol> Message-ID: <530B7E31.8000805@stoneleaf.us> On 02/23/2014 02:54 PM, Nick Coghlan wrote: > > It's a harm containment tactic, based on the assumption people *will* > want to include the output of ascii() in binary protocols containing > ASCII segments, regardless of whether or not we consider their reasons > for doing so to be particularly good. One possible problem with %a -- it becomes the bytes equivalent of %s in Python 2 strings, with the minor exception of how unicode strings are handled (quote marks are added). In other words, instead of %d, one could use %a. On the other hand, %a is so much more user-friendly than b'%s' % ('%d' % 123).encode('ascii', errors='backslashreplace'). -- ~Ethan~ From solipsis at pitrou.net Mon Feb 24 18:43:04 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 24 Feb 2014 18:43:04 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> <530A5D53.7070108@stoneleaf.us> <20140223223745.231a5b5e@fsol> <530B7E31.8000805@stoneleaf.us> Message-ID: <20140224184304.7b95f66c@fsol> On Mon, 24 Feb 2014 09:15:29 -0800 Ethan Furman wrote: > On 02/23/2014 02:54 PM, Nick Coghlan wrote: > > > > It's a harm containment tactic, based on the assumption people *will* > > want to include the output of ascii() in binary protocols containing > > ASCII segments, regardless of whether or not we consider their reasons > > for doing so to be particularly good. > > One possible problem with %a -- it becomes the bytes equivalent of %s in Python 2 strings, with the minor exception of > how unicode strings are handled (quote marks are added). In other words, instead of %d, one could use %a. > > On the other hand, %a is so much more user-friendly than b'%s' % ('%d' % 123).encode('ascii', errors='backslashreplace'). But why not b'%d' % 123 ? Regards Antoine. From rob.cliffe at btinternet.com Mon Feb 24 18:52:43 2014 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 24 Feb 2014 17:52:43 +0000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <5307A34D.6050101@gmail.com> <20140222050114.GE3684@ando> <87r46vvcx0.fsf@uwakimon.sk.tsukuba.ac.jp> <20140222102051.16b65fda@fsol> <20140222105829.376aab18@fsol> <20140222111756.17bb8a4c@fsol> <20140222122717.6b7c9785@fsol> <87lhx3uyya.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <530B86EB.50700@btinternet.com> On 22/02/2014 21:26, Tim Delaney wrote: > > On 23 February 2014 02:29, Nick Coghlan > wrote: > > On 22 Feb 2014 22:15, "Stephen J. Turnbull" > wrote: > > Antoine Pitrou writes: > > > Chris Angelico > > wrote: > > > > hasattr(x,"y") <-> (x.y or True except AttributeError: False) > > > But it's not the same. hasattr() returns a boolean, not an > arbitrary > > > value. > > I think he meant > > hasattr(x,"y") <-> (x.y and True except AttributeError: False) > > With PEP 463, the explicit equivalent of hasattr() would be > something like : > > hasattr(x,"y") <-> (bool(x.y) or True except AttributeError: False) > > That would work, but I think I'd prefer: > > hasattr(x,"y") <-> bool(x.y or True except AttributeError: False) > > Makes it clearer IMO that the entire expression will always return a > boolean. > This also works: hasattr(x,"y") <-> (lambda v: True)(x.y) except AttributeError: False Which is less obscure is a matter of judgement. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Feb 24 18:58:30 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 24 Feb 2014 09:58:30 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <20140224184304.7b95f66c@fsol> References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> <530A5D53.7070108@stoneleaf.us> <20140223223745.231a5b5e@fsol> <530B7E31.8000805@stoneleaf.us> <20140224184304.7b95f66c@fsol> Message-ID: <530B8846.5080502@stoneleaf.us> On 02/24/2014 09:43 AM, Antoine Pitrou wrote: > On Mon, 24 Feb 2014 09:15:29 -0800 > Ethan Furman wrote: >> On 02/23/2014 02:54 PM, Nick Coghlan wrote: >>> >>> It's a harm containment tactic, based on the assumption people *will* >>> want to include the output of ascii() in binary protocols containing >>> ASCII segments, regardless of whether or not we consider their reasons >>> for doing so to be particularly good. >> >> One possible problem with %a -- it becomes the bytes equivalent of %s in Python 2 strings, with the minor exception of >> how unicode strings are handled (quote marks are added). In other words, instead of %d, one could use %a. >> >> On the other hand, %a is so much more user-friendly than b'%s' % ('%d' % 123).encode('ascii', errors='backslashreplace'). > > But why not b'%d' % 123 ? I was just using 123 as an example of the user-unfriendliness of the rest of that line. -- ~Ethan~ From solipsis at pitrou.net Mon Feb 24 19:06:13 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 24 Feb 2014 19:06:13 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 References: <53095562.8010304@stoneleaf.us> <20140223123352.5aa94595@fsol> <530A5D53.7070108@stoneleaf.us> <20140223223745.231a5b5e@fsol> <530B7E31.8000805@stoneleaf.us> <20140224184304.7b95f66c@fsol> <530B8846.5080502@stoneleaf.us> Message-ID: <20140224190613.5d33af4c@fsol> On Mon, 24 Feb 2014 09:58:30 -0800 Ethan Furman wrote: > On 02/24/2014 09:43 AM, Antoine Pitrou wrote: > > On Mon, 24 Feb 2014 09:15:29 -0800 > > Ethan Furman wrote: > >> On 02/23/2014 02:54 PM, Nick Coghlan wrote: > >>> > >>> It's a harm containment tactic, based on the assumption people *will* > >>> want to include the output of ascii() in binary protocols containing > >>> ASCII segments, regardless of whether or not we consider their reasons > >>> for doing so to be particularly good. > >> > >> One possible problem with %a -- it becomes the bytes equivalent of %s in Python 2 strings, with the minor exception of > >> how unicode strings are handled (quote marks are added). In other words, instead of %d, one could use %a. > >> > >> On the other hand, %a is so much more user-friendly than b'%s' % ('%d' % 123).encode('ascii', errors='backslashreplace'). > > > > But why not b'%d' % 123 ? > > I was just using 123 as an example of the user-unfriendliness of the rest of that line. The thing is, we don't have any believable example of a data type for which '%a' would be useful. IME, most formatting happens with basic data types such as str, int, etc., and '%a' can't be useful for those. Regards Antoine. From rob.cliffe at btinternet.com Mon Feb 24 19:30:52 2014 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 24 Feb 2014 18:30:52 +0000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307A34D.6050101@gmail.com> References: <5307A34D.6050101@gmail.com> Message-ID: <530B8FDC.1030506@btinternet.com> Some of your points have been answered by others, I'll try to avoid repetition. On 21/02/2014 19:04, Yury Selivanov wrote: > [snip] > > Inconvenience of dict[] raising KeyError was solved by > introducing the dict.get() method. And I think that > > dct.get('a', 'b') > > is 1000 times better than > > dct['a'] except KeyError: 'b' Do you? I don't. "Explicit is better than implicit". I think this may be partly a matter of familiarity. > >> Translate numbers to names, falling back on the numbers:: >> g = grp.getgrnam(tarinfo.gname)[2] except KeyError: >> tarinfo.gid >> u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: >> tarinfo.uid >> >> # Lib/tarfile.py:2198: >> try: >> g = grp.getgrnam(tarinfo.gname)[2] >> except KeyError: >> g = tarinfo.gid >> try: >> u = pwd.getpwnam(tarinfo.uname)[2] >> except KeyError: >> u = tarinfo.uid > This one is a valid example, but totally unparseable by > humans. Again, I find the more concise version easier to read than the original. It is dense, yes. It takes time to read and absorb, sure - but so does the original 8-line version. And it makes the repetition of the same code structure much more obvious. >> > > I think all of the above more readable with try statement. >> >> Retrieving a message from either a cache or the internet, with auth >> check:: >> >> logging.info("Message shown to user: %s",((cache[k] >> except LookupError: >> (backend.read(k) except OSError: 'Resource not available') >> ) >> if check_permission(k) else 'Access denied' >> ) except BaseException: "This is like a bare except clause") >> >> try: >> if check_permission(k): >> try: >> _ = cache[k] >> except LookupError: >> try: >> _ = backend.read(k) >> except OSError: >> _ = 'Resource not available' >> else: >> _ = 'Access denied' >> except BaseException: >> _ = "This is like a bare except clause" >> logging.info("Message shown to user: %s", _) > I think there is a consensus that this was a poor example, unless intended to show how the new construction (like any other) could be abused to produce hard-to-understand code. > If you replace '_' with a 'msg' (why did you use '_'??) > then try statements are *much* more readable. > >> [snip] >> >> Lib/ipaddress.py:343:: >> try: >> ips.append(ip.ip) >> except AttributeError: >> ips.append(ip.network_address) >> Becomes:: >> ips.append(ip.ip except AttributeError: ip.network_address) > or it may become: > > ips.append(getattr(ip, 'ip', ip.network_address)) > > or > > address = getattr(ip, 'ip', ip.network_address) > ips.append(address) Please note that any of these is an improvement on the original, in that they don't trap an AttributeError evaluating ips.append. (The old try ... except syntax can be used incorrectly, just as any new syntax can.) > > --- > > All in all, your proposal scares me. I doesn't make python > code readable, Again, a personal judgement. > it doesn't solve the problem of overbroad > exceptions handling (you have couple examples of overbroad > handling in your PEP examples section). > > Yes, some examples look neat. But your syntax is much easier > to abuse, than 'if..else' expression, Why? Any syntax can be abused. I can easily abuse if..else if I want to: ErrorMessage = (None if eggs else "No eggs") if ham else "No ham" if eggs else "No ham or eggs" # Tested Suppose you were learning Python. Which is easier, to learn lots of special methods (dict.get, getattr etc.), or to learn ONE self-explanatory form of syntax that covers them all: Dict[Key] except KeyError: default List[Index] except IndexError: default x.y except AttributeError: default (Of course, I'm not saying "Don't use getattr". Just that you could get by if you've never heard of it.) From ethan at stoneleaf.us Mon Feb 24 19:40:46 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 24 Feb 2014 10:40:46 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <53095562.8010304@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> Message-ID: <530B922E.7050709@stoneleaf.us> Okay, types corrected, most comments taken into account. %b is right out, %a is still suffering scrutiny. The arguments seem to boil down to: We don't need it. vs Somebody might, and it's better than having them inappropriately add a __bytes__ method if we don't have it. "We don't need it" doesn't really need any further explanation. Does anybody have any examples where %a could be useful? Considering the work-arounds are either wrong or painful, it wouldn't take much to sway me towards keeping it, but at the moment it seems to be a YAGNI, plus we could always add it later if it turns out to be useful. (For that matter, we could implement the main portion of the PEP now, and maybe a %a use-case will show up before 3.5 is released and we could add it then.) So, any last thoughts about %a? From v+python at g.nevcal.com Mon Feb 24 20:29:41 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 24 Feb 2014 11:29:41 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <530B922E.7050709@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> <530B922E.7050709@stoneleaf.us> Message-ID: <530B9DA5.8020209@g.nevcal.com> On 2/24/2014 10:40 AM, Ethan Furman wrote: > Somebody might, and it's better than having them inappropriately add a > __bytes__ method if we don't have it. I'll admit my first thought on reading the initial discussions about adding bytes % formatting was "Oh, if I want to display custom objects in a byte stream, just add a __bytes__ method." I don't believe there is any verbiage in the PEP (that might get transferred to the documentation) that explains why that would be a bad idea. Should there be? Whether or not %a is implemented sooner or later? -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 24 20:42:37 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 24 Feb 2014 20:42:37 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 References: <53095562.8010304@stoneleaf.us> <530B922E.7050709@stoneleaf.us> Message-ID: <20140224204237.6a513c1f@fsol> On Mon, 24 Feb 2014 10:40:46 -0800 Ethan Furman wrote: > Okay, types corrected, most comments taken into account. > > %b is right out, %a is still suffering scrutiny. > > The arguments seem to boil down to: > > We don't need it. > > vs > > Somebody might, and it's better than having them inappropriately add a __bytes__ method if we don't have it. Don't forget that Python is a language for consenting adults. Adding a near-useless feature for fear that otherwise people might shoot themselves in the foot by using another feature must be one of the worst arguments I've ever heard :-) Regards Antoine. From breamoreboy at yahoo.co.uk Mon Feb 24 20:54:06 2014 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Mon, 24 Feb 2014 19:54:06 +0000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <530B922E.7050709@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> <530B922E.7050709@stoneleaf.us> Message-ID: On 24/02/2014 18:40, Ethan Furman wrote: > Okay, types corrected, most comments taken into account. > > %b is right out, %a is still suffering scrutiny. > > The arguments seem to boil down to: > > We don't need it. > > vs > > Somebody might, and it's better than having them inappropriately add a > __bytes__ method if we don't have it. > > > "We don't need it" doesn't really need any further explanation. > > Does anybody have any examples where %a could be useful? Considering > the work-arounds are either wrong or painful, it wouldn't take much to > sway me towards keeping it, but at the moment it seems to be a YAGNI, > plus we could always add it later if it turns out to be useful. (For > that matter, we could implement the main portion of the PEP now, and > maybe a %a use-case will show up before 3.5 is released and we could add > it then.) > > So, any last thoughts about %a? I placed it under your nose https://mail.python.org/pipermail/python-dev/2014-January/131636.html but personally I wouldn't lose any sleep whether it stays or goes. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com From ethan at stoneleaf.us Mon Feb 24 21:06:23 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 24 Feb 2014 12:06:23 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: <53095562.8010304@stoneleaf.us> <530B922E.7050709@stoneleaf.us> Message-ID: <530BA63F.4030309@stoneleaf.us> On 02/24/2014 11:54 AM, Mark Lawrence wrote: > On 24/02/2014 18:40, Ethan Furman wrote: >> >> So, any last thoughts about %a? > > I placed it under your nose https://mail.python.org/pipermail/python-dev/2014-January/131636.html but personally I > wouldn't lose any sleep whether it stays or goes. So you did, sorry I forgot about it. So the argument, then, is that %a should be included because it is present in str? Note that %r, while it works for str, is rejected from this proposal (primarily because of the possibility of having non-ASCII characters); while %a doesn't suffer from that possibility (obviously ;) , I think the case needs to be made that %a is useful for including ... in a mixed binary/ASCII format, but so far nobody has filled in the ... . -- ~Ethan~ From rob.cliffe at btinternet.com Mon Feb 24 21:52:19 2014 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 24 Feb 2014 20:52:19 +0000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307E2E2.3070007@stoneleaf.us> References: <5307CDDE.8090106@canterbury.ac.nz> <5307D2AE.8030608@trueblade.com> <5307E2E2.3070007@stoneleaf.us> Message-ID: <530BB103.8040204@btinternet.com> On 21/02/2014 23:36, Ethan Furman wrote: > On 02/21/2014 02:26 PM, Eric V. Smith wrote: >> On 2/21/2014 5:06 PM, Greg Ewing wrote: >>>> On 21 February 2014 13:15, Chris Angelico wrote: >>>> >>>>> Generator expressions require parentheses, unless they would be >>>>> strictly redundant. Ambiguities with except expressions could be >>>>> resolved in the same way, forcing nested except-in-except trees to be >>>>> correctly parenthesized > > There would be no ambiguity if only nested excepts are allowed. If one > wants to catch multiple exceptions from one expression, /and do > something different for each one/, use the statement form as it's > going to be clearer. For example: > > try: > value = 1/x > except ZeroDivisionError: > try: > value = 1/default['denominator'] > except KeyError: > value = NaN > > is much cleaner as: > > value = 1/x except ZeroDivisionError: 1/default['denominator'] > except KeyError: NaN > > However, this: > > try: > result = Parse(some_stuff) > except MissingOperator: > result = ... > except InvalidOperand: > result = ... > except SomethingElse: > result = ... > > would not benefit from being condensed into a single expression > > -- > ~Ethan~ Funny, my feeling was exactly the reverse. :-) Probably because the latter seems to me to be a more natural thing to want to do (I find it easier to imagine use cases for it). And also because there is no way of getting exactly the same effect with a parenthesized except-expression: (expr except ValueError: ValueErrrorMessage") except NameError:NameErrorMessage # doesn't quite do what I want (here if expr raises a ValueError, evaluating ValueErrrorMessage which is mis-spelt will raise a NameError which will be misleadingly caught). But I guess the truth is that any except-expression which gets too long and complicated should be written some other way. From rob.cliffe at btinternet.com Mon Feb 24 21:56:17 2014 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 24 Feb 2014 20:56:17 +0000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <53080696.3050000@g.nevcal.com> References: <20140221145205.0b8b670d@fsol> <20140221183250.2b44f860@fsol> <52fb1d300a923efdc57920d4ef8f3a35@chopin.edu.pl> <53080696.3050000@g.nevcal.com> Message-ID: <530BB1F1.1050807@btinternet.com> On 22/02/2014 02:08, Glenn Linderman wrote: > On 2/21/2014 5:06 PM, Jan Kaliszewski wrote: >> Or even (still being my favorite): >> >> msg = seq[i] except (IndexError: "nothing") > > This syntax actually has a benefit: the parenthesized syntax after > except could become a list, to allow handling different exceptions > from the tried expression with different results: > > msg = seq[dictionary[i]] except (IndexError: "nothing", KeyError: > "serious problems") It shouldn't be a true list. We need lazy evaluation of the default values. And if an unlisted exception is raised, we don't want any of the defaults evaluated. Rob Cliffe > > And still allows nesting: > > msg = seq[i] except (IndexError: dictionary[i] except (KeyError: "no > fallback data for %s" % i)) > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/rob.cliffe%40btinternet.com > > > No virus found in this message. > Checked by AVG - www.avg.com > Version: 2012.0.2247 / Virus Database: 3705/6616 - Release Date: 02/22/14 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Mon Feb 24 22:08:54 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Mon, 24 Feb 2014 13:08:54 -0800 (PST) Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: Message-ID: <530bb4e6.2b2f8c0a.1dcc.3c9d@mx.google.com> Victor Stinner wrote: >> Will ascii() ever emit an antislash representation? > Try ascii(chr(0x1fffff)). In which version? I get: ValueError: chr() arg not in range(0x110000) > How do you plan to use this output? Write it into a socket or a file? > When I debug, I use print & logging which both expect text string. So I > think that b'%a' is useless. Sad Use Case 1: There is not yet a working implementation of the file or wire format. Either I am still writing it, or the file I need to parse is coming from a "partner" who "configured" rather than wrote the original program. I write (or request that they write) something recognizable to the actual stream, as a landmark. Case 1a: I want a repr of the same object that is supposedly being represented in the official format, so I can see whether the problem is bad data or bad serialization. Use Case 2: Fallback for some sort of serialization format; I expect not to ever use the fallback in production, but better something ugly than a failure, let alone a crash. Use Case 3: Shortcut for serialization of objects whose repr is "good enough". (My first instinct would probably be to implement the __bytes__ special method, but if I thought that was supposed to expose the real data, as opposed to a serialized copy, then I would go for %a.) -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From victor.stinner at gmail.com Mon Feb 24 23:10:10 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 24 Feb 2014 23:10:10 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <530bb4e6.2b2f8c0a.1dcc.3c9d@mx.google.com> References: <530bb4e6.2b2f8c0a.1dcc.3c9d@mx.google.com> Message-ID: 2014-02-24 22:08 GMT+01:00 Jim J. Jewett : >>> Will ascii() ever emit an antislash representation? Sorry, it's chr(0x10ffff): >>> print(ascii(chr(0x10ffff))) '\U0010ffff' Victor From ncoghlan at gmail.com Mon Feb 24 23:33:53 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Feb 2014 08:33:53 +1000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <20140224204237.6a513c1f@fsol> References: <53095562.8010304@stoneleaf.us> <530B922E.7050709@stoneleaf.us> <20140224204237.6a513c1f@fsol> Message-ID: On 25 Feb 2014 05:44, "Antoine Pitrou" wrote: > > On Mon, 24 Feb 2014 10:40:46 -0800 > Ethan Furman wrote: > > > Okay, types corrected, most comments taken into account. > > > > %b is right out, %a is still suffering scrutiny. > > > > The arguments seem to boil down to: > > > > We don't need it. > > > > vs > > > > Somebody might, and it's better than having them inappropriately add a __bytes__ method if we don't have it. > > Don't forget that Python is a language for consenting adults. Adding a > near-useless feature for fear that otherwise people might shoot > themselves in the foot by using another feature must be one of the > worst arguments I've ever heard :-) That's not quite the argument. The argument is that __bytes__ is expected to work in arbitrary binary contexts and hence should *never* assume ASCII compatibility (the PEP should probably propose a new addition to PEP 8 to that effect), so the question is then "OK, since it isn't defining __bytes__, what is the preferred spelling for getting the ASCII compatible representation of an object as a byte sequence?". If we do nothing, then that spelling is "ascii(obj).encode('ascii')". If %a is allowed as part of a binary interpolation pattern, then it becomes "b'%a' % obj" Allowing %a also improves the consistency with text interpolation. In the case of %r, the inconsistency is based on needing to disallow arbitrary Unicode code points in the result and not wanting to redefine %r as a second way to spell %a. There's no corresponding reason to disallow %a - the result is guaranteed to be ASCII compatible, so there's no risk of data driven encoding errors, and no difference between doing the binary interpolation directly, or doing text interpolation and then encoding the result as ASCII. As far as use cases go, as someone else mentioned, the main one is likely to be binary logging and error reporting formats, as it becomes a quick and easy way to embed a backslash escaped string. However, my interest is more in providing an obvious way to do it and in minimising the differences between text and binary interpolation. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 24 23:40:45 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 24 Feb 2014 23:40:45 +0100 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: <53095562.8010304@stoneleaf.us> <530B922E.7050709@stoneleaf.us> <20140224204237.6a513c1f@fsol> Message-ID: <20140224234045.2718d80d@fsol> On Tue, 25 Feb 2014 08:33:53 +1000 Nick Coghlan wrote: > As far as use cases go, as someone else mentioned, the main one is likely > to be binary logging and error reporting formats, as it becomes a quick and > easy way to embed a backslash escaped string. That's a fringe use case, though. Also, your binary logging format probably has a well-defined character set that's not necessarily ASCII (perhaps UTF-8), so using the proposed "%a" is sub-optimal and potentially confusing (if lots of non-ASCII characters get escaped as \uxxxx). > However, my interest is more > in providing an obvious way to do it and in minimising the differences > between text and binary interpolation. That sounds very theoretical. Regards Antoine. From ethan at stoneleaf.us Mon Feb 24 23:45:22 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 24 Feb 2014 14:45:22 -0800 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: <53095562.8010304@stoneleaf.us> <530B922E.7050709@stoneleaf.us> <20140224204237.6a513c1f@fsol> Message-ID: <530BCB82.7030805@stoneleaf.us> On 02/24/2014 02:33 PM, Nick Coghlan wrote: > > Allowing %a also improves the consistency with text interpolation. In the case of %r, the inconsistency is based on > needing to disallow arbitrary Unicode code points in the result and not wanting to redefine %r as a second way to spell > %a. There's no corresponding reason to disallow %a - the result is guaranteed to be ASCII compatible, so there's no risk > of data driven encoding errors, and no difference between doing the binary interpolation directly, or doing text > interpolation and then encoding the result as ASCII. > > As far as use cases go, as someone else mentioned, the main one is likely to be binary logging and error reporting > formats, as it becomes a quick and easy way to embed a backslash escaped string. However, my interest is more in > providing an obvious way to do it and in minimising the differences between text and binary interpolation. Jim Jewett had some use-cases that I'm happy to run with. (Thanks jJ!) So final question for %a: %a can only be used in Python 3 (3.2+, I believe) -- do we want to be able to use %a as a short way of including text? In Python2/3 code bases it will need to be '%s' % 'a string'.encode('ascii'). In Python 3 only code bases that could be shortened to '%a' % 'a string': pro: much easier if "mojibake" ( \x and \u sequences ) sneak in, the original data can still be retrieved cons: has surrounding quotes (would need to have bytes.__mod__ remove them) -- ~Ethan~ From jimjjewett at gmail.com Tue Feb 25 00:54:26 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Mon, 24 Feb 2014 15:54:26 -0800 (PST) Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307A34D.6050101@gmail.com> Message-ID: <530bdbb2.8c79e00a.3cc3.5b81@mx.google.com> Yury Selivanov wrote: > I think the Motivation section is pretty weak. I have normally wished for this when I was (semi- interactively) exploring a weakly structured dataset. Often, I start with a string, split it into something hopefully like records, and then start applying filters and transforms. I would prefer to write a comprehension instead of a for loop. Alas, without pre-editing, I can be fairly confident that the data is dirty. Sometimes I can solve it with a filter (assuming that I remember and don't mind the out-of-order evaluation): # The "if value" happens first, # so the 1/value turns out to be safe. [1/value for value in working_list if value] Note that this means dropping the bad data, so that items in this list will have different indices than those in the parent working_list. I would rather have written: [1/value except (TypeError, ZeroDivisionError): None] which would keep the matching indices, and clearly indicate where I now had missing/invalid data. Sometimes I solve it with a clumsy workaround: sum((e.weight if hasattr(e, 'weight') else 1.0) for e in working_list) But the "hasattr" implies that I am doing some sort of classification based on whether or not the element has a weight. The true intent was to recognize that while every element does have a weight, the representation that I'm starting from didn't always bother to store it -- so I am repairing that before processing. sum(e.weight except AttributeError: 1) Often I give up, and create a junky helper function, or several. But to avoid polluting the namespace, I may leave it outside the class, or give it a truly bad name: def __only_n2(worklist): results = [] for line in worklist: line=line.strip() if not line: # or maybe just edit the input file... continue split1=line.split(", ") if 7 != len(split1): continue if "n2" == split1[3]: results.append(split1) return results worklist_n2 = __only_n2(worklist7) In real life code, even after hand-editing the input data to fix a few cases, I recently ended up with: class VoteMark: ... @classmethod def from_property(cls, voteline): # print (voteline) count, _junk, prefs = voteline.partition(": ") return cls(count, prefs) ... # module level scope def make_votes(vs=votestring): return [VoteMark.from_property(e) for e in vs.splitlines()] vs=make_votes() You can correctly point out that I was being sloppy, and that I *should* have gone back to clean it up. But I wouldn't have had to clean up either the code or the data (well, not as much), if I had been able to just keep the step-at-a-time transformations I was building up during development: vs=[(VoteMark(*e.strip().split(": ")) except (TypeError, ValueError): None) for e in votestring.splitlines()] Yes, the first line is still doing too much, and might be worth a helper function during cleanup. But it is already better than an alternate constructor that exists only to simplify a single (outside the class) function that is only called once. Which in turn is better than the first draft that was so ugly that I actually did fix it during that same work session. > Inconvenience of dict[] raising KeyError was solved by > introducing the dict.get() method. And I think that > dct.get('a', 'b') > is 1000 times better than > dct['a'] except KeyError: 'b' I don't. dct.get('a', default='b') would be considerably better, but it would still imply that missing values are normal. So even after argclinic is fully integrated, there will still be times when I prefer to make it explicit that I consider this an abnormal case. (And, as others have pointed out, .get isn't a good solution when the default is expensive to compute.) >> Consider this example of a two-level cache:: >> for key in sequence: >> x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key))) > I'm sorry, it took me a minute to understand what your > example is doing. I would rather see two try..except blocks > than this. Agreed -- like my semi-interactive code above, it does too much on one line. I don't object as much to: for key in sequence: x = (lvl1[key] except KeyError: (lvl2[key] except KeyError: f(key))) >> Retrieve an argument, defaulting to None:: >> cond = args[1] except IndexError: None >> >> # Lib/pdb.py:803: >> try: >> cond = args[1] >> except IndexError: >> cond = None > cond = None if (len(args) < 2) else args[1] This is an area where tastes will differ. I view the first as saying that not having a cond would be unusual, or at least a different kind of call. I view your version as a warning that argument parsing will be complex, and that there may be specific combinations of arguments that are only valid depending on the values of other arguments. Obviously, not everyone will share that intuition, but looking at the actual code, the first serves me far better. (It is a do_condition method, and falsy values -- such as None -- trigger a clear rather than a set.) >> Attempt a translation, falling back on the original:: >> e.widget = self._nametowidget(W) except KeyError: W >> >> # Lib/tkinter/__init__.py:1222: >> try: >> e.widget = self._nametowidget(W) >> except KeyError: >> e.widget = W Note that if the value were being stored in a variable instead of an attribute, it would often be written more like: try: W=_nametowidget(W) except: pass > I'm not sure this is a good example either. ... > Your new syntax just helps to work with this error prone api. Exactly. You think that is bad, because it encourages use of a sub-optimal API. I think it is good, because getting an external API fixed is ... unlikely to happen. > # sys.abiflags may not be defined on all platforms. > _CONFIG_VARS['abiflags'] = sys.abiflags except AttributeError: '' > Ugly. > _CONFIG_VARS['abiflags'] = getattr(sys, 'abiflags', '') > Much more readable. Again, tastes differ. To me, getattr looks too much like internals, and I wonder if I will need to look up getattribute or __getattr__, because maybe this code is doing something strange. After an unpleasant pause, I realize that, no, it really is safe. Then later, I wonder if _CONFIG_VARS is some odd mapping with special limits. Then I see that it doesn't matter, because that isn't what you're passing to getattr. Then I remember that sys often is a special case, and start wondering if I need extra tests around this. Wait, why was I looking at this code again? (And as others pointed out, getattr with a constant has its own code smell.) The "except" form clearly indicates that sys.abiflags *ought* to be there, and the code is just providing some (probably reduced) services to oddball systems, instead of failing. >> Retrieve an indexed item, defaulting to None (similar to dict.get):: >> def getNamedItem(self, name): >> return self._attrs[name] except KeyError: None > _attrs there is a dict (or at least it's something that quaks > like a dict, and has [] and keys()), so > return self._attrs.get(name) Why do you assume it has .get? Or that .get does what a python mapping normally does with .get? The last time I dove into code like that, it was written that way specifically because the DOM (and _attrs in particular) might well be created by some other program, in support of a very unpythonic API. Note that the method itself is called getNamedItem, rather than just get; that also suggests an external source of API expectations. >> Translate numbers to names, falling back on the numbers:: >> g = grp.getgrnam(tarinfo.gname)[2] except KeyError: tarinfo.gid >> u = pwd.getpwnam(tarinfo.uname)[2] except KeyError: tarinfo.uid >> >> # Lib/tarfile.py:2198: >> try: >> g = grp.getgrnam(tarinfo.gname)[2] >> except KeyError: >> g = tarinfo.gid >> try: >> u = pwd.getpwnam(tarinfo.uname)[2] >> except KeyError: >> u = tarinfo.uid > This one is a valid example, but totally unparseable by > humans. Moreover, it promotes a bad pattern, as you > mask KeyErrors in 'grp.getgrnam(tarinfo.gname)' call. Do you find the existing try: except: code any better, or are you just worried that it doesn't solve enough of the problem? FWIW, I think it doesn't matter where the KeyError came from; even if the problem were finding the getgrnam method, the right answer (once deployed, as opposed to during development) would still be to fall back to the best available information. >> Perform some lengthy calculations in EAFP mode, handling division by >> zero as a sort of sticky NaN:: >> value = calculate(x) except ZeroDivisionError: float("nan") vs >> try: >> value = calculate(x) >> except ZeroDivisionError: >> value = float("nan") > I think all of the above more readable with try statement. I don't, though I would wrap the except. The calculation is important; everything else is boilerplate, and the more you can get it out of the way, the better. > Yes, some examples look neat. But your syntax is much easier > to abuse, than 'if..else' expression, and if people start > abusing it, Python will simply loose it's readability > advantage. This is also an argument for mandatory parentheses. ()-continuation makes it easier to wrap the except clause out of the way. (((( )) (() ()) ))-proliferation provides pushback when the expressions start to get too complicated. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From jimjjewett at gmail.com Tue Feb 25 01:27:44 2014 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Mon, 24 Feb 2014 16:27:44 -0800 (PST) Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <5307E359.6080603@stoneleaf.us> Message-ID: <530be380.461ce00a.586c.ffffadd5@mx.google.com> Greg Ewing suggested: >> This version might be more readable: >> >> value = lst[2] except "No value" if IndexError Ethan Furman asked: > It does read nicely, and is fine for the single, non-nested, case > (which is probably the vast majority), but how would > it handle nested exceptions? With parentheses. Sometimes, the parentheses will make a complex expression ugly. Sometimes, a complex expression should really be factored into pieces anyway. Hopefully, these times are highly correlated. The above syntax does lend itself somewhat naturally to multiple *short* except clauses: value = (lst[2] except "No value" if IndexError except "Bad Input" if TypeError) and nested exception expressions are at least possible, but deservedly ugly: value = (lvl1[name] except (lvl2[name] except (compute_new_answer(name) except None if AppValueError) if KeyError) if KeyError) This also makes me wonder whether the cost of a subscope (for exception capture) could be limited to when an exception actually occurs, and whether that might lower the cost enough to make the it a good tradeoff. def myfunc1(a, b, e): assert "main scope e value" == e e = "main scope e value" value = (myfunc1(val1, val2, e) except e.reason if AppError as e) assert "main scope e value" == e -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From rosuav at gmail.com Tue Feb 25 01:42:29 2014 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 25 Feb 2014 11:42:29 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <530be380.461ce00a.586c.ffffadd5@mx.google.com> References: <5307E359.6080603@stoneleaf.us> <530be380.461ce00a.586c.ffffadd5@mx.google.com> Message-ID: On Tue, Feb 25, 2014 at 11:27 AM, Jim J. Jewett wrote: > This also makes me wonder whether the cost of a subscope > (for exception capture) could be limited to when an > exception actually occurs, and whether that might lower > the cost enough to make the it a good tradeoff. > > def myfunc1(a, b, e): > assert "main scope e value" == e > > e = "main scope e value" > value = (myfunc1(val1, val2, e) > except e.reason if AppError as e) > assert "main scope e value" == e I'm sure it could. But there aren't many use-cases. Look at the one example I was able to find in the stdlib: http://legacy.python.org/dev/peps/pep-0463/#capturing-the-exception-object It's hardly a shining example of the value of the proposal. Got any really awesome demo that requires 'as'? Most of the supporting examples use something like KeyError where it's simply "was an exception thrown or not". ChrisA From christian at python.org Tue Feb 25 08:39:40 2014 From: christian at python.org (Christian Heimes) Date: Tue, 25 Feb 2014 08:39:40 +0100 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() Message-ID: <530C48BC.2070804@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, this looks pretty serious -- and it caught me off guard, too. :( https://www.trustedsec.com/february-2014/python-remote-code-execution-socket-recvfrom_into/ Next time please inform the Python Security Response Team about any and all issues that are related to buffer overflows or similar bugs. In fact please drop a note about anything that even remotely look like an exploitable issue. Even public bug reports should be forwarded to PSRT. I have requested a CVE number. How about security releases? The upcoming 3.3 and 3.4 release should contain the fix (not verified yet). Python 2.7 to 3.2 will need a security release, though. Regards Christian -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCgAGBQJTDEi1AAoJEMeIxMHUVQ1FdAwP/j36bioIzz+kFvX9AEo2Bxtq H+JsvRxiWHJrXLG0YUf1AolW+s92/2dRAYLq86DQa7PK2rvrqR4bQUOP+fLi9hdT 5b9YF4mHhBtte9lTDwESYw4IXtoOz4gbhXpY/dGGLjiEeWYNgRl40xSZYXf6cZfR okRRE0c6EZ9WnAYWl1vW1oUzPjua0KOpVOhvabog/YNSPL3SW8shWANpu0fg/n+G guBYTP90pgUEz7Jc20xeVAB9BeZoC/jjDPv1QRMu+PWjyFeaI4iLdNe3loRXBYy1 xmzHxQACzQR45lxAzCoBwBopC49JIF7o7pnTBrY9id9j0yRMAPC/N1uQCceLO1yc GoKardxzUT9IX++yfLTOYwdGnpXDQeXUbHAImcWNGMN8QfsWUFBezPmqKM7tfTNR I/khqTaLPewr58z4d/erfJ5wSEHVdyWASmUWGniS9jjfFNVBDNA2pSPBCP9TJhK5 30BOnKB+MMNG+LCe5chiQOyKje/pbfwrEwwrdiJYCOSXK+w/hbPClNBBq4w9XXVk sIIk5xO1IZ4rMG/YLkg9vaWzn7Yi6O0GJXOmQWp+22kYwaQK+3l6qqSSo2laMVN8 c6sLFng3loO1v3SDO0AOTTU2VcdsS0SdYLkEXMwHK/tRWeXWrwC5HrYnFS0Hu0iI EQlaImb433lu8mHrvPEx =K5ZL -----END PGP SIGNATURE----- From stuart at stuartbishop.net Tue Feb 25 08:43:00 2014 From: stuart at stuartbishop.net (Stuart Bishop) Date: Tue, 25 Feb 2014 14:43:00 +0700 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <53095562.8010304@stoneleaf.us> References: <53095562.8010304@stoneleaf.us> Message-ID: On 23 February 2014 08:56, Ethan Furman wrote: > ``%a`` will call :func:``ascii()`` on the interpolated value's > :func:``repr()``. > This is intended as a debugging aid, rather than something that should be > used > in production. Non-ascii values will be encoded to either ``\xnn`` or > ``\unnnn`` > representation. So we use %a for exactly the same purposes that we used to use %r. > Unsupported codes > ----------------- > > ``%r`` (which calls ``__repr__`` and returns a :class:`str`) is not > supported. But you propose changing the code. I think there would have been a lot less discussion if you just defined %r to do what you propose for %a, as everything would work as people expected. -- Stuart Bishop http://www.stuartbishop.net/ From ncoghlan at gmail.com Tue Feb 25 08:53:49 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Feb 2014 17:53:49 +1000 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: <530C48BC.2070804@python.org> References: <530C48BC.2070804@python.org> Message-ID: On 25 February 2014 17:39, Christian Heimes wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Hi, > > this looks pretty serious -- and it caught me off guard, too. :( > > https://www.trustedsec.com/february-2014/python-remote-code-execution-socket-recvfrom_into/ > > Next time please inform the Python Security Response Team about any > and all issues that are related to buffer overflows or similar bugs. > In fact please drop a note about anything that even remotely look like > an exploitable issue. Even public bug reports should be forwarded to PSRT. > > I have requested a CVE number. How about security releases? The > upcoming 3.3 and 3.4 release should contain the fix (not verified > yet). I've checked these, and noted the relevant hg.python.org links on the tracker issue at http://bugs.python.org/issue20246 > Python 2.7 to 3.2 will need a security release, though. Agreed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Feb 25 09:27:13 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Feb 2014 18:27:13 +1000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: <53095562.8010304@stoneleaf.us> Message-ID: On 25 February 2014 17:43, Stuart Bishop wrote: > On 23 February 2014 08:56, Ethan Furman wrote: > >> ``%a`` will call :func:``ascii()`` on the interpolated value's >> :func:``repr()``. >> This is intended as a debugging aid, rather than something that should be >> used >> in production. Non-ascii values will be encoded to either ``\xnn`` or >> ``\unnnn`` >> representation. > > So we use %a for exactly the same purposes that we used to use %r. > >> Unsupported codes >> ----------------- >> >> ``%r`` (which calls ``__repr__`` and returns a :class:`str`) is not >> supported. > > But you propose changing the code. > > I think there would have been a lot less discussion if you just > defined %r to do what you propose for %a, as everything would work as > people expected. No, it wouldn't. [Python 3] >>> "%r" % "\xe9" "'?'" >>> "%a" % "\xe9" "'\\xe9'" %r is being disallowed in PEP 461 because it doesn't guarantee ASCII compatibility in Python 3 the way it did in Python 2. That's not up for discussion, as having %r behave like %a in binary interpolation but not in text interpolation would be far too confusing. However, %a *already* guarantees ASCII compatible output for text interpolation (by escaping everything below 0x20 or above 0x7F, the same way %r did in Python 2), so some of us think %a *should* be allowed for consistency with text interpolation, both because there's no compelling reason to disallow it and because it's the obvious way to interpolate representations of arbitrary objects into binary formats that contain ASCII compatible segments. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From victor.stinner at gmail.com Tue Feb 25 10:13:07 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 25 Feb 2014 10:13:07 +0100 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: Hi, 2014-02-25 8:53 GMT+01:00 Nick Coghlan : > I've checked these, and noted the relevant hg.python.org links on the > tracker issue at http://bugs.python.org/issue20246 Would it be possible to have a table with all known Python security vulnerabilities and the Python versions which are fixed? Bonus point if we provide a link to the changeset fixing it for each branch. Maybe put this table on http://www.python.org/security/ ? Last issues: - hash DoS - sock.recvfrom_into() - DoS with very long lines in HTTP, FTP, etc. protocols - etc. Victor From victor.stinner at gmail.com Tue Feb 25 11:07:13 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 25 Feb 2014 11:07:13 +0100 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: <530C48BC.2070804@python.org> References: <530C48BC.2070804@python.org> Message-ID: Hi, 2014-02-25 8:39 GMT+01:00 Christian Heimes : > this looks pretty serious -- and it caught me off guard, too. :( > https://www.trustedsec.com/february-2014/python-remote-code-execution-socket-recvfrom_into/ I don't think that the issue is critical. Extract of the article "Diving into SocketServer() luckily socket.recvfrom_into() isn?t even used". In fact, I didn't find any usage of the method except of unit test. Do you know which applications are vulnerable? Victor From gitaarik at gmail.com Tue Feb 25 12:25:27 2014 From: gitaarik at gmail.com (Rik) Date: Tue, 25 Feb 2014 12:25:27 +0100 Subject: [Python-Dev] Running 2.7 tests on OS X Message-ID: I want to try to submit a patch for 2.7, but I don't know how to run the tests for the 2.7 branch. `./configure` doesn't seem to create a `python.exe` file on the 2.7 branch on OS X Mavericks, and I do need this file according to this guide: http://docs.python.org/devguide/ Anybody know how I should do this? -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Feb 25 12:46:30 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 25 Feb 2014 12:46:30 +0100 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() References: <530C48BC.2070804@python.org> Message-ID: <20140225124630.22c03670@fsol> On Tue, 25 Feb 2014 08:39:40 +0100 Christian Heimes wrote: > > this looks pretty serious -- and it caught me off guard, too. :( > > https://www.trustedsec.com/february-2014/python-remote-code-execution-socket-recvfrom_into/ > > Next time please inform the Python Security Response Team about any > and all issues that are related to buffer overflows or similar bugs. > In fact please drop a note about anything that even remotely look like > an exploitable issue. Even public bug reports should be forwarded to PSRT. If that's the case, then can't we have an email hook on bugs.python.org every time an issue is classified as security? (either when created or later when modified) "Bug reports should be forwarded to PSRT" just adds a tedious and unnecessary manual step. Regards Antoine. From solipsis at pitrou.net Tue Feb 25 12:49:23 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 25 Feb 2014 12:49:23 +0100 Subject: [Python-Dev] Running 2.7 tests on OS X References: Message-ID: <20140225124923.6188a902@fsol> Hi Rik, On Tue, 25 Feb 2014 12:25:27 +0100 Rik wrote: > I want to try to submit a patch for 2.7, but I don't know how to run the > tests for the 2.7 branch. `./configure` doesn't seem to create a > `python.exe` file on the 2.7 branch on OS X Mavericks, and I do need this > file according to this guide: > http://docs.python.org/devguide/ To quote the devguide, the complete command is: ./configure --with-pydebug && make -j2 which you can decompose into: ./configure --with-pydebug make -j2 (note: "--with-pydebug" is kind of optional, but enables many debugging guards and assertions that help catch low-level issues; it also makes the resulting executable slower, somewhere between 2x and 10x slower) python.exe will be created by the "make" step, not the "./configure" step. (as a side note, "-j2" means two processes will be used concurrently for compiling; you can adjust according to the number of CPU cores in your system) Regards Antoine. From fijall at gmail.com Tue Feb 25 13:59:16 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 25 Feb 2014 14:59:16 +0200 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: On Tue, Feb 25, 2014 at 11:13 AM, Victor Stinner wrote: > Hi, > > 2014-02-25 8:53 GMT+01:00 Nick Coghlan : >> I've checked these, and noted the relevant hg.python.org links on the >> tracker issue at http://bugs.python.org/issue20246 > > Would it be possible to have a table with all known Python security > vulnerabilities and the Python versions which are fixed? Bonus point > if we provide a link to the changeset fixing it for each branch. Maybe > put this table on http://www.python.org/security/ ? > > Last issues: > - hash DoS is this fixed? From donald at stufft.io Tue Feb 25 14:01:29 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 25 Feb 2014 08:01:29 -0500 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: On Feb 25, 2014, at 7:59 AM, Maciej Fijalkowski wrote: > On Tue, Feb 25, 2014 at 11:13 AM, Victor Stinner > wrote: >> Hi, >> >> 2014-02-25 8:53 GMT+01:00 Nick Coghlan : >>> I've checked these, and noted the relevant hg.python.org links on the >>> tracker issue at http://bugs.python.org/issue20246 >> >> Would it be possible to have a table with all known Python security >> vulnerabilities and the Python versions which are fixed? Bonus point >> if we provide a link to the changeset fixing it for each branch. Maybe >> put this table on http://www.python.org/security/ ? >> >> Last issues: >> - hash DoS > > is this fixed? > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io It is in 3.4. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From fijall at gmail.com Tue Feb 25 14:03:54 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 25 Feb 2014 15:03:54 +0200 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: On Tue, Feb 25, 2014 at 3:01 PM, Donald Stufft wrote: > > On Feb 25, 2014, at 7:59 AM, Maciej Fijalkowski wrote: > >> On Tue, Feb 25, 2014 at 11:13 AM, Victor Stinner >> wrote: >>> Hi, >>> >>> 2014-02-25 8:53 GMT+01:00 Nick Coghlan : >>>> I've checked these, and noted the relevant hg.python.org links on the >>>> tracker issue at http://bugs.python.org/issue20246 >>> >>> Would it be possible to have a table with all known Python security >>> vulnerabilities and the Python versions which are fixed? Bonus point >>> if we provide a link to the changeset fixing it for each branch. Maybe >>> put this table on http://www.python.org/security/ ? >>> >>> Last issues: >>> - hash DoS >> >> is this fixed? >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io > > It is in 3.4. Oh, I thought security fixes go to all python releases. From rosuav at gmail.com Tue Feb 25 14:06:01 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Feb 2014 00:06:01 +1100 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: On Tue, Feb 25, 2014 at 11:59 PM, Maciej Fijalkowski wrote: >> Last issues: >> - hash DoS > > is this fixed? Yes, hash randomization was added as an option in 2.7.3 or 2.7.4 or thereabouts, and is on by default in 3.3+. You do have to set an environment variable for 2.7 (and I think 2.6 got that too (??)), as it can break code. ChrisA From fijall at gmail.com Tue Feb 25 14:07:20 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 25 Feb 2014 15:07:20 +0200 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: On Tue, Feb 25, 2014 at 3:06 PM, Chris Angelico wrote: > On Tue, Feb 25, 2014 at 11:59 PM, Maciej Fijalkowski wrote: >>> Last issues: >>> - hash DoS >> >> is this fixed? > > Yes, hash randomization was added as an option in 2.7.3 or 2.7.4 or > thereabouts, and is on by default in 3.3+. You do have to set an > environment variable for 2.7 (and I think 2.6 got that too (??)), as > it can break code. No, the hash randomization is broken, it does not provide enough randomness (without changing the hash function which only happened in 3.4+) From donald at stufft.io Tue Feb 25 14:08:09 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 25 Feb 2014 08:08:09 -0500 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: On Feb 25, 2014, at 8:06 AM, Chris Angelico wrote: > On Tue, Feb 25, 2014 at 11:59 PM, Maciej Fijalkowski wrote: >>> Last issues: >>> - hash DoS >> >> is this fixed? > > Yes, hash randomization was added as an option in 2.7.3 or 2.7.4 or > thereabouts, and is on by default in 3.3+. You do have to set an > environment variable for 2.7 (and I think 2.6 got that too (??)), as > it can break code. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io Hash randomization is broken and doesn?t fix anything. It?s only SipHash in 3.4+ that actually fixes it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue Feb 25 14:10:13 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 25 Feb 2014 08:10:13 -0500 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: <58644C44-69D8-4FA5-A3C6-8EB7E3FEC46A@stufft.io> On Feb 25, 2014, at 8:07 AM, Maciej Fijalkowski wrote: > On Tue, Feb 25, 2014 at 3:06 PM, Chris Angelico wrote: >> On Tue, Feb 25, 2014 at 11:59 PM, Maciej Fijalkowski wrote: >>>> Last issues: >>>> - hash DoS >>> >>> is this fixed? >> >> Yes, hash randomization was added as an option in 2.7.3 or 2.7.4 or >> thereabouts, and is on by default in 3.3+. You do have to set an >> environment variable for 2.7 (and I think 2.6 got that too (??)), as >> it can break code. > > No, the hash randomization is broken, it does not provide enough > randomness (without changing the hash function which only happened in > 3.4+) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io More information available here: http://legacy.python.org/dev/peps/pep-0456/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rosuav at gmail.com Tue Feb 25 14:12:15 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Feb 2014 00:12:15 +1100 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: On Wed, Feb 26, 2014 at 12:07 AM, Maciej Fijalkowski wrote: > No, the hash randomization is broken, it does not provide enough > randomness (without changing the hash function which only happened in > 3.4+) Hmm, I don't remember reading about that - got a link to more info? Or was that report kept quieter? ChrisA From solipsis at pitrou.net Tue Feb 25 14:17:28 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 25 Feb 2014 14:17:28 +0100 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() References: <530C48BC.2070804@python.org> Message-ID: <20140225141728.40090f42@fsol> On Tue, 25 Feb 2014 08:08:09 -0500 Donald Stufft wrote: > > Hash randomization is broken and doesn?t fix anything. Not sure what you mean with "doesn't fix anything". Hash collisions were easy to exploit pre-hash randomization, they doesn't seem as easy to exploit with it. Regards Antoine. From donald at stufft.io Tue Feb 25 14:21:46 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 25 Feb 2014 08:21:46 -0500 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: <20140225141728.40090f42@fsol> References: <530C48BC.2070804@python.org> <20140225141728.40090f42@fsol> Message-ID: <7C0D41B8-8F3D-4203-885F-BFB38D42F53A@stufft.io> On Feb 25, 2014, at 8:17 AM, Antoine Pitrou wrote: > On Tue, 25 Feb 2014 08:08:09 -0500 > Donald Stufft wrote: >> >> Hash randomization is broken and doesn?t fix anything. > > Not sure what you mean with "doesn't fix anything". Hash collisions were > easy to exploit pre-hash randomization, they doesn't seem as easy to > exploit with it. Instead of pre-generating one set of values that can be be used to DoS things you have to pre-generate 256 sets of values and try them until you get the right one. It?s like putting on armor made of paper and saying it?s harder to stab you now. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From rosuav at gmail.com Tue Feb 25 14:32:55 2014 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 26 Feb 2014 00:32:55 +1100 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: <7C0D41B8-8F3D-4203-885F-BFB38D42F53A@stufft.io> References: <530C48BC.2070804@python.org> <20140225141728.40090f42@fsol> <7C0D41B8-8F3D-4203-885F-BFB38D42F53A@stufft.io> Message-ID: On Wed, Feb 26, 2014 at 12:21 AM, Donald Stufft wrote: > Instead of pre-generating one set of values that can be be used to DoS things > you have to pre-generate 256 sets of values and try them until you get the > right one. It?s like putting on armor made of paper and saying it?s harder to > stab you now. Paper armor? You mean the stuff they tested here? http://www.imdb.com/title/tt1980597/ Cutting the problem by a factor of 256 isn't nothing, but yes, it's still a problem. But why is it only 256? I tried Googling for 'python hash randomization problem' and other keywords but all I can find is the original. I'm sure someone's done the analysis somewhere as to why it's still a problem in 2.7, and why (presumably) it can't be fixed in 2.7 - after all, that's the version that's not going anywhere any time soon. ChrisA From ncoghlan at gmail.com Tue Feb 25 14:33:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Feb 2014 23:33:27 +1000 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: On 25 Feb 2014 23:09, "Maciej Fijalkowski" wrote: > > On Tue, Feb 25, 2014 at 3:06 PM, Chris Angelico wrote: > > On Tue, Feb 25, 2014 at 11:59 PM, Maciej Fijalkowski wrote: > >>> Last issues: > >>> - hash DoS > >> > >> is this fixed? > > > > Yes, hash randomization was added as an option in 2.7.3 or 2.7.4 or > > thereabouts, and is on by default in 3.3+. You do have to set an > > environment variable for 2.7 (and I think 2.6 got that too (??)), as > > it can break code. > > No, the hash randomization is broken, it does not provide enough > randomness (without changing the hash function which only happened in > 3.4+) The blind hash collision DOS attack was fixed in all applicable branches. There was then a second vulnerability in the randomisation that still allowed a relatively straightforward invocation specific secret recovery attack against earlier versions that is only fixed with the SipHash change in 3.4. You and the other PyPy devs apparently feel that the existence of the second vulnerability means it isn't worth your while to fix the original one either. While this mirrors the core team's original position that it was up to applications and frameworks to deal with the problem, conflating the two vulnerabilities like that is still just your perspective, not ours (in addition to all the same measures that limited the impact of the original issue, there are many measures that specifically mitigate the latter one, with process recycling being one of the simplest). Regards, Nick. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 25 14:41:54 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Feb 2014 23:41:54 +1000 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: <7C0D41B8-8F3D-4203-885F-BFB38D42F53A@stufft.io> References: <530C48BC.2070804@python.org> <20140225141728.40090f42@fsol> <7C0D41B8-8F3D-4203-885F-BFB38D42F53A@stufft.io> Message-ID: On 25 Feb 2014 23:23, "Donald Stufft" wrote: > > > On Feb 25, 2014, at 8:17 AM, Antoine Pitrou wrote: > > > On Tue, 25 Feb 2014 08:08:09 -0500 > > Donald Stufft wrote: > >> > >> Hash randomization is broken and doesn't fix anything. > > > > Not sure what you mean with "doesn't fix anything". Hash collisions were > > easy to exploit pre-hash randomization, they doesn't seem as easy to > > exploit with it. > > Instead of pre-generating one set of values that can be be used to DoS things > you have to pre-generate 256 sets of values and try them until you get the > right one. It's like putting on armor made of paper and saying it's harder to > stab you now. This isn't quite correct - the hash randomisation can at least be combined with aggressive process recycling to present a moving target that is harder to attack. Without any hash randomisation at all, process recycling can't help in the slightest. SIPHash is still the real fix, although the reality remains that an attacker that really wants to bring a site down is likely to achieve their aims, regardless of whether or not there's a specific DoS vulnerability in the application server. Cheers, Nick. > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Feb 25 15:20:24 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 25 Feb 2014 09:20:24 -0500 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: <7F5E4631-690D-4E3D-8B97-AD3A65AF3BA2@stufft.io> On Feb 25, 2014, at 8:33 AM, Nick Coghlan wrote: > > On 25 Feb 2014 23:09, "Maciej Fijalkowski" wrote: > > > > On Tue, Feb 25, 2014 at 3:06 PM, Chris Angelico wrote: > > > On Tue, Feb 25, 2014 at 11:59 PM, Maciej Fijalkowski wrote: > > >>> Last issues: > > >>> - hash DoS > > >> > > >> is this fixed? > > > > > > Yes, hash randomization was added as an option in 2.7.3 or 2.7.4 or > > > thereabouts, and is on by default in 3.3+. You do have to set an > > > environment variable for 2.7 (and I think 2.6 got that too (??)), as > > > it can break code. > > > > No, the hash randomization is broken, it does not provide enough > > randomness (without changing the hash function which only happened in > > 3.4+) > > The blind hash collision DOS attack was fixed in all applicable branches. There was then a second vulnerability in the randomisation that still allowed a relatively straightforward invocation specific secret recovery attack against earlier versions that is only fixed with the SipHash change in 3.4. > > You and the other PyPy devs apparently feel that the existence of the second vulnerability means it isn't worth your while to fix the original one either. While this mirrors the core team's original position that it was up to applications and frameworks to deal with the problem, conflating the two vulnerabilities like that is still just your perspective, not ours (in addition to all the same measures that limited the impact of the original issue, there are many measures that specifically mitigate the latter one, with process recycling being one of the simplest). > > I don?t have a PoC but my gut is that process cycling is not nearly enough to mitigate the second class of attack. You?d only need 256 * N requests to recover the secret where N is the number of samples you need to factor out network noise. However if you decide you don?t care about recovering the secret and you just care about DoSing a server you can just hit it with colliding values for all 256 possible values. 255 of them will complete quickly and one of them will take a long time, repeat this a few times and you?ll have the server DoS?d in a short amount of time. I?m not sure what you mean by I don?t think it?s worth my while. I don?t run any project where I would fix this in. I do agree with the sentiment of the PyPy developers that implementing this fix is more or less pointless for PyPy because it?s so trivially worked around. Saying that the original fix fixed the original issue is taking a very narrow view of what the original issue even is. It?s not *wrong* to take that view, but I don?t think it?s very useful outside the context of not wanting to admit that the original fix wasn?t good enough. I *do* believe that calling it fixed is misleading to people who will assume it means they no longer have to worry about a trivial DoS via hash collisions when they still do need to, just slightly different than before. In the end, it?s good that it was fixed in 3.4, I wish it had been back ported and applied to 2.7 and the relevant 3.x branches. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From benjamin at python.org Tue Feb 25 15:41:17 2014 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 25 Feb 2014 06:41:17 -0800 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: <530C48BC.2070804@python.org> References: <530C48BC.2070804@python.org> Message-ID: <1393339277.26501.87591749.13312CA9@webmail.messagingengine.com> On Mon, Feb 24, 2014, at 11:39 PM, Christian Heimes wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > Hi, > > this looks pretty serious -- and it caught me off guard, too. :( > > https://www.trustedsec.com/february-2014/python-remote-code-execution-socket-recvfrom_into/ > > Next time please inform the Python Security Response Team about any > and all issues that are related to buffer overflows or similar bugs. > In fact please drop a note about anything that even remotely look like > an exploitable issue. Even public bug reports should be forwarded to > PSRT. I'm not sure why you think it wasn't sent to security@ https://mail.python.org/mailman/private/psrt/2014-January/001297.html From barry at python.org Tue Feb 25 16:22:17 2014 From: barry at python.org (Barry Warsaw) Date: Tue, 25 Feb 2014 10:22:17 -0500 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: <20140225102217.410f0ff5@anarchist.wooz.org> On Feb 25, 2014, at 03:03 PM, Maciej Fijalkowski wrote: >Oh, I thought security fixes go to all python releases. Well, not the EOL'd ones of course. Where's the analysis on backporting SIPHash to older Python versions? Would such a backport break backward compatibility? What other impacts would backporting have? Would it break pickles, marshals, or other serialization protocols? Are there performance penalties? While security should be a top priority, it isn't the only consideration in such cases. A *lot* of discussion went into how to effect the hash randomization in Python 2.7, because of questions like these. The same analysis would have to be done for backporting this change to active older Python versions. -Barry From saimadhavheblikar at gmail.com Tue Feb 25 17:59:18 2014 From: saimadhavheblikar at gmail.com (Saimadhav Heblikar) Date: Tue, 25 Feb 2014 22:29:18 +0530 Subject: [Python-Dev] GSOC 2014 - IDLE Project Message-ID: Hi, Saimadhav Heblikar here.I would like to express my interest in working on IDLE improvement project as a part of Google Summer of Code 2014 for Python Core projects under the Python Software Foundation.I am currently a freshman Computer Science undergraduate student at PESIT , Bangalore. Similar to most Python programmers, i started my python journey on IDLE,and once i came to know that IDLE was one of the prospective projects,i knew this was great opportunity to give back to IDLE and the community as whole. I have created an account on the bug tracker and submitted the PSF contributor agreement.My username on the tracker is sahutd(saimadhav heblikar) http://bugs.python.org/user18939. I use the same nickname on the IRC channel. To enhance my understanding of the codebase, i have submitted few patches, some which have been committed http://bugs.python.org/issue20634 http://bugs.python.org/issue20677 and some which are under review http://bugs.python.org/issue20403 (IDLE related) http://bugs.python.org/issue20640 (Adds IDLE unittest) http://bugs.python.org/issue20451 http://bugs.python.org/issue20466 -------------- Coming to the point,as the theme of the project which i am interested in and the theme for the IDLE GSOC `13 projects by Jayakrishnan and Phil Webster are the same, i have a few questions to ask the python mentors/devs. 1.What will be the scope of the idle gsoc 2014?By scope,i want to know,whether i will be expected to continue the work done in 2013, or start afresh. 2. If i were to start afresh,i have explored the following two possibilities, a)Code recommender feature for IDLE - Similar to the feature in Eclipse but made for Python(would work from the shell or in IDLE).As an example of how it would work,we could think of it as a web api, which would return information about most used functions in a module.It would help beginners and experts alike to make better use of their dev time.It would also go a long way in making python even more social.If any questions on this proposal,i am looking forward to answer it. (I would like to know the community's opinion on this,GSOC or otherwise)If this generates a positive response, i will immediately come up with a plan to develop it. b)Working on a project to integrate a passive checker like pyflakes with the ability to highlight and possibly correct errors on-the-fly and on demand.Automatically integrate the above feature to work on folders/directories, selection of files ,projects etc. 2.If i were to continue the work done in gsoc 2013,would it involve a)building on features for PEP8(or other style checker,though after reading http://bugs.python.org/issue18704 i am inclined to believe it is not a must have atm. ) b)extending the unittest framework. Would i be completing(or to an extent) , the missing tests in idlelib/idle_test?What would be the priorities? c)features which don't seem to have been completed from gsoc 2013 like line numbering,improving cursor behavior and making right click more intuitive.(Anything gsoc 13 related which i have missed out?) -------------- I believe from the abstracts of GSOC 13 projects which were selected,that , i am expected to increase test coverage for idle AND add a new feature . Is my understanding correct? I would like to know whom i can contact(mentors) if have questions about the proposal writing and/or other technical questions relating to GSOC 14. I would also like to come up with overall outline based on your feedback so that it can critiqued before i submit the application on the google page. Awaiting your replies. Saimadhav Heblikar(sahutd) Bug Tracker username: sahutd (http://bugs.python.org/user18939) Email : saimadhavheblikar at gmail.com Github : https://github.com/sahutd -------------- next part -------------- An HTML attachment was scrubbed... URL: From saimadhavheblikar at gmail.com Tue Feb 25 18:13:33 2014 From: saimadhavheblikar at gmail.com (Saimadhav Heblikar) Date: Tue, 25 Feb 2014 22:43:33 +0530 Subject: [Python-Dev] Fwd: GSOC 2014 - IDLE Project In-Reply-To: References: Message-ID: Hi, Saimadhav Heblikar here.I would like to express my interest in working on IDLE improvement project as a part of Google Summer of Code 2014 for Python Core projects under the Python Software Foundation.I am currently a freshman Computer Science undergraduate student at PESIT , Bangalore. Similar to most Python programmers, i started my python journey on IDLE,and once i came to know that IDLE was one of the prospective projects,i knew this was great opportunity to give back to IDLE and the community as whole. I have created an account on the bug tracker and submitted the PSF contributor agreement.My username on the tracker is sahutd(saimadhav heblikar) http://bugs.python.org/user18939. I use the same nickname on the IRC channel. To enhance my understanding of the codebase, i have submitted few patches, some which have been committed http://bugs.python.org/issue20634 http://bugs.python.org/issue20677 and some which are under review http://bugs.python.org/issue20403 (IDLE related) http://bugs.python.org/issue20640 (Adds IDLE unittest) http://bugs.python.org/issue20451 http://bugs.python.org/issue20466 -------------- Coming to the point,as the theme of the project which i am interested in and the theme for the IDLE GSOC `13 projects by Jayakrishnan and Phil Webster are the same, i have a few questions to ask the python mentors/devs. 1.What will be the scope of the idle gsoc 2014?By scope,i want to know,whether i will be expected to continue the work done in 2013, or start afresh. 2. If i were to start afresh,i have explored the following two possibilities, a)Code recommender feature for IDLE - Similar to the feature in Eclipse but made for Python(would work from the shell or in IDLE).As an example of how it would work,we could think of it as a web api, which would return information about most used functions in a module.It would help beginners and experts alike to make better use of their dev time.It would also go a long way in making python even more social.If any questions on this proposal,i am looking forward to answer it. (I would like to know the community's opinion on this,GSOC or otherwise)If this generates a positive response, i will immediately come up with a plan to develop it. b)Working on a project to integrate a passive checker like pyflakes with the ability to highlight and possibly correct errors on-the-fly and on demand.Automatically integrate the above feature to work on folders/directories, selection of files ,projects etc. 2.If i were to continue the work done in gsoc 2013,would it involve a)building on features for PEP8(or other style checker,though after reading http://bugs.python.org/issue18704 i am inclined to believe it is not a must have atm. ) b)extending the unittest framework. Would i be completing(or to an extent) , the missing tests in idlelib/idle_test?What would be the priorities? c)features which don't seem to have been completed from gsoc 2013 like line numbering,improving cursor behavior and making right click more intuitive.(Anything gsoc 13 related which i have missed out?) -------------- I believe from the abstracts of GSOC 13 projects which were selected,that , i am expected to increase test coverage for idle AND add a new feature . Is my understanding correct? I would like to know whom i can contact(mentors) if have questions about the proposal writing and/or other technical questions relating to GSOC 14. I would also like to come up with overall outline based on your feedback so that it can critiqued before i submit the application on the google page. Awaiting your replies. Saimadhav Heblikar(sahutd) Bug Tracker username: sahutd (http://bugs.python.org/user18939) Email : saimadhavheblikar at gmail.com Github : https://github.com/sahutd -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Feb 25 18:48:48 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 25 Feb 2014 19:48:48 +0200 Subject: [Python-Dev] cpython: whatsnew: textwrap.shorten. In-Reply-To: <3fXByW4Fszz7Mj3@mail.python.org> References: <3fXByW4Fszz7Mj3@mail.python.org> Message-ID: 23.02.14 18:42, r.david.murray ???????(??): > http://hg.python.org/cpython/rev/4d615ab37804 > changeset: 89337:4d615ab37804 > user: R David Murray > date: Sun Feb 23 10:22:07 2014 -0500 > summary: > whatsnew: textwrap.shorten. > > Also add the missing TextWrapper.shorten method doc. There is no the TextWrapper.shorten method. The textwrap.shorten() function is implemented via new max_lines parameter. From christian at python.org Tue Feb 25 19:21:28 2014 From: christian at python.org (Christian Heimes) Date: Tue, 25 Feb 2014 19:21:28 +0100 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: <1393339277.26501.87591749.13312CA9@webmail.messagingengine.com> References: <530C48BC.2070804@python.org> <1393339277.26501.87591749.13312CA9@webmail.messagingengine.com> Message-ID: <530CDF28.5000700@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 25.02.2014 15:41, Benjamin Peterson wrote: > I'm not sure why you think it wasn't sent to security@ > https://mail.python.org/mailman/private/psrt/2014-January/001297.html Because > I can't find the mail in my inbox. Perhaps it fell victim to python.org's spam filter like several other mails to my python.org address? I really don't know. :( -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCgAGBQJTDN8iAAoJEMeIxMHUVQ1FNAIQANHAiRBvDfLZ15+GQ/QcMCC/ 8Vfcvc+DbQ2f36qFiuGsTwMnEAXX3RbKoqdi1oKbHm8yEFhIIYtKge7UATYKVZ66 YtypiWMVFUXQJf7/JnSzMbEbK8AZ8XCN0aKio18DXH4lG5DGSC0z69uj6rgxVyVm lt84oOSFxRojUVHYUSjc1eOUDdyJ8IHc0uC2v6O8x75c2X09ftkjMik7XcU1H0iL m303bI2lWVJLNAjFBjYVo4erwTqP4lD+LSr6lpmEn0eNfQc5ZawCL3gMdzmU87gY 0pQkRaJk2axESzDEpVFnvbX5ugzq2y+qeTKuCO8pOzE7nkknQp2RKDPNwhSjj4aX +JDjVaNpd3/in6YRPGzR4kjNKbMwC56Rgz4bvha9kg4Ldh+QAGbnUJhK8GmM1IVD 7TnEOdVwkaq7n/i/HVuIQZE6G0kKeunz0QJ5vtR4xUMNp24wvid1IJM5GdSR73hQ tTWJjXP4F85cu0gcy81nTTOLd+7ryPoyF9xOVXUvlwwYWDv4w0hLXXOJ/y/pOIMl B7sYyEaAcgFTjUhOpHamhLgqL1HuUHDsZdDy5ISOciX1nBxodjelcEmlpeGN+bH6 YUFgXvFu8GO+OsTsZ3UHJzrdzRiUcwk8ECTNgyEYAGg/ztpqsyXU8wNBI4KDSnB7 9ivcsc5aTUBrSiJExkRs =x7QU -----END PGP SIGNATURE----- From fijall at gmail.com Tue Feb 25 19:38:46 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 25 Feb 2014 20:38:46 +0200 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: <20140225102217.410f0ff5@anarchist.wooz.org> References: <530C48BC.2070804@python.org> <20140225102217.410f0ff5@anarchist.wooz.org> Message-ID: On Tue, Feb 25, 2014 at 5:22 PM, Barry Warsaw wrote: > On Feb 25, 2014, at 03:03 PM, Maciej Fijalkowski wrote: > >>Oh, I thought security fixes go to all python releases. > > Well, not the EOL'd ones of course. yes of course sorry. > > Where's the analysis on backporting SIPHash to older Python versions? Would > such a backport break backward compatibility? What other impacts would > backporting have? Would it break pickles, marshals, or other serialization > protocols? Are there performance penalties? > > While security should be a top priority, it isn't the only consideration in > such cases. A *lot* of discussion went into how to effect the hash > randomization in Python 2.7, because of questions like these. The same > analysis would have to be done for backporting this change to active older > Python versions. My impression is that a lot of discussion went into hash randomization, because it was a high profile issue. It got "fixed", then later someone discovered that the fix is completely broken and was left at that without much discussion because it's no longer "high visibility". I would really *like* to perceive this process as a lot of discussion going into because of ramification of changes. Cheers, fijal From solipsis at pitrou.net Tue Feb 25 19:49:24 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 25 Feb 2014 19:49:24 +0100 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() References: <530C48BC.2070804@python.org> <20140225102217.410f0ff5@anarchist.wooz.org> Message-ID: <20140225194924.42953ea0@fsol> On Tue, 25 Feb 2014 20:38:46 +0200 Maciej Fijalkowski wrote: > > My impression is that a lot of discussion went into hash > randomization, because it was a high profile issue. It got "fixed", > then later someone discovered that the fix is completely broken and > was left at that without much discussion because it's no longer "high > visibility". I would really *like* to perceive this process as a lot > of discussion going into because of ramification of changes. Most of the discussion, AFAIR, was about the potential backwards compatibility issues (which led to the decision of adding hash randomization in 2.7, but disabled by default). But you're right that for some reason it suddenly became a "high profile issue" while the general attack mechanism had apparently been known for years. (and AFAIK there's no proof of actual attacks in the wild) Regards Antoine. From storchaka at gmail.com Tue Feb 25 20:10:19 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 25 Feb 2014 21:10:19 +0200 Subject: [Python-Dev] Add Py_REPLACE and Py_XREPLACE macros In-Reply-To: References: Message-ID: 17.02.14 01:27, Nick Coghlan ???????(??): > This change doesn't fix any of the known crashers in Lib/test/crashers, > though - I applied the patch locally and checked. It fixes other crasher (http://bugs.python.org/issue20440#msg209713). > The point is that people already know what Py_CLEAR does. This operation > is like Py_CLEAR (the old reference is only removed *after* the pointer > has been updated), except that the value it is being replaced with can > be something other than NULL. If the replacement value *is* NULL, then > the new operation is *exactly* equivalent to Py_CLEAR. > > Operations that do related things should ideally have related names. The > point of my deliberately erroneous expansion is that it's an error a > reader can make while still correctly understanding the *logic* of the > code, even though they're missing a subtlety of the mechanics. Py_CLEAR and Py_DECREF have no related names. I think that the clarity and briefness are important. I assume that these macros will be widely used (they allow existing code to write shorter), perhaps even more than Py_CLEAR. Therefore people will know what they do. > An explicit name like Py_SET_AND_DECREF would also be reasonable. It's > substantially less confusing than Py_REPLACE, as it is less ambiguous > about whether or not the refcount on the new value is adjusted. I agree if it will satisfy Martin, although this name looks ugly to me. From tjreedy at udel.edu Tue Feb 25 20:30:51 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 25 Feb 2014 14:30:51 -0500 Subject: [Python-Dev] Running 2.7 tests on OS X In-Reply-To: References: Message-ID: On 2/25/2014 6:25 AM, Rik wrote: > I want to try to submit a patch for 2.7, but I don't know how to run the > tests for the 2.7 branch. `./configure` doesn't seem to create a > `python.exe` file on the 2.7 branch on OS X Mavericks, and I do need > this file according to this guide: > http://docs.python.org/devguide/ While testing with a fresh build is definitely best, 2nd best is testing against the latest release, as long as it is installed with Lib/test included. So don't let build problems indefinitely stop you from submitting the patch ;-). -- Terry Jan Reedy From tjreedy at udel.edu Tue Feb 25 21:13:33 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 25 Feb 2014 15:13:33 -0500 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> <20140225141728.40090f42@fsol> <7C0D41B8-8F3D-4203-885F-BFB38D42F53A@stufft.io> Message-ID: On 2/25/2014 8:32 AM, Chris Angelico wrote: > On Wed, Feb 26, 2014 at 12:21 AM, Donald Stufft wrote: >> Instead of pre-generating one set of values that can be be used to DoS things >> you have to pre-generate 256 sets of values and try them until you get the >> right one. It?s like putting on armor made of paper and saying it?s harder to >> stab you now. > > Paper armor? You mean the stuff they tested here? > http://www.imdb.com/title/tt1980597/ OT, but I just watched that Mythbusters episode and they confirmed that on a weight basis, paper armor from multiple folded sheets was comparable to steel. They have also shown that phonebooks stop ordinary bullets. -- Terry Jan Reedy From ethan at stoneleaf.us Tue Feb 25 21:17:51 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 25 Feb 2014 12:17:51 -0800 Subject: [Python-Dev] [OT] Paper armor [was Python Remote Code Execution in socket.recvfrom_into()] In-Reply-To: References: <530C48BC.2070804@python.org> <20140225141728.40090f42@fsol> <7C0D41B8-8F3D-4203-885F-BFB38D42F53A@stufft.io> Message-ID: <530CFA6F.5030301@stoneleaf.us> On 02/25/2014 12:13 PM, Terry Reedy wrote: > On 2/25/2014 8:32 AM, Chris Angelico wrote: >> On Wed, Feb 26, 2014 at 12:21 AM, Donald Stufft wrote: >>> >>> Instead of pre-generating one set of values that can be be used to DoS things >>> you have to pre-generate 256 sets of values and try them until you get the >>> right one. It?s like putting on armor made of paper and saying it?s harder to >>> stab you now. >> >> Paper armor? You mean the stuff they tested here? >> http://www.imdb.com/title/tt1980597/ > > OT, but I just watched that Mythbusters episode and they confirmed that on a weight basis, paper armor from multiple > folded sheets was comparable to steel. They have also shown that phonebooks stop ordinary bullets. Those must be big-city phone books. About the only thing my phone book would stop is a bb. :/ -- ~Ethan~ From ncoghlan at gmail.com Tue Feb 25 21:46:00 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Feb 2014 06:46:00 +1000 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: <20140225194924.42953ea0@fsol> References: <530C48BC.2070804@python.org> <20140225102217.410f0ff5@anarchist.wooz.org> <20140225194924.42953ea0@fsol> Message-ID: On 26 Feb 2014 04:51, "Antoine Pitrou" wrote: > > On Tue, 25 Feb 2014 20:38:46 +0200 > Maciej Fijalkowski wrote: > > > > My impression is that a lot of discussion went into hash > > randomization, because it was a high profile issue. It got "fixed", > > then later someone discovered that the fix is completely broken and > > was left at that without much discussion because it's no longer "high > > visibility". I would really *like* to perceive this process as a lot > > of discussion going into because of ramification of changes. > > Most of the discussion, AFAIR, was about the potential backwards > compatibility issues (which led to the decision of adding hash > randomization in 2.7, but disabled by default). > > But you're right that for some reason it suddenly became a "high > profile issue" while the general attack mechanism had apparently been > known for years. > (and AFAIK there's no proof of actual attacks in the wild) Remote DOS attacks are so easy and so prevalent that if all a CVE does is make them slightly easier when untrusted input isn't properly validated and resource consumption per request isn't capped, security teams rank it as a pretty low threat and not very interesting. However, there was a paper that presented this one like it was a big revelation, it stirred up a lot of complaints, so doing something about it ended up being easier than repeatedly explaining why it didn't really matter that much. Donald's right that the original fix just increased the required attack payload size from 1 MB (if I recall the number correctly) to 256 MB total, but at that point attempts to exploit it are starting to look like a more conventional DoS (one that doesn't rely on any particular vulnerability to be effective) anyway. It's still nice to finally have it fixed properly in 3.4, though, and the original change at least took care of the problem of getting people to update their code that previously relied on consistent dict ordering. Without that previous work, the hash change for 3.4 would have been more challenging. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Feb 25 22:58:06 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Feb 2014 07:58:06 +1000 Subject: [Python-Dev] [Python-checkins] cpython: whatsnew: DynanicClassAttribute (#19030), Py_SetStandardStreamEncoding (#16129) In-Reply-To: <3fYXgP6KZQz7LpV@mail.python.org> References: <3fYXgP6KZQz7LpV@mail.python.org> Message-ID: On 26 Feb 2014 07:04, "r.david.murray" wrote: > > http://hg.python.org/cpython/rev/4cd620d8c3f6 > changeset: 89392:4cd620d8c3f6 > user: R David Murray > date: Tue Feb 25 16:03:14 2014 -0500 > summary: > whatsnew: DynanicClassAttribute (#19030), Py_SetStandardStreamEncoding (#16129) > > Adding missing docs for DynamicClassAttribute by copying the docstring. The > doc entry could stand some expansion, which I will note on the issue. > > files: > Doc/library/types.rst | 21 +++++++++++++++++++++ > Doc/whatsnew/3.4.rst | 17 +++++++++++++++++ > 2 files changed, 38 insertions(+), 0 deletions(-) > > > diff --git a/Doc/library/types.rst b/Doc/library/types.rst > --- a/Doc/library/types.rst > +++ b/Doc/library/types.rst > @@ -15,6 +15,9 @@ > Python interpreter, but not exposed as builtins like :class:`int` or > :class:`str` are. > > +Finally, it provides some additional type-related utility classes and functions > +that are not fundamental enough to be builtins. > + > > Dynamic Type Creation > --------------------- > @@ -220,6 +223,9 @@ > Return a new view of the underlying mapping's values. > > > +Additional Utility Classes and Functions > +---------------------------------------- > + > .. class:: SimpleNamespace > > A simple :class:`object` subclass that provides attribute access to its > @@ -246,3 +252,18 @@ > instead. > > .. versionadded:: 3.3 > + > + > +.. function:: DynamicClassAttribute(fget=None, fset=None, fdel=None, doc=None) > + > + Route attribute access on a class to __getattr__. > + > + This is a descriptor, used to define attributes that act differently when > + accessed through an instance and through a class. Instance access remains > + normal, but access to an attribute through a class will be routed to the > + class's __getattr__ method; this is done by raising AttributeError. > + > + This allows one to have properties active on an instance, and have virtual > + attributes on the class with the same name (see Enum for an example). > + > + .. versionadded:: 3.4 > diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst > --- a/Doc/whatsnew/3.4.rst > +++ b/Doc/whatsnew/3.4.rst > @@ -1270,6 +1270,17 @@ > :issue:`1565525`). > > > +types > +----- > + > +A new :func:`~types.DynamicClassAttribute` descriptor provides a way to define > +an attribute that acts normally when looked up through an instance object, but > +which is routed to the *class* ``__getattr__`` when looked up through the > +class. This allows one to have properties active on a class, and have virtual > +attributes on the class with the same name (see :mod:`Enum` for an example). > +(Contributed by Ethan Furman in :issue:`19030`.) > + > + > urllib > ------ > > @@ -1512,6 +1523,12 @@ > object allocator have been silenced. (Contributed by Dhiru Kholia in > :issue:`18596`.) > > +* New function :c:func:`Py_SetStandardStreamEncoding` allows an application > + that is embedding Python to do the equivalent of setting > + :envvar:`PYTHONIOENCODING`. Its arguments override the equivalent > + values from :envvar:`PYTHONIOENCODING` if it exists. (Contributed > + by Bastien Montagne and Nick Coghlan in :issue:`16129`.) > + There was an existing entry for this one: second bullet under http://docs.python.org/dev/whatsnew/3.4.html#other-build-and-c-api-changes Cheers, Nick. > > .. _other-improvements-3.4: > > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kasturisurya at gmail.com Wed Feb 26 02:56:38 2014 From: kasturisurya at gmail.com (Surya) Date: Wed, 26 Feb 2014 07:26:38 +0530 Subject: [Python-Dev] GSoC 2014 - Email Module Message-ID: Hey there, I am Surya, studying final year of Engineering. I have looked into Core Python's ideas list and got interested in Email module. I've been working on Django over the past few years, and now like to work on slightly a different layer of protocols and this idea happened to be it. That said, I have been just using the Email module earlier along with other similar modules and never dwell into the internal protocol standards before. It would be great if someone initially can let me know what documentation I should start reading to get up to the speed and probably bugs related to this idea I can look into. Also, I am looking forward to talk about the project with the mentor! Thanks Surya -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Feb 26 03:46:10 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 25 Feb 2014 21:46:10 -0500 Subject: [Python-Dev] GSoC 2014 - Email Module In-Reply-To: References: Message-ID: On 2/25/2014 8:56 PM, Surya wrote: > Hey there, > > I am Surya, studying final year of Engineering. I have looked into Core > Python's ideas list and got interested in Email module. > > I've been working on Django over the past few years, and now like to > work on slightly a different layer of protocols and this idea happened > to be it. > > That said, I have been just using the Email module earlier along with > other similar modules and never dwell into the internal protocol > standards before. > > It would be great if someone initially can let me know what > documentation I should start reading to get up to the speed and probably > bugs related to this idea I can look into. Please sign up for core-mentorship list and ask questions there. -- Terry Jan Reedy From stephen at xemacs.org Wed Feb 26 04:57:14 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 26 Feb 2014 12:57:14 +0900 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: <53095562.8010304@stoneleaf.us> Message-ID: <87vbw2ttk5.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes that b'%a' is > the obvious way to interpolate representations of arbitrary objects > into binary formats that contain ASCII compatible segments. The only argument that I have sympathy for is > %a *should* be allowed for consistency with text interpolation although introduction of a new format character is a poor man's consistency, and this is consistency for consistency's sake. (I don't have a big problem with that, though. I *like* consistency!) But TOOWTDI where I get off the bus. I don't I agree that this consistency is terribly useful, given how easy it is to def ascify(obj): # You could also do this with UTF-8. return ascii(obj).encode('ascii', errors='backslashescape') I think the obvious way to interpolate representations of arbitrary objects into binary formats that may contain ASCII-compatible *segments* is a '__bytes__' method. (Yes, I'm cheating, that's not the sense of "arbitrary" Nick meant. But see below for what happens when I *do* consider Nick's sense of "arbitrary".) If it makes sense to represent an object using only ASCII bytes (eg, a BASE64 encoding for binary blobs), why not a '__bytes__' method? If non-ASCII- compatible segments are allowed, why not use __repr__, or a '__bytes__' method that gives you a full representation of the object (eg, a pickle)? So we're really talking about formats that are 100% ASCII-compatible. What are the use cases? Debugging logs? I don't see it. As far as human-readability goes, I read 100% incompatible-with-anything debug logs (aka, containing Japanese in several of its 4 commonly-used wire-format encodings) into XEmacs buffers regularly with no problems. Decoding them can be a bitch, of course -- life would be simple if only they *were* Python reprs! Of course Emacsen provide a huge amount of help with such things, but most of what I need to do would work fine as long as the editor doesn't crash, has an ASCII printable visual representation of non-printing-ASCII bytes, and allows both truncation and wrap-at-screen-edge printing of long lines. OTOH, maybe you have an automatic log-analysis tool or the like that snafus on non-ASCII-compatible stuff. if you are truly serious about keeping your debug logs 100% ASCII-compatible (whether pure ASCII or some ASCII-compatible "universal" encoding like UTF-8 or GB18030), you really have your work cut out for you, especially if you want it to be automatically parseable. Ascification is the least of your worries. Or you can do something like def log_debug_msg(msg_or_obj): write_to_log(ascify(msg_or_obj)) and get rid of the annoying "b" prefix on all your log message formats, too! YMMV, but *I* don't see debug logs as a plausible justification. The only plausible case I can think of is Glenn's web app where you actually directly insert debug information into wire protocol destined to appear in end-user output -- but then, this web app itself is only usable in Kansas and other places where the nearest place that a language other than Middle American English is spoken is a megameter away. Industrial strength frameworks will do that work using str, and then .encode() to the user's requested encoding. So this probably isn't an app, but rather the web server itself (which speaks bytes to clients, not text to users). But then, typical reprs (whether restricted to ASCII or not) have insufficient information about an object to reproduce it. Why is it a good idea to encourage people writing objects to a debug log to use a broken-for-the-purpose repr? (I can see it could go either way. For example, if the alternative is a "something went wrong" error message. But I'd like to see a stronger argument that a feature which is intended to encourage people to take shortcuts -- and otherwise has no justification -- is Pythonic. :-) The "inappropriate '__bytes__' method" seems to be a imaginary bogeyman, in any case. If people really want to dump arbitrary objects (in Nick's sense) to a byte-oriented stream *outside* of the stream's protocol, I think it would be easier to do that with 'ascify' then by altering *every* class definition by adding 'ascify' as the '__bytes__' definition. Note that 'ascify' is fully general in case you don't know what the type of the object you are dumping is; '__bytes__' may not be. Some objects may have existing incompatible definitions for '__bytes__': eg, in HTTP, there's no problem with sending an object in binary format, and it might very well be a complex object with internal structure that gets flattened for transmission (into a pickle, for example, or a Python list of frames from a streaming server: "stream % frames[secs_to_frame(28):]"). Surely you're not going to replace that '__bytes__' with def __bytes__(self): return ascify(self)" Steve From stephen at xemacs.org Wed Feb 26 05:11:32 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 26 Feb 2014 13:11:32 +0900 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: <7C0D41B8-8F3D-4203-885F-BFB38D42F53A@stufft.io> References: <530C48BC.2070804@python.org> <20140225141728.40090f42@fsol> <7C0D41B8-8F3D-4203-885F-BFB38D42F53A@stufft.io> Message-ID: <87txbmtswb.fsf@uwakimon.sk.tsukuba.ac.jp> Donald Stufft writes: > Instead of pre-generating one set of values that can be be used to > DoS things you have to pre-generate 256 sets of values and try them > until you get the right one. It?s like putting on armor made of > paper and saying it?s harder to stab you now. You obviously don't watch "Burn Notice." Paper armor worked great for Michael Weston! Unpacking, not all crackers are serious. I'd be willing to bet that there are a number of script kiddies who are *still* running scripts that only know about the first hole. What's to lose by beating them? From ncoghlan at gmail.com Wed Feb 26 06:00:34 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Feb 2014 15:00:34 +1000 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: <87vbw2ttk5.fsf@uwakimon.sk.tsukuba.ac.jp> References: <53095562.8010304@stoneleaf.us> <87vbw2ttk5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On 26 February 2014 13:57, Stephen J. Turnbull wrote: > Nick Coghlan writes that b'%a' is > > > the obvious way to interpolate representations of arbitrary objects > > into binary formats that contain ASCII compatible segments. > > The only argument that I have sympathy for is > > > %a *should* be allowed for consistency with text interpolation > > although introduction of a new format character is a poor man's > consistency, and this is consistency for consistency's sake. (I don't > have a big problem with that, though. I *like* consistency!) It's *not* a new format character, unless you mean "new in Python 3". Python 3 text interpolation has included %a for as long as I can recall, specifically as a way of spelling the old Python 2 %r interpolation behaviour now that the Python 3 %r allows Unicode text. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wes.turner at gmail.com Wed Feb 26 06:33:09 2014 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 25 Feb 2014 23:33:09 -0600 Subject: [Python-Dev] Python Remote Code Execution in socket.recvfrom_into() In-Reply-To: References: <530C48BC.2070804@python.org> Message-ID: On 2/25/14, Victor Stinner wrote: > Hi, > > 2014-02-25 8:53 GMT+01:00 Nick Coghlan : >> I've checked these, and noted the relevant hg.python.org links on the >> tracker issue at http://bugs.python.org/issue20246 > > Would it be possible to have a table with all known Python security > vulnerabilities and the Python versions which are fixed? Bonus point > if we provide a link to the changeset fixing it for each branch. Maybe > put this table on http://www.python.org/security/ ? For http://www.python.org/security/ : Here's a start at an issue tracker query for open and closed issues with 'Type: Security': http://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=&creation=&%40sort=creation&creator=&activity=&%40columns=activity&actor=&nosy=&type=4&components=&versions=&%40columns=versions&dependencies=&assignee=&keywords=&priority=&%40group=priority&status=&%40columns=status&resolution=&nosy_count=&message_count=&%40pagesize=200&%40startwith=0&%40action=search Here's a list of filed CVEs with Python in the vendor field: http://www.cvedetails.com/vulnerability-list/vendor_id-10210/product_id-18230/Python-Python.html When referring to security issues, it may be helpful to reference the CVE codes and tracker IDs. > > Last issues: > - hash DoS > - sock.recvfrom_into() > - DoS with very long lines in HTTP, FTP, etc. protocols > - etc. > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com > -- -- Wes Turner From storchaka at gmail.com Wed Feb 26 10:40:01 2014 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 26 Feb 2014 11:40:01 +0200 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc Message-ID: There were several suggestions for naming new macros which replace old value with new value and then (x)decref old value. #define Py_XXX(ptr, value) \ { \ PyObject *__tmp__ = ptr; \ ptr = new_value; \ Py_DECREF(__tmp__); \ } Listed in order of receipt: 1. Py_(X)SETREF. 2. Py_(X)ASSIGN. 3. Py_(X)DECREC_REPLACE. 4. Py_REF_ASSIGN (Py_XREF_ASSIGN? Py_REF_XASSIGN?). 5. Py_(X)REPLACE. 6. Py_(X)STORE 7. Py_CLEAR_AND_SET. 8. Py_SET_AND_(X)DECREF. 9. Py_(X)DECREF_AND_ASSIGN. 10. Py_ASSIGN_AND_(X)DECREF. 11. Other... Let's choose the least confusing names. See discussions at: http://bugs.python.org/issue3081 http://bugs.python.org/issue16447 http://bugs.python.org/issue20440 http://comments.gmane.org/gmane.comp.python.devel/145346 From stephen at xemacs.org Wed Feb 26 11:56:40 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 26 Feb 2014 19:56:40 +0900 Subject: [Python-Dev] PEP 461: Adding % formatting to bytes and bytearray -- Final, Take 2 In-Reply-To: References: <53095562.8010304@stoneleaf.us> <87vbw2ttk5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87k3cita53.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > sjt writes: > > although introduction of a new format character is a poor man's > > consistency, and this is consistency for consistency's sake. (I don't > > have a big problem with that, though. I *like* consistency!) > > It's *not* a new format character, unless you mean "new in Python > 3". Ah, my bad. Obviously I meant "new to Python 3", and therefore was just plain wrong. From kasturi.surya at gmail.com Wed Feb 26 10:46:44 2014 From: kasturi.surya at gmail.com (Surya) Date: Wed, 26 Feb 2014 15:16:44 +0530 Subject: [Python-Dev] GSoC 2014 - Email Module In-Reply-To: References: Message-ID: On Wed, Feb 26, 2014 at 8:16 AM, Terry Reedy wrote: > On 2/25/2014 8:56 PM, Surya wrote: > >> Hey there, >> >> I am Surya, studying final year of Engineering. I have looked into Core >> Python's ideas list and got interested in Email module. >> >> I've been working on Django over the past few years, and now like to >> work on slightly a different layer of protocols and this idea happened >> to be it. >> >> That said, I have been just using the Email module earlier along with >> other similar modules and never dwell into the internal protocol >> standards before. >> >> It would be great if someone initially can let me know what >> documentation I should start reading to get up to the speed and probably >> bugs related to this idea I can look into. >> > > Please sign up for core-mentorship list and ask questions there. Is this for students too? Because I want to join as a student.?? > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > kasturi.surya%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Feb 26 17:13:23 2014 From: brett at python.org (Brett Cannon) Date: Wed, 26 Feb 2014 11:13:23 -0500 Subject: [Python-Dev] GSoC 2014 - Email Module In-Reply-To: References: Message-ID: On Wed, Feb 26, 2014 at 4:46 AM, Surya wrote: > > > > On Wed, Feb 26, 2014 at 8:16 AM, Terry Reedy wrote: > >> On 2/25/2014 8:56 PM, Surya wrote: >> >>> Hey there, >>> >>> I am Surya, studying final year of Engineering. I have looked into Core >>> Python's ideas list and got interested in Email module. >>> >>> I've been working on Django over the past few years, and now like to >>> work on slightly a different layer of protocols and this idea happened >>> to be it. >>> >>> That said, I have been just using the Email module earlier along with >>> other similar modules and never dwell into the internal protocol >>> standards before. >>> >>> It would be great if someone initially can let me know what >>> documentation I should start reading to get up to the speed and probably >>> bugs related to this idea I can look into. >>> >> >> Please sign up for core-mentorship list and ask questions there. > > > Is this for students too? Because I want to join as a student.?? > The mailing list is for anyone who wants to contribute to Python's development and wants some help getting started, student or not. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed Feb 26 17:09:53 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 26 Feb 2014 10:09:53 -0600 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: Message-ID: I like Py_DECREF_REPLACE. It gives the impression that it decrefs the original and replaces it. On Wed, Feb 26, 2014 at 3:40 AM, Serhiy Storchaka wrote: > There were several suggestions for naming new macros which replace old > value with new value and then (x)decref old value. > > #define Py_XXX(ptr, value) \ > { \ > PyObject *__tmp__ = ptr; \ > ptr = new_value; \ > Py_DECREF(__tmp__); \ > } > > Listed in order of receipt: > > 1. Py_(X)SETREF. > 2. Py_(X)ASSIGN. > 3. Py_(X)DECREC_REPLACE. > 4. Py_REF_ASSIGN (Py_XREF_ASSIGN? Py_REF_XASSIGN?). > 5. Py_(X)REPLACE. > 6. Py_(X)STORE > 7. Py_CLEAR_AND_SET. > 8. Py_SET_AND_(X)DECREF. > 9. Py_(X)DECREF_AND_ASSIGN. > 10. Py_ASSIGN_AND_(X)DECREF. > 11. Other... > > Let's choose the least confusing names. > > See discussions at: > > http://bugs.python.org/issue3081 > http://bugs.python.org/issue16447 > http://bugs.python.org/issue20440 > http://comments.gmane.org/gmane.comp.python.devel/145346 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > rymg19%40gmail.com > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Feb 26 18:34:07 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 26 Feb 2014 17:34:07 +0000 Subject: [Python-Dev] GSoC 2014 - Email Module In-Reply-To: References: Message-ID: On 26 February 2014 16:13, Brett Cannon wrote: >>> >>> Please sign up for core-mentorship list and ask questions there. >> >> >> Is this for students too? Because I want to join as a student. > > > The mailing list is for anyone who wants to contribute to Python's > development and wants some help getting started, student or not. The subject of this email mentions GSoC, it's probably worth clarifying that the GSoC process is still under way and there isn't (as far as I know, I'm not involved myself) a confirmed list of mentors and projects in place yet. The OP is more than welcome to get involved, and the core-mentorship list will provide help in getting up to speed and starting to contribute, but it's not directly related to GSoC. Becoming part of the development community is a great way to prepare for a GSoC project when that starts up, though! Paul From solipsis at pitrou.net Wed Feb 26 19:26:42 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Feb 2014 19:26:42 +0100 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc References: Message-ID: <20140226192642.769b2377@fsol> On Wed, 26 Feb 2014 11:40:01 +0200 Serhiy Storchaka wrote: > There were several suggestions for naming new macros which replace old > value with new value and then (x)decref old value. > > #define Py_XXX(ptr, value) \ > { \ > PyObject *__tmp__ = ptr; \ > ptr = new_value; \ > Py_DECREF(__tmp__); \ > } > 1. Py_(X)SETREF. My vote is on this one. I'm also -1 on any name which doesn't have "REF" in it; the name should clearly suggest that it's a refcounting operation. Regards Antoine. From tjreedy at udel.edu Wed Feb 26 19:53:14 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 26 Feb 2014 13:53:14 -0500 Subject: [Python-Dev] GSoC 2014 - Status for Core Python In-Reply-To: References: Message-ID: On 2/26/2014 12:34 PM, Paul Moore wrote: > The subject of this email mentions GSoC, it's probably worth > clarifying that the GSoC process is still under way and there isn't > (as far as I know, I'm not involved myself) a confirmed list of > mentors and projects in place yet. There is a confirmed list of 190 sponsor organizations, including PSF. https://www.google-melange.com/gsoc/org/list/public/google/gsoc2014 50 include 'python' in their tags. PSF acts as an umbrella organization for multiple Python projects https://wiki.python.org/moin/SummerOfCode/2014 Core Python is the first listed of about 15. core-mentorship is listed as the list for core Python GSOC projects. The core Python ideas page is https://wiki.python.org/moin/SummerOfCode/2014/python-core Email (R. David Murray, Antoine Pitrou) and Idle (me, Tal Einat) are the two project areas lists. According to the GSOC timeline https://www.google-melange.com/gsoc/events/google/gsoc2014 this is an 'Interim Period' and 'Would-be students discuss project ideas with potential mentoring organizations.' > The OP is more than welcome to get > involved, and the core-mentorship list will provide help in getting up > to speed and starting to contribute, but it's not directly related to > GSoC. Becoming part of the development community is a great way to > prepare for a GSoC project when that starts up, though! Our intention is that GSOC students be a part of the core development community in that their code should be posted and reviewed on the tracker and hopefully committed, just as with any other new developer on the list. -- Terry Jan Reedy From p.f.moore at gmail.com Wed Feb 26 20:50:08 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 26 Feb 2014 19:50:08 +0000 Subject: [Python-Dev] GSoC 2014 - Status for Core Python In-Reply-To: References: Message-ID: On 26 February 2014 18:53, Terry Reedy wrote: > On 2/26/2014 12:34 PM, Paul Moore wrote: > >> The subject of this email mentions GSoC, it's probably worth >> clarifying that the GSoC process is still under way and there isn't >> (as far as I know, I'm not involved myself) a confirmed list of >> mentors and projects in place yet. > > > There is a confirmed list of 190 sponsor organizations, including PSF. > https://www.google-melange.com/gsoc/org/list/public/google/gsoc2014 > 50 include 'python' in their tags. > > PSF acts as an umbrella organization for multiple Python projects > https://wiki.python.org/moin/SummerOfCode/2014 > Core Python is the first listed of about 15. [...] That's fantastic. My apologies for spreading misinformation, I was going off the last things I remembered seeing on this list, which is not what I should have done. I should have checked my facts before posting. Paul From timothy.c.delaney at gmail.com Wed Feb 26 23:05:36 2014 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Thu, 27 Feb 2014 09:05:36 +1100 Subject: [Python-Dev] GSoC 2014 - Status for Core Python In-Reply-To: References: Message-ID: On 27 February 2014 05:53, Terry Reedy wrote: > > PSF acts as an umbrella organization for multiple Python projects > https://wiki.python.org/moin/SummerOfCode/2014 > Core Python is the first listed of about 15. > I'm guessing Mercurial will appear under the umbrella in the not to distant future (Mercurial was rejected as a sponsor organisation - Giovanni Gherdovich is liaising with the PSF about it). Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Feb 27 01:11:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Feb 2014 10:11:51 +1000 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: <20140226192642.769b2377@fsol> References: <20140226192642.769b2377@fsol> Message-ID: On 27 Feb 2014 04:28, "Antoine Pitrou" wrote: > > On Wed, 26 Feb 2014 11:40:01 +0200 > Serhiy Storchaka wrote: > > > There were several suggestions for naming new macros which replace old > > value with new value and then (x)decref old value. > > > > #define Py_XXX(ptr, value) \ > > { \ > > PyObject *__tmp__ = ptr; \ > > ptr = new_value; \ > > Py_DECREF(__tmp__); \ > > } > > > > 1. Py_(X)SETREF. > > My vote is on this one. > I'm also -1 on any name which doesn't have "REF" in it; the name should > clearly suggest that it's a refcounting operation. Yeah, I think SETREF is my favourite as well (even though some of the later suggestions were mine). Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Feb 27 01:21:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Feb 2014 10:21:05 +1000 Subject: [Python-Dev] GSoC 2014 - Status for Core Python In-Reply-To: References: Message-ID: On 27 Feb 2014 08:07, "Tim Delaney" wrote: > > On 27 February 2014 05:53, Terry Reedy wrote: >> >> >> PSF acts as an umbrella organization for multiple Python projects >> https://wiki.python.org/moin/SummerOfCode/2014 >> Core Python is the first listed of about 15. > > > I'm guessing Mercurial will appear under the umbrella in the not to distant future (Mercurial was rejected as a sponsor organisation - Giovanni Gherdovich is liaising with the PSF about it). I saw a comment from Matt Mackall about that earlier, so it sounds like it's happening: https://twitter.com/mpmselenic/status/438793983251083264 Cheers, Nick. > > Tim Delaney > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Thu Feb 27 08:13:46 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 27 Feb 2014 08:13:46 +0100 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: Message-ID: Am 26.02.2014 17:09, schrieb Ryan Gonzalez: > I like Py_DECREF_REPLACE. It gives the impression that it decrefs the original > and replaces it. Agreed, most other suggestions are not really explicit enough. Georg From ronaldoussoren at mac.com Thu Feb 27 09:44:45 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 27 Feb 2014 09:44:45 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> Message-ID: On 21 Feb 2014, at 16:52, Chris Angelico wrote: > On Sat, Feb 22, 2014 at 1:34 AM, Brett Cannon wrote: >> While I like the general concept, I agree that it looks too much like a >> crunched statement; the use of the colon is a non-starter for me. I'm sure >> I'm not the only one whose brain has been trained to view a colon in Python >> to mean "statement", period. This goes against that syntactic practice and >> just doesn't work for me. >> >> I'm -1 with the current syntax, but it can go into the + range if a better >> syntax can be chosen. > > We bikeshedded that extensively on -ideas. The four best options are: > > value = (expr except Exception: default) > value = (expr except Exception -> default) > value = (expr except Exception pass default) > value = (expr except Exception then default) > > Note that the last option involves the creation of a new keyword. > > Would any of the others feel better to you? What about (also mentioned in the PEP)? value = (expr except Exception try default) This seems to read nicely, although ?try? is at a completely different position than it is in the equivalent try statement. I like the general idea, but like Brett I don?t like using a colon here at all. Ronald P.S. Sorry if this way already brought up, I?ve browsed through most of the threads on this on -ideas and -dev, but haven?t read all messages. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From rosuav at gmail.com Thu Feb 27 11:09:18 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Feb 2014 21:09:18 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> Message-ID: On Thu, Feb 27, 2014 at 7:44 PM, Ronald Oussoren wrote: > What about (also mentioned in the PEP)? > > value = (expr except Exception try default) > > This seems to read nicely, although ?try? is at a completely different position than it is in the equivalent try statement. > > I like the general idea, but like Brett I don?t like using a colon here at all. I see your "although" clause to be quite a strong objection. In the statement form of an if, you have: if cond: true_suite else: false_suite In the expression form, you have: true_expr if cond else false_expr Personally, I think it's a weakness of the if-expression that they're not in the same order (cond, true_expr, false_expr), but they're still introduced with the same keywords. The 'if' keyword is followed by the condition, and the 'else' keyword by the false "stuff". Putting "try" followed by the default is confusing, because any exception raised in the default-expr will bubble up. Stealing any other keyword from the try/except block would make just as little sense: expr except Exception finally default # "finally" implies something that always happens expr except Exception else default # "else" implies *no* exception expr except Exception try default # "try" indicates the initial expr, not the default default except Exception try expr # breaks L->R evaluation order Left to right evaluation order is extremely important to me. I don't know about anyone else, but since I'm the one championing the PEP, you're going to have to show me a *really* strong incentive to reword it to advocate something like the last one :) This is stated in the PEP: http://www.python.org/dev/peps/pep-0463/#alternative-proposals Using try and except leaves the notation "mentally ambiguous" as to which of the two outer expressions is which. It doesn't make perfect sense either way, and I expect a lot of people would be flicking back to the docs constantly to make sure they had it right. It's hard when there's confusion across languages (try writing some code in REXX and Python that uses division and modulo operators - 1234/10 -> 123.4 in Py3 and REXX, but 1234//10 and 1234%10 have opposite meaning); it's unnecessarily hard to have the same confusion in different places in the same language. ChrisA From kristjan at ccpgames.com Thu Feb 27 11:22:05 2014 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 27 Feb 2014 10:22:05 +0000 Subject: [Python-Dev] Start writing inlines rather than macros? Message-ID: Hi there. The discussion on http://bugs.python.org/issue20440 started me thinking that much of this bikeshedding could be avoided if we weren't constrained to writing macros for all of this stuff. For example, a Py_INLINE(PyObject *) Py_Incref(PyObject *obj) { Py_INCREF(obj); return obj; } could be used in a Py_Assign() function, if a new reference were wanted: Py_INLINE(void) Py_Assign(PyObject ** target, PyObject *obj) { PyObject *tmp = *target; *target = tmp; Py_DECREF(tmp); } So that you could then safely write code like Py_Assign(&MyVar, Py_Incref(obj)); This would also allow you to stop writing various super macros to try to cater to all possible permutations. Now, Larry Hastings pointed out that we support C89 which doesn't support Inlines. Rather than suggesting here that we update that compatibility requirement, how about adding a Py_INLINE() macro.? This would be like Py_LOCAL_INLINE() except that it would drop the "static" keyword, unless inline isn't supported: #if defined(_MSC_VER) #define Py_INLINE(type) __inline type #elif defined(USE_INLINE) #define Py_INLINE(type) inline type #else #define Py_INLINE(type) static type #endif The only question is with the last line. How many platforms actually _do_not_ have inlines? Would writing stuff like this be considered problematic for those platforms? Note, that I'm not suggesting replacing macros all over the place, only for new code. Another question is: Is "static inline" in any practical way different from "inline"? It would be great to get rid of macros in code. It would be great for debugging too! K -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronaldoussoren at mac.com Thu Feb 27 11:44:48 2014 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 27 Feb 2014 11:44:48 +0100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> Message-ID: <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> On 27 Feb 2014, at 11:09, Chris Angelico wrote: > On Thu, Feb 27, 2014 at 7:44 PM, Ronald Oussoren wrote: >> What about (also mentioned in the PEP)? >> >> value = (expr except Exception try default) >> >> This seems to read nicely, although ?try? is at a completely different position than it is in the equivalent try statement. >> >> I like the general idea, but like Brett I don?t like using a colon here at all. > > I see your "although" clause to be quite a strong objection. In the > statement form of an if, you have: I?m not convinced that this is a strong objection. The order of keywords is different, but that doesn?t have to be problem. > > if cond: true_suite > else: false_suite > > In the expression form, you have: > > true_expr if cond else false_expr [?] > > > Putting "try" followed by the default is confusing, because any > exception raised in the default-expr will bubble up. Stealing any > other keyword from the try/except block would make just as little > sense: > > expr except Exception finally default # "finally" implies something > that always happens > expr except Exception else default # "else" implies *no* exception > expr except Exception try default # "try" indicates the initial expr, > not the default I didn?t parse the expression this way at all, but quite naturally parsed is as ?use expr, and try using default if expr raises Exception? and not as a RTL expression. > default except Exception try expr # breaks L->R evaluation order > > Left to right evaluation order is extremely important to me. I agree with that, RTL evaluation would be pretty odd in Python. > I don't > know about anyone else, but since I'm the one championing the PEP, > you're going to have to show me a *really* strong incentive to reword > it to advocate something like the last one :) This is stated in the > PEP: > > http://www.python.org/dev/peps/pep-0463/#alternative-proposals > > Using try and except leaves the notation "mentally ambiguous" as to > which of the two outer expressions is which. It doesn't make perfect > sense either way, and I expect a lot of people would be flicking back > to the docs constantly to make sure they had it right. Really? The evaluation order you mention in above didn?t make sense to me until I tried to make sense of it. Ronald From victor.stinner at gmail.com Thu Feb 27 11:47:06 2014 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 27 Feb 2014 11:47:06 +0100 Subject: [Python-Dev] Start writing inlines rather than macros? In-Reply-To: References: Message-ID: Hi, 2014-02-27 11:22 GMT+01:00 Kristj?n Valur J?nsson : > Now, Larry Hastings pointed out that we support C89 which doesn?t support > Inlines. Rather than suggesting here that we update that compatibility > requirement, In practice, recent versions of GCC and Clang are used. On Windows, it's Visual Studio 2010. I'm pretty sure that these compilers support inline functions. I'm also in favor of using inline functions instead of long macros using ugly hacks like "instr1,instr2" syntax where instr1 used assert(). See for example unicodeobject.c to have an idea of what horrible macros mean. I'm in favor of dropping C89 support and require at least C99. There is now C11, it's time to drop the old C89. http://en.wikipedia.org/wiki/C11_%28C_standard_revision%29 Victor From rosuav at gmail.com Thu Feb 27 12:07:08 2014 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 27 Feb 2014 22:07:08 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> Message-ID: On Thu, Feb 27, 2014 at 9:44 PM, Ronald Oussoren wrote: >> expr except Exception try default # "try" indicates the initial expr, >> not the default > > I didn?t parse the expression this way at all, but quite naturally parsed is as ?use expr, and try using default if expr raises Exception? and not as a RTL expression. Thing is, in the statement form, "try doing this" means "do this, and you might get an exception, so deal with it". In the 'try default' form, "try this" means "oops, you got an exception, so try this instead". It's using "try" in the opposite way. >> default except Exception try expr # breaks L->R evaluation order >> >> Left to right evaluation order is extremely important to me. > > I agree with that, RTL evaluation would be pretty odd in Python. > > Really? The evaluation order you mention in above didn?t make sense to me until I tried to make sense of it. The "default except Exception try expr" notation has "try" followed by the thing that might raise an exception, which mimics the statement. The "expr except Exception try default" notation evaluates from left to right. Both make some sense, and I'd say it's on balance which is the more likely to be expected. Imagine an overall expression where it's ambiguous: value = (d["foo"] except KeyError try d["spam"]) Which one do you expect to be tried first? I'd say you could poll a bunch of moderately-experienced Python programmers and get both answers in reasonable numbers. ChrisA From ncoghlan at gmail.com Thu Feb 27 13:57:31 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Feb 2014 22:57:31 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> Message-ID: On 27 February 2014 20:44, Ronald Oussoren wrote: > > On 27 Feb 2014, at 11:09, Chris Angelico wrote: > >> On Thu, Feb 27, 2014 at 7:44 PM, Ronald Oussoren wrote: >>> What about (also mentioned in the PEP)? >>> >>> value = (expr except Exception try default) >>> >>> This seems to read nicely, although "try" is at a completely different position than it is in the equivalent try statement. >>> >>> I like the general idea, but like Brett I don't like using a colon here at all. >> >> I see your "although" clause to be quite a strong objection. In the >> statement form of an if, you have: > > I'm not convinced that this is a strong objection. The order of keywords is different, but that doesn't have to be problem. As Chris notes, the problem is that if you use "try", the two plausible interpretations are: 1. Evaluates left-to-right, try introduces a different part of the syntax (different from every past statement->expression conversion where the clauses are reordered, but almost always introduced by the same keywords as they are in the statement form) 2. Evaluates right-to-left, try introduces the expressions covered by the exception handler (this is just backwards, and significantly harder to follow than even the "middle first" conditional expression) Neither interpretation is particularly acceptable, and the fact that the other interpretation would remain plausible regardless is a further strike against both of them. Personally, I think the PEP makes a good case for particular semantics with a spelling that isn't great, but isn't completely abhorrent either. I definitely think it represents an improvement over the status quo, which is an ongoing proliferation of function-based special cases for doing particular kinds of exception handling as an expression, and the spelling advocated for in the PEP seems like the best of the (many) alternatives that have been proposed. The way I get the colon in the proposed syntax to make sense to my brain is to view it as being more like the colon in a dictionary key:value pair than it is like the one that introduces a suite or the body of a lambda expression: (lst[2] except {IndexError: "No value"}) The analogy isn't exact (since exception handling is isinstance() based rather than equality based), but I think it gives the right general flavour in terms of the intended meaning of the colon in this construct. The analogy could likely be encouraged and strengthened by changing the parentheses requirements to make this read more like a binary except expression with a parenthesised RHS rather than a ternary expression: lst[2] except (IndexError: "No value") Catching multiple errors would involve a parenthesised tuple as the "key": f() except ((TypeError, AttributeError): "No value") The deferred "multiple except clauses" part of the PEP could also change to be more dict display like: value = expr except ( Exception1: default1, Exception2: default2, Exception3: default3, ) Writing out those examples, I actually like that version (where the parentheses are still mandatory, but the left paren is after the except keyword rather than before the first expression) better than the one currently in the PEP. It also avoids the weirdly unbalanced look of the variant that had the left paren *before* the except keyword. The main downside I see is that the exception handling definition syntax would only be permitted in that specific location, but still look a lot like an ordinary parenthesised expression. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From kristjan at ccpgames.com Thu Feb 27 14:06:45 2014 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Thu, 27 Feb 2014 13:06:45 +0000 Subject: [Python-Dev] Start writing inlines rather than macros? In-Reply-To: References: Message-ID: > -----Original Message----- > From: Victor Stinner [mailto:victor.stinner at gmail.com] > Sent: 27. febr?ar 2014 10:47 > To: Kristj?n Valur J?nsson > Cc: Python-Dev (python-dev at python.org) > Subject: Re: [Python-Dev] Start writing inlines rather than macros? > In practice, recent versions of GCC and Clang are used. On Windows, it's > Visual Studio 2010. I'm pretty sure that these compilers support inline > functions. > > I'm also in favor of using inline functions instead of long macros using ugly > hacks like "instr1,instr2" syntax where instr1 used assert(). See for example > unicodeobject.c to have an idea of what horrible macros mean. > > I'm in favor of dropping C89 support and require at least C99. There is now > C11, it's time to drop the old C89. > http://en.wikipedia.org/wiki/C11_%28C_standard_revision%29 well, requiring C99 is another discussion which I'm not so keen on instigating :) As you point out, most of our target platforms probably do support inline already. My question is more of the nature: What about those that don't support inline, is there any harm in defaulting to "static" in that case and leave the inlining to the optimizer on those platforms? K From kristjan at ccpgames.com Thu Feb 27 14:10:32 2014 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 27 Feb 2014 13:10:32 +0000 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: <20140226192642.769b2377@fsol> Message-ID: I agree with NICK. having REF in it is a good idea. So, I'm +1 on setref. Having long explicit macros with exact semantics in the name is a bad one. so I'm -1 on any Py_DECREF_AND_REPLACE or similar daschhunds. Also, is there any real requirement for having separate non-X versions of these? The Xs constitue a permutation explosion, particularly if you want then also versions that INCREF the source :) K From: Python-Dev [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Nick Coghlan Sent: 27. febr?ar 2014 00:12 To: Antoine Pitrou Cc: python-dev at python.org Subject: Re: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc On 27 Feb 2014 04:28, "Antoine Pitrou" > wrote: > > On Wed, 26 Feb 2014 11:40:01 +0200 > Serhiy Storchaka > wrote: > > > There were several suggestions for naming new macros which replace old > > value with new value and then (x)decref old value. > > > > #define Py_XXX(ptr, value) \ > > { \ > > PyObject *__tmp__ = ptr; \ > > ptr = new_value; \ > > Py_DECREF(__tmp__); \ > > } > > > > 1. Py_(X)SETREF. > > My vote is on this one. > I'm also -1 on any name which doesn't have "REF" in it; the name should > clearly suggest that it's a refcounting operation. Yeah, I think SETREF is my favourite as well (even though some of the later suggestions were mine). Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Feb 27 17:30:24 2014 From: brett at python.org (Brett Cannon) Date: Thu, 27 Feb 2014 11:30:24 -0500 Subject: [Python-Dev] Start writing inlines rather than macros? In-Reply-To: References: Message-ID: On Thu, Feb 27, 2014 at 5:47 AM, Victor Stinner wrote: > Hi, > > 2014-02-27 11:22 GMT+01:00 Kristj?n Valur J?nsson : > > Now, Larry Hastings pointed out that we support C89 which doesn?t support > > Inlines. Rather than suggesting here that we update that compatibility > > requirement, > > In practice, recent versions of GCC and Clang are used. On Windows, > it's Visual Studio 2010. I'm pretty sure that these compilers support > inline functions. > > I'm also in favor of using inline functions instead of long macros > using ugly hacks like "instr1,instr2" syntax where instr1 used > assert(). See for example unicodeobject.c to have an idea of what > horrible macros mean. > > I'm in favor of dropping C89 support and require at least C99. There > is now C11, it's time to drop the old C89. > http://en.wikipedia.org/wiki/C11_%28C_standard_revision%29 The Visual Studio team has publicly stated they will never support C99, so dropping C89 blindly is going to alienate a big part of our user base unless we switch to C++ instead. I'm fine with trying to pull in C99 features, though, that we can somehow support in a backwards-compatible way with VS. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sturla.molden at gmail.com Thu Feb 27 18:22:37 2014 From: sturla.molden at gmail.com (Sturla Molden) Date: Thu, 27 Feb 2014 17:22:37 +0000 (UTC) Subject: [Python-Dev] Start writing inlines rather than macros? References: Message-ID: <1900238651415214229.843004sturla.molden-gmail.com@news.gmane.org> Brett Cannon wrote: > The Visual Studio team has publicly stated they will never support C99, > so dropping C89 blindly is going to alienate a big part of our user base > unless we switch to C++ instead. I'm fine with trying to pull in C99 > features, though, that we can somehow support in a backwards-compatible way with VS. So you are saying that Python should use "the C that Visual Studio supports"? I believe Microsoft is not competent to define the C standard. If they cannot provide a compiler that is their bad. There are plenty of other standard-compliant compilers we can use, including Intel, clang and gcc (MinGW). Sturla From brett at python.org Thu Feb 27 19:01:56 2014 From: brett at python.org (Brett Cannon) Date: Thu, 27 Feb 2014 13:01:56 -0500 Subject: [Python-Dev] Start writing inlines rather than macros? In-Reply-To: <1900238651415214229.843004sturla.molden-gmail.com@news.gmane.org> References: <1900238651415214229.843004sturla.molden-gmail.com@news.gmane.org> Message-ID: On Thu, Feb 27, 2014 at 12:22 PM, Sturla Molden wrote: > Brett Cannon wrote: > > > The Visual Studio team has publicly stated they will never support C99, > > so dropping C89 blindly is going to alienate a big part of our user base > > unless we switch to C++ instead. I'm fine with trying to pull in C99 > > features, though, that we can somehow support in a backwards-compatible > way with VS. > > So you are saying that Python should use "the C that Visual Studio > supports"? Well, C89 + Amendments which happens to be what all C compilers support, including VS. > I believe Microsoft is not competent to define the C standard. > Maybe, but they still control a large install base. > If they cannot provide a compiler that is their bad. And unfortunately ours if we want Windows developers to be able to use CPython for things like embedding, > There are plenty of > other standard-compliant compilers we can use, including Intel, clang and > gcc (MinGW). > You manage to convince a majority of Windows developers to switch compilers and then I would be happy to promote we drop VS support and switch entirely to C99. Until then, though, this is like suggesting we cut off Windows XP because MS doesn't have long-term support anymore: it's a choice between being pragmatic for serving our install base or doing something to simplify our lives. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at hotpy.org Thu Feb 27 19:23:14 2014 From: mark at hotpy.org (Mark Shannon) Date: Thu, 27 Feb 2014 18:23:14 +0000 Subject: [Python-Dev] Start writing inlines rather than macros? In-Reply-To: References: Message-ID: <530F8292.3020405@hotpy.org> On 27/02/14 13:06, Kristj?n Valur J?nsson wrote: > > >> -----Original Message----- >> From: Victor Stinner [mailto:victor.stinner at gmail.com] >> Sent: 27. febr?ar 2014 10:47 >> To: Kristj?n Valur J?nsson >> Cc: Python-Dev (python-dev at python.org) >> Subject: Re: [Python-Dev] Start writing inlines rather than macros? >> In practice, recent versions of GCC and Clang are used. On Windows, it's >> Visual Studio 2010. I'm pretty sure that these compilers support inline >> functions. >> >> I'm also in favor of using inline functions instead of long macros using ugly >> hacks like "instr1,instr2" syntax where instr1 used assert(). See for example >> unicodeobject.c to have an idea of what horrible macros mean. >> >> I'm in favor of dropping C89 support and require at least C99. There is now >> C11, it's time to drop the old C89. >> http://en.wikipedia.org/wiki/C11_%28C_standard_revision%29 > > well, requiring C99 is another discussion which I'm not so keen on instigating :) > As you point out, most of our target platforms probably do support inline > already. My question is more of the nature: What about those that don't support > inline, is there any harm in defaulting to "static" in that case and leave the inlining > to the optimizer on those platforms? I agree, modern compilers will inline quite aggressively, so declaring a function static is as good as declaring it inline, provided the function is small. Static functions are a lot easier to read and maintain than LOUD_BUT_UNTYPED_MACRO(x) :) Cheers, Mark. From solipsis at pitrou.net Thu Feb 27 19:51:56 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Feb 2014 19:51:56 +0100 Subject: [Python-Dev] Start writing inlines rather than macros? References: <1900238651415214229.843004sturla.molden-gmail.com@news.gmane.org> Message-ID: <20140227195156.1750ddf9@fsol> On Thu, 27 Feb 2014 17:22:37 +0000 (UTC) Sturla Molden wrote: > Brett Cannon wrote: > > > The Visual Studio team has publicly stated they will never support C99, > > so dropping C89 blindly is going to alienate a big part of our user base > > unless we switch to C++ instead. I'm fine with trying to pull in C99 > > features, though, that we can somehow support in a backwards-compatible way with VS. > > So you are saying that Python should use "the C that Visual Studio > supports"? I believe Microsoft is not competent to define the C standard. > If they cannot provide a compiler that is their bad. There are plenty of > other standard-compliant compilers we can use, including Intel, clang and > gcc (MinGW). Other C compilers don't support all of C99, AFAIR. We sometimes get such bug reports from e.g. AIX users. I'd be all for accepting C99, ideally. But we must care about our users' constraints. Regards Antoine. From skip at pobox.com Thu Feb 27 20:12:24 2014 From: skip at pobox.com (Skip Montanaro) Date: Thu, 27 Feb 2014 13:12:24 -0600 Subject: [Python-Dev] Start writing inlines rather than macros? In-Reply-To: <20140227195156.1750ddf9@fsol> References: <1900238651415214229.843004sturla.molden-gmail.com@news.gmane.org> <20140227195156.1750ddf9@fsol> Message-ID: I think it's at least worthwhile to investigate the use of inline/static functions over the current macros. It's been many years since I looked at them. I doubt they have gotten any easier to read or edit with all their backslashes. I do have one question though. Suppose you encounter a compiler that doesn't understand the inline keyword, so you choose the static declaration as Kristj?n suggested. The resulting Python executable should be functionally correct, but if the optimizer doesn't happen to inline a given static function you might be stuck with some bad performance across-the-board (if it never inlines, or doesn't inline where we really need it to), or only under some circumstances (as a hypothetical example, inlining in dictobject.c, but not in ceval.c). Is there a configurable way to tell if a compiler will inline functions which are declared static, and possibly under what conditions they might not? It might still be necessary to maintain macros for those platforms. Skip From solipsis at pitrou.net Thu Feb 27 20:23:02 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Feb 2014 20:23:02 +0100 Subject: [Python-Dev] Start writing inlines rather than macros? References: <1900238651415214229.843004sturla.molden-gmail.com@news.gmane.org> <20140227195156.1750ddf9@fsol> Message-ID: <20140227202302.134d2a90@fsol> On Thu, 27 Feb 2014 13:12:24 -0600 Skip Montanaro wrote: > I think it's at least worthwhile to investigate the use of > inline/static functions over the current macros. It's been many years > since I looked at them. I doubt they have gotten any easier to read or > edit with all their backslashes. I can assure you they haven't :-) > I do have one question though. Suppose you encounter a compiler that > doesn't understand the inline keyword, so you choose the static > declaration as Kristj?n suggested. The resulting Python executable > should be functionally correct, but if the optimizer doesn't happen to > inline a given static function you might be stuck with some bad > performance across-the-board You're right. Since we only define macros where performance is critical (such as INCREF and DECREF), it would definitely have a very significant impact on performance. > Is there a configurable way to tell if a compiler will inline > functions which are declared static, and possibly under what > conditions they might not? It might still be necessary to maintain > macros for those platforms. Well, if we must maintain macros, let's maintain them everywhere and avoid the burden of two different implementations for the same thing. Regards Antoine. From skip at pobox.com Thu Feb 27 20:27:02 2014 From: skip at pobox.com (Skip Montanaro) Date: Thu, 27 Feb 2014 13:27:02 -0600 Subject: [Python-Dev] Start writing inlines rather than macros? In-Reply-To: <20140227202302.134d2a90@fsol> References: <1900238651415214229.843004sturla.molden-gmail.com@news.gmane.org> <20140227195156.1750ddf9@fsol> <20140227202302.134d2a90@fsol> Message-ID: On Thu, Feb 27, 2014 at 1:23 PM, Antoine Pitrou wrote: > Well, if we must maintain macros, let's maintain them everywhere and > avoid the burden of two different implementations for the same thing. Would it be possible to generate the macro versions from the inline/static versions where necessary, as some sort of pre-compile step? S From v+python at g.nevcal.com Thu Feb 27 20:36:01 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 27 Feb 2014 11:36:01 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> Message-ID: <530F93A1.4050106@g.nevcal.com> On 2/27/2014 4:57 AM, Nick Coghlan wrote: > The way I get the colon in the proposed syntax to make sense to my > brain is to view it as being more like the colon in a dictionary > key:value pair than it is like the one that introduces a suite or the > body of a lambda expression: > > (lst[2] except {IndexError: "No value"}) > > The analogy isn't exact (since exception handling is isinstance() > based rather than equality based), but I think it gives the right > general flavour in terms of the intended meaning of the colon in this > construct. The analogy could likely be encouraged and strengthened by > changing the parentheses requirements to make this read more like a > binary except expression with a parenthesised RHS rather than a > ternary expression: > > lst[2] except (IndexError: "No value") > > Catching multiple errors would involve a parenthesised tuple as the "key": > > f() except ((TypeError, AttributeError): "No value") > > The deferred "multiple except clauses" part of the PEP could also > change to be more dict display like: > > value = expr except ( > Exception1: default1, > Exception2: default2, > Exception3: default3, > ) > > Writing out those examples, I actually like that version (where the > parentheses are still mandatory, but the left paren is after the > except keyword rather than before the first expression) better than > the one currently in the PEP. It also avoids the weirdly unbalanced > look of the variant that had the left paren*before* the except > keyword. The main downside I see is that the exception handling > definition syntax would only be permitted in that specific location, > but still look a lot like an ordinary parenthesised expression. > > Cheers, > Nick. +1 f() except ((TypeError, AttributeError): "No value") is a nice extension to the idea of value = expr except ( Exception1: default1, Exception2: default2, Exception3: default3, ) Which I've liked since I first saw it, as it neatly solves handling multiple except clauses. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Feb 27 20:36:56 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 27 Feb 2014 20:36:56 +0100 Subject: [Python-Dev] Start writing inlines rather than macros? In-Reply-To: References: <1900238651415214229.843004sturla.molden-gmail.com@news.gmane.org> <20140227195156.1750ddf9@fsol> <20140227202302.134d2a90@fsol> Message-ID: <20140227203656.66eaca5b@fsol> On Thu, 27 Feb 2014 13:27:02 -0600 Skip Montanaro wrote: > On Thu, Feb 27, 2014 at 1:23 PM, Antoine Pitrou wrote: > > Well, if we must maintain macros, let's maintain them everywhere and > > avoid the burden of two different implementations for the same thing. > > Would it be possible to generate the macro versions from the > inline/static versions where necessary, as some sort of pre-compile > step? Uh... well, perhaps, but one still has to check if the macros *are* correct. (I'm not sure a static-to-macro converter would be trivial to write) Regards Antoine. From rosuav at gmail.com Thu Feb 27 20:55:24 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 28 Feb 2014 06:55:24 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <530F93A1.4050106@g.nevcal.com> References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: On Fri, Feb 28, 2014 at 6:36 AM, Glenn Linderman wrote: > +1 > > f() except ((TypeError, AttributeError): "No value") > > is a nice extension to the idea of > > value = expr except ( > Exception1: default1, > Exception2: default2, > Exception3: default3, > ) > > Which I've liked since I first saw it, as it neatly solves handling multiple > except clauses. You can already list multiple exception types. The definition of the exception_list is exactly as per the try/except statement (bar the 'as' keyword, which I would love to see implemented some day). ChrisA From rosuav at gmail.com Thu Feb 27 22:29:53 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 28 Feb 2014 08:29:53 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: I've added another utility script to my PEP draft repo: https://github.com/Rosuav/ExceptExpr/blob/master/replace_except_expr.py It's built out of some pretty horrendous hacks, it makes some assumptions about code layout (eg spaces for indentation, and a try/except block never has anything else on the same line(s)), and the code's a bit messy, but it does work. I ran it on the Python stdlib and it produced a whole lot of edits (I had it keep the old version in comments, bracketed with markers with the text "PEP 463" in them, which makes it easy to find and analyze); one file (with two try blocks) causes a test failure, but all the rest still pass. If people can try the script on their own codebases and see how it looks, that'd be great. I recommend first running the script with ast.Expr handling removed (as per the file you see above), and then maybe running it with that one line commented out. You'll get a lot of unhelpful change suggestions from the double-expression handler. Could a core committer please apply the last few changes to the PEP? https://raw.github.com/Rosuav/ExceptExpr/master/pep-0463.txt or the diff is below. I think this is the last important change left; things have gone fairly quiet here, and I think the PEP's about ready to request pronouncement, unless someone knows of something I've forgotten. (Have I said "I'll write up a paragraph about that" about anything and not done it yet? Now's the perfect time to remind me.) Thanks! ChrisA diff -r 5f63c8a92d1c pep-0463.txt --- a/pep-0463.txt Mon Feb 24 14:22:46 2014 -0800 +++ b/pep-0463.txt Fri Feb 28 08:25:33 2014 +1100 @@ -43,6 +43,34 @@ * statistics.mean(data) - no way to handle an empty iterator +Had this facility existed early in Python's history, there would have been +no need to create dict.get() and related methods; the one obvious way to +handle an absent key would be to respond to the exception. One method is +written which signal the absence in one way, and one consistent technique +is used to respond to the absence. Instead, we have dict.get(), and as of +Python 3.4, we also have min(... default=default), and myriad others. We +have a LBYL syntax for testing inside an expression, but there is currently +no EAFP notation; compare the following:: + + # LBYL: + if key in dic: + process(dic[key]) + else: + process(None) + # As an expression: + process(dic[key] if key in dic else None) + + # EAFP: + try: + process(dic[key]) + except KeyError: + process(None) + # As an expression: + process(dic[key] except KeyError: None) + +Python generally recommends the EAFP policy, but must then proliferate +utility functions like dic.get(key,None) to enable this. + Rationale ========= @@ -338,6 +366,19 @@ except KeyError: u = tarinfo.uid +Look up an attribute, falling back on a default:: + mode = (f.mode except AttributeError: 'rb') + + # Lib/aifc.py:882: + if hasattr(f, 'mode'): + mode = f.mode + else: + mode = 'rb' + + return (sys._getframe(1) except AttributeError: None) + # Lib/inspect.py:1350: + return sys._getframe(1) if hasattr(sys, "_getframe") else None + Perform some lengthy calculations in EAFP mode, handling division by zero as a sort of sticky NaN:: @@ -616,7 +657,7 @@ it would be with the statement form, and as its syntax is a point on which consensus has not been reached, the entire feature is deferred. -Multiple 'except' keywords can be used, and they will all catch +Multiple 'except' keywords could be used, and they will all catch exceptions raised in the original expression (only):: # Will catch any of the listed exceptions thrown by expr; -- end -- From rosuav at gmail.com Thu Feb 27 22:32:00 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 28 Feb 2014 08:32:00 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: On Fri, Feb 28, 2014 at 8:29 AM, Chris Angelico wrote: > @@ -43,6 +43,34 @@ > > * statistics.mean(data) - no way to handle an empty iterator > > +Had this facility existed early in Python's history, there would have been > +no need to create dict.get() and related methods; the one obvious way to > +handle an absent key would be to respond to the exception. One method is > +written which signal the absence in one way, and one consistent technique Doh! Typical... I notice the typo only after hitting send. This should be "which signals". The linked-to draft file has been updated. ChrisA From ncoghlan at gmail.com Fri Feb 28 00:33:52 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Feb 2014 09:33:52 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: On 28 Feb 2014 05:56, "Chris Angelico" wrote: > > On Fri, Feb 28, 2014 at 6:36 AM, Glenn Linderman wrote: > > +1 > > > > f() except ((TypeError, AttributeError): "No value") > > > > is a nice extension to the idea of > > > > value = expr except ( > > Exception1: default1, > > Exception2: default2, > > Exception3: default3, > > ) > > > > Which I've liked since I first saw it, as it neatly solves handling multiple > > except clauses. > > You can already list multiple exception types. The definition of the > exception_list is exactly as per the try/except statement (bar the > 'as' keyword, which I would love to see implemented some day). Note that this example is covering the deferred case of multiple except clauses that resolve to different values, not the already handled case of multiple exception types that resolve to the same value. Anyway, even if you choose not to switch the parenthesis requirement in PEP, it should at least make the "colon as in dict display, not as in suite introduction" comparison, and note this as an alternative proposal for the required parentheses. Cheers, Nick. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 28 00:40:55 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 28 Feb 2014 10:40:55 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: On Fri, Feb 28, 2014 at 10:33 AM, Nick Coghlan wrote: > > On 28 Feb 2014 05:56, "Chris Angelico" wrote: >> >> On Fri, Feb 28, 2014 at 6:36 AM, Glenn Linderman >> wrote: >> > +1 >> > >> > f() except ((TypeError, AttributeError): "No value") >> > >> > is a nice extension to the idea of >> > >> > value = expr except ( >> > Exception1: default1, >> > Exception2: default2, >> > Exception3: default3, >> > ) >> > >> > Which I've liked since I first saw it, as it neatly solves handling >> > multiple >> > except clauses. >> >> You can already list multiple exception types. The definition of the >> exception_list is exactly as per the try/except statement (bar the >> 'as' keyword, which I would love to see implemented some day). > > Note that this example is covering the deferred case of multiple except > clauses that resolve to different values, not the already handled case of > multiple exception types that resolve to the same value. The "nice extension" to that notation is already handled, though. > Anyway, even if you choose not to switch the parenthesis requirement in PEP, > it should at least make the "colon as in dict display, not as in suite > introduction" comparison, and note this as an alternative proposal for the > required parentheses. Hmm. Not sure that it really helps, but okay. Will word something up. ChrisA From v+python at g.nevcal.com Fri Feb 28 03:12:05 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 27 Feb 2014 18:12:05 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> Message-ID: <530FF075.8020303@g.nevcal.com> On 2/27/2014 3:40 PM, Chris Angelico wrote: > On Fri, Feb 28, 2014 at 10:33 AM, Nick Coghlan wrote: >> On 28 Feb 2014 05:56, "Chris Angelico" wrote: >>> On Fri, Feb 28, 2014 at 6:36 AM, Glenn Linderman >>> wrote: >>>> +1 >>>> >>>> f() except ((TypeError, AttributeError): "No value") >>>> >>>> is a nice extension to the idea of >>>> >>>> value = expr except ( >>>> Exception1: default1, >>>> Exception2: default2, >>>> Exception3: default3, >>>> ) >>>> >>>> Which I've liked since I first saw it, as it neatly solves handling >>>> multiple >>>> except clauses. >>> You can already list multiple exception types. The definition of the >>> exception_list is exactly as per the try/except statement (bar the >>> 'as' keyword, which I would love to see implemented some day). >> Note that this example is covering the deferred case of multiple except >> clauses that resolve to different values, not the already handled case of >> multiple exception types that resolve to the same value. > The "nice extension" to that notation is already handled, though. Yes. But the point is really the location of the (), sorry if my "nice extension" comment is throwing you off that track. value = expr except ( Exception1: default1, Exception2: default2, Exception3: default3, ) > >> Anyway, even if you choose not to switch the parenthesis requirement in PEP, >> it should at least make the "colon as in dict display, not as in suite >> introduction" comparison, and note this as an alternative proposal for the >> required parentheses. > Hmm. Not sure that it really helps, but okay. Will word something up. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 28 04:46:48 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 28 Feb 2014 14:46:48 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <530FF075.8020303@g.nevcal.com> References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> Message-ID: On Fri, Feb 28, 2014 at 1:12 PM, Glenn Linderman wrote: > Yes. But the point is really the location of the (), sorry if my "nice > extension" comment is throwing you off that track. Ah! I see. We touched on this syntax on -ideas, but at the time, the proposed syntax didn't have parens around the outside. Either location will solve most of the same problems (like precedence/associativity versus multiple except clauses), but shrinking down to just the exception list and default value makes it look very different. It looks like what's inside should be an expression, which it isn't. What's the advantage of this form? What's its key advantage over the parens-around-the-whole-thing form? ChrisA From stephen at xemacs.org Fri Feb 28 07:26:32 2014 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 28 Feb 2014 15:26:32 +0900 Subject: [Python-Dev] Start writing inlines rather than macros? In-Reply-To: References: <1900238651415214229.843004sturla.molden-gmail.com@news.gmane.org> <20140227195156.1750ddf9@fsol> <20140227202302.134d2a90@fsol> Message-ID: <87vbvzsqg7.fsf@uwakimon.sk.tsukuba.ac.jp> Skip Montanaro writes: > On Thu, Feb 27, 2014 at 1:23 PM, Antoine Pitrou wrote: > > Well, if we must maintain macros, let's maintain them everywhere and > > avoid the burden of two different implementations for the same thing. > > Would it be possible to generate the macro versions from the > inline/static versions where necessary, as some sort of pre-compile > step? To do that in general is a hard problem, unless you're a C compiler that already supports inlining. I'm not sure how easy it would be to define rules that support the inlines Python demands for performance, and yet makes function to macro conversion easy. I think it makes much more sense to look at rewriting the macros by hand so that they emulate the necessary amount of function semantics (eg, providing argument type-checking in debug builds, not mutating arguments unless passed by reference, evaluating arguments exactly once, etc). XEmacs has a lot of these for precisely the reason this thread started (plus the recognition at the time we started this practice that a number of our platforms could not be trusted to inline certain performance-critical routines). They're spelled using the same conventions as functions, so macros that don't obey those rules stick out. Then these function-like macros could be #ifdef'd with the inline functions, with some (automated) comparison testing on appropriate benchmarks by building with and without -dDONT_TRUST_INLINE. And then beta-testing by defaulting to #undef DONT_TRUST_INLINE. Then wait for cries of pain -- if none, remove the dead #ifdef branches in the last beta or so. Yes, this involves a certain amount of work, but I think it's bounded. There should be no extra maintenance of the #ifdef'd parts compared to the maintenance required to fix bugs resulting from change from macro semantics to inline semantics anyway, unless changing the macros is mistake-prone -- which will be recognized immediately. From v+python at g.nevcal.com Fri Feb 28 08:38:57 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 27 Feb 2014 23:38:57 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> Message-ID: <53103D11.9030204@g.nevcal.com> On 2/27/2014 7:46 PM, Chris Angelico wrote: > On Fri, Feb 28, 2014 at 1:12 PM, Glenn Linderman wrote: >> Yes. But the point is really the location of the (), sorry if my "nice >> extension" comment is throwing you off that track. > Ah! I see. > > We touched on this syntax on -ideas, but at the time, the proposed > syntax didn't have parens around the outside. Either location will > solve most of the same problems (like precedence/associativity versus > multiple except clauses), but shrinking down to just the exception > list and default value makes it look very different. It looks like > what's inside should be an expression, which it isn't. What's the > advantage of this form? What's its key advantage over the > parens-around-the-whole-thing form? Key advantage to me is that if a function call or other expression may produce multiple exceptions, the syntax doesn't require repeating the "except" over and over. By not repeating the except over and over, there is less ambiguity about what expression the Exception-lists apply to, when there is more than one Exception list in the expression. Whereas the current PEP syntax has ambiguity regarding how to interpret a-expr except except-list-b: b-expr except except-list-c: c-expr (does the 2nd except apply to a-expr or b-expr?), without parentheses, and, as far as I am concerned, even with the parentheses, this syntax makes it very clear that each of the Exception-lists apply to a-expr. Key advantage to others may be that because the : is within the () [and the leading ( is quite nearby, making it obvious], it is less likely to be considered a statement boundary, and more easily explained as a special type of list syntax... not _really_ a list, because it is really code to be executed somewhat sequentially rather than data, and lists don't have : ... and not _really_ a dict constant, which does have :, because the Exception is not _really_ a key, but the syntax can draw on analogies with the dict constant syntax which will help people remember it, and even sort of understand that there is a pair-wise relationship between the Exception-list and the expression after the :, without repeating the except over and over. -------------- next part -------------- An HTML attachment was scrubbed... URL: From sppook at live.com Fri Feb 28 05:15:14 2014 From: sppook at live.com (sppook at live.com) Date: Thu, 27 Feb 2014 21:15:14 -0700 Subject: [Python-Dev] [PATCH] unicode subtypes broken in latest py3k? debug builds Message-ID: Sent from my Windows Phone -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 28 09:41:49 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 28 Feb 2014 19:41:49 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <53103D11.9030204@g.nevcal.com> References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> <53103D11.9030204@g.nevcal.com> Message-ID: On Fri, Feb 28, 2014 at 6:38 PM, Glenn Linderman wrote: > Whereas the current PEP syntax has ambiguity regarding how to interpret > a-expr except except-list-b: b-expr except except-list-c: c-expr (does the > 2nd except apply to a-expr or b-expr?), without parentheses, and, as far as > I am concerned, even with the parentheses, this syntax makes it very clear > that each of the Exception-lists apply to a-expr. Fair enough. It's a bit hard to talk about multiple except expressions, though, as they're a problem unless formally supported - and they're almost never needed. Really, all you need to do is say "never abut except-expressions without parentheses" (which the current proposal and the "parens around the exception bit only" proposal both enforce), and then there's no problem. I expect that normal use of this won't include any form of chaining. Yes, it can - like any feature - be used abnormally, but at some point it's better to just drop it out as a statement. > Key advantage to others may be that because the : is within the () [and the > leading ( is quite nearby, making it obvious], it is less likely to be > considered a statement boundary, and more easily explained as a special type > of list syntax... not _really_ a list, because it is really code to be > executed somewhat sequentially rather than data, and lists don't have : ... > and not _really_ a dict constant, which does have :, because the Exception > is not _really_ a key, but the syntax can draw on analogies with the dict > constant syntax which will help people remember it, and even sort of > understand that there is a pair-wise relationship between the Exception-list > and the expression after the :, without repeating the except over and over. See the confusing terminology we have here? It might be called a "list" of except-expressions, but the brackets are round, and a list's are square. It's kinda like dict syntax, only again, the other sort of bracket, and it's wrong to try to construct a dict; it'd be too tempting to conflate this with some of the other current proposals for lazily-evaluated expressions (aka "simpler syntax for lambda" or other terms). This is, fundamentally, a multi-part expression on par with "and" and "or": first, evaluate the primary expression; then, if an exception is raised, evaluate the exception list and see if it matches; then, if it matches, squash the exception and evaluate the default expression. You can't turn that into a dict, partly because you'd need to sort out lazy evaluation, and partly because a dict is unordered - if this is expanded to support multiple except clauses, they have to be processed in order. (You might, for instance, catch ZeroDivisionError, and then Exception, with different handling. It'd be VERY confusing for them to be processed in the wrong order, particularly if it happens unpredictably.) Are there any other expressions that allow parens around a part of the expression, without the stuff inside them becoming a completely separate sub-expression? ChrisA From larry at hastings.org Fri Feb 28 09:56:11 2014 From: larry at hastings.org (Larry Hastings) Date: Fri, 28 Feb 2014 00:56:11 -0800 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: Message-ID: <53104F2B.5080200@hastings.org> On 02/26/2014 11:13 PM, Georg Brandl wrote: > Am 26.02.2014 17:09, schrieb Ryan Gonzalez: >> I like Py_DECREF_REPLACE. It gives the impression that it decrefs the original >> and replaces it. > Agreed, most other suggestions are not really explicit enough. +1 from me too. When I saw Py_SETREF I thought, oh, it sets the thing and increfs it. FWIW this vote is just on the name. I haven't stared at the whole Py_REPLACE idea enough to have an opinion about whether or not to use it. But if we use it I'm +1 on Py_DECREF_REPLACE. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Fri Feb 28 10:30:43 2014 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 28 Feb 2014 01:30:43 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> <53103D11.9030204@g.nevcal.com> Message-ID: <53105743.5080407@g.nevcal.com> On 2/28/2014 12:41 AM, Chris Angelico wrote: > On Fri, Feb 28, 2014 at 6:38 PM, Glenn Linderman wrote: >> Whereas the current PEP syntax has ambiguity regarding how to interpret >> a-expr except except-list-b: b-expr except except-list-c: c-expr (does the >> 2nd except apply to a-expr or b-expr?), without parentheses, and, as far as >> I am concerned, even with the parentheses, this syntax makes it very clear >> that each of the Exception-lists apply to a-expr. > Fair enough. It's a bit hard to talk about multiple except > expressions, though, as they're a problem unless formally supported - > and they're almost never needed. Really, all you need to do is say > "never abut except-expressions without parentheses" (which the current > proposal and the "parens around the exception bit only" proposal both > enforce), and then there's no problem. I expect that normal use of > this won't include any form of chaining. Yes, it can - like any > feature - be used abnormally, but at some point it's better to just > drop it out as a statement. > >> Key advantage to others may be that because the : is within the () [and the >> leading ( is quite nearby, making it obvious], it is less likely to be >> considered a statement boundary, and more easily explained as a special type >> of list syntax... not _really_ a list, because it is really code to be >> executed somewhat sequentially rather than data, and lists don't have : ... >> and not _really_ a dict constant, which does have :, because the Exception >> is not _really_ a key, but the syntax can draw on analogies with the dict >> constant syntax which will help people remember it, and even sort of >> understand that there is a pair-wise relationship between the Exception-list >> and the expression after the :, without repeating the except over and over. > See the confusing terminology we have here? It might be called a > "list" of except-expressions, but the brackets are round, and a list's > are square. It's kinda like dict syntax, only again, the other sort of > bracket, and it's wrong to try to construct a dict; it'd be too > tempting to conflate this with some of the other current proposals for > lazily-evaluated expressions (aka "simpler syntax for lambda" or other > terms). This is, fundamentally, a multi-part expression on par with > "and" and "or": first, evaluate the primary expression; then, if an > exception is raised, evaluate the exception list and see if it > matches; then, if it matches, squash the exception and evaluate the > default expression. You can't turn that into a dict, partly because > you'd need to sort out lazy evaluation, and partly because a dict is > unordered - if this is expanded to support multiple except clauses, > they have to be processed in order. (You might, for instance, catch > ZeroDivisionError, and then Exception, with different handling. It'd > be VERY confusing for them to be processed in the wrong order, > particularly if it happens unpredictably.) That's why I kept saying "not _really_" :) It isn't a list, but it has an order requirement; it isn't a dict, but it has pairs; and finally, it isn't data, but code. But nonetheless the analogies are somewhat useful. > Are there any other expressions that allow parens around a part of the > expression, without the stuff inside them becoming a completely > separate sub-expression? Sure. Function invocation. You can claim (and it is accurate) that the stuff inside is somewhat independent of the actual function called, but the syntax is function-name open-paren parameter-list close-paren, and the stuff in the parens would be a tuple if it were purely data, except not quite a tuple because some items are pairs (name and value), but it winds up being neither a tuple, nor a list, nor a dict, but instead a complex structure related to code execution :) Actually, this sounds quite similar to value = expr except ( Exception1: default1, Exception2: default2, Exception3: default3, ) except that to get the pairing aspect of some parameters for a function call, you use = instead of :, and instead of a named function it has an expression and a keyword. Not being an expert parser generator, I don't know if the () could be made optional if there is only one exception-list, but that would also remove one of the benefits some folks might perceive with using this syntax, and would also make the analogy with function call syntax a little less comparable. Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristjan at ccpgames.com Fri Feb 28 10:47:07 2014 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Fri, 28 Feb 2014 09:47:07 +0000 Subject: [Python-Dev] Start writing inlines rather than macros? In-Reply-To: References: <1900238651415214229.843004sturla.molden-gmail.com@news.gmane.org> <20140227195156.1750ddf9@fsol> Message-ID: > -----Original Message----- > From: Python-Dev [mailto:python-dev- > bounces+kristjan=ccpgames.com at python.org] On Behalf Of Skip Montanaro > Sent: 27. febr?ar 2014 19:12 > To: python-dev Dev > Subject: Re: [Python-Dev] Start writing inlines rather than macros? > one question though. Suppose you encounter a compiler that > doesn't understand the inline keyword, so you choose the static declaration > as Kristj?n suggested. The resulting Python executable should be functionally > correct, but if the optimizer doesn't happen to inline a given static function > you might be stuck with some bad performance across-the-board (if it never > inlines, or doesn't inline where we really need it to), or only under some > circumstances (as a hypothetical example, inlining in dictobject.c, but not in > ceval.c). > Is there a configurable way to tell if a compiler will inline functions which are > declared static, and possibly under what conditions they might not? It might > still be necessary to maintain macros for those platforms. It would be horrible to have to maintain both macros and functions. My suggestion would be to use functions for new code, and new use cases. We would stick with Py_INCREF() , and continue using that in obvious cases, (such as in the implementation of the new functions) but at the same time introduce a Py_IncRef() function returning a new reference, which can be used in new code where it is convenient. The new macros under discussion, Py_INCREF_AND_REPLACE_WITH_GUSTO() and all of them would then be redesigned in a more composable functional form. This way, only new code would be compiled with different efficiency on different platforms, thus avoiding introducing performance regressions. K From thomas at python.org Fri Feb 28 11:59:19 2014 From: thomas at python.org (Thomas Wouters) Date: Fri, 28 Feb 2014 02:59:19 -0800 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <53105743.5080407@g.nevcal.com> References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> <53103D11.9030204@g.nevcal.com> <53105743.5080407@g.nevcal.com> Message-ID: On Fri, Feb 28, 2014 at 1:30 AM, Glenn Linderman wrote: > > value = expr except ( > Exception1: default1, > Exception2: default2, > Exception3: default3, > ) > > except that to get the pairing aspect of some parameters for a function call, you use = instead of :, and instead of a named function it has an expression and a keyword. > > [ Cue people suggesting the use of '=' or '=>' or '->' instead of ':' ] > Not being an expert parser generator, I don't know if the () could > be made optional if there is only one exception-list, but that would > also remove one of the benefits some folks might perceive with using > this syntax, and would also make the analogy with function call > syntax a little less comparable. > > I believe it's possible, but it's probably tricky: the parser has to consider two or three cases: 1. expr1 except expr2: expr4 2. expr1 except (expr2: expr4) 3. expr1 except (expr2, expr3): expr4 4. expr1 except ((expr2, expr3): expr4) (for simplicity I use 'expr' here, which is entirely the wrong thing to do in terms of CPython's grammar: there's different types of expressions for different situations. #2 and #4 are actually automatically derived from #1 and #3 by 'expr' including tuples and parenthesized expressions.) CPython's parser is a LL(1) parser, which means it can look ahead 1 character to choose between alternatives (and that's not going to change, Guido likes it this way -- and so I do, personally :). Looking at #2 and #3 you can't tell which alternative to use until you see the ':' or ',' after 'expr2', which is too late. The only way to handle that, I think, is to include all possible alternatives in the grammar for the 'except' statement, duplicating logic and definitions from a lot of places. We already do this kind of thing to deal with other potentially-ambiguous situations (to handle dicts and sets and dict comprehensions and set comprehensions, for example) but this would be even more duplication. It would be easier if we weren't talking about '(' or any other token that can be part of a normal expression ;-P [ Cue people suggesting the use of 'expr1 except < expr2: expr4 >'... ] -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 28 12:46:51 2014 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 28 Feb 2014 22:46:51 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: <53105743.5080407@g.nevcal.com> References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> <53103D11.9030204@g.nevcal.com> <53105743.5080407@g.nevcal.com> Message-ID: On Fri, Feb 28, 2014 at 8:30 PM, Glenn Linderman wrote: > > Are there any other expressions that allow parens around a part of the > > expression, without the stuff inside them becoming a completely > > separate sub-expression? > > > Sure. Function invocation. You can claim (and it is accurate) that the > stuff inside is somewhat independent of the actual function called, but the > syntax is function-name open-paren parameter-list close-paren, and the > stuff in the parens would be a tuple if it were purely data, except not > quite a tuple because some items are pairs (name and value), but it winds up > being neither a tuple, nor a list, nor a dict, but instead a complex > structure related to code execution :) Thanks, that's exactly the sort of example I was looking for :) For some reason I mentally blanked and didn't think of that. My point is, if you're going to look for parens around the bit after except, we should look at styling and layout that follows how that's done. As TW says: > [ Cue people suggesting the use of '=' or '=>' or '->' instead of ':' ] If this is to be laid out like a function call, '=' would indeed make sense. But let's stick to the colon for now, so this is a mix of dict-style and function-call-style. expr except(Exception: default) In a more function-call-style layout, that would be: expr except(Exception=default) Both of those look reasonable, and they're tying the parens to the word "except" so it can't *actually* be a function call. Then it looks like it should be laid out like this: except(expr, Exception=default) and it really *is* looking like a function call - and now it's hit the uncanny valley [1], where it's close enough to be a problem (eg because of lazy evaluation, or because "Exception" could be "(AttributeError, KeyError)", which can't be a keyword argument). I don't like this last form; does anyone else support it? Removing the space after the word "except" makes me a lot less averse to this form. Funny how stylistic choices can influence a structural one! Here are a few more examples. How do other people feel about them? cond = args[1] except(IndexError: None) pwd = os.getcwd() except(OSError: None) e.widget = self._nametowidget(W) except(KeyError=W) line = readline() except(StopIteration='') _CONFIG_VARS['abiflags'] = sys.abiflags except(AttributeError: '') def getNamedItem(self, name): return self._attrs[name] except(KeyError: None) g = grp.getgrnam(tarinfo.gname)[2] except(KeyError=tarinfo.gid) u = pwd.getpwnam(tarinfo.uname)[2] except(KeyError=tarinfo.uid) mode = f.mode except(AttributeError: 'rb') return sys._getframe(1) except(AttributeError=None) ips.append(ip.ip except(AttributeError: ip.network_address)) dirlist.append(_os.getcwd() except((AttributeError, OSError)=_os.curdir)) The last one is really critical here, I think. It's the only stdlib example I could find that catches two different errors (IIRC). Here it is with the colon: dirlist.append(_os.getcwd() except((AttributeError, OSError): _os.curdir)) Thoughts? ChrisA [1] http://tvtropes.org/pmwiki/pmwiki.php/Main/UncannyValley From ncoghlan at gmail.com Fri Feb 28 13:27:00 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Feb 2014 22:27:00 +1000 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: <53104F2B.5080200@hastings.org> References: <53104F2B.5080200@hastings.org> Message-ID: On 28 Feb 2014 19:05, "Larry Hastings" wrote: > > On 02/26/2014 11:13 PM, Georg Brandl wrote: >> >> Am 26.02.2014 17:09, schrieb Ryan Gonzalez: >>> >>> I like Py_DECREF_REPLACE. It gives the impression that it decrefs the original >>> and replaces it. >> >> Agreed, most other suggestions are not really explicit enough. > > > +1 from me too. When I saw Py_SETREF I thought, oh, it sets the thing and increfs it. > > FWIW this vote is just on the name. I haven't stared at the whole Py_REPLACE idea enough to have an opinion about whether or not to use it. But if we use it I'm +1 on Py_DECREF_REPLACE. For additional context, the idea itself is necessary for the same reason Py_CLEAR was added: to help ensure that an object's state is never pointing at another object that is in the process of being deleted. The difference is that Py_CLEAR only allows setting the pointer to NULL, while the point of the new macro is to set it to an arbitrary existing point. There is no implicit incref as that isn't needed for correctness (you can do the incref before the pointer replacement, and often the reference count will already be correct without an explicit incref anyway). With the new macro in place, the existing Py_CLEAR(x) macro would be equivalent to Py_SETREF(x, NULL). Originally I was also concerned about the "how will people know there's no implicit incref?", but I've since become satisfied with the fact that the precedent set by the reference stealing SET_ITEM macros is strong enough to justify the shorter name. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Feb 28 13:51:03 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Feb 2014 22:51:03 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> <53103D11.9030204@g.nevcal.com> <53105743.5080407@g.nevcal.com> Message-ID: On 28 February 2014 21:46, Chris Angelico wrote: > On Fri, Feb 28, 2014 at 8:30 PM, Glenn Linderman wrote: >> > Are there any other expressions that allow parens around a part of the >> > expression, without the stuff inside them becoming a completely >> > separate sub-expression? >> >> Sure. Function invocation. You can claim (and it is accurate) that the >> stuff inside is somewhat independent of the actual function called, but the >> syntax is function-name open-paren parameter-list close-paren, and the >> stuff in the parens would be a tuple if it were purely data, except not >> quite a tuple because some items are pairs (name and value), but it winds up >> being neither a tuple, nor a list, nor a dict, but instead a complex >> structure related to code execution :) > > Thanks, that's exactly the sort of example I was looking for :) For > some reason I mentally blanked and didn't think of that. My point is, > if you're going to look for parens around the bit after except, we > should look at styling and layout that follows how that's done. Also generator expressions and most uses of yield or yield from as embedded expressions. Parentheses are our general "this next bit may not be following the normal syntax rules" utility, in addition to being used to override the normal precedence rules (square brackets and curly braces similarly denote regions where the parsing rules may differ from the top level ones). As far as the specific problem I'm attacking with this variant of the proposal goes, one of the recurring reactions from different people has been "but colons are used to introduce suites, not expressions". This impression is not, in fact, correct, as the colon is already used as a subexpression separator in four cases: * dict display key:value pairs * slice notation start:stop:step triples * function parameter : annotation separation * lambda expression arg list : return value separation PEP 463 just adds a fifth such case, as an exception spec:result separator. The preferred notation in the PEP most resembles the existing lambda use case, with "except" instead of "lambda", an exception handling spec instead of an argument list and an additional leading expression: (expr except Exception: default) Lots of people don't like the lambda notation though, so it isn't necessarily a particularly compelling parallel to use. By contrast, it's rare to hear any objections to the {key:value} dict display syntax. Hence the proposed tweak to the syntax to define an "exception handler expression" syntax that is analogous to a dict display rather than a lambda expression: expr except (Exception: default) However, I have realised that there *is* a major downside to that notation, which is that it lacks the connotations of lazy evaluation associated with lambda expressions, whereas the default result of an except expression won't be evaluated at all if the exception isn't thrown. So I think that on balance, I actually do prefer your current proposal. That said, I do think this is a variant worth discussing explicitly in the PEP, if only to remind people that there's definitely precedent for using a colon to separate two subexpressions inside a larger expression element - it's not *just* used to introduce suites, even though that is by far the most *common* use case. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Fri Feb 28 14:07:09 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 1 Mar 2014 00:07:09 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> <53103D11.9030204@g.nevcal.com> <53105743.5080407@g.nevcal.com> Message-ID: On Fri, Feb 28, 2014 at 11:51 PM, Nick Coghlan wrote: > So I think that on balance, I actually do prefer your current > proposal. That said, I do think this is a variant worth discussing > explicitly in the PEP, if only to remind people that there's > definitely precedent for using a colon to separate two subexpressions > inside a larger expression element - it's not *just* used to introduce > suites, even though that is by far the most *common* use case. I've added a bit more to the PEP about that. https://github.com/Rosuav/ExceptExpr/commit/f32387 Does that explain it, do you think? ChrisA From ncoghlan at gmail.com Fri Feb 28 14:24:17 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Feb 2014 23:24:17 +1000 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> <53103D11.9030204@g.nevcal.com> <53105743.5080407@g.nevcal.com> Message-ID: On 28 February 2014 23:07, Chris Angelico wrote: > On Fri, Feb 28, 2014 at 11:51 PM, Nick Coghlan wrote: >> So I think that on balance, I actually do prefer your current >> proposal. That said, I do think this is a variant worth discussing >> explicitly in the PEP, if only to remind people that there's >> definitely precedent for using a colon to separate two subexpressions >> inside a larger expression element - it's not *just* used to introduce >> suites, even though that is by far the most *common* use case. > > I've added a bit more to the PEP about that. > > https://github.com/Rosuav/ExceptExpr/commit/f32387 > > Does that explain it, do you think? Yeah, that works. You may also want to add a "common objections" section to explicitly cover the "but colons introduce suites" objection. That would provide a space to explicitly list all of the current "not introducing a suite" use cases for the colon in Python's syntax, since increasing that tally from four to five is less radical than introducing the first non-suite related use case for the symbol. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Fri Feb 28 14:39:46 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 1 Mar 2014 00:39:46 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> <53103D11.9030204@g.nevcal.com> <53105743.5080407@g.nevcal.com> Message-ID: On Fri, Feb 28, 2014 at 11:51 PM, Nick Coghlan wrote: >>> > Are there any other expressions that allow parens around a part of the >>> > expression, without the stuff inside them becoming a completely >>> > separate sub-expression? > > Also generator expressions and most uses of yield or yield from as > embedded expressions. Parentheses are our general "this next bit may > not be following the normal syntax rules" utility, in addition to > being used to override the normal precedence rules (square brackets > and curly braces similarly denote regions where the parsing rules may > differ from the top level ones). In those cases, the "stuff inside the parens" is the entire syntactic structure of that sub-element, right? I was looking for something where there's syntax inside and syntax outside the parens, where what's in the parens isn't just an expression or tuple, so I could try laying out an except expression to imitate that style. The function call is excellent; removing one space from the layout did make the expression look better, but as explained above, I still prefer the current form. Would like to make the parens optional; maybe make them mandatory if there are two except-expressions abutting, or something, but I'd like to simplify the syntax for the common case. Otherwise, current syntax is still my personal preferred. Of course, we still have to convince Guido that this is a good idea at all, syntax or no :) ChrisA From rosuav at gmail.com Fri Feb 28 14:49:41 2014 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 1 Mar 2014 00:49:41 +1100 Subject: [Python-Dev] PEP 463: Exception-catching expressions In-Reply-To: References: <20140221145205.0b8b670d@fsol> <6A7884AB-44B2-49BF-AF07-BEA598871864@mac.com> <530F93A1.4050106@g.nevcal.com> <530FF075.8020303@g.nevcal.com> <53103D11.9030204@g.nevcal.com> <53105743.5080407@g.nevcal.com> Message-ID: On Sat, Mar 1, 2014 at 12:24 AM, Nick Coghlan wrote: > Yeah, that works. You may also want to add a "common objections" > section to explicitly cover the "but colons introduce suites" > objection. That would provide a space to explicitly list all of the > current "not introducing a suite" use cases for the colon in Python's > syntax, since increasing that tally from four to five is less radical > than introducing the first non-suite related use case for the symbol. https://github.com/Rosuav/ExceptExpr/commit/b770f9 Any other objections that should be listed? ChrisA From kristjan at ccpgames.com Fri Feb 28 14:58:29 2014 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Fri, 28 Feb 2014 13:58:29 +0000 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: <53104F2B.5080200@hastings.org> Message-ID: +1 Also, for the equivalence to hold there is no separate Py_XSETREF, the X behaviour is implied, which I favour. Enough of this X-proliferation already! But also see the discussion on inlines. It would be great to make this an inline rather than a macro. K From: Python-Dev [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Nick Coghlan Sent: 28. febr?ar 2014 12:27 To: Larry Hastings Cc: python-dev at python.org Subject: Re: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc For additional context, the idea itself is necessary for the same reason Py_CLEAR was added: to help ensure that an object's state is never pointing at another object that is in the process of being deleted. The difference is that Py_CLEAR only allows setting the pointer to NULL, while the point of the new macro is to set it to an arbitrary existing point. There is no implicit incref as that isn't needed for correctness (you can do the incref before the pointer replacement, and often the reference count will already be correct without an explicit incref anyway). With the new macro in place, the existing Py_CLEAR(x) macro would be equivalent to Py_SETREF(x, NULL). Originally I was also concerned about the "how will people know there's no implicit incref?", but I've since become satisfied with the fact that the precedent set by the reference stealing SET_ITEM macros is strong enough to justify the shorter name. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Feb 28 16:19:48 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 28 Feb 2014 10:19:48 -0500 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: <53104F2B.5080200@hastings.org> Message-ID: <20140228101948.29966a30@anarchist.wooz.org> On Feb 28, 2014, at 10:27 PM, Nick Coghlan wrote: >With the new macro in place, the existing Py_CLEAR(x) macro would be >equivalent to Py_SETREF(x, NULL). > >Originally I was also concerned about the "how will people know there's no >implicit incref?", but I've since become satisfied with the fact that the >precedent set by the reference stealing SET_ITEM macros is strong enough to >justify the shorter name. I haven't had time to follow this discussion at all, but for a macro to be called Py_SETREF and *not* increment the reference counter seems at best confusing. Despite my hesitation to paint a bike shed I haven't had time to inspect, something more akin to Py_SET_POINTER seems more appropriate (i.e. don't put "REF" in the name if it isn't playing refcounting games). -Barry From status at bugs.python.org Fri Feb 28 18:07:50 2014 From: status at bugs.python.org (Python tracker) Date: Fri, 28 Feb 2014 18:07:50 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20140228170750.0F8B256A7E@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2014-02-21 - 2014-02-28) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4582 (+24) closed 27999 (+62) total 32581 (+86) Open issues with patches: 2073 Issues opened (53) ================== #20693: Sidebar scrolls down 2x as fast as page content http://bugs.python.org/issue20693 reopened by georg.brandl #20726: inspect: Make Signature instances picklable http://bugs.python.org/issue20726 opened by yselivanov #20727: Improved roundrobin itertools recipe http://bugs.python.org/issue20727 opened by david.lindquist #20728: Remove unused import from base64 http://bugs.python.org/issue20728 opened by Claudiu.Popa #20729: mailbox.Mailbox does odd hasattr() check http://bugs.python.org/issue20729 opened by Rosuav #20736: test_socket: testSendmsgDontWait needlessly skipped on Linux http://bugs.python.org/issue20736 opened by baikie #20737: 3.3 _thread lock.acquire() timeout and threading.Event().wait( http://bugs.python.org/issue20737 opened by raruler #20739: PEP 463 (except expression) implementation http://bugs.python.org/issue20739 opened by twouters #20741: Documentation archives should be available also in tar.xz form http://bugs.python.org/issue20741 opened by Arfrever #20742: 2to3 zip fixer doesn't fix for loops. http://bugs.python.org/issue20742 opened by drj #20744: shutil should not use distutils http://bugs.python.org/issue20744 opened by doko #20745: test_statistics fails in refleak mode http://bugs.python.org/issue20745 opened by pitrou #20746: test_pdb fails in refleak mode http://bugs.python.org/issue20746 opened by pitrou #20747: Charset.header_encode in email.charset doesn't take a maxlinel http://bugs.python.org/issue20747 opened by rednaw #20748: 3.4rc2 MSI uninstallation leaves behind ensurepip _uninstall . http://bugs.python.org/issue20748 opened by loewis #20749: shutil.unpack_archive(): security concerns not documented http://bugs.python.org/issue20749 opened by jwilk #20750: Roundtrip-test tokenize.untokenize(iterable_of_5_tuples) http://bugs.python.org/issue20750 opened by terry.reedy #20751: Misleading examples in the descriptor protocol documentation http://bugs.python.org/issue20751 opened by zuo #20752: Difflib should provide the option of overriding the SequenceMa http://bugs.python.org/issue20752 opened by offby1 #20753: disable test_robotparser test that uses an invalid URL http://bugs.python.org/issue20753 opened by larry #20754: distutils should use SafeConfigParser http://bugs.python.org/issue20754 opened by alunduil #20756: Segmentation fault with unoconv http://bugs.python.org/issue20756 opened by Sworddragon #20758: mimetypes initialization order http://bugs.python.org/issue20758 opened by chid #20760: test_compileall test getting failed on 3.4 RC http://bugs.python.org/issue20760 opened by vipulb #20761: os.path.join doesn't strip LF or CR http://bugs.python.org/issue20761 opened by ExtraVeral #20762: SSLSocket.read() not documented http://bugs.python.org/issue20762 opened by ebianchi #20766: reference leaks in pdb http://bugs.python.org/issue20766 opened by xdegaye #20767: Some python extensions can't be compiled with clang 3.4 http://bugs.python.org/issue20767 opened by Antoine.Brodin.FreeBSD #20768: pyconfig.h #defines macros in global namespace http://bugs.python.org/issue20768 opened by fsateler #20769: Reload() description is unclear http://bugs.python.org/issue20769 opened by roysmith #20770: Inform caller of smtplib STARTTLS failures http://bugs.python.org/issue20770 opened by aclover #20773: Improve docs for DynamicClassAttribute http://bugs.python.org/issue20773 opened by ethan.furman #20774: collections.deque should ship with a stdlib json serializer http://bugs.python.org/issue20774 opened by acdha #20776: Add tests for importlib.machinery.PathFinder http://bugs.python.org/issue20776 opened by brett.cannon #20779: Add pathlib.chown method http://bugs.python.org/issue20779 opened by vajrasky #20780: Shadowed (duplicate name but different body) test in test_stat http://bugs.python.org/issue20780 opened by vajrasky #20782: base64 module docs do not use the terms 'bytes' and 'string' c http://bugs.python.org/issue20782 opened by r.david.murray #20784: 'collections.abc' is no longer defined when collections is imp http://bugs.python.org/issue20784 opened by r.david.murray #20785: Missing symbols in Python27.lib (Windows 64bit) http://bugs.python.org/issue20785 opened by Victor.Lazzarini #20786: inspect.getargspec() returns wrong answer with property.__dele http://bugs.python.org/issue20786 opened by zzzeek #20787: typo in asyncio docs for subprocess_exec() http://bugs.python.org/issue20787 opened by akira #20788: distutils.msvccompiler - flags are hidden inside initialize() http://bugs.python.org/issue20788 opened by Matt.Goodman #20789: [3.4] cherrypick 5dec1604322c http://bugs.python.org/issue20789 opened by brett.cannon #20792: IDLE: Extend tests for PathBrowser http://bugs.python.org/issue20792 opened by sahutd #20797: zipfile.extractall should accept bytes path as parameter http://bugs.python.org/issue20797 opened by Laurent.Mazuel #20799: Turn on all tests in test.test_importlib.source.test_file_load http://bugs.python.org/issue20799 opened by brett.cannon #20800: Cannot run gui tests twice. http://bugs.python.org/issue20800 opened by terry.reedy #20803: struct.pack_into writes 0x00 for pad bytes http://bugs.python.org/issue20803 opened by bsder #20804: Sentinels identity lost when pickled (unittest.mock) http://bugs.python.org/issue20804 opened by Vlastimil.Z??ma #20806: os.times document points to wrong section of non-Linux manual http://bugs.python.org/issue20806 opened by geoffreyspear #20807: 3.4 cherry pick: 82ec02db7fe6 & ec42ab5e0cb3 Windows installer http://bugs.python.org/issue20807 opened by ncoghlan #20808: 3.4 cherry pick: 6a1711c96fa6 (Popen.__del__ traceback) http://bugs.python.org/issue20808 opened by barry #20809: isabspath fails if path is None http://bugs.python.org/issue20809 opened by jordannh Most recent 15 issues with no replies (15) ========================================== #20809: isabspath fails if path is None http://bugs.python.org/issue20809 #20799: Turn on all tests in test.test_importlib.source.test_file_load http://bugs.python.org/issue20799 #20797: zipfile.extractall should accept bytes path as parameter http://bugs.python.org/issue20797 #20789: [3.4] cherrypick 5dec1604322c http://bugs.python.org/issue20789 #20788: distutils.msvccompiler - flags are hidden inside initialize() http://bugs.python.org/issue20788 #20787: typo in asyncio docs for subprocess_exec() http://bugs.python.org/issue20787 #20785: Missing symbols in Python27.lib (Windows 64bit) http://bugs.python.org/issue20785 #20782: base64 module docs do not use the terms 'bytes' and 'string' c http://bugs.python.org/issue20782 #20780: Shadowed (duplicate name but different body) test in test_stat http://bugs.python.org/issue20780 #20776: Add tests for importlib.machinery.PathFinder http://bugs.python.org/issue20776 #20754: distutils should use SafeConfigParser http://bugs.python.org/issue20754 #20751: Misleading examples in the descriptor protocol documentation http://bugs.python.org/issue20751 #20749: shutil.unpack_archive(): security concerns not documented http://bugs.python.org/issue20749 #20748: 3.4rc2 MSI uninstallation leaves behind ensurepip _uninstall . http://bugs.python.org/issue20748 #20746: test_pdb fails in refleak mode http://bugs.python.org/issue20746 Most recent 15 issues waiting for review (15) ============================================= #20809: isabspath fails if path is None http://bugs.python.org/issue20809 #20792: IDLE: Extend tests for PathBrowser http://bugs.python.org/issue20792 #20787: typo in asyncio docs for subprocess_exec() http://bugs.python.org/issue20787 #20780: Shadowed (duplicate name but different body) test in test_stat http://bugs.python.org/issue20780 #20779: Add pathlib.chown method http://bugs.python.org/issue20779 #20767: Some python extensions can't be compiled with clang 3.4 http://bugs.python.org/issue20767 #20766: reference leaks in pdb http://bugs.python.org/issue20766 #20758: mimetypes initialization order http://bugs.python.org/issue20758 #20753: disable test_robotparser test that uses an invalid URL http://bugs.python.org/issue20753 #20752: Difflib should provide the option of overriding the SequenceMa http://bugs.python.org/issue20752 #20739: PEP 463 (except expression) implementation http://bugs.python.org/issue20739 #20736: test_socket: testSendmsgDontWait needlessly skipped on Linux http://bugs.python.org/issue20736 #20729: mailbox.Mailbox does odd hasattr() check http://bugs.python.org/issue20729 #20728: Remove unused import from base64 http://bugs.python.org/issue20728 #20727: Improved roundrobin itertools recipe http://bugs.python.org/issue20727 Top 10 most discussed issues (10) ================================= #20727: Improved roundrobin itertools recipe http://bugs.python.org/issue20727 12 msgs #20440: Use Py_REPLACE/Py_XREPLACE macros http://bugs.python.org/issue20440 10 msgs #16484: pydoc generates invalid docs.python.org link for xml.etree.Ele http://bugs.python.org/issue16484 8 msgs #19021: AttributeError in Popen.__del__ http://bugs.python.org/issue19021 8 msgs #20630: Add sorting helpers for collections containing None values http://bugs.python.org/issue20630 8 msgs #20747: Charset.header_encode in email.charset doesn't take a maxlinel http://bugs.python.org/issue20747 8 msgs #20784: 'collections.abc' is no longer defined when collections is imp http://bugs.python.org/issue20784 8 msgs #9291: mimetypes initialization fails on Windows because of non-Latin http://bugs.python.org/issue9291 7 msgs #20804: Sentinels identity lost when pickled (unittest.mock) http://bugs.python.org/issue20804 7 msgs #19619: Blacklist base64, hex, ... codecs from bytes.decode() and str. http://bugs.python.org/issue19619 6 msgs Issues closed (61) ================== #6815: UnicodeDecodeError in os.path.expandvars http://bugs.python.org/issue6815 closed by serhiy.storchaka #9974: tokenizer.untokenize not invariant with line continuations http://bugs.python.org/issue9974 closed by terry.reedy #14513: IDLE icon switched and switches on Windows taskbar http://bugs.python.org/issue14513 closed by terry.reedy #16074: Bad error message in os.rename, os.link, and os.symlink http://bugs.python.org/issue16074 closed by georg.brandl #17053: pydoc should use inspect.signature instead of inspect.getfulla http://bugs.python.org/issue17053 closed by yselivanov #19470: email.header.Header - should not allow two newlines in a row http://bugs.python.org/issue19470 closed by r.david.murray #20146: UserDict module docs link is obsolete http://bugs.python.org/issue20146 closed by ned.deily #20199: status of module_for_loader and utils._module_to_load http://bugs.python.org/issue20199 closed by brett.cannon #20227: Argument Clinic: rename arguments in generated C? http://bugs.python.org/issue20227 closed by zach.ware #20261: Cannot pickle some objects that have a __getattr__() http://bugs.python.org/issue20261 closed by larry #20429: 3.3.4rc1 install deleted Windows taskbar icons http://bugs.python.org/issue20429 closed by terry.reedy #20484: calling pydoc.Helper()('modules') in the test suite sometimes http://bugs.python.org/issue20484 closed by eric.snow #20501: fileinput module will read whole file into memory when using f http://bugs.python.org/issue20501 closed by serhiy.storchaka #20535: run_tests.py: Work around issues 20355 and 20361 http://bugs.python.org/issue20535 closed by serhiy.storchaka #20568: Pass --default-install to ensurepip in the Windows installers http://bugs.python.org/issue20568 closed by python-dev #20570: Bundle pip 1.5.3 in Python 3.4rc2 http://bugs.python.org/issue20570 closed by larry #20628: Improve doc for csv.DictReader 'fieldnames' parameter http://bugs.python.org/issue20628 closed by r.david.murray #20637: Support key-sharing dictionaries in subclasses http://bugs.python.org/issue20637 closed by pitrou #20641: Python installer needs elevated rights to install pip http://bugs.python.org/issue20641 closed by loewis #20650: asyncio.BaseEventLoop.run_in_executor docs have awkward wordin http://bugs.python.org/issue20650 closed by brett.cannon #20677: Minor typo in enum docs http://bugs.python.org/issue20677 closed by ezio.melotti #20679: 3.4 cherry-pick: 587fd4b91120 improve Enum subclass behavior http://bugs.python.org/issue20679 closed by larry #20690: IDLE Indents convert to spaces and then throws error http://bugs.python.org/issue20690 closed by terry.reedy #20714: Allow for ]]> in CDATA in minidom http://bugs.python.org/issue20714 closed by loewis #20715: 3.4 cherry-pick: 2000c27ebe80 inspect: Fix getfullargspec to s http://bugs.python.org/issue20715 closed by larry #20721: 3.4 cherry-pick: 005d0678f93c Update pip to 1.5.4 http://bugs.python.org/issue20721 closed by larry #20723: Make test (Python 3.3.4) http://bugs.python.org/issue20723 closed by ned.deily #20724: 3.4 cherry-pick: d6aa3fa646e2 inspect.signature: Check for fun http://bugs.python.org/issue20724 closed by larry #20725: Fail to build on Windows x64 with VS Express http://bugs.python.org/issue20725 closed by zach.ware #20730: Typo in idlelib.GrepDialog http://bugs.python.org/issue20730 closed by terry.reedy #20731: Python 3.3.4: SyntaxError with correct source code encoding # http://bugs.python.org/issue20731 closed by loewis #20732: Custom logging formatter doesnt work in 3.3.4 http://bugs.python.org/issue20732 closed by vinay.sajip #20733: Typo in itertools docs - "itertool-functions" http://bugs.python.org/issue20733 closed by rhettinger #20734: 3.4 cherry-pick: 13edfab6c3c0 disable 2 pydoc tests http://bugs.python.org/issue20734 closed by larry #20735: Documentation: remove stringprep deprecation mark in docs http://bugs.python.org/issue20735 closed by python-dev #20738: 3.4 cherry-pick: 7b80f57f904e fix 20641 http://bugs.python.org/issue20738 closed by larry #20740: Remove invalid number from squares in introduction section http://bugs.python.org/issue20740 closed by ezio.melotti #20743: test_tcl memory leak http://bugs.python.org/issue20743 closed by pitrou #20755: 3.3.4: doc build fails on Sphinx < 1.2 http://bugs.python.org/issue20755 closed by python-dev #20757: 3.4rc2 Traceback on Windows pip uninstall http://bugs.python.org/issue20757 closed by python-dev #20759: unittest.mock documentation typos http://bugs.python.org/issue20759 closed by python-dev #20763: old sys.path_hooks importer does not work with Python 3.4.0rc1 http://bugs.python.org/issue20763 closed by brett.cannon #20764: os.walk recurses down even with dirnames deleted http://bugs.python.org/issue20764 closed by ned.deily #20765: Pathlib docs fail to mention with_name, with_suffix http://bugs.python.org/issue20765 closed by pitrou #20771: spam http://bugs.python.org/issue20771 closed by zach.ware #20772: Spam http://bugs.python.org/issue20772 closed by berker.peksag #20775: Modifications to global variables ignored after instantiating http://bugs.python.org/issue20775 closed by Naftali.Harris #20777: PyArg_ParseTupleAndKeywords does not respect arguments format. http://bugs.python.org/issue20777 closed by Claymore #20778: ModuleFinder.load_module skips incorrect number of bytes in py http://bugs.python.org/issue20778 closed by brett.cannon #20781: BZ2File doesn't decompress some .bz2 files correctly http://bugs.python.org/issue20781 closed by serhiy.storchaka #20783: bytearray init fails when \x00 is present http://bugs.python.org/issue20783 closed by bsder #20790: Make sure exec_module() implementations are documented http://bugs.python.org/issue20790 closed by brett.cannon #20791: copy.copy(bytes) is slow http://bugs.python.org/issue20791 closed by pitrou #20793: locale.setlocale() http://bugs.python.org/issue20793 closed by ned.deily #20794: ImportError: Bad magic number in .pyc file http://bugs.python.org/issue20794 closed by ned.deily #20795: Spam http://bugs.python.org/issue20795 closed by berker.peksag #20796: Test failures when running with PYTHONDONTWRITEBYTECODE http://bugs.python.org/issue20796 closed by python-dev #20798: Reversed order in testtools messages http://bugs.python.org/issue20798 closed by r.david.murray #20801: WindowsError: [Error 2] The system cannot find the file specif http://bugs.python.org/issue20801 closed by ned.deily #20802: AttributeError: 'NoneType' object has no attribute 'returncode http://bugs.python.org/issue20802 closed by ned.deily #20805: Error in 3.3 Tutorial http://bugs.python.org/issue20805 closed by r.david.murray From jason.burgoon at transamerica.com Fri Feb 28 18:05:18 2014 From: jason.burgoon at transamerica.com (Burgoon, Jason) Date: Fri, 28 Feb 2014 17:05:18 +0000 Subject: [Python-Dev] Python 3.3.4150 Message-ID: <9DB8A1E75492864D8C3016CE2126AADD28BDD1C8@CREXHTMAIL02.us.aegon.com> Good day Python Dev Team - One of our users has reported the following: I have installed the given msi on 64 bit as per the install instructions document. One of the shortcuts 'Start Menu\Programs\Python 3.3\ Module Docs' is not getting launched. When I launch this shortcut, it is not opening any window. I have tried with admin user and non-admin user. Is this expected behavior? Please advise and thanks for your help. Jason [http://intranet.ds.global/technology/employees/Documents/Communication%20Resource%20Center/Small%20logo%20for%20signature.png] Jason Burgoon Application Management Services - Coordinator 4333 Edgewood Rd NE, Cedar Rapids, IA 52499 Phone: (319) 355-4534 Email: jason.burgoon at transamerica.com Click here to visit the AGT Website -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 5574 bytes Desc: image001.png URL: From ncoghlan at gmail.com Fri Feb 28 23:15:12 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 1 Mar 2014 08:15:12 +1000 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: <20140228101948.29966a30@anarchist.wooz.org> References: <53104F2B.5080200@hastings.org> <20140228101948.29966a30@anarchist.wooz.org> Message-ID: On 1 Mar 2014 01:22, "Barry Warsaw" wrote: > > On Feb 28, 2014, at 10:27 PM, Nick Coghlan wrote: > > >With the new macro in place, the existing Py_CLEAR(x) macro would be > >equivalent to Py_SETREF(x, NULL). > > > >Originally I was also concerned about the "how will people know there's no > >implicit incref?", but I've since become satisfied with the fact that the > >precedent set by the reference stealing SET_ITEM macros is strong enough to > >justify the shorter name. > > I haven't had time to follow this discussion at all, but for a macro to be > called Py_SETREF and *not* increment the reference counter seems at best > confusing. Despite my hesitation to paint a bike shed I haven't had time to > inspect, something more akin to Py_SET_POINTER seems more appropriate > (i.e. don't put "REF" in the name if it isn't playing refcounting games). It *is* playing refcounting games - it's decrefing the existing target while stealing a reference to the new target, just like the existing SET_ITEM macros and somewhat like Py_CLEAR (although in that case, it's more obvious that we will never incref NULL). The whole point of this macro is to take an *existing* reference and safely *overwrite* another existing reference, exactly as the SET_ITEM macros do. That actually gives me an idea that wasn't on Serhiy's original list: Py_SET_ATTR(target, value). After all, setting attributes safely from C is the main use case for this, and I think it strengthens the parallel with the SET_ITEM macros on the concrete types. Cheers, Nick. > > -Barry > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Feb 28 23:49:17 2014 From: barry at python.org (Barry Warsaw) Date: Fri, 28 Feb 2014 17:49:17 -0500 Subject: [Python-Dev] Poll: Py_REPLACE/Py_ASSIGN/etc In-Reply-To: References: <53104F2B.5080200@hastings.org> <20140228101948.29966a30@anarchist.wooz.org> Message-ID: <20140228174917.48ec22da@anarchist.wooz.org> On Mar 01, 2014, at 08:15 AM, Nick Coghlan wrote: >It *is* playing refcounting games - it's decrefing the existing target >while stealing a reference to the new target, just like the existing >SET_ITEM macros and somewhat like Py_CLEAR (although in that case, it's >more obvious that we will never incref NULL). Okay, but "setting the reference" isn't one of them, which is what I read when I see Py_SETREF. ;) >The whole point of this macro is to take an *existing* reference and safely >*overwrite* another existing reference, exactly as the SET_ITEM macros do. > >That actually gives me an idea that wasn't on Serhiy's original list: >Py_SET_ATTR(target, value). That does seem better. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: