From steve at pearwood.info  Sun Nov  1 01:06:30 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 1 Nov 2015 17:06:30 +1100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <n0ugob$mvt$1@ger.gmane.org>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org>
Message-ID: <20151101060629.GE10946@ando.pearwood.info>

CC'ing Python-Ideas. Follow-ups to Python-Ideas please.

On Thu, Oct 29, 2015 at 09:22:15PM -0400, Terry Reedy wrote:

> Leaving IDLE aside, the reason '' is added to sys.path is so that people 
> can import their own modules.  This is very useful.  Shadowing is the 
> result of putting it at the front.  I have long thought this a dubious 
> choice.  If '' were instead appended, people could still import modules 
> that did not duplicate stdlib names.  Anyone who wanted shadowing could 
> move '' to the front.  But then shadowing would be intentional, not an 
> accident.

Terry is right. Shadowing should be possible, and it should require a 
deliberate decision on the part of the programmer.

Consider the shell, say, bash or similar. My understanding is that the 
shell PATH deliberately excludes the current directory because of the 
possibility of malicious software shadowing usual commands in /bin etc. 
If you want to run an executable in the current directory, you have to 
explicitly provide the path to it: ./myscript rather than just myscript.

Now Python isn't exactly the shell, and so I'm not proposing that Python 
does the same thing. But surely we can agree on the following?

- Shadowing explicitly installed packages, including the stdlib, is 
  *occasionally* useful.

- But when shadowing occurs, it is *nearly always* accidental.

- Such accidental shadowing often causes problems.

- And further more, debugging shadowing problems is sometimes tricky 
  even for experienced coders, and almost impossible for beginners.

  (It's not until you've been burned once or thrice by shadowing that 
  you recognise the symptoms, at which point it is then usually easy to 
  debug.)

- Hence, we should put the onus on those who want to shadow installed 
  packages) to do so *explicitly*, or at least make it easier to avoid 
  accidental shadowing.


I propose the following two changes:


(1) Beginning with Python 3.6, the default is that the current directory 
is put at the end of sys.path rather than the beginning. Instead of:

    >>> print(sys.path)
    ['', '/this', '/that', '/another']

we will have this instead:

    >>> print(sys.path)
    ['/this', '/that', '/another', '']

Those who don't shadow installed packages won't notice any 
difference.

Scripts which deliberately or unintentionally shadow installed packages 
will break from this change. I don't have a problem with this. You can't 
fix harmful behaviour without breaking code that depends on that harmful 
behaviour. Additionally, I expect that those who rely on the current 
behaviour will be in a small minority, much fewer than those who will be 
bitten by accidental shadowing into the indefinite future. And if you 
want the old behaviour back, it is easy to do so, by changing the path 
before doing your imports:

    import sys
    if sys.path[-1] == "":  sys.path = [""] + sys.path[:-1]

or equivalent.

I do not belive that it is onerous for those who want shadowing to have 
to take steps to do so explicitly. That can be added to your scripts on 
a case-by-case basis, or your PYTHONSTARTUP file, by modifying your 
site.py, or (I think) by putting the code into the sitecustomize or 
usercustomize modules.

(2) IDLE doesn't need to wait for Python 3.6 to make this change. I 
believe that IDLE is permitted to make backwards incompatible changes in 
minor releases, so there is no reason why it can't change the path 
effective immediately.

That's a simpler fix than scanning the entire path, raising warnings 
(which beginners won't understand and will either ignore or panic over) 
or other complex solutions. It may not prevent *every* shadowing 
incident, but it will improve the situation immeasurably.


Thoughts?



-- 
Steve

From storchaka at gmail.com  Sun Nov  1 01:41:03 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 1 Nov 2015 08:41:03 +0200
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <20151101060629.GE10946@ando.pearwood.info>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
Message-ID: <n14c60$gcp$1@ger.gmane.org>

On 01.11.15 08:06, Steven D'Aprano wrote:
> (1) Beginning with Python 3.6, the default is that the current directory
> is put at the end of sys.path rather than the beginning. Instead of:
>
>      >>> print(sys.path)
>      ['', '/this', '/that', '/another']
>
> we will have this instead:
>
>      >>> print(sys.path)
>      ['/this', '/that', '/another', '']
>
> Those who don't shadow installed packages won't notice any
> difference.
>
> Scripts which deliberately or unintentionally shadow installed packages
> will break from this change. I don't have a problem with this. You can't
> fix harmful behaviour without breaking code that depends on that harmful
> behaviour. Additionally, I expect that those who rely on the current
> behaviour will be in a small minority, much fewer than those who will be
> bitten by accidental shadowing into the indefinite future.

Unfortunately this is not such small minority

https://code.openhub.net/search?s=%22sys.path.pop(0)%22&p=0

But we can workaround this problem by adding harmless path at the strt 
of sys.path.

     sys.path.insert(0, '/non-existing-stub-path')

or

     sys.path.insert(0, sys.path[0])



From tjreedy at udel.edu  Sun Nov  1 02:26:35 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 1 Nov 2015 02:26:35 -0500
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <20151101060629.GE10946@ando.pearwood.info>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
Message-ID: <n14eri$io7$1@ger.gmane.org>

On 11/1/2015 1:06 AM, Steven D'Aprano wrote:
> CC'ing Python-Ideas. Follow-ups to Python-Ideas please.
>
> On Thu, Oct 29, 2015 at 09:22:15PM -0400, Terry Reedy wrote:
>
>> Leaving IDLE aside, the reason '' is added to sys.path is so that people
>> can import their own modules.  This is very useful.  Shadowing is the
>> result of putting it at the front.  I have long thought this a dubious
>> choice.  If '' were instead appended, people could still import modules
>> that did not duplicate stdlib names.  Anyone who wanted shadowing could
>> move '' to the front.  But then shadowing would be intentional, not an
>> accident.
>
> Terry is right. Shadowing should be possible, and it should require a
> deliberate decision on the part of the programmer.
>
> Consider the shell, say, bash or similar. My understanding is that the
> shell PATH deliberately excludes the current directory because of the
> possibility of malicious software shadowing usual commands in /bin etc.
> If you want to run an executable in the current directory, you have to
> explicitly provide the path to it: ./myscript rather than just myscript.
>
> Now Python isn't exactly the shell, and so I'm not proposing that Python
> does the same thing. But surely we can agree on the following?
>
> - Shadowing explicitly installed packages, including the stdlib, is
>    *occasionally* useful.
>
> - But when shadowing occurs, it is *nearly always* accidental.
>
> - Such accidental shadowing often causes problems.
>
> - And further more, debugging shadowing problems is sometimes tricky
>    even for experienced coders, and almost impossible for beginners.
>
>    (It's not until you've been burned once or thrice by shadowing that
>    you recognise the symptoms, at which point it is then usually easy to
>    debug.)
>
> - Hence, we should put the onus on those who want to shadow installed
>    packages) to do so *explicitly*, or at least make it easier to avoid
>    accidental shadowing.
>
>
> I propose the following two changes:
>
>
> (1) Beginning with Python 3.6, the default is that the current directory
> is put at the end of sys.path rather than the beginning. Instead of:
>
>      >>> print(sys.path)
>      ['', '/this', '/that', '/another']
>
> we will have this instead:
>
>      >>> print(sys.path)
>      ['/this', '/that', '/another', '']
>
> Those who don't shadow installed packages won't notice any
> difference.

Serhiy pointed out that code that unconditionally executes 
sys.path.pop(0) to avoid shadowing of its stdlib imports might break if 
it imports from what is now sys.path[1].  This currently (2.7, 3.4, 3.5) 
is <somewhere>pythonxy.zip.  (In 2.7 and 3.4 on Windows, somewhere = 
/windows/system32, In 3.5, the install directory). I say 'might' because 
I don't know if the presence of pythonxy.zip always means that 
<installdir>/lib is absent. If it is gone, the imports will fail.

Since use of zip is rare, the transition might be doable.  Adding "if 
sys.path[0] == '':" is backwark compatible.  If the import system had 
available the set of /lib modules, as I proposed on python-list, then 
the error handler could diagnose the situation of a lib module import 
failing because of an existing pythonxy.zip not being on sys.path. I 
believe the set would also help for generating deprecation messages.

> Scripts which deliberately or unintentionally shadow installed packages
> will break from this change. I don't have a problem with this. You can't
> fix harmful behaviour without breaking code that depends on that harmful
> behaviour. Additionally, I expect that those who rely on the current
> behaviour will be in a small minority, much fewer than those who will be
> bitten by accidental shadowing into the indefinite future. And if you
> want the old behaviour back, it is easy to do so, by changing the path
> before doing your imports:
>
>      import sys
>      if sys.path[-1] == "":  sys.path = [""] + sys.path[:-1]
>
> or equivalent.
>
> I do not belive that it is onerous for those who want shadowing to have
> to take steps to do so explicitly.

The question is whether it would be too onerous for those explicitly 
avoiding shadowing now to have to change their explicit code.

 > That can be added to your scripts on
> a case-by-case basis, or your PYTHONSTARTUP file, by modifying your
> site.py, or (I think) by putting the code into the sitecustomize or
> usercustomize modules.
>
> (2) IDLE doesn't need to wait for Python 3.6 to make this change. I
> believe that IDLE is permitted to make backwards incompatible changes in
> minor releases, so there is no reason why it can't change the path
> effective immediately.

IDLE can change how it operates with its own code.  I plan to add a 
conditional pop before IDLE does its own imports.  However, IDLE is not 
free to change how user code is executed, except to better imitate 
CPython than it does now. Hence '' will have to be put back for user 
code execution.

-- 
Terry Jan Reedy


From rosuav at gmail.com  Sun Nov  1 02:36:08 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 1 Nov 2015 18:36:08 +1100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <20151101060629.GE10946@ando.pearwood.info>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org>
 <20151101060629.GE10946@ando.pearwood.info>
Message-ID: <CAPTjJmpeGmu6dG4BygHXGnBp5rzmGOAwUkU4L3ZTABBRtd7wbA@mail.gmail.com>

On Sun, Nov 1, 2015 at 5:06 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> (1) Beginning with Python 3.6, the default is that the current directory
> is put at the end of sys.path rather than the beginning. Instead of:
>
>     >>> print(sys.path)
>     ['', '/this', '/that', '/another']
>
> we will have this instead:
>
>     >>> print(sys.path)
>     ['/this', '/that', '/another', '']
>
> Those who don't shadow installed packages won't notice any
> difference.
>
> Scripts which deliberately or unintentionally shadow installed packages
> will break from this change. I don't have a problem with this. You can't
> fix harmful behaviour without breaking code that depends on that harmful
> behaviour. Additionally, I expect that those who rely on the current
> behaviour will be in a small minority, much fewer than those who will be
> bitten by accidental shadowing into the indefinite future.

+1. As Serhiy says, though, this will additionally break scripts that
_protect against_ shadowing. We could have an easily-recognized shim
in slot zero, as a compatibility measure; not duplicating the next
one, but a clearly invalid entry. I suggest None, which appears to
work:

rosuav at sikorsky:~$ python3
Python 3.6.0a0 (default:9ca59b3cc18b, Oct 16 2015, 15:25:11)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path[0]=None; sys.path.append("")
>>> open("string.py","w").write("print('I am a string!')")
23
>>> import string
>>> string
<module 'string' from '/usr/local/lib/python3.6/string.py'>
>>> import doesntexist
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'doesntexist'
>>> open("uniquename.py","w").write("print('I have a unique name.')")
30
>>> import uniquename
I have a unique name.

The None entry is happily ignored. Shim removal with "if not
sys.path[0]: sys.path.pop(0)" will still remove it.

But I had a quick look at the link Serhiy posted, and two of the three
entries that I dug into were actually not removing the blank entry
(the other removed it and replaced it with a hard path); they had
previously inserted something at the beginning of sys.path, and were
removing _that_ entry. They won't need to be changed. I did a similar
search on GitHub and, again, most of the results weren't actually
removing the '' entry from the beginning. So the None could be put
there as a compatibility measure for 3.6, and then dropped in 3.7 - or
even just left out altogether, so a small number of programs break.

Would have been nice to do this at Python 3.0, as it'd have been on
par with absolute imports and other edge cases. Oh well.

ChrisA

From lac at openend.se  Sun Nov  1 02:53:17 2015
From: lac at openend.se (Laura Creighton)
Date: Sun, 01 Nov 2015 08:53:17 +0100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <20151101060629.GE10946@ando.pearwood.info>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
Message-ID: <201511010753.tA17rHs3025816@fido.openend.se>

In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes:

>I propose the following two changes:
>
>
>(1) Beginning with Python 3.6, the default is that the current directory 
>is put at the end of sys.path rather than the beginning. Instead of:
>
>    >>> print(sys.path)
>    ['', '/this', '/that', '/another']
>
>we will have this instead:
>
>    >>> print(sys.path)
>    ['/this', '/that', '/another', '']
>
>Those who don't shadow installed packages won't notice any 
>difference.
>
>Scripts which deliberately or unintentionally shadow installed packages 
>will break from this change. I don't have a problem with this. You can't 
>fix harmful behaviour without breaking code that depends on that harmful 
>behaviour. 

This is a bad idea, if you mean 'shadows anything in site-packages'.
I write a perfectly good working program, which then silently breaks
because somebody happens to install a site package with a name
conflict with my code.  Can you imagine being in the middle of
writing and debugging such a thing and have everything start
failing because suddenly your program isn't the one being found?
How long is it going to take you to stop looking at your own code,
and your own setup for the problem and begin looking at what
packages got installed by anybody else sharing this machine, and
what they are named?

It is not necessarily going to make the teachers' lives any better.
They will trade the confusion of 'things are acting strangely
around here' for 'I just wrote a program and the stupid langauge
cannot find it'.  

People whose code inadvertantly shadow something are better off with
a warning about the potential problem.

Laura

From rosuav at gmail.com  Sun Nov  1 02:58:36 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 1 Nov 2015 18:58:36 +1100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <201511010753.tA17rHs3025816@fido.openend.se>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org>
 <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
Message-ID: <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>

On Sun, Nov 1, 2015 at 6:53 PM, Laura Creighton <lac at openend.se> wrote:
> In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes:
>
>>I propose the following two changes:
>>
>>
>>(1) Beginning with Python 3.6, the default is that the current directory
>>is put at the end of sys.path rather than the beginning. Instead of:
>>
>>    >>> print(sys.path)
>>    ['', '/this', '/that', '/another']
>>
>>we will have this instead:
>>
>>    >>> print(sys.path)
>>    ['/this', '/that', '/another', '']
>>
>>Those who don't shadow installed packages won't notice any
>>difference.
>>
>>Scripts which deliberately or unintentionally shadow installed packages
>>will break from this change. I don't have a problem with this. You can't
>>fix harmful behaviour without breaking code that depends on that harmful
>>behaviour.
>
> This is a bad idea, if you mean 'shadows anything in site-packages'.
> I write a perfectly good working program, which then silently breaks
> because somebody happens to install a site package with a name
> conflict with my code.  Can you imagine being in the middle of
> writing and debugging such a thing and have everything start
> failing because suddenly your program isn't the one being found?
> How long is it going to take you to stop looking at your own code,
> and your own setup for the problem and begin looking at what
> packages got installed by anybody else sharing this machine, and
> what they are named?

I'm fairly sure Steven's talking about the standard library, not
arbitrary packages people might happen to have installed. Whether he
thought about it or not, the solution to both your problems seems to
be this:

sys.path = ['/usr/local/lib/python36.zip', '/usr/local/lib/python3.6',
'/usr/local/lib/python3.6/plat-linux',
'/usr/local/lib/python3.6/lib-dynload', '',
'/home/rosuav/.local/lib/python3.6/site-packages',
'/usr/local/lib/python3.6/site-packages']

Equivalently, for the system Python:

sys.path = ['/usr/lib/python3.4',
'/usr/lib/python3.4/plat-x86_64-linux-gnu',
'/usr/lib/python3.4/lib-dynload', '',
'/usr/local/lib/python3.4/dist-packages',
'/usr/lib/python3/dist-packages']


So the current directory is after everything that's logically the
standard library, but before things that would be installed separately
(site-packages, dist-packages, etc). It'd no longer be possible to
shadow wave.py, but you could shadow sqlalchemy.py.

ChrisA

From lac at openend.se  Sun Nov  1 03:08:36 2015
From: lac at openend.se (Laura Creighton)
Date: Sun, 01 Nov 2015 09:08:36 +0100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
Message-ID: <201511010808.tA188a77027105@fido.openend.se>

In a message of Sun, 01 Nov 2015 18:58:36 +1100, Chris Angelico writes:
>sys.path = ['/usr/local/lib/python36.zip', '/usr/local/lib/python3.6',
>'/usr/local/lib/python3.6/plat-linux',
>'/usr/local/lib/python3.6/lib-dynload', '',
>'/home/rosuav/.local/lib/python3.6/site-packages',
>'/usr/local/lib/python3.6/site-packages']
>
>Equivalently, for the system Python:
>
>sys.path = ['/usr/lib/python3.4',
>'/usr/lib/python3.4/plat-x86_64-linux-gnu',
>'/usr/lib/python3.4/lib-dynload', '',
>'/usr/local/lib/python3.4/dist-packages',
>'/usr/lib/python3/dist-packages']
>
>
>So the current directory is after everything that's logically the
>standard library, but before things that would be installed separately
>(site-packages, dist-packages, etc). It'd no longer be possible to
>shadow wave.py, but you could shadow sqlalchemy.py.
>
>ChrisA

But the kid who just wrote string.py or turtle.py will still have
the 'why isn't this working at all?' experience instead of something
that warns her what her problem is.

Laura

From rosuav at gmail.com  Sun Nov  1 04:02:23 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 1 Nov 2015 20:02:23 +1100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <201511010808.tA188a77027105@fido.openend.se>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org>
 <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <201511010808.tA188a77027105@fido.openend.se>
Message-ID: <CAPTjJmofHSfE4kKmN8dBeyY6YZEM6YVruAvmyQ8gKXkbEemyAA@mail.gmail.com>

On Sun, Nov 1, 2015 at 7:08 PM, Laura Creighton <lac at openend.se> wrote:
> In a message of Sun, 01 Nov 2015 18:58:36 +1100, Chris Angelico writes:
>>sys.path = ['/usr/local/lib/python36.zip', '/usr/local/lib/python3.6',
>>'/usr/local/lib/python3.6/plat-linux',
>>'/usr/local/lib/python3.6/lib-dynload', '',
>>'/home/rosuav/.local/lib/python3.6/site-packages',
>>'/usr/local/lib/python3.6/site-packages']
>>
>>Equivalently, for the system Python:
>>
>>sys.path = ['/usr/lib/python3.4',
>>'/usr/lib/python3.4/plat-x86_64-linux-gnu',
>>'/usr/lib/python3.4/lib-dynload', '',
>>'/usr/local/lib/python3.4/dist-packages',
>>'/usr/lib/python3/dist-packages']
>>
>>
>>So the current directory is after everything that's logically the
>>standard library, but before things that would be installed separately
>>(site-packages, dist-packages, etc). It'd no longer be possible to
>>shadow wave.py, but you could shadow sqlalchemy.py.
>>
>>ChrisA
>
> But the kid who just wrote string.py or turtle.py will still have
> the 'why isn't this working at all?' experience instead of something
> that warns her what her problem is.

Right. The warning when you save a file of that name is still a useful
thing; it's orthogonal to this, though. (FWIW I think the warning's a
good idea, but it's no panacea.)

ChrisA

From lac at openend.se  Sun Nov  1 04:49:49 2015
From: lac at openend.se (Laura Creighton)
Date: Sun, 01 Nov 2015 10:49:49 +0100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <CAPTjJmofHSfE4kKmN8dBeyY6YZEM6YVruAvmyQ8gKXkbEemyAA@mail.gmail.com>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <201511010808.tA188a77027105@fido.openend.se>
 <CAPTjJmofHSfE4kKmN8dBeyY6YZEM6YVruAvmyQ8gKXkbEemyAA@mail.gmail.com>
Message-ID: <201511010949.tA19nnwj001815@fido.openend.se>

In a message of Sun, 01 Nov 2015 20:02:23 +1100, Chris Angelico writes:
>> But the kid who just wrote string.py or turtle.py will still have
>> the 'why isn't this working at all?' experience instead of something
>> that warns her what her problem is.
>
>Right. The warning when you save a file of that name is still a useful
>thing; it's orthogonal to this, though. (FWIW I think the warning's a
>good idea, but it's no panacea.)
>
>ChrisA

Adding warnings to IDLE when you save a file is a fine idea, and will
help IDLE users avoid pain.  I want to help _everybody_ with a change
to Python, so that it issues a warning when you shadow something in
the standard library.

Something like:
Warning: local file /u/lac/junk/string.py shadows module named string in the 
Standard Library

Note, for somebody implementing this.  Teachers will find the message slightly
more useful if it prints the full path name.

Laura

From agtoever at hotmail.com  Sun Nov  1 06:08:44 2015
From: agtoever at hotmail.com (Albert ten Oever)
Date: Sun, 1 Nov 2015 11:08:44 +0000
Subject: [Python-ideas] Include partitioning in itertools
Message-ID: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>

Hi all,
I really hope this isn't proposed before - I couldn't find anything in the archives.
I want to propose to include a 'partition' (mathematically more correct: a set partition) function in itertools. To my knowledge, partitioning of a set (iterable) is quite a common thing to do and a logic extension of the combinatoric generators in itertools.
It is implemented as a Python recipe (http://code.activestate.com/recipes/576795-partitioning-a-sequence/). I found that implementing a partition generator of an iterable isn't very straightforward, which, in my opinion, strengthens the case for implementing it as a separate function.
Definitions of partitions: http://mathworld.wolfram.com/SetPartition.html or https://en.wikipedia.org/wiki/Partition_of_a_set
Humble regards,
Albert. 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151101/2650b5d4/attachment.html>

From steve at pearwood.info  Sun Nov  1 06:26:30 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 1 Nov 2015 22:26:30 +1100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <CAPTjJmofHSfE4kKmN8dBeyY6YZEM6YVruAvmyQ8gKXkbEemyAA@mail.gmail.com>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <201511010808.tA188a77027105@fido.openend.se>
 <CAPTjJmofHSfE4kKmN8dBeyY6YZEM6YVruAvmyQ8gKXkbEemyAA@mail.gmail.com>
Message-ID: <20151101112630.GF10946@ando.pearwood.info>

On Sun, Nov 01, 2015 at 08:02:23PM +1100, Chris Angelico wrote:
> On Sun, Nov 1, 2015 at 7:08 PM, Laura Creighton <lac at openend.se> wrote:
[...]
> > But the kid who just wrote string.py or turtle.py will still have
> > the 'why isn't this working at all?' experience instead of something
> > that warns her what her problem is.
> 
> Right. The warning when you save a file of that name is still a useful
> thing; it's orthogonal to this, though. (FWIW I think the warning's a
> good idea, but it's no panacea.)

I disagree.

As shown by the tutor and python-list mailing lists, beginners don't 
read error messages. Warning them that they're about to save a file 
using the name of a stdlib module will be no different. They'll either 
ignore the warning, and still be in the dark as to why their code is 
broken, or they'll panic that they did something wrong, and possibly 
lose their as yet unsaved work.

What produces the warning? Python, or the editor? If Python, you're 
annoying experienced programmers who intend to do what they do. If the 
editor, you do nothing about people using a different editor, or people 
who move and rename files in the shell.


-- 
Steve

From steve at pearwood.info  Sun Nov  1 06:42:50 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 1 Nov 2015 22:42:50 +1100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <201511010949.tA19nnwj001815@fido.openend.se>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <201511010808.tA188a77027105@fido.openend.se>
 <CAPTjJmofHSfE4kKmN8dBeyY6YZEM6YVruAvmyQ8gKXkbEemyAA@mail.gmail.com>
 <201511010949.tA19nnwj001815@fido.openend.se>
Message-ID: <20151101114250.GG10946@ando.pearwood.info>

On Sun, Nov 01, 2015 at 10:49:49AM +0100, Laura Creighton wrote:

> Adding warnings to IDLE when you save a file is a fine idea, and will
> help IDLE users avoid pain.

IDLE users must be different from most computer users then, because most 
computer users don't read warnings or errors.

http://ignorethecode.net/blog/2008/10/31/nobody-reads/

http://stackoverflow.com/questions/125269/how-would-you-handle-users-who-dont-read-dialog-boxes

> I want to help _everybody_ with a change
> to Python, so that it issues a warning when you shadow something in
> the standard library.

It's not enough to protect the stdlib. I've seen people accidently 
shadow numpy, or other third-party modules.



-- 
Steve

From steve at pearwood.info  Sun Nov  1 06:44:03 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 1 Nov 2015 22:44:03 +1100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <n14c60$gcp$1@ger.gmane.org>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <n14c60$gcp$1@ger.gmane.org>
Message-ID: <20151101114402.GH10946@ando.pearwood.info>

On Sun, Nov 01, 2015 at 08:41:03AM +0200, Serhiy Storchaka wrote:
> On 01.11.15 08:06, Steven D'Aprano wrote:
[...]
> >Scripts which deliberately or unintentionally shadow installed packages
> >will break from this change. I don't have a problem with this. You can't
> >fix harmful behaviour without breaking code that depends on that harmful
> >behaviour. Additionally, I expect that those who rely on the current
> >behaviour will be in a small minority, much fewer than those who will be
> >bitten by accidental shadowing into the indefinite future.
> 
> Unfortunately this is not such small minority
> 
> https://code.openhub.net/search?s=%22sys.path.pop(0)%22&p=0

The search results contain MANY duplicates. For example, in the first 
ten results, there are three duplicates of "common.py" from "kongove's 
autotest", and two duplicates of "common.py" from "Chromium OS".

The first hit does:

    sys.path.insert(0, path.dirname(__file__))
    import objects
    from objects import constants
    sys.path.pop(0)

which seems to be a very common pattern: insert something into the start 
of the path, then pop it out later. That's harmless, and won't be 
effected by shifting where "" in inserted. So I think that this search 
is not a good test for code that will be effected.

Besides, anyone who unconditionally pops the first item from sys.path is 
already on shakey ground. You should not assume that the first item will 
always be "", since it may have been changed before your code runs, e.g. 
by the PYTHONSTARTUP file, usercustomize, etc. Now, we shouldn't break 
people's code for no good reason, not even it it is already broken, but 
we have a good reason: having "" at the start of sys.path breaks code 
that inadvertently shadows other modules.

(And it may even be a security risk.)


-- 
Steve

From lac at openend.se  Sun Nov  1 07:20:43 2015
From: lac at openend.se (Laura Creighton)
Date: Sun, 01 Nov 2015 13:20:43 +0100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <20151101114250.GG10946@ando.pearwood.info>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <201511010808.tA188a77027105@fido.openend.se>
 <CAPTjJmofHSfE4kKmN8dBeyY6YZEM6YVruAvmyQ8gKXkbEemyAA@mail.gmail.com>
 <201511010949.tA19nnwj001815@fido.openend.se>
 <20151101114250.GG10946@ando.pearwood.info>
Message-ID: <201511011220.tA1CKhsS013079@fido.openend.se>

In a message of Sun, 01 Nov 2015 22:42:50 +1100, "Steven D'Aprano" writes:
>On Sun, Nov 01, 2015 at 10:49:49AM +0100, Laura Creighton wrote:
>
>> Adding warnings to IDLE when you save a file is a fine idea, and will
>> help IDLE users avoid pain.
>
>IDLE users must be different from most computer users then, because most 
>computer users don't read warnings or errors.
>
>http://ignorethecode.net/blog/2008/10/31/nobody-reads/
>
>http://stackoverflow.com/questions/125269/how-would-you-handle-users-who-dont-read-dialog-boxes

This is actually my old field of research.

The reason that most users don't read dialog boxes is that they are using
operating systems who chat them up using dialog boxes all the time.
They are used to having a dialog box mean 'the computer is working
as usual'.  They do not associate this with warnings or errors.  Thus
they are oblivious to them.

This is the argument against running your C code with lint every single
time.  People get used to seeing:

__Warning: long assignment may lose accuracy__

treat it as noise, and then when you actually want them to find such errors
are unable to find them.  It's quite astonishing.  They can read the
lint output to you out loud, and _still_ not be able to do the exercise
where they were to find rounding errors in their programs.

However, when you divide people into groups and make one group use
lint all the time, and tell the others not to use it, until they
get to this question -- poof.  The second group finds the rounding
errors.

So if, unknownst to me, lots and lots of people are shadowing the
stdlib, then if we issue warnings we may blunt their ability to
see warnings in general.  And that would be a downside.  But the
upside is that all of the people who did this inadvertantly would
get a warning that actually explained what is going on.  And the other
thing that is well understood is that people who are learning tend
to see warnings and whatnot -- simply because they haven't had the
time and the experience to file the warnings as 'just more noise'.

>> I want to help _everybody_ with a change
>> to Python, so that it issues a warning when you shadow something in
>> the standard library.
>
>It's not enough to protect the stdlib. I've seen people accidently 
>shadow numpy, or other third-party modules.

Yes, but I very much don't want any warnings when you shadow third
party modules.  If shadowing third party modules produce a warning,
then we will end up with 'all warnings get ignored' in very short
order.

We want warnings to be rare, rare enough that people don't get used
to seeing them.  If you can point me at a large community of regular
intentional shadowers of the standard library, that quite likely would
be enough for me to think that warning people is a bad idea -- having 
people get into the habit of ignoring warnings is a really dreadful 
outcome.

But the argument isn't 'we shouldn't issue warnings because people
don't read them' but rather 'we shouldn't make warnings commonplace
because that makes people incapable of reading them'.

>-- 
>Steve

(This ignores a different group of learners who have been studied,
 people who cannot learn and read at the same time.  Only some of
 these people have the sorts of problems that get called dyslexia.
 But video tutorials are making life a lot easier for these people
 nowadays.  And it is hard to see how adding a warning could harm
 these learners.)

Laura

From tritium-list at sdamon.com  Sun Nov  1 08:25:54 2015
From: tritium-list at sdamon.com (Alexander Walters)
Date: Sun, 01 Nov 2015 08:25:54 -0500
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
Message-ID: <563612E2.1060201@sdamon.com>

Honestly, shadowing modules is something that should be solved by 
renaming modules.  If you are worrying about shadowing ONLY the standard 
library - guess what?  those names don't change often, and are well 
known.  Don't use those names.  If you are talking about shadowing 
site-packages, or any package anywhere on sys.path, you are bound to 
break something somewhere in a hard to debug way.  It is sufficient to 
allow things to break and instruct people to jump in the repl, import a 
module and check __file__.

On 11/1/2015 02:58, Chris Angelico wrote:
> On Sun, Nov 1, 2015 at 6:53 PM, Laura Creighton <lac at openend.se> wrote:
>> In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes:
>>
>>> I propose the following two changes:
>>>
>>>
>>> (1) Beginning with Python 3.6, the default is that the current directory
>>> is put at the end of sys.path rather than the beginning. Instead of:
>>>
>>>     >>> print(sys.path)
>>>     ['', '/this', '/that', '/another']
>>>
>>> we will have this instead:
>>>
>>>     >>> print(sys.path)
>>>     ['/this', '/that', '/another', '']
>>>
>>> Those who don't shadow installed packages won't notice any
>>> difference.
>>>
>>> Scripts which deliberately or unintentionally shadow installed packages
>>> will break from this change. I don't have a problem with this. You can't
>>> fix harmful behaviour without breaking code that depends on that harmful
>>> behaviour.
>> This is a bad idea, if you mean 'shadows anything in site-packages'.
>> I write a perfectly good working program, which then silently breaks
>> because somebody happens to install a site package with a name
>> conflict with my code.  Can you imagine being in the middle of
>> writing and debugging such a thing and have everything start
>> failing because suddenly your program isn't the one being found?
>> How long is it going to take you to stop looking at your own code,
>> and your own setup for the problem and begin looking at what
>> packages got installed by anybody else sharing this machine, and
>> what they are named?
> I'm fairly sure Steven's talking about the standard library, not
> arbitrary packages people might happen to have installed. Whether he
> thought about it or not, the solution to both your problems seems to
> be this:
>
> sys.path = ['/usr/local/lib/python36.zip', '/usr/local/lib/python3.6',
> '/usr/local/lib/python3.6/plat-linux',
> '/usr/local/lib/python3.6/lib-dynload', '',
> '/home/rosuav/.local/lib/python3.6/site-packages',
> '/usr/local/lib/python3.6/site-packages']
>
> Equivalently, for the system Python:
>
> sys.path = ['/usr/lib/python3.4',
> '/usr/lib/python3.4/plat-x86_64-linux-gnu',
> '/usr/lib/python3.4/lib-dynload', '',
> '/usr/local/lib/python3.4/dist-packages',
> '/usr/lib/python3/dist-packages']
>
>
> So the current directory is after everything that's logically the
> standard library, but before things that would be installed separately
> (site-packages, dist-packages, etc). It'd no longer be possible to
> shadow wave.py, but you could shadow sqlalchemy.py.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


From rosuav at gmail.com  Sun Nov  1 08:44:27 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 2 Nov 2015 00:44:27 +1100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <563612E2.1060201@sdamon.com>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org>
 <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <563612E2.1060201@sdamon.com>
Message-ID: <CAPTjJmpkT64WURYxhOXugg=pUEJ9fw8i0de738DTuYxPsJoGWA@mail.gmail.com>

On Mon, Nov 2, 2015 at 12:25 AM, Alexander Walters
<tritium-list at sdamon.com> wrote:
>   If you are worrying about shadowing ONLY the standard library - guess
> what?  those names don't change often, and are well known.  Don't use those
> names.

Well known? Okay. No cheating now! Which of these names will shadow
something from the standard library, and which don't?

code.py
cgi.py
chunk.py
cmd.py
cprofile.py
gc.py
html.py
imp.py
mailbox.py
numbers.py
test.py
this.py
types.py
wave.py

Every one of these is a reasonably plausible name for a quick
throw-away script (maybe a test script as you learn how to use
something). How many of them will be problematic?

ChrisA

From python at lucidity.plus.com  Sun Nov  1 08:45:21 2015
From: python at lucidity.plus.com (Erik)
Date: Sun, 1 Nov 2015 13:45:21 +0000
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <201511010949.tA19nnwj001815@fido.openend.se>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <201511010808.tA188a77027105@fido.openend.se>
 <CAPTjJmofHSfE4kKmN8dBeyY6YZEM6YVruAvmyQ8gKXkbEemyAA@mail.gmail.com>
 <201511010949.tA19nnwj001815@fido.openend.se>
Message-ID: <56361771.9090702@lucidity.plus.com>

On 01/11/15 09:49, Laura Creighton wrote:
> Adding warnings to IDLE when you save a file is a fine idea, and will
> help IDLE users avoid pain.  I want to help _everybody_ with a change
> to Python, so that it issues a warning when you shadow something in
> the standard library.

I think that a solution to this could be something along the lines of 
having a command line switch (let's say '-w') or other way of enabling a 
"shadow module checking" mechanism.

When an exception is thrown that is caught by the default handler and 
output to stderr, an additional line of output could be appended:

"Note: This problem could be caused by a module that shadows a standard 
library or third party module. Run Python with the '-w' switch for more 
detail."

I'm not sure if this would be suitable for _all_ exceptions or only 
those from certain points in the exception hierarchy.

When the '-w' switch is enabled, the default exception handler would 
instead enumerate all active modules and, for any which has another 
module with the same name further down the search path than the one that 
is loaded, would output some sort of diagnostic message (i.e., a list of 
which loaded modules are shadowing _something_else_ that would have been 
loaded otherwise) - similar to your example below:

> Something like:
> Warning: local file /u/lac/junk/string.py shadows module named string in the
> Standard Library

This would prevent "power users" from always having to see a potentially 
large diagnostic after each uncaught exception, but generally remind 
everyone that such a thing is possible and "here's the easy way to check 
for it".

A bit like Valgrind's memcheck tool's "--leak-check=full" option - when 
run without it, if there are leaks then memcheck reminds the user that 
that's the way to start digging down into what might be causing them.

E.

From lac at openend.se  Sun Nov  1 08:53:23 2015
From: lac at openend.se (Laura Creighton)
Date: Sun, 01 Nov 2015 14:53:23 +0100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <563612E2.1060201@sdamon.com>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <563612E2.1060201@sdamon.com>
Message-ID: <201511011353.tA1DrNkI019815@fido.openend.se>

In a message of Sun, 01 Nov 2015 08:25:54 -0500, Alexander Walters writes:
>Honestly, shadowing modules is something that should be solved by 
>renaming modules.  If you are worrying about shadowing ONLY the standard 
>library - guess what?  those names don't change often, and are well 
>known.  Don't use those names.  

The problem is that the sort of people who make these errors don't
know the names.  Knowing that 'turtle.py' is a bad name for 'my
very first program that does turtle graphics' is a more advanced
skill than the turtle program writers, or even their teachers, can
be expected to have.  

I'm willing to have the kids and their teachers (and all the other
beginners) suffer if the downside is that an even only
moderately-sized set of legitimate stdlib shadowers end up with an
inability to notice warning messages, because they are now ubiquitous.
I just think that legitimate stdlib shadowers are rare.  And I 
suspect that many of them do this once, for a short period, not
every day of their lives.  Thus the 'ignore this noise' is much
less likely to develop with them.

Python warnings are rare.  Choosing to add a new one is a serious
step.  I do not want to open the floodgates and end up warning about all
possibly-silly practice -- I am one of the cognitive psychology boffins
who go around asking/nudging/reminding people to warn less often if you
want your most serious warnings to be heeded.

I just think the tradeoffs here are worth it.

Laura

From steve at pearwood.info  Sun Nov  1 10:06:33 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 2 Nov 2015 02:06:33 +1100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <CAPTjJmpkT64WURYxhOXugg=pUEJ9fw8i0de738DTuYxPsJoGWA@mail.gmail.com>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <563612E2.1060201@sdamon.com>
 <CAPTjJmpkT64WURYxhOXugg=pUEJ9fw8i0de738DTuYxPsJoGWA@mail.gmail.com>
Message-ID: <20151101150633.GI10946@ando.pearwood.info>

On Mon, Nov 02, 2015 at 12:44:27AM +1100, Chris Angelico wrote:
> On Mon, Nov 2, 2015 at 12:25 AM, Alexander Walters
> <tritium-list at sdamon.com> wrote:
> >   If you are worrying about shadowing ONLY the standard library - guess
> > what?  those names don't change often, and are well known.  Don't use those
> > names.
> 
> Well known? Okay. No cheating now! Which of these names will shadow
> something from the standard library, and which don't?
> 
> code.py
> cgi.py
> chunk.py
[...]

Does it matter? It really only matters if you shadow something that gets 
imported.

It's not exactly great practice to name your script "code.py", but if 
you do, and it doesn't directly or indirectly try to import the code 
library, it's harmless. That's why it is harmful to present users with 
an alert (which they won't read) warning them that they're shadowing a 
stdlib module.



-- 
Steve

From rosuav at gmail.com  Sun Nov  1 10:24:37 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 2 Nov 2015 02:24:37 +1100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <20151101150633.GI10946@ando.pearwood.info>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org>
 <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <563612E2.1060201@sdamon.com>
 <CAPTjJmpkT64WURYxhOXugg=pUEJ9fw8i0de738DTuYxPsJoGWA@mail.gmail.com>
 <20151101150633.GI10946@ando.pearwood.info>
Message-ID: <CAPTjJmp1Oyjx5Yzcq4N5_3hEQn5QcmW3s2wGg5sq+TRk5h95-w@mail.gmail.com>

On Mon, Nov 2, 2015 at 2:06 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Mon, Nov 02, 2015 at 12:44:27AM +1100, Chris Angelico wrote:
>> On Mon, Nov 2, 2015 at 12:25 AM, Alexander Walters
>> <tritium-list at sdamon.com> wrote:
>> >   If you are worrying about shadowing ONLY the standard library - guess
>> > what?  those names don't change often, and are well known.  Don't use those
>> > names.
>>
>> Well known? Okay. No cheating now! Which of these names will shadow
>> something from the standard library, and which don't?
>>
>> code.py
>> cgi.py
>> chunk.py
> [...]
>
> Does it matter? It really only matters if you shadow something that gets
> imported.
>
> It's not exactly great practice to name your script "code.py", but if
> you do, and it doesn't directly or indirectly try to import the code
> library, it's harmless. That's why it is harmful to present users with
> an alert (which they won't read) warning them that they're shadowing a
> stdlib module.

It matters under two circumstances:

1) Currently, if the stdlib module ends up getting imported *even
indirectly* by something you use. The indirect import is the subtle
one here.

2) Under the proposal to reorder sys.path, if you want to import
*your* module, you wouldn't be able to.

I'm responding to the suggestion that the standard library names are
"well known" to the point where people can be told not to use them. I
say they are not; sure, everyone knows that 'sys.py' would be a bad
name, and you wouldn't call your module "pprint.py" unless you
deliberately want to shadow the normal standard library module, but
there are a lot that are more obscure. Suppose you start learning
about Python's data types, and you create a "types.py" that lets you
play with strings, integers, floats, and so on. Then, in the next
lesson, you meet enumerations, and enum.py.

$ touch types.py; python3 -c "import enum"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.6/enum.py", line 2, in <module>
    from types import MappingProxyType, DynamicClassAttribute
ImportError: cannot import name 'MappingProxyType'

Tell me, is this an implausible scenario? Should a person just
learning about Python's type system have to also learn not to use the
name "types.py"?

ChrisA

From steve at pearwood.info  Sun Nov  1 10:44:10 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 2 Nov 2015 02:44:10 +1100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <201511010753.tA17rHs3025816@fido.openend.se>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
Message-ID: <20151101154410.GJ10946@ando.pearwood.info>

On Sun, Nov 01, 2015 at 08:53:17AM +0100, Laura Creighton wrote:
> In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes:

> >Scripts which deliberately or unintentionally shadow installed packages 
> >will break from this change. I don't have a problem with this. You can't 
> >fix harmful behaviour without breaking code that depends on that harmful 
> >behaviour. 
> 
> This is a bad idea, if you mean 'shadows anything in site-packages'.
> I write a perfectly good working program, which then silently breaks
> because somebody happens to install a site package with a name
> conflict with my code.

I'm willing to consider that "" should appear in the middle of sys.path:

[standard-library-directories, "", site-package-directories]

but that still allows accidental shadowing of third-party packages. I 
don't think that the problem of accidently shadowing numpy is less 
deserving of a solution than accidently shadowing random. They're both 
problems.

The downside of using a search path, instead of explicit full pathnames, 
is that you can have conflicts where two or more modules have the same 
name, and only the first can be imported. That's unavoidable. We can 
only try to minimize the risk of accidental shadowing, not prevent it 
altogether.


> Can you imagine being in the middle of
> writing and debugging such a thing and have everything start failing 
> because suddenly your program isn't the one being found? How long is 
> it going to take you to stop looking at your own code, and your own 
> setup for the problem and begin looking at what packages got installed 
> by anybody else sharing this machine, and what they are named?

Now just a minute, I'm not proposing that this change occur at random, 
in the middle of a debugging session. Not even in a point (micro) 
release. So the code will only break (if it breaks) when you upgrade 
from version 3.x to 3.6 or higher. Moving to a new minor version number 
is a big step, and people expect that things *could* break.

(They don't necessarily expect that they *will* break, but it's 
certainly a possibility that they might.)

As an experienced developer, what do you do when code that works in one 
version stops working in the next? Everyone I know immediately assumes 
that something has changed in the new Python version, and proceeds from 
there. In this case, they'll be right. Hopefully they will read the 
release notes. Or the documentation. Or ask on StackOverflow.

The same applies for when you install a new package: "Everything worked 
fine yesterday." "What changed?" "Well, I installed a new package with 
the same name as one of my modules." "Well there you go."

Of course we can come up with scenarios that are more confusing. Suppose 
you have a program that you don't use very often, say, once a year, that 
relies on "" being at the front of the path, and then somebody who isn't 
you installs a site-wide third-party package that now shadows one of 
your modules, and you don't find out about for six months.

But this sort of thing can already happen: any time you add a 
third-party module, it may shadow *other* third party modules that are 
found later in the path. So I'm not creating a brand new failure mode, 
I'm just changing how it applies a little.

There are all sorts of debugging strategies available to an experienced 
developer, starting from "print out module.__file__" to "use the 
debugger", which a beginner who has just written "turtle.py" would never 
think of. If I'm going to break anyone's code, I'd rather it be yours 
than theirs, since I have every confidence you can debug any problems 
fairly quickly.

I don't suggest breaking working code lightly, but the issue of 
shadowing the stdlib is an ever-present millstone around every Python 
developer's neck. The stdlib alone, never mind site packages, is too big 
to expect everyone to memorise what names they should not use. We can't 
realistically expect people to avoid shadowing. But we can shift it from 
"easy to shadow, and mostly affect those who are least able to cope" to 
"harder to shadow, and mostly affect those who can cope".



-- 
Steve

From lac at openend.se  Sun Nov  1 11:59:40 2015
From: lac at openend.se (Laura Creighton)
Date: Sun, 01 Nov 2015 17:59:40 +0100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <20151101154410.GJ10946@ando.pearwood.info>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <20151101154410.GJ10946@ando.pearwood.info>
Message-ID: <201511011659.tA1GxeiG001458@fido.openend.se>

In a message of Mon, 02 Nov 2015 02:44:10 +1100, "Steven D'Aprano" writes:
>On Sun, Nov 01, 2015 at 08:53:17AM +0100, Laura Creighton wrote:
>> In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano" writes:
>but that still allows accidental shadowing of third-party packages. I 
>don't think that the problem of accidently shadowing numpy is less 
>deserving of a solution than accidently shadowing random. They're both 
>problems.

And I do.

seriously.  I think that shadowing third party packages is something
I do all the time.  I am not willing to share a namespace with all
the package writers in the universe, and should I write package
name with the same name as somebody else's  third party package,
I damn well want mine, not some other one.

But learners shadowing the stdlib are a separate case.

We've had similar arguments before, the slipperly slope sort.
Either you don't want to solve any unless all get solved, or
else you are afraid that minor precedent here will lead to
all hell breaking loose when it is extended to its ultimate
conclusion.

But I think the entire purpose of being a human being is to
make such judgments.  And I think that this is an easy one.
Shadowing the stdlib can be treated as a catchable mistake.
Shadowing an outside package is something we want to do all the
time.

>> because suddenly your program isn't the one being found? How long is 
>> it going to take you to stop looking at your own code, and your own 
>> setup for the problem and begin looking at what packages got installed 
>> by anybody else sharing this machine, and what they are named?
>
>Now just a minute, I'm not proposing that this change occur at random, 
>in the middle of a debugging session. Not even in a point (micro) 
>release. So the code will only break (if it breaks) when you upgrade 
>from version 3.x to 3.6 or higher. Moving to a new minor version number 
>is a big step, and people expect that things *could* break.

Ah, I think your thinking is less than clear on this one.

1. my sysadmin does an update and suddently the system python is a
different one than the one where I wrote my code.

some poor sysadmin soul who never wrote anything gets to deal
with the fact that my code now has name conflicts with
packages that we have, but I never knew about and never imported.

2. things fall over because the new verision of python,
in effect, shadows my code.

I am not willing to share my namespace with every other python
package creator on the planet.

I am a consultant.  When my code breaks, they call me.  I am 
unwilling to have my code break every time some human being
decides to make a package named something I already used over
the last 18 years of writing python code.  (I know that
3.x has not been out that long, but 18 years from now, 
the same condition holds.)

Laura

From srkunze at mail.de  Sun Nov  1 12:14:20 2015
From: srkunze at mail.de (Sven R. Kunze)
Date: Sun, 1 Nov 2015 18:14:20 +0100
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
Message-ID: <5636486C.6080401@mail.de>

Hi Albert,

sounds useful. I can remember needing it myself.

On 01.11.2015 12:08, Albert ten Oever wrote:
> Hi all,
>
> I really hope this isn't proposed before - I couldn't find anything in 
> the archives.
>
> I want to propose to include a 'partition' (mathematically more 
> correct: a set partition) function in itertools.

This operates over iterables, right? So, it's not quite about sets alone.

> To my knowledge, partitioning of a set (iterable) is quite a common 
> thing to do and a logic extension of the combinatoric generators in 
> itertools.
>
> It is implemented as a Python recipe 
> (http://code.activestate.com/recipes/576795-partitioning-a-sequence/). 
> I found that implementing a partition generator of an iterable isn't 
> very straightforward, which, in my opinion, strengthens the case for 
> implementing it as a separate function.

Not quite sure if I understand the implementation. Reading this 
http://mathworld.wolfram.com/BellNumber.html for the given example of 6, 
it should have 203 possible partitions. What am I missing?

The implementation over there reads as if one gets a list of all 
possible partitions. Picking a specific subset ("all 4-part-partitions") 
could be useful as well.

> Definitions of partitions: 
> http://mathworld.wolfram.com/SetPartition.html or 
> https://en.wikipedia.org/wiki/Partition_of_a_set
>
> Humble regards,
>
> Albert.

Best,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151101/07c9e4e0/attachment.html>

From njs at pobox.com  Sun Nov  1 12:21:23 2015
From: njs at pobox.com (Nathaniel Smith)
Date: Sun, 1 Nov 2015 09:21:23 -0800
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <20151101154410.GJ10946@ando.pearwood.info>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org>
 <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <20151101154410.GJ10946@ando.pearwood.info>
Message-ID: <CAPJVwB=QZqtgdnEdxJya9kSv7eqG9JtUBr-gQPhYK8q3aeuSbw@mail.gmail.com>

On Nov 1, 2015 7:44 AM, "Steven D'Aprano" <steve at pearwood.info> wrote:
>
> On Sun, Nov 01, 2015 at 08:53:17AM +0100, Laura Creighton wrote:
> > In a message of Sun, 01 Nov 2015 17:06:30 +1100, "Steven D'Aprano"
writes:
>
> > >Scripts which deliberately or unintentionally shadow installed packages
> > >will break from this change. I don't have a problem with this. You
can't
> > >fix harmful behaviour without breaking code that depends on that
harmful
> > >behaviour.
> >
> > This is a bad idea, if you mean 'shadows anything in site-packages'.
> > I write a perfectly good working program, which then silently breaks
> > because somebody happens to install a site package with a name
> > conflict with my code.
>
> I'm willing to consider that "" should appear in the middle of sys.path:
>
> [standard-library-directories, "", site-package-directories]
>
> but that still allows accidental shadowing of third-party packages. I
> don't think that the problem of accidently shadowing numpy is less
> deserving of a solution than accidently shadowing random. They're both
> problems.

In this case there actually is a common use case [1] that involves issuing
using the "" entry to intentionally shadow third-party packages: if you're
hacking on foo v1.3.dev, and want to test out the code you're writing as
opposed to the installed version of foo v1.2, then switching to the root of
your source checkout before running python or nosetests or whatever
suffices.

(OTOH it is also the true that numpy has had enough users run into problems
due to trying to run python while accidentally in the root of a source
checkout that numpy/__init__.py contains special case code to check for
this case and error out early.)

-n

[1] Commonality here is assessed using the standard experimental procedure,
i.e., "*I* do it so it's common".
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151101/cfd0eb7a/attachment.html>

From brett at python.org  Sun Nov  1 12:40:23 2015
From: brett at python.org (Brett Cannon)
Date: Sun, 01 Nov 2015 17:40:23 +0000
Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module
In-Reply-To: <E1ZsaxV-0006p7-8h@se2-syd.hostedmail.net.au>
References: <n114bh$ih1$1@ger.gmane.org>
 <CAP7+vJ+zmEaJfyXNsBK2=fk42aTfVxA4o3tH7297_XEjM9jpkg@mail.gmail.com>
 <n11l6m$q68$1@ger.gmane.org>
 <CAP1=2W7pQAsGqUj7MrZP=6pAOzLBTDQigyaowipFUOpbhxG5eQ@mail.gmail.com>
 <E1ZsaxV-0006p7-8h@se2-syd.hostedmail.net.au>
Message-ID: <CAP1=2W51xqoGVGP6XVa=nKMZELB4r4SqN+UC66d7JRs+XUTJRw@mail.gmail.com>

On Sat, 31 Oct 2015 at 11:33 Steve Dower <steve.dower at python.org> wrote:

> Could we add a context manager to importlib (or perhaps site or sys) to
> temporarily disable imports from non-standard paths? I don't see any safe
> way to change the default behavior, but no reason we can't make it easy for
> applications to self-isolate.
>

There's https://www.python.org/dev/peps/pep-0406/ which tried to get
isolated import data. As for an immediate solution, trick is whether we
have stored somewhere what  the standard path entries are or if we have a
reliable way to figure out which paths are standard or not. There is also
the problem of not being thread-safe when swapping out sys.path entries.

-Brett


>
> Top-posted from my Windows Phone
> ------------------------------
> From: Brett Cannon <brett at python.org>
> Sent: ?10/?31/?2015 9:50
> To: Terry Reedy <tjreedy at udel.edu>; python-ideas at python.org
> Subject: Re: [Python-ideas] Add 'module' module, similar to 'keyword'
> module
>
>
>
> On Fri, 30 Oct 2015 at 22:57 Terry Reedy <tjreedy at udel.edu> wrote:
>
>> On 10/30/2015 9:19 PM, Guido van Rossum wrote:
>> > There's sys.builtin_module_names which returns the names of the
>> > hardcoded builtin modules.
>>
>> Great.  With this solved, I opened an issue for IDLE.
>> https://bugs.python.org/issue25522
>>
>> > Dynamically loaded modules can be found by
>> > searching sys.path in the usual way -- importlib shoul know. I wonder if
>> > just asking importlib whether it can locate a given module would be
>> enough?
>>
>> The default search order is stdlib builtins, local user files, /lib
>> files, so the shadowing issue the opposite for builtin and /lib modules.
>>   Hence a different message is needed.
>>
>
> Quick and dirty way to use importlib is to get the location of the stdlib
> (os.__file__ should work since it's hard-coded in the interpreter as
> representing where the stdlib is) and then use importlib.find_spec() for a
> module name to check if the file location in the spec has the same location
> prefix as os or not (make sure you use absolute paths since it isn't
> guaranteed if you don't execute site.py).
>
> I've now seen this use case, the logging one, and the 2to3 module rename.
> I'm starting to wonder if there some general solution that should get added
> to the import machinery that  can serve these cases more easily than with
> importers or __import__ overrides that can be tricky to get right.
>
> -Brett
>
>
>>
>> > On Fri, Oct 30, 2015 at 6:09 PM, Terry Reedy
>> > <tjreedy at udel.edu
>> > <mailto:tjreedy at udel.edu>> wrote:
>> >
>> >     This idea results from issue of user files shadowing stdlib files on
>> >     import.  There was a thread on pydev about this yesterday.  There is
>> >     also an opposite issue of builtin modules shadowing user files.
>> >
>> >     The keyword module provides kwlist and iskeyword function.  One use
>> >     of kwlist is used in some other stdlib modules and can be used by
>> >     syntax highlighters (as in IDLE).  Kwlist is updated by the main
>> >     function.
>> >
>> >     A module module would have at least liblist and islibmodule
>> >     function. Liblist would contain all directories with __init__.py and
>> >     all .py files.  (I don't think files within package directories
>> >     should be included, as there is no direct shadowing problem.)  A
>> >     python oriented editor could then warn on save requests "This name
>> >     matches a stdlib name in /Lib. If you run python in this directory,
>> >     you will not be able to import the stdlib module.  Continue?".
>> >
>> >     The module should also have binlist and isbinmodule for builtin
>> >     modules.  (I do not know how to get such a list.  If necessary, an
>> >     api could be added.)  An editor could than warn "This name matches a
>> >     builtin stdlib name.  You will not be able to import this file.
>> >     Continue?".
>> >
>> >     --
>> >     Terry Jan Reedy
>> >
>> >     _______________________________________________
>> >     Python-ideas mailing list
>> >     Python-ideas at python.org
>> >     <mailto:Python-ideas at python.org>
>> >     https://mail.python.org/mailman/listinfo/python-ideas
>> >     Code of Conduct: http://python.org/psf/codeofconduct/
>> >
>> >
>> >
>> >
>> > --
>> > --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
>> >
>> >
>> > _______________________________________________
>> > Python-ideas mailing list
>> > Python-ideas at python.org
>> > https://mail.python.org/mailman/listinfo/python-ideas
>> > Code of Conduct: http://python.org/psf/codeofconduct/
>> >
>>
>>
>> --
>> Terry Jan Reedy
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151101/62b42f6d/attachment-0001.html>

From storchaka at gmail.com  Sun Nov  1 13:10:08 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 1 Nov 2015 20:10:08 +0200
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
Message-ID: <n15ki1$crv$1@ger.gmane.org>

On 01.11.15 13:08, Albert ten Oever wrote:
> I really hope this isn't proposed before - I couldn't find anything in
> the archives.
>
> I want to propose to include a 'partition' (mathematically more correct:
> a set partition) function in itertools. To my knowledge, partitioning of
> a set (iterable) is quite a common thing to do and a logic extension of
> the combinatoric generators in itertools.
>
> It is implemented as a Python recipe
> (http://code.activestate.com/recipes/576795-partitioning-a-sequence/). I
> found that implementing a partition generator of an iterable isn't very
> straightforward, which, in my opinion, strengthens the case for
> implementing it as a separate function.

The implementation can be simpler and don't require itertools:

def partition(seq):
     if seq:
         yield [seq]
     for i in range(1, len(seq)):
         s = seq[i:]
         for p in partition(seq[:i]):
             p.append(s)
             yield p

 >>> from pprint import pprint
 >>> pprint(sorted(partition('abcde'), key=len))
[['abcde'],
  ['a', 'bcde'],
  ['ab', 'cde'],
  ['abc', 'de'],
  ['abcd', 'e'],
  ['a', 'b', 'cde'],
  ['a', 'bc', 'de'],
  ['ab', 'c', 'de'],
  ['a', 'bcd', 'e'],
  ['ab', 'cd', 'e'],
  ['abc', 'd', 'e'],
  ['a', 'b', 'c', 'de'],
  ['a', 'b', 'cd', 'e'],
  ['a', 'bc', 'd', 'e'],
  ['ab', 'c', 'd', 'e'],
  ['a', 'b', 'c', 'd', 'e']]



From lukasjuhrich at wh2.tu-dresden.de  Sun Nov  1 13:30:09 2015
From: lukasjuhrich at wh2.tu-dresden.de (Lukas Juhrich)
Date: Sun, 1 Nov 2015 19:30:09 +0100
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <5636486C.6080401@mail.de>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
 <5636486C.6080401@mail.de>
Message-ID: <56365A31.6020803@wh2.tu-dresden.de>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi Sven,

The issue here is that such an implementation does not allow ?gaps?
and implicitly enforces that the elements of such a partition only
contain elements directly following each other, thus making it not a
complete implementation of producing a set partition.  To give a
specific counterexample, `['acd', 'bef']` will never be produced.

The question to ask is whether you want to implement exact
mathematical set partition, or if you implement something with the
above constraint, and call it something different like ?iterable
partition?.  If choosing the latter, however, it should be made clear
by the function name and in the documentation that this is *not*
comparable to set partition.

Perhaps it will be useful to implement both concepts, since such an
?iterable partition? seems to be easier to implement and will perhaps
be faster (which, however, is a blind guess and depends on the
implementation of producing a set partition).

- --
Sincerely,
Lukas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJWNlotAAoJENBP4KTXGFXeSJ8P/33IfUqVe+LqKjdjghOVrb2A
2wcj85GJHV5hu8UIYNfchVwvBML7w0LpgvHsopD7NkXLYepOy1AuSDTlePYZGu2g
TvVFLd861HZVAQFtFwuuY1bAkNlCDDtNOUZYrortkc7F05TFBaGiTu7Z7zrHG9PN
5eLZ7dpr5y9Ok8nCaerUVnxAn81+A+mr0O2KPAAOfTLA0C0bo4uWJsp1WJMcLF1N
o0RpzrZ68Qy1vzfnRaf5T/MpcxIFaPxTYee8wxg/pGse5gPJidvK83G5MV/ylHbK
INqG7iPTcD8DJ0OpvR/462QaxBf225fP3Dg+xUtKvpBw2SMrSfc8X9VXOE7EFMJ+
m9Wajx5d1+wVvlTzdEa5Nu5A7I5DGDO+zW7jeuZ2Ri6q6aeKHRt1JLsi/5VIJ+/B
0xb7/a5JKcjskrZuQrlCzVKlvOUq2Wf1SZKlPLq6jsB3xkT1qJhF2QssoR3s2aNn
998gwRl3sJzrdVh9+QxCXjoQ92aqQCLRsvHll+uZ9bjJEhD3SXlyFhqmhLf396fq
BMWpxFrSuf4ltqke2WQrzrvpJnxd1MH7yH+2tVQej8/kUHJyCWdBLZeQJeFljiJa
3zYh6BQ8KjwtjNcSHzmhANO/0FSwzVDPSvP2ocEbjYjv5WMXXk0/sNdRgsb3gkQ1
5zuBS6j2n5n9SeUZ0IcM
=Ki11
-----END PGP SIGNATURE-----

From tjreedy at udel.edu  Sun Nov  1 20:10:57 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 1 Nov 2015 20:10:57 -0500
Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module
In-Reply-To: <56354996.8030807@brenbarn.net>
References: <n114bh$ih1$1@ger.gmane.org>
 <20151031081450.GD10946@ando.pearwood.info> <n13ga1$scg$1@ger.gmane.org>
 <56354996.8030807@brenbarn.net>
Message-ID: <n16d79$rac$1@ger.gmane.org>

On 10/31/2015 7:07 PM, Brendan Barnwell wrote:
> On 2015-10-31 15:45, Terry Reedy wrote:
>> Revised and reduced proposal: If other people would find it useful, add
>> all_stdlib_toplevel_module_names - builtin_module_names to sys as
>> something equivalent to .other_stdlib_module_names or
>> .python_coded_module_names.  Anyone wanting all_toplevel_module names
>> could add the two.  Or add the latter, and let others subtract.
>
>      One thing I've sometimes wondered about for Python 4000 is the idea
> of putting the whole standard library under a single top-level package
> (e.g., from stdlib import sys).  This would be a big change but would
> reduce the surprises that can arise from stdlib modules named things
> like "string", "parser", etc.  I see a brief mention of this in PEP 3108
> but that's it.  Was there more discussion of the idea?

It has been discussed and rejected by Guido.  I think it would be more 
sensible to require 'from . import mymod' for local imports.

-- 
Terry Jan Reedy


From stephen at xemacs.org  Sun Nov  1 20:25:21 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 2 Nov 2015 10:25:21 +0900
Subject: [Python-ideas]  Include partitioning in itertools
In-Reply-To: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
Message-ID: <22070.48001.916960.173974@turnbull.sk.tsukuba.ac.jp>

Albert ten Oever writes:

 > I want to propose to include a 'partition' (mathematically more
 > correct: a set partition) function in itertools. To my knowledge,
 > partitioning of a set (iterable) is quite a common thing to do and
 > a logic extension of the combinatoric generators in itertools.

Even after other discussion, I don't understand.  Sets and iterables
are rather different things.  The recipe from ActiveState would be
perfectly happy with partitioning "aaaaaa", but that sequence is not a
reasonable representation of a set (I would normally say it has only
one element despite being repeated 6 times), and the results makes no
sense as a set, either.  For example, as sets ["aa", "aaaa"] and
["aaaa", "aa"] are the same.  And I suspect that you might get
anomolies using a sequence or iterable-based algorithm (for example,
(["a"] is not ["a"]) and ("a" is "a") both evaluate to true because of
the way the CPython implementation handles different objects).

I'm also not clear on what you would hope to get from an infinite
iterator.  I assume passing an infinite iterator would be a programmer
error, but you could also restrict the function to working on concrete
sequences.

All that doesn't mean it's a bad idea, but at this point it's
definitely underspecified.

I'd also prefer a more explicit name, such as enumerate_partitions.

Steve



From brenbarn at brenbarn.net  Sun Nov  1 20:29:28 2015
From: brenbarn at brenbarn.net (Brendan Barnwell)
Date: Sun, 01 Nov 2015 17:29:28 -0800
Subject: [Python-ideas] Add 'module' module, similar to 'keyword' module
In-Reply-To: <n16d79$rac$1@ger.gmane.org>
References: <n114bh$ih1$1@ger.gmane.org>
 <20151031081450.GD10946@ando.pearwood.info> <n13ga1$scg$1@ger.gmane.org>
 <56354996.8030807@brenbarn.net> <n16d79$rac$1@ger.gmane.org>
Message-ID: <5636BC78.1040200@brenbarn.net>

On 2015-11-01 17:10, Terry Reedy wrote:
> On 10/31/2015 7:07 PM, Brendan Barnwell wrote:
>> On 2015-10-31 15:45, Terry Reedy wrote:
>>> Revised and reduced proposal: If other people would find it useful, add
>>> all_stdlib_toplevel_module_names - builtin_module_names to sys as
>>> something equivalent to .other_stdlib_module_names or
>>> .python_coded_module_names.  Anyone wanting all_toplevel_module names
>>> could add the two.  Or add the latter, and let others subtract.
>>
>>      One thing I've sometimes wondered about for Python 4000 is the idea
>> of putting the whole standard library under a single top-level package
>> (e.g., from stdlib import sys).  This would be a big change but would
>> reduce the surprises that can arise from stdlib modules named things
>> like "string", "parser", etc.  I see a brief mention of this in PEP 3108
>> but that's it.  Was there more discussion of the idea?
>
> It has been discussed and rejected by Guido.  I think it would be more
> sensible to require 'from . import mymod' for local imports.

	That doesn't handle the situation where you want to import a 
third-party module that is neither part of the stdlib nor part of your 
own project.  But I guess my question is answered anyhow.


-- 
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."
    --author unknown

From mertz at gnosis.cx  Sun Nov  1 20:49:22 2015
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 1 Nov 2015 17:49:22 -0800
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
Message-ID: <CAEbHw4bqtxcOc5=gBzb4dcwikspS9knxrdpYWwK2=yntMj0OJg@mail.gmail.com>

This doesn't make sense to me for `itertools`.  There is no way to
"partition" an infinite sequence in the style of the recipe you show (nor
any sensible one I can think of).  I think a better name would be
`powerset()` anyway; but in either case, it's not an `itertools` type of
operation.

I think what might be more useful is to make a mix-in class that gave
concrete collections a `.powerset()` method that preserved the types of the
original collection.  I think you can even do this as an iterable without
having to concretize the whole set of subsets (but not from an iterable,
from a concrete collection, as input).

On Sun, Nov 1, 2015 at 3:08 AM, Albert ten Oever <agtoever at hotmail.com>
wrote:

> Hi all,
>
> I really hope this isn't proposed before - I couldn't find anything in the
> archives.
>
> I want to propose to include a 'partition' (mathematically more correct: a
> set partition) function in itertools. To my knowledge, partitioning of a
> set (iterable) is quite a common thing to do and a logic extension of the
> combinatoric generators in itertools.
>
> It is implemented as a Python recipe (
> http://code.activestate.com/recipes/576795-partitioning-a-sequence/). I
> found that implementing a partition generator of an iterable isn't very
> straightforward, which, in my opinion, strengthens the case for
> implementing it as a separate function.
>
> Definitions of partitions: http://mathworld.wolfram.com/SetPartition.html
> or https://en.wikipedia.org/wiki/Partition_of_a_set
>
> Humble regards,
>
> Albert.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151101/25ec8d67/attachment-0001.html>

From tim.peters at gmail.com  Sun Nov  1 20:54:22 2015
From: tim.peters at gmail.com (Tim Peters)
Date: Sun, 1 Nov 2015 19:54:22 -0600
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <5636486C.6080401@mail.de>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
 <5636486C.6080401@mail.de>
Message-ID: <CAExdVN=CYqqawp4bncHVY5UyrUyPpxbbF=TTBszfi3dEdtS6tQ@mail.gmail.com>

[Albert ten Oever]
> ...
> I want to propose to include a 'partition' (mathematically more correct: a
> set partition) function in itertools.
>
> To my knowledge, partitioning of a set (iterable) is quite a common thing to
> do and a logic extension of the combinatoric generators in itertools.
>
> It is implemented as a Python recipe
> (http://code.activestate.com/recipes/576795-partitioning-a-sequence/).


[Sven R. Kunze]
> ...
> Not quite sure if I understand the implementation. Reading this
> http://mathworld.wolfram.com/BellNumber.html for the given example of 6, it
> should have 203 possible partitions. What am I missing?

The ActiveState recipe has nothing to do with set partitions.  Instead
it returns all ways of breaking a sequence into a sequence of
subsequences whose sum (catenation) is the original sequence.  If
there are N elements in the original sequence, there are 2**(N-1) ways
to do that.  That's why there are 32 results in the recipe's example
output (2**(len("abcdef")-1) == 32).  There would be 203 "set
partitions" of a 6-element set.

From mertz at gnosis.cx  Sun Nov  1 20:58:42 2015
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 1 Nov 2015 17:58:42 -0800
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <CAEbHw4bqtxcOc5=gBzb4dcwikspS9knxrdpYWwK2=yntMj0OJg@mail.gmail.com>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
 <CAEbHw4bqtxcOc5=gBzb4dcwikspS9knxrdpYWwK2=yntMj0OJg@mail.gmail.com>
Message-ID: <CAEbHw4ahcESzFjxWinWxFg_dsOCq6ebgn37Rdc63zizPP38iaQ@mail.gmail.com>

Oh, yeah... I see what you are asking for isn't quite the same thing as
power set.  But still, it's not something you can do in finite time with
infinite iterators (nor in tractable time with *large* iterators).  So
`itertools` isn't where it belongs.

On Sun, Nov 1, 2015 at 5:49 PM, David Mertz <mertz at gnosis.cx> wrote:

> This doesn't make sense to me for `itertools`.  There is no way to
> "partition" an infinite sequence in the style of the recipe you show (nor
> any sensible one I can think of).  I think a better name would be
> `powerset()` anyway; but in either case, it's not an `itertools` type of
> operation.
>
> I think what might be more useful is to make a mix-in class that gave
> concrete collections a `.powerset()` method that preserved the types of the
> original collection.  I think you can even do this as an iterable without
> having to concretize the whole set of subsets (but not from an iterable,
> from a concrete collection, as input).
>
> On Sun, Nov 1, 2015 at 3:08 AM, Albert ten Oever <agtoever at hotmail.com>
> wrote:
>
>> Hi all,
>>
>> I really hope this isn't proposed before - I couldn't find anything in
>> the archives.
>>
>> I want to propose to include a 'partition' (mathematically more correct:
>> a set partition) function in itertools. To my knowledge, partitioning of a
>> set (iterable) is quite a common thing to do and a logic extension of the
>> combinatoric generators in itertools.
>>
>> It is implemented as a Python recipe (
>> http://code.activestate.com/recipes/576795-partitioning-a-sequence/). I
>> found that implementing a partition generator of an iterable isn't very
>> straightforward, which, in my opinion, strengthens the case for
>> implementing it as a separate function.
>>
>> Definitions of partitions: http://mathworld.wolfram.com/SetPartition.html
>>  or https://en.wikipedia.org/wiki/Partition_of_a_set
>>
>> Humble regards,
>>
>> Albert.
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151101/202c8c5c/attachment.html>

From brenbarn at brenbarn.net  Sun Nov  1 21:37:15 2015
From: brenbarn at brenbarn.net (Brendan Barnwell)
Date: Sun, 01 Nov 2015 18:37:15 -0800
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <CAEbHw4ahcESzFjxWinWxFg_dsOCq6ebgn37Rdc63zizPP38iaQ@mail.gmail.com>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
 <CAEbHw4bqtxcOc5=gBzb4dcwikspS9knxrdpYWwK2=yntMj0OJg@mail.gmail.com>
 <CAEbHw4ahcESzFjxWinWxFg_dsOCq6ebgn37Rdc63zizPP38iaQ@mail.gmail.com>
Message-ID: <5636CC5B.7020806@brenbarn.net>

On 2015-11-01 17:58, David Mertz wrote:
> Oh, yeah... I see what you are asking for isn't quite the same thing as
> power set.  But still, it's not something you can do in finite time with
> infinite iterators (nor in tractable time with *large* iterators).  So
> `itertools` isn't where it belongs.

	I don't think that reasoning is sound.  Itertools already includes 
several combinatoric generators like permutations and combinations, 
which aren't computable for infinite iterators and which take 
intractable time for large finite iterators.

-- 
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."
    --author unknown

From mertz at gnosis.cx  Sun Nov  1 22:42:42 2015
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 1 Nov 2015 19:42:42 -0800
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <5636CC5B.7020806@brenbarn.net>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
 <CAEbHw4bqtxcOc5=gBzb4dcwikspS9knxrdpYWwK2=yntMj0OJg@mail.gmail.com>
 <CAEbHw4ahcESzFjxWinWxFg_dsOCq6ebgn37Rdc63zizPP38iaQ@mail.gmail.com>
 <5636CC5B.7020806@brenbarn.net>
Message-ID: <CAEbHw4YF9PF4oF6mEwxj4Yd=+1TQTDkDpm5HjyiUx_mRF1SHjg@mail.gmail.com>

I'm surprised to have just noticed that `itertools` is oddly broken this
way (under Python 3.4.3):

This is perfectly happy:

>>> def fibs():

    a, b = 1, 1
    while True:
        yield b
        b, a = a+b, b
...
>>> f = ((a,b) for a in fibs() for b in fibs())
>>> next(f)
Out[3]: (1, 1)
>>> next(f)
Out[4]: (1, 2)
>>> next(f)
Out[5]: (1, 3)

And yet this freezes be greedily trying to go through the infinite iterator:

>>> from itertools import product
>>> product(fibs(), fibs())


On Sun, Nov 1, 2015 at 6:37 PM, Brendan Barnwell <brenbarn at brenbarn.net>
wrote:

> On 2015-11-01 17:58, David Mertz wrote:
>
>> Oh, yeah... I see what you are asking for isn't quite the same thing as
>> power set.  But still, it's not something you can do in finite time with
>> infinite iterators (nor in tractable time with *large* iterators).  So
>> `itertools` isn't where it belongs.
>>
>
>         I don't think that reasoning is sound.  Itertools already includes
> several combinatoric generators like permutations and combinations, which
> aren't computable for infinite iterators and which take intractable time
> for large finite iterators.
>
> --
> Brendan Barnwell
> "Do not follow where the path may lead.  Go, instead, where there is no
> path, and leave a trail."
>    --author unknown
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151101/78ea3879/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Sun Nov  1 18:22:39 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 02 Nov 2015 12:22:39 +1300
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <20151101060629.GE10946@ando.pearwood.info>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
Message-ID: <56369EBF.2000806@canterbury.ac.nz>

Steven D'Aprano wrote:

> (1) Beginning with Python 3.6, the default is that the current directory 
> is put at the end of sys.path rather than the beginning. Instead of:
> 
> Scripts which deliberately or unintentionally shadow installed packages 
> will break from this change. I don't have a problem with this. You can't 
> fix harmful behaviour without breaking code that depends on that harmful 
> behaviour.

It will also break scripts that rely on looking at the
first element of sys.path to find the directory they
are running from. Such code is NOT relying on harmful
behaviour.

-- 
Greg


From rosuav at gmail.com  Sun Nov  1 23:43:43 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 2 Nov 2015 15:43:43 +1100
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <CAEbHw4YF9PF4oF6mEwxj4Yd=+1TQTDkDpm5HjyiUx_mRF1SHjg@mail.gmail.com>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
 <CAEbHw4bqtxcOc5=gBzb4dcwikspS9knxrdpYWwK2=yntMj0OJg@mail.gmail.com>
 <CAEbHw4ahcESzFjxWinWxFg_dsOCq6ebgn37Rdc63zizPP38iaQ@mail.gmail.com>
 <5636CC5B.7020806@brenbarn.net>
 <CAEbHw4YF9PF4oF6mEwxj4Yd=+1TQTDkDpm5HjyiUx_mRF1SHjg@mail.gmail.com>
Message-ID: <CAPTjJmr-iYGVOoW7DkSSZuLrrekK30+oz_E0EGNsXjdf8oi3EA@mail.gmail.com>

On Mon, Nov 2, 2015 at 2:42 PM, David Mertz <mertz at gnosis.cx> wrote:
> I'm surprised to have just noticed that `itertools` is oddly broken this way
> (under Python 3.4.3):
>
> This is perfectly happy:
>
>>>> def fibs():
>
>     a, b = 1, 1
>     while True:
>         yield b
>         b, a = a+b, b
> ...
>>>> f = ((a,b) for a in fibs() for b in fibs())
>>>> next(f)
> Out[3]: (1, 1)
>>>> next(f)
> Out[4]: (1, 2)
>>>> next(f)
> Out[5]: (1, 3)
>
> And yet this freezes be greedily trying to go through the infinite iterator:
>
>>>> from itertools import product
>>>> product(fibs(), fibs())

Yeah. The problem is that you're looking at a simplified version of
the reference implementation. Further down it says that it's
equivalent to a rather longer function, which (critically) includes a
call to tuple(), to coalesce the iterables into their values. When you
use the naive genexp, what you're actually doing is constructing a new
instance of b's fibs() for every iteration of a's one (you won't see
that on an infinite iterator, but try a finite iterator with some
print() calls and you'll see); itertools.product obviously can't do
that, as it's given the iterable and not the means of constructing
more. The tiny example up the top is inaccurate in its handling of
edge cases, but for stable, repeatable iterables, it will return the
same results.

ChrisA

From tritium-list at sdamon.com  Mon Nov  2 01:54:48 2015
From: tritium-list at sdamon.com (Alexander Walters)
Date: Mon, 02 Nov 2015 01:54:48 -0500
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <201511011353.tA1DrNkI019815@fido.openend.se>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <563612E2.1060201@sdamon.com> <201511011353.tA1DrNkI019815@fido.openend.se>
Message-ID: <563708B8.8070808@sdamon.com>

On 11/1/2015 08:53, Laura Creighton wrote:
> In a message of Sun, 01 Nov 2015 08:25:54 -0500, Alexander Walters writes:
>> Honestly, shadowing modules is something that should be solved by
>> renaming modules.  If you are worrying about shadowing ONLY the standard
>> library - guess what?  those names don't change often, and are well
>> known.  Don't use those names.
> The problem is that the sort of people who make these errors don't
> know the names.  Knowing that 'turtle.py' is a bad name for 'my
> very first program that does turtle graphics' is a more advanced
> skill than the turtle program writers, or even their teachers, can
> be expected to have.
This is what is called a teachable moment.  It is BETTER that python 
does nothing to prevent or warn the user.  Let them fail, and fail 
hard.  Its better for them to fail.  I see any argument against this as 
pure detriment to the user.  Wanting these warnings is wanting to 
disadvantage the new user - debugging shadowed names is a skill you need 
in your toolbox, and this deprives people of that vital early lesson.

From toddrjen at gmail.com  Mon Nov  2 03:19:41 2015
From: toddrjen at gmail.com (Todd)
Date: Mon, 2 Nov 2015 09:19:41 +0100
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <56361771.9090702@lucidity.plus.com>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org>
 <20151101060629.GE10946@ando.pearwood.info>
 <201511010753.tA17rHs3025816@fido.openend.se>
 <CAPTjJmr8EXXCSqorEaxF2LSf9XFnbf-5hWhT82gj4gNzH9G7jQ@mail.gmail.com>
 <201511010808.tA188a77027105@fido.openend.se>
 <CAPTjJmofHSfE4kKmN8dBeyY6YZEM6YVruAvmyQ8gKXkbEemyAA@mail.gmail.com>
 <201511010949.tA19nnwj001815@fido.openend.se>
 <56361771.9090702@lucidity.plus.com>
Message-ID: <CAFpSVpJP=VDOw10OMHvpmXE96Qvzz7-OPwVOsNLzAua8GnJjSg@mail.gmail.com>

On Nov 1, 2015 14:45, "Erik" <python at lucidity.plus.com> wrote:
>
> On 01/11/15 09:49, Laura Creighton wrote:
>>
>> Adding warnings to IDLE when you save a file is a fine idea, and will
>> help IDLE users avoid pain.  I want to help _everybody_ with a change
>> to Python, so that it issues a warning when you shadow something in
>> the standard library.
>
>
> I think that a solution to this could be something along the lines of
having a command line switch (let's say '-w') or other way of enabling a
"shadow module checking" mechanism.
>
> When an exception is thrown that is caught by the default handler and
output to stderr, an additional line of output could be appended:
>
> "Note: This problem could be caused by a module that shadows a standard
library or third party module. Run Python with the '-w' switch for more
detail."
>
> I'm not sure if this would be suitable for _all_ exceptions or only those
from certain points in the exception hierarchy.
>
> When the '-w' switch is enabled, the default exception handler would
instead enumerate all active modules and, for any which has another module
with the same name further down the search path than the one that is
loaded, would output some sort of diagnostic message (i.e., a list of which
loaded modules are shadowing _something_else_ that would have been loaded
otherwise) - similar to your example below:
>
>
>> Something like:
>> Warning: local file /u/lac/junk/string.py shadows module named string in
the
>> Standard Library
>
>
> This would prevent "power users" from always having to see a potentially
large diagnostic after each uncaught exception, but generally remind
everyone that such a thing is possible and "here's the easy way to check
for it".
>
> A bit like Valgrind's memcheck tool's "--leak-check=full" option - when
run without it, if there are leaks then memcheck reminds the user that
that's the way to start digging down into what might be causing them.

I was thinking of something like this too.  A list of stdlib modules could
be created statically when python is originally built (if it isn't
already).  This would have no changes in behaviour unless an error actually
happens.  And maybe to avoid any performance cost the message can only be
inserted when the interpreter actually exits, if that makes any sense at
all.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151102/4bd89f40/attachment.html>

From tjreedy at udel.edu  Mon Nov  2 06:14:12 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 2 Nov 2015 06:14:12 -0500
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <56369EBF.2000806@canterbury.ac.nz>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <56369EBF.2000806@canterbury.ac.nz>
Message-ID: <n17gic$1nd$1@ger.gmane.org>

On 11/1/2015 6:22 PM, Greg Ewing wrote:

> It will also break scripts that rely on looking at the
> first element of sys.path to find the directory they
> are running from.

'' is not very informative ;-).
I suspect the full path added in the past, but it is not now.
import os; os.getcwd() works better

-- 
Terry Jan Reedy


From greg.ewing at canterbury.ac.nz  Mon Nov  2 19:02:56 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 03 Nov 2015 13:02:56 +1300
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <n17gic$1nd$1@ger.gmane.org>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <56369EBF.2000806@canterbury.ac.nz> <n17gic$1nd$1@ger.gmane.org>
Message-ID: <5637F9B0.3010201@canterbury.ac.nz>

Terry Reedy wrote:
> '' is not very informative ;-).
> I suspect the full path added in the past, but it is not now.
> import os; os.getcwd() works better

You misunderstand. The point is *not* to find the
cwd, it's to find the directory containing the main
script, so you can load resources related to it.

There are probably other and possibly better ways
to go about that, but it works and is sometimes
used. The proposed change would break it.

-- 
Greg

From python at mrabarnett.plus.com  Mon Nov  2 20:22:29 2015
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 3 Nov 2015 01:22:29 +0000
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the
 standard library that IDLE depends on, bad things happen
In-Reply-To: <5637F9B0.3010201@canterbury.ac.nz>
References: <201510291559.t9TFxi4i003479@fido.openend.se>
 <n0ugob$mvt$1@ger.gmane.org> <20151101060629.GE10946@ando.pearwood.info>
 <56369EBF.2000806@canterbury.ac.nz> <n17gic$1nd$1@ger.gmane.org>
 <5637F9B0.3010201@canterbury.ac.nz>
Message-ID: <56380C55.4020405@mrabarnett.plus.com>

On 2015-11-03 00:02, Greg Ewing wrote:
> Terry Reedy wrote:
>> '' is not very informative ;-).
>> I suspect the full path added in the past, but it is not now.
>> import os; os.getcwd() works better
>
> You misunderstand. The point is *not* to find the
> cwd, it's to find the directory containing the main
> script, so you can load resources related to it.
>
> There are probably other and possibly better ways
> to go about that, but it works and is sometimes
> used. The proposed change would break it.
>
I use os.path.dirname(__file__) in the main script.


From agtoever at hotmail.com  Tue Nov  3 10:46:41 2015
From: agtoever at hotmail.com (Albert ten Oever)
Date: Tue, 3 Nov 2015 15:46:41 +0000
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <mailman.217.1446435764.4465.python-ideas@python.org>
References: <mailman.217.1446435764.4465.python-ideas@python.org>
Message-ID: <DUB129-W718C910470215D83B36391D62B0@phx.gbl>

> [Sven R. Kunze]
> > ...
> > Not quite sure if I understand the implementation. Reading this
> > http://mathworld.wolfram.com/BellNumber.html for the given example of 6, it
> > should have 203 possible partitions. What am I missing?
> > [Tim Peters]
> The ActiveState recipe has nothing to do with set partitions.  Instead
> it returns all ways of breaking a sequence into a sequence of
> subsequences whose sum (catenation) is the original sequence.  If
> there are N elements in the original sequence, there are 2**(N-1) ways
> to do that.  That's why there are 32 results in the recipe's example
> output (2**(len("abcdef")-1) == 32).  There would be 203 "set
> partitions" of a 6-element set.

@Tim: you are completely right. Sorry for the confusion referencing this implementation. Forget it.

> On Sun, Nov 1, 2015 at 6:37 PM, Brendan Barnwell <brenbarn at brenbarn.net> wrote:
> > On 2015-11-01 17:58, David Mertz wrote:
> >> Oh, yeah... I see what you are asking for isn't quite the same thing as
> >> power set.  But still, it's not something you can do in finite time with
> >> infinite iterators (nor in tractable time with *large* iterators).  So
> >> `itertools` isn't where it belongs.
> >>
> >
> >         I don't think that reasoning is sound.  Itertools already includes
> > several combinatoric generators like permutations and combinations, which
> > aren't computable for infinite iterators and which take intractable time
> > for large finite iterators.
> >

@Brendan: exactly! 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151103/4dad6c4a/attachment.html>

From srkunze at mail.de  Tue Nov  3 11:58:31 2015
From: srkunze at mail.de (Sven R. Kunze)
Date: Tue, 3 Nov 2015 17:58:31 +0100
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <56365A31.6020803@wh2.tu-dresden.de>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
 <5636486C.6080401@mail.de> <56365A31.6020803@wh2.tu-dresden.de>
Message-ID: <5638E7B7.2060608@mail.de>

Hey Lukas,

I agree both version (+ the n-part-partition-version of it) might be 
very useful to have.

But where do they go into?

itertools might be the right place for the iter_partition; but where to 
with the set_partition?

Best,
Sven

On 01.11.2015 19:30, Lukas Juhrich wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Hi Sven,
>
> The issue here is that such an implementation does not allow ?gaps?
> and implicitly enforces that the elements of such a partition only
> contain elements directly following each other, thus making it not a
> complete implementation of producing a set partition.  To give a
> specific counterexample, `['acd', 'bef']` will never be produced.
>
> The question to ask is whether you want to implement exact
> mathematical set partition, or if you implement something with the
> above constraint, and call it something different like ?iterable
> partition?.  If choosing the latter, however, it should be made clear
> by the function name and in the documentation that this is *not*
> comparable to set partition.
>
> Perhaps it will be useful to implement both concepts, since such an
> ?iterable partition? seems to be easier to implement and will perhaps
> be faster (which, however, is a blind guess and depends on the
> implementation of producing a set partition).
>
> - --
> Sincerely,
> Lukas
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQIcBAEBCAAGBQJWNlotAAoJENBP4KTXGFXeSJ8P/33IfUqVe+LqKjdjghOVrb2A
> 2wcj85GJHV5hu8UIYNfchVwvBML7w0LpgvHsopD7NkXLYepOy1AuSDTlePYZGu2g
> TvVFLd861HZVAQFtFwuuY1bAkNlCDDtNOUZYrortkc7F05TFBaGiTu7Z7zrHG9PN
> 5eLZ7dpr5y9Ok8nCaerUVnxAn81+A+mr0O2KPAAOfTLA0C0bo4uWJsp1WJMcLF1N
> o0RpzrZ68Qy1vzfnRaf5T/MpcxIFaPxTYee8wxg/pGse5gPJidvK83G5MV/ylHbK
> INqG7iPTcD8DJ0OpvR/462QaxBf225fP3Dg+xUtKvpBw2SMrSfc8X9VXOE7EFMJ+
> m9Wajx5d1+wVvlTzdEa5Nu5A7I5DGDO+zW7jeuZ2Ri6q6aeKHRt1JLsi/5VIJ+/B
> 0xb7/a5JKcjskrZuQrlCzVKlvOUq2Wf1SZKlPLq6jsB3xkT1qJhF2QssoR3s2aNn
> 998gwRl3sJzrdVh9+QxCXjoQ92aqQCLRsvHll+uZ9bjJEhD3SXlyFhqmhLf396fq
> BMWpxFrSuf4ltqke2WQrzrvpJnxd1MH7yH+2tVQej8/kUHJyCWdBLZeQJeFljiJa
> 3zYh6BQ8KjwtjNcSHzmhANO/0FSwzVDPSvP2ocEbjYjv5WMXXk0/sNdRgsb3gkQ1
> 5zuBS6j2n5n9SeUZ0IcM
> =Ki11
> -----END PGP SIGNATURE-----
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


From steve at pearwood.info  Tue Nov  3 19:15:54 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 4 Nov 2015 11:15:54 +1100
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <5638E7B7.2060608@mail.de>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
 <5636486C.6080401@mail.de> <56365A31.6020803@wh2.tu-dresden.de>
 <5638E7B7.2060608@mail.de>
Message-ID: <20151104001553.GM10946@ando.pearwood.info>

On Tue, Nov 03, 2015 at 05:58:31PM +0100, Sven R. Kunze wrote:
> Hey Lukas,
> 
> I agree both version (+ the n-part-partition-version of it) might be 
> very useful to have.
> 
> But where do they go into?
> 
> itertools might be the right place for the iter_partition; but where to 
> with the set_partition?

I think the right place for both of them is your own personal Python 
toolbox.


-- 
Steve

From abarnert at yahoo.com  Tue Nov  3 20:06:14 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 3 Nov 2015 17:06:14 -0800
Subject: [Python-ideas] Include partitioning in itertools
In-Reply-To: <5638E7B7.2060608@mail.de>
References: <DUB129-W21B85B93655B341464D845D62D0@phx.gbl>
 <5636486C.6080401@mail.de> <56365A31.6020803@wh2.tu-dresden.de>
 <5638E7B7.2060608@mail.de>
Message-ID: <A14ABAFC-F576-44EC-980A-41E1A60B51BC@yahoo.com>

On Nov 3, 2015, at 08:58, Sven R. Kunze <srkunze at mail.de> wrote:
> 
> Hey Lukas,
> 
> I agree both version (+ the n-part-partition-version of it) might be very useful to have.
> 
> But where do they go into?
> 
> itertools might be the right place for the iter_partition; but where to with the set_partition?

One more consideration: if you look at what's in itertools today, it isn't really everything you might want to do with and/or to produce iterators. Instead, it's (mostly) a collection of inner-loop building blocks that are accelerated with C code, and can be assembled together in pure Python to do almost everything you might want. The recipes give extensive examples of such things that are useful but don't need to be in the module itself. So, the question isn't just whether iter_partitions is like permutations in what it does, but whether it's like permutations in not having any more minimal building blocks?if there are such building blocks, it might be better to add those to itertools, and iter_partitions to the recipes.

Meanwhile, if you think there's a wider need for these functions than can be served by a blog post or ActiveState recipe, surely they should be on PyPI (whether or not they're in the stdlib?after all, if they're that useful, people will want them in 3.5 and 2.7). That also gives you a way to prove their usefulness?look at all these open source projects that use the library, and these comments from users saying "this really should be in the stdlib", and so on. At least for iter_partition, you might want to consider submitting it as an addition to more-itertools instead of posting your own library. He's already got a bunch of other stuff that's itertools-ish that may not belong in the stdlib (including all the recipes from the stdlib docs) but may still be useful enough for people to install the package.


From sjoerdjob at sjec.nl  Wed Nov  4 17:02:41 2015
From: sjoerdjob at sjec.nl (sjoerdjob at sjec.nl)
Date: Wed, 4 Nov 2015 23:02:41 +0100
Subject: [Python-ideas] Use `isinstance` check during exception handling
Message-ID: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>

Hi all,

TL;DR: Change exception checking logic from
    err.__class__ in exc.__mro__
to
    isinstance(err, exc)
Because it is more powerful, more consistent and allows for cleaner
code.


The reason I am proposing this change is that quite often, one has to
deal with libraries behaving 'badly' with regard to exception
granularity.

I will take the Python base library before PEP3151 [1] as an example. It
is not hard to imagine there are a lot of other cases of libraries that
behave with little granularity in the exception classes.

    try:
        some(operation)
    except OSError as err:
        if err.errno == errno.ENOENT:
            # handle file-not-exists
        elif err.errno == errno.EACCES:
            # handle permission stuff.
        else:
            raise

After PEP3151 one can now write:

    try:
        some(operation)
    except FileNotFoundError:
        # handle file-not-exists
    except PermissionError:
        # handle permission stuff.

Now, PEP3151 took over a year to be resolved, judging by the dates. If
we have to go through changes like this more often, one will have to
wait quite a while. Also, consider backwards compatibility issues. Code
using FileNotFoundError will only run on Python3.3+.

Thus I propose to have the exception handling code use the normal
`isinstance` machinery, as it is more powerful, and allows one to use
virtual base classes to check if the instance of the exception is
relevant to catch. An example:

    import errno

    # Fake `FileNotFoundError` in Python3.2
    class ENOENTCheck(type):
        def __instancecheck__(cls, inst):
            return isinstance(inst, EnvironmentError) and \
                   inst.errno == errno.ENOENT
    class FileNotFoundError(EnvironmentError, metaclass=ENOENTCheck):
        pass

    try:
        open("no-such-file")
    except Exception as e:
        print(isinstance(e, FileNotFoundError))

This would print `True`. But, using `FileNotFoundError` as a class to
catch will not work, because the `__mro__` is checked instead.

I proposed the change on the Python issue tracker [2] already, but it
was (rightfully, I think) decided that it needs some more discussion
because it will bring a (potentially huge) semantic change.

As an opening, here are some pros and cons I can think of:

Cons:

- *Code already written might break*: Highly unlikely. If you have
  written code that overrides `isinstance` logic, you'd more than likely
  not write code that depends on there being a difference in matching
  logic between the exception handling and normal `isinstance` calls.

- *Performance will go down*: Yes, instance-checking is probably a lot
  more expensive then subclass-checking. I haven't made the change and
  benchmarked it yet, but based on Martin Panter's comment [3], it might
  only cost a lot when actually checking against a virtual base class.
  If cost is limited to those cases, I think the cost is negligible.

- *Exception handling like this is just magic*: Yes, anything regarding
  the metaclasses might be considered as magic. But metaclasses also
  allow you to build great abstractions, like for instance the ORM
  included with Django. I think this is one of the cases where the
  benefits outweigh the cost.

Pros:

- *Cleaner code*: The same benefit PEP 3151 brought for the base library
  and OS errors can be extended to all the Python libraries that lack
  granularity.

- *Consistency*: Who would ever expect that you can not always catch an
  exception `foo` using class `Bar` when `isinstance(foo, Bar)`.

Even though I can currently think of more downsides than upsides, I do
think that the downsides are really minor, while the upsides are huge:
clean code and consistency.

Also, I set up a proof-of-concept PEP3151 compatibility layer for
Python2.7 that sort-of fakes this feature. It fakes it by checking the
instance `sys.exc_info()[1]` in the `__subclasscheck__`. For the proof
of concept see [4]. I only found out during porting to Python3 that it
does not actually work, because of the direct checking of the `__mro__`,
but that's hopefully going to be fixed soon-ish [5].

I'd be interested to hear your thoughts about this. Any more pros and
cons would be welcome, as well as your judgement as to whether the
breaking of backwards compatibility is potentially a bad idea.

Kind regards,
Sjoerd Job Postmus

[1] PEP 3151 -- Reworking the OS and IO exception hierarchy
    https://www.python.org/dev/peps/pep-3151/
[2] Call `isinstance` instead of `issubclass` during exception handling
    http://bugs.python.org/issue25537
[3] http://bugs.python.org/issue25537#msg253963
[4] https://github.com/sjoerdjob/exhacktion/
[5] Catching virtual subclasses in except clauses
    http://bugs.python.org/issue12029

From tjreedy at udel.edu  Wed Nov  4 20:39:40 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 4 Nov 2015 20:39:40 -0500
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
Message-ID: <n1ec0t$1s7$1@ger.gmane.org>



On 11/4/2015 5:02 PM, sjoerdjob at sjec.nl wrote:
> TL;DR: Change exception checking logic  from
>      err.__class__ in exc.__mro__

This is backwards. err.__class__ must be a (true) subclass, not a 
superclass, of exc, where err = exception instance, exc = exception class.

There are two checks in the exception process.  First, for 'raise err', 
is that err is a instance of StopIteration, which is to say, 
err.__class__ is a subclass of StopIteration.  To avoid the fakery of 
__instancecheck__ and __subclasscheck__, I presume this is implemented 
as something like one of
   StopIteration in err.__class__.__mro__
   StopIteration in type(err).__mro__
(There was a recent pydev discussion about when and why these may diverge.)

The second, for 'except exc', is that exc is a superclass (other than 
object) of type(err).  I presume this is implemented as something like
   exc in type(err)[:-1]  # exclude object class
This assume that err has already passed the first check above.

The `try` doc says "An object is compatible with an exception if it is 
the class or a base class of the exception object".  This text fails to 
exclude the object class as a compatible object.  CPython code rejects 
'except object:' with "TypeError: catching classes that do not inherit 
from BaseException is not allowed"

> to
>      isinstance(err, exc)
> Because it is more powerful,

?? This predicate is True more often than the current one because it a) 
does not exclude exc = object

 >>> isinstance(Exception(), object)
True

and b) accesses exc.__instancecheck__ (which is part of your point). 
This makes it a weaker predicate.

I don't have strong opinions at the moment about the proposal itself, 
but semantic changes in core python syntax need a PEP (once you think 
there is *some* support).

-- 
Terry Jan Reedy


From mike at selik.org  Wed Nov  4 21:42:58 2015
From: mike at selik.org (Michael Selik)
Date: Thu, 05 Nov 2015 02:42:58 +0000
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <n1ec0t$1s7$1@ger.gmane.org>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <n1ec0t$1s7$1@ger.gmane.org>
Message-ID: <CADqi795fqfhaibsn_WfqDvoO6R2HZskChHbwOt71=D0CWG5apw@mail.gmail.com>

Hi Sjoerd,

I'm not clear on what problem this suggestion is solving. PEP 3151 was
solving the problem of existing code failing to be specific with IOError
and OSError error numbers. Can you provide some less abstract snippets as
examples, including surrounding context or a link to a project that has the
problem?

If I saw `except Exception as e` followed by `isinstance(e, SomeError)` I
would be very confused as to why `except SomeError` was not better. Is this
simply for backporting FileNotFoundError?

Michael

On Wed, Nov 4, 2015 at 8:39 PM Terry Reedy <tjreedy at udel.edu> wrote:

>
>
> On 11/4/2015 5:02 PM, sjoerdjob at sjec.nl wrote:
> > TL;DR: Change exception checking logic  from
> >      err.__class__ in exc.__mro__
>
> This is backwards. err.__class__ must be a (true) subclass, not a
> superclass, of exc, where err = exception instance, exc = exception class.
>
> There are two checks in the exception process.  First, for 'raise err',
> is that err is a instance of StopIteration, which is to say,
> err.__class__ is a subclass of StopIteration.  To avoid the fakery of
> __instancecheck__ and __subclasscheck__, I presume this is implemented
> as something like one of
>    StopIteration in err.__class__.__mro__
>    StopIteration in type(err).__mro__
> (There was a recent pydev discussion about when and why these may diverge.)
>
> The second, for 'except exc', is that exc is a superclass (other than
> object) of type(err).  I presume this is implemented as something like
>    exc in type(err)[:-1]  # exclude object class
> This assume that err has already passed the first check above.
>
> The `try` doc says "An object is compatible with an exception if it is
> the class or a base class of the exception object".  This text fails to
> exclude the object class as a compatible object.  CPython code rejects
> 'except object:' with "TypeError: catching classes that do not inherit
> from BaseException is not allowed"
>
> > to
> >      isinstance(err, exc)
> > Because it is more powerful,
>
> ?? This predicate is True more often than the current one because it a)
> does not exclude exc = object
>
>  >>> isinstance(Exception(), object)
> True
>
> and b) accesses exc.__instancecheck__ (which is part of your point).
> This makes it a weaker predicate.
>
> I don't have strong opinions at the moment about the proposal itself,
> but semantic changes in core python syntax need a PEP (once you think
> there is *some* support).
>
> --
> Terry Jan Reedy
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151105/f58f3486/attachment.html>

From mike at selik.org  Wed Nov  4 22:01:30 2015
From: mike at selik.org (Michael Selik)
Date: Thu, 05 Nov 2015 03:01:30 +0000
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <CADqi795fqfhaibsn_WfqDvoO6R2HZskChHbwOt71=D0CWG5apw@mail.gmail.com>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <n1ec0t$1s7$1@ger.gmane.org>
 <CADqi795fqfhaibsn_WfqDvoO6R2HZskChHbwOt71=D0CWG5apw@mail.gmail.com>
Message-ID: <CADqi796-SS-oEC-0p-voQd9USAu1bBE7J4tH_wstLmzNwPzRpA@mail.gmail.com>

After reading https://github.com/sjoerdjob/exhacktion/ I have a better
understanding of what you want to accomplish. However, I'm still not clear
on where the pain is coming from. If you find yourself writing an if in an
except many times as you described:

    try:
        os.chown("README.md", 100, 100)
    except EnvironmentError as e:
        if e.errno == errno.ENOENT:
            pass
        else:
            raise


Would using a context manager solve your problem?

    class IgnoreNotFound:
        def __enter__(self):
            return self
        def __exit__(self, etyp, e, etb):
            if issubclass(etyp, EnvironmentError) and e.message ==
errno.ENOENT:
                return True

    with IgnoreNotFound:
        os.chown("README.md", 100, 100)


On Wed, Nov 4, 2015 at 9:42 PM Michael Selik <mike at selik.org> wrote:

> Hi Sjoerd,
>
> I'm not clear on what problem this suggestion is solving. PEP 3151 was
> solving the problem of existing code failing to be specific with IOError
> and OSError error numbers. Can you provide some less abstract snippets as
> examples, including surrounding context or a link to a project that has the
> problem?
>
> If I saw `except Exception as e` followed by `isinstance(e, SomeError)` I
> would be very confused as to why `except SomeError` was not better. Is this
> simply for backporting FileNotFoundError?
>
> Michael
>
> On Wed, Nov 4, 2015 at 8:39 PM Terry Reedy <tjreedy at udel.edu> wrote:
>
>>
>>
>> On 11/4/2015 5:02 PM, sjoerdjob at sjec.nl wrote:
>> > TL;DR: Change exception checking logic  from
>> >      err.__class__ in exc.__mro__
>>
>> This is backwards. err.__class__ must be a (true) subclass, not a
>> superclass, of exc, where err = exception instance, exc = exception class.
>>
>> There are two checks in the exception process.  First, for 'raise err',
>> is that err is a instance of StopIteration, which is to say,
>> err.__class__ is a subclass of StopIteration.  To avoid the fakery of
>> __instancecheck__ and __subclasscheck__, I presume this is implemented
>> as something like one of
>>    StopIteration in err.__class__.__mro__
>>    StopIteration in type(err).__mro__
>> (There was a recent pydev discussion about when and why these may
>> diverge.)
>>
>> The second, for 'except exc', is that exc is a superclass (other than
>> object) of type(err).  I presume this is implemented as something like
>>    exc in type(err)[:-1]  # exclude object class
>> This assume that err has already passed the first check above.
>>
>> The `try` doc says "An object is compatible with an exception if it is
>> the class or a base class of the exception object".  This text fails to
>> exclude the object class as a compatible object.  CPython code rejects
>> 'except object:' with "TypeError: catching classes that do not inherit
>> from BaseException is not allowed"
>>
>> > to
>> >      isinstance(err, exc)
>> > Because it is more powerful,
>>
>> ?? This predicate is True more often than the current one because it a)
>> does not exclude exc = object
>>
>>  >>> isinstance(Exception(), object)
>> True
>>
>> and b) accesses exc.__instancecheck__ (which is part of your point).
>> This makes it a weaker predicate.
>>
>> I don't have strong opinions at the moment about the proposal itself,
>> but semantic changes in core python syntax need a PEP (once you think
>> there is *some* support).
>>
>> --
>> Terry Jan Reedy
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151105/fd8488bf/attachment-0001.html>

From mike at selik.org  Wed Nov  4 22:35:56 2015
From: mike at selik.org (Michael Selik)
Date: Thu, 05 Nov 2015 03:35:56 +0000
Subject: [Python-ideas] [Python-Dev] If you shadow a module in the standard
 library that IDLE depends on, bad things happen
Message-ID: <CADqi796qW=bPCwC68kpFiucWttjWOvejLMsAtLgtg+FjcMrQOw@mail.gmail.com>

Placing current working directory at the beginning of sys.path or the end
of sys.path is a tradeoff between two debugging tasks given the error
message, `AttributeError: 'module' object has no attribute 'blah'`. The
traceback tells you the name of the module. Which would you prefer: (1)
needing to notice that you made a module of the same name, or (2) needing
to notice that somewhere in the cryptic locations on sys.path there's a
module of the same name?

I'll pick option 1. Sure, it's tricky the first time it happens, but I've
seen complete beginners figure it out after a few minutes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151105/c04562a8/attachment.html>

From mike at selik.org  Wed Nov  4 22:45:01 2015
From: mike at selik.org (Michael Selik)
Date: Thu, 05 Nov 2015 03:45:01 +0000
Subject: [Python-ideas] Consider making enumerate a sequence if its
 argument is a sequence
Message-ID: <CADqi795xb7b0WEX9p9eoRF+o2o3NEs69YZ5w8ynQSoz3nKm5vA@mail.gmail.com>

On 02.10.2015 05:49, Steven D'Aprano wrote:
> At most, some have suggested that we don't have a good
> word for those iterables which are not iterators.

Isn't that a "sequence"? The collections module defines a Sequence as
having a __getitem__ and __len__. Is a "non-iterator iterable" something
different?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151105/60b19d47/attachment.html>

From sjoerdjob at sjec.nl  Thu Nov  5 01:00:12 2015
From: sjoerdjob at sjec.nl (sjoerdjob at sjec.nl)
Date: Thu, 5 Nov 2015 07:00:12 +0100
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <n1ec0t$1s7$1@ger.gmane.org>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <n1ec0t$1s7$1@ger.gmane.org>
Message-ID: <744456a7ee014b3774f4bccbf673cacd.squirrel@mail.sjec.nl>

> On 11/4/2015 5:02 PM, sjoerdjob at sjec.nl wrote:
>> TL;DR: Change exception checking logic  from
>>      err.__class__ in exc.__mro__
>
> This is backwards. err.__class__ must be a (true) subclass, not a
> superclass, of exc, where err = exception instance, exc = exception class.

Ah, a minor glitch when writing the e-mail. My apologies. It should be
    exc in err.__class__.__mro__

> There are two checks in the exception process.  First, for 'raise err',
> is that err is a instance of StopIteration, which is to say,
> err.__class__ is a subclass of StopIteration.  To avoid the fakery of
> __instancecheck__ and __subclasscheck__, I presume this is implemented
> as something like one of
>    StopIteration in err.__class__.__mro__
>    StopIteration in type(err).__mro__
> (There was a recent pydev discussion about when and why these may
> diverge.)

I can't actually find the logic which handles the `StopIteration`,
unless you are talking specifically about the exception handling in
for-loops and everything else that relates to iterables. (The code I'm
looking at is Python/errors.c:PyErr_GivenExceptionMatches).

> The second, for 'except exc', is that exc is a superclass (other than
> object) of type(err).  I presume this is implemented as something like
>    exc in type(err)[:-1]  # exclude object class
> This assume that err has already passed the first check above.

>From what I can find (again: errors.c), the current logic is:

  (note: `err` is what is raised, `exc` is what's matched against)
  * if `exc` is a tuple, recurse over the elements of the tuple.
  * if the `err` is an instance, replace it with its class
  * if both `err` and `exc` are subclasses of exception:
        check PyType_IsSubtype(err, exc)
    which basically is the same as (see Objects/typeobject.c)
        exc in err.__mro__
    note: not err.__mro__[:-1].

> The `try` doc says "An object is compatible with an exception if it is
> the class or a base class of the exception object".  This text fails to
> exclude the object class as a compatible object.  CPython code rejects
> 'except object:' with "TypeError: catching classes that do not inherit
> from BaseException is not allowed"
>
>> to
>>      isinstance(err, exc)
>> Because it is more powerful,
>
> ?? This predicate is True more often than the current one because it a)
> does not exclude exc = object

My idea was basically to still demand that both `err` and `exc` must
sub-class `BaseException`, but that only the check for
`PyType_IsSubtype` would be replaced with `PyObject_IsInstance`.

>  >>> isinstance(Exception(), object)
> True
>
> and b) accesses exc.__instancecheck__ (which is part of your point).
> This makes it a weaker predicate.

Yes, the predicate is weaker. And that gives us more power.

> I don't have strong opinions at the moment about the proposal itself,
> but semantic changes in core python syntax need a PEP (once you think
> there is *some* support).

It's not a syntax-change, but a semantics-change. Still I would not be
surprised if it indeed does need a PEP, but there will only be a point
in doing that when there is *some* support. Before writing that PEP, I
was checking here to see if I could find some support, or if it the pros
don't outweigh the cons for the 'masses'.

> --
> Terry Jan Reedy

Regards,
Sjoerd Job Postmus

From storchaka at gmail.com  Thu Nov  5 03:05:55 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 5 Nov 2015 10:05:55 +0200
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
Message-ID: <n1f2l3$fr2$1@ger.gmane.org>

On 05.11.15 00:02, sjoerdjob at sjec.nl wrote:
> Now, PEP3151 took over a year to be resolved, judging by the dates. If
> we have to go through changes like this more often, one will have to
> wait quite a while. Also, consider backwards compatibility issues. Code
> using FileNotFoundError will only run on Python3.3+.
>
> Thus I propose to have the exception handling code use the normal
> `isinstance` machinery, as it is more powerful, and allows one to use
> virtual base classes to check if the instance of the exception is
> relevant to catch. An example:
>
>      import errno
>
>      # Fake `FileNotFoundError` in Python3.2

Do you propose to add a new feature in 3.2?



From sjoerdjob at sjec.nl  Thu Nov  5 03:13:49 2015
From: sjoerdjob at sjec.nl (Sjoerd Job Postmus)
Date: Thu, 5 Nov 2015 09:13:49 +0100
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <n1f2l3$fr2$1@ger.gmane.org>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <n1f2l3$fr2$1@ger.gmane.org>
Message-ID: <7D22F391-69AF-46C1-9FCD-DFE3D1B01D13@sjec.nl>



> On 5 Nov 2015, at 09:05, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 
>> On 05.11.15 00:02, sjoerdjob at sjec.nl wrote:
>> Now, PEP3151 took over a year to be resolved, judging by the dates. If
>> we have to go through changes like this more often, one will have to
>> wait quite a while. Also, consider backwards compatibility issues. Code
>> using FileNotFoundError will only run on Python3.3+.
>> 
>> Thus I propose to have the exception handling code use the normal
>> `isinstance` machinery, as it is more powerful, and allows one to use
>> virtual base classes to check if the instance of the exception is
>> relevant to catch. An example:
>> 
>>     import errno
>> 
>>     # Fake `FileNotFoundError` in Python3.2
> 
> Do you propose to add a new feature in 3.2?

Definitely not! PEP3151 was just meant as an example of an improvement of exception granularity taking a long time.

> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From mike at selik.org  Thu Nov  5 03:18:12 2015
From: mike at selik.org (Michael Selik)
Date: Thu, 05 Nov 2015 08:18:12 +0000
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <7D22F391-69AF-46C1-9FCD-DFE3D1B01D13@sjec.nl>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <n1f2l3$fr2$1@ger.gmane.org> <7D22F391-69AF-46C1-9FCD-DFE3D1B01D13@sjec.nl>
Message-ID: <CADqi797TSqOKpVb6z5qzRSxrY9HYnbAQ-6GLW0G38hrqikTJzg@mail.gmail.com>

I think with good reason. It's impossible to enumerate all possible ways a
program can error. Eventually one must fall back to inspecting various
error payload information and perhaps even other state. Increasing error
granularity is a slippery slope. The shift should be driven by evidence of
annoying boilerplate code written in the standard library and major
community projects.

On Thu, Nov 5, 2015 at 3:13 AM Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:

> Definitely not! PEP3151 was just meant as an example of an improvement of
> exception granularity taking a long time.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151105/b96728fe/attachment-0001.html>

From rosuav at gmail.com  Thu Nov  5 03:21:17 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 5 Nov 2015 19:21:17 +1100
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <n1f2l3$fr2$1@ger.gmane.org>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <n1f2l3$fr2$1@ger.gmane.org>
Message-ID: <CAPTjJmqeGgsEbfQF=wGSCwpVyp=eWWWpujMHO4E_vU5AqrQSyg@mail.gmail.com>

On Thu, Nov 5, 2015 at 7:05 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 05.11.15 00:02, sjoerdjob at sjec.nl wrote:
>>
>> Now, PEP3151 took over a year to be resolved, judging by the dates. If
>> we have to go through changes like this more often, one will have to
>> wait quite a while. Also, consider backwards compatibility issues. Code
>> using FileNotFoundError will only run on Python3.3+.
>>
>> Thus I propose to have the exception handling code use the normal
>> `isinstance` machinery, as it is more powerful, and allows one to use
>> virtual base classes to check if the instance of the exception is
>> relevant to catch. An example:
>>
>>      import errno
>>
>>      # Fake `FileNotFoundError` in Python3.2
>
>
> Do you propose to add a new feature in 3.2?

I think the point is to future-proof Python against further such
incidents, and to allow some measure of control even over third-party
libraries. For instance, psycopg2 dumps ProgrammingError on you for
lots of all SQL errors, but has a few special cases like
IntegrityError; if there's some special case that isn't handled by
psycopg2 itself, you have no recourse but to catch something too broad
and then manually filter.

I do like the idea of being able to refactor exception specialization code:

try:
    ...
except BroadError as e:
    if is_narrower_error(e):
        # handle NarrowerError
    else:
        raise

but I don't know whether it's worth complicating the core exception
handling system with such a rare case. Also, what happens if you have
a bug in your __instancecheck__ method? It's going to be *extremely*
hard to track down. How often does this sort of except-level
specialization come up?

ChrisA

From abarnert at yahoo.com  Thu Nov  5 03:43:43 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 5 Nov 2015 00:43:43 -0800
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
Message-ID: <A8D09FB9-19BE-4453-B2E8-6077A8000FF3@yahoo.com>

On Nov 4, 2015, at 14:02, sjoerdjob at sjec.nl wrote:
> 
> Hi all,
> 
> TL;DR: Change exception checking logic from
>    err.__class__ in exc.__mro__
> to
>    isinstance(err, exc)
> Because it is more powerful, more consistent and allows for cleaner
> code.

Correct me if I'm wrong, but doesn't Python allow you to raise a class and handle it without ever creating an instance? If an instance is needed (e.g., because the exception is caught by a handler with an as clause), one is created by calling exc(), but if it's not needed, that doesn't happen.

Assuming I'm right, your proposal would mean the instance is _always_ needed, so if you raise a type, exc() is always called. So, for example:

    class MyException(Exception):
        def __init__(self, thingy, *a, **kw):
            super().__init__(*a, **kw)
            self.thingy = thingy

    try:
        raise MyException
    except MyException:
        print('whoops')

Instead of printing "whoops", it would now (I think) abort with a TypeError about trying to call MyException with the wrong number of arguments.

Plus, there's almost certainly code where someone skips creating the instance as an optimization?probably it's usually a useless premature one, but maybe someone has tested and it makes a difference in a real program.

One obvious possibility is: if exc is itself a subclass of BaseException, you use the existing subclass check; if not, instead of using the subclass check on its type, you use the new instance check. But you'd have to think that through to make sure it makes sense.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151105/0c7037cc/attachment.html>

From abarnert at yahoo.com  Thu Nov  5 04:17:24 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 5 Nov 2015 01:17:24 -0800
Subject: [Python-ideas] Consider making enumerate a sequence if its
 argument is a sequence
In-Reply-To: <CADqi795xb7b0WEX9p9eoRF+o2o3NEs69YZ5w8ynQSoz3nKm5vA@mail.gmail.com>
References: <CADqi795xb7b0WEX9p9eoRF+o2o3NEs69YZ5w8ynQSoz3nKm5vA@mail.gmail.com>
Message-ID: <510DA2AB-4BB6-4B36-9ADD-2FC48B9CE1EC@yahoo.com>

On Nov 4, 2015, at 19:45, Michael Selik <mike at selik.org> wrote:
> 
> On 02.10.2015 05:49, Steven D'Aprano wrote:
> > At most, some have suggested that we don't have a good
> > word for those iterables which are not iterators. 
> 
> Isn't that a "sequence"? The collections module defines a Sequence as having a __getitem__ and __len__. Is a "non-iterator iterable" something different?

A mapping isn't a sequence (even though it has __getitem__ and __len__, and also has the same supertypes, which add additional restrictions that you forgot like __contains__). Neither is a set, or any of the kinds of mapping views, or any intentionally-not-quite-sequence (like range before 3.2, or, IIRC, blist.sortedlist), or an iterable that, each time it's iterated, gives you 20 new values in [0, 1) that add up to 1. But they're all non-iterator iterables.

In fact, there are a number of concepts that largely overlap, but don't completely, including:

* Iterables that aren't iterators
* Iterables that don't return self from __iter__
* Iterables that always return a new iterator
* Iterables that can repeatably produce a "correct" sequence of values (as long as they aren't mutated in between iterations)
* Iterables that can repeatably produce equal values (as long as they aren't mutated)
* Iterables that can repeatedly produce identical values (as long as they aren't mutated)

There are also variations of the last three that represent some data that can be "externally mutated" (e.g., a list of all broadcasting SMB servers on the local subnet will always produce equal values unless a server goes up or down between iterations?or, more simply, any view can change values if the thing it's a view of is mutated, without the view itself being mutated).

Anyway, all of the above are true for sequences, mappings, sets, mapping views, and most not-quite-sequences, but not for the random example, or the SMB example.

I don't think we need official names for all of these things, just one?as long as it's clear which one, and exactly what it means, I think that will cover well over 90% of all reasonable uses.

I believe that earlier in the thread, Terry suggested "collection" to mean the last definition, which seems fine at first glance, so let's assume that. If we added that to the documentation (and fixed all the places that sloppily talk about sequences when they don't really mean sequences, etc.), and ideally added an ABC and a static type to match, then code could very easily explain that it produces or requires a collection. Most code that needs two passes over an iterable can just say it requires a collection. The fact that it might also work with some sort-of-repeatable-but-not-quite-collection types is fine (and we don't have to say that such types are "defective" or anything, as long as the documentation makes it clear that there are some very rare iterables that are neither iterators nor collections, and what that means).

Of course Collection defined this way can't be an implicit ABC, because it doesn't have any interface beyond that of Iterator. But then Sequence and Mapping can't be implicit either, because they have the same interface as each other. In fact, that's the main reason we need ABCs in the first place?for cases where structural/duck typing can't cover the intended semantics.

From vgr255 at live.ca  Thu Nov  5 08:08:07 2015
From: vgr255 at live.ca (Emanuel Barry)
Date: Thu, 5 Nov 2015 08:08:07 -0500
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <A8D09FB9-19BE-4453-B2E8-6077A8000FF3@yahoo.com>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>,
 <A8D09FB9-19BE-4453-B2E8-6077A8000FF3@yahoo.com>
Message-ID: <BLU172-W454E33B542AC916E6C34D491290@phx.gbl>

I think the main concern here is not "Would that benefit the corner cases?" - it surely would. What we should be worrying is, "Will this impede the common cases?". I think that it should be possible, for all the (very) common cases, to use what CPython already uses internally. If it matches, I don't see why we'd even bother going through the __subclasscheck__ and __instancecheck__ hooks. IMO, it should only use those (whichever ends up being the prefered choice - I have no strong preference for now either way) if the standard exception lookup fails, and possibly including the fact that BaseException doesn't need to be a direct subclass (e.g. virtual subclasses).
Just my 2 cents.
-Emanuel 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151105/c9b7ebea/attachment.html>

From sjoerdjob at sjec.nl  Thu Nov  5 10:00:34 2015
From: sjoerdjob at sjec.nl (sjoerdjob at sjec.nl)
Date: Thu, 5 Nov 2015 16:00:34 +0100
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <A8D09FB9-19BE-4453-B2E8-6077A8000FF3@yahoo.com>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <A8D09FB9-19BE-4453-B2E8-6077A8000FF3@yahoo.com>
Message-ID: <afbc97894d853b3acf5604263b650627.squirrel@mail.sjec.nl>

> On Nov 4, 2015, at 14:02, sjoerdjob at sjec.nl wrote:
>>
>> Hi all,
>>
>> TL;DR: Change exception checking logic from
>>    err.__class__ in exc.__mro__
>> to
>>    isinstance(err, exc)
>> Because it is more powerful, more consistent and allows for cleaner
>> code.
>
> Correct me if I'm wrong, but doesn't Python allow you to raise a class and
> handle it without ever creating an instance? If an instance is needed
> (e.g., because the exception is caught by a handler with an as clause),
> one is created by calling exc(), but if it's not needed, that doesn't
> happen.
>
> Assuming I'm right, your proposal would mean the instance is _always_
> needed, so if you raise a type, exc() is always called. So, for example:
>
>     class MyException(Exception):
>         def __init__(self, thingy, *a, **kw):
>             super().__init__(*a, **kw)
>             self.thingy = thingy
>
>     try:
>         raise MyException
>     except MyException:
>         print('whoops')
>
> Instead of printing "whoops", it would now (I think) abort with a
> TypeError about trying to call MyException with the wrong number of
> arguments.

Yes, the exception is instantiated as soon as it is raised (not when it's
matched). This happens in Python/ceval.c. So you actually have to catch
`TypeError` in this case.

> Plus, there's almost certainly code where someone skips creating the
> instance as an optimization???probably it's usually a useless premature
> one, but maybe someone has tested and it makes a difference in a real
> program.

It will get instantiated anyway.

> One obvious possibility is: if exc is itself a subclass of BaseException,
> you use the existing subclass check; if not, instead of using the subclass
> check on its type, you use the new instance check. But you'd have to think
> that through to make sure it makes sense.


From mike at selik.org  Thu Nov  5 10:39:33 2015
From: mike at selik.org (Michael Selik)
Date: Thu, 05 Nov 2015 15:39:33 +0000
Subject: [Python-ideas] Consider making enumerate a sequence if its
 argument is a sequence
In-Reply-To: <510DA2AB-4BB6-4B36-9ADD-2FC48B9CE1EC@yahoo.com>
References: <CADqi795xb7b0WEX9p9eoRF+o2o3NEs69YZ5w8ynQSoz3nKm5vA@mail.gmail.com>
 <510DA2AB-4BB6-4B36-9ADD-2FC48B9CE1EC@yahoo.com>
Message-ID: <CADqi796hZQuaNtFapUNDpw-2uaSs_T8zquPWM87XQW769G6+TQ@mail.gmail.com>

I'd like to see several examples where a new term would improve the reading
of the documentation. So far I agree with Akira Li and Brandon Barnwell.
The current usage of "iterable" provides the correct level of specificity.
I do not think `list(finite_iterable)` improves the function tooltip. I
prefer `list(iterable)`. There's no need to cover all edge cases in the
tooltip.

Two tasks that require viewing more than one element simultaneously are
sorting and finding a median. The functions `sorted(iterable)` and
`statistics.median(data)` have reasonable documentation without mentioning
the fact that inputting an infinite iterator would be a bad choice.

On Thu, Nov 5, 2015 at 4:17 AM Andrew Barnert <abarnert at yahoo.com> wrote:

> On Nov 4, 2015, at 19:45, Michael Selik <mike at selik.org> wrote:
> >
> > On 02.10.2015 05:49, Steven D'Aprano wrote:
> > > At most, some have suggested that we don't have a good
> > > word for those iterables which are not iterators.
> >
> > Isn't that a "sequence"? The collections module defines a Sequence as
> having a __getitem__ and __len__. Is a "non-iterator iterable" something
> different?
>
> A mapping isn't a sequence (even though it has __getitem__ and __len__,
> and also has the same supertypes, which add additional restrictions that
> you forgot like __contains__). Neither is a set, or any of the kinds of
> mapping views, or any intentionally-not-quite-sequence (like range before
> 3.2, or, IIRC, blist.sortedlist), or an iterable that, each time it's
> iterated, gives you 20 new values in [0, 1) that add up to 1. But they're
> all non-iterator iterables.
>
> In fact, there are a number of concepts that largely overlap, but don't
> completely, including:
>
> * Iterables that aren't iterators
> * Iterables that don't return self from __iter__
> * Iterables that always return a new iterator
> * Iterables that can repeatably produce a "correct" sequence of values (as
> long as they aren't mutated in between iterations)
> * Iterables that can repeatably produce equal values (as long as they
> aren't mutated)
> * Iterables that can repeatedly produce identical values (as long as they
> aren't mutated)
>
> There are also variations of the last three that represent some data that
> can be "externally mutated" (e.g., a list of all broadcasting SMB servers
> on the local subnet will always produce equal values unless a server goes
> up or down between iterations?or, more simply, any view can change values
> if the thing it's a view of is mutated, without the view itself being
> mutated).
>
> Anyway, all of the above are true for sequences, mappings, sets, mapping
> views, and most not-quite-sequences, but not for the random example, or the
> SMB example.
>
> I don't think we need official names for all of these things, just one?as
> long as it's clear which one, and exactly what it means, I think that will
> cover well over 90% of all reasonable uses.
>
> I believe that earlier in the thread, Terry suggested "collection" to mean
> the last definition, which seems fine at first glance, so let's assume
> that. If we added that to the documentation (and fixed all the places that
> sloppily talk about sequences when they don't really mean sequences, etc.),
> and ideally added an ABC and a static type to match, then code could very
> easily explain that it produces or requires a collection. Most code that
> needs two passes over an iterable can just say it requires a collection.
> The fact that it might also work with some
> sort-of-repeatable-but-not-quite-collection types is fine (and we don't
> have to say that such types are "defective" or anything, as long as the
> documentation makes it clear that there are some very rare iterables that
> are neither iterators nor collections, and what that means).
>
> Of course Collection defined this way can't be an implicit ABC, because it
> doesn't have any interface beyond that of Iterator. But then Sequence and
> Mapping can't be implicit either, because they have the same interface as
> each other. In fact, that's the main reason we need ABCs in the first
> place?for cases where structural/duck typing can't cover the intended
> semantics.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151105/99c5095a/attachment.html>

From random832 at fastmail.com  Thu Nov  5 10:45:32 2015
From: random832 at fastmail.com (Random832)
Date: Thu, 05 Nov 2015 10:45:32 -0500
Subject: [Python-ideas] Use `isinstance` check during exception handling
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <n1ec0t$1s7$1@ger.gmane.org>
 <CADqi795fqfhaibsn_WfqDvoO6R2HZskChHbwOt71=D0CWG5apw@mail.gmail.com>
 <CADqi796-SS-oEC-0p-voQd9USAu1bBE7J4tH_wstLmzNwPzRpA@mail.gmail.com>
Message-ID: <87lhactppf.fsf@fastmail.com>

Michael Selik <mike at selik.org> writes:
> After reading https://github.com/sjoerdjob/exhacktion/ I have a better
> understanding of what you want to accomplish. However, I'm still not
> clear on where the pain is coming from. If you find yourself writing
> an if in an except many times as you described:
>
> try:
>     os.chown("README.md", 100, 100)
> except EnvironmentError as e:
>     if e.errno == errno.ENOENT:
>         pass
>     else:
>         raise

You know, it occurs to me, there are some languages that support this as
a high-level feature. I.e. in Visual Basic

Try
    [Statements]
Catch ex As Exception When [Boolean-valued expression]
    [Statements]
End Try

This is in fact the *only* way that MS Windows "Structured Exception
Handling" exceptions work in C:

__try { ... }
__except([BOOL-valued expression]) { ... }

Both of these have the effect that the exception "was never caught" if
it returns false.


From random832 at fastmail.com  Thu Nov  5 10:58:55 2015
From: random832 at fastmail.com (Random832)
Date: Thu, 05 Nov 2015 10:58:55 -0500
Subject: [Python-ideas] Use `isinstance` check during exception handling
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <n1f2l3$fr2$1@ger.gmane.org>
 <CAPTjJmqeGgsEbfQF=wGSCwpVyp=eWWWpujMHO4E_vU5AqrQSyg@mail.gmail.com>
Message-ID: <87h9l0tp34.fsf@fastmail.com>

Chris Angelico <rosuav at gmail.com> writes:
> I do like the idea of being able to refactor exception specialization code:
>
> try:
>     ...
> except BroadError as e:
>     if is_narrower_error(e):
>         # handle NarrowerError
>     else:
>         raise
>
> but I don't know whether it's worth complicating the core exception
> handling system with such a rare case.

The syntax could be this:
except BroadError as e if is_narrower_error(e):

I think the bytecode *already* evaluates a boolean expression
which today happens to always be a comparison of the exception's
type. Notably, the exception type itself can apparently be an
arbitrary expression, which (I was mildly surprised to discover)
has access to sys.exc_info.

class NeverThrown(Exception): pass

def exfilter():
   etype, e = sys.exc_info()[:2]
   if is_narrower_error(e): return etype
   else: return NeverThrown

try:
   ...
except exfilter() as e:
   ...

This is, of course, a *profoundly* ugly way to do this.


From sjoerdjob at sjec.nl  Thu Nov  5 11:08:51 2015
From: sjoerdjob at sjec.nl (Sjoerd Job Postmus)
Date: Thu, 5 Nov 2015 17:08:51 +0100
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <87h9l0tp34.fsf@fastmail.com>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <n1f2l3$fr2$1@ger.gmane.org>
 <CAPTjJmqeGgsEbfQF=wGSCwpVyp=eWWWpujMHO4E_vU5AqrQSyg@mail.gmail.com>
 <87h9l0tp34.fsf@fastmail.com>
Message-ID: <81F19DD4-F43A-4C8F-9056-58124F327ACE@sjec.nl>

Yes. This way is really ugly.

Also, wrt to the `if` statement: that only works nicely if you don't really want support for the tuple-case. How would you model different conditions for different elements of the tuple?

> On 5 Nov 2015, at 16:58, Random832 <random832 at fastmail.com> wrote:
> 
> Chris Angelico <rosuav at gmail.com> writes:
>> I do like the idea of being able to refactor exception specialization code:
>> 
>> try:
>>    ...
>> except BroadError as e:
>>    if is_narrower_error(e):
>>        # handle NarrowerError
>>    else:
>>        raise
>> 
>> but I don't know whether it's worth complicating the core exception
>> handling system with such a rare case.
> 
> The syntax could be this:
> except BroadError as e if is_narrower_error(e):
> 
> I think the bytecode *already* evaluates a boolean expression
> which today happens to always be a comparison of the exception's
> type. Notably, the exception type itself can apparently be an
> arbitrary expression, which (I was mildly surprised to discover)
> has access to sys.exc_info.
> 
> class NeverThrown(Exception): pass
> 
> def exfilter():
>   etype, e = sys.exc_info()[:2]
>   if is_narrower_error(e): return etype
>   else: return NeverThrown
> 
> try:
>   ...
> except exfilter() as e:
>   ...
> 
> This is, of course, a *profoundly* ugly way to do this.
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From ian.g.kelly at gmail.com  Thu Nov  5 11:41:44 2015
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Thu, 5 Nov 2015 09:41:44 -0700
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <87h9l0tp34.fsf@fastmail.com>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <n1f2l3$fr2$1@ger.gmane.org>
 <CAPTjJmqeGgsEbfQF=wGSCwpVyp=eWWWpujMHO4E_vU5AqrQSyg@mail.gmail.com>
 <87h9l0tp34.fsf@fastmail.com>
Message-ID: <CALwzidm_qk60-b0ZEQus3HtgzM9ZT0kiv2sgidXQUP6uNsfzcQ@mail.gmail.com>

On Thu, Nov 5, 2015 at 8:58 AM, Random832 <random832 at fastmail.com> wrote:
> The syntax could be this:
> except BroadError as e if is_narrower_error(e):
>
> I think the bytecode *already* evaluates a boolean expression
> which today happens to always be a comparison of the exception's
> type. Notably, the exception type itself can apparently be an
> arbitrary expression, which (I was mildly surprised to discover)
> has access to sys.exc_info.
>
> class NeverThrown(Exception): pass
>
> def exfilter():
>    etype, e = sys.exc_info()[:2]
>    if is_narrower_error(e): return etype
>    else: return NeverThrown
>
> try:
>    ...
> except exfilter() as e:
>    ...
>
> This is, of course, a *profoundly* ugly way to do this.

That's pretty neat. You could parameterize the condition and allow a
library function to handle the ugly part.

def conditional(cls, cond):
    exc_type, exc = sys.exc_info()[:2]
    if issubclass(exc_type, cls) and cond(exc):
        return exc_type
    return NeverThrown

try:
    ...
except conditional(ValueError, lambda e: e.args[0] == 42):
    ...

From sjoerdjob at sjec.nl  Thu Nov  5 11:53:24 2015
From: sjoerdjob at sjec.nl (Sjoerd Job Postmus)
Date: Thu, 5 Nov 2015 17:53:24 +0100
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <CALwzidm_qk60-b0ZEQus3HtgzM9ZT0kiv2sgidXQUP6uNsfzcQ@mail.gmail.com>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <n1f2l3$fr2$1@ger.gmane.org>
 <CAPTjJmqeGgsEbfQF=wGSCwpVyp=eWWWpujMHO4E_vU5AqrQSyg@mail.gmail.com>
 <87h9l0tp34.fsf@fastmail.com>
 <CALwzidm_qk60-b0ZEQus3HtgzM9ZT0kiv2sgidXQUP6uNsfzcQ@mail.gmail.com>
Message-ID: <D72D175D-6F01-48B6-9ED8-F0BCF4AEEEF6@sjec.nl>

This does sound like a nice approach. Neat! 

However: it *does* depend on only being resolved during the exception being matched. So you can't pre-define it.

Still, I like it. 

> On 5 Nov 2015, at 17:41, Ian Kelly <ian.g.kelly at gmail.com> wrote:
> 
>> On Thu, Nov 5, 2015 at 8:58 AM, Random832 <random832 at fastmail.com> wrote:
>> The syntax could be this:
>> except BroadError as e if is_narrower_error(e):
>> 
>> I think the bytecode *already* evaluates a boolean expression
>> which today happens to always be a comparison of the exception's
>> type. Notably, the exception type itself can apparently be an
>> arbitrary expression, which (I was mildly surprised to discover)
>> has access to sys.exc_info.
>> 
>> class NeverThrown(Exception): pass
>> 
>> def exfilter():
>>   etype, e = sys.exc_info()[:2]
>>   if is_narrower_error(e): return etype
>>   else: return NeverThrown
>> 
>> try:
>>   ...
>> except exfilter() as e:
>>   ...
>> 
>> This is, of course, a *profoundly* ugly way to do this.
> 
> That's pretty neat. You could parameterize the condition and allow a
> library function to handle the ugly part.
> 
> def conditional(cls, cond):
>    exc_type, exc = sys.exc_info()[:2]
>    if issubclass(exc_type, cls) and cond(exc):
>        return exc_type
>    return NeverThrown
> 
> try:
>    ...
> except conditional(ValueError, lambda e: e.args[0] == 42):
>    ...
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From abarnert at yahoo.com  Thu Nov  5 15:06:40 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 5 Nov 2015 12:06:40 -0800
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <afbc97894d853b3acf5604263b650627.squirrel@mail.sjec.nl>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <A8D09FB9-19BE-4453-B2E8-6077A8000FF3@yahoo.com>
 <afbc97894d853b3acf5604263b650627.squirrel@mail.sjec.nl>
Message-ID: <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com>

On Nov 5, 2015, at 07:00, sjoerdjob at sjec.nl wrote:

>>> On Nov 4, 2015, at 14:02, sjoerdjob at sjec.nl wrote:
>>> 
>>> Hi all,
>>> 
>>> TL;DR: Change exception checking logic from
>>>   err.__class__ in exc.__mro__
>>> to
>>>   isinstance(err, exc)
>>> Because it is more powerful, more consistent and allows for cleaner
>>> code.
>> 
>> Correct me if I'm wrong, but doesn't Python allow you to raise a class and
>> handle it without ever creating an instance? If an instance is needed
>> (e.g., because the exception is caught by a handler with an as clause),
>> one is created by calling exc(), but if it's not needed, that doesn't
>> happen.
>> 
>> Assuming I'm right, your proposal would mean the instance is _always_
>> needed, so if you raise a type, exc() is always called. So, for example:
>> 
>>    class MyException(Exception):
>>        def __init__(self, thingy, *a, **kw):
>>            super().__init__(*a, **kw)
>>            self.thingy = thingy
>> 
>>    try:
>>        raise MyException
>>    except MyException:
>>        print('whoops')
>> 
>> Instead of printing "whoops", it would now (I think) abort with a
>> TypeError about trying to call MyException with the wrong number of
>> arguments.
> 
> Yes, the exception is instantiated as soon as it is raised (not when it's
> matched). This happens in Python/ceval.c. So you actually have to catch
> `TypeError` in this case.

Then what's the point of the code at Python/errors.c#186 that deals with the fact that the given exception (err) can be either an instance or a type? If it must always be an instance, surely those two lines are unnecessary and misleading.

In fact, looking into it a bit more: the normal way to check for an exception from C is with PyErr_ExceptionMatches, which calls PyErr_GivenExceptionMatches with the type, not the value, of the exception. So, even if an instance has been created, you don't usually have it inside PyErr_GivenExceptionMatches anyway, so the proposed change wouldn't work without further changes. At the very least you'd need to change PyErr_ExceptionMatches to pass the value instead of the type. But presumably there's a reason it passes the type in the first place (maybe in CPython it's not possible to raise a type without an instance from pure Python code, but it is from C API code?), which means there's probably other code that has to change as well. 

And that's assuming there's no other code than calls PyErr_GivenExceptionMatches with the value of PyErr_Occurred() (or tstate->curexc_type or any other way of getting an exception type) anywhere in CPython or in any extension module anyone's ever written. Since there's nothing about the docs that implies that this is invalid, and in fact the docs for PyErr_ExceptionMatches and the source code for PyErr_GivenExceptionMatches both strongly imply that passing a type is the normal way to check, I doubt that assumption is true.

>> Plus, there's almost certainly code where someone skips creating the
>> instance as an optimization?probably it's usually a useless premature
>> one, but maybe someone has tested and it makes a difference in a real
>> program.
> 
> It will get instantiated anyway.
> 
>> One obvious possibility is: if exc is itself a subclass of BaseException,
>> you use the existing subclass check; if not, instead of using the subclass
>> check on its type, you use the new instance check. But you'd have to think
>> that through to make sure it makes sense.
> 

From sjoerdjob at sjec.nl  Thu Nov  5 15:18:22 2015
From: sjoerdjob at sjec.nl (Sjoerd Job Postmus)
Date: Thu, 5 Nov 2015 21:18:22 +0100
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <A8D09FB9-19BE-4453-B2E8-6077A8000FF3@yahoo.com>
 <afbc97894d853b3acf5604263b650627.squirrel@mail.sjec.nl>
 <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com>
Message-ID: <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl>



> On 5 Nov 2015, at 21:06, Andrew Barnert <abarnert at yahoo.com> wrote:
> 
> On Nov 5, 2015, at 07:00, sjoerdjob at sjec.nl wrote:
> 
>>>> On Nov 4, 2015, at 14:02, sjoerdjob at sjec.nl wrote:
>>>> 
>>>> Hi all,
>>>> 
>>>> TL;DR: Change exception checking logic from
>>>>  err.__class__ in exc.__mro__
>>>> to
>>>>  isinstance(err, exc)
>>>> Because it is more powerful, more consistent and allows for cleaner
>>>> code.
>>> 
>>> Correct me if I'm wrong, but doesn't Python allow you to raise a class and
>>> handle it without ever creating an instance? If an instance is needed
>>> (e.g., because the exception is caught by a handler with an as clause),
>>> one is created by calling exc(), but if it's not needed, that doesn't
>>> happen.
>>> 
>>> Assuming I'm right, your proposal would mean the instance is _always_
>>> needed, so if you raise a type, exc() is always called. So, for example:
>>> 
>>>   class MyException(Exception):
>>>       def __init__(self, thingy, *a, **kw):
>>>           super().__init__(*a, **kw)
>>>           self.thingy = thingy
>>> 
>>>   try:
>>>       raise MyException
>>>   except MyException:
>>>       print('whoops')
>>> 
>>> Instead of printing "whoops", it would now (I think) abort with a
>>> TypeError about trying to call MyException with the wrong number of
>>> arguments.
>> 
>> Yes, the exception is instantiated as soon as it is raised (not when it's
>> matched). This happens in Python/ceval.c. So you actually have to catch
>> `TypeError` in this case.
> 
> Then what's the point of the code at Python/errors.c#186 that deals with the fact that the given exception (err) can be either an instance or a type? If it must always be an instance, surely those two lines are unnecessary and misleading.
> 
> In fact, looking into it a bit more: the normal way to check for an exception from C is with PyErr_ExceptionMatches, which calls PyErr_GivenExceptionMatches with the type, not the value, of the exception. So, even if an instance has been created, you don't usually have it inside PyErr_GivenExceptionMatches anyway, so the proposed change wouldn't work without further changes. At the very least you'd need to change PyErr_ExceptionMatches to pass the value instead of the type. But presumably there's a reason it passes the type in the first place (maybe in CPython it's not possible to raise a type without an instance from pure Python code, but it is from C API code?), which means there's probably other code that has to change as well. 
> 
> And that's assuming there's no other code than calls PyErr_GivenExceptionMatches with the value of PyErr_Occurred() (or tstate->curexc_type or any other way of getting an exception type) anywhere in CPython or in any extension module anyone's ever written. Since there's nothing about the docs that implies that this is invalid, and in fact the docs for PyErr_ExceptionMatches and the source code for PyErr_GivenExceptionMatches both strongly imply that passing a type is the normal way to check, I doubt that assumption is true.

I added an assert before the except, asserting that no instance is passed, and surprisingly that assert triggers.

Also: in some cases, tstate->cur_value is actually a PyString object instead of an exception subclass (for instance raised by PyErr_FormatV (or something like that, don't have the code in front of me right now).

So yes, it does appear that a lot of code might need changing. But it also seems odd to me that exceptions get raised with strings for values instead of exception instances.

It might seem troublesome to change these locations to raise actual instances instead of strings, but those might actually just be hidden bugs just the same.

Do you think anyone would object if I'd go over the code looking for places where type(tstate->cur_value) != tstate->cur_type and propose patches for these?

>>> Plus, there's almost certainly code where someone skips creating the
>>> instance as an optimization?probably it's usually a useless premature
>>> one, but maybe someone has tested and it makes a difference in a real
>>> program.
>> 
>> It will get instantiated anyway.
>> 
>>> One obvious possibility is: if exc is itself a subclass of BaseException,
>>> you use the existing subclass check; if not, instead of using the subclass
>>> check on its type, you use the new instance check. But you'd have to think
>>> that through to make sure it makes sense.
>> 

From abarnert at yahoo.com  Thu Nov  5 17:03:14 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 5 Nov 2015 14:03:14 -0800
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <A8D09FB9-19BE-4453-B2E8-6077A8000FF3@yahoo.com>
 <afbc97894d853b3acf5604263b650627.squirrel@mail.sjec.nl>
 <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com>
 <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl>
Message-ID: <B738B29C-D696-48E7-95B9-84F120A67A44@yahoo.com>

On Nov 5, 2015, at 12:18, Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
> 
>> On 5 Nov 2015, at 21:06, Andrew Barnert <abarnert at yahoo.com> wrote:
>> 
>> On Nov 5, 2015, at 07:00, sjoerdjob at sjec.nl wrote:
>> 
>>>>> On Nov 4, 2015, at 14:02, sjoerdjob at sjec.nl wrote:
>>>>> 
>>>>> Hi all,
>>>>> 
>>>>> TL;DR: Change exception checking logic from
>>>>> err.__class__ in exc.__mro__
>>>>> to
>>>>> isinstance(err, exc)
>>>>> Because it is more powerful, more consistent and allows for cleaner
>>>>> code.
>>>> 
>>>> Correct me if I'm wrong, but doesn't Python allow you to raise a class and
>>>> handle it without ever creating an instance? If an instance is needed
>>>> (e.g., because the exception is caught by a handler with an as clause),
>>>> one is created by calling exc(), but if it's not needed, that doesn't
>>>> happen.
>>>> 
>>>> Assuming I'm right, your proposal would mean the instance is _always_
>>>> needed, so if you raise a type, exc() is always called. So, for example:
>>>> 
>>>>  class MyException(Exception):
>>>>      def __init__(self, thingy, *a, **kw):
>>>>          super().__init__(*a, **kw)
>>>>          self.thingy = thingy
>>>> 
>>>>  try:
>>>>      raise MyException
>>>>  except MyException:
>>>>      print('whoops')
>>>> 
>>>> Instead of printing "whoops", it would now (I think) abort with a
>>>> TypeError about trying to call MyException with the wrong number of
>>>> arguments.
>>> 
>>> Yes, the exception is instantiated as soon as it is raised (not when it's
>>> matched). This happens in Python/ceval.c. So you actually have to catch
>>> `TypeError` in this case.
>> 
>> Then what's the point of the code at Python/errors.c#186 that deals with the fact that the given exception (err) can be either an instance or a type? If it must always be an instance, surely those two lines are unnecessary and misleading.
>> 
>> In fact, looking into it a bit more: the normal way to check for an exception from C is with PyErr_ExceptionMatches, which calls PyErr_GivenExceptionMatches with the type, not the value, of the exception. So, even if an instance has been created, you don't usually have it inside PyErr_GivenExceptionMatches anyway, so the proposed change wouldn't work without further changes. At the very least you'd need to change PyErr_ExceptionMatches to pass the value instead of the type. But presumably there's a reason it passes the type in the first place (maybe in CPython it's not possible to raise a type without an instance from pure Python code, but it is from C API code?), which means there's probably other code that has to change as well. 
>> 
>> And that's assuming there's no other code than calls PyErr_GivenExceptionMatches with the value of PyErr_Occurred() (or tstate->curexc_type or any other way of getting an exception type) anywhere in CPython or in any extension module anyone's ever written. Since there's nothing about the docs that implies that this is invalid, and in fact the docs for PyErr_ExceptionMatches and the source code for PyErr_GivenExceptionMatches both strongly imply that passing a type is the normal way to check, I doubt that assumption is true.
> 
> I added an assert before the except, asserting that no instance is passed, and surprisingly that assert triggers.
> 
> Also: in some cases, tstate->cur_value is actually a PyString object instead of an exception subclass (for instance raised by PyErr_FormatV (or something like that, don't have the code in front of me right now).
> 
> So yes, it does appear that a lot of code might need changing. But it also seems odd to me that exceptions get raised with strings for values instead of exception instances.
> 
> It might seem troublesome to change these locations to raise actual instances instead of strings, but those might actually just be hidden bugs just the same.
> 
> Do you think anyone would object if I'd go over the code looking for places where type(tstate->cur_value) != tstate->cur_type and propose patches for these?

It would be really nice if someone remembered _why_ the code is this way. It may be a performance issue?if the exception is raised in C code and handled in C code and never displayed, the exception object never gets used, so why waste time (and complicate the reference graph) creating one? Or it may just be a holdover from Python 1.x's string exceptions. Or there may be some other reason neither of us can guess. Without knowing which it is, it's hard to know what to do about it.

But meanwhile, even if you patch all of CPython, that doesn't change the fact that there are probably extension libraries out there setting exceptions with a null or string value, or checking the exception by using PyErr_ExceptionOccurred or other means that ignore the value (without going through PyErr_GivenExceptionMatches or anything else you could change), which are perfectly valid things to do according to the docs, and have worked for decades. How are you going to fix all of those?

Also, even if it's true that pure Python code can't create an exception without a value, only C code can, the way the language reference is written, that seems to be something specific to CPython, not guaranteed by the language. So, other implementations might be doing something different, and you'd have to fix all of them as well. (That may not be a big deal?are there any other major 3.x implementations besides PyPy, which is still on 3.2?)

The best thing I can think of is to add a new API around exception values instead of types (a replacement for ExceptionOccurred that returns the value instead of the type, a replacement for ExceptionMatches that requires an instance instead of taking an instance or type, etc.), 
change GivenExceptionMatches (and the except clause implementation) to use that API, change SetNone and friends to always create a type, deprecate ExceptionOccurred and friends, change the documentation to make it clear than raising a type in Python is the same as raising Type(), wait a couple versions for the C API deprecation, and only then can you assume that ExceptionMatches always gets an instance (and decide what you do if not), so you can make your proposed change.

From rosuav at gmail.com  Thu Nov  5 17:12:28 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 6 Nov 2015 09:12:28 +1100
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <B738B29C-D696-48E7-95B9-84F120A67A44@yahoo.com>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <A8D09FB9-19BE-4453-B2E8-6077A8000FF3@yahoo.com>
 <afbc97894d853b3acf5604263b650627.squirrel@mail.sjec.nl>
 <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com>
 <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl>
 <B738B29C-D696-48E7-95B9-84F120A67A44@yahoo.com>
Message-ID: <CAPTjJmqRDLRTiUvgotYFEkUuD0F6gWtVyzQPoKDTYxSC5k86HA@mail.gmail.com>

On Fri, Nov 6, 2015 at 9:03 AM, Andrew Barnert via Python-ideas
<python-ideas at python.org> wrote:
> It would be really nice if someone remembered _why_ the code is this way. It may be a performance issue?if the exception is raised in C code and handled in C code and never displayed, the exception object never gets used, so why waste time (and complicate the reference graph) creating one?
>

I have some recollection that it's most commonly hit when you're
working with StopIteration - an internal iterator raises it, a 'for'
loop catches it, and no instance is ever constructed.

ChrisA

From stephen at xemacs.org  Fri Nov  6 00:33:24 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 6 Nov 2015 14:33:24 +0900
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <B738B29C-D696-48E7-95B9-84F120A67A44@yahoo.com>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <A8D09FB9-19BE-4453-B2E8-6077A8000FF3@yahoo.com>
 <afbc97894d853b3acf5604263b650627.squirrel@mail.sjec.nl>
 <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com>
 <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl>
 <B738B29C-D696-48E7-95B9-84F120A67A44@yahoo.com>
Message-ID: <22076.15268.375643.440588@turnbull.sk.tsukuba.ac.jp>

Andrew Barnert via Python-ideas writes:
 > Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:

 > > It might seem troublesome to change these locations to raise
 > > actual instances instead of strings, but those might actually
 > > just be hidden bugs just the same.

No, I'm sure mostly they're not.  They're working code that has been
there since the early days, perhaps a few from the 2.x days in
modules.  Python is generally very conservative about changing working
code.

 > > Do you think anyone would object if I'd go over the code looking
 > > for places where type(tstate->cur_value) != tstate->cur_type and
 > > propose patches for these?

I think, yes, though I don't know enough about it to object myself.
Code churn of this kind is generally frowned upon, unless you can show
that the risk of hidden bugs of the kind you refer to above is
substantially greater than the risk that *you* will introduce bugs
with your patches.

 > It would be really nice if someone remembered _why_ the code is
 > this way. It may be a performance issue?

Or it could be bootstrapping -- it may be that "something" can be
raised before the exception machinery is fully initialized.  (On
reflection, that seems unlikely, if raise is available, BaseException
probably is too.  But you never know. :)

I would guess it was discussed, decided, and pronounced in one of the
early Python 3 PEPs.  (I'm sure that for Python 2 it was purely an
issue of backward compatibility.)


From sjoerdjob at sjec.nl  Fri Nov  6 00:57:22 2015
From: sjoerdjob at sjec.nl (sjoerdjob at sjec.nl)
Date: Fri, 6 Nov 2015 06:57:22 +0100
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <22076.15268.375643.440588@turnbull.sk.tsukuba.ac.jp>
References: <752adf5dfc75fcf898f576b5ddff7fc7.squirrel@mail.sjec.nl>
 <A8D09FB9-19BE-4453-B2E8-6077A8000FF3@yahoo.com>
 <afbc97894d853b3acf5604263b650627.squirrel@mail.sjec.nl>
 <83547380-3947-42BA-B8B6-D97AF52D296D@yahoo.com>
 <3540DB5E-C07D-4240-82EE-30966B45B2D8@sjec.nl>
 <B738B29C-D696-48E7-95B9-84F120A67A44@yahoo.com>
 <22076.15268.375643.440588@turnbull.sk.tsukuba.ac.jp>
Message-ID: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl>

> Andrew Barnert via Python-ideas writes:
>  > Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
>
>  > > It might seem troublesome to change these locations to raise
>  > > actual instances instead of strings, but those might actually
>  > > just be hidden bugs just the same.
>
> No, I'm sure mostly they're not.  They're working code that has been
> there since the early days, perhaps a few from the 2.x days in
> modules.  Python is generally very conservative about changing working
> code.
>
>  > > Do you think anyone would object if I'd go over the code looking
>  > > for places where type(tstate->cur_value) != tstate->cur_type and
>  > > propose patches for these?
>
> I think, yes, though I don't know enough about it to object myself.
> Code churn of this kind is generally frowned upon, unless you can show
> that the risk of hidden bugs of the kind you refer to above is
> substantially greater than the risk that *you* will introduce bugs
> with your patches.

After digging more into the code, I concur. It's quite easy to make some
changes that break some stuff in this logic.

>  > It would be really nice if someone remembered _why_ the code is
>  > this way. It may be a performance issue???
>
> Or it could be bootstrapping -- it may be that "something" can be
> raised before the exception machinery is fully initialized.  (On
> reflection, that seems unlikely, if raise is available, BaseException
> probably is too.  But you never know. :)
>
> I would guess it was discussed, decided, and pronounced in one of the
> early Python 3 PEPs.  (I'm sure that for Python 2 it was purely an
> issue of backward compatibility.)

To all: thank you for at least considering my suggestion, and helping me
consider the pros and cons.

Even though I myself still think it would be a major benefit to change
from `issubclass` (or `exc in err.__mro__`) to `isinstance`, I do now see
that the changes needed to the Python codebase are quite substantial. The
consideration about third-party C-modules is also something to further
think about.

As I understand it now, the changes needed---and the possibility of
introduced bugs---outweigh the added benefit of such a change, at least
for now.

(Still, it send me along a nice cursory browse of the Python code base,
including the bytecode compiler and the evaluator. The compilation of the
bytecode for exception handling is nicely documented. To the author(s) of
that code: great job, it's very understandable!)

From abarnert at yahoo.com  Fri Nov  6 02:49:07 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 6 Nov 2015 07:49:07 +0000 (UTC)
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl>
References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl>
Message-ID: <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com>

On Thursday, November 5, 2015 9:57 PM, "sjoerdjob at sjec.nl" <sjoerdjob at sjec.nl> wrote:

>>  Andrew Barnert via Python-ideas writes:
>>   > Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
>> 
>>   > > It might seem troublesome to change these locations to raise
>>   > > actual instances instead of strings, but those might actually
>>   > > just be hidden bugs just the same.
>> 
>>  No, I'm sure mostly they're not.  They're working code that has 
> been
>>  there since the early days, perhaps a few from the 2.x days in
>>  modules.  Python is generally very conservative about changing working
>>  code.
>> 
>>   > > Do you think anyone would object if I'd go over the code 
> looking
>>   > > for places where type(tstate->cur_value) != 
> tstate->cur_type and
>>   > > propose patches for these?
>> 
>>  I think, yes, though I don't know enough about it to object myself.
>>  Code churn of this kind is generally frowned upon, unless you can show
>>  that the risk of hidden bugs of the kind you refer to above is
>>  substantially greater than the risk that *you* will introduce bugs
>>  with your patches.
> 
> After digging more into the code, I concur. It's quite easy to make some
> changes that break some stuff in this logic.

Looking at this a bit further, I think there may be something you can do that gets everything you actually wanted, without changes all over the place and serious risks.

Basically, it's just two pretty small changes in errors.c:

* Change PyErr_GivenExceptionMatches so that if given a value that's an instance of an exception type (PyExceptionInstance_Check(err) is true), instead of just getting its class and then ignoring the value, it does a standard isinstance check on the value.

* Change PyErr_ExceptionMatches so that if there is an exception value, and its type matches the exception type, it is passed to PyErr_GivenExceptionMatches; otherwise, the exception type is passed.


So, all Python code that raises and handles exceptions will get the benefit of isinstance checks. Any new C code that wants that benefit can get it very easily (only raise with PyErr_SetObject or something that uses it like PyErr_SetFromErrno; only handle with PyErr_GivenExceptionMatches). Any existing C code that skips creating an instance will continue to do so, and to get the same fast-path check used today, except for one extra call to PyExceptionInstance_Check(tstate->curexc_value).

Of course it's not just that small patch to errors.c. You also need tests to exercise the new functionality. And a docs patch for the Exceptions section of the C API. And also a docs patch to the Exceptions section in the Reference. But no other code patches.

And you'd definitely need to performance-test two things: First, that the extra PyExceptionInstance_Check doesn't slow down fast-path handling (e.g., in an empty for loops). Second, whatever performance tests your original proposal would have needed, to make sure the isinstance check doesn't noticeably slow down pure-Python code that handles a lot of exceptions.


I think the only C API docs change needed is to PyErr_ExceptionMatches. However, a note explaining how to make sure to get and/or to bypass the isinstance check as desired might be useful.

For the reference docs, I think just this note at the end of the section:

> Note: Exceptions raised and handled by Python code use the same mechanism as the isinstance function for determining whether an exception matches. However, exceptions raised or handled by the implementation (or implementation-level extensions) may bypass that and check the type directly.


> New in 3.6: In previous versions of Python, exception handling always bypassed the isinstance function and checked the type directly.

From amir at rachum.com  Fri Nov  6 17:11:14 2015
From: amir at rachum.com (Amir Rachum)
Date: Sat, 7 Nov 2015 00:11:14 +0200
Subject: [Python-ideas] Adding collections.abc.Ordered
Message-ID: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>

I am suggesting the addition of a collections abstract base class called
"Ordered". Its meaning is that a collection's iteration order is part of
its API. The bulk of this mail describes a use case for this. The reason I
believe that such abstract base class is required is that there is no way
to test this behavior in a given class. An ordered collection has the exact
same interface as an unordered collection (e.g, dict and OrderedDict),
other than a _promise_ of the API that the order in which this collection
will be iterated has some sort of meaning (In OrderedDict, it is the order
in which keys were added to it.)


As examples, set, frozenset, dict and defaultdict should *not* be
considered as ordered. list, OrderedDict, deque and tuple should be
considered ordered.


Before I dive into the use case I am presenting, I would like to point out
that a similar discussion was already done on the subject (suggesting at
first that OrderedDict would abstract-subclass from Sequence) at the
following thread* - http://code.activestate.com/lists/python-ideas/29532/.
That thread was abandoned largely from a lack of a clear use case, which I
hope will be more clear in this suggestion.


I am working on package called basicstruct (
https://pypi.python.org/pypi/basicstruct). The way it works is that you
define an object that inherits from basicstruct.BasicStruct, define
__slots__ , and automatically get behaviors such as a nice __init__,
__str__, comparison functions, attribute access, dict-like access,
pickleability, etc. It is similar to namedtuple, except that it is mutable.


In the Python documentation, the following is said regarding __slots__:


   Any non-string iterable may be assigned to __slots__. Mappings may also
be used; however, in the future, special meaning may be assigned to the
values corresponding to each key.


Here's how the current__init__ method of BasicStruct looks like:


   class BasicStruct(object):

       """Class for holding struct-like objects."""


       __slots__ = ()  # should be extended by deriving classes


       def __init__(self, *args, **kwargs):

           arg_pairs = zip(self.__slots__, args)

           for key, value in chain(arg_pairs, six.iteritems(kwargs)):

               setattr(self, key, value)


           for key in self.__slots__:

               if not hasattr(self, key):

                   setattr(self, key, None)


Notice the following line:



   arg_pairs = zip(self.__slots__, args)


It assumes that __slots__ defines attributes in a certain order. So as a
use I would expect that the following code


   class MyStruct(BasicStruct):

      __slots__ = ('x', 'y')



   MyStruct(0, 1)


... will create a struct in which x is 0 and y is 1.


However, if I define __slots__ as a set, or dict, the result is undefined.


   class MyStruct(BasicStruct):

      __slots__ = {'x', 'y'}  # No order is defined here



   MyStruct(0, 1)  # Which is which?


So, In BasicStruct's __init__ method it is required to test whether
__slots__ was defined with a collection that has a meaningful order. If it
was _not_, using non-keyword arguments in __init__ should be forbidden,
since the order of __slots__ is arbitrary. Here is how I wish the code
would look like:


   class BasicStruct(object):

       """Class for holding struct-like objects."""



       __slots__ = ()  # should be extended by deriving classes



       def __init__(self, *args, **kwargs):

           ordered = isinstance(self.__slots__, Ordered)



           if args and not ordered:

               raise ValueError("Can't pass non-keyword arguments to {},
since "

                                "__slots__ was declared with an unordered "

                                "iterable.".format(self.__class__.__name__))



           arg_pairs = zip(self.__slots__, args)

           for key, value in chain(arg_pairs, six.iteritems(kwargs)):

               setattr(self, key, value)



           for key in self.__slots__:

               if not hasattr(self, key):

                   setattr(self, key, None)


Thanks,

Amir Rachum




* The author of the original thread is Ram Rachum. To avoid some confusion
- yes, we are related - Ram is my big brother. It's just so happens that I
was looking around for this issue and found that he did so in the past as
well.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151107/0a55fd7f/attachment-0001.html>

From abarnert at yahoo.com  Fri Nov  6 23:47:41 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 6 Nov 2015 20:47:41 -0800
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
Message-ID: <2B772FD7-9B1A-45E1-A38C-415F194DA503@yahoo.com>

As I understand it, the only reason you want this ABC is so that if someone uses your struct-sequence class and defines his attributes with no inherent order, you want to be able to raise an error if he uses positional initialization. Has anyone ever done that? Has anyone complained "I created a struct type with attributes a and b in arbitrary order, and then when I do Spam(1, 2), sometimes a gets 1 and sometimes it gets 2"?

It seems to me that this is obviously the same as, say, a user explicitly zipping two sets together. Sure, it's a stupid thing to do?but it's an _obvious_ stupid thing to do, so zip doesn't have to prevent them from doing it or go out of its way to warn them. And it's the same here. The order of the attributes is the order of the slots, so if you specify them in arbitrary order, that will be an arbitrary order, obviously.

Maybe there's something about this use case that causes people to not notice they've screwed up like this? If so, I think that's what you have to make a case for. (That's why I asked whether anyone has actually complained about it.)

One more thing: what's stopping you from defining Ordered in your module and registering all the relevant builtin types there? In fact, you pretty much have to do this even if your proposal is accepted, unless you want your module to require 3.6+ (which seems unlikely, given that you're using six). So what would you gain by having it in the stdlib?

> On Nov 6, 2015, at 14:11, Amir Rachum <amir at rachum.com> wrote:
> 
> I am suggesting the addition of a collections abstract base class called "Ordered". Its meaning is that a collection's iteration order is part of its API. The bulk of this mail describes a use case for this. The reason I believe that such abstract base class is required is that there is no way to test this behavior in a given class. An ordered collection has the exact same interface as an unordered collection (e.g, dict and OrderedDict), other than a _promise_ of the API that the order in which this collection will be iterated has some sort of meaning (In OrderedDict, it is the order in which keys were added to it.)
> 
> As examples, set, frozenset, dict and defaultdict should *not* be considered as ordered. list, OrderedDict, deque and tuple should be considered ordered.
> 
> Before I dive into the use case I am presenting, I would like to point out that a similar discussion was already done on the subject (suggesting at first that OrderedDict would abstract-subclass from Sequence) at the following thread* - http://code.activestate.com/lists/python-ideas/29532/. That thread was abandoned largely from a lack of a clear use case, which I hope will be more clear in this suggestion.
> 
> I am working on package called basicstruct (https://pypi.python.org/pypi/basicstruct). The way it works is that you define an object that inherits from basicstruct.BasicStruct, define __slots__ , and automatically get behaviors such as a nice __init__, __str__, comparison functions, attribute access, dict-like access, pickleability, etc. It is similar to namedtuple, except that it is mutable.
> 
> In the Python documentation, the following is said regarding __slots__:
> 
>     Any non-string iterable may be assigned to __slots__. Mappings may also be used; however, in the future, special meaning may be assigned to the values corresponding to each key.
> 
> Here's how the current__init__ method of BasicStruct looks like:
> 
>     class BasicStruct(object):
>         """Class for holding struct-like objects."""
> 
>         __slots__ = ()  # should be extended by deriving classes
> 
>         def __init__(self, *args, **kwargs):
>             arg_pairs = zip(self.__slots__, args)
>             for key, value in chain(arg_pairs, six.iteritems(kwargs)):
>                 setattr(self, key, value)
> 
>             for key in self.__slots__:
>                 if not hasattr(self, key):
>                     setattr(self, key, None)
> 
> Notice the following line:
>     
>     arg_pairs = zip(self.__slots__, args)
> 
> It assumes that __slots__ defines attributes in a certain order. So as a use I would expect that the following code
> 
>     class MyStruct(BasicStruct):
>        __slots__ = ('x', 'y')
>     
>     MyStruct(0, 1)
> 
> ... will create a struct in which x is 0 and y is 1.
> 
> However, if I define __slots__ as a set, or dict, the result is undefined.
> 
>     class MyStruct(BasicStruct):
>        __slots__ = {'x', 'y'}  # No order is defined here
>     
>     MyStruct(0, 1)  # Which is which?
> 
> So, In BasicStruct's __init__ method it is required to test whether __slots__ was defined with a collection that has a meaningful order. If it was _not_, using non-keyword arguments in __init__ should be forbidden, since the order of __slots__ is arbitrary. Here is how I wish the code would look like:
> 
>     class BasicStruct(object):
>         """Class for holding struct-like objects."""
>     
>         __slots__ = ()  # should be extended by deriving classes
>     
>         def __init__(self, *args, **kwargs):
>             ordered = isinstance(self.__slots__, Ordered)
>     
>             if args and not ordered:
>                 raise ValueError("Can't pass non-keyword arguments to {}, since "
>                                  "__slots__ was declared with an unordered "
>                                  "iterable.".format(self.__class__.__name__))
>     
>             arg_pairs = zip(self.__slots__, args)
>             for key, value in chain(arg_pairs, six.iteritems(kwargs)):
>                 setattr(self, key, value)
>     
>             for key in self.__slots__:
>                 if not hasattr(self, key):
>                     setattr(self, key, None)
> 
> Thanks,
> Amir Rachum
> 
> 
> 
> * The author of the original thread is Ram Rachum. To avoid some confusion - yes, we are related - Ram is my big brother. It's just so happens that I was looking around for this issue and found that he did so in the past as well.
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151106/7dae3f92/attachment-0001.html>

From stephen at xemacs.org  Sat Nov  7 00:00:49 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 7 Nov 2015 14:00:49 +0900
Subject: [Python-ideas]  Adding collections.abc.Ordered
In-Reply-To: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
Message-ID: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>

Amir Rachum writes:

 > An ordered collection has the exact same interface as an unordered
 > collection (e.g, dict and OrderedDict), other than a _promise_ of
 > the API that the order in which this collection will be iterated
 > has some sort of meaning (In OrderedDict, it is the order in which
 > keys were added to it.)

I think this is underspecified.  One of the main reasons why ordered
versions of initially unordered types, especially dict, take so long
to standardize is that people (even the Dutch!) disagree about the One
Obvious Order to impose.  After repeated refusals, only the factions
that continue clamoring- and preferably, only one -need be considered.

This ABC would be a tar pit of recriminations between class authors
and class clients.  At the very least, the author's intended
comparison function should be introspectable so that client subclasses
can include validation functions.  (I guess you could just use
"lambda slot: list(self.__slots__).index(slot)" as the key function,
but that doesn't say much about the conceptual semantics.)  Then the
presence of the ordering function would provide a way to duck-type
abc.Ordered.

I haven't thought carefully about it, but a method with semantics of
"insert next slot at end according to natural order" (named
"natural_append"?  "ordered_append"? or maybe the existing "append"
used by sequences could be given this semantic as well?

Nevertheless, I don't think this shold be added to Python yet.  I
don't think your use case is a valid one for supporting it.  I have
two reasons:

(1) In principle, record types should not be initialized with
    positional arguments.  I experienced cogniztive dissonance as soon
    as I saw your __init__():

    > Here's how the current__init__ method of BasicStruct looks like:

    >    class BasicStruct(object):
    >        """Class for holding struct-like objects."""

    "with naturally-ordered slots" should be appended.

    >        __slots__ = ()  # should be extended by deriving classes

    >        def __init__(self, *args, **kwargs):

    Yes, we do it in C all the time, and it works fine[1] there.  But
    the semantics are mismatched at the conceptual level, and it is
    error-prone in practice if the type has more than about 3 slots of
    differing types.  If a concrete BasicStruct "wants" to provide a
    positional constructor, it can use a factory function.

    Unfortunately this would be impossible to do automatically in the
    exactly the cases of interest -- you need the *source code* order
    of slots, but that's not available for introspection at runtime.
    So it would violate DRY.  I don't consider that an argument for
    the proposal at the moment, but feel free to try to convince the
    list (and maybe me too<wink/>) otherwise.)  (In fact I consider it
    the reverse: for Sequences, The Order You See Is The Order You
    Get, and the DRY violation occurs *because* you are using an
    inappropriate container for your purpose (ordered slots).

(2) BasicStruct isn't the relevant use case: you can always just
    document that __slots__ is restricted to ordered finite iterables
    except string-like ones which use sequences of length 1 to
    represent individual members, and perhaps trap some of the more
    obvious mistakes (set and dict and maybe their subclasses).

    The use case you need to present is one where a non-Sequence (eg,
    a set or dict) is the obvious representation for __slots__ in a
    BasicStruct.  Note that __slots__ allows nearly arbitrary finite
    iterables of strings only because there's no reason not to, and it
    simplifies the implementation.  In Python practice, that is indeed
    an invitation to experiment with concrete classes that use
    unordered __slots__ for some reason.  But there's no sense that
    abstract classes that use __slots__ in an abstract way (as yours
    does) must preserve that generality, and therefore no reason (yet)
    for Python to help you preserve that generality.


Footnotes: 
[1]  According to the usual definition of "works fine in C": is no
more than twice as error-prone as other features of C.  Given the
existence of pointers....


From abarnert at yahoo.com  Sat Nov  7 04:54:56 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 7 Nov 2015 01:54:56 -0800
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
Message-ID: <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>

On Nov 6, 2015, at 21:00, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> 
> Amir Rachum writes:
> 
>> An ordered collection has the exact same interface as an unordered
>> collection (e.g, dict and OrderedDict), other than a _promise_ of
>> the API that the order in which this collection will be iterated
>> has some sort of meaning (In OrderedDict, it is the order in which
>> keys were added to it.)
> 
> I think this is underspecified.  One of the main reasons why ordered
> versions of initially unordered types, especially dict, take so long
> to standardize is that people (even the Dutch!) disagree about the One
> Obvious Order to impose.  After repeated refusals, only the factions
> that continue clamoring- and preferably, only one -need be considered.
> 
> This ABC would be a tar pit of recriminations between class authors
> and class clients.  

I think you've missed the point of his proposal. It doesn't matter _what_ the order is?insertion order, sorted by __lt__ (possibly with a key function), looked up by a function that maps keys to indices, whatever?only that there is some meaningful order (and that iteration follows that order). If so, your type can declare that it implements Ordered.

Of course the user has to know what that order is and what it means in order to do anything with it. In his example, if you use, say, a blist.sortedlist instead of a list for your slots, then you have to pass positional parameters to the initializer in sorted order instead of in source code order. But if that's what makes sense for your code, then great.

The reason I'm not sure it's helpful is that "ordered in some way that makes sense to me that you don't know about" isn't really that useful of a thing for you to check for me. After all, I can use a blist.sortedlist('bac') and then pass the initializer values in b-a-c order, or use list('bac') and pass them in a-b-c order. Each of those is at least as easy to screw up, and just as wrong, as using a set('bac'), and his code can't possibly check for the first two, so what's the point of checking for the third?

But, whether the proposal is useful or not, it is sufficiently specified.

> At the very least, the author's intended
> comparison function should be introspectable so that client subclasses
> can include validation functions.  (I guess you could just use
> "lambda slot: list(self.__slots__).index(slot)" as the key function,
> but that doesn't say much about the conceptual semantics.)  Then the
> presence of the ordering function would provide a way to duck-type
> abc.Ordered.

How would you define the comparison function for OrderedDict in a way that makes any semantic sense? Or, for that matter, list? The order is the result of a dynamic sequence of operations that nobody's kept track of and that in general can't be recovered. But it doesn't matter why the values are in some particular order, only that they are. Surely OrderedDict and list are perfectly reasonable types for __slots__, despite the fact that their comparison function doesn't exist.

> I haven't thought carefully about it, but a method with semantics of
> "insert next slot at end according to natural order" (named
> "natural_append"?  "ordered_append"? or maybe the existing "append"
> used by sequences could be given this semantic as well?

You're talking about the API for a mutable sorted collection here. That's an interesting question (and I've written a blog post about it, and had long conversations with the authors of various different sorted collection libs on PyPI), but it's not relevant here. If we wanted to add sorted collection ABCs to the stdlib, that still wouldn't help the OP at all, because things like list and OrderedDict are Ordered in his sense but not sorted in any useful sense.

(If you're curious: I think the best answer is to use add. A sorted collection is more like a multiset, at least from the mutation side, than like a list. Append, insert, and __setitem__ are all nonsense; add makes perfect sense. But not everyone agrees with me. If you want to discuss this, you should probably take it off-list, or start a new thread on adding sorted collections ABCs, because they have very little to do with adding an Ordered ABC.)

> Nevertheless, I don't think this shold be added to Python yet.  I
> don't think your use case is a valid one for supporting it.  I have
> two reasons:
> 
> (1) In principle, record types should not be initialized with
>    positional arguments.  I experienced cogniztive dissonance as soon
>    as I saw your __init__():
> 
>> Here's how the current__init__ method of BasicStruct looks like:
> 
>>   class BasicStruct(object):
>>       """Class for holding struct-like objects."""
> 
>    "with naturally-ordered slots" should be appended.
> 
>>       __slots__ = ()  # should be extended by deriving classes
> 
>>       def __init__(self, *args, **kwargs):
> 
>    Yes, we do it in C all the time, and it works fine[1] there. But
>    the semantics are mismatched at the conceptual level, and it is
>    error-prone in practice if the type has more than about 3 slots of
>    differing types

It's worth noting that the C committee agrees with you, which is why they added designated initializers to the language, which are often considered among the best upgrades from C89. Unless you need C89 support or (portable) C++ compatibility (sadly, you usually do need one or the other?), unless you're initializing something with only a few attributes with an obvious-to-the-reader order (like a point, where it's pretty clear that { 1, 2, 3 } is initializing x, y, and z), you use a designated initializer like { .spam=1, .eggs=2, .cheese=3 }. Which looks sort of like Python, but with extra line noise.

>  If a concrete BasicStruct "wants" to provide a
>    positional constructor, it can use a factory function.
> 
>    Unfortunately this would be impossible to do automatically in the
>    exactly the cases of interest -- you need the *source code* order
>    of slots, but that's not available for introspection at runtime.
>    So it would violate DRY.  I don't consider that an argument for
>    the proposal at the moment, but feel free to try to convince the
>    list (and maybe me too<wink/>) otherwise.)  (In fact I consider it
>    the reverse: for Sequences, The Order You See Is The Order You
>    Get, and the DRY violation occurs *because* you are using an
>    inappropriate container for your purpose (ordered slots).
> 
> (2) BasicStruct isn't the relevant use case: you can always just
>    document that __slots__ is restricted to ordered finite iterables

It had better be a repeatable one too; after all, iter(['a', 'b']) is an ordered finite iterable, but not a very good candidate for slots. But that's a whole other thread. :)

>    except string-like ones which use sequences of length 1 to
>    represent individual members, and perhaps trap some of the more
>    obvious mistakes (set and dict and maybe their subclasses).
> 
>    The use case you need to present is one where a non-Sequence (eg,
>    a set or dict) is the obvious representation for __slots__ in a
>    BasicStruct.  Note that __slots__ allows nearly arbitrary finite
>    iterables of strings only because there's no reason not to, and it
>    simplifies the implementation.  In Python practice, that is indeed
>    an invitation to experiment with concrete classes that use
>    unordered __slots__ for some reason.  But there's no sense that
>    abstract classes that use __slots__ in an abstract way (as yours
>    does) must preserve that generality, and therefore no reason (yet)
>    for Python to help you preserve that generality.
> 
> 
> Footnotes: 
> [1]  According to the usual definition of "works fine in C": is no
> more than twice as error-prone as other features of C.  Given the
> existence of pointers....
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From stephen at xemacs.org  Sat Nov  7 09:41:39 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 7 Nov 2015 23:41:39 +0900
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
Message-ID: <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>

Andrew Barnert writes:
 > On Nov 6, 2015, at 21:00, Stephen J. Turnbull <stephen at xemacs.org> wrote:

 > > This ABC would be a tar pit of recriminations between class authors
 > > and class clients.  

 > I think you've missed the point of his proposal. It doesn't matter
 > _what_ the order is

That's precisely what I see as problematic, unless the author of the
subclass is the same as the author of the code using the subclass.

 > But, whether the proposal is useful or not, it is sufficiently
 > specified.

I wrote inaccurately, I guess.  My contention is that it's more
dangerous than useful as currently specified but that could be
reversed with additional restrictions.

 > How would you define the comparison function for OrderedDict in a
 > way that makes any semantic sense?  Or, for that matter, list?

Again, bad nomenclature, sorry.[1]  I should have used "index".  For
this application, I have in mind

class Ordered:

    def index(self, key):
        return list(z for z in self).index(key)

    def compare(self, key1, key2):
        return self.index(key1) < self.index(key2)

as the default implementation (except that for Amir's application it's
.index() that's interesting, not .compare() -- .compare() could be
omitted I guess, fn. [1] applies).  The problem with list would be
that it can have multiple entries of the same value that are not
adjacent, of course, but then it wouldn't work for Amir's application
anyway.  This is part of what I have in mind when I say
"underspecified".

I don't think thought about how this collections.abc.Ordered should
interact with objects that could be considered to represent multisets.
Do you?

 > Surely OrderedDict and list are perfectly reasonable types for
 > __slots__, despite the fact that their comparison function doesn't
 > exist.

If given the above renaming you still say it doesn't exist, I don't
understand what you're trying to say.



Footnotes: 
[1]  As an excuse for the poor nomenclature, in my field I have to
worry about (pre-) ordered structures for which an index function does
not exist, thus "comparison function".


From mistersheik at gmail.com  Sat Nov  7 10:35:41 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Sat, 7 Nov 2015 07:35:41 -0800 (PST)
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
Message-ID: <61c756e9-ffbf-45bd-8eb4-024a6329dded@googlegroups.com>



On Saturday, November 7, 2015 at 9:42:19 AM UTC-5, Stephen J. Turnbull 
wrote:
>
> Andrew Barnert writes: 
>  > On Nov 6, 2015, at 21:00, Stephen J. Turnbull <ste... at xemacs.org 
> <javascript:>> wrote: 
>
>  > > This ABC would be a tar pit of recriminations between class authors 
>  > > and class clients.   
>
>  > I think you've missed the point of his proposal. It doesn't matter 
>  > _what_ the order is 
>
> That's precisely what I see as problematic, unless the author of the 
> subclass is the same as the author of the code using the subclass. 
>
>  > But, whether the proposal is useful or not, it is sufficiently 
>  > specified. 
>
> I wrote inaccurately, I guess.  My contention is that it's more 
> dangerous than useful as currently specified but that could be 
> reversed with additional restrictions. 
>

It's not "dangerous"!!  Talk about hyperbole.  It's just a question of 
utility.  Personally I would like to see a much richer collections.abc for 
tests like this.
 

>
>  > How would you define the comparison function for OrderedDict in a 
>  > way that makes any semantic sense?  Or, for that matter, list? 
>
> Again, bad nomenclature, sorry.[1]  I should have used "index".  For 
> this application, I have in mind 
>
> class Ordered: 
>
>     def index(self, key): 
>         return list(z for z in self).index(key) 
>
>     def compare(self, key1, key2): 
>         return self.index(key1) < self.index(key2) 
>
> as the default implementation (except that for Amir's application it's 
> .index() that's interesting, not .compare() -- .compare() could be 
> omitted I guess, fn. [1] applies).  The problem with list would be 
> that it can have multiple entries of the same value that are not 
> adjacent, of course, but then it wouldn't work for Amir's application 
> anyway.  This is part of what I have in mind when I say 
> "underspecified". 
>
> I don't think thought about how this collections.abc.Ordered should 
> interact with objects that could be considered to represent multisets. 
> Do you? 
>

Did you read the original post?  All you need is a consistent order.  It 
doesn't matter if the same element turns up twice in order to be Ordered. 
 He can check uniqueness after the fact.
 

>
>  > Surely OrderedDict and list are perfectly reasonable types for 
>  > __slots__, despite the fact that their comparison function doesn't 
>  > exist. 
>
> If given the above renaming you still say it doesn't exist, I don't 
> understand what you're trying to say. 
>
>
>
> Footnotes: 
> [1]  As an excuse for the poor nomenclature, in my field I have to 
> worry about (pre-) ordered structures for which an index function does 
> not exist, thus "comparison function". 
>
> _______________________________________________ 
> Python-ideas mailing list 
> Python... at python.org <javascript:> 
> https://mail.python.org/mailman/listinfo/python-ideas 
> Code of Conduct: http://python.org/psf/codeofconduct/ 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151107/a51a00df/attachment-0001.html>

From storchaka at gmail.com  Sat Nov  7 12:06:03 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 7 Nov 2015 19:06:03 +0200
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
Message-ID: <n1lb1s$bi6$1@ger.gmane.org>

On 07.11.15 00:11, Amir Rachum wrote:
> I am suggesting the addition of a collections abstract base class called
> "Ordered". Its meaning is that a collection's iteration order is part of
> its API. The bulk of this mail describes a use case for this. The reason
> I believe that such abstract base class is required is that there is no
> way to test this behavior in a given class. An ordered collection has
> the exact same interface as an unordered collection (e.g, dict and
> OrderedDict), other than a _promise_ of the API that the order in which
> this collection will be iterated has some sort of meaning (In
> OrderedDict, it is the order in which keys were added to it.)
>
>
> As examples, set, frozenset, dict and defaultdict should *not* be
> considered as ordered. list, OrderedDict, deque and tuple should be
> considered ordered.

I just wanted to offer this idea. I have two use cases:

1. Human-readable output. If the collection is not ordered, it is 
preferable to sort it before output. pprint and testing frameworks will 
benefit from this.

2. Serialization. If the mapping is not ordered, we can serialize its 
content as dict (that can be more efficient), otherwise we have to 
serialize it as a sequence of key-value pairs.

Actually we need only two functions: test if the collection is ordered, 
and a way to register the class as ordered (or unordered).


From mistersheik at gmail.com  Sat Nov  7 12:19:57 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Sat, 07 Nov 2015 17:19:57 +0000
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <n1lb1s$bi6$1@ger.gmane.org>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <n1lb1s$bi6$1@ger.gmane.org>
Message-ID: <CAA68w_m8SfEa_95XMKp2SZgv2GShydkHYFVuH-rxuixdJm_ujg@mail.gmail.com>

FYI: 1. might not work since the elements in a dict may not be sortable.

On Sat, Nov 7, 2015 at 12:06 PM Serhiy Storchaka <storchaka at gmail.com>
wrote:

> On 07.11.15 00:11, Amir Rachum wrote:
> > I am suggesting the addition of a collections abstract base class called
> > "Ordered". Its meaning is that a collection's iteration order is part of
> > its API. The bulk of this mail describes a use case for this. The reason
> > I believe that such abstract base class is required is that there is no
> > way to test this behavior in a given class. An ordered collection has
> > the exact same interface as an unordered collection (e.g, dict and
> > OrderedDict), other than a _promise_ of the API that the order in which
> > this collection will be iterated has some sort of meaning (In
> > OrderedDict, it is the order in which keys were added to it.)
> >
> >
> > As examples, set, frozenset, dict and defaultdict should *not* be
> > considered as ordered. list, OrderedDict, deque and tuple should be
> > considered ordered.
>
> I just wanted to offer this idea. I have two use cases:
>
> 1. Human-readable output. If the collection is not ordered, it is
> preferable to sort it before output. pprint and testing frameworks will
> benefit from this.
>
> 2. Serialization. If the mapping is not ordered, we can serialize its
> content as dict (that can be more efficient), otherwise we have to
> serialize it as a sequence of key-value pairs.
>
> Actually we need only two functions: test if the collection is ordered,
> and a way to register the class as ordered (or unordered).
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/B1Pt76OtJi8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151107/abd07cd8/attachment.html>

From mike at selik.org  Sat Nov  7 14:18:29 2015
From: mike at selik.org (Michael Selik)
Date: Sat, 07 Nov 2015 19:18:29 +0000
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com>
References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl>
 <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <CADqi796TzPSvoBoxVsDyse4HOgMQDYEieZWjnBrOw5sVgFDugA@mail.gmail.com>

Sjoerd, glad we agree, happy to help.

In case someone is still interested in making a patch, I'd like to clarify
why it makes me nervous.

Though Sjoerd led with a TL;DR that summarized his suggestion as a swap of
isinstance for issubclass, a better summary would have been: "I prefer more
specific exception types". The proposed solution was to enable custom
exception types that conditionally insert themselves into the class
hierarchy depending on data stored on a raised exception instance.

Unfortunately, that solution requires changing an obscure error handling
feature. As Sjoerd noted when he sent the first message in this thread, it
is possible this would introduce a backwards-incompatibility. Perhaps no
one in the standard library or major public projects would write such weird
code as to get tripped up by the proposed change, but it's always possible
that someone out there is using this feature for an important project. I
think we've learned the lesson from the long slog to Python 3 adoption (and
how Perl 6 harmed that community, and countless other examples) that
backwards compatibility is of utmost importance.

The current code pattern for a single try/except is to catch the general
exception and have an if-condition in the except block.That's a frequent
and readable practice.

    try:
        cursor.execute(query)
    except sqlite3.OperationalError as str(e):
        if 'syntax error' in str(e):
            print('Your SQL is bad')

If that code or similar code appears many times, a good way to refactor
with the current features of Python is to use a context manager.

    class SqlSyntaxErrorHandler:
        def __enter__(self):
            return self
        def __exit__(self, etyp, einstance, etraceback):
            if (issubclass(etyp, sqlite3.OperationalError)
                and 'syntax error' in str(einstance)):
                    print('Your SQL is bad')

    with SqlSyntaxErrorHandler:
        cursor.execute(first_query)

    with SqlSyntaxErrorHandler:
        cursor.execute(second_query)

While context managers may not be a beginner topic, they are typically
easier to read and debug than using metaclasses for conditional runtime
inheritance. Even if there would be some benefit to the metaclass style,
it's not worth the risk of changing current error handling behavior.

-- Michael


On Fri, Nov 6, 2015, 2:49 AM Andrew Barnert via Python-ideas <
python-ideas at python.org> wrote:

> On Thursday, November 5, 2015 9:57 PM, "sjoerdjob at sjec.nl" <
> sjoerdjob at sjec.nl> wrote:
>
> >>  Andrew Barnert via Python-ideas writes:
> >>   > Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
> >>
> >>   > > It might seem troublesome to change these locations to raise
> >>   > > actual instances instead of strings, but those might actually
> >>   > > just be hidden bugs just the same.
> >>
> >>  No, I'm sure mostly they're not.  They're working code that has
> > been
> >>  there since the early days, perhaps a few from the 2.x days in
> >>  modules.  Python is generally very conservative about changing working
> >>  code.
> >>
> >>   > > Do you think anyone would object if I'd go over the code
> > looking
> >>   > > for places where type(tstate->cur_value) !=
> > tstate->cur_type and
> >>   > > propose patches for these?
> >>
> >>  I think, yes, though I don't know enough about it to object myself.
> >>  Code churn of this kind is generally frowned upon, unless you can show
> >>  that the risk of hidden bugs of the kind you refer to above is
> >>  substantially greater than the risk that *you* will introduce bugs
> >>  with your patches.
> >
> > After digging more into the code, I concur. It's quite easy to make some
> > changes that break some stuff in this logic.
>
> Looking at this a bit further, I think there may be something you can do
> that gets everything you actually wanted, without changes all over the
> place and serious risks.
>
> Basically, it's just two pretty small changes in errors.c:
>
> * Change PyErr_GivenExceptionMatches so that if given a value that's an
> instance of an exception type (PyExceptionInstance_Check(err) is true),
> instead of just getting its class and then ignoring the value, it does a
> standard isinstance check on the value.
>
> * Change PyErr_ExceptionMatches so that if there is an exception value,
> and its type matches the exception type, it is passed to
> PyErr_GivenExceptionMatches; otherwise, the exception type is passed.
>
>
> So, all Python code that raises and handles exceptions will get the
> benefit of isinstance checks. Any new C code that wants that benefit can
> get it very easily (only raise with PyErr_SetObject or something that uses
> it like PyErr_SetFromErrno; only handle with PyErr_GivenExceptionMatches).
> Any existing C code that skips creating an instance will continue to do so,
> and to get the same fast-path check used today, except for one extra call
> to PyExceptionInstance_Check(tstate->curexc_value).
>
> Of course it's not just that small patch to errors.c. You also need tests
> to exercise the new functionality. And a docs patch for the Exceptions
> section of the C API. And also a docs patch to the Exceptions section in
> the Reference. But no other code patches.
>
> And you'd definitely need to performance-test two things: First, that the
> extra PyExceptionInstance_Check doesn't slow down fast-path handling (e.g.,
> in an empty for loops). Second, whatever performance tests your original
> proposal would have needed, to make sure the isinstance check doesn't
> noticeably slow down pure-Python code that handles a lot of exceptions.
>
>
> I think the only C API docs change needed is to PyErr_ExceptionMatches.
> However, a note explaining how to make sure to get and/or to bypass the
> isinstance check as desired might be useful.
>
> For the reference docs, I think just this note at the end of the section:
>
> > Note: Exceptions raised and handled by Python code use the same
> mechanism as the isinstance function for determining whether an exception
> matches. However, exceptions raised or handled by the implementation (or
> implementation-level extensions) may bypass that and check the type
> directly.
>
>
> > New in 3.6: In previous versions of Python, exception handling always
> bypassed the isinstance function and checked the type directly.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151107/f1073fea/attachment-0001.html>

From tjreedy at udel.edu  Sat Nov  7 16:29:04 2015
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 7 Nov 2015 16:29:04 -0500
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <CADqi796TzPSvoBoxVsDyse4HOgMQDYEieZWjnBrOw5sVgFDugA@mail.gmail.com>
References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl>
 <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com>
 <CADqi796TzPSvoBoxVsDyse4HOgMQDYEieZWjnBrOw5sVgFDugA@mail.gmail.com>
Message-ID: <n1lqf5$h9h$1@ger.gmane.org>

On 11/7/2015 2:18 PM, Michael Selik wrote:

> The current code pattern for a single try/except is to catch the general
> exception and have an if-condition in the except block.That's a frequent
> and readable practice.
>
>      try:
>          cursor.execute(query)
>      except sqlite3.OperationalError as str(e):
>          if 'syntax error' in str(e):
>              print('Your SQL is bad')
>
> If that code or similar code appears many times, a good way to refactor
> with the current features of Python is to use a context manager.
>
>      class SqlSyntaxErrorHandler:
>          def __enter__(self):
>              return self
>          def __exit__(self, etyp, einstance, etraceback):
>              if (issubclass(etyp, sqlite3.OperationalError)
>                  and 'syntax error' in str(einstance)):
>                      print('Your SQL is bad')

I like this example of how to use the error handling ability of __exit__ 
to set up an 'error handling context.  This is much clearer than the 
examples in the pep.  However, I checked and
issubclass(None, whatever) does not work.  True should be returned when 
the exception is handled.

     def __exit__(self, etyp, einstance, etraceback):
         if (etyp and issubclass(etyp, sqlite3.OperationalError)
                  and 'syntax error' in str(einstance)):
             print('Your SQL is bad')
             return True

>      with SqlSyntaxErrorHandler:
>          cursor.execute(first_query)
>
>      with SqlSyntaxErrorHandler:
>          cursor.execute(second_query)

-- 
Terry Jan Reedy


From abarnert at yahoo.com  Sat Nov  7 16:41:09 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 7 Nov 2015 13:41:09 -0800
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
Message-ID: <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>

On Nov 7, 2015, at 06:41, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> 
> Andrew Barnert writes:
>> On Nov 6, 2015, at 21:00, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> 
>>> This ABC would be a tar pit of recriminations between class authors
>>> and class clients.  
> 
>> I think you've missed the point of his proposal. It doesn't matter
>> _what_ the order is
> 
> That's precisely what I see as problematic, unless the author of the
> subclass is the same as the author of the code using the subclass.
> 
>> But, whether the proposal is useful or not, it is sufficiently
>> specified.
> 
> I wrote inaccurately, I guess.  My contention is that it's more
> dangerous than useful as currently specified but that could be
> reversed with additional restrictions.

Not very useful, sure?I argued that myself?but how is it dangerous? The only risk here is the risk that comes from adding more code, docs, and complexity to the stdlib.

>> How would you define the comparison function for OrderedDict in a
>> way that makes any semantic sense?  Or, for that matter, list?
> 
> Again, bad nomenclature, sorry.[1]  I should have used "index".  For
> this application, I have in mind
> 
> class Ordered:
> 
>    def index(self, key):
>        return list(z for z in self).index(key)
> 
>    def compare(self, key1, key2):
>        return self.index(key1) < self.index(key2)

For, How is that at all useful? What code can you imagine that needs to compare elements from an arbitrary iterable, and would rather have a comparison that's linear in the length of the iterable than none at all?

Second, why list(z for z in self) instead of just list(self) or [z for z in self]? It seems like you're deliberately trying to find the slowest and least straightforward way of implementing it?

But, most importantly, if you want an index method, why aren't you just requiring Sequence, which has one? The notion of index implies the notion of random accessibility?which is exactly what Sequence adds on top of Sized Iterable Container.

Of course there are types for which compare might exist but index might not (e.g., a range of reals, or a set that's only partially ordered), but they're not iterables. (Well, a partially ordered set could Iterate in partially-arbitrary order, but then it's not Ordered in the OP's sense.)

His proposal really does specify exactly what he wants, it's dead simple, and it does all he asks of it. The only question is whether anyone needs it (and, even if they do, whether it needs to be in the stdlib).

> as the default implementation (except that for Amir's application it's
> .index() that's interesting, not .compare() -- .compare() could be
> omitted I guess, fn. [1] applies).  The problem with list would be
> that it can have multiple entries of the same value that are not
> adjacent, of course, but then it wouldn't work for Amir's application
> anyway.  This is part of what I have in mind when I say
> "underspecified".

Well, yes, what he _really_ requires here is an ordered, _unique_, finite, repeatable iterable. Adding a test for Ordered is clearly insufficient for that, and doesn't seem particularly useful. But again, that doesn't mean that Ordered is underspecified in any way; it just means he doesn't have a good use case for it.

> I don't think thought about how this collections.abc.Ordered should
> interact with objects that could be considered to represent multisets.
> Do you?

I'm not sure what you're asking here. But clearly list, OrderedCounter, and a SortedMultiSet type are all ordered, and can all be used to represent multisets in rather obvious ways. And the only way his proposed Ordered has to interact with them is that they all inherit from or get registered with the Ordered ABC (and aren't lying about it).

Of course your proposal for some kind of sorting-related ABC that generates the sort order from indexes would have a problem with multisets, and especially with list as you mentioned above, but that has nothing to do with his proposal.

>> Surely OrderedDict and list are perfectly reasonable types for
>> __slots__, despite the fact that their comparison function doesn't
>> exist.
> 
> If given the above renaming you still say it doesn't exist, I don't
> understand what you're trying to say.

OK, they're perfectly reasonable types for __slots__, despite the fact that no useful, acceptably efficient, or in any way desirable comparison function exists. Better?

> Footnotes: 
> [1]  As an excuse for the poor nomenclature, in my field I have to
> worry about (pre-) ordered structures for which an index function does
> not exist, thus "comparison function".

From mike at selik.org  Sat Nov  7 17:28:49 2015
From: mike at selik.org (Michael Selik)
Date: Sat, 07 Nov 2015 22:28:49 +0000
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <n1lqf5$h9h$1@ger.gmane.org>
References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl>
 <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com>
 <CADqi796TzPSvoBoxVsDyse4HOgMQDYEieZWjnBrOw5sVgFDugA@mail.gmail.com>
 <n1lqf5$h9h$1@ger.gmane.org>
Message-ID: <CADqi796nQQ3qVDDVLJ5Lz+SBnHxMrW2hG+9=MWqW+mFAG9wcfA@mail.gmail.com>

Sorry about that. I also forgot to instantiate the class. Here's
functioning code...

    class SqlSyntaxErrorHandler:
        def __enter__(self):
            return self
        def __exit__(self, etyp, einstance, etraceback):
            if (etyp is not None
                and issubclass(etyp, sqlite3.OperationalError)
                and 'syntax error' in str(einstance)):
                    print('Your SQL is bad')
                    return True

    with SqlSyntaxErrorHandler():
        cursor.execute(query)


On Sat, Nov 7, 2015 at 4:29 PM Terry Reedy <tjreedy at udel.edu> wrote:

> On 11/7/2015 2:18 PM, Michael Selik wrote:
>
> > The current code pattern for a single try/except is to catch the general
> > exception and have an if-condition in the except block.That's a frequent
> > and readable practice.
> >
> >      try:
> >          cursor.execute(query)
> >      except sqlite3.OperationalError as str(e):
> >          if 'syntax error' in str(e):
> >              print('Your SQL is bad')
> >
> > If that code or similar code appears many times, a good way to refactor
> > with the current features of Python is to use a context manager.
> >
> >      class SqlSyntaxErrorHandler:
> >          def __enter__(self):
> >              return self
> >          def __exit__(self, etyp, einstance, etraceback):
> >              if (issubclass(etyp, sqlite3.OperationalError)
> >                  and 'syntax error' in str(einstance)):
> >                      print('Your SQL is bad')
>
> I like this example of how to use the error handling ability of __exit__
> to set up an 'error handling context.  This is much clearer than the
> examples in the pep.  However, I checked and
> issubclass(None, whatever) does not work.  True should be returned when
> the exception is handled.
>
>      def __exit__(self, etyp, einstance, etraceback):
>          if (etyp and issubclass(etyp, sqlite3.OperationalError)
>                   and 'syntax error' in str(einstance)):
>              print('Your SQL is bad')
>              return True
>
> >      with SqlSyntaxErrorHandler:
> >          cursor.execute(first_query)
> >
> >      with SqlSyntaxErrorHandler:
> >          cursor.execute(second_query)
>
> --
> Terry Jan Reedy
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151107/7ee1010e/attachment.html>

From mike at selik.org  Sat Nov  7 17:44:08 2015
From: mike at selik.org (Michael Selik)
Date: Sat, 07 Nov 2015 22:44:08 +0000
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
Message-ID: <CADqi79729CFhrLUW3axaQ2Lyz31j9DiJaMKnLARc4UmMLH7aWA@mail.gmail.com>

Hi Amir,

Could you (or someone else) help me understand the use case for this base
class? Rather than describing the use case, could you point me to somewhere
in the standard library or a public project where it would clarify the code?

I currently share Andrew's concern -- knowing a thing is ordered does not
help us understand *how* it's ordered. Therefore, despite an `Ordered` base
class, we would still need to rely on error messages, tests, and
documentation. As he said, calling this "dangerous" is not hyperbole but a
proper description of how any change to the standard library will increase
the maintenance burden, may reduce usability/learnability, and may have
unpredictable consequences.

-- Michael

On Sat, Nov 7, 2015 at 4:41 PM Andrew Barnert via Python-ideas <
python-ideas at python.org> wrote:

> On Nov 7, 2015, at 06:41, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> >
> > Andrew Barnert writes:
> >> On Nov 6, 2015, at 21:00, Stephen J. Turnbull <stephen at xemacs.org>
> wrote:
> >
> >>> This ABC would be a tar pit of recriminations between class authors
> >>> and class clients.
> >
> >> I think you've missed the point of his proposal. It doesn't matter
> >> _what_ the order is
> >
> > That's precisely what I see as problematic, unless the author of the
> > subclass is the same as the author of the code using the subclass.
> >
> >> But, whether the proposal is useful or not, it is sufficiently
> >> specified.
> >
> > I wrote inaccurately, I guess.  My contention is that it's more
> > dangerous than useful as currently specified but that could be
> > reversed with additional restrictions.
>
> Not very useful, sure?I argued that myself?but how is it dangerous? The
> only risk here is the risk that comes from adding more code, docs, and
> complexity to the stdlib.
>
> >> How would you define the comparison function for OrderedDict in a
> >> way that makes any semantic sense?  Or, for that matter, list?
> >
> > Again, bad nomenclature, sorry.[1]  I should have used "index".  For
> > this application, I have in mind
> >
> > class Ordered:
> >
> >    def index(self, key):
> >        return list(z for z in self).index(key)
> >
> >    def compare(self, key1, key2):
> >        return self.index(key1) < self.index(key2)
>
> For, How is that at all useful? What code can you imagine that needs to
> compare elements from an arbitrary iterable, and would rather have a
> comparison that's linear in the length of the iterable than none at all?
>
> Second, why list(z for z in self) instead of just list(self) or [z for z
> in self]? It seems like you're deliberately trying to find the slowest and
> least straightforward way of implementing it?
>
> But, most importantly, if you want an index method, why aren't you just
> requiring Sequence, which has one? The notion of index implies the notion
> of random accessibility?which is exactly what Sequence adds on top of Sized
> Iterable Container.
>
> Of course there are types for which compare might exist but index might
> not (e.g., a range of reals, or a set that's only partially ordered), but
> they're not iterables. (Well, a partially ordered set could Iterate in
> partially-arbitrary order, but then it's not Ordered in the OP's sense.)
>
> His proposal really does specify exactly what he wants, it's dead simple,
> and it does all he asks of it. The only question is whether anyone needs it
> (and, even if they do, whether it needs to be in the stdlib).
>
> > as the default implementation (except that for Amir's application it's
> > .index() that's interesting, not .compare() -- .compare() could be
> > omitted I guess, fn. [1] applies).  The problem with list would be
> > that it can have multiple entries of the same value that are not
> > adjacent, of course, but then it wouldn't work for Amir's application
> > anyway.  This is part of what I have in mind when I say
> > "underspecified".
>
> Well, yes, what he _really_ requires here is an ordered, _unique_, finite,
> repeatable iterable. Adding a test for Ordered is clearly insufficient for
> that, and doesn't seem particularly useful. But again, that doesn't mean
> that Ordered is underspecified in any way; it just means he doesn't have a
> good use case for it.
>
> > I don't think thought about how this collections.abc.Ordered should
> > interact with objects that could be considered to represent multisets.
> > Do you?
>
> I'm not sure what you're asking here. But clearly list, OrderedCounter,
> and a SortedMultiSet type are all ordered, and can all be used to represent
> multisets in rather obvious ways. And the only way his proposed Ordered has
> to interact with them is that they all inherit from or get registered with
> the Ordered ABC (and aren't lying about it).
>
> Of course your proposal for some kind of sorting-related ABC that
> generates the sort order from indexes would have a problem with multisets,
> and especially with list as you mentioned above, but that has nothing to do
> with his proposal.
>
> >> Surely OrderedDict and list are perfectly reasonable types for
> >> __slots__, despite the fact that their comparison function doesn't
> >> exist.
> >
> > If given the above renaming you still say it doesn't exist, I don't
> > understand what you're trying to say.
>
> OK, they're perfectly reasonable types for __slots__, despite the fact
> that no useful, acceptably efficient, or in any way desirable comparison
> function exists. Better?
>
> > Footnotes:
> > [1]  As an excuse for the poor nomenclature, in my field I have to
> > worry about (pre-) ordered structures for which an index function does
> > not exist, thus "comparison function".
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151107/817160a4/attachment-0001.html>

From stephen at xemacs.org  Sun Nov  8 07:34:37 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 8 Nov 2015 21:34:37 +0900
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
Message-ID: <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>

Andrew Barnert writes:

 > Not very useful, sure?I argued that myself?but how is it
 > dangerous? The only risk here is the risk that comes from adding
 > more code, docs, and complexity to the stdlib.

This proposal requires that users not only read the docs in the
stdlib, but also the docs of each class, perhaps each object, using
the functionality.  Asking users to read docs thoroughly is a recipe
for bugs, especially since the order used is likely to be documented
as "the natural order", and people will have different ideas about
that.  This is the "complication" the Zen warns about, as well as
being "implicit".  You may not consider that dangerous, but I do.

And on top of that, not only doesn't it provide any guarantees, you've
pretty much convinced me it's impossible to do so.  If that's not an
attractive nuisance, what is?

 > But, most importantly, if you want an index method, why aren't you
 > just requiring Sequence, which has one?

Because "ordered sets" and "ordered dicts" aren't Sequences.

But don't ask me, ask the OP.  As I wrote earlier, no use case for
set- or dict-valued __slots__ has been given.  I have nothing invested
in anything I wrote except that.  I'm happy to concede that everything
else I wrote was boneheaded, retract it, and apologize for wasting
your time.


From guido at python.org  Sun Nov  8 13:45:10 2015
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 Nov 2015 10:45:10 -0800
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>

I'm warming up slightly to the idea of this ABC.

I've re-read Amir's post, and if I found myself in his situation I would
have used a combination of documenting that __slots__ needs to be ordered
and at runtime only checking for a few common definitely-bad cases. E.g.
for exactly set or dict (not subclasses, since OrderedDict inherits from
dict) would cover most of the common mistakes: most people will use a
literal in their __slots__ definition so we just need to watch out for the
common literals. Those users who are sophisticated enough to use some
advanced mapping type in their __slots__ should just be expected to deal
with the consequences.

But the Ordered ABC, while not essential (unless you're a perfectionist, in
which case you're in the wrong language community anyways :-) still fills a
niche.

The Ordered ABC should have no additional methods, and no default
implementations. I think it should apply to collections but not to
iterators. It should apply at the level of the read-only interface.
Sequence is always Ordered. Mapping and Set are not by default, but can
have it added. OrderedDict is the prime (maybe only) example -- it's a
MutableMapping and Ordered. We might eventually get an OrderedSet.

A sorted set or mapping (e.g. one implemented using some kind of tree)
should also be considered Ordered, even though otherwise this is a totally
different topic -- while some other languages or branches of math use
"order" to refer to sorting (e.g. "partial ordering"), in Python we make a
distinction: on the one hand there's sorted() and list.sort(), and on the
other hand there's OrderedDict.

So, I think that the Ordered ABC proposed here is totally well-defined and
mildly useful. It may be somewhat confusing (because many people when they
first encounter the term "ordered" they think it's about sorting -- witness
some posts in this thread). The use cases are not very important. I guess
I'm -0 on adding it -- if better use cases than Amir's are developed I
might change to +0. (I don't care much about Serhiy's use cases.)

Sorry for the rambling.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151108/a2b70687/attachment.html>

From amir at rachum.com  Sun Nov  8 14:06:03 2015
From: amir at rachum.com (Amir Rachum)
Date: Sun, 8 Nov 2015 21:06:03 +0200
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
Message-ID: <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>

I would like to expand on my original use case, which I hope will provide a
more persuading argument.
As part of BasicStruct I intend to allow the use of mapping types as
__slots__, with the semantics of default values.
So it'll look something like this:

    class Point(BasicStruct):
        __slots__ = {'x': 5, 'y': 7}

My fear is that someone will just use a dict literal as stated above and
try to create an object with positional arguments:

    Point(0)

The problem is that iteration order over a dict is not guaranteed, so this
might initialize Point(x=0, y=7) like the use intends, or Point(x=5, y=0).
So in this case I want to disable positional arguments. However, positional
arguments for initializing struct are pretty convenient, so I would like to
make that available as far as possible. An implementation using Ordered
might look like this:

    class BasicStruct(object):
        """Class for holding struct-like objects."""

        __slots__ = ()  # should be extended by deriving classes

        def __init__(self, *args, **kwargs):
            default_values = isinstance(self.__slots__, Mapping)
            ordered = isinstance(self.__slots__, Ordered)

            if args and not ordered:
                raise ValueError("Can't pass non-keyword arguments to {},
since "
                                 "__slots__ was declared with an unordered "

 "iterable.".format(self.__class__.__name__))

            arg_pairs = zip(self.__slots__, args)
            for key, value in chain(arg_pairs, six.iteritems(kwargs)):
                setattr(self, key, value)

            for key in self.__slots__:
                if not hasattr(self, key):
                    default_value = None
                    if default_values:
                        default_value = self.__slots__[key]
                    setattr(self, key, default_value)


This will allow users who are interested in positional argument
initialization to use an Ordered mapping (such as OrderedDict, or a custom
collection).
I would like to emphasize again that the most compelling reason for me that
this should be part of the stdlib is that there is no workaround for this
that also allows users to use ordered custom collections. There is no way
to check for "promised ordering" in any functional manner.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151108/8fff1fee/attachment.html>

From mistersheik at gmail.com  Sun Nov  8 14:10:01 2015
From: mistersheik at gmail.com (Neil Girdhar)
Date: Sun, 08 Nov 2015 19:10:01 +0000
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
Message-ID: <CAA68w_=Dc39jFVLJadwFTQCgy3Jq_qH=g9vxBi917LMnj4HXBQ@mail.gmail.com>

On Sun, Nov 8, 2015 at 2:06 PM Amir Rachum <amir at rachum.com> wrote:

> I would like to expand on my original use case, which I hope will provide
> a more persuading argument.
> As part of BasicStruct I intend to allow the use of mapping types as
> __slots__, with the semantics of default values.
> So it'll look something like this:
>
>     class Point(BasicStruct):
>         __slots__ = {'x': 5, 'y': 7}
>
> My fear is that someone will just use a dict literal as stated above and
> try to create an object with positional arguments:
>
>     Point(0)
>
> The problem is that iteration order over a dict is not guaranteed, so this
> might initialize Point(x=0, y=7) like the use intends, or Point(x=5, y=0).
> So in this case I want to disable positional arguments. However, positional
> arguments for initializing struct are pretty convenient, so I would like to
> make that available as far as possible. An implementation using Ordered
> might look like this:
>
>     class BasicStruct(object):
>         """Class for holding struct-like objects."""
>
>         __slots__ = ()  # should be extended by deriving classes
>
>         def __init__(self, *args, **kwargs):
>             default_values = isinstance(self.__slots__, Mapping)
>             ordered = isinstance(self.__slots__, Ordered)
>
>             if args and not ordered:
>                 raise ValueError("Can't pass non-keyword arguments to {},
> since "
>                                  "__slots__ was declared with an unordered
> "
>
>  "iterable.".format(self.__class__.__name__))
>
>             arg_pairs = zip(self.__slots__, args)
>             for key, value in chain(arg_pairs, six.iteritems(kwargs)):
>                 setattr(self, key, value)
>
>             for key in self.__slots__:
>                 if not hasattr(self, key):
>                     default_value = None
>                     if default_values:
>                         default_value = self.__slots__[key]
>                     setattr(self, key, default_value)
>
>
> This will allow users who are interested in positional argument
> initialization to use an Ordered mapping (such as OrderedDict, or a custom
> collection).
> I would like to emphasize again that the most compelling reason for me
> that this should be part of the stdlib is that there is no workaround for
> this that also allows users to use ordered custom collections. There is no
> way to check for "promised ordering" in any functional manner.
>

Did you know that you can register your custom collection with an abc?

> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/B1Pt76OtJi8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/B1Pt76OtJi8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151108/adf690d4/attachment-0001.html>

From sjoerdjob at sjec.nl  Sun Nov  8 16:12:31 2015
From: sjoerdjob at sjec.nl (Sjoerd Job Postmus)
Date: Sun, 8 Nov 2015 22:12:31 +0100
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
Message-ID: <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl>



> On 8 Nov 2015, at 20:06, Amir Rachum <amir at rachum.com> wrote:
> 
> I would like to expand on my original use case, which I hope will provide a more persuading argument.
> As part of BasicStruct I intend to allow the use of mapping types as __slots__, with the semantics of default values.
> So it'll look something like this:
> 
>     class Point(BasicStruct):
>         __slots__ = {'x': 5, 'y': 7}

So instead they'll write
    __slots__ = OrderedDict({'x': 5, 'y': 7})
Causing the same issues?

> 
> My fear is that someone will just use a dict literal as stated above and try to create an object with positional arguments:
> 
>     Point(0)
> 
> The problem is that iteration order over a dict is not guaranteed, so this might initialize Point(x=0, y=7) like the use intends, or Point(x=5, y=0). So in this case I want to disable positional arguments. However, positional arguments for initializing struct are pretty convenient, so I would like to make that available as far as possible. An implementation using Ordered might look like this:
> 
>     class BasicStruct(object):
>         """Class for holding struct-like objects."""
> 
>         __slots__ = ()  # should be extended by deriving classes
> 
>         def __init__(self, *args, **kwargs):
>             default_values = isinstance(self.__slots__, Mapping)
>             ordered = isinstance(self.__slots__, Ordered)
> 
>             if args and not ordered:
>                 raise ValueError("Can't pass non-keyword arguments to {}, since "
>                                  "__slots__ was declared with an unordered "
>                                  "iterable.".format(self.__class__.__name__))
> 
>             arg_pairs = zip(self.__slots__, args)
>             for key, value in chain(arg_pairs, six.iteritems(kwargs)):
>                 setattr(self, key, value)
> 
>             for key in self.__slots__:
>                 if not hasattr(self, key):
>                     default_value = None
>                     if default_values:
>                         default_value = self.__slots__[key]
>                     setattr(self, key, default_value)
> 
> 
> This will allow users who are interested in positional argument initialization to use an Ordered mapping (such as OrderedDict, or a custom collection).
> I would like to emphasize again that the most compelling reason for me that this should be part of the stdlib is that there is no workaround for this that also allows users to use ordered custom collections. There is no way to check for "promised ordering" in any functional manner.

Besides the fact that you can create that base-class yourself, and register OrderedDict as an instance of it already, I think there's a better solution to your problem (not quite tested, but should work):

1. Create a Field class which maintains a creation-counter to be able to query ordering.
2. In the __init__ check against the creation ordering.

Added bonus: you can choose to have some attributes have a default value, and others not having a default value. 

If you're willing to do some metaclass trickery, you could even end up with a syntax like:

    class Point(Struct):
        x = Field(5)
        y = Field(7)
        z = Field()  # we force the user to supply the z value, as it has no default. 

And generate the __slots__ from that. Maybe look at Django's ModelBase implementation, and how it uses metaclasses for that?

Even more added bonus would be defining special Field sub-classes which also check the type/value of the supplied values (NumericField raising ValueError when you pass it the string "foo").

Not really sure what the added benefit of an Ordered base-class would be here.

> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151108/4f2a89d5/attachment.html>

From storchaka at gmail.com  Sun Nov  8 17:10:06 2015
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 9 Nov 2015 00:10:06 +0200
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
 <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl>
Message-ID: <n1oh7u$s9j$1@ger.gmane.org>

On 08.11.15 23:12, Sjoerd Job Postmus wrote:
> On 8 Nov 2015, at 20:06, Amir Rachum <amir at rachum.com
> <mailto:amir at rachum.com>> wrote:
>> As part of BasicStruct I intend to allow the use of mapping types as
>> __slots__, with the semantics of default values..
>> So it'll look something like this:
>>
>>     class Point(BasicStruct):
>>         __slots__ = {'x': 5, 'y': 7}
>
> So instead they'll write
>      __slots__ = OrderedDict({'x': 5, 'y': 7})
> Causing the same issues?

Perhaps OrderedDict should reject unordered sources. Hey, here is yet 
one use case!



From abarnert at yahoo.com  Sun Nov  8 19:42:05 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 8 Nov 2015 16:42:05 -0800
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <n1oh7u$s9j$1@ger.gmane.org>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
 <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl> <n1oh7u$s9j$1@ger.gmane.org>
Message-ID: <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com>

On Nov 8, 2015, at 14:10, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 
>> On 08.11.15 23:12, Sjoerd Job Postmus wrote:
>> On 8 Nov 2015, at 20:06, Amir Rachum <amir at rachum.com
>> <mailto:amir at rachum.com>> wrote:
>>> As part of BasicStruct I intend to allow the use of mapping types as
>>> __slots__, with the semantics of default values..
>>> So it'll look something like this:
>>> 
>>>    class Point(BasicStruct):
>>>        __slots__ = {'x': 5, 'y': 7}
>> 
>> So instead they'll write
>>     __slots__ = OrderedDict({'x': 5, 'y': 7})
>> Causing the same issues?
> 
> Perhaps OrderedDict should reject unordered sources. Hey, here is yet one use case!

I've maintained code that does this:

    self.headers = OrderedDict(headers)
    self.origheaders = len(headers)

? so it can later do this:

    altheaders = list(self.headers.items())[self.origheaders:]

Not a great design, but one that exists in the wild, and would be broken by OrderedDict not allowing a dict as an argument.

Also, this wouldn't allow creating an OrderedDict from an empty dict (which seems far less stupid, but I didn't lead with it because I can't remember seeing it in real code).


From guido at python.org  Sun Nov  8 23:28:31 2015
From: guido at python.org (Guido van Rossum)
Date: Sun, 8 Nov 2015 20:28:31 -0800
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
 <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl>
 <n1oh7u$s9j$1@ger.gmane.org>
 <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com>
Message-ID: <CAP7+vJL1vTtx5zXA+aJJL+e=M0c-FBNDYS9RxS9VLvm5pJA5BQ@mail.gmail.com>

So if OrderedDict had always rejected construction from a dict, how would
you have written this?

On Sunday, November 8, 2015, Andrew Barnert via Python-ideas <
python-ideas at python.org> wrote:

> On Nov 8, 2015, at 14:10, Serhiy Storchaka <storchaka at gmail.com
> <javascript:;>> wrote:
> >
> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote:
> >> On 8 Nov 2015, at 20:06, Amir Rachum <amir at rachum.com <javascript:;>
> >> <mailto:amir at rachum.com <javascript:;>>> wrote:
> >>> As part of BasicStruct I intend to allow the use of mapping types as
> >>> __slots__, with the semantics of default values..
> >>> So it'll look something like this:
> >>>
> >>>    class Point(BasicStruct):
> >>>        __slots__ = {'x': 5, 'y': 7}
> >>
> >> So instead they'll write
> >>     __slots__ = OrderedDict({'x': 5, 'y': 7})
> >> Causing the same issues?
> >
> > Perhaps OrderedDict should reject unordered sources. Hey, here is yet
> one use case!
>
> I've maintained code that does this:
>
>     self.headers = OrderedDict(headers)
>     self.origheaders = len(headers)
>
> ? so it can later do this:
>
>     altheaders = list(self.headers.items())[self.origheaders:]
>
> Not a great design, but one that exists in the wild, and would be broken
> by OrderedDict not allowing a dict as an argument.
>
> Also, this wouldn't allow creating an OrderedDict from an empty dict
> (which seems far less stupid, but I didn't lead with it because I can't
> remember seeing it in real code).
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org <javascript:;>
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
--Guido (mobile)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151108/fab86b9b/attachment-0001.html>

From sjoerdjob at sjec.nl  Mon Nov  9 00:30:51 2015
From: sjoerdjob at sjec.nl (Sjoerd Job Postmus)
Date: Mon, 9 Nov 2015 06:30:51 +0100
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <CADqi796TzPSvoBoxVsDyse4HOgMQDYEieZWjnBrOw5sVgFDugA@mail.gmail.com>
References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl>
 <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com>
 <CADqi796TzPSvoBoxVsDyse4HOgMQDYEieZWjnBrOw5sVgFDugA@mail.gmail.com>
Message-ID: <AB61E1C1-6376-4A97-AE0E-B1527F6AAEC9@sjec.nl>

Hi again everybody,

So I've had a couple of days extra to think about it, and yes: it might still break some things, but I expect extremely little.

After the discussions here, I came to think it would require a lot of changes to the codebase, and decided to let it rest. 

Last night, it dawned to me that there might be a fix with very little code-changes needed:

* add an extra operator `PyCmp_EXC_VAL_MATCH`
* in the compiler, change `DUP_TOP` to `DUP_TOP_TWO` just above the generation of the compare. I don't have the exact location, as I'm typing on my phone right now.
* create an extra function PyErr_GivenExceptionValueMatches which calls PyObject_isInstance
* call that from ceval for the `PyCmp_EXC_VAL_MATCH.
* most important: update the docs.

A little bit of code duplication, but not that much.

As for the scary part.
* existing code: Because this would only introduce changes for Python code, existing C-extensions would not be affected. Python code would, but I think the fallout would be very little, as the only noticeable changes are when a class overrides its __instancecheck__, which I believe is too negligible a corner-case.
* performance: with the changes already proposed for the moving towards __subclasscheck__ from the current behaviour, there are some optimizations included which should also benefit this proposal. Exact measurements are of course needed.

One of the changes I do see needed in Python-code is having the unit-testing libraries change their 'assertRaises' code to reflect the new behavior.

I agree this might cause a backward-compatibility issue, however after some thinking about it, I don't think it will affect more than 1 percent of already existing code, probably even way less. I don't have any numbers to support that feeling, though.

As far as an important project: if you're depending on obscure features, you should probably be ready to expect changes there. Yes, the current behavior is precisely documented, but in nearly all of the cases the actual implementation and the proposed change agree, unless one did magic.

I made the changes I listed above and ran `make; make test`, and got only 1 error due to socket.socket running out of memory or something. I'll re-run the tests on a clean checkout as well, but for now I think it's unrelated.

Regards,
Sjoerd Job Postmus

> On 7 Nov 2015, at 20:18, Michael Selik <mike at selik.org> wrote:
> 
> Sjoerd, glad we agree, happy to help.
> 
> In case someone is still interested in making a patch, I'd like to clarify why it makes me nervous.
> 
> Though Sjoerd led with a TL;DR that summarized his suggestion as a swap of isinstance for issubclass, a better summary would have been: "I prefer more specific exception types". The proposed solution was to enable custom exception types that conditionally insert themselves into the class hierarchy depending on data stored on a raised exception instance.
> 
> Unfortunately, that solution requires changing an obscure error handling feature. As Sjoerd noted when he sent the first message in this thread, it is possible this would introduce a backwards-incompatibility. Perhaps no one in the standard library or major public projects would write such weird code as to get tripped up by the proposed change, but it's always possible that someone out there is using this feature for an important project. I think we've learned the lesson from the long slog to Python 3 adoption (and how Perl 6 harmed that community, and countless other examples) that backwards compatibility is of utmost importance.
> 
> The current code pattern for a single try/except is to catch the general exception and have an if-condition in the except block.That's a frequent and readable practice.
> 
>     try:
>         cursor.execute(query)
>     except sqlite3.OperationalError as str(e):
>         if 'syntax error' in str(e):
>             print('Your SQL is bad')
> 
> If that code or similar code appears many times, a good way to refactor with the current features of Python is to use a context manager.
> 
>     class SqlSyntaxErrorHandler:
>         def __enter__(self):
>             return self
>         def __exit__(self, etyp, einstance, etraceback):
>             if (issubclass(etyp, sqlite3.OperationalError)
>                 and 'syntax error' in str(einstance)):
>                     print('Your SQL is bad')
> 
>     with SqlSyntaxErrorHandler:
>         cursor.execute(first_query)
> 
>     with SqlSyntaxErrorHandler:
>         cursor.execute(second_query)
> 
> While context managers may not be a beginner topic, they are typically easier to read and debug than using metaclasses for conditional runtime inheritance. Even if there would be some benefit to the metaclass style, it's not worth the risk of changing current error handling behavior.
> 
> -- Michael
> 
> 
>> On Fri, Nov 6, 2015, 2:49 AM Andrew Barnert via Python-ideas <python-ideas at python.org> wrote:
>> On Thursday, November 5, 2015 9:57 PM, "sjoerdjob at sjec.nl" <sjoerdjob at sjec.nl> wrote:
>> 
>> >>  Andrew Barnert via Python-ideas writes:
>> >>   > Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
>> >>
>> >>   > > It might seem troublesome to change these locations to raise
>> >>   > > actual instances instead of strings, but those might actually
>> >>   > > just be hidden bugs just the same.
>> >>
>> >>  No, I'm sure mostly they're not.  They're working code that has
>> > been
>> >>  there since the early days, perhaps a few from the 2.x days in
>> >>  modules.  Python is generally very conservative about changing working
>> >>  code.
>> >>
>> >>   > > Do you think anyone would object if I'd go over the code
>> > looking
>> >>   > > for places where type(tstate->cur_value) !=
>> > tstate->cur_type and
>> >>   > > propose patches for these?
>> >>
>> >>  I think, yes, though I don't know enough about it to object myself.
>> >>  Code churn of this kind is generally frowned upon, unless you can show
>> >>  that the risk of hidden bugs of the kind you refer to above is
>> >>  substantially greater than the risk that *you* will introduce bugs
>> >>  with your patches.
>> >
>> > After digging more into the code, I concur. It's quite easy to make some
>> > changes that break some stuff in this logic.
>> 
>> Looking at this a bit further, I think there may be something you can do that gets everything you actually wanted, without changes all over the place and serious risks.
>> 
>> Basically, it's just two pretty small changes in errors.c:
>> 
>> * Change PyErr_GivenExceptionMatches so that if given a value that's an instance of an exception type (PyExceptionInstance_Check(err) is true), instead of just getting its class and then ignoring the value, it does a standard isinstance check on the value.
>> 
>> * Change PyErr_ExceptionMatches so that if there is an exception value, and its type matches the exception type, it is passed to PyErr_GivenExceptionMatches; otherwise, the exception type is passed.
>> 
>> 
>> So, all Python code that raises and handles exceptions will get the benefit of isinstance checks. Any new C code that wants that benefit can get it very easily (only raise with PyErr_SetObject or something that uses it like PyErr_SetFromErrno; only handle with PyErr_GivenExceptionMatches). Any existing C code that skips creating an instance will continue to do so, and to get the same fast-path check used today, except for one extra call to PyExceptionInstance_Check(tstate->curexc_value).
>> 
>> Of course it's not just that small patch to errors.c. You also need tests to exercise the new functionality. And a docs patch for the Exceptions section of the C API. And also a docs patch to the Exceptions section in the Reference. But no other code patches.
>> 
>> And you'd definitely need to performance-test two things: First, that the extra PyExceptionInstance_Check doesn't slow down fast-path handling (e.g., in an empty for loops). Second, whatever performance tests your original proposal would have needed, to make sure the isinstance check doesn't noticeably slow down pure-Python code that handles a lot of exceptions.
>> 
>> 
>> I think the only C API docs change needed is to PyErr_ExceptionMatches. However, a note explaining how to make sure to get and/or to bypass the isinstance check as desired might be useful.
>> 
>> For the reference docs, I think just this note at the end of the section:
>> 
>> > Note: Exceptions raised and handled by Python code use the same mechanism as the isinstance function for determining whether an exception matches. However, exceptions raised or handled by the implementation (or implementation-level extensions) may bypass that and check the type directly.
>> 
>> 
>> > New in 3.6: In previous versions of Python, exception handling always bypassed the isinstance function and checked the type directly.
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151109/371b658a/attachment.html>

From sjoerdjob at sjec.nl  Mon Nov  9 00:41:25 2015
From: sjoerdjob at sjec.nl (sjoerdjob at sjec.nl)
Date: Mon, 9 Nov 2015 06:41:25 +0100
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <AB61E1C1-6376-4A97-AE0E-B1527F6AAEC9@sjec.nl>
References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl>
 <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com>
 <CADqi796TzPSvoBoxVsDyse4HOgMQDYEieZWjnBrOw5sVgFDugA@mail.gmail.com>
 <AB61E1C1-6376-4A97-AE0E-B1527F6AAEC9@sjec.nl>
Message-ID: <6f22ce0f98ae5b4f7a290343bad92564.squirrel@mail.sjec.nl>

> I made the changes I listed above and ran `make; make test`, and got only
> 1 error due to socket.socket running out of memory or something. I'll
> re-run the tests on a clean checkout as well, but for now I think it's
> unrelated.

After running the tests against a clean checkout (without my changes): I
get the same errors from test_socket. So it's unrelated.

> Regards,
> Sjoerd Job Postmus
>
>> On 7 Nov 2015, at 20:18, Michael Selik <mike at selik.org> wrote:
>>
>> Sjoerd, glad we agree, happy to help.
>>
>> In case someone is still interested in making a patch, I'd like to
>> clarify why it makes me nervous.
>>
>> Though Sjoerd led with a TL;DR that summarized his suggestion as a swap
>> of isinstance for issubclass, a better summary would have been: "I
>> prefer more specific exception types". The proposed solution was to
>> enable custom exception types that conditionally insert themselves into
>> the class hierarchy depending on data stored on a raised exception
>> instance.
>>
>> Unfortunately, that solution requires changing an obscure error handling
>> feature. As Sjoerd noted when he sent the first message in this thread,
>> it is possible this would introduce a backwards-incompatibility. Perhaps
>> no one in the standard library or major public projects would write such
>> weird code as to get tripped up by the proposed change, but it's always
>> possible that someone out there is using this feature for an important
>> project. I think we've learned the lesson from the long slog to Python 3
>> adoption (and how Perl 6 harmed that community, and countless other
>> examples) that backwards compatibility is of utmost importance.
>>
>> The current code pattern for a single try/except is to catch the general
>> exception and have an if-condition in the except block.That's a frequent
>> and readable practice.
>>
>>     try:
>>         cursor.execute(query)
>>     except sqlite3.OperationalError as str(e):
>>         if 'syntax error' in str(e):
>>             print('Your SQL is bad')
>>
>> If that code or similar code appears many times, a good way to refactor
>> with the current features of Python is to use a context manager.
>>
>>     class SqlSyntaxErrorHandler:
>>         def __enter__(self):
>>             return self
>>         def __exit__(self, etyp, einstance, etraceback):
>>             if (issubclass(etyp, sqlite3.OperationalError)
>>                 and 'syntax error' in str(einstance)):
>>                     print('Your SQL is bad')
>>
>>     with SqlSyntaxErrorHandler:
>>         cursor.execute(first_query)
>>
>>     with SqlSyntaxErrorHandler:
>>         cursor.execute(second_query)
>>
>> While context managers may not be a beginner topic, they are typically
>> easier to read and debug than using metaclasses for conditional runtime
>> inheritance. Even if there would be some benefit to the metaclass style,
>> it's not worth the risk of changing current error handling behavior.
>>
>> -- Michael
>>
>>
>>> On Fri, Nov 6, 2015, 2:49 AM Andrew Barnert via Python-ideas
>>> <python-ideas at python.org> wrote:
>>> On Thursday, November 5, 2015 9:57 PM, "sjoerdjob at sjec.nl"
>>> <sjoerdjob at sjec.nl> wrote:
>>>
>>> >>  Andrew Barnert via Python-ideas writes:
>>> >>   > Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
>>> >>
>>> >>   > > It might seem troublesome to change these locations to raise
>>> >>   > > actual instances instead of strings, but those might actually
>>> >>   > > just be hidden bugs just the same.
>>> >>
>>> >>  No, I'm sure mostly they're not.  They're working code that has
>>> > been
>>> >>  there since the early days, perhaps a few from the 2.x days in
>>> >>  modules.  Python is generally very conservative about changing
>>> working
>>> >>  code.
>>> >>
>>> >>   > > Do you think anyone would object if I'd go over the code
>>> > looking
>>> >>   > > for places where type(tstate->cur_value) !=
>>> > tstate->cur_type and
>>> >>   > > propose patches for these?
>>> >>
>>> >>  I think, yes, though I don't know enough about it to object myself.
>>> >>  Code churn of this kind is generally frowned upon, unless you can
>>> show
>>> >>  that the risk of hidden bugs of the kind you refer to above is
>>> >>  substantially greater than the risk that *you* will introduce bugs
>>> >>  with your patches.
>>> >
>>> > After digging more into the code, I concur. It's quite easy to make
>>> some
>>> > changes that break some stuff in this logic.
>>>
>>> Looking at this a bit further, I think there may be something you can
>>> do that gets everything you actually wanted, without changes all over
>>> the place and serious risks.
>>>
>>> Basically, it's just two pretty small changes in errors.c:
>>>
>>> * Change PyErr_GivenExceptionMatches so that if given a value that's an
>>> instance of an exception type (PyExceptionInstance_Check(err) is true),
>>> instead of just getting its class and then ignoring the value, it does
>>> a standard isinstance check on the value.
>>>
>>> * Change PyErr_ExceptionMatches so that if there is an exception value,
>>> and its type matches the exception type, it is passed to
>>> PyErr_GivenExceptionMatches; otherwise, the exception type is passed.
>>>
>>>
>>> So, all Python code that raises and handles exceptions will get the
>>> benefit of isinstance checks. Any new C code that wants that benefit
>>> can get it very easily (only raise with PyErr_SetObject or something
>>> that uses it like PyErr_SetFromErrno; only handle with
>>> PyErr_GivenExceptionMatches). Any existing C code that skips creating
>>> an instance will continue to do so, and to get the same fast-path check
>>> used today, except for one extra call to
>>> PyExceptionInstance_Check(tstate->curexc_value).
>>>
>>> Of course it's not just that small patch to errors.c. You also need
>>> tests to exercise the new functionality. And a docs patch for the
>>> Exceptions section of the C API. And also a docs patch to the
>>> Exceptions section in the Reference. But no other code patches.
>>>
>>> And you'd definitely need to performance-test two things: First, that
>>> the extra PyExceptionInstance_Check doesn't slow down fast-path
>>> handling (e.g., in an empty for loops). Second, whatever performance
>>> tests your original proposal would have needed, to make sure the
>>> isinstance check doesn't noticeably slow down pure-Python code that
>>> handles a lot of exceptions.
>>>
>>>
>>> I think the only C API docs change needed is to PyErr_ExceptionMatches.
>>> However, a note explaining how to make sure to get and/or to bypass the
>>> isinstance check as desired might be useful.
>>>
>>> For the reference docs, I think just this note at the end of the
>>> section:
>>>
>>> > Note: Exceptions raised and handled by Python code use the same
>>> mechanism as the isinstance function for determining whether an
>>> exception matches. However, exceptions raised or handled by the
>>> implementation (or implementation-level extensions) may bypass that
>>> and check the type directly.
>>>
>>>
>>> > New in 3.6: In previous versions of Python, exception handling always
>>> bypassed the isinstance function and checked the type directly.
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


From ram at rachum.com  Mon Nov  9 03:03:18 2015
From: ram at rachum.com (Ram Rachum)
Date: Mon, 9 Nov 2015 10:03:18 +0200
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CAP7+vJL1vTtx5zXA+aJJL+e=M0c-FBNDYS9RxS9VLvm5pJA5BQ@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
 <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl> <n1oh7u$s9j$1@ger.gmane.org>
 <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com>
 <CAP7+vJL1vTtx5zXA+aJJL+e=M0c-FBNDYS9RxS9VLvm5pJA5BQ@mail.gmail.com>
Message-ID: <CANXboVbtzDC7GD_+yGXDkAcy8QCa3hOHypH6hcxgGPVB-KpxEQ@mail.gmail.com>

I'm not Andrew, but I'm guessing simply writing
`OrderedDict(headers.items())`.

On Mon, Nov 9, 2015 at 6:28 AM, Guido van Rossum <guido at python.org> wrote:

> So if OrderedDict had always rejected construction from a dict, how would
> you have written this?
>
>
> On Sunday, November 8, 2015, Andrew Barnert via Python-ideas <
> python-ideas at python.org> wrote:
>
>> On Nov 8, 2015, at 14:10, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> >
>> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote:
>> >> On 8 Nov 2015, at 20:06, Amir Rachum <amir at rachum.com
>> >> <mailto:amir at rachum.com>> wrote:
>> >>> As part of BasicStruct I intend to allow the use of mapping types as
>> >>> __slots__, with the semantics of default values..
>> >>> So it'll look something like this:
>> >>>
>> >>>    class Point(BasicStruct):
>> >>>        __slots__ = {'x': 5, 'y': 7}
>> >>
>> >> So instead they'll write
>> >>     __slots__ = OrderedDict({'x': 5, 'y': 7})
>> >> Causing the same issues?
>> >
>> > Perhaps OrderedDict should reject unordered sources. Hey, here is yet
>> one use case!
>>
>> I've maintained code that does this:
>>
>>     self.headers = OrderedDict(headers)
>>     self.origheaders = len(headers)
>>
>> ? so it can later do this:
>>
>>     altheaders = list(self.headers.items())[self.origheaders:]
>>
>> Not a great design, but one that exists in the wild, and would be broken
>> by OrderedDict not allowing a dict as an argument.
>>
>> Also, this wouldn't allow creating an OrderedDict from an empty dict
>> (which seems far less stupid, but I didn't lead with it because I can't
>> remember seeing it in real code).
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> --Guido (mobile)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151109/f281c29d/attachment.html>

From guido at python.org  Mon Nov  9 11:29:21 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Nov 2015 08:29:21 -0800
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CANXboVbtzDC7GD_+yGXDkAcy8QCa3hOHypH6hcxgGPVB-KpxEQ@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
 <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl> <n1oh7u$s9j$1@ger.gmane.org>
 <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com>
 <CAP7+vJL1vTtx5zXA+aJJL+e=M0c-FBNDYS9RxS9VLvm5pJA5BQ@mail.gmail.com>
 <CANXboVbtzDC7GD_+yGXDkAcy8QCa3hOHypH6hcxgGPVB-KpxEQ@mail.gmail.com>
Message-ID: <CAP7+vJJeXy1Fqpep9pG7_MgUEs+-bY9Uc3GZHO1LQ-8Nn-RFXg@mail.gmail.com>

Well, that would still defeat the purpose, wouldn't it? The items are no
more ordered than the headers dict itself. Also, items() doesn't return a
sequence -- it's an ItemsView (which inherits from Set) and presumably it's
not Ordered.

I guess my question is not so much how to prevent getting an exception --
I'm trying to tease out what the right order for the headers would be. Or
perhaps I'm just trying to understand what the code is doing (the snippet
shown mostly looks like bad code to me).

On Mon, Nov 9, 2015 at 12:03 AM, Ram Rachum <ram at rachum.com> wrote:

> I'm not Andrew, but I'm guessing simply writing
> `OrderedDict(headers.items())`.
>
> On Mon, Nov 9, 2015 at 6:28 AM, Guido van Rossum <guido at python.org> wrote:
>
>> So if OrderedDict had always rejected construction from a dict, how would
>> you have written this?
>>
>>
>> On Sunday, November 8, 2015, Andrew Barnert via Python-ideas <
>> python-ideas at python.org> wrote:
>>
>>> On Nov 8, 2015, at 14:10, Serhiy Storchaka <storchaka at gmail.com> wrote:
>>> >
>>> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote:
>>> >> On 8 Nov 2015, at 20:06, Amir Rachum <amir at rachum.com
>>> >> <mailto:amir at rachum.com>> wrote:
>>> >>> As part of BasicStruct I intend to allow the use of mapping types as
>>> >>> __slots__, with the semantics of default values..
>>> >>> So it'll look something like this:
>>> >>>
>>> >>>    class Point(BasicStruct):
>>> >>>        __slots__ = {'x': 5, 'y': 7}
>>> >>
>>> >> So instead they'll write
>>> >>     __slots__ = OrderedDict({'x': 5, 'y': 7})
>>> >> Causing the same issues?
>>> >
>>> > Perhaps OrderedDict should reject unordered sources. Hey, here is yet
>>> one use case!
>>>
>>> I've maintained code that does this:
>>>
>>>     self.headers = OrderedDict(headers)
>>>     self.origheaders = len(headers)
>>>
>>> ? so it can later do this:
>>>
>>>     altheaders = list(self.headers.items())[self.origheaders:]
>>>
>>> Not a great design, but one that exists in the wild, and would be broken
>>> by OrderedDict not allowing a dict as an argument.
>>>
>>> Also, this wouldn't allow creating an OrderedDict from an empty dict
>>> (which seems far less stupid, but I didn't lead with it because I can't
>>> remember seeing it in real code).
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>>
>> --
>> --Guido (mobile)
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151109/109cf282/attachment.html>

From mike at selik.org  Mon Nov  9 19:20:51 2015
From: mike at selik.org (Michael Selik)
Date: Mon, 9 Nov 2015 19:20:51 -0500
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CAP7+vJJeXy1Fqpep9pG7_MgUEs+-bY9Uc3GZHO1LQ-8Nn-RFXg@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
 <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl> <n1oh7u$s9j$1@ger.gmane.org>
 <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com>
 <CAP7+vJL1vTtx5zXA+aJJL+e=M0c-FBNDYS9RxS9VLvm5pJA5BQ@mail.gmail.com>
 <CANXboVbtzDC7GD_+yGXDkAcy8QCa3hOHypH6hcxgGPVB-KpxEQ@mail.gmail.com>
 <CAP7+vJJeXy1Fqpep9pG7_MgUEs+-bY9Uc3GZHO1LQ-8Nn-RFXg@mail.gmail.com>
Message-ID: <CADqi796yPPC9Z0aa=vFxw4ttHnfCjmStp1xfd7m6WY9DZKXzpA@mail.gmail.com>

I found a use case, out in the wild!

I've been searching through GitHub for "ordered" and similar phrases.
Much of the usage is OrderedDict and OrderedSet, which really don't
benefit from an essentially redundant inheritance from a hypothetical
collections.Ordered. There are some tools that enable UI reordering,
and a bunch of tree- and graph-traversal ordering functions. Again,
not much benefit from a collections.Ordered.

However, Django has a class factory that creates an attribute `can_order`:

def formset_factory(form, formset=BaseFormSet, extra=1, can_order=False,
                    can_delete=False, max_num=None, validate_max=False,
                    min_num=None, validate_min=False):
    """Return a FormSet for the given form class."""
    if min_num is None:
        min_num = DEFAULT_MIN_NUM
    if max_num is None:
        max_num = DEFAULT_MAX_NUM
    # hard limit on forms instantiated, to prevent memory-exhaustion attacks
    # limit is simply max_num + DEFAULT_MAX_NUM (which is 2*DEFAULT_MAX_NUM
    # if max_num is None in the first place)
    absolute_max = max_num + DEFAULT_MAX_NUM
    attrs = {'form': form, 'extra': extra,
             'can_order': can_order, 'can_delete': can_delete,
             'min_num': min_num, 'max_num': max_num,
             'absolute_max': absolute_max, 'validate_min': validate_min,
             'validate_max': validate_max}
    return type(form.__name__ + str('FormSet'), (formset,), attrs)



This attribute gets set in several places, but checked only twice in
the Django codebase. Unfortunately, I think switching to the proposed
inheritance mechanism would make the code worse, not better:
`self.can_order` would become `isinstance(self, collections.Ordered)`.
The readability of the Django internal code would not be much
different, but users would lose consistency of introspectability as
the other features like `can_delete` are simple class attributes.



On Mon, Nov 9, 2015 at 11:29 AM, Guido van Rossum <guido at python.org> wrote:
> Well, that would still defeat the purpose, wouldn't it? The items are no
> more ordered than the headers dict itself. Also, items() doesn't return a
> sequence -- it's an ItemsView (which inherits from Set) and presumably it's
> not Ordered.
>
> I guess my question is not so much how to prevent getting an exception --
> I'm trying to tease out what the right order for the headers would be. Or
> perhaps I'm just trying to understand what the code is doing (the snippet
> shown mostly looks like bad code to me).
>
> On Mon, Nov 9, 2015 at 12:03 AM, Ram Rachum <ram at rachum.com> wrote:
>>
>> I'm not Andrew, but I'm guessing simply writing
>> `OrderedDict(headers.items())`.
>>
>> On Mon, Nov 9, 2015 at 6:28 AM, Guido van Rossum <guido at python.org> wrote:
>>>
>>> So if OrderedDict had always rejected construction from a dict, how would
>>> you have written this?
>>>
>>>
>>> On Sunday, November 8, 2015, Andrew Barnert via Python-ideas
>>> <python-ideas at python.org> wrote:
>>>>
>>>> On Nov 8, 2015, at 14:10, Serhiy Storchaka <storchaka at gmail.com> wrote:
>>>> >
>>>> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote:
>>>> >> On 8 Nov 2015, at 20:06, Amir Rachum <amir at rachum.com
>>>> >> <mailto:amir at rachum.com>> wrote:
>>>> >>> As part of BasicStruct I intend to allow the use of mapping types as
>>>> >>> __slots__, with the semantics of default values..
>>>> >>> So it'll look something like this:
>>>> >>>
>>>> >>>    class Point(BasicStruct):
>>>> >>>        __slots__ = {'x': 5, 'y': 7}
>>>> >>
>>>> >> So instead they'll write
>>>> >>     __slots__ = OrderedDict({'x': 5, 'y': 7})
>>>> >> Causing the same issues?
>>>> >
>>>> > Perhaps OrderedDict should reject unordered sources. Hey, here is yet
>>>> > one use case!
>>>>
>>>> I've maintained code that does this:
>>>>
>>>>     self.headers = OrderedDict(headers)
>>>>     self.origheaders = len(headers)
>>>>
>>>> ? so it can later do this:
>>>>
>>>>     altheaders = list(self.headers.items())[self.origheaders:]
>>>>
>>>> Not a great design, but one that exists in the wild, and would be broken
>>>> by OrderedDict not allowing a dict as an argument.
>>>>
>>>> Also, this wouldn't allow creating an OrderedDict from an empty dict
>>>> (which seems far less stupid, but I didn't lead with it because I can't
>>>> remember seeing it in real code).
>>>>
>>>> _______________________________________________
>>>> Python-ideas mailing list
>>>> Python-ideas at python.org
>>>> https://mail.python.org/mailman/listinfo/python-ideas
>>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>>
>>>
>>> --
>>> --Guido (mobile)
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From guido at python.org  Mon Nov  9 19:47:08 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 9 Nov 2015 16:47:08 -0800
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CADqi796yPPC9Z0aa=vFxw4ttHnfCjmStp1xfd7m6WY9DZKXzpA@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
 <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl> <n1oh7u$s9j$1@ger.gmane.org>
 <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com>
 <CAP7+vJL1vTtx5zXA+aJJL+e=M0c-FBNDYS9RxS9VLvm5pJA5BQ@mail.gmail.com>
 <CANXboVbtzDC7GD_+yGXDkAcy8QCa3hOHypH6hcxgGPVB-KpxEQ@mail.gmail.com>
 <CAP7+vJJeXy1Fqpep9pG7_MgUEs+-bY9Uc3GZHO1LQ-8Nn-RFXg@mail.gmail.com>
 <CADqi796yPPC9Z0aa=vFxw4ttHnfCjmStp1xfd7m6WY9DZKXzpA@mail.gmail.com>
Message-ID: <CAP7+vJ+edk=6vuoOn_G9AxupF-NCuo-A2i6zjqSecWHBpE9khA@mail.gmail.com>

Hm, this just indicates whether some UI appears by which the user can
reorder the forms in a list? Is it even appropriate to move this from the
instance to the class?

On Mon, Nov 9, 2015 at 4:20 PM, Michael Selik <mike at selik.org> wrote:

> I found a use case, out in the wild!
>
> I've been searching through GitHub for "ordered" and similar phrases.
> Much of the usage is OrderedDict and OrderedSet, which really don't
> benefit from an essentially redundant inheritance from a hypothetical
> collections.Ordered. There are some tools that enable UI reordering,
> and a bunch of tree- and graph-traversal ordering functions. Again,
> not much benefit from a collections.Ordered.
>
> However, Django has a class factory that creates an attribute `can_order`:
>
> def formset_factory(form, formset=BaseFormSet, extra=1, can_order=False,
>                     can_delete=False, max_num=None, validate_max=False,
>                     min_num=None, validate_min=False):
>     """Return a FormSet for the given form class."""
>     if min_num is None:
>         min_num = DEFAULT_MIN_NUM
>     if max_num is None:
>         max_num = DEFAULT_MAX_NUM
>     # hard limit on forms instantiated, to prevent memory-exhaustion
> attacks
>     # limit is simply max_num + DEFAULT_MAX_NUM (which is 2*DEFAULT_MAX_NUM
>     # if max_num is None in the first place)
>     absolute_max = max_num + DEFAULT_MAX_NUM
>     attrs = {'form': form, 'extra': extra,
>              'can_order': can_order, 'can_delete': can_delete,
>              'min_num': min_num, 'max_num': max_num,
>              'absolute_max': absolute_max, 'validate_min': validate_min,
>              'validate_max': validate_max}
>     return type(form.__name__ + str('FormSet'), (formset,), attrs)
>
>
>
> This attribute gets set in several places, but checked only twice in
> the Django codebase. Unfortunately, I think switching to the proposed
> inheritance mechanism would make the code worse, not better:
> `self.can_order` would become `isinstance(self, collections.Ordered)`.
> The readability of the Django internal code would not be much
> different, but users would lose consistency of introspectability as
> the other features like `can_delete` are simple class attributes.
>
>
>
> On Mon, Nov 9, 2015 at 11:29 AM, Guido van Rossum <guido at python.org>
> wrote:
> > Well, that would still defeat the purpose, wouldn't it? The items are no
> > more ordered than the headers dict itself. Also, items() doesn't return a
> > sequence -- it's an ItemsView (which inherits from Set) and presumably
> it's
> > not Ordered.
> >
> > I guess my question is not so much how to prevent getting an exception --
> > I'm trying to tease out what the right order for the headers would be. Or
> > perhaps I'm just trying to understand what the code is doing (the snippet
> > shown mostly looks like bad code to me).
> >
> > On Mon, Nov 9, 2015 at 12:03 AM, Ram Rachum <ram at rachum.com> wrote:
> >>
> >> I'm not Andrew, but I'm guessing simply writing
> >> `OrderedDict(headers.items())`.
> >>
> >> On Mon, Nov 9, 2015 at 6:28 AM, Guido van Rossum <guido at python.org>
> wrote:
> >>>
> >>> So if OrderedDict had always rejected construction from a dict, how
> would
> >>> you have written this?
> >>>
> >>>
> >>> On Sunday, November 8, 2015, Andrew Barnert via Python-ideas
> >>> <python-ideas at python.org> wrote:
> >>>>
> >>>> On Nov 8, 2015, at 14:10, Serhiy Storchaka <storchaka at gmail.com>
> wrote:
> >>>> >
> >>>> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote:
> >>>> >> On 8 Nov 2015, at 20:06, Amir Rachum <amir at rachum.com
> >>>> >> <mailto:amir at rachum.com>> wrote:
> >>>> >>> As part of BasicStruct I intend to allow the use of mapping types
> as
> >>>> >>> __slots__, with the semantics of default values..
> >>>> >>> So it'll look something like this:
> >>>> >>>
> >>>> >>>    class Point(BasicStruct):
> >>>> >>>        __slots__ = {'x': 5, 'y': 7}
> >>>> >>
> >>>> >> So instead they'll write
> >>>> >>     __slots__ = OrderedDict({'x': 5, 'y': 7})
> >>>> >> Causing the same issues?
> >>>> >
> >>>> > Perhaps OrderedDict should reject unordered sources. Hey, here is
> yet
> >>>> > one use case!
> >>>>
> >>>> I've maintained code that does this:
> >>>>
> >>>>     self.headers = OrderedDict(headers)
> >>>>     self.origheaders = len(headers)
> >>>>
> >>>> ? so it can later do this:
> >>>>
> >>>>     altheaders = list(self.headers.items())[self.origheaders:]
> >>>>
> >>>> Not a great design, but one that exists in the wild, and would be
> broken
> >>>> by OrderedDict not allowing a dict as an argument.
> >>>>
> >>>> Also, this wouldn't allow creating an OrderedDict from an empty dict
> >>>> (which seems far less stupid, but I didn't lead with it because I
> can't
> >>>> remember seeing it in real code).
> >>>>
> >>>> _______________________________________________
> >>>> Python-ideas mailing list
> >>>> Python-ideas at python.org
> >>>> https://mail.python.org/mailman/listinfo/python-ideas
> >>>> Code of Conduct: http://python.org/psf/codeofconduct/
> >>>
> >>>
> >>>
> >>> --
> >>> --Guido (mobile)
> >>>
> >>> _______________________________________________
> >>> Python-ideas mailing list
> >>> Python-ideas at python.org
> >>> https://mail.python.org/mailman/listinfo/python-ideas
> >>> Code of Conduct: http://python.org/psf/codeofconduct/
> >>
> >>
> >
> >
> >
> > --
> > --Guido van Rossum (python.org/~guido)
> >
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151109/2372bc7c/attachment-0001.html>

From mike at selik.org  Mon Nov  9 20:03:11 2015
From: mike at selik.org (Michael Selik)
Date: Tue, 10 Nov 2015 01:03:11 +0000
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CAP7+vJ+edk=6vuoOn_G9AxupF-NCuo-A2i6zjqSecWHBpE9khA@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
 <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl> <n1oh7u$s9j$1@ger.gmane.org>
 <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com>
 <CAP7+vJL1vTtx5zXA+aJJL+e=M0c-FBNDYS9RxS9VLvm5pJA5BQ@mail.gmail.com>
 <CANXboVbtzDC7GD_+yGXDkAcy8QCa3hOHypH6hcxgGPVB-KpxEQ@mail.gmail.com>
 <CAP7+vJJeXy1Fqpep9pG7_MgUEs+-bY9Uc3GZHO1LQ-8Nn-RFXg@mail.gmail.com>
 <CADqi796yPPC9Z0aa=vFxw4ttHnfCjmStp1xfd7m6WY9DZKXzpA@mail.gmail.com>
 <CAP7+vJ+edk=6vuoOn_G9AxupF-NCuo-A2i6zjqSecWHBpE9khA@mail.gmail.com>
Message-ID: <CADqi796M0wYJAs0ZEis+To9nVGxBCoW+88FMHj1VytU9K0JPfw@mail.gmail.com>

If my thoughts got lost in the mess of text: No, I don't think it is
appropriate.

On Mon, Nov 9, 2015 at 7:47 PM Guido van Rossum <guido at python.org> wrote:

> Hm, this just indicates whether some UI appears by which the user can
> reorder the forms in a list? Is it even appropriate to move this from the
> instance to the class?
>
> On Mon, Nov 9, 2015 at 4:20 PM, Michael Selik <mike at selik.org> wrote:
>
>> I found a use case, out in the wild!
>>
>> I've been searching through GitHub for "ordered" and similar phrases.
>> Much of the usage is OrderedDict and OrderedSet, which really don't
>> benefit from an essentially redundant inheritance from a hypothetical
>> collections.Ordered. There are some tools that enable UI reordering,
>> and a bunch of tree- and graph-traversal ordering functions. Again,
>> not much benefit from a collections.Ordered.
>>
>> However, Django has a class factory that creates an attribute `can_order`:
>>
>> def formset_factory(form, formset=BaseFormSet, extra=1, can_order=False,
>>                     can_delete=False, max_num=None, validate_max=False,
>>                     min_num=None, validate_min=False):
>>     """Return a FormSet for the given form class."""
>>     if min_num is None:
>>         min_num = DEFAULT_MIN_NUM
>>     if max_num is None:
>>         max_num = DEFAULT_MAX_NUM
>>     # hard limit on forms instantiated, to prevent memory-exhaustion
>> attacks
>>     # limit is simply max_num + DEFAULT_MAX_NUM (which is
>> 2*DEFAULT_MAX_NUM
>>     # if max_num is None in the first place)
>>     absolute_max = max_num + DEFAULT_MAX_NUM
>>     attrs = {'form': form, 'extra': extra,
>>              'can_order': can_order, 'can_delete': can_delete,
>>              'min_num': min_num, 'max_num': max_num,
>>              'absolute_max': absolute_max, 'validate_min': validate_min,
>>              'validate_max': validate_max}
>>     return type(form.__name__ + str('FormSet'), (formset,), attrs)
>>
>>
>>
>> This attribute gets set in several places, but checked only twice in
>> the Django codebase. Unfortunately, I think switching to the proposed
>> inheritance mechanism would make the code worse, not better:
>> `self.can_order` would become `isinstance(self, collections.Ordered)`.
>> The readability of the Django internal code would not be much
>> different, but users would lose consistency of introspectability as
>> the other features like `can_delete` are simple class attributes.
>>
>>
>>
>> On Mon, Nov 9, 2015 at 11:29 AM, Guido van Rossum <guido at python.org>
>> wrote:
>> > Well, that would still defeat the purpose, wouldn't it? The items are no
>> > more ordered than the headers dict itself. Also, items() doesn't return
>> a
>> > sequence -- it's an ItemsView (which inherits from Set) and presumably
>> it's
>> > not Ordered.
>> >
>> > I guess my question is not so much how to prevent getting an exception
>> --
>> > I'm trying to tease out what the right order for the headers would be.
>> Or
>> > perhaps I'm just trying to understand what the code is doing (the
>> snippet
>> > shown mostly looks like bad code to me).
>> >
>> > On Mon, Nov 9, 2015 at 12:03 AM, Ram Rachum <ram at rachum.com> wrote:
>> >>
>> >> I'm not Andrew, but I'm guessing simply writing
>> >> `OrderedDict(headers.items())`.
>> >>
>> >> On Mon, Nov 9, 2015 at 6:28 AM, Guido van Rossum <guido at python.org>
>> wrote:
>> >>>
>> >>> So if OrderedDict had always rejected construction from a dict, how
>> would
>> >>> you have written this?
>> >>>
>> >>>
>> >>> On Sunday, November 8, 2015, Andrew Barnert via Python-ideas
>> >>> <python-ideas at python.org> wrote:
>> >>>>
>> >>>> On Nov 8, 2015, at 14:10, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>> >>>> >
>> >>>> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote:
>> >>>> >> On 8 Nov 2015, at 20:06, Amir Rachum <amir at rachum.com
>> >>>> >> <mailto:amir at rachum.com>> wrote:
>> >>>> >>> As part of BasicStruct I intend to allow the use of mapping
>> types as
>> >>>> >>> __slots__, with the semantics of default values..
>> >>>> >>> So it'll look something like this:
>> >>>> >>>
>> >>>> >>>    class Point(BasicStruct):
>> >>>> >>>        __slots__ = {'x': 5, 'y': 7}
>> >>>> >>
>> >>>> >> So instead they'll write
>> >>>> >>     __slots__ = OrderedDict({'x': 5, 'y': 7})
>> >>>> >> Causing the same issues?
>> >>>> >
>> >>>> > Perhaps OrderedDict should reject unordered sources. Hey, here is
>> yet
>> >>>> > one use case!
>> >>>>
>> >>>> I've maintained code that does this:
>> >>>>
>> >>>>     self.headers = OrderedDict(headers)
>> >>>>     self.origheaders = len(headers)
>> >>>>
>> >>>> ? so it can later do this:
>> >>>>
>> >>>>     altheaders = list(self.headers.items())[self.origheaders:]
>> >>>>
>> >>>> Not a great design, but one that exists in the wild, and would be
>> broken
>> >>>> by OrderedDict not allowing a dict as an argument.
>> >>>>
>> >>>> Also, this wouldn't allow creating an OrderedDict from an empty dict
>> >>>> (which seems far less stupid, but I didn't lead with it because I
>> can't
>> >>>> remember seeing it in real code).
>> >>>>
>> >>>> _______________________________________________
>> >>>> Python-ideas mailing list
>> >>>> Python-ideas at python.org
>> >>>> https://mail.python.org/mailman/listinfo/python-ideas
>> >>>> Code of Conduct: http://python.org/psf/codeofconduct/
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> --Guido (mobile)
>> >>>
>> >>> _______________________________________________
>> >>> Python-ideas mailing list
>> >>> Python-ideas at python.org
>> >>> https://mail.python.org/mailman/listinfo/python-ideas
>> >>> Code of Conduct: http://python.org/psf/codeofconduct/
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > --Guido van Rossum (python.org/~guido)
>> >
>> > _______________________________________________
>> > Python-ideas mailing list
>> > Python-ideas at python.org
>> > https://mail.python.org/mailman/listinfo/python-ideas
>> > Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151110/e9b2345c/attachment.html>

From mike at selik.org  Mon Nov  9 20:14:05 2015
From: mike at selik.org (Michael Selik)
Date: Tue, 10 Nov 2015 01:14:05 +0000
Subject: [Python-ideas] Use `isinstance` check during exception handling
In-Reply-To: <6f22ce0f98ae5b4f7a290343bad92564.squirrel@mail.sjec.nl>
References: <20dea45bc5e012ecd5db6f000c668e12.squirrel@mail.sjec.nl>
 <1277750399.598478.1446796147897.JavaMail.yahoo@mail.yahoo.com>
 <CADqi796TzPSvoBoxVsDyse4HOgMQDYEieZWjnBrOw5sVgFDugA@mail.gmail.com>
 <AB61E1C1-6376-4A97-AE0E-B1527F6AAEC9@sjec.nl>
 <6f22ce0f98ae5b4f7a290343bad92564.squirrel@mail.sjec.nl>
Message-ID: <CADqi794xgEhiSmDr5Ze+pQX7L12j6JMDwmHCVMCEF9-C5F3=eA@mail.gmail.com>

I searched cpython/Lib/*.py for except blocks that have an if-statement as
the first line. r'except \w+ as\s+\w+:\n\s+if .*:' Not the world's most
thorough regex, but it seemed to do the trick.

There are very few repeated except/if blocks that would be appropriate for
refactoring with any technique. The majority were in two modules --
mailbox.py and pathlib.py -- checking if the errorno was ENOINT and/or
ENOTDIR. I suppose those could be improved with FileNotFound and
NotADirectoryError, but it doesn't look like we need anything more potent.

Is there a different place we should look for refactoring candidates?


On Mon, Nov 9, 2015, 12:41 AM  <sjoerdjob at sjec.nl> wrote:

> > I made the changes I listed above and ran `make; make test`, and got only
> > 1 error due to socket.socket running out of memory or something. I'll
> > re-run the tests on a clean checkout as well, but for now I think it's
> > unrelated.
>
> After running the tests against a clean checkout (without my changes): I
> get the same errors from test_socket. So it's unrelated.
>
> > Regards,
> > Sjoerd Job Postmus
> >
> >> On 7 Nov 2015, at 20:18, Michael Selik <mike at selik.org> wrote:
> >>
> >> Sjoerd, glad we agree, happy to help.
> >>
> >> In case someone is still interested in making a patch, I'd like to
> >> clarify why it makes me nervous.
> >>
> >> Though Sjoerd led with a TL;DR that summarized his suggestion as a swap
> >> of isinstance for issubclass, a better summary would have been: "I
> >> prefer more specific exception types". The proposed solution was to
> >> enable custom exception types that conditionally insert themselves into
> >> the class hierarchy depending on data stored on a raised exception
> >> instance.
> >>
> >> Unfortunately, that solution requires changing an obscure error handling
> >> feature. As Sjoerd noted when he sent the first message in this thread,
> >> it is possible this would introduce a backwards-incompatibility. Perhaps
> >> no one in the standard library or major public projects would write such
> >> weird code as to get tripped up by the proposed change, but it's always
> >> possible that someone out there is using this feature for an important
> >> project. I think we've learned the lesson from the long slog to Python 3
> >> adoption (and how Perl 6 harmed that community, and countless other
> >> examples) that backwards compatibility is of utmost importance.
> >>
> >> The current code pattern for a single try/except is to catch the general
> >> exception and have an if-condition in the except block.That's a frequent
> >> and readable practice.
> >>
> >>     try:
> >>         cursor.execute(query)
> >>     except sqlite3.OperationalError as str(e):
> >>         if 'syntax error' in str(e):
> >>             print('Your SQL is bad')
> >>
> >> If that code or similar code appears many times, a good way to refactor
> >> with the current features of Python is to use a context manager.
> >>
> >>     class SqlSyntaxErrorHandler:
> >>         def __enter__(self):
> >>             return self
> >>         def __exit__(self, etyp, einstance, etraceback):
> >>             if (issubclass(etyp, sqlite3.OperationalError)
> >>                 and 'syntax error' in str(einstance)):
> >>                     print('Your SQL is bad')
> >>
> >>     with SqlSyntaxErrorHandler:
> >>         cursor.execute(first_query)
> >>
> >>     with SqlSyntaxErrorHandler:
> >>         cursor.execute(second_query)
> >>
> >> While context managers may not be a beginner topic, they are typically
> >> easier to read and debug than using metaclasses for conditional runtime
> >> inheritance. Even if there would be some benefit to the metaclass style,
> >> it's not worth the risk of changing current error handling behavior.
> >>
> >> -- Michael
> >>
> >>
> >>> On Fri, Nov 6, 2015, 2:49 AM Andrew Barnert via Python-ideas
> >>> <python-ideas at python.org> wrote:
> >>> On Thursday, November 5, 2015 9:57 PM, "sjoerdjob at sjec.nl"
> >>> <sjoerdjob at sjec.nl> wrote:
> >>>
> >>> >>  Andrew Barnert via Python-ideas writes:
> >>> >>   > Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
> >>> >>
> >>> >>   > > It might seem troublesome to change these locations to raise
> >>> >>   > > actual instances instead of strings, but those might actually
> >>> >>   > > just be hidden bugs just the same.
> >>> >>
> >>> >>  No, I'm sure mostly they're not.  They're working code that has
> >>> > been
> >>> >>  there since the early days, perhaps a few from the 2.x days in
> >>> >>  modules.  Python is generally very conservative about changing
> >>> working
> >>> >>  code.
> >>> >>
> >>> >>   > > Do you think anyone would object if I'd go over the code
> >>> > looking
> >>> >>   > > for places where type(tstate->cur_value) !=
> >>> > tstate->cur_type and
> >>> >>   > > propose patches for these?
> >>> >>
> >>> >>  I think, yes, though I don't know enough about it to object myself.
> >>> >>  Code churn of this kind is generally frowned upon, unless you can
> >>> show
> >>> >>  that the risk of hidden bugs of the kind you refer to above is
> >>> >>  substantially greater than the risk that *you* will introduce bugs
> >>> >>  with your patches.
> >>> >
> >>> > After digging more into the code, I concur. It's quite easy to make
> >>> some
> >>> > changes that break some stuff in this logic.
> >>>
> >>> Looking at this a bit further, I think there may be something you can
> >>> do that gets everything you actually wanted, without changes all over
> >>> the place and serious risks.
> >>>
> >>> Basically, it's just two pretty small changes in errors.c:
> >>>
> >>> * Change PyErr_GivenExceptionMatches so that if given a value that's an
> >>> instance of an exception type (PyExceptionInstance_Check(err) is true),
> >>> instead of just getting its class and then ignoring the value, it does
> >>> a standard isinstance check on the value.
> >>>
> >>> * Change PyErr_ExceptionMatches so that if there is an exception value,
> >>> and its type matches the exception type, it is passed to
> >>> PyErr_GivenExceptionMatches; otherwise, the exception type is passed.
> >>>
> >>>
> >>> So, all Python code that raises and handles exceptions will get the
> >>> benefit of isinstance checks. Any new C code that wants that benefit
> >>> can get it very easily (only raise with PyErr_SetObject or something
> >>> that uses it like PyErr_SetFromErrno; only handle with
> >>> PyErr_GivenExceptionMatches). Any existing C code that skips creating
> >>> an instance will continue to do so, and to get the same fast-path check
> >>> used today, except for one extra call to
> >>> PyExceptionInstance_Check(tstate->curexc_value).
> >>>
> >>> Of course it's not just that small patch to errors.c. You also need
> >>> tests to exercise the new functionality. And a docs patch for the
> >>> Exceptions section of the C API. And also a docs patch to the
> >>> Exceptions section in the Reference. But no other code patches.
> >>>
> >>> And you'd definitely need to performance-test two things: First, that
> >>> the extra PyExceptionInstance_Check doesn't slow down fast-path
> >>> handling (e.g., in an empty for loops). Second, whatever performance
> >>> tests your original proposal would have needed, to make sure the
> >>> isinstance check doesn't noticeably slow down pure-Python code that
> >>> handles a lot of exceptions.
> >>>
> >>>
> >>> I think the only C API docs change needed is to PyErr_ExceptionMatches.
> >>> However, a note explaining how to make sure to get and/or to bypass the
> >>> isinstance check as desired might be useful.
> >>>
> >>> For the reference docs, I think just this note at the end of the
> >>> section:
> >>>
> >>> > Note: Exceptions raised and handled by Python code use the same
> >>> mechanism as the isinstance function for determining whether an
> >>> exception matches. However, exceptions raised or handled by the
> >>> implementation (or implementation-level extensions) may bypass that
> >>> and check the type directly.
> >>>
> >>>
> >>> > New in 3.6: In previous versions of Python, exception handling always
> >>> bypassed the isinstance function and checked the type directly.
> >>> _______________________________________________
> >>> Python-ideas mailing list
> >>> Python-ideas at python.org
> >>> https://mail.python.org/mailman/listinfo/python-ideas
> >>> Code of Conduct: http://python.org/psf/codeofconduct/
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151110/3182236d/attachment-0001.html>

From stephen at xemacs.org  Mon Nov  9 22:52:55 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 10 Nov 2015 12:52:55 +0900
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CAP7+vJJeXy1Fqpep9pG7_MgUEs+-bY9Uc3GZHO1LQ-8Nn-RFXg@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
 <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl>
 <n1oh7u$s9j$1@ger.gmane.org>
 <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com>
 <CAP7+vJL1vTtx5zXA+aJJL+e=M0c-FBNDYS9RxS9VLvm5pJA5BQ@mail.gmail.com>
 <CANXboVbtzDC7GD_+yGXDkAcy8QCa3hOHypH6hcxgGPVB-KpxEQ@mail.gmail.com>
 <CAP7+vJJeXy1Fqpep9pG7_MgUEs+-bY9Uc3GZHO1LQ-8Nn-RFXg@mail.gmail.com>
Message-ID: <22081.27159.81864.553297@turnbull.sk.tsukuba.ac.jp>

Guido van Rossum writes:

 > I guess my question is not so much how to prevent getting an
 > exception -- I'm trying to tease out what the right order for the
 > headers would be.

If they are email headers, then each MTA is supposed to *prepend* its
trace fields (Received and friends) to the existing header block.
Those new fields would correspond to Andrew's "alternate" headers.

Perhaps you are thinking of the fact that MTAs are also supposed to
preserve the existing order of headers.  However, the mail system as a
whole is in practice not reliable because of intermediating agents
like mailing lists and user forwarding.  These sometimes insert
application fields (such as List-* and Resent-*) before the trace
fields block rather than prepend their fields to the existing header.
As an indication of this lack of reliability, DKIM does not prescribe
a set or block of fields to sign, but rather the fields to sign are
specified as a sequence in the DKIM-Signature header, and verifiers
must extract and append the fields to the content to be signed in that
order regardless of physical order.

Python's email package provides a header abstraction which is an
ordered mapping.  The point is to be able to reproduce the entire
message with full accuracy by Message.flatten() (ie, without affecting
signatures).  So the relevant order is order found in the input
message (ie, FIFO).  However, this order is not stable in the sense
required by the OP of this thread: existing fields may be deleted and
new ones inserted at arbitrary positions.  Nor does it refer to any
externally-defined collation order for fields.

BTW, I do understand the difference between sorting and ordering.
What I missed is that that Amir (the OP) didn't ask for his
BasicStruct to be Ordered (although I guess it would be, considered as
a container for its slots); he wants to check that the initializer for
__slots__ is Ordered.

But that is insufficient in this use case (an alphabetically-ordered
instance would be a problematic initializer: consider a derived
BasicStruct that adds members to its parent BasicStruct).  I still
question the general usefulness of this ABC on the one hand, and ask
how far to proliferate its more useful subclasses on the other.

From abarnert at yahoo.com  Mon Nov  9 05:32:06 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 9 Nov 2015 02:32:06 -0800
Subject: [Python-ideas] Adding collections.abc.Ordered
In-Reply-To: <CAP7+vJL1vTtx5zXA+aJJL+e=M0c-FBNDYS9RxS9VLvm5pJA5BQ@mail.gmail.com>
References: <CANxN+D7MrkX1=WvQvWODDWdPM2PqKYofcdk52xnftTfDv0S2qw@mail.gmail.com>
 <22077.34177.66228.508517@turnbull.sk.tsukuba.ac.jp>
 <4D51ABF1-2961-44E2-B7D3-F77FAFD05EAC@yahoo.com>
 <22078.3491.476646.759893@turnbull.sk.tsukuba.ac.jp>
 <67E66963-8086-4809-8EA1-23E69B264E75@yahoo.com>
 <22079.16733.991880.97360@turnbull.sk.tsukuba.ac.jp>
 <CAP7+vJLUH1D71G469=wN2cQtP71Vcz4pa+TWKzchmun1Lr93mg@mail.gmail.com>
 <CANxN+D6htqQYCA5PHTRx7ZZLqHGDxwiPzPSWSU5Z4+ami_P7UQ@mail.gmail.com>
 <A2452054-2E30-402D-BCA3-F38231F8737C@sjec.nl> <n1oh7u$s9j$1@ger.gmane.org>
 <9F6BA67D-29D3-4CB8-9542-E5C9DA36740F@yahoo.com>
 <CAP7+vJL1vTtx5zXA+aJJL+e=M0c-FBNDYS9RxS9VLvm5pJA5BQ@mail.gmail.com>
Message-ID: <AB7F2279-A385-494D-8BE1-DE444DCE45F4@yahoo.com>

On Nov 8, 2015, at 20:28, Guido van Rossum <guido at python.org> wrote:
> 
> So if OrderedDict had always rejected construction from a dict, how would you have written this?

Well, I wouldn't have written it this way in the first place. The obvious way to do it is to store the original headers and new headers separately. ChainMapping two dicts together is much more obvious than cramming them into one OrderedDict and keeping track of where the dividing index is. So, I'd do this:

    self.original_headers = copy.copy(headers)
    self.alt_headers = OrderedDict()
    self.headers = ChainMap(self.alt_headers, self.original_headers)

Now origheaders is just len(self.original_headers), altheaders is self.alt_headers.items().

I didn't make this change to the code in question because it worked, and had no edits for over 2 years, and it only took me a couple minutes to figure out what it was doing, so it made more sense just to leave it alone.

If Python 3.7 forced me to change it by not allowing OrderedDict to be constructed from a dict, I'd have to decide whether to make this change, or the obvious mechanical fix of just calling OrderedDict on iter(headers.items()) instead of headers to trick the Ordered check.

By the way, the fact that this simple mechanical workaround works, and provides no additional semantics to tell the reader what's going on, seems like a minor strike against the proposal. (And if you ban iterators too, I'd just change the iter(?) to list(?).) All Ordered can really check is that the type provides some way to preserve a meaningful order, not that the data really are meaningfully ordered, after all.

>> On Sunday, November 8, 2015, Andrew Barnert via Python-ideas <python-ideas at python.org> wrote:
>> On Nov 8, 2015, at 14:10, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> >
>> >> On 08.11.15 23:12, Sjoerd Job Postmus wrote:
>> >> On 8 Nov 2015, at 20:06, Amir Rachum <amir at rachum.com
>> >> <mailto:amir at rachum.com>> wrote:
>> >>> As part of BasicStruct I intend to allow the use of mapping types as
>> >>> __slots__, with the semantics of default values..
>> >>> So it'll look something like this:
>> >>>
>> >>>    class Point(BasicStruct):
>> >>>        __slots__ = {'x': 5, 'y': 7}
>> >>
>> >> So instead they'll write
>> >>     __slots__ = OrderedDict({'x': 5, 'y': 7})
>> >> Causing the same issues?
>> >
>> > Perhaps OrderedDict should reject unordered sources. Hey, here is yet one use case!
>> 
>> I've maintained code that does this:
>> 
>>     self.headers = OrderedDict(headers)
>>     self.origheaders = len(headers)
>> 
>> ? so it can later do this:
>> 
>>     altheaders = list(self.headers.items())[self.origheaders:]
>> 
>> Not a great design, but one that exists in the wild, and would be broken by OrderedDict not allowing a dict as an argument.
>> 
>> Also, this wouldn't allow creating an OrderedDict from an empty dict (which seems far less stupid, but I didn't lead with it because I can't remember seeing it in real code).
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> -- 
> --Guido (mobile)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151109/f33eb03e/attachment.html>

From wes.turner at gmail.com  Fri Nov 13 23:46:44 2015
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 13 Nov 2015 22:46:44 -0600
Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict
In-Reply-To: <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com>
References: <5620D33A.3050709@feete.org>
 <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com>
 <C94FB06F-6919-4D78-9E53-F94A6DB2DFC7@yahoo.com>
 <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com>
Message-ID: <CACfEFw9AGeW6vq8rNS33a2TT31pnq3X1cVdkQHDbtaxr3jnJZw@mail.gmail.com>

On Fri, Oct 16, 2015 at 9:08 PM, Andrew Barnert via Python-ideas <
python-ideas at python.org> wrote:

> Actually, forget all that; it's even simpler.
>
> At least in recent 3.x, the only thing wrong with inheriting from both
> types, assuming you put OrderedDict first, is the __init__ signature. So:
>
>     class OrderedDefaultDict(OrderedDict, defaultdict):
>         def __init__(self, default_factory=None, *a, **kw):
>             OrderedDict.__init__(self, *a, **kw)
>             self.default_factory = default_factory
>
> More importantly, because __missing__ support is built into dict, despite
> the confusing docs for defaultdict, you don't really need defaultdict at
> all here:
>
>     class OrderedDefaultDict(OrderedDict):
>         def __init__(self, default_factory=None, *a, **kw):
>             OrderedDict.__init__(self, *a, **kw)
>             self.default_factory = default_factory
>         def __missing__(self, key):
>             self[key] = value = default_factory()
>             return value
>
> And either of these should work with 2.5+ (according to
> https://docs.python.org/2/library/stdtypes.html#dict that's when
> dict.__missing__ was added).
>
>
Thanks!

- [ ] Could/should maybe either of these make it into the standard library,
that would save a fair amount of copying.

.. Great in combination w/ dict views:
https://docs.python.org/2/library/stdtypes.html#dictionary-view-objects





> Sent from my iPhone
>
> On Oct 16, 2015, at 18:57, Andrew Barnert <abarnert at yahoo.com> wrote:
>
> Sorry, sent too soon?
>
> Sent from my iPhone
>
> On Oct 16, 2015, at 18:39, Andrew Barnert via Python-ideas <
> python-ideas at python.org> wrote:
>
> On Oct 16, 2015, at 03:36, Ian Foote <ian at feete.org> wrote:
>
> I recently wanted to use an OrderedCounter and an OrderedDefaultDict.
> After a bit of googling I discovered that OrderedCounter is quite easy to
> implement:
>
>
> This is already in the docs for OrderedDict, so it shouldn't have taken
> any googling.
>
> from collections import Counter, OrderedDict
>
> class OrderedCounter(Counter, OrderedDict):
>      'Counter that remembers the order elements are first seen'
>      def __repr__(self):
>          return '%s(%r)' % (self.__class__.__name__,
>                             OrderedDict(self))
>      def __reduce__(self):
>          return self.__class__, (OrderedDict(self),)
>
>
> from https://rhettinger.wordpress.com/2011/05/26/super-considered-super/
>
> Unfortunately an OrderedDefaultDict did not seem so easy. I did find
> http://stackoverflow.com/questions/6190331/can-i-do-an-ordered-default-dict-in-python
> which suggests:
>
>
> Most of the trickiness here is handling None as a factory, which is only
> useful for subclasses that implement a custom __missing__
>
>
> ? or that requires two-stage initialization, where it can initialize the
> base with None and then assign the factory later. Still not all that
> common, but it is a separate reason for the extra complexity besides a
> custom __missing__ that doesn't need the factory.
>
> To make something that works like you'd expect except for that part is a
> lot simpler.
>
>
> Off the top of my head (untested, since I'm on my phone):
>
> class DefaultOrderedDict(OrderedDict):
>     def __init__(self, default_factory, *a, **kw):
>         super().__init__(*a, **kw)
>         self.default_factory = default_factory
>     def __getitem__(self, key):
>         try:
>             return super().__getitem__(key)
>         except KeyError:
>             return self.__missing__(key)
>     def __missing__(self, key):
>         self[key] = value = self.default_factory()
>         return value
>     def __repr__(self):
>         return '{}({}, {})'.format(type(self).__name__,
> self.default_factory, dict.__repr__(self))
>
> In fact, if you aren't worried about subclassing, or about being able to
> directly call or pass around a bound__missing__, you can make this even
> simpler.
>
> I may be wrong about this being picklable/copyable without any extra help;
> if so, it still doesn't need the whole complicated "treat a None factory as
> if it doesn't exist" part.
>
> Also, I think the SO recipe works in Python 2.4+, or whichever first added
> OrderedDict, while this requires 3.1 (although it should be obvious how to
> backport it, make sure that OrderedDict isn't an old-style class?).
>
> Anyway, other than those issues, it seems pretty obvious, and I don't
> think there's any part of that functionality that's easy to get wrong.
> Handling a None factory in __missing__ and pickle/copy is easy to get
> wrong, but if you aren't going to subclass this and therefore aren't going
> to try to build that part, you're not going to build it wrong.
>
> And I'm not sure a docs recipe that's mostly code that's unnecessary for
> most uses, hiding the important and simple part, would be a great idea.
>
> from collections import OrderedDict, Callable
> class DefaultOrderedDict(OrderedDict):
>     # Source: http://stackoverflow.com/a/6190500/562769
>     def __init__(self, default_factory=None, *a, **kw):
>         if (default_factory is not None and
>            not isinstance(default_factory, Callable)):
>             raise TypeError('first argument must be callable')
>         OrderedDict.__init__(self, *a, **kw)
>         self.default_factory = default_factory
>
>     def __getitem__(self, key):
>         try:
>             return OrderedDict.__getitem__(self, key)
>         except KeyError:
>             return self.__missing__(key)
>
>     def __missing__(self, key):
>         if self.default_factory is None:
>             raise KeyError(key)
>         self[key] = value = self.default_factory()
>         return value
>
>     def __reduce__(self):
>         if self.default_factory is None:
>             args = tuple()
>         else:
>             args = self.default_factory,
>         return type(self), args, None, None, self.items()
>
>     def copy(self):
>         return self.__copy__()
>
>     def __copy__(self):
>         return type(self)(self.default_factory, self)
>
>     def __deepcopy__(self, memo):
>         import copy
>         return type(self)(self.default_factory,
>                           copy.deepcopy(self.items()))
>
>     def __repr__(self):
>         return 'OrderedDefaultDict(%s, %s)' % (self.default_factory,
>                                                OrderedDict.__repr__(self))
>
>
> This seems to me both easy to get wrong and hard to discover. It also
> risks getting out of sync with updates to collections.
> I'm wondering if this is sufficient justification to add OrderedCounter
> and OrderedDict to collections, either directly or as recipes.
>
>
> Since one of them is already a recipe and the other one would probably not
> be useful as one, I don't think that's a great idea.
>
> Adding OrderedDefaultDict to the module itself might be useful. Or,
> alternatively, put it in PyPI or ActiveState and add a link to it from the
> docs? (I'm pretty sure there are already implementations in both places,
> actually. Also, it would be nice to be able to point at a third-party
> implementation that works in 2.6+/3.2+ or whatever even if it only appears
> in the 3.6 docs.)
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151113/8b8a9993/attachment.html>

From abarnert at yahoo.com  Sat Nov 14 00:17:20 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 13 Nov 2015 21:17:20 -0800
Subject: [Python-ideas] OrderedCounter and OrderedDefaultDict
In-Reply-To: <CACfEFw9AGeW6vq8rNS33a2TT31pnq3X1cVdkQHDbtaxr3jnJZw@mail.gmail.com>
References: <5620D33A.3050709@feete.org>
 <83EDFBB7-679E-4193-8765-7A2B46F0E2F8@yahoo.com>
 <C94FB06F-6919-4D78-9E53-F94A6DB2DFC7@yahoo.com>
 <9BD02B4D-EB84-4D21-B232-DA023882503E@yahoo.com>
 <CACfEFw9AGeW6vq8rNS33a2TT31pnq3X1cVdkQHDbtaxr3jnJZw@mail.gmail.com>
Message-ID: <F8E20356-2A1B-43D7-9352-4472B5AE50D2@yahoo.com>

On Nov 13, 2015, at 20:46, Wes Turner <wes.turner at gmail.com> wrote:
> 
>> On Fri, Oct 16, 2015 at 9:08 PM, Andrew Barnert via Python-ideas <python-ideas at python.org> wrote:
>> Actually, forget all that; it's even simpler.
>> 
>> At least in recent 3.x, the only thing wrong with inheriting from both types, assuming you put OrderedDict first, is the __init__ signature. So:
>> 
>>     class OrderedDefaultDict(OrderedDict, defaultdict):
>>         def __init__(self, default_factory=None, *a, **kw):
>>             OrderedDict.__init__(self, *a, **kw)
>>             self.default_factory = default_factory
>> 
>> More importantly, because __missing__ support is built into dict, despite the confusing docs for defaultdict, you don't really need defaultdict at all here:
>> 
>>     class OrderedDefaultDict(OrderedDict):
>>         def __init__(self, default_factory=None, *a, **kw):
>>             OrderedDict.__init__(self, *a, **kw)
>>             self.default_factory = default_factory
>>         def __missing__(self, key):
>>             self[key] = value = default_factory()
>>             return value
>> 
>> And either of these should work with 2.5+ (according to https://docs.python.org/2/library/stdtypes.html#dict that's when dict.__missing__ was added).
> 
> Thanks!
> 
> - [ ] Could/should maybe either of these make it into the standard library,
> that would save a fair amount of copying.

You could say the same about everything in the recipes, every one-liner like "def identity(x): return x", and so on. But most such things aren't used that often, and there's a cost to putting them in the stdlib?more for people to learn and remember, more code to be maintained, bigger downloads, etc. So just saying "maybe it would save some copying" isn't an argument for adding it to the stdlib.

Adding OrderedDefaultDict as a docs recipe (as OrderedCounter already is) might be worth doing. 
Reorganizing the docs a bit to make it more obvious (by better highlighting __missing__, and making it clear that it's a method of dict rather than something special about defaultdict) seems even more likely to be worth it. If someone has an idea of how it should read, just file a docs bug and submit a patch and see if whoever's in charge of that area says it needs to come back here.

> .. Great in combination w/ dict views:
> https://docs.python.org/2/library/stdtypes.html#dictionary-view-objects

Of course, and that works out of the box.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151113/c212374c/attachment-0001.html>

From matt at whoosh.ca  Mon Nov 16 01:17:37 2015
From: matt at whoosh.ca (Matt Chaput)
Date: Mon, 16 Nov 2015 01:17:37 -0500
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
Message-ID: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>

Since PyCharm added support for pep484 I've been working on adding type hints to the Whoosh code base. It makes it a lot more enjoyable to work with the code in the IDE. As I add more and more hints, it's already revealed bugs where arguments were accidentally swapped, or interfaces weren't updated everywhere. I feel much more confident working in the (quite large) codebase with typing.

Basically, I'm loving the idea of PEP484. For me, it's a huge improvement in the development experience. However, I've already run into a few issues. I'm not sure if these were discussed during the development of PEP484.

1. The biggest problem, most perplexing problem for me is circular imports. That type hints must actually be imported at the top level is a MASSIVE limitation, at least in a big codebase. I started off importing the types I needed, then switched to the "import the module, specify the type in a string" trick, but even that has failed.

In a large/complex codebase, the more things you have to import at the top level (to use for typing), the more potential import circles you create. Now I've hit an impasse where every step in the circle is using the module/string trick, but it doesn't help because I need to subclass something in the circle.

For now, I'm going to have to workaround this, either by trying to move "interface" base classes out to their own side package, or just not type hinting some things. Neither is appetizing at all.

I think a good solution might be if the typing system could have its own special "imports" that are only used by the typing system, e.g. something like this at the top of a file:

    # "real" import
    from foo import bar

    # imports just for type checking
    __typeimports__ = """
    from baz import Qux
    """

2. It would be really nice if we could have "type aliasing" (or whatever it's called). I have a lot of types that are just something like "Tuple[int, int]", so type checking doesn't help much. It would be much more useful if I have a value that Python sees as (e.g.) an int, but have the type system track it as something more specific. Something like this:

    DocId = typing.TypeAlias(int)
    DocLength = typing.TypeAlias(int)

    def id_and_length() -> Tuple[DocId, Length]:
        docid = 5  # type: DocId
        length = 10  # type: Length
        return docid, length


3. I can't tell from the PEP484 documentation... If I have a fully hinted base class, and subclass it with the same signature for every method, am I still supposed to hint all the arguments and returns in the subclass? That's what I've been doing, but it's pretty tedious.

Cheers,


From vito.detullio at gmail.com  Mon Nov 16 01:51:02 2015
From: vito.detullio at gmail.com (Vito De Tullio)
Date: Mon, 16 Nov 2015 07:51:02 +0100
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
Message-ID: <n2buco$sts$1@ger.gmane.org>

Matt Chaput wrote:

> 2. It would be really nice if we could have "type aliasing" (or whatever
> it's called). I have a lot of types that are just something like
> "Tuple[int, int]", so type checking doesn't help much. It would be much
> more useful if I have a value that Python sees as (e.g.) an int, but have
> the type system track it as something more specific. Something like this:
> 
>     DocId = typing.TypeAlias(int)
>     DocLength = typing.TypeAlias(int)
> 
>     def id_and_length() -> Tuple[DocId, Length]:
>         docid = 5  # type: DocId
>         length = 10  # type: Length
>         return docid, length

like... type aliases? https://www.python.org/dev/peps/pep-0484/#type-aliases

-- 
By ZeD


From spinenkoia at gmail.com  Mon Nov 16 10:03:32 2015
From: spinenkoia at gmail.com (=?UTF-8?B?0JjQstCw0L0g0KHQv9C40L3QtdC90LrQvg==?=)
Date: Mon, 16 Nov 2015 20:03:32 +0500
Subject: [Python-ideas] The prohibition overrides the class method
Message-ID: <CAM+FtgWTKvYhcxGeEeXnYcCF+G7dYgD19SJWMEjmbiaXmxsZJQ@mail.gmail.com>

Hello, I want to apologize for my English.
I propose to make an analogue of the final specifier in C ++. In class
abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod.

I have a sketch of implementation:
def finalmethod(funcobj):
    funcobj.__isfinalmethod__ = True
    return funcobj

class Override(ABCMeta):
    def __init__(cls, name, bases, namespace):
        super(Override, cls).__init__(name, bases, namespace)
        finals = {name
                        for name, value in namespace.items()
                        if getattr(value, "__isfinalmethod__", False)
        }

        for base in bases:
            for name in getattr(base, "__finalmethods__", set()):
                value = getattr(cls, name, None)
                if getattr(value, "__isfinalmethod__", False):
                    finals.add(name)
                else:
                    raise TypeError("function '" + str(value.__name__) + "'
in class '" + str(cls.__name__) + "' is final")
        cls.__finalmethods__ = frozenset(finals)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/dde95697/attachment.html>

From rosuav at gmail.com  Mon Nov 16 10:44:45 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 17 Nov 2015 02:44:45 +1100
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CAM+FtgWTKvYhcxGeEeXnYcCF+G7dYgD19SJWMEjmbiaXmxsZJQ@mail.gmail.com>
References: <CAM+FtgWTKvYhcxGeEeXnYcCF+G7dYgD19SJWMEjmbiaXmxsZJQ@mail.gmail.com>
Message-ID: <CAPTjJmroqtDXqm0mEN7G8Vwrbd+YVnWdf=cZbTXZUe7NKR3hEg@mail.gmail.com>

On Tue, Nov 17, 2015 at 2:03 AM, ???? ???????? <spinenkoia at gmail.com> wrote:
> Hello, I want to apologize for my English.

It's fine - and a lot better than my Russian! (Or Ukrainian, or any of
the other languages that use a Cyrillic script. See, I can't even tell
one from another - all I know is which script you've used.)

> I propose to make an analogue of the final specifier in C ++. In class
> abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod.

It's not common in Python to actually enforce this kind of thing.
What's the advantage of stopping people from overriding some methods?
In Java, as I understand it, the compiler optimizes calls to final
methods; but in Python, you can even add methods to individual
instances of a class:

>>> class X:
...     def method(self):
...         print("I am the regular method.")
...
>>> x = X()
>>> x.method()
I am the regular method.
>>> x.method = lambda: print("I am the custom method.")
>>> x.method()
I am the custom method.

Can you show an example of how this decorator would help your project?
If so, it might make a useful recipe, rather than actually a part of
the language or standard library - it can go into a collection of
"here's some neat tricks you can do with function decorators" tips.

ChrisA

From matt at whoosh.ca  Mon Nov 16 10:48:54 2015
From: matt at whoosh.ca (Matt Chaput)
Date: Mon, 16 Nov 2015 10:48:54 -0500
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <n2buco$sts$1@ger.gmane.org>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <n2buco$sts$1@ger.gmane.org>
Message-ID: <0142D3D9-B73C-4D7C-990D-8C3D191B461A@whoosh.ca>

> like... type aliases? https://www.python.org/dev/peps/pep-0484/#type-aliases

As I read it, that's just a way to avoid having to spell out the same type definition more than once. It's not the same as being able to define a new type for the type system which Python sees as a plain old primitive type.

I'm probably not using the right programming language theory word for this concept :)

Matt


From rosuav at gmail.com  Mon Nov 16 10:53:31 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 17 Nov 2015 02:53:31 +1100
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <0142D3D9-B73C-4D7C-990D-8C3D191B461A@whoosh.ca>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <n2buco$sts$1@ger.gmane.org>
 <0142D3D9-B73C-4D7C-990D-8C3D191B461A@whoosh.ca>
Message-ID: <CAPTjJmrWT3OQ8c23OPyxnuLccQ76R_NLLw-RSeA9gHEv6uTWrg@mail.gmail.com>

On Tue, Nov 17, 2015 at 2:48 AM, Matt Chaput <matt at whoosh.ca> wrote:
> As I read it, that's just a way to avoid having to spell out the same type definition more than once. It's not the same as being able to define a new type for the type system which Python sees as a plain old primitive type.
>
> I'm probably not using the right programming language theory word for this concept :)

I'm not entirely sure, but I think you might be able to subclass
something to create a special form. A length might be a subclass of
integer, and a docid could be a different subclass of integer. Lengths
and document IDs could then be counted as distinct from each other,
although any function that expects an integer would happily accept
either. (That might not be appropriate for docid, but it would for
length (eg range() should accept them), so you might want to do docid
some other way.)

ChrisA

From ian.g.kelly at gmail.com  Mon Nov 16 10:55:24 2015
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Mon, 16 Nov 2015 08:55:24 -0700
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CAPTjJmroqtDXqm0mEN7G8Vwrbd+YVnWdf=cZbTXZUe7NKR3hEg@mail.gmail.com>
References: <CAM+FtgWTKvYhcxGeEeXnYcCF+G7dYgD19SJWMEjmbiaXmxsZJQ@mail.gmail.com>
 <CAPTjJmroqtDXqm0mEN7G8Vwrbd+YVnWdf=cZbTXZUe7NKR3hEg@mail.gmail.com>
Message-ID: <CALwzidnGVzJGYz_H4D3QmobpjGmRXRvOWPc7SMxfi3qFe4Qi5Q@mail.gmail.com>

On Mon, Nov 16, 2015 at 8:44 AM, Chris Angelico <rosuav at gmail.com> wrote:
> On Tue, Nov 17, 2015 at 2:03 AM, ???? ???????? <spinenkoia at gmail.com> wrote:
>> Hello, I want to apologize for my English.
>
> It's fine - and a lot better than my Russian! (Or Ukrainian, or any of
> the other languages that use a Cyrillic script. See, I can't even tell
> one from another - all I know is which script you've used.)
>
>> I propose to make an analogue of the final specifier in C ++. In class
>> abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod.
>
> It's not common in Python to actually enforce this kind of thing.
> What's the advantage of stopping people from overriding some methods?

I would add the point that while abstractmethod helps the programmer
by warning when they've forgotten to implement a method, finalmethod
is more likely to just get in the way when the programmer needs to
override a method, but the class author has arbitrarily decided that
they can't. Python usually tries to give the programmer what they need
while staying out of their way as much as possible.

From srkunze at mail.de  Mon Nov 16 11:33:53 2015
From: srkunze at mail.de (Sven R. Kunze)
Date: Mon, 16 Nov 2015 17:33:53 +0100
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <n2buco$sts$1@ger.gmane.org>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <n2buco$sts$1@ger.gmane.org>
Message-ID: <564A0571.1010107@mail.de>

Or like structural typing - there has already been some discussion on 
this list:

https://github.com/ambv/typehinting/issues/11#issuecomment-138133867

On 16.11.2015 07:51, Vito De Tullio wrote:
> Matt Chaput wrote:
>
>> 2. It would be really nice if we could have "type aliasing" (or whatever
>> it's called). I have a lot of types that are just something like
>> "Tuple[int, int]", so type checking doesn't help much. It would be much
>> more useful if I have a value that Python sees as (e.g.) an int, but have
>> the type system track it as something more specific. Something like this:
>>
>>      DocId = typing.TypeAlias(int)
>>      DocLength = typing.TypeAlias(int)
>>
>>      def id_and_length() -> Tuple[DocId, Length]:
>>          docid = 5  # type: DocId
>>          length = 10  # type: Length
>>          return docid, length
> like... type aliases? https://www.python.org/dev/peps/pep-0484/#type-aliases
>


From srkunze at mail.de  Mon Nov 16 11:27:20 2015
From: srkunze at mail.de (Sven R. Kunze)
Date: Mon, 16 Nov 2015 17:27:20 +0100
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CALwzidnGVzJGYz_H4D3QmobpjGmRXRvOWPc7SMxfi3qFe4Qi5Q@mail.gmail.com>
References: <CAM+FtgWTKvYhcxGeEeXnYcCF+G7dYgD19SJWMEjmbiaXmxsZJQ@mail.gmail.com>
 <CAPTjJmroqtDXqm0mEN7G8Vwrbd+YVnWdf=cZbTXZUe7NKR3hEg@mail.gmail.com>
 <CALwzidnGVzJGYz_H4D3QmobpjGmRXRvOWPc7SMxfi3qFe4Qi5Q@mail.gmail.com>
Message-ID: <564A03E8.2050709@mail.de>

It's a well-known feature from other languages ("sealed"). Though, I 
tend to agree here that Python should not create barriers.


What would a real-world example look like? I cannot imagine where people 
can decide *properly***when to place that modifier.


Best,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/45d9e0ac/attachment.html>

From abarnert at yahoo.com  Mon Nov 16 12:07:43 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 16 Nov 2015 09:07:43 -0800
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CAM+FtgWTKvYhcxGeEeXnYcCF+G7dYgD19SJWMEjmbiaXmxsZJQ@mail.gmail.com>
References: <CAM+FtgWTKvYhcxGeEeXnYcCF+G7dYgD19SJWMEjmbiaXmxsZJQ@mail.gmail.com>
Message-ID: <23E8DA7D-CBCB-410E-8A0A-595C0AFCADCF@yahoo.com>

On Nov 16, 2015, at 07:03, ???? ???????? <spinenkoia at gmail.com> wrote:
> 
> Hello, I want to apologize for my English.
> I propose to make an analogue of the final specifier in C ++. In class abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod.

Since the C++ standard is (mostly) written based on public proposals on open-std.org, presumably there's some discussion of the motivation behind adding this feature to C++11. It would be worth knowing what that motivation is, to see whether it applies to Python. For example, if it was proposed just to provide optimization opportunities, or to be closer to C++/CLI, then it probably isn't worth adding to Python (unless you have Python-specific use cases to offer, and it's just a coincidence that C++ and Python could use very similar features). On the other hand, if it solves a real problem with making some hierarchies easier to read, or removing opportunities for hard-to-diagnose errors, then it may be (assuming those problems have equivalents in Python).

Sometimes there are also blog posts presenting the idea to C++ users instead of the C++ committee, which often get the motivation across better (e.g., explaining how rvalue references solve the forwarding and move problems that boost ran into with the previous version of C++).

So, if you want to suggest a feature copied from C++11, it's probably worth searching for those references and providing links and summaries.

I suspect that it's primarily for optimization. In C++, by default, methods can't be overridden, so if the compiler sees spam.cook() it can just compile in a direct call to Spam::cook. But if cook() is declared virtual in either Spam or any of its superclasses, it instead has to compile an indirect call to spam->vtable[1], which also means it will usually keep Spam::vtable in L2 cache whenever a Spam object is around, and so on, which can make a big difference. So, when the compiler can prove that spam really is a Spam, and not some subtype (e.g., because the function constructed it locally, or accepted it by value), it will optimize out the virtual call. What final adds is that if cook() is declared final in Spam or any of its superclasses, the compiler can optimize out the virtual call even if the value could actually be a subtype (e.g., because the function accepted it by reference), because it would be illegal for any such subtype to have overridden the method. So, that can be a huge benefit. Obviously, none of this applies in Python. Methods are always looked up by name in the value's dict, then the class and each of its bases at runtime; there's no concept of a variable's type to start the search at, and no way to record something that can be directly called at compile time (especially since type, function, and method objects are themselves constructed at runtime).

> I have a sketch of implementation:
> def finalmethod(funcobj):
>     funcobj.__isfinalmethod__ = True
>     return funcobj
> 
> class Override(ABCMeta):
>     def __init__(cls, name, bases, namespace):
>         super(Override, cls).__init__(name, bases, namespace)
>         finals = {name
>                         for name, value in namespace.items()
>                         if getattr(value, "__isfinalmethod__", False)
>         }
> 
>         for base in bases:
>             for name in getattr(base, "__finalmethods__", set()):
>                 value = getattr(cls, name, None)
>                 if getattr(value, "__isfinalmethod__", False):
>                     finals.add(name)
>                 else:
>                     raise TypeError("function '" + str(value.__name__) + "' in class '" + str(cls.__name__) + "' is final")
>         cls.__finalmethods__ = frozenset(finals)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/0c98f2c2/attachment.html>

From abarnert at yahoo.com  Mon Nov 16 12:19:14 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 16 Nov 2015 09:19:14 -0800
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
Message-ID: <CEBE6791-2136-4F3C-8EBA-B476E2A9DF38@yahoo.com>

On Nov 15, 2015, at 22:17, Matt Chaput <matt at whoosh.ca> wrote:
> 
> 2. It would be really nice if we could have "type aliasing" (or whatever it's called). I have a lot of types that are just something like "Tuple[int, int]", so type checking doesn't help much. It would be much more useful if I have a value that Python sees as (e.g.) an int, but have the type system track it as something more specific. Something like this:
> 
>    DocId = typing.TypeAlias(int)
>    DocLength = typing.TypeAlias(int)
> 
>    def id_and_length() -> Tuple[DocId, Length]:
>        docid = 5  # type: DocId
>        length = 10  # type: Length
>        return docid, length

It sounds like you want a subtype that adds no new semantics or other runtime effects. Like this:

    class DocId(int): pass
    class DocLength(int): pass

    def id_and_length() -> Tuple[DocId, DocLength]:
        return DocId(5), DocLength(10)

These will behave exactly like int objects, except that you can (dynamically or statically) type check them. It does mean that if someone uses "type(spam[0]) == int" it will fail, but I think if you care either way, you'd actually want it to fail. Meanwhile, "isinstance(spam[0], int)" or "spam[0] + eggs" or even using it in a function that requires something usable as a C long will work as expected. The object will also be the same size as an int in memory (although it will pickle a little bigger). It can't be optimized into a constant at compile time, but I doubt that's ever an issue. And it makes your intentions perfectly clear.

From rosuav at gmail.com  Mon Nov 16 12:29:00 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 17 Nov 2015 04:29:00 +1100
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <CEBE6791-2136-4F3C-8EBA-B476E2A9DF38@yahoo.com>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <CEBE6791-2136-4F3C-8EBA-B476E2A9DF38@yahoo.com>
Message-ID: <CAPTjJmpBDWLPSQT0Q0Kp0t2WyYogZ2-ObyBHHjOgS+U6M0jjpQ@mail.gmail.com>

On Tue, Nov 17, 2015 at 4:19 AM, Andrew Barnert via Python-ideas
<python-ideas at python.org> wrote:
> The object will also be the same size as an int in memory (although it will pickle a little bigger).

To make that true, you have to use slots:

>>> import sys
>>> sys.getsizeof(12341234123412341234)
36
>>> class X(int): pass
...
>>> sys.getsizeof(X(12341234123412341234))
60
>>> class X(int): __slots__=()
...
>>> sys.getsizeof(X(12341234123412341234))
36

ChrisA

From guido at python.org  Mon Nov 16 12:31:39 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Nov 2015 09:31:39 -0800
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <CEBE6791-2136-4F3C-8EBA-B476E2A9DF38@yahoo.com>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <CEBE6791-2136-4F3C-8EBA-B476E2A9DF38@yahoo.com>
Message-ID: <CAP7+vJKhDGgUMMLdBj1_v9U5x_UEWgZyyNW=Hq0npWH+_ka3Xw@mail.gmail.com>

If you really want DocId to be as small as int you should add
`__slots__ = ()` to the class def. It's still a bit slower (some code
special-cases exactly int) and it also has some other imperfections --
e.g. DocId(2) + DocId(2) == int(4), not DocId(4). But that's probably
okay for a document ID (and the solution is way too messy to use).

But in general this is a pretty verbose option. If the main purpose is
for documentation maybe type aliases, only used on annotations and
other types are enough (`DocId = int`). But yeah, the type checker
won't track it. (That's a possible future feature though.)

On Mon, Nov 16, 2015 at 9:19 AM, Andrew Barnert via Python-ideas
<python-ideas at python.org> wrote:
> On Nov 15, 2015, at 22:17, Matt Chaput <matt at whoosh.ca> wrote:
>>
>> 2. It would be really nice if we could have "type aliasing" (or whatever it's called). I have a lot of types that are just something like "Tuple[int, int]", so type checking doesn't help much. It would be much more useful if I have a value that Python sees as (e.g.) an int, but have the type system track it as something more specific. Something like this:
>>
>>    DocId = typing.TypeAlias(int)
>>    DocLength = typing.TypeAlias(int)
>>
>>    def id_and_length() -> Tuple[DocId, Length]:
>>        docid = 5  # type: DocId
>>        length = 10  # type: Length
>>        return docid, length
>
> It sounds like you want a subtype that adds no new semantics or other runtime effects. Like this:
>
>     class DocId(int): pass
>     class DocLength(int): pass
>
>     def id_and_length() -> Tuple[DocId, DocLength]:
>         return DocId(5), DocLength(10)
>
> These will behave exactly like int objects, except that you can (dynamically or statically) type check them. It does mean that if someone uses "type(spam[0]) == int" it will fail, but I think if you care either way, you'd actually want it to fail. Meanwhile, "isinstance(spam[0], int)" or "spam[0] + eggs" or even using it in a function that requires something usable as a C long will work as expected. The object will also be the same size as an int in memory (although it will pickle a little bigger). It can't be optimized into a constant at compile time, but I doubt that's ever an issue. And it makes your intentions perfectly clear.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
--Guido van Rossum (python.org/~guido)

From spinenkoia at gmail.com  Mon Nov 16 13:15:55 2015
From: spinenkoia at gmail.com (=?UTF-8?B?0JjQstCw0L0g0KHQv9C40L3QtdC90LrQvg==?=)
Date: Mon, 16 Nov 2015 23:15:55 +0500
Subject: [Python-ideas] The prohibition overrides the class method
Message-ID: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>

Thank you for the constructive responses, perhaps, I agree with you that
this is not in the spirit of Python.
As for the real example of the use, sometimes there to make the
implementation of the method in the base class, but that none of the rest
of the development team could not accidentally or unknowingly override this
method (perhaps due to the need to be unsuccessful architecture).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/91a5f583/attachment.html>

From rymg19 at gmail.com  Mon Nov 16 13:24:51 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 16 Nov 2015 12:24:51 -0600
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
References: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
Message-ID: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>

You could always put this in the docstring:

DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE.


Short, the-the-point, and evokes fear.

...

Seriously, though, I personally think this might go well with the whole PEP 484 type-annotation thing. I mean, if there are already generics and abstract methods. If we're going for static typing, adding final would fit well. Something like:

@finalmethod
def whatever(self): ...

The type checker (in this case, Mypy) would check it, but there would be no runtime effect.

On November 16, 2015 12:15:55 PM CST, "???? ????????" <spinenkoia at gmail.com> wrote:
>Thank you for the constructive responses, perhaps, I agree with you
>that
>this is not in the spirit of Python.
>As for the real example of the use, sometimes there to make the
>implementation of the method in the base class, but that none of the
>rest
>of the development team could not accidentally or unknowingly override
>this
>method (perhaps due to the need to be unsuccessful architecture).
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>https://mail.python.org/mailman/listinfo/python-ideas
>Code of Conduct: http://python.org/psf/codeofconduct/

-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/9cad8f4b/attachment.html>

From joejev at gmail.com  Mon Nov 16 14:19:28 2015
From: joejev at gmail.com (Joseph Jevnik)
Date: Mon, 16 Nov 2015 14:19:28 -0500
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
References: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
 <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
Message-ID: <CAHGq92UnYmpQTLW-W_BbcW1sr-fUVFyvfwrAn3L8Wpngp_DKmw@mail.gmail.com>

def finalmethod(f):
    f.__doc__ += 'DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD,
YOU WILL DIE!'
    return f

On Mon, Nov 16, 2015 at 1:24 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:

> You could always put this in the docstring:
>
> DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE.
>
>
> Short, the-the-point, and evokes fear.
>
> ...
>
> Seriously, though, I personally think this might go well with the whole
> PEP 484 type-annotation thing. I mean, if there are already generics and
> abstract methods. If we're going for static typing, adding final would fit
> well. Something like:
>
> @finalmethod
> def whatever(self): ...
>
> The type checker (in this case, Mypy) would check it, but there would be
> no runtime effect.
>
> On November 16, 2015 12:15:55 PM CST, "???? ????????" <
> spinenkoia at gmail.com> wrote:
>>
>> Thank you for the constructive responses, perhaps, I agree with you that
>> this is not in the spirit of Python.
>> As for the real example of the use, sometimes there to make the
>> implementation of the method in the base class, but that none of the rest
>> of the development team could not accidentally or unknowingly override this
>> method (perhaps due to the need to be unsuccessful architecture).
>>
>> ------------------------------
>>
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
> --
> Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/f0b7ef72/attachment.html>

From abarnert at yahoo.com  Mon Nov 16 16:59:20 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 16 Nov 2015 13:59:20 -0800
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
References: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
 <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
Message-ID: <AB6A1F80-1512-4090-971F-7728AA2546E0@yahoo.com>

On Nov 16, 2015, at 10:24, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> 
> You could always put this in the docstring:
> 
> DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE.
> 
> 
> Short, the-the-point, and evokes fear.

Sure, until the first time someone accidentally violates it and doesn't die. Then you will have trained him to ignore your docstrings, comments, type annotations, and warnings completely. (Of course you could take the OP's code and add some stuff that causes the hard drive casing and spindle to break, firing the platter at the developer's skull like one of those discs from the Dolph Lundgren movie I Come in Peace, but that would break compatibility with computers that only have SSDs, and I don't think that's feasible in 2015.)
> 
> ...
> 
> Seriously, though, I personally think this might go well with the whole PEP 484 type-annotation thing. I mean, if there are already generics and abstract methods. If we're going for static typing, adding final would fit well. Something like:
> 
> @finalmethod
> def whatever(self): ...
> 
> The type checker (in this case, Mypy) would check it, but there would be no runtime effect.

Yes, that makes sense. If the feature isn't supposed to have any runtime effect, and is modeled after something that's definitely compile-time-centric in C++ and friends, signaling that by importing it from typing seems to be a pretty clear way to get that across.

Although I still don't know why we need it. See below:

>> On November 16, 2015 12:15:55 PM CST, "???? ????????" <spinenkoia at gmail.com> wrote:
>> Thank you for the constructive responses, perhaps, I agree with you that this is not in the spirit of Python.
>> As for the real example of the use, sometimes there to make the implementation of the method in the base class, but that none of the rest of the development team could not accidentally or unknowingly override this method (perhaps due to the need to be unsuccessful architecture).

If it's just part of the implementation of the base class, just double-underscore it so that it can't be accidentally overridden (or called).

It's only when something is also part of the interface that this is a problem. And it's kind of strange to have something be part of the interface of a type, but not be refinable by subtypes. When would you want that? (And as for accidental overriding, if a class claims doesn't even know what the interface of some supertype is, it probably shouldn't be inheriting?either it should be a mixin for someone who does know the supertype to compose, or it should be encapsulating rather than inheriting.)

Do you have a concrete example of some code where you wanted finalmethod, and would use it to prevent the rest of your team from overriding it?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/3c3e7fdb/attachment-0001.html>

From abarnert at yahoo.com  Mon Nov 16 17:16:38 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 16 Nov 2015 14:16:38 -0800
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <CAP7+vJKhDGgUMMLdBj1_v9U5x_UEWgZyyNW=Hq0npWH+_ka3Xw@mail.gmail.com>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <CEBE6791-2136-4F3C-8EBA-B476E2A9DF38@yahoo.com>
 <CAP7+vJKhDGgUMMLdBj1_v9U5x_UEWgZyyNW=Hq0npWH+_ka3Xw@mail.gmail.com>
Message-ID: <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com>

On Nov 16, 2015, at 09:31, Guido van Rossum <guido at python.org> wrote:
> 
> If you really want DocId to be as small as int you should add
> `__slots__ = ()` to the class def. It's still a bit slower (some code
> special-cases exactly int) and it also has some other imperfections --
> e.g. DocId(2) + DocId(2) == int(4), not DocId(4). But that's probably
> okay for a document ID (and the solution is way too messy to use).

Good points. But I don't think any of those things matter for the OP's use case. And surely being able to runtime-check the type and get the same results as compile-time checks, and not requiring any new language features, are advantages?

I'm sure there are cases where the performance matters, but are there enough cases where the performance matters, runtime typing (e.g., in logs and debugger) doesn't matter, and static typing does matter?

As a side note, most of the C++ family have something akin to "typedef" and/or "using" type aliases that are explicitly _not_ treated as different types by the static typing algorithm: if your function wants a DocId and I pass an int, you get neither an error nor a warning, because they are literally just different names for the same type. So, if Python really does need a feature that's stricter than what C++ and friends do, using a name that evokes comparisons to their feature is probably not a good idea. The way C++ does what the OP is asking for is exactly what I suggested: an empty class that publicly inherits the type. (Of course it only works with class types, not simple types like int, in C++. But Python only has class types, so that wouldn't be a problem.)

> But in general this is a pretty verbose option.

Even with __slots__, it's fewer characters, and the same number of concepts, as the OP's proposed solution.

I think it's also more obvious to read: you're saying that DocId is a brand-new type that's a subtype of int (that adds nothing), which by definition means it can be used anywhere an int can be used, but not vice-versa. You don't have to know anything about MyPy or isinstance or anything else to figure out how they'll handle it (except the basic knowledge that Python's type system is generally a sensible OO system).

> If the main purpose is
> for documentation maybe type aliases, only used on annotations and
> other types are enough (`DocId = int`). But yeah, the type checker
> won't track it. (That's a possible future feature though.)
> 
> On Mon, Nov 16, 2015 at 9:19 AM, Andrew Barnert via Python-ideas
> <python-ideas at python.org> wrote:
>> On Nov 15, 2015, at 22:17, Matt Chaput <matt at whoosh.ca> wrote:
>>> 
>>> 2. It would be really nice if we could have "type aliasing" (or whatever it's called). I have a lot of types that are just something like "Tuple[int, int]", so type checking doesn't help much. It would be much more useful if I have a value that Python sees as (e.g.) an int, but have the type system track it as something more specific. Something like this:
>>> 
>>>   DocId = typing.TypeAlias(int)
>>>   DocLength = typing.TypeAlias(int)
>>> 
>>>   def id_and_length() -> Tuple[DocId, Length]:
>>>       docid = 5  # type: DocId
>>>       length = 10  # type: Length
>>>       return docid, length
>> 
>> It sounds like you want a subtype that adds no new semantics or other runtime effects. Like this:
>> 
>>    class DocId(int): pass
>>    class DocLength(int): pass
>> 
>>    def id_and_length() -> Tuple[DocId, DocLength]:
>>        return DocId(5), DocLength(10)
>> 
>> These will behave exactly like int objects, except that you can (dynamically or statically) type check them. It does mean that if someone uses "type(spam[0]) == int" it will fail, but I think if you care either way, you'd actually want it to fail. Meanwhile, "isinstance(spam[0], int)" or "spam[0] + eggs" or even using it in a function that requires something usable as a C long will work as expected. The object will also be the same size as an int in memory (although it will pickle a little bigger). It can't be optimized into a constant at compile time, but I doubt that's ever an issue. And it makes your intentions perfectly clear.
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)

From guido at python.org  Mon Nov 16 17:32:42 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Nov 2015 14:32:42 -0800
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <CEBE6791-2136-4F3C-8EBA-B476E2A9DF38@yahoo.com>
 <CAP7+vJKhDGgUMMLdBj1_v9U5x_UEWgZyyNW=Hq0npWH+_ka3Xw@mail.gmail.com>
 <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com>
Message-ID: <CAP7+vJLpD1ZaSc_XvumMiZdwtxz5_e+S=wSx6iTRGJJ1E3jHCw@mail.gmail.com>

On Mon, Nov 16, 2015 at 2:16 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> On Nov 16, 2015, at 09:31, Guido van Rossum <guido at python.org> wrote:
>>
>> If you really want DocId to be as small as int you should add
>> `__slots__ = ()` to the class def. It's still a bit slower (some code
>> special-cases exactly int) and it also has some other imperfections --
>> e.g. DocId(2) + DocId(2) == int(4), not DocId(4). But that's probably
>> okay for a document ID (and the solution is way too messy to use).
>
> Good points. But I don't think any of those things matter for the OP's use case.

Well, *you* claimed there was no size difference.

> And surely being able to runtime-check the type and get the same results as compile-time checks, and not requiring any new language features, are advantages?

Now you're changing the subject -- Matt very specifically asked for
something that at runtime was just an int but was tracked more
specifically by the type checker. We currently don't have that:
there's either some runtime overhead (your solution) or the type
checker doesn't track it (type aliasing as PEP 484 defines it).

Anyway, lots of things PEP 484 tracks cannot be checked at runtime
(e.g. anything involving TypeVar).

> I'm sure there are cases where the performance matters, but are there enough cases where the performance matters, runtime typing (e.g., in logs and debugger) doesn't matter, and static typing does matter?

A key requirement for PEP 484 is that there's no runtime overhead
(apart from modules loading a tiny bit slower because the annotations
are evaluated). Otherwise people will be afraid of using it. Having
zillions of subclasses of builtin types at runtime just so the type
checker can track them separately is in direct contradiction to this
requirement. (Note that the BDFL-delegate specifically insisted that
we remove isinstance() support from PEP-484's types.)

> As a side note, most of the C++ family have something akin to "typedef" and/or "using" type aliases that are explicitly _not_ treated as different types by the static typing algorithm: if your function wants a DocId and I pass an int, you get neither an error nor a warning, because they are literally just different names for the same type.

Yes, that's what PEP 484 type aliases do too.

> So, if Python really does need a feature that's stricter than what C++ and friends do, using a name that evokes comparisons to their feature is probably not a good idea. The way C++ does what the OP is asking for is exactly what I suggested: an empty class that publicly inherits the type. (Of course it only works with class types, not simple types like int, in C++. But Python only has class types, so that wouldn't be a problem.)

But C++ classes like that have no runtime overhead. The equivalent
Python syntax does, alas.

>> But in general this is a pretty verbose option.
>
> Even with __slots__, it's fewer characters, and the same number of concepts, as the OP's proposed solution.

But such a hack -- everytime someone sees that they'll wonder why the
`__slots__ = ()`. And AFAICT Matt didn't propose any solution -- he
just showed some example code indicating what he wanted to be taken
care of automatically.

> I think it's also more obvious to read: you're saying that DocId is a brand-new type that's a subtype of int (that adds nothing), which by definition means it can be used anywhere an int can be used, but not vice-versa. You don't have to know anything about MyPy or isinstance or anything else to figure out how they'll handle it (except the basic knowledge that Python's type system is generally a sensible OO system).

Yeah, but I'd still prefer a solution that is only read by the type
checker. And defining subclasses of int is pretty uncommon, so it'll
confuse the heck out of a lot of readers -- much more so than other
parts of type annotations (which are ignorable).

Anyway, let's stop bickering until Matt has had the time to read the
thread and respond.

-- 
--Guido van Rossum (python.org/~guido)

From rymg19 at gmail.com  Mon Nov 16 17:45:34 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 16 Nov 2015 16:45:34 -0600
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <AB6A1F80-1512-4090-971F-7728AA2546E0@yahoo.com>
References: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
 <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
 <AB6A1F80-1512-4090-971F-7728AA2546E0@yahoo.com>
Message-ID: <CAO41-mOa8ijF6YX-4qJZ4oUwPaV4NVnR-Lv419RSHJmn34pFZA@mail.gmail.com>

On Mon, Nov 16, 2015 at 3:59 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Nov 16, 2015, at 10:24, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>
> You could always put this in the docstring:
>
> DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE.
>
>
>
> Short, the-the-point, and evokes fear.
>
>
> Sure, until the first time someone accidentally violates it and doesn't
> die. Then you will have trained him to ignore your docstrings, comments,
> type annotations, and warnings completely. (Of course you could take the
> OP's code and add some stuff that causes the hard drive casing and spindle
> to break, firing the platter at the developer's skull like one of those
> discs from the Dolph Lundgren movie I Come in Peace, but that would break
> compatibility with computers that only have SSDs, and I don't think that's
> feasible in 2015.)
>

Well, it's just as easy to ignore type annotations and call `int(None)`.

(You should definitely patent that hard drive stuff, though! ;)


>
> ...
>
> Seriously, though, I personally think this might go well with the whole
> PEP 484 type-annotation thing. I mean, if there are already generics and
> abstract methods. If we're going for static typing, adding final would fit
> well. Something like:
>
> @finalmethod
> def whatever(self): ...
>
> The type checker (in this case, Mypy) would check it, but there would be
> no runtime effect.
>
>
> Yes, that makes sense. If the feature isn't supposed to have any runtime
> effect, and is modeled after something that's definitely
> compile-time-centric in C++ and friends, signaling that by importing it
> from typing seems to be a pretty clear way to get that across.
>
> Although I still don't know why we need it. See below:
>
> On November 16, 2015 12:15:55 PM CST, "???? ????????" <
> spinenkoia at gmail.com> wrote:
>>
>> Thank you for the constructive responses, perhaps, I agree with you that
>> this is not in the spirit of Python.
>> As for the real example of the use, sometimes there to make the
>> implementation of the method in the base class, but that none of the rest
>> of the development team could not accidentally or unknowingly override this
>> method (perhaps due to the need to be unsuccessful architecture).
>>
>
> If it's just part of the implementation of the base class, just
> double-underscore it so that it can't be accidentally overridden (or
> called).
>
> It's only when something is also part of the interface that this is a
> problem. And it's kind of strange to have something be part of the
> interface of a type, but not be refinable by subtypes. When would you want
> that? (And as for accidental overriding, if a class claims doesn't even
> know what the interface of some supertype is, it probably shouldn't be
> inheriting?either it should be a mixin for someone who does know the
> supertype to compose, or it should be encapsulating rather than inheriting.)
>
> Do you have a concrete example of some code where you wanted finalmethod,
> and would use it to prevent the rest of your team from overriding it?
>

What about http://programmers.stackexchange.com/a/288974/118754?

-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/6e9fe64b/attachment-0001.html>

From mike at selik.org  Mon Nov 16 18:15:11 2015
From: mike at selik.org (Michael Selik)
Date: Mon, 16 Nov 2015 23:15:11 +0000
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CAO41-mOa8ijF6YX-4qJZ4oUwPaV4NVnR-Lv419RSHJmn34pFZA@mail.gmail.com>
References: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
 <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
 <AB6A1F80-1512-4090-971F-7728AA2546E0@yahoo.com>
 <CAO41-mOa8ijF6YX-4qJZ4oUwPaV4NVnR-Lv419RSHJmn34pFZA@mail.gmail.com>
Message-ID: <CADqi797ynrD3042GAvQ=LHrqzFyHYSbOhqABNQadvESHGohTkw@mail.gmail.com>

For that envelopes example on StackExchange, I'd probably use name mangling
to reduce chances of subclassing mistakes and have prepare() call stuff(),
seal(), stamp() in the correct order or whatever those were. I'm not sold
in the need for "final".

Btw, why does my iPad not have a back tick on the keyboard? My Android can
do it...

On Mon, Nov 16, 2015 at 5:53 PM Ryan Gonzalez <rymg19 at gmail.com> wrote:

> On Mon, Nov 16, 2015 at 3:59 PM, Andrew Barnert <abarnert at yahoo.com>
> wrote:
>
>> On Nov 16, 2015, at 10:24, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>
>> You could always put this in the docstring:
>>
>> DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE.
>>
>>
>>
>> Short, the-the-point, and evokes fear.
>>
>>
>> Sure, until the first time someone accidentally violates it and doesn't
>> die. Then you will have trained him to ignore your docstrings, comments,
>> type annotations, and warnings completely. (Of course you could take the
>> OP's code and add some stuff that causes the hard drive casing and spindle
>> to break, firing the platter at the developer's skull like one of those
>> discs from the Dolph Lundgren movie I Come in Peace, but that would break
>> compatibility with computers that only have SSDs, and I don't think that's
>> feasible in 2015.)
>>
>
> Well, it's just as easy to ignore type annotations and call `int(None)`.
>
> (You should definitely patent that hard drive stuff, though! ;)
>
>
>>
>> ...
>>
>> Seriously, though, I personally think this might go well with the whole
>> PEP 484 type-annotation thing. I mean, if there are already generics and
>> abstract methods. If we're going for static typing, adding final would fit
>> well. Something like:
>>
>> @finalmethod
>> def whatever(self): ...
>>
>> The type checker (in this case, Mypy) would check it, but there would be
>> no runtime effect.
>>
>>
>> Yes, that makes sense. If the feature isn't supposed to have any runtime
>> effect, and is modeled after something that's definitely
>> compile-time-centric in C++ and friends, signaling that by importing it
>> from typing seems to be a pretty clear way to get that across.
>>
>> Although I still don't know why we need it. See below:
>>
>> On November 16, 2015 12:15:55 PM CST, "???? ????????" <
>> spinenkoia at gmail.com> wrote:
>>>
>>> Thank you for the constructive responses, perhaps, I agree with you that
>>> this is not in the spirit of Python.
>>> As for the real example of the use, sometimes there to make the
>>> implementation of the method in the base class, but that none of the rest
>>> of the development team could not accidentally or unknowingly override this
>>> method (perhaps due to the need to be unsuccessful architecture).
>>>
>>
>> If it's just part of the implementation of the base class, just
>> double-underscore it so that it can't be accidentally overridden (or
>> called).
>>
>> It's only when something is also part of the interface that this is a
>> problem. And it's kind of strange to have something be part of the
>> interface of a type, but not be refinable by subtypes. When would you want
>> that? (And as for accidental overriding, if a class claims doesn't even
>> know what the interface of some supertype is, it probably shouldn't be
>> inheriting?either it should be a mixin for someone who does know the
>> supertype to compose, or it should be encapsulating rather than inheriting.)
>>
>> Do you have a concrete example of some code where you wanted finalmethod,
>> and would use it to prevent the rest of your team from overriding it?
>>
>
> What about http://programmers.stackexchange.com/a/288974/118754?
>
> --
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
> program. Something?s wrong.
> http://kirbyfan64.github.io/
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/28a88e48/attachment.html>

From mike at selik.org  Mon Nov 16 18:24:55 2015
From: mike at selik.org (Michael Selik)
Date: Mon, 16 Nov 2015 23:24:55 +0000
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CAO41-mNzVZiZYr0dMmLPjk4qhxAWk=CV=wYKyW47C1+DotX81Q@mail.gmail.com>
References: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
 <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
 <AB6A1F80-1512-4090-971F-7728AA2546E0@yahoo.com>
 <CAO41-mOa8ijF6YX-4qJZ4oUwPaV4NVnR-Lv419RSHJmn34pFZA@mail.gmail.com>
 <CADqi797ynrD3042GAvQ=LHrqzFyHYSbOhqABNQadvESHGohTkw@mail.gmail.com>
 <CAO41-mNzVZiZYr0dMmLPjk4qhxAWk=CV=wYKyW47C1+DotX81Q@mail.gmail.com>
Message-ID: <CADqi795TPb_EW542wo=SpsBB8ya-t=QYhQ_APYTUuDjXFkLLcg@mail.gmail.com>

Ryan, are you saying that

def helper(self):
pass

__helper = helper

Would not be sufficient protection against clumsy subclassing?

On Mon, Nov 16, 2015 at 6:22 PM Ryan Gonzalez <rymg19 at gmail.com> wrote:

> On Mon, Nov 16, 2015 at 5:15 PM, Michael Selik <mike at selik.org> wrote:
>
>> For that envelopes example on StackExchange, I'd probably use name
>> mangling to reduce chances of subclassing mistakes and have prepare() call
>> stuff(), seal(), stamp() in the correct order or whatever those were. I'm
>> not sold in the need for "final".
>>
>>
> Maybe the subclasses should be able to access the functions. They just
> shouldn't override it.
>
>
>> Btw, why does my iPad not have a back tick on the keyboard? My Android
>> can do it...
>>
>>
> Android > iPad.
>
> *ducks from incoming bullets*
>
>
>> On Mon, Nov 16, 2015 at 5:53 PM Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>
>>> On Mon, Nov 16, 2015 at 3:59 PM, Andrew Barnert <abarnert at yahoo.com>
>>> wrote:
>>>
>>>> On Nov 16, 2015, at 10:24, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>>>
>>>> You could always put this in the docstring:
>>>>
>>>> DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE.
>>>>
>>>>
>>>>
>>>> Short, the-the-point, and evokes fear.
>>>>
>>>>
>>>> Sure, until the first time someone accidentally violates it and doesn't
>>>> die. Then you will have trained him to ignore your docstrings, comments,
>>>> type annotations, and warnings completely. (Of course you could take
>>>> the OP's code and add some stuff that causes the hard drive casing and
>>>> spindle to break, firing the platter at the developer's skull like one of
>>>> those discs from the Dolph Lundgren movie I Come in Peace, but that would
>>>> break compatibility with computers that only have SSDs, and I don't think
>>>> that's feasible in 2015.)
>>>>
>>>
>>> Well, it's just as easy to ignore type annotations and call `int(None)`.
>>>
>>> (You should definitely patent that hard drive stuff, though! ;)
>>>
>>>
>>>>
>>>> ...
>>>>
>>>> Seriously, though, I personally think this might go well with the whole
>>>> PEP 484 type-annotation thing. I mean, if there are already generics and
>>>> abstract methods. If we're going for static typing, adding final would fit
>>>> well. Something like:
>>>>
>>>> @finalmethod
>>>> def whatever(self): ...
>>>>
>>>> The type checker (in this case, Mypy) would check it, but there would
>>>> be no runtime effect.
>>>>
>>>>
>>>> Yes, that makes sense. If the feature isn't supposed to have any
>>>> runtime effect, and is modeled after something that's definitely
>>>> compile-time-centric in C++ and friends, signaling that by importing it
>>>> from typing seems to be a pretty clear way to get that across.
>>>>
>>>> Although I still don't know why we need it. See below:
>>>>
>>>> On November 16, 2015 12:15:55 PM CST, "???? ????????" <
>>>> spinenkoia at gmail.com> wrote:
>>>>>
>>>>> Thank you for the constructive responses, perhaps, I agree with you
>>>>> that this is not in the spirit of Python.
>>>>> As for the real example of the use, sometimes there to make the
>>>>> implementation of the method in the base class, but that none of the rest
>>>>> of the development team could not accidentally or unknowingly override this
>>>>> method (perhaps due to the need to be unsuccessful architecture).
>>>>>
>>>>
>>>> If it's just part of the implementation of the base class, just
>>>> double-underscore it so that it can't be accidentally overridden (or
>>>> called).
>>>>
>>>> It's only when something is also part of the interface that this is a
>>>> problem. And it's kind of strange to have something be part of the
>>>> interface of a type, but not be refinable by subtypes. When would you want
>>>> that? (And as for accidental overriding, if a class claims doesn't even
>>>> know what the interface of some supertype is, it probably shouldn't be
>>>> inheriting?either it should be a mixin for someone who does know the
>>>> supertype to compose, or it should be encapsulating rather than inheriting.)
>>>>
>>>> Do you have a concrete example of some code where you wanted
>>>> finalmethod, and would use it to prevent the rest of your team from
>>>> overriding it?
>>>>
>>>
>>> What about http://programmers.stackexchange.com/a/288974/118754?
>>>
>>> --
>>> Ryan
>>> [ERROR]: Your autotools build scripts are 200 lines longer than your
>>> program. Something?s wrong.
>>> http://kirbyfan64.github.io/
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>
>
> --
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
> program. Something?s wrong.
> http://kirbyfan64.github.io/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/aa874815/attachment-0001.html>

From random832 at fastmail.com  Mon Nov 16 18:39:27 2015
From: random832 at fastmail.com (Random832)
Date: Mon, 16 Nov 2015 18:39:27 -0500
Subject: [Python-ideas] The prohibition overrides the class method
References: <CAM+FtgWTKvYhcxGeEeXnYcCF+G7dYgD19SJWMEjmbiaXmxsZJQ@mail.gmail.com>
Message-ID: <m2twolo6og.fsf@fastmail.com>

I'm not sure how this thread has gone on this long without anyone
mentioning __methods.

If you prefix a method with two underscores, it will transform the name
of the method in order to make it difficult to accidentally override.

So in other methods inside the base class that need to call your method,
you can call the underscored version and subclasses won't override it (I
think it's possible for them to 

e.g.

class A:
    def __test(self):
        print("123")
    def test(self):
        return self.__internal_test()
    def other_function(self):
        ...
        self.__test()
        ...


class B(A):
    def __test(self):
        print("456")


>>> B().test()
123


B can still override test if it wants to, but it won't affect any calls
to the __test method from A's other methods. If it *really* wants to, it
can override _A__test(), but someone's not likely to do that by mistake.


From ian.g.kelly at gmail.com  Mon Nov 16 18:40:26 2015
From: ian.g.kelly at gmail.com (Ian Kelly)
Date: Mon, 16 Nov 2015 16:40:26 -0700
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CADqi797ynrD3042GAvQ=LHrqzFyHYSbOhqABNQadvESHGohTkw@mail.gmail.com>
References: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
 <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
 <AB6A1F80-1512-4090-971F-7728AA2546E0@yahoo.com>
 <CAO41-mOa8ijF6YX-4qJZ4oUwPaV4NVnR-Lv419RSHJmn34pFZA@mail.gmail.com>
 <CADqi797ynrD3042GAvQ=LHrqzFyHYSbOhqABNQadvESHGohTkw@mail.gmail.com>
Message-ID: <CALwzidmis_ZPvCr=y4TULoC_pqnq-hPH8fSaojzOR6pSrCLb9w@mail.gmail.com>

On Nov 16, 2015 4:15 PM, "Michael Selik" <mike at selik.org> wrote:
>
> For that envelopes example on StackExchange, I'd probably use name
mangling to reduce chances of subclassing mistakes and have prepare() call
stuff(), seal(), stamp() in the correct order or whatever those were. I'm
not sold in the need for "final".

Name mangling would not work in that example. The prepare method is defined
by a base class and overridden in a subclass; it is the subclass that
wishes to declare the method final. Name mangling would obstruct overriding
the method at all, which would be a problem for the base class that invites
the method to be overridden.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/9cc915fb/attachment.html>

From greg.ewing at canterbury.ac.nz  Mon Nov 16 17:09:59 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 17 Nov 2015 11:09:59 +1300
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
References: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
 <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
Message-ID: <564A5437.7000704@canterbury.ac.nz>

Ryan Gonzalez wrote:
> You could always put this in the docstring:
> 
> DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE.

And don't turn on the headlight, either. :-)

-- 
Greg

From mike at selik.org  Mon Nov 16 19:12:58 2015
From: mike at selik.org (Michael Selik)
Date: Tue, 17 Nov 2015 00:12:58 +0000
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CALwzidmis_ZPvCr=y4TULoC_pqnq-hPH8fSaojzOR6pSrCLb9w@mail.gmail.com>
References: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
 <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
 <AB6A1F80-1512-4090-971F-7728AA2546E0@yahoo.com>
 <CAO41-mOa8ijF6YX-4qJZ4oUwPaV4NVnR-Lv419RSHJmn34pFZA@mail.gmail.com>
 <CADqi797ynrD3042GAvQ=LHrqzFyHYSbOhqABNQadvESHGohTkw@mail.gmail.com>
 <CALwzidmis_ZPvCr=y4TULoC_pqnq-hPH8fSaojzOR6pSrCLb9w@mail.gmail.com>
Message-ID: <CADqi794K3UrZQAOyb-MFg0c9yp79WV4=8n+afOFxdvCLPjH=Yw@mail.gmail.com>

Thanks, Ian, I missed that detail.

While my initial thought was that name mangling would achieve the desired
level of subclassing-clumsiness defense. As below...

    # using 2 steps instead of 3, for brevity

    class Envelope:
        def prepare(self):
            print('omnibus!')

    class Child(Envelope):

        def stuff(self):
            print('step 1')
        __stuff = stuff

        def seal(self):
            print('step 2')
        __seal = seal

        def prepare(self):
            self.__stuff()
            self.__seal()


The post on StackExchange added the extra wrinkle that both classes have
C++ virtual functions, kind-of like this...

    from abc import ABC, abstractmethod

    class Envelope(ABC):

        def something_useful(self):
            ''' I'd normally avoid tricky inheritance
            but there must be some really useful code
            in here that I don't want to refactor
            '''

        @abstractmethod
        def prepare(self):
            pass


    class Child(Envelope):

        @abstractmethod
        def stuff(self):
            print('step 1')

        @abstractmethod
        def seal(self):
            print('step 2')

        def prepare(self):
            'should be FINAL, Override at your OWN PERIL'
            self.stuff()
            self.seal()

This contrived scenario seems better resolved by refactoring the `Envelope`
class. Moreover, I don't understand why the `Child` class would exist at
all. Why override `prepare` only to create more abstract methods?

On Mon, Nov 16, 2015 at 6:40 PM Ian Kelly <ian.g.kelly at gmail.com> wrote:

> On Nov 16, 2015 4:15 PM, "Michael Selik" <mike at selik.org> wrote:
> >
> > For that envelopes example on StackExchange, I'd probably use name
> mangling to reduce chances of subclassing mistakes and have prepare() call
> stuff(), seal(), stamp() in the correct order or whatever those were. I'm
> not sold in the need for "final".
>
> Name mangling would not work in that example. The prepare method is
> defined by a base class and overridden in a subclass; it is the subclass
> that wishes to declare the method final. Name mangling would obstruct
> overriding the method at all, which would be a problem for the base class
> that invites the method to be overridden.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151117/36fffbb2/attachment.html>

From chris.barker at noaa.gov  Mon Nov 16 20:27:03 2015
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Mon, 16 Nov 2015 17:27:03 -0800
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <CAP7+vJLpD1ZaSc_XvumMiZdwtxz5_e+S=wSx6iTRGJJ1E3jHCw@mail.gmail.com>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <CEBE6791-2136-4F3C-8EBA-B476E2A9DF38@yahoo.com>
 <CAP7+vJKhDGgUMMLdBj1_v9U5x_UEWgZyyNW=Hq0npWH+_ka3Xw@mail.gmail.com>
 <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com>
 <CAP7+vJLpD1ZaSc_XvumMiZdwtxz5_e+S=wSx6iTRGJJ1E3jHCw@mail.gmail.com>
Message-ID: <3007695256009891190@unknownmsgid>

>> As a side note, most of the C++ family have something akin to "typedef" and/or "using" type aliases that are explicitly _not_ treated as different types by the static typing algorithm:

...
> Yes, that's what PEP 484 type aliases do too.

Darn -- C typedefs have their uses, but improving type safety is not
one of them.

PEP 484 is all about type checking/safety -- it seems it would be a
lot more useful if aliases were treated as different types.

Oh well.

CHB

From matt at whoosh.ca  Mon Nov 16 20:35:04 2015
From: matt at whoosh.ca (Matt Chaput)
Date: Mon, 16 Nov 2015 20:35:04 -0500
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <CAP7+vJLpD1ZaSc_XvumMiZdwtxz5_e+S=wSx6iTRGJJ1E3jHCw@mail.gmail.com>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <CEBE6791-2136-4F3C-8EBA-B476E2A9DF38@yahoo.com>
 <CAP7+vJKhDGgUMMLdBj1_v9U5x_UEWgZyyNW=Hq0npWH+_ka3Xw@mail.gmail.com>
 <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com>
 <CAP7+vJLpD1ZaSc_XvumMiZdwtxz5_e+S=wSx6iTRGJJ1E3jHCw@mail.gmail.com>
Message-ID: <BC8DED79-A050-4D6D-A046-88C1B73BB167@whoosh.ca>

> Yeah, but I'd still prefer a solution that is only read by the type
> checker. And defining subclasses of int is pretty uncommon, so it'll
> confuse the heck out of a lot of readers -- much more so than other
> parts of type annotations (which are ignorable).

I think a solution that only affected the type system would be much better, but it's cool that subclassing int exists as a workaround (didn't know that would work :). I'll have to decide if it's worth the trouble for me.

Thanks all of you for learning me a thing!

Matt


From guido at python.org  Mon Nov 16 20:35:42 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Nov 2015 17:35:42 -0800
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <3007695256009891190@unknownmsgid>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <CEBE6791-2136-4F3C-8EBA-B476E2A9DF38@yahoo.com>
 <CAP7+vJKhDGgUMMLdBj1_v9U5x_UEWgZyyNW=Hq0npWH+_ka3Xw@mail.gmail.com>
 <670B43D6-2BB2-4CBB-AE74-26E4C57B7782@yahoo.com>
 <CAP7+vJLpD1ZaSc_XvumMiZdwtxz5_e+S=wSx6iTRGJJ1E3jHCw@mail.gmail.com>
 <3007695256009891190@unknownmsgid>
Message-ID: <CAP7+vJJu1nFRuHCfvbqyn-55Dn=SVBG_4aQ_kkKfmtYz3Z1oeQ@mail.gmail.com>

No, PEP 484 is about a pragmatic compromise that doesn't require
adding new syntax and yet allows a reasonable amount of checking. Type
checking religion has no place here.

On Mon, Nov 16, 2015 at 5:27 PM, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
>>> As a side note, most of the C++ family have something akin to "typedef" and/or "using" type aliases that are explicitly _not_ treated as different types by the static typing algorithm:
>
> ...
>> Yes, that's what PEP 484 type aliases do too.
>
> Darn -- C typedefs have their uses, but improving type safety is not
> one of them.
>
> PEP 484 is all about type checking/safety -- it seems it would be a
> lot more useful if aliases were treated as different types.
>
> Oh well.
>
> CHB



-- 
--Guido van Rossum (python.org/~guido)

From rymg19 at gmail.com  Mon Nov 16 18:21:57 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 16 Nov 2015 17:21:57 -0600
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CADqi797ynrD3042GAvQ=LHrqzFyHYSbOhqABNQadvESHGohTkw@mail.gmail.com>
References: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
 <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
 <AB6A1F80-1512-4090-971F-7728AA2546E0@yahoo.com>
 <CAO41-mOa8ijF6YX-4qJZ4oUwPaV4NVnR-Lv419RSHJmn34pFZA@mail.gmail.com>
 <CADqi797ynrD3042GAvQ=LHrqzFyHYSbOhqABNQadvESHGohTkw@mail.gmail.com>
Message-ID: <CAO41-mNzVZiZYr0dMmLPjk4qhxAWk=CV=wYKyW47C1+DotX81Q@mail.gmail.com>

On Mon, Nov 16, 2015 at 5:15 PM, Michael Selik <mike at selik.org> wrote:

> For that envelopes example on StackExchange, I'd probably use name
> mangling to reduce chances of subclassing mistakes and have prepare() call
> stuff(), seal(), stamp() in the correct order or whatever those were. I'm
> not sold in the need for "final".
>
>
Maybe the subclasses should be able to access the functions. They just
shouldn't override it.


> Btw, why does my iPad not have a back tick on the keyboard? My Android can
> do it...
>
>
Android > iPad.

*ducks from incoming bullets*


> On Mon, Nov 16, 2015 at 5:53 PM Ryan Gonzalez <rymg19 at gmail.com> wrote:
>
>> On Mon, Nov 16, 2015 at 3:59 PM, Andrew Barnert <abarnert at yahoo.com>
>> wrote:
>>
>>> On Nov 16, 2015, at 10:24, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>>
>>> You could always put this in the docstring:
>>>
>>> DO NOT OVERRIDE THIS METHOD. IF YOU OVERRIDE THIS METHOD, YOU WILL DIE.
>>>
>>>
>>>
>>> Short, the-the-point, and evokes fear.
>>>
>>>
>>> Sure, until the first time someone accidentally violates it and doesn't
>>> die. Then you will have trained him to ignore your docstrings, comments,
>>> type annotations, and warnings completely. (Of course you could take
>>> the OP's code and add some stuff that causes the hard drive casing and
>>> spindle to break, firing the platter at the developer's skull like one of
>>> those discs from the Dolph Lundgren movie I Come in Peace, but that would
>>> break compatibility with computers that only have SSDs, and I don't think
>>> that's feasible in 2015.)
>>>
>>
>> Well, it's just as easy to ignore type annotations and call `int(None)`.
>>
>> (You should definitely patent that hard drive stuff, though! ;)
>>
>>
>>>
>>> ...
>>>
>>> Seriously, though, I personally think this might go well with the whole
>>> PEP 484 type-annotation thing. I mean, if there are already generics and
>>> abstract methods. If we're going for static typing, adding final would fit
>>> well. Something like:
>>>
>>> @finalmethod
>>> def whatever(self): ...
>>>
>>> The type checker (in this case, Mypy) would check it, but there would be
>>> no runtime effect.
>>>
>>>
>>> Yes, that makes sense. If the feature isn't supposed to have any runtime
>>> effect, and is modeled after something that's definitely
>>> compile-time-centric in C++ and friends, signaling that by importing it
>>> from typing seems to be a pretty clear way to get that across.
>>>
>>> Although I still don't know why we need it. See below:
>>>
>>> On November 16, 2015 12:15:55 PM CST, "???? ????????" <
>>> spinenkoia at gmail.com> wrote:
>>>>
>>>> Thank you for the constructive responses, perhaps, I agree with you
>>>> that this is not in the spirit of Python.
>>>> As for the real example of the use, sometimes there to make the
>>>> implementation of the method in the base class, but that none of the rest
>>>> of the development team could not accidentally or unknowingly override this
>>>> method (perhaps due to the need to be unsuccessful architecture).
>>>>
>>>
>>> If it's just part of the implementation of the base class, just
>>> double-underscore it so that it can't be accidentally overridden (or
>>> called).
>>>
>>> It's only when something is also part of the interface that this is a
>>> problem. And it's kind of strange to have something be part of the
>>> interface of a type, but not be refinable by subtypes. When would you want
>>> that? (And as for accidental overriding, if a class claims doesn't even
>>> know what the interface of some supertype is, it probably shouldn't be
>>> inheriting?either it should be a mixin for someone who does know the
>>> supertype to compose, or it should be encapsulating rather than inheriting.)
>>>
>>> Do you have a concrete example of some code where you wanted
>>> finalmethod, and would use it to prevent the rest of your team from
>>> overriding it?
>>>
>>
>> What about http://programmers.stackexchange.com/a/288974/118754?
>>
>> --
>> Ryan
>> [ERROR]: Your autotools build scripts are 200 lines longer than your
>> program. Something?s wrong.
>> http://kirbyfan64.github.io/
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/b4a34b34/attachment.html>

From abarnert at yahoo.com  Mon Nov 16 21:21:28 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 16 Nov 2015 18:21:28 -0800
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CADqi794K3UrZQAOyb-MFg0c9yp79WV4=8n+afOFxdvCLPjH=Yw@mail.gmail.com>
References: <CAM+FtgUnTsHnthfwbYrwi4NYwuZn-i-hcD9ZW7b2Ng15zLMynQ@mail.gmail.com>
 <0F32DD15-7F44-456A-B485-CD285CE67DAD@gmail.com>
 <AB6A1F80-1512-4090-971F-7728AA2546E0@yahoo.com>
 <CAO41-mOa8ijF6YX-4qJZ4oUwPaV4NVnR-Lv419RSHJmn34pFZA@mail.gmail.com>
 <CADqi797ynrD3042GAvQ=LHrqzFyHYSbOhqABNQadvESHGohTkw@mail.gmail.com>
 <CALwzidmis_ZPvCr=y4TULoC_pqnq-hPH8fSaojzOR6pSrCLb9w@mail.gmail.com>
 <CADqi794K3UrZQAOyb-MFg0c9yp79WV4=8n+afOFxdvCLPjH=Yw@mail.gmail.com>
Message-ID: <422CD3EE-03A0-4EDF-B2FF-D71A0DF965A3@yahoo.com>

On Nov 16, 2015, at 16:12, Michael Selik <mike at selik.org> wrote:
> 
> Thanks, Ian, I missed that detail.
> 
> While my initial thought was that name mangling would achieve the desired level of subclassing-clumsiness defense. As below...
> 
>     # using 2 steps instead of 3, for brevity
> 
>     class Envelope:
>         def prepare(self):
>             print('omnibus!')
> 
>     class Child(Envelope):
> 
>         def stuff(self):
>             print('step 1')
>         __stuff = stuff
> 
>         def seal(self):
>             print('step 2')
>         __seal = seal
> 
>         def prepare(self):
>             self.__stuff()
>             self.__seal()
> 
> 
> The post on StackExchange added the extra wrinkle that both classes have C++ virtual functions, kind-of like this...
> 
>     from abc import ABC, abstractmethod
> 
>     class Envelope(ABC):
> 
>         def something_useful(self):
>             ''' I'd normally avoid tricky inheritance
>             but there must be some really useful code
>             in here that I don't want to refactor
>             '''
> 
>         @abstractmethod
>         def prepare(self):
>             pass
> 
> 
>     class Child(Envelope):
> 
>         @abstractmethod
>         def stuff(self):
>             print('step 1')
> 
>         @abstractmethod
>         def seal(self):
>             print('step 2')
> 
>         def prepare(self):
>             'should be FINAL, Override at your OWN PERIL'
>             self.stuff()
>             self.seal()
> 
> This contrived scenario seems better resolved by refactoring the `Envelope` class. Moreover, I don't understand why the `Child` class would exist at all. Why override `prepare` only to create more abstract methods?

Apparently, because they want to use the strategy pattern, twice, but in a way that gets none of the advantages of that pattern.

To actually get the benefits, those new abstract methods would not be in Child; instead, you'd create a new EnvelopePrepaper interface that Child uses. Why can't you just merge the EnvelopePreparer into Child itself? Because then you lose all the decoupling benefits that the strategy pattern is meant to provide, and you add new problems in that a single class now has multiple separate interfaces that can accidentally step on each other's toes. In fact, what they clearly wanted was for Envelope to use a preparer strategy, and for Child to _be_ such a preparer strategy that itself uses a 2-stage-preparer strategy, and for other people to implement 2-stage-preparer strategies. But then they've merged both interfaces into the class that uses those strategies instead for no good reason. If you fix either of those problems, or just use a simpler design instead of misusing strategy, or even if you only misuse strategy once instead of multiple times, the problem wouldn't arise.

In other words, it's a subset of exactly the problem I suggested before seeing the example: If you want to inherit from a class, you have to know that class's interface, because you're promising to implement it. This design forces users to inherit from a class whose interface they don't know, and then hopes for a new language feature to hide one of the symptoms of that problem instead of fixing the problem.

>> On Mon, Nov 16, 2015 at 6:40 PM Ian Kelly <ian.g.kelly at gmail.com> wrote:
>> On Nov 16, 2015 4:15 PM, "Michael Selik" <mike at selik.org> wrote:
>> >
>> > For that envelopes example on StackExchange, I'd probably use name mangling to reduce chances of subclassing mistakes and have prepare() call stuff(), seal(), stamp() in the correct order or whatever those were. I'm not sold in the need for "final".
>> 
>> Name mangling would not work in that example. The prepare method is defined by a base class and overridden in a subclass; it is the subclass that wishes to declare the method final. Name mangling would obstruct overriding the method at all, which would be a problem for the base class that invites the method to be overridden.
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151116/38e1dc72/attachment-0001.html>

From matt at whoosh.ca  Mon Nov 16 21:41:45 2015
From: matt at whoosh.ca (Matt Chaput)
Date: Mon, 16 Nov 2015 21:41:45 -0500
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
Message-ID: <FE176A55-9AB9-4298-871C-4BBC0F5789F6@whoosh.ca>

I forgot to say, there's been a couple of places where I haven't been able to add hinting that hopefully can be improved:

1. Standard library types that should be public but aren't (regular expression object)
2. Specifying an argument/return that's a class (that should be a subclass of some class).

There might be a way to do 2 that I don't know about :)

Cheers,

Matt


From guido at python.org  Mon Nov 16 21:51:38 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Nov 2015 18:51:38 -0800
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <FE176A55-9AB9-4298-871C-4BBC0F5789F6@whoosh.ca>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <FE176A55-9AB9-4298-871C-4BBC0F5789F6@whoosh.ca>
Message-ID: <CAP7+vJ+hie47aWSrfZBqa0MHFebCYQe-pW7H2t2bNE_y3GtUpg@mail.gmail.com>

#1: typing.re defines the re types. For the others, the typeshed
project takes contributions.

#2: You can use def f() -> type: ... to specify a class as return
type; but we currently don't have a way to contrain that class.

On Mon, Nov 16, 2015 at 6:41 PM, Matt Chaput <matt at whoosh.ca> wrote:
> I forgot to say, there's been a couple of places where I haven't been able to add hinting that hopefully can be improved:
>
> 1. Standard library types that should be public but aren't (regular expression object)
> 2. Specifying an argument/return that's a class (that should be a subclass of some class).
>
> There might be a way to do 2 that I don't know about :)
>
> Cheers,
>
> Matt
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
--Guido van Rossum (python.org/~guido)

From rymg19 at gmail.com  Mon Nov 16 22:36:22 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 16 Nov 2015 21:36:22 -0600
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <CAP7+vJ+hie47aWSrfZBqa0MHFebCYQe-pW7H2t2bNE_y3GtUpg@mail.gmail.com>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <FE176A55-9AB9-4298-871C-4BBC0F5789F6@whoosh.ca>
 <CAP7+vJ+hie47aWSrfZBqa0MHFebCYQe-pW7H2t2bNE_y3GtUpg@mail.gmail.com>
Message-ID: <DEF4567C-1370-4529-BFA0-AFF45A69B865@gmail.com>



On November 16, 2015 8:51:38 PM CST, Guido van Rossum <guido at python.org> wrote:
>#1: typing.re defines the re types. For the others, the typeshed
>project takes contributions.
>
>#2: You can use def f() -> type: ... to specify a class as return
>type; but we currently don't have a way to contrain that class.
>

What about a typing.Class or typing.Type generic? Like:

def f() -> Type[MyBaseClass]: ...

>On Mon, Nov 16, 2015 at 6:41 PM, Matt Chaput <matt at whoosh.ca> wrote:
>> I forgot to say, there's been a couple of places where I haven't been
>able to add hinting that hopefully can be improved:
>>
>> 1. Standard library types that should be public but aren't (regular
>expression object)
>> 2. Specifying an argument/return that's a class (that should be a
>subclass of some class).
>>
>> There might be a way to do 2 that I don't know about :)
>>
>> Cheers,
>>
>> Matt
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/

-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.

From guido at python.org  Mon Nov 16 22:39:33 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 16 Nov 2015 19:39:33 -0800
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <DEF4567C-1370-4529-BFA0-AFF45A69B865@gmail.com>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <FE176A55-9AB9-4298-871C-4BBC0F5789F6@whoosh.ca>
 <CAP7+vJ+hie47aWSrfZBqa0MHFebCYQe-pW7H2t2bNE_y3GtUpg@mail.gmail.com>
 <DEF4567C-1370-4529-BFA0-AFF45A69B865@gmail.com>
Message-ID: <CAP7+vJKRdH3Mqt0si+-J=wd82tmrwRRrgScKt=kJvGqNvdSDWQ@mail.gmail.com>

There's an existing (but postponed) proposal to use Type[X]:
https://github.com/ambv/typehinting/issues/107

On Mon, Nov 16, 2015 at 7:36 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>
>
> On November 16, 2015 8:51:38 PM CST, Guido van Rossum <guido at python.org> wrote:
>>#1: typing.re defines the re types. For the others, the typeshed
>>project takes contributions.
>>
>>#2: You can use def f() -> type: ... to specify a class as return
>>type; but we currently don't have a way to contrain that class.
>>
>
> What about a typing.Class or typing.Type generic? Like:
>
> def f() -> Type[MyBaseClass]: ...
>
>>On Mon, Nov 16, 2015 at 6:41 PM, Matt Chaput <matt at whoosh.ca> wrote:
>>> I forgot to say, there's been a couple of places where I haven't been
>>able to add hinting that hopefully can be improved:
>>>
>>> 1. Standard library types that should be public but aren't (regular
>>expression object)
>>> 2. Specifying an argument/return that's a class (that should be a
>>subclass of some class).
>>>
>>> There might be a way to do 2 that I don't know about :)
>>>
>>> Cheers,
>>>
>>> Matt
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
> Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.



-- 
--Guido van Rossum (python.org/~guido)

From spinenkoia at gmail.com  Tue Nov 17 00:30:01 2015
From: spinenkoia at gmail.com (=?UTF-8?B?0JjQstCw0L0g0KHQv9C40L3QtdC90LrQvg==?=)
Date: Tue, 17 Nov 2015 10:30:01 +0500
Subject: [Python-ideas] The prohibition overrides the class method
Message-ID: <CAM+FtgXQ2cE75+oaF=a6zw1UWZWga6+72eyUesbpTfOQ31NE-Q@mail.gmail.com>

Thank you all very much. Perhaps I'll leave that idea, Python too flexible
language for such things.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151117/9abe2881/attachment.html>

From victor.stinner at gmail.com  Tue Nov 17 06:17:42 2015
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 17 Nov 2015 12:17:42 +0100
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CAM+FtgWTKvYhcxGeEeXnYcCF+G7dYgD19SJWMEjmbiaXmxsZJQ@mail.gmail.com>
References: <CAM+FtgWTKvYhcxGeEeXnYcCF+G7dYgD19SJWMEjmbiaXmxsZJQ@mail.gmail.com>
Message-ID: <CAMpsgwZe12Oxth10UxHqDjKKDY70xgfXvj9S8rLe5shUTJdS8Q@mail.gmail.com>

2015-11-16 16:03 GMT+01:00 ???? ???????? <spinenkoia at gmail.com>:
> I propose to make an analogue of the final specifier in C ++. In class
> abc.ABCMeta has decorator abstractmethod, you can add decorator finalmethod.

Would it be allowed to override the method in the class where the
method is defined using unittest.mock? Mocking looks popular and
importang in the Python community to write unit tests.

Victor

From matt at whoosh.ca  Tue Nov 17 11:25:18 2015
From: matt at whoosh.ca (Matt Chaput)
Date: Tue, 17 Nov 2015 11:25:18 -0500
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <CAP7+vJ+hie47aWSrfZBqa0MHFebCYQe-pW7H2t2bNE_y3GtUpg@mail.gmail.com>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <FE176A55-9AB9-4298-871C-4BBC0F5789F6@whoosh.ca>
 <CAP7+vJ+hie47aWSrfZBqa0MHFebCYQe-pW7H2t2bNE_y3GtUpg@mail.gmail.com>
Message-ID: <BCF710FC-68B9-4A63-B4CE-CD5FDE9976EE@whoosh.ca>

> #1: typing.re defines the re types. For the others, the typeshed
> project takes contributions.

Me: Really? I don't remember seeing...
Me: (Searches pep484 document)
Me: (Embarrassed)
Me: (Crawls in hole)

Thanks! :)

Matt



From rymg19 at gmail.com  Tue Nov 17 12:40:09 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 17 Nov 2015 11:40:09 -0600
Subject: [Python-ideas] Lessons from typing hinting Whoosh (PEP484)
In-Reply-To: <BCF710FC-68B9-4A63-B4CE-CD5FDE9976EE@whoosh.ca>
References: <EF5A7165-AB18-43DD-83C6-7D1C777DB297@whoosh.ca>
 <FE176A55-9AB9-4298-871C-4BBC0F5789F6@whoosh.ca>
 <CAP7+vJ+hie47aWSrfZBqa0MHFebCYQe-pW7H2t2bNE_y3GtUpg@mail.gmail.com>
 <BCF710FC-68B9-4A63-B4CE-CD5FDE9976EE@whoosh.ca>
Message-ID: <111A0DAD-8B32-494E-BD65-5507806E867F@gmail.com>



On November 17, 2015 10:25:18 AM CST, Matt Chaput <matt at whoosh.ca> wrote:
>> #1: typing.re defines the re types. For the others, the typeshed
>> project takes contributions.
>
>Me: Really? I don't remember seeing...
>Me: (Searches pep484 document)
>Me: (Embarrassed)
>Me: (Crawls in hole)
>
>Thanks! :)
>

Don't worry; I do stuff like that *all the time*. Literally.

>Matt
>
>
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>https://mail.python.org/mailman/listinfo/python-ideas
>Code of Conduct: http://python.org/psf/codeofconduct/

-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.

From mike at selik.org  Tue Nov 17 17:48:37 2015
From: mike at selik.org (Michael Selik)
Date: Tue, 17 Nov 2015 22:48:37 +0000
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CAM+FtgXQ2cE75+oaF=a6zw1UWZWga6+72eyUesbpTfOQ31NE-Q@mail.gmail.com>
References: <CAM+FtgXQ2cE75+oaF=a6zw1UWZWga6+72eyUesbpTfOQ31NE-Q@mail.gmail.com>
Message-ID: <CADqi796XaDOSWomPk9x1u7zO+47fZf9r=GbQPdo0=5u=XWnZNw@mail.gmail.com>

Cython has a `final` for preventing a class from being overridden (
http://docs.cython.org/src/userguide/extension_types.html#subclassing). I
don't know if Cython supports `final` methods outside of final classes.
They might be more interested in implementing that feature.


On Tue, Nov 17, 2015, 12:30 AM ???? ???????? <spinenkoia at gmail.com> wrote:

> Thank you all very much. Perhaps I'll leave that idea, Python too flexible
> language for such things.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151117/801c6998/attachment.html>

From ncoghlan at gmail.com  Tue Nov 17 21:41:29 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 18 Nov 2015 12:41:29 +1000
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CADqi796XaDOSWomPk9x1u7zO+47fZf9r=GbQPdo0=5u=XWnZNw@mail.gmail.com>
References: <CAM+FtgXQ2cE75+oaF=a6zw1UWZWga6+72eyUesbpTfOQ31NE-Q@mail.gmail.com>
 <CADqi796XaDOSWomPk9x1u7zO+47fZf9r=GbQPdo0=5u=XWnZNw@mail.gmail.com>
Message-ID: <CADiSq7fn8zuyfzdwmnMUt+T6mLX52-ENV-Sd3M4sG90n=NXRjA@mail.gmail.com>

On 18 November 2015 at 08:48, Michael Selik <mike at selik.org> wrote:
> Cython has a `final` for preventing a class from being overridden
> (http://docs.cython.org/src/userguide/extension_types.html#subclassing). I
> don't know if Cython supports `final` methods outside of final classes. They
> might be more interested in implementing that feature.

+1, as a lot of Cython's power comes from allowing developers to
deliberately (and selectively) *opt out* of Python's dynamic features.
They even provide an annotation mode that shows how far you've
progressed in rendering a section of code entirely static:
http://docs.cython.org/src/quickstart/cythonize.html#determining-where-to-add-types

Regards,
Nick.

P.S. For folks interested in learning more about that, I'd recommend
Caleb Hattingh's PyCon Australia talk about using Cython to speed up
and parallelise code hotspots:
http://docs.cython.org/src/quickstart/cythonize.html#determining-where-to-add-types

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Nov 17 22:18:17 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 18 Nov 2015 13:18:17 +1000
Subject: [Python-ideas] The prohibition overrides the class method
In-Reply-To: <CADiSq7fn8zuyfzdwmnMUt+T6mLX52-ENV-Sd3M4sG90n=NXRjA@mail.gmail.com>
References: <CAM+FtgXQ2cE75+oaF=a6zw1UWZWga6+72eyUesbpTfOQ31NE-Q@mail.gmail.com>
 <CADqi796XaDOSWomPk9x1u7zO+47fZf9r=GbQPdo0=5u=XWnZNw@mail.gmail.com>
 <CADiSq7fn8zuyfzdwmnMUt+T6mLX52-ENV-Sd3M4sG90n=NXRjA@mail.gmail.com>
Message-ID: <CADiSq7dh1am7r=1xmWjm4izw6pAKyFCZREgyjzGs-hNQ-KbN+Q@mail.gmail.com>

On 18 November 2015 at 12:41, Nick Coghlan <ncoghlan at gmail.com> wrote:
> P.S. For folks interested in learning more about that, I'd recommend
> Caleb Hattingh's PyCon Australia talk about using Cython to speed up
> and parallelise code hotspots:
> http://docs.cython.org/src/quickstart/cythonize.html#determining-where-to-add-types

Oops, the actual video link: https://www.youtube.com/watch?v=NfnMJMkhDoQ

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From me at jeltef.nl  Mon Nov 23 14:09:17 2015
From: me at jeltef.nl (Jelte Fennema)
Date: Mon, 23 Nov 2015 20:09:17 +0100
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
Message-ID: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>

Hi,

After reading PEP0465 <https://www.python.org/dev/peps/pep-0465/> about the
dedicated matrix multiplication I started wondering if the same solution
couldn't be applied to boolean operators as well. There currently are a lot
of high profile libraries that have their own functions for boolean
operators, like Numpy, Pandas or SQLAlchemy. They do this because the
current boolean operators can't be overloaded. PEP0335
<https://www.python.org/dev/peps/pep-0335/> was created to solve this
problem (and makes the problem more clear), but was rejected because it
needed changes to the byte code for the boolean operators, which would make
them slower. Currently some of these libraries resort to the bitwise
operators (at least Pandas), but those don't bind as strong as comparison
operators, which means you have to do comparisons like this: (series1 == 2)
& (series2 == 3)

That is why I propose to create new operators just like for matrix
multiplication which can be used in libraries that need one. I'm not sure
what the operators should look like, but my first guess would be &&, || and
! for and, or and not respectively.

Is this an idea that sounds reasonable?

Jelte
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151123/8b802217/attachment.html>

From vgr255 at live.ca  Mon Nov 23 15:22:53 2015
From: vgr255 at live.ca (Emanuel Barry)
Date: Mon, 23 Nov 2015 15:22:53 -0500
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
Message-ID: <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>

Could you provide some links to where this could be useful, and how code could be rewritten? I can see the desire for such a feature, I myself would have liked such an operator or keyword. If you get general approval on this list, you can then move on to write a PEP (as that's what's needed if you wish to add a new keyword and/or operator to the language).
I'm +0 for now, and may change once you provide us with use cases in the wild.-Emanuel
From: me at jeltef.nl
Date: Mon, 23 Nov 2015 20:09:17 +0100
To: python-ideas at python.org
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean operators

Hi,
After reading PEP0465 about the dedicated matrix multiplication I started wondering if the same solution couldn't be applied to boolean operators as well. There currently are a lot of high profile libraries that have their own functions for boolean operators, like Numpy, Pandas or SQLAlchemy. They do this because the current boolean operators can't be overloaded. PEP0335 was created to solve this problem (and makes the problem more clear), but was rejected because it needed changes to the byte code for the boolean operators, which would make them slower. Currently some of these libraries resort to the bitwise operators (at least Pandas), but those don't bind as strong as comparison operators, which means you have to do comparisons like this: (series1 == 2) & (series2 == 3) 
That is why I propose to create new operators just like for matrix multiplication which can be used in libraries that need one. I'm not sure what the operators should look like, but my first guess would be &&, || and ! for and, or and not respectively. 
Is this an idea that sounds reasonable?
Jelte

_______________________________________________
Python-ideas mailing list
Python-ideas at python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/ 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151123/49533ef4/attachment.html>

From me at jeltef.nl  Mon Nov 23 18:49:30 2015
From: me at jeltef.nl (Jelte Fennema)
Date: Mon, 23 Nov 2015 15:49:30 -0800 (PST)
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
Message-ID: <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>

Some examples of current practice for SQLAlchemy (an SQL ORM) can be found 
here: http://docs.sqlalchemy.org/en/rel_1_0/orm/tutorial.html#common-filter-operators
An slightly adapted example is this: 

from sqlalchemy import or_, and_
query.filter(or_(User.name == 'ed', and_(User.name == 'wendy', User.age > 
20)))

With the new operators this could simply be rewritten to:
query.filter(User.name == 'ed' || (User.name == 'wendy' && User.age > 20))
This is much clearer in my opinion.

Pandas overloads the binary and and  or operators, which causes the small 
issue that it needs an extra pair of braces around expressions, see 
https://stackoverflow.com/questions/24775648/element-wise-logcial-or-in-pandas 
or http://stackoverflow.com/a/19581644/2570866

This means that this (which selects the rows that either are lower than 
three or equal to five):
df[(df < 3) | (df == 5)]
Can be rewritten to this:
df[df < 3 || df == 5]
This is clearly little advantage, but it also means that there is no need 
to override the binary or and and. That way they can be used for their 
original purpose.

For Numpy the case is again like with SQLAlchemy see 
: https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_and.html 
and https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_or.html

I hope this made the use for the overloadable logical and and or operators 
clear. The not operator might be a bit less useful and I'm not sure it's 
needed as much. Currently SQLAlchemy and Pandas overload the "~" (invert) 
operator and Numpy has a function again: 
https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_not.html 

Lastly, for SQLAlchemy an in operator that does not return a boolean could 
also be useful. I can't think of use cases for the others though and I also 
can't directly think of an operator that would be as clear as the &&, || 
and ! operators.

As for the PEP, I have no problem writing one if this is accepted as a 
useful addition. Also any suggestions and critiques are very welcome of 
course. 

On Monday, 23 November 2015 21:24:23 UTC+1, Emanuel Barry wrote:
>
> Could you provide some links to where this could be useful, and how code 
> could be rewritten? I can see the desire for such a feature, I myself would 
> have liked such an operator or keyword. If you get general approval on this 
> list, you can then move on to write a PEP (as that's what's needed if you 
> wish to add a new keyword and/or operator to the language).
>
> I'm +0 for now, and may change once you provide us with use cases in the 
> wild.
> -Emanuel
>
> ------------------------------
> From: m... at jeltef.nl <javascript:>
> Date: Mon, 23 Nov 2015 20:09:17 +0100
> To: python... at python.org <javascript:>
> Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean 
> operators
>
> Hi,
>
> After reading PEP0465 <https://www.python.org/dev/peps/pep-0465/> about 
> the dedicated matrix multiplication I started wondering if the same 
> solution couldn't be applied to boolean operators as well. There currently 
> are a lot of high profile libraries that have their own functions for 
> boolean operators, like Numpy, Pandas or SQLAlchemy. They do this because 
> the current boolean operators can't be overloaded. PEP0335 
> <https://www.python.org/dev/peps/pep-0335/> was created to solve this 
> problem (and makes the problem more clear), but was rejected because it 
> needed changes to the byte code for the boolean operators, which would make 
> them slower. Currently some of these libraries resort to the bitwise 
> operators (at least Pandas), but those don't bind as strong as comparison 
> operators, which means you have to do comparisons like this: (series1 == 2) 
> & (series2 == 3) 
>
> That is why I propose to create new operators just like for matrix 
> multiplication which can be used in libraries that need one. I'm not sure 
> what the operators should look like, but my first guess would be &&, || and 
> ! for and, or and not respectively. 
>
> Is this an idea that sounds reasonable?
>
> Jelte
>
> _______________________________________________ Python-ideas mailing list 
> Python... at python.org <javascript:> 
> https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: 
> http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151123/79e9f330/attachment-0001.html>

From me at jeltef.nl  Mon Nov 23 18:49:30 2015
From: me at jeltef.nl (Jelte Fennema)
Date: Mon, 23 Nov 2015 15:49:30 -0800 (PST)
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
Message-ID: <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>

Some examples of current practice for SQLAlchemy (an SQL ORM) can be found 
here: http://docs.sqlalchemy.org/en/rel_1_0/orm/tutorial.html#common-filter-operators
An slightly adapted example is this: 

from sqlalchemy import or_, and_
query.filter(or_(User.name == 'ed', and_(User.name == 'wendy', User.age > 
20)))

With the new operators this could simply be rewritten to:
query.filter(User.name == 'ed' || (User.name == 'wendy' && User.age > 20))
This is much clearer in my opinion.

Pandas overloads the binary and and  or operators, which causes the small 
issue that it needs an extra pair of braces around expressions, see 
https://stackoverflow.com/questions/24775648/element-wise-logcial-or-in-pandas 
or http://stackoverflow.com/a/19581644/2570866

This means that this (which selects the rows that either are lower than 
three or equal to five):
df[(df < 3) | (df == 5)]
Can be rewritten to this:
df[df < 3 || df == 5]
This is clearly little advantage, but it also means that there is no need 
to override the binary or and and. That way they can be used for their 
original purpose.

For Numpy the case is again like with SQLAlchemy see 
: https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_and.html 
and https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_or.html

I hope this made the use for the overloadable logical and and or operators 
clear. The not operator might be a bit less useful and I'm not sure it's 
needed as much. Currently SQLAlchemy and Pandas overload the "~" (invert) 
operator and Numpy has a function again: 
https://docs.scipy.org/doc/numpy/reference/generated/numpy.logical_not.html 

Lastly, for SQLAlchemy an in operator that does not return a boolean could 
also be useful. I can't think of use cases for the others though and I also 
can't directly think of an operator that would be as clear as the &&, || 
and ! operators.

As for the PEP, I have no problem writing one if this is accepted as a 
useful addition. Also any suggestions and critiques are very welcome of 
course. 

On Monday, 23 November 2015 21:24:23 UTC+1, Emanuel Barry wrote:
>
> Could you provide some links to where this could be useful, and how code 
> could be rewritten? I can see the desire for such a feature, I myself would 
> have liked such an operator or keyword. If you get general approval on this 
> list, you can then move on to write a PEP (as that's what's needed if you 
> wish to add a new keyword and/or operator to the language).
>
> I'm +0 for now, and may change once you provide us with use cases in the 
> wild.
> -Emanuel
>
> ------------------------------
> From: m... at jeltef.nl <javascript:>
> Date: Mon, 23 Nov 2015 20:09:17 +0100
> To: python... at python.org <javascript:>
> Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean 
> operators
>
> Hi,
>
> After reading PEP0465 <https://www.python.org/dev/peps/pep-0465/> about 
> the dedicated matrix multiplication I started wondering if the same 
> solution couldn't be applied to boolean operators as well. There currently 
> are a lot of high profile libraries that have their own functions for 
> boolean operators, like Numpy, Pandas or SQLAlchemy. They do this because 
> the current boolean operators can't be overloaded. PEP0335 
> <https://www.python.org/dev/peps/pep-0335/> was created to solve this 
> problem (and makes the problem more clear), but was rejected because it 
> needed changes to the byte code for the boolean operators, which would make 
> them slower. Currently some of these libraries resort to the bitwise 
> operators (at least Pandas), but those don't bind as strong as comparison 
> operators, which means you have to do comparisons like this: (series1 == 2) 
> & (series2 == 3) 
>
> That is why I propose to create new operators just like for matrix 
> multiplication which can be used in libraries that need one. I'm not sure 
> what the operators should look like, but my first guess would be &&, || and 
> ! for and, or and not respectively. 
>
> Is this an idea that sounds reasonable?
>
> Jelte
>
> _______________________________________________ Python-ideas mailing list 
> Python... at python.org <javascript:> 
> https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: 
> http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151123/79e9f330/attachment-0002.html>

From rosuav at gmail.com  Mon Nov 23 19:08:25 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 24 Nov 2015 11:08:25 +1100
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
Message-ID: <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>

On Tue, Nov 24, 2015 at 10:49 AM, Jelte Fennema <me at jeltef.nl> wrote:
> As for the PEP, I have no problem writing one if this is accepted as a
> useful addition. Also any suggestions and critiques are very welcome of
> course.

I think it's reasonable, except for the potential confusion of having
*three* "and" operators. The one with the word is never going to
change - its semantics demand that it not be overridable. When should
you use & and when &&? Judging by how @ has gone, I think the answer
will be simple: "Always use &, unless the docs for some third-party
library say to use &&", in which case I think it should be okay.

ChrisA

From guido at python.org  Mon Nov 23 19:13:42 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 23 Nov 2015 16:13:42 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
Message-ID: <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>

I honestly think the added confusion makes it a non-starter. It's also
confusing that in other languages that have && and ||, they are
shortcut operators, but the proposed operators here won't be. And the
real question isn't "when to use & vs. &&", it's "when to use 'and'
vs. &&".

On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Tue, Nov 24, 2015 at 10:49 AM, Jelte Fennema <me at jeltef.nl> wrote:
>> As for the PEP, I have no problem writing one if this is accepted as a
>> useful addition. Also any suggestions and critiques are very welcome of
>> course.
>
> I think it's reasonable, except for the potential confusion of having
> *three* "and" operators. The one with the word is never going to
> change - its semantics demand that it not be overridable. When should
> you use & and when &&? Judging by how @ has gone, I think the answer
> will be simple: "Always use &, unless the docs for some third-party
> library say to use &&", in which case I think it should be okay.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
--Guido van Rossum (python.org/~guido)

From me at jeltef.nl  Mon Nov 23 19:17:58 2015
From: me at jeltef.nl (Jelte Fennema)
Date: Tue, 24 Nov 2015 01:17:58 +0100
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
Message-ID: <CAGECzQSG2g3SzcfbdGStMUjUGwsBZPX8VdhLQnDZC5BcVYtUCA@mail.gmail.com>

This confusion could quite simply be solved by just not implementing the
operations on the standard types, even though they would be trivial to
implement. This just leaves the possibility for library developers to do
something useful with the operators, like with the new @ operator.

On 24 November 2015 at 01:13, Guido van Rossum <guido at python.org> wrote:

> I honestly think the added confusion makes it a non-starter. It's also
> confusing that in other languages that have && and ||, they are
> shortcut operators, but the proposed operators here won't be. And the
> real question isn't "when to use & vs. &&", it's "when to use 'and'
> vs. &&".
>
> On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico <rosuav at gmail.com> wrote:
> > On Tue, Nov 24, 2015 at 10:49 AM, Jelte Fennema <me at jeltef.nl> wrote:
> >> As for the PEP, I have no problem writing one if this is accepted as a
> >> useful addition. Also any suggestions and critiques are very welcome of
> >> course.
> >
> > I think it's reasonable, except for the potential confusion of having
> > *three* "and" operators. The one with the word is never going to
> > change - its semantics demand that it not be overridable. When should
> > you use & and when &&? Judging by how @ has gone, I think the answer
> > will be simple: "Always use &, unless the docs for some third-party
> > library say to use &&", in which case I think it should be okay.
> >
> > ChrisA
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/220d5dbb/attachment.html>

From bruce at leban.us  Mon Nov 23 19:26:18 2015
From: bruce at leban.us (Bruce Leban)
Date: Mon, 23 Nov 2015 16:26:18 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
Message-ID: <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>

On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico <rosuav at gmail.com> wrote:

> I think it's reasonable, except for the potential confusion of having
> *three* "and" operators.
>

I think using && and || would be an attractive nuisance for people
switching from another programming language. Right now if I accidentally
right && in Python or "and" in another language, I get an immediate syntax
error. With this proposal, I get unexpected results.

If this idea were to fly, a better name would be something that doesn't
have that problem, e.g., .and. .or. .not. I don't want to bikeshed the
exact syntax**, but I think it should be clear that with something like
this: (1) no one is going to accidentally type them and (2) they are pretty
clearly some variation of the standard and/or/not.

**Lots of other possibilities that are syntax errors right now: @and, (and),
etc. I like .and. because it's less visual clutter and it's easy to type.

--- Bruce
Check out my puzzle book and get it free here:
http://J.mp/ingToConclusionsFree (available on iOS)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151123/bd2c799c/attachment.html>

From me at jeltef.nl  Mon Nov 23 20:11:47 2015
From: me at jeltef.nl (Jelte Fennema)
Date: Tue, 24 Nov 2015 02:11:47 +0100
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
Message-ID: <CAGECzQQTAa+oOvEsfSO0OK96TYdcJSQaVsmb2RQ-xaqrtdXjeA@mail.gmail.com>

On 24 November 2015 at 01:26, Bruce Leban <bruce at leban.us> wrote:

> If this idea were to fly, a better name would be something that doesn't
> have that problem, e.g., .and. .or. .not. I don't want to bikeshed the
> exact syntax**, but I think it should be clear that with something like
> this: (1) no one is going to accidentally type them and (2) they are pretty
> clearly some variation of the standard and/or/not.


I think a naming scheme like that is indeed a good way to solve the
confusion issues, since it is also immediately clear that these would be a
special version of the normal operators. Another advantage is that it could
also be extended to the in operator, if that one is to be included.

I'm not sure I like the dots version very much though, but like you said
there are lots of syntax error options to choose from.

On 24 November 2015 at 01:26, Bruce Leban <bruce at leban.us> wrote:

>
> On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
>> I think it's reasonable, except for the potential confusion of having
>> *three* "and" operators.
>>
>
> I think using && and || would be an attractive nuisance for people
> switching from another programming language. Right now if I accidentally
> right && in Python or "and" in another language, I get an immediate syntax
> error. With this proposal, I get unexpected results.
>
> If this idea were to fly, a better name would be something that doesn't
> have that problem, e.g., .and. .or. .not. I don't want to bikeshed the
> exact syntax**, but I think it should be clear that with something like
> this: (1) no one is going to accidentally type them and (2) they are pretty
> clearly some variation of the standard and/or/not.
>
> **Lots of other possibilities that are syntax errors right now: @and, (and),
> etc. I like .and. because it's less visual clutter and it's easy to type.
>
> --- Bruce
> Check out my puzzle book and get it free here:
> http://J.mp/ingToConclusionsFree (available on iOS)
>
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/cbcc9bf7/attachment-0001.html>

From chris.barker at noaa.gov  Mon Nov 23 20:38:38 2015
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Mon, 23 Nov 2015 17:38:38 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
Message-ID: <-334649383431326154@unknownmsgid>

> I honestly think the added confusion makes it a non-starter.

Coming from my experience from the Numpy world, the fact that you get
"rich comparisons" for most what seem like Boolean operators, but not
for and and or is very confusing to newbies.

Much of the time, you can use the bitwise operators, as you often have
done a comparison first :

(A < x) & (A > y)

But it's kind of a coincidence that it works, so it only makes thing
more confusing for newbies. Bitwise operators really are kind of
obscure these days.

Explaining that you use ".and." Instead of " and" would be a much
lighter lift than getting into the whole explanation for why we can't
overload "and".

-CHB

I think first we decide it is or isn't a good idea, and then decide
how to spell it, but .and. and .or. kind of appeal to me.





> It's also
> confusing that in other languages that have && and ||, they are
> shortcut operators, but the proposed operators here won't be. And the
> real question isn't "when to use & vs. &&", it's "when to use 'and'
> vs. &&".
>
>> On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico <rosuav at gmail.com> wrote:
>>> On Tue, Nov 24, 2015 at 10:49 AM, Jelte Fennema <me at jeltef.nl> wrote:
>>> As for the PEP, I have no problem writing one if this is accepted as a
>>> useful addition. Also any suggestions and critiques are very welcome of
>>> course.
>>
>> I think it's reasonable, except for the potential confusion of having
>> *three* "and" operators. The one with the word is never going to
>> change - its semantics demand that it not be overridable. When should
>> you use & and when &&? Judging by how @ has gone, I think the answer
>> will be simple: "Always use &, unless the docs for some third-party
>> library say to use &&", in which case I think it should be okay.
>>
>> ChrisA
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From mike at selik.org  Mon Nov 23 20:44:03 2015
From: mike at selik.org (Michael Selik)
Date: Tue, 24 Nov 2015 01:44:03 +0000
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAGECzQQTAa+oOvEsfSO0OK96TYdcJSQaVsmb2RQ-xaqrtdXjeA@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
 <CAGECzQQTAa+oOvEsfSO0OK96TYdcJSQaVsmb2RQ-xaqrtdXjeA@mail.gmail.com>
Message-ID: <CADqi7965MTn=-+u7jr5W8gsDDeDVJ3Ds=a5C+pPgT_yCqznwhA@mail.gmail.com>

Why hasn't SQLAlchemy gone the route of NumPy with overloaded operators?
Perhaps whatever reason it is would prevent using any new operators as well.

With NumPy I make that mistake constantly:
    A == a & B == b
rather than
    (A == a) & (B == b)

I'd put that in the category of parentheses tax along with the print
function and old style % string interpolation. Annoying, but it's
inappropriate to use a gun to swat a fly. (In case my metaphor is unclear,
creating a new operator is the gun -- risking collateral damage and all
that)

As Guido said, the real usability problem is that the ``and`` operator is a
new Python programmer's first instinct. Adding yet another operator would
make Python harder to learn and read. Even if you advertise a new operator,
many libraries will be slow to change and we'll have 3 different techniques
to teach. Let's weigh the benefits against the negative consequences.

On Mon, Nov 23, 2015 at 8:12 PM Jelte Fennema <me at jeltef.nl> wrote:

> On 24 November 2015 at 01:26, Bruce Leban <bruce at leban.us> wrote:
>
>> If this idea were to fly, a better name would be something that doesn't
>> have that problem, e.g., .and. .or. .not. I don't want to bikeshed the
>> exact syntax**, but I think it should be clear that with something like
>> this: (1) no one is going to accidentally type them and (2) they are pretty
>> clearly some variation of the standard and/or/not.
>
>
> I think a naming scheme like that is indeed a good way to solve the
> confusion issues, since it is also immediately clear that these would be a
> special version of the normal operators. Another advantage is that it could
> also be extended to the in operator, if that one is to be included.
>
> I'm not sure I like the dots version very much though, but like you said
> there are lots of syntax error options to choose from.
>
> On 24 November 2015 at 01:26, Bruce Leban <bruce at leban.us> wrote:
>
>>
>> On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico <rosuav at gmail.com> wrote:
>>
>>> I think it's reasonable, except for the potential confusion of having
>>> *three* "and" operators.
>>>
>>
>> I think using && and || would be an attractive nuisance for people
>> switching from another programming language. Right now if I accidentally
>> right && in Python or "and" in another language, I get an immediate syntax
>> error. With this proposal, I get unexpected results.
>>
>> If this idea were to fly, a better name would be something that doesn't
>> have that problem, e.g., .and. .or. .not. I don't want to bikeshed the
>> exact syntax**, but I think it should be clear that with something like
>> this: (1) no one is going to accidentally type them and (2) they are pretty
>> clearly some variation of the standard and/or/not.
>>
>> **Lots of other possibilities that are syntax errors right now: @and, (and),
>> etc. I like .and. because it's less visual clutter and it's easy to type.
>>
>> --- Bruce
>> Check out my puzzle book and get it free here:
>> http://J.mp/ingToConclusionsFree (available on iOS)
>>
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "python-ideas" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> python-ideas+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "python-ideas" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> python-ideas+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/a52a2bb2/attachment.html>

From me at jeltef.nl  Mon Nov 23 21:09:27 2015
From: me at jeltef.nl (Jelte Fennema)
Date: Tue, 24 Nov 2015 03:09:27 +0100
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CADqi7965MTn=-+u7jr5W8gsDDeDVJ3Ds=a5C+pPgT_yCqznwhA@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
 <CAGECzQQTAa+oOvEsfSO0OK96TYdcJSQaVsmb2RQ-xaqrtdXjeA@mail.gmail.com>
 <CADqi7965MTn=-+u7jr5W8gsDDeDVJ3Ds=a5C+pPgT_yCqznwhA@mail.gmail.com>
Message-ID: <CAGECzQSPx+hv_XjYD9ktofGi1ru2fxTjJTA9uGacwzdYAp1sbg@mail.gmail.com>

On 24 November 2015 at 02:44, Michael Selik <mike at selik.org> wrote:

> Why hasn't SQLAlchemy gone the route of NumPy with overloaded operators?
>
It seems I was wrong about that, they apparently do:
http://stackoverflow.com/a/14185275/2570866

I'd put that in the category of parentheses tax along with the print
> function and old style % string interpolation.

This seems like a bit of a weird argument since the parentheses for the
print function are put there for a reason (see PEP3105) and the old style %
string interpolation will be replaced by the new format string literal.

Adding yet another operator would make Python harder to learn and read.
> Even if you advertise a new operator, many libraries will be slow to change
> and we'll have 3 different techniques to teach.

I don't think that much confusion will arise, since the normal way is to to
use is the short `and` version. Only in libraries where it would be
explicitly told the new operator would be used. It would also do away with
the confusion about why and cannot be overridden and why the precedence of
the & operator is "wrong".

On 24 November 2015 at 02:44, Michael Selik <mike at selik.org> wrote:

> Why hasn't SQLAlchemy gone the route of NumPy with overloaded operators?
> Perhaps whatever reason it is would prevent using any new operators as
> well.
>
> With NumPy I make that mistake constantly:
>     A == a & B == b
> rather than
>     (A == a) & (B == b)
>
> I'd put that in the category of parentheses tax along with the print
> function and old style % string interpolation. Annoying, but it's
> inappropriate to use a gun to swat a fly. (In case my metaphor is unclear,
> creating a new operator is the gun -- risking collateral damage and all
> that)
>
> As Guido said, the real usability problem is that the ``and`` operator is
> a new Python programmer's first instinct. Adding yet another operator would
> make Python harder to learn and read. Even if you advertise a new operator,
> many libraries will be slow to change and we'll have 3 different techniques
> to teach. Let's weigh the benefits against the negative consequences.
>
> On Mon, Nov 23, 2015 at 8:12 PM Jelte Fennema <me at jeltef.nl> wrote:
>
>> On 24 November 2015 at 01:26, Bruce Leban <bruce at leban.us> wrote:
>>
>>> If this idea were to fly, a better name would be something that doesn't
>>> have that problem, e.g., .and. .or. .not. I don't want to bikeshed the
>>> exact syntax**, but I think it should be clear that with something like
>>> this: (1) no one is going to accidentally type them and (2) they are pretty
>>> clearly some variation of the standard and/or/not.
>>
>>
>> I think a naming scheme like that is indeed a good way to solve the
>> confusion issues, since it is also immediately clear that these would be a
>> special version of the normal operators. Another advantage is that it could
>> also be extended to the in operator, if that one is to be included.
>>
>> I'm not sure I like the dots version very much though, but like you said
>> there are lots of syntax error options to choose from.
>>
>> On 24 November 2015 at 01:26, Bruce Leban <bruce at leban.us> wrote:
>>
>>>
>>> On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico <rosuav at gmail.com>
>>> wrote:
>>>
>>>> I think it's reasonable, except for the potential confusion of having
>>>> *three* "and" operators.
>>>>
>>>
>>> I think using && and || would be an attractive nuisance for people
>>> switching from another programming language. Right now if I accidentally
>>> right && in Python or "and" in another language, I get an immediate syntax
>>> error. With this proposal, I get unexpected results.
>>>
>>> If this idea were to fly, a better name would be something that doesn't
>>> have that problem, e.g., .and. .or. .not. I don't want to bikeshed the
>>> exact syntax**, but I think it should be clear that with something like
>>> this: (1) no one is going to accidentally type them and (2) they are pretty
>>> clearly some variation of the standard and/or/not.
>>>
>>> **Lots of other possibilities that are syntax errors right now: @and, (and),
>>> etc. I like .and. because it's less visual clutter and it's easy to
>>> type.
>>>
>>> --- Bruce
>>> Check out my puzzle book and get it free here:
>>> http://J.mp/ingToConclusionsFree (available on iOS)
>>>
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "python-ideas" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> python-ideas+unsubscribe at googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "python-ideas" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> python-ideas+unsubscribe at googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/d9fe555b/attachment-0001.html>

From chris.barker at noaa.gov  Mon Nov 23 21:38:28 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 23 Nov 2015 18:38:28 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CADqi7965MTn=-+u7jr5W8gsDDeDVJ3Ds=a5C+pPgT_yCqznwhA@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
 <CAGECzQQTAa+oOvEsfSO0OK96TYdcJSQaVsmb2RQ-xaqrtdXjeA@mail.gmail.com>
 <CADqi7965MTn=-+u7jr5W8gsDDeDVJ3Ds=a5C+pPgT_yCqznwhA@mail.gmail.com>
Message-ID: <CALGmxE+9ryhDA4qm-=w-kEB4a95q4+ZB=3XLHFgcHauU4eFspQ@mail.gmail.com>

On Mon, Nov 23, 2015 at 5:44 PM, Michael Selik <mike at selik.org> wrote:

> With NumPy I make that mistake constantly:
>     A == a & B == b
> rather than
>     (A == a) & (B == b)
>
> I'd put that in the category of parentheses tax
>

It's not just the operator precedence that's an issue here -- this also
means something different than what people expect. As it happens a bitwise
and works "just like" and when the values are boolean (like above), but
that isn't always the case -- confusion waiting to happen.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151123/3988933a/attachment.html>

From greg.ewing at canterbury.ac.nz  Mon Nov 23 23:55:02 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 24 Nov 2015 17:55:02 +1300
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
Message-ID: <5653EDA6.5020903@canterbury.ac.nz>

Bruce Leban wrote:
> If this idea were to fly, a better name would be something that doesn't 
> have that problem, e.g., .and. .or. .not.

Dots are a bit on the ugly side.

Some other possibilities:

AND, OR, NOT

And, Or, Not

en, of, nlet

-- 
Greg

From abarnert at yahoo.com  Tue Nov 24 00:35:52 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 23 Nov 2015 21:35:52 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <5653EDA6.5020903@canterbury.ac.nz>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
 <5653EDA6.5020903@canterbury.ac.nz>
Message-ID: <7CEF161C-557F-4AF1-ADF8-FD83C301ED2D@yahoo.com>

(Top-posting because this is really a reply to a combination of three earlier posts, not to Greg Ewing's post, except at the very end.)

-1 on "&&" and "||". To anyone familiar with C and friends, it seems like they ought to be like C (short-circuiting, and more generally the same thing we already spell "and" and "or"); to anyone else, it would make no sense for them to have different precedence than "&" and "|".

-0.5 on ".and." and ".or." for multiple reasons, most of which apply just as well to anything similar:

* That looks like a general custom-infix syntax. Anyone coming to Python (including Python 2 users) will expect that they can just as easily define ".spam." (and then be disappointed that they can't...), or will be worried that 3.7 will add a bunch of new dot operators they'll have to learn. Not a _huge_ negative, but already enough to turn me off the idea.

* What would you call the dunder methods (and operator module functions), and how would you deal with the fact that any novice/transplant is going to assume "__and__" means ".and." rather than "&"?

* Operators starting with "." are ambiguous with float literals and/or attribute access ("spam.and.eggs" looks like a member of a member of spam), at least to humans, if not to the parser.

* Operators starting and/or ending with "." are hard to talk about because of natural-language punctuation (and even harder on a mobile keyboard, or an overly clever text editor or word processor).

-0 on actually adding a general custom-infix syntax, come to think of it. Any solution to the problems above should work just as well here. And it means that in the future, libraries don't have to cram things into inappropriate symbols or be stuck with prefix or dot-method notation. And I don't think it would be any harder to learn than a few special cases. And I think ".between." would be just as useful in an ORM as ".in.". More generally, the whole point of "@" was that everyone agreed that it was the only new operator anyone would need (except maybe "@@") for a decade or two; if that's not true, infinity is a better number than 4.

+1 on the idea if someone can come up with a good spelling that avoids all the above problems and reads as naturally as "@".

> On Nov 23, 2015, at 20:55, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> 
> Bruce Leban wrote:
>> If this idea were to fly, a better name would be something that doesn't have that problem, e.g., .and. .or. .not.
> 
> Dots are a bit on the ugly side.
> 
> Some other possibilities:
> 
> AND, OR, NOT
> 
> And, Or, Not
> 
> en, of, nlet

Is that last one supposed to be "niet"? When I try "nlet", Google assumes it's a typo for "net", which I guess could be a unary boolean identity operator, but I don't think we need that. :)

Anyway, that pattern is a bit hard to extend to an overloadable "in" operator, because Dutch for "in" is "in".

From ncoghlan at gmail.com  Tue Nov 24 00:40:53 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 24 Nov 2015 15:40:53 +1000
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <-334649383431326154@unknownmsgid>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
Message-ID: <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>

On 24 November 2015 at 11:38, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
> I think first we decide it is or isn't a good idea, and then decide
> how to spell it, but .and. and .or. kind of appeal to me.

I think it's reasonably clear that rich logical operators with
appropriate precedence and non-shortcircuiting behaviour would be a
nice feature to have, the question is whether or not they can be
introduced without making things even more confusing than they already
are. Using placeholder syntax, let's consider the three operations:

  A rich_and B
  A rich_or B
  rich_not A

To explain this fully will require explaining how they differ from:

  A and B  -> A if not bool(A) else B
  A or B  -> A if bool(A) else B
  not A -> not bool(A)

and:

  A & B -> operator.and_(A, B)
  A | B -> operator.or_(A, B)
  ~A -> operator.not_(A)

Depending on a user's background, it will also potentially require
explaining how they differ from these operations in C and other
languages:

  A && B
  A || B
  !A

Casting any new forms specifically as "matrix" operators (like the new
matmul operator Jelte referenced in the opening message of the thread)
also runs into problems, since "@" is a genuinely distinct operation
only applicable to matrices, while the goal here is instead to
broadcast an existing operation over the array elements, which matrix
objects are already able to do implicitly for most binary operators.

Something that *could* potentially be comprehensible is the idea of
allowing "elementwise" logical operators, with a suitable syntactic
spelling. There'd still be a slight niggle to explain why and/or/not
have explicitly elementwise variants when other binary operations
don't, but that would likely still be less confusing than explaining
the use of the bitwise operators.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From random832 at fastmail.com  Tue Nov 24 01:34:20 2015
From: random832 at fastmail.com (Random832)
Date: Tue, 24 Nov 2015 06:34:20 +0000 (UTC)
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
Message-ID: <n310dc$92q$1@ger.gmane.org>

On 2015-11-24, Chris Angelico <rosuav at gmail.com> wrote:
> I think it's reasonable, except for the potential confusion of having
> *three* "and" operators. The one with the word is never going to
> change - its semantics demand that it not be overridable.

How? Are you referring to the short-circuit? C# allows
overloading the short-circuit operators by doing it in two
parts - in pseudocode:

a and b: a if a.__false__() else a & b
a or b: a if a.__true__() else a | b

There's no fundamental reason short-circuiting "demands" that it
not be overridable, just because C++ can't do it.


From rosuav at gmail.com  Tue Nov 24 01:52:13 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 24 Nov 2015 17:52:13 +1100
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <n310dc$92q$1@ger.gmane.org>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org>
Message-ID: <CAPTjJmot3sCFo0xjTQoELGKW0dbvkhkmDZDZNnTxZ2Rqg4-WGQ@mail.gmail.com>

On Tue, Nov 24, 2015 at 5:34 PM, Random832 <random832 at fastmail.com> wrote:
> On 2015-11-24, Chris Angelico <rosuav at gmail.com> wrote:
>> I think it's reasonable, except for the potential confusion of having
>> *three* "and" operators. The one with the word is never going to
>> change - its semantics demand that it not be overridable.
>
> How? Are you referring to the short-circuit? C# allows
> overloading the short-circuit operators by doing it in two
> parts - in pseudocode:
>
> a and b: a if a.__false__() else a & b
> a or b: a if a.__true__() else a | b
>
> There's no fundamental reason short-circuiting "demands" that it
> not be overridable, just because C++ can't do it.

The Python semantics are defined more tightly than that, though - the
"else" clause in each case would simply be "b". Changing that is not
something you can do with operator overloading, so it would mean a
fundamental change in the operator's semantics. And once you do that,
you end up with a completely different operator.

So, yes, the semantics of Python's short-circuiting 'and' and 'or'
operators precludes any form of overriding. Yes, it's possible to have
overridable short-circuiting operators, but Python's ones are not
those. (Also, I think I prefer the simpler semantics. But that's a
matter of personal choice.)

ChrisA

From abarnert at yahoo.com  Tue Nov 24 02:15:16 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 23 Nov 2015 23:15:16 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <n310dc$92q$1@ger.gmane.org>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org>
Message-ID: <8FAFB343-875C-4365-A918-D7146D338FF2@yahoo.com>

On Nov 23, 2015, at 22:34, Random832 <random832 at fastmail.com> wrote:
> 
>> On 2015-11-24, Chris Angelico <rosuav at gmail.com> wrote:
>> I think it's reasonable, except for the potential confusion of having
>> *three* "and" operators. The one with the word is never going to
>> change - its semantics demand that it not be overridable.
> 
> How? Are you referring to the short-circuit? C# allows
> overloading the short-circuit operators by doing it in two
> parts - in pseudocode:
> 
> a and b: a if a.__false__() else a & b
> a or b: a if a.__true__() else a | b
> 
> There's no fundamental reason short-circuiting "demands" that it
> not be overridable, just because C++ can't do it.

Actually, it _is_ possible in C++. You can pretty much do anything in C++, as long as you're an expert, you don't care about anyone reading your code, and you don't mind 1970s-style build times, and this is no exception. You just write an expression template library that uses non-short-circuiting-at-compile-time operators to generate functions that are short-circuiting at runtime, and then add the boilerplate to wrap any constants or normal variables up into your expression template types, and you're set.

A simpler solution is to use lambda lifting, like Swift: "a and b" just means "a.__and__(lambda: b)".

You could also split the operator into two method calls, without combining with bool conversion a la C#. So "a and b" becomes "tmp = a.__and1__()", then "tmp.__and2__(b) if tmp2 else tmp2". That way, "and" can return a non-boolean value, just as it does when not overloaded. I can't think of any language that does this off the top of my head, but I'm sure they exist.

Or, in languages with macros or equivalent, you just use a macro instead of a function.

Or, in languages with lazy evaluation by default, it's even simpler--if you don't use the result of "b" anywhere, it never gets evaluated.



From greg.ewing at canterbury.ac.nz  Tue Nov 24 01:05:09 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 24 Nov 2015 19:05:09 +1300
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
Message-ID: <5653FE15.8080402@canterbury.ac.nz>

Nick Coghlan wrote:
> Something that *could* potentially be comprehensible is the idea of
> allowing "elementwise" logical operators, with a suitable syntactic
> spelling.

-1 on tying any of this explicity to a notion of
elementwise operations. The range of potential uses
is wider than that.

-- 
Greg

From sjoerdjob at sjec.nl  Tue Nov 24 03:03:50 2015
From: sjoerdjob at sjec.nl (Sjoerd Job Postmus)
Date: Tue, 24 Nov 2015 09:03:50 +0100
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
Message-ID: <20151124080350.GA30034@sjoerdjob.com>

Regarding the short-circuiting `and` and `or`, I think there's a way we
can have our overloading-cake and only eat the pieces we need too.

Even though it might not be totally intuitive, it should be possible to
make the overloaded method get a 0-argument function instead of a value.

In that case, `a and b` would become `a.__land__(lambda: b)`. As far as
performance goes, there probably would have to be some special casing
that first checks if `a` has an overloaded `and`, and if not use the
default behaviour.

The default implementation:

    class object:
        def __land__(self, other_f):
    	    if not self:
    	        return self
             else:
                return other_f()

        def __lor__(self, other_f):
            if self:
                return self
            else:
                return other_f()

And for some python-expressions-to-AST nodeclass:

    class BaseExpression:
        def __land__(self, other_f):
            return AndExpression(self, other_f())
        def __lor__(self, other_f):
            return OrExpression(self, other_f())

Again, the fact that the second argument to `__land__`/`__or__` is a
function might be a bit confusing, but that's probably going to be the
only way to make short-circuiting work for an overloaded and/or without
going the route of lazy evaluation.

On Mon, Nov 23, 2015 at 04:13:42PM -0800, Guido van Rossum wrote:
> I honestly think the added confusion makes it a non-starter. It's also
> confusing that in other languages that have && and ||, they are
> shortcut operators, but the proposed operators here won't be. And the
> real question isn't "when to use & vs. &&", it's "when to use 'and'
> vs. &&".
> 
> On Mon, Nov 23, 2015 at 4:08 PM, Chris Angelico <rosuav at gmail.com> wrote:
> > On Tue, Nov 24, 2015 at 10:49 AM, Jelte Fennema <me at jeltef.nl> wrote:
> >> As for the PEP, I have no problem writing one if this is accepted as a
> >> useful addition. Also any suggestions and critiques are very welcome of
> >> course.
> >
> > I think it's reasonable, except for the potential confusion of having
> > *three* "and" operators. The one with the word is never going to
> > change - its semantics demand that it not be overridable. When should
> > you use & and when &&? Judging by how @ has gone, I think the answer
> > will be simple: "Always use &, unless the docs for some third-party
> > library say to use &&", in which case I think it should be okay.
> >
> > ChrisA
> > _______________________________________________
> > Python-ideas mailing list
> > Python-ideas at python.org
> > https://mail.python.org/mailman/listinfo/python-ideas
> > Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From njs at pobox.com  Tue Nov 24 03:30:57 2015
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 24 Nov 2015 00:30:57 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <n310dc$92q$1@ger.gmane.org>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org>
Message-ID: <CAPJVwB=xgF6DQiV4kLR0xd0tL_q-qHwaj_2jT1vqjLZTHvLe3Q@mail.gmail.com>

On Nov 23, 2015 22:34, "Random832" <random832 at fastmail.com> wrote:
>
> On 2015-11-24, Chris Angelico <rosuav at gmail.com> wrote:
> > I think it's reasonable, except for the potential confusion of having
> > *three* "and" operators. The one with the word is never going to
> > change - its semantics demand that it not be overridable.
>
> How? Are you referring to the short-circuit? C# allows
> overloading the short-circuit operators by doing it in two
> parts - in pseudocode:
>
> a and b: a if a.__false__() else a & b
> a or b: a if a.__true__() else a | b
>
> There's no fundamental reason short-circuiting "demands" that it
> not be overridable, just because C++ can't do it.

No, I disagree -- for us short circuiting makes overriding extremely
difficult, probably impossible, because in python in general and in the use
cases being discussed here in particular, the right-hand side argument
should have a chance to overload. It's going to be brutal on newbies and
general code comprehensibility if

  True and array_of_bools

or

  True and sqlalchemy_expression

act totally differently than the mirrored versions, but this is an
unavoidable if you have both short circuiting and overloading.

(Example of where you'd see code like this:

if restrict_user_id is not None:
  user_id_constraint = (MyClass.user_id == restrict_user_id)
else:
  user_id_constraint = True
if restrict_month is not None:
  month_constraint = (MyClass.month == restrict_month)
else:
  month_constraint = True
...
return query(MyClass).filter(user_id_constraint and month_constraint and
...)

Yes, this could be written differently, but this pattern comes up fairly
often IME. And it's otherwise very reliable; in numpy in particular 'True
op array' and 'array(True) op array' act identically for every overloaded
binary operator.)

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/52fe14f3/attachment-0001.html>

From mike at selik.org  Tue Nov 24 04:14:58 2015
From: mike at selik.org (Michael Selik)
Date: Tue, 24 Nov 2015 09:14:58 +0000
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAGECzQSPx+hv_XjYD9ktofGi1ru2fxTjJTA9uGacwzdYAp1sbg@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
 <CAGECzQQTAa+oOvEsfSO0OK96TYdcJSQaVsmb2RQ-xaqrtdXjeA@mail.gmail.com>
 <CADqi7965MTn=-+u7jr5W8gsDDeDVJ3Ds=a5C+pPgT_yCqznwhA@mail.gmail.com>
 <CAGECzQSPx+hv_XjYD9ktofGi1ru2fxTjJTA9uGacwzdYAp1sbg@mail.gmail.com>
Message-ID: <CADqi7970HqAOMAUEEy=cbaKDcKO0hWQwkvwYfy9reEcuaWXEGA@mail.gmail.com>

As I think of people's reactions to seeing & and | for the first time, the
typical response is, "What do you mean by bitwise?" Not, "Why are you using
bitwise operators for non-bitwise operations?" Interestingly, no one in
this thread seems to have a problem with ``&`` and ``|`` for set
intersection and union. The primary complaint is that NumPy users
instinctively reach for ``and``/``or`` and then forget the operator
precedence of ``&``/``|``.


On Mon, Nov 23, 2015 at 9:09 PM Jelte Fennema <me at jeltef.nl> wrote:

> On 24 November 2015 at 02:44, Michael Selik <mike at selik.org> wrote:
>
>> Why hasn't SQLAlchemy gone the route of NumPy with overloaded operators?
>>
> It seems I was wrong about that, they apparently do:
> http://stackoverflow.com/a/14185275/2570866
>

Using pipe and ampersand looks readable. The use of bitwise operators for
overloading and/or/not seems standard for many objects, in standard library
and major projects. In fact, I use those operators as logical far more
often than as bitwise. I'd bet a great number of NumPy users in the science
community are completely unaware of their bitwise effects.


I'd put that in the category of parentheses tax along with the print
>> function and old style % string interpolation.
>
> This seems like a bit of a weird argument since the parentheses for the
> print function are put there for a reason (see PEP3105) and the old style %
> string interpolation will be replaced by the new format string literal.
>

I picked ``print`` as an example to show that requiring parens is fine. I
picked string interpolation as an example because the ``.format`` solution
also accepted parens as necessary. The rationale for f-string syntax (as
written in the PEP) does not complain about parens but instead mentions
that they will be necessary inside the string in several circumstances,
like lambdas.

Adding yet another operator would make Python harder to learn and read.
>> Even if you advertise a new operator, many libraries will be slow to change
>> and we'll have 3 different techniques to teach.
>
> I don't think that much confusion will arise, since the normal way is to
> to use is the short `and` version. Only in libraries where it would be
> explicitly told the new operator would be used. It would also do away with
> the confusion about why and cannot be overridden and why the precedence of
> the & operator is "wrong".
>

That precedence issue doesn't cause problems when teaching, in my
experience. People accept readily that there's issues with operator
precedence. On the other hand, they have trouble reading code that uses an
older style (perhaps using ``&``) instead of the newer style that they
learned (the hypothetical ``&&``). Even the switch between ``%``
interpolation and ``.format`` is still causing problems in large
organizations.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/1ade5c2b/attachment.html>

From me at jeltef.nl  Tue Nov 24 04:59:44 2015
From: me at jeltef.nl (Jelte Fennema)
Date: Tue, 24 Nov 2015 10:59:44 +0100
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CADqi7970HqAOMAUEEy=cbaKDcKO0hWQwkvwYfy9reEcuaWXEGA@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
 <CAGECzQQTAa+oOvEsfSO0OK96TYdcJSQaVsmb2RQ-xaqrtdXjeA@mail.gmail.com>
 <CADqi7965MTn=-+u7jr5W8gsDDeDVJ3Ds=a5C+pPgT_yCqznwhA@mail.gmail.com>
 <CAGECzQSPx+hv_XjYD9ktofGi1ru2fxTjJTA9uGacwzdYAp1sbg@mail.gmail.com>
 <CADqi7970HqAOMAUEEy=cbaKDcKO0hWQwkvwYfy9reEcuaWXEGA@mail.gmail.com>
Message-ID: <CAGECzQQLaczs+k4EcT_byiOpFvn7C7XrQs90Ori9UVcnUCS+Rg@mail.gmail.com>

On 24 November 2015 at 10:14, Michael Selik <mike at selik.org> wrote:

> As I think of people's reactions to seeing & and | for the first time, the
> typical response is, "What do you mean by bitwise?" Not, "Why are you using
> bitwise operators for non-bitwise operations?" Interestingly, no one in
> this thread seems to have a problem with ``&`` and ``|`` for set
> intersection and union. The primary complaint is that NumPy users
> instinctively reach for ``and``/``or`` and then forget the operator
> precedence of ``&``/``|``.
>


Using pipe and ampersand looks readable. The use of bitwise operators for
> overloading and/or/not seems standard for many objects, in standard library
> and major projects. In fact, I use those operators as logical far more
> often than as bitwise. I'd bet a great number of NumPy users in the science
> community are completely unaware of their bitwise effects.
>
> I picked ``print`` as an example to show that requiring parens is fine. I
> picked string interpolation as an example because the ``.format`` solution
> also accepted parens as necessary. The rationale for f-string syntax (as
> written in the PEP) does not complain about parens but instead mentions
> that they will be necessary inside the string in several circumstances,
> like lambdas.
>
> That precedence issue doesn't cause problems when teaching, in my
> experience. People accept readily that there's issues with operator
> precedence. On the other hand, they have trouble reading code that uses an
> older style (perhaps using ``&``) instead of the newer style that they
> learned (the hypothetical ``&&``). Even the switch between ``%``
> interpolation and ``.format`` is still causing problems in large
> organizations.
>
> I get your points that it is bad to confuse people with even more and/or
operators they have to learn. But I think it's weird that you are saying
that people accept the wrong operator precedence readily, even though
before you said you do it wrong yourself constantly. This is also why I
think the set intersection and union are not a problem is because the
operator precedence is correct there.

Your point that the problem is mostly that the old code will still be usin
the ``&`` operator, which could confuse people is true. But I also think
that would eventually disappear, which would make Python better in the
future.

Another thing is that it seems some people are worried about the form of
the new operators. Some options (which don't use the C style &&, because
that would indeed cause confusion):
.and.
*and*
+and+
%and%
@and@
?and?
^and^
<and>
{and}
(and)
[and]
|and|
:and:
_and_ (could currently be a variablename)
en (the Dutch version)

Some of these could also be used with just one special character, like
@and, but I think the surrounded ones look more visually pleasing. These
are just a couple of examples and some of them seem fine to me. I do think
that it is important though to not focus on the form already. It seems
better to first figure out if the new operators would not confuse the
newcomers to much in whatever form they come.


On 24 November 2015 at 10:14, Michael Selik <mike at selik.org> wrote:

> As I think of people's reactions to seeing & and | for the first time, the
> typical response is, "What do you mean by bitwise?" Not, "Why are you using
> bitwise operators for non-bitwise operations?" Interestingly, no one in
> this thread seems to have a problem with ``&`` and ``|`` for set
> intersection and union. The primary complaint is that NumPy users
> instinctively reach for ``and``/``or`` and then forget the operator
> precedence of ``&``/``|``.
>
>
> On Mon, Nov 23, 2015 at 9:09 PM Jelte Fennema <me at jeltef.nl> wrote:
>
>> On 24 November 2015 at 02:44, Michael Selik <mike at selik.org> wrote:
>>
>>> Why hasn't SQLAlchemy gone the route of NumPy with overloaded operators?
>>>
>> It seems I was wrong about that, they apparently do:
>> http://stackoverflow.com/a/14185275/2570866
>>
>
> Using pipe and ampersand looks readable. The use of bitwise operators for
> overloading and/or/not seems standard for many objects, in standard library
> and major projects. In fact, I use those operators as logical far more
> often than as bitwise. I'd bet a great number of NumPy users in the science
> community are completely unaware of their bitwise effects.
>
>
> I'd put that in the category of parentheses tax along with the print
>>> function and old style % string interpolation.
>>
>> This seems like a bit of a weird argument since the parentheses for the
>> print function are put there for a reason (see PEP3105) and the old style %
>> string interpolation will be replaced by the new format string literal.
>>
>
> I picked ``print`` as an example to show that requiring parens is fine. I
> picked string interpolation as an example because the ``.format`` solution
> also accepted parens as necessary. The rationale for f-string syntax (as
> written in the PEP) does not complain about parens but instead mentions
> that they will be necessary inside the string in several circumstances,
> like lambdas.
>
> Adding yet another operator would make Python harder to learn and read.
>>> Even if you advertise a new operator, many libraries will be slow to change
>>> and we'll have 3 different techniques to teach.
>>
>> I don't think that much confusion will arise, since the normal way is to
>> to use is the short `and` version. Only in libraries where it would be
>> explicitly told the new operator would be used. It would also do away with
>> the confusion about why and cannot be overridden and why the precedence of
>> the & operator is "wrong".
>>
>
> That precedence issue doesn't cause problems when teaching, in my
> experience. People accept readily that there's issues with operator
> precedence. On the other hand, they have trouble reading code that uses an
> older style (perhaps using ``&``) instead of the newer style that they
> learned (the hypothetical ``&&``). Even the switch between ``%``
> interpolation and ``.format`` is still causing problems in large
> organizations.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/c0fffde5/attachment-0001.html>

From me at jeltef.nl  Tue Nov 24 05:10:54 2015
From: me at jeltef.nl (Jelte Fennema)
Date: Tue, 24 Nov 2015 11:10:54 +0100
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <7CEF161C-557F-4AF1-ADF8-FD83C301ED2D@yahoo.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
 <5653EDA6.5020903@canterbury.ac.nz>
 <7CEF161C-557F-4AF1-ADF8-FD83C301ED2D@yahoo.com>
Message-ID: <CAGECzQTzKg+KwkevWfefBx4RFvkAD_E_RZqzsc4ahLtmxgoh3A@mail.gmail.com>

On 24 November 2015 at 06:35, Andrew Barnert via Python-ideas <
python-ideas at python.org> wrote:

> -0 on actually adding a general custom-infix syntax, come to think of it.
> Any solution to the problems above should work just as well here. And it
> means that in the future, libraries don't have to cram things into
> inappropriate symbols or be stuck with prefix or dot-method notation. And I
> don't think it would be any harder to learn than a few special cases. And I
> think ".between." would be just as useful in an ORM as ".in.". More
> generally, the whole point of "@" was that everyone agreed that it was the
> only new operator anyone would need (except maybe "@@") for a decade or
> two; if that's not true, infinity is a better number than 4.


This seems like an interesting idea, but I think it would be hard finding a
notation that does not conflict with current operations. For the and/or
almost any symbol would work, since they are keywords and the operators
have no meaning on them, but if any name can be used operations on
variables will become unclear. So another symbol would probably have to be
used that isn't used yet, like the $ symbol. Also, I think there are
probably some other issues that I haven't thought of, since this would add
a pretty big language feature.

PS. I didn't include the dollar convention in my last options list for the
new and operator, but it is ofcourse also a possibility: $and$, or maybe
$and.

On 24 November 2015 at 06:35, Andrew Barnert via Python-ideas <
python-ideas at python.org> wrote:

> (Top-posting because this is really a reply to a combination of three
> earlier posts, not to Greg Ewing's post, except at the very end.)
>
> -1 on "&&" and "||". To anyone familiar with C and friends, it seems like
> they ought to be like C (short-circuiting, and more generally the same
> thing we already spell "and" and "or"); to anyone else, it would make no
> sense for them to have different precedence than "&" and "|".
>
> -0.5 on ".and." and ".or." for multiple reasons, most of which apply just
> as well to anything similar:
>
> * That looks like a general custom-infix syntax. Anyone coming to Python
> (including Python 2 users) will expect that they can just as easily define
> ".spam." (and then be disappointed that they can't...), or will be worried
> that 3.7 will add a bunch of new dot operators they'll have to learn. Not a
> _huge_ negative, but already enough to turn me off the idea.
>
> * What would you call the dunder methods (and operator module functions),
> and how would you deal with the fact that any novice/transplant is going to
> assume "__and__" means ".and." rather than "&"?
>
> * Operators starting with "." are ambiguous with float literals and/or
> attribute access ("spam.and.eggs" looks like a member of a member of spam),
> at least to humans, if not to the parser.
>
> * Operators starting and/or ending with "." are hard to talk about because
> of natural-language punctuation (and even harder on a mobile keyboard, or
> an overly clever text editor or word processor).
>
> -0 on actually adding a general custom-infix syntax, come to think of it.
> Any solution to the problems above should work just as well here. And it
> means that in the future, libraries don't have to cram things into
> inappropriate symbols or be stuck with prefix or dot-method notation. And I
> don't think it would be any harder to learn than a few special cases. And I
> think ".between." would be just as useful in an ORM as ".in.". More
> generally, the whole point of "@" was that everyone agreed that it was the
> only new operator anyone would need (except maybe "@@") for a decade or
> two; if that's not true, infinity is a better number than 4.
>
> +1 on the idea if someone can come up with a good spelling that avoids all
> the above problems and reads as naturally as "@".
>
> > On Nov 23, 2015, at 20:55, Greg Ewing <greg.ewing at canterbury.ac.nz>
> wrote:
> >
> > Bruce Leban wrote:
> >> If this idea were to fly, a better name would be something that doesn't
> have that problem, e.g., .and. .or. .not.
> >
> > Dots are a bit on the ugly side.
> >
> > Some other possibilities:
> >
> > AND, OR, NOT
> >
> > And, Or, Not
> >
> > en, of, nlet
>
> Is that last one supposed to be "niet"? When I try "nlet", Google assumes
> it's a typo for "net", which I guess could be a unary boolean identity
> operator, but I don't think we need that. :)
>
> Anyway, that pattern is a bit hard to extend to an overloadable "in"
> operator, because Dutch for "in" is "in".
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/7199f23b/attachment.html>

From mike at selik.org  Tue Nov 24 05:36:42 2015
From: mike at selik.org (Michael Selik)
Date: Tue, 24 Nov 2015 10:36:42 +0000
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAGECzQQLaczs+k4EcT_byiOpFvn7C7XrQs90Ori9UVcnUCS+Rg@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
 <CAGECzQQTAa+oOvEsfSO0OK96TYdcJSQaVsmb2RQ-xaqrtdXjeA@mail.gmail.com>
 <CADqi7965MTn=-+u7jr5W8gsDDeDVJ3Ds=a5C+pPgT_yCqznwhA@mail.gmail.com>
 <CAGECzQSPx+hv_XjYD9ktofGi1ru2fxTjJTA9uGacwzdYAp1sbg@mail.gmail.com>
 <CADqi7970HqAOMAUEEy=cbaKDcKO0hWQwkvwYfy9reEcuaWXEGA@mail.gmail.com>
 <CAGECzQQLaczs+k4EcT_byiOpFvn7C7XrQs90Ori9UVcnUCS+Rg@mail.gmail.com>
Message-ID: <CADqi797u-hYG1uh4GwYJ8czhG8Dksa+S9cNvasOUwXtMds0W=A@mail.gmail.com>

On Tue, Nov 24, 2015 at 5:00 AM Jelte Fennema <me at jeltef.nl> wrote:

> On 24 November 2015 at 10:14, Michael Selik <mike at selik.org> wrote:
>
>> As I think of people's reactions to seeing & and | for the first time,
>> the typical response is, "What do you mean by bitwise?" Not, "Why are you
>> using bitwise operators for non-bitwise operations?" Interestingly, no one
>> in this thread seems to have a problem with ``&`` and ``|`` for set
>> intersection and union. The primary complaint is that NumPy users
>> instinctively reach for ``and``/``or`` and then forget the operator
>> precedence of ``&``/``|``.
>>
>

> I get your points that it is bad to confuse people with even more and/or
> operators they have to learn. But I think it's weird that you are saying
> that people accept the wrong operator precedence readily, even though
> before you said you do it wrong yourself constantly.
>

I make that mistake in an interactive environment and fix it moments later,
so it's not a big thing for me. I also occasionally forget to put a colon
at the end of my for-loops, etc. ;-)


> This is also why I think the set intersection and union are not a problem
> is because the operator precedence is correct there.
>
> Your point that the problem is mostly that the old code will still be usin
> the ``&`` operator, which could confuse people is true. But I also think
> that would eventually disappear, which would make Python better in the
> future.
>

What's the half-life of deprecated code? That stuff is like nuclear waste.
Or more like a bacteria you're spot-treating with antibiotic. It keeps
replicating while you apply ointment and might evolve a resistance. Ok, ok,
I'm getting a little too colorful there.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/e3f9a462/attachment.html>

From random832 at fastmail.com  Tue Nov 24 10:11:45 2015
From: random832 at fastmail.com (Random832)
Date: Tue, 24 Nov 2015 15:11:45 +0000 (UTC)
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org>
 <CAPTjJmot3sCFo0xjTQoELGKW0dbvkhkmDZDZNnTxZ2Rqg4-WGQ@mail.gmail.com>
Message-ID: <n31unh$vj7$1@ger.gmane.org>

On 2015-11-24, Chris Angelico wrote:
> The Python semantics are defined more tightly than that, though - the
> "else" clause in each case would simply be "b". Changing that is not
> something you can do with operator overloading, so it would mean a
> fundamental change in the operator's semantics. And once you do that,
> you end up with a completely different operator.

I don't see that as "fundamental". Certainly it can't _actually_
be a & b, but it could certainly be "a __foo__ b" where the
default implementation of the __foo__ method (on object) simply
returns b.

If that's "fundamental" then adding overloading to _any_ operator
that didn't support it in Python 1.0 violates those operators'
"fundamental" behavior of raising a TypeError when applied to
types they did not support in Python 1.0.

This would also allow b.__rfoo__, to address the objection
Nathaniel Smith raised, of the right-hand-side not being able to
override. It can't override "does it short-circuit", since
that'd defeat the point of short-circuiting, but it can override
the value in the case where it doesn't, and a well-behaved
'elementwise-"and"able' type would work either way, only
short-circuiting if every element of the left side is false (or
true for "or").


From chris.barker at noaa.gov  Tue Nov 24 12:31:30 2015
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 24 Nov 2015 09:31:30 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CADqi797u-hYG1uh4GwYJ8czhG8Dksa+S9cNvasOUwXtMds0W=A@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
 <CAGECzQQTAa+oOvEsfSO0OK96TYdcJSQaVsmb2RQ-xaqrtdXjeA@mail.gmail.com>
 <CADqi7965MTn=-+u7jr5W8gsDDeDVJ3Ds=a5C+pPgT_yCqznwhA@mail.gmail.com>
 <CAGECzQSPx+hv_XjYD9ktofGi1ru2fxTjJTA9uGacwzdYAp1sbg@mail.gmail.com>
 <CADqi7970HqAOMAUEEy=cbaKDcKO0hWQwkvwYfy9reEcuaWXEGA@mail.gmail.com>
 <CAGECzQQLaczs+k4EcT_byiOpFvn7C7XrQs90Ori9UVcnUCS+Rg@mail.gmail.com>
 <CADqi797u-hYG1uh4GwYJ8czhG8Dksa+S9cNvasOUwXtMds0W=A@mail.gmail.com>
Message-ID: <CALGmxEKsEKPt4ZpePfemXhCytZYEp23c2S81HfrH6p+7T5OQ5A@mail.gmail.com>

>
> As I think of people's reactions to seeing & and | for the first time, the
>>> typical response is, "What do you mean by bitwise?" Not, "Why are you using
>>> bitwise operators for non-bitwise operations?"
>>>
>>
agreed -- but unfortunately, numpy may be one place where people really do
want bitwise operations sometimes -- and it's way too late now to re-define
them anyway.


> Interestingly, no one in this thread seems to have a problem with ``&``
>>> and ``|`` for set intersection and union.
>>>
>>
I'd say that's because for sets, it would have been a very rare case to do
bitwise operations -- I'm sure I lack imagination, but I can't think of
even one case where it would make sense. So why not re-use them?

The primary complaint is that NumPy users instinctively reach for
>>> ``and``/``or`` and then forget the operator precedence of ``&``/``|``.
>>>
>>
well, that is an issue, though I think the fact that you can only use & |
when you want  "and" "or" when it happens to make sense -- i.e. for boolean
arrays, is the bigger problem. Also, newbies will either not think to use
them, or will see them in code and think they ARE another way to spell and
and or.

It also doesn't allow element-wise short circuiting behavior. you get:

In [*2*]: np.array([0,1,2,3]) &  np.array([3,2,1,0])

Out[*2*]: array([0, 0, 0, 0])
when you want:

In [*3*]: [ a and b for a,b in zip([0,1,2,3], [3,2,1,0])]

Out[*3*]: [0, 2, 1, 0]
oh, and this:

In [*4*]: np.array([0,1,2,3]) |  np.array([3,2,1,0])

Out[*4*]: array([3, 3, 3, 3])
would surely confuse people that don't "get" what bitwise means, even
though it results in teh same thing in a boolean context:

In [*5*]: (np.array([0,1,2,3]) |  np.array([3,2,1,0])).astype(np.bool)

Out[*5*]: array([ True,  True,  True,  True], dtype=bool)

Honestly, I'm not sure element-wise short circuiting is desired, but it
would be nice to be consistent about that.

As for precedence, it's a annoying, but at least in my experience, the very
first test case fails, and then I remember to add the parens.

As for the more and more operators issue -- for the most part, this would
be used for special purposes -- most people wouldn't even notice there were
there, and numpy users might well think it' s special numpy thing (same for
SQLAlchemy folks, or...)

Unless folks wanted to add support for element-wise and and or to the
built-in sequence types. I don't think anyone has proposed that.

Your point that the problem is mostly that the old code will still be using
>> the ``&`` operator, which could confuse people is true.
>>
>
well, yes - but code using & would continue to work the same way -- and
anyone confused would, in fact, have been mis-interpreting what & meant
already :-)

As for element-wise everything -- THAT discussion has happened multiple
times in the past, and been rejected. Time to go dig into the archives --
not sure if there is rejected PEP for it.


-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/be39fc85/attachment.html>

From brenbarn at brenbarn.net  Tue Nov 24 13:12:43 2015
From: brenbarn at brenbarn.net (Brendan Barnwell)
Date: Tue, 24 Nov 2015 10:12:43 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <n310dc$92q$1@ger.gmane.org>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org>
Message-ID: <5654A89B.40607@brenbarn.net>

On 2015-11-23 22:34, Random832 wrote:
> On 2015-11-24, Chris Angelico<rosuav at gmail.com>  wrote:
>> >I think it's reasonable, except for the potential confusion of having
>> >*three*  "and" operators. The one with the word is never going to
>> >change - its semantics demand that it not be overridable.
> How? Are you referring to the short-circuit? C# allows
> overloading the short-circuit operators by doing it in two
> parts - in pseudocode:
>
> a and b: a if a.__false__() else a & b
> a or b: a if a.__true__() else a | b
>
> There's no fundamental reason short-circuiting "demands" that it
> not be overridable, just because C++ can't do it.

	The problem is that this kind of overriding doesn't handle the main use
case, which is elementwise and-ing/or-ing.  If a is some numpy-like
array of [1, 1], then it may be boolean true, but you still want "a
magic_or b" to do the elementwise operation, not just return "a" by
itself.  (I say "numpy-like" because a real numpy array will fail here
precisely because its boolean truth value is undefined, so it raises an
error.)

	The problem is that, with the existing and/or, the issue of the boolean
truth values of the operands is entangled with the actual and/or
operation.  For an elementwise operation, you *always* want to do it
elementwise, regardless of whether the operands "count" as boolean false
in some other situation.

-- 
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."
    --author unknown

From guido at python.org  Tue Nov 24 13:20:09 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 24 Nov 2015 10:20:09 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <5654A89B.40607@brenbarn.net>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
Message-ID: <CAP7+vJ+rPPzy-BX1q+oATgKJyBfTLaVv9m-1hwceQs4DH14_Tg@mail.gmail.com>

To everyone claiming that you can't overload and/or because they are
shortcut operators, please re-read PEP 335. It provides a clean
solution -- it was rejected because it adds an extra byte code to all
code using those operators (the majority of which don't need it).

-- 
--Guido van Rossum (python.org/~guido)

From abarnert at yahoo.com  Tue Nov 24 13:15:43 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 24 Nov 2015 10:15:43 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAGECzQTzKg+KwkevWfefBx4RFvkAD_E_RZqzsc4ahLtmxgoh3A@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAGu0AnuVF38pueqYgj89HcQrvE+YPB7ZHY9vt89UvyjsoeUdJg@mail.gmail.com>
 <5653EDA6.5020903@canterbury.ac.nz>
 <7CEF161C-557F-4AF1-ADF8-FD83C301ED2D@yahoo.com>
 <CAGECzQTzKg+KwkevWfefBx4RFvkAD_E_RZqzsc4ahLtmxgoh3A@mail.gmail.com>
Message-ID: <B22F3E0B-E91B-46A6-8463-0970BF7A00A9@yahoo.com>

On Nov 24, 2015, at 02:10, Jelte Fennema <me at jeltef.nl> wrote:
> 
>> On 24 November 2015 at 06:35, Andrew Barnert via Python-ideas <python-ideas at python.org> wrote: 
>> -0 on actually adding a general custom-infix syntax, come to think of it. Any solution to the problems above should work just as well here. And it means that in the future, libraries don't have to cram things into inappropriate symbols or be stuck with prefix or dot-method notation. And I don't think it would be any harder to learn than a few special cases. And I think ".between." would be just as useful in an ORM as ".in.". More generally, the whole point of "@" was that everyone agreed that it was the only new operator anyone would need (except maybe "@@") for a decade or two; if that's not true, infinity is a better number than 4.
> 
> This seems like an interesting idea, but I think it would be hard finding a notation that does not conflict with current operations.

The obvious notation is "a `spam` b", as used in Haskell and various other languages.

It doesn't conflict with current operations. It doesn't look like a pair of operators, but like a matched bracketing or quoting, which is exactly what we'd want. It has just about the right screen density (. can be easy to miss, while $ is so heavy that it makes the "in" in $in$ harder to read). It doesn't conflict with a different meaning elsewhere, like $(in), which can look like part of string template to a human reader.

Also, experience with those other languages shows that it works. In particular, in Haskell, I could always just define any new string of symbols to call spam on its operands, and with any precedence and associativity I want--but it's often much more readable to just use `spam` instead.

Its biggest problem is that Guido hates backticks, and is on record as promising that they will never be reused in Python now that they no longer mean repr. Since I don't think he'd accept the idea anyway (IIRC, last time it came up, he said it wasn't even worth writing a PEP to categorically dismiss), that makes this the perfect syntax for it. :)

More seriously, my point that infinity is a better number than 4 is more an argument against adding custom and, or, not , and in operators than an argument for adding custom arbitrary operators, which means the spelling isn't that important anyway. I don't think you're going to do much better than `and`, and I don't think that's good enough for Python.

> For the and/or almost any symbol would work, since they are keywords and the operators have no meaning on them,

No, because code has to be readable and parseable by humans, not just by compilers. Even if the compiler can tell that "spam.and.eggs" or "2.in.e" aren't using dots for member access or float purposes, that isn't clear to a human reader until you look at it carefully.

(Of course you can always use `and` or some other syntax that wouldn't be confusing even if and weren't a keyword, but at that point you're back to "no better than custom infixes".)

Also, bear in mind that what you're competing with is a.spam(b), so anything that looks too heavyweight or too weird, like a $(spam) b, isn't going to be much improvement except in very long expressions where closing the parens gets troublesome (which often aren't going to be readable anyway, and maybe it's not so terrible to encourage people to break them up and name subparts).

> but if any name can be used operations on variables will become unclear. So another symbol would probably have to be used that isn't used yet, like the $ symbol. Also, I think there are probably some other issues that I haven't thought of, since this would add a pretty big language feature.

Actually, last time I looked into it (around 3.4), it didn't have any other big issues or complex interactions with other features (and I don't think static typing, await, or any other changes will affect that, but I'd have to look more carefully). And it's pretty simple to implement, and pretty easy to describe. If you're interested, I may have written up a blog post, and if I didn't, I could write one now. (I think I also have a hack that implements it with a retokenizing import hook, if you want to play with it.)

While we're at it, every time custom infix operators come up, someone points out that you can already fake it pretty well. For example, "a &And& b" just needs a global named "And" whose __rand__(self, a) returns an object whose __and__(self, b) does what you want. And this allows you to use whatever precedence you want by picking the surrounding operators (much like Swift's custom infix operators get their precedence from the first symbol in the name), and lets you implement operator sectioning (you can make __rand__(self, a).__call__ the same as its __and__, and then you can pass around "a &And" as a partial function), and so on. There are a few glitches, but it works well with the kind of code you'd write with an ORM. And again, this is just as true for a proposal to add four custom operators as for a proposal to add a general feature. So, you have to think about why sqlanywhere, numpy, etc. have decided not to use this trick so far, when it's been well known since around the days of Python 2.4, and why they'd be better off with custom operators that don't look that different from the fake ones.

> PS. I didn't include the dollar convention in my last options list for the new and operator, but it is ofcourse also a possibility: $and$, or maybe $and. 
> 
>> On 24 November 2015 at 06:35, Andrew Barnert via Python-ideas <python-ideas at python.org> wrote:
>> (Top-posting because this is really a reply to a combination of three earlier posts, not to Greg Ewing's post, except at the very end.)
>> 
>> -1 on "&&" and "||". To anyone familiar with C and friends, it seems like they ought to be like C (short-circuiting, and more generally the same thing we already spell "and" and "or"); to anyone else, it would make no sense for them to have different precedence than "&" and "|".
>> 
>> -0.5 on ".and." and ".or." for multiple reasons, most of which apply just as well to anything similar:
>> 
>> * That looks like a general custom-infix syntax. Anyone coming to Python (including Python 2 users) will expect that they can just as easily define ".spam." (and then be disappointed that they can't...), or will be worried that 3.7 will add a bunch of new dot operators they'll have to learn. Not a _huge_ negative, but already enough to turn me off the idea.
>> 
>> * What would you call the dunder methods (and operator module functions), and how would you deal with the fact that any novice/transplant is going to assume "__and__" means ".and." rather than "&"?
>> 
>> * Operators starting with "." are ambiguous with float literals and/or attribute access ("spam.and.eggs" looks like a member of a member of spam), at least to humans, if not to the parser.
>> 
>> * Operators starting and/or ending with "." are hard to talk about because of natural-language punctuation (and even harder on a mobile keyboard, or an overly clever text editor or word processor).
>> 
>> -0 on actually adding a general custom-infix syntax, come to think of it. Any solution to the problems above should work just as well here. And it means that in the future, libraries don't have to cram things into inappropriate symbols or be stuck with prefix or dot-method notation. And I don't think it would be any harder to learn than a few special cases. And I think ".between." would be just as useful in an ORM as ".in.". More generally, the whole point of "@" was that everyone agreed that it was the only new operator anyone would need (except maybe "@@") for a decade or two; if that's not true, infinity is a better number than 4.
>> 
>> +1 on the idea if someone can come up with a good spelling that avoids all the above problems and reads as naturally as "@".
>> 
>> > On Nov 23, 2015, at 20:55, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>> >
>> > Bruce Leban wrote:
>> >> If this idea were to fly, a better name would be something that doesn't have that problem, e.g., .and. .or. .not.
>> >
>> > Dots are a bit on the ugly side.
>> >
>> > Some other possibilities:
>> >
>> > AND, OR, NOT
>> >
>> > And, Or, Not
>> >
>> > en, of, nlet
>> 
>> Is that last one supposed to be "niet"? When I try "nlet", Google assumes it's a typo for "net", which I guess could be a unary boolean identity operator, but I don't think we need that. :)
>> 
>> Anyway, that pattern is a bit hard to extend to an overloadable "in" operator, because Dutch for "in" is "in".
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>> 
>> --
>> 
>> ---
>> You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/5bEW_wwNJcM/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe at googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/42064fe2/attachment-0001.html>

From brenbarn at brenbarn.net  Tue Nov 24 01:41:45 2015
From: brenbarn at brenbarn.net (Brendan Barnwell)
Date: Mon, 23 Nov 2015 22:41:45 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <n310dc$92q$1@ger.gmane.org>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org>
Message-ID: <565406A9.90605@brenbarn.net>

On 2015-11-23 22:34, Random832 wrote:
> On 2015-11-24, Chris Angelico <rosuav at gmail.com> wrote:
>> I think it's reasonable, except for the potential confusion of having
>> *three* "and" operators. The one with the word is never going to
>> change - its semantics demand that it not be overridable.
>
> How? Are you referring to the short-circuit? C# allows
> overloading the short-circuit operators by doing it in two
> parts - in pseudocode:
>
> a and b: a if a.__false__() else a & b
> a or b: a if a.__true__() else a | b
>
> There's no fundamental reason short-circuiting "demands" that it
> not be overridable, just because C++ can't do it.

	The problem is that this kind of overriding doesn't handle the main use 
case, which is elementwise and-ing/or-ing.  If a is some numpy-like 
array of [1, 1], then it may be boolean true, but you still want "a 
magic_or b" to do the elementwise operation, not just return "a" by 
itself.  (I say "numpy-like" because a real numpy array will fail here 
precisely because its boolean truth value is undefined, so it raises an 
error.)

	The problem is that, with the existing and/or, the issue of the boolean 
truth values of the operands is entangled with the actual and/or 
operation.  For an elementwise operation, you *always* want to do it 
elementwise, regardless of whether the operands "count" as boolean false 
in some other situation.

-- 
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."
    --author unknown

From random832 at fastmail.com  Tue Nov 24 14:07:35 2015
From: random832 at fastmail.com (Random832)
Date: Tue, 24 Nov 2015 19:07:35 +0000 (UTC)
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
Message-ID: <n32chn$2fq$2@ger.gmane.org>

On 2015-11-24, Brendan Barnwell wrote:
> 	The problem is that this kind of overriding doesn't handle the main use
> case, which is elementwise and-ing/or-ing.  If a is some numpy-like
> array of [1, 1], then it may be boolean true, but you still want "a
> magic_or b" to do the elementwise operation, not just return "a" by
> itself.

Why? I am asking specifically because *all* elements of a are true, so all
elements of the result will also be true, and taken from a. Clearly [0, 1]
should *not* return true for a.__true__(), which is why I didn't use __bool__.
But the elementwise operation [1, 1] magic_or [0, 2] returns [1, 1].


From brenbarn at brenbarn.net  Tue Nov 24 14:27:51 2015
From: brenbarn at brenbarn.net (Brendan Barnwell)
Date: Tue, 24 Nov 2015 11:27:51 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <n32chn$2fq$2@ger.gmane.org>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
 <n32chn$2fq$2@ger.gmane.org>
Message-ID: <5654BA37.8080208@brenbarn.net>

On 2015-11-24 11:07, Random832 wrote:
> On 2015-11-24, Brendan Barnwell wrote:
>> 	The problem is that this kind of overriding doesn't handle the main use
>> case, which is elementwise and-ing/or-ing.  If a is some numpy-like
>> array of [1, 1], then it may be boolean true, but you still want "a
>> magic_or b" to do the elementwise operation, not just return "a" by
>> itself.
>
> Why? I am asking specifically because *all* elements of a are true, so all
> elements of the result will also be true, and taken from a. Clearly [0, 1]
> should *not* return true for a.__true__(), which is why I didn't use __bool__.
> But the elementwise operation [1, 1] magic_or [0, 2] returns [1, 1].

	I guess I'm not understanding what you mean by __true__ then.  What is 
this __true__ for which ([0, 1]).__true__() is not true?

-- 
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."
    --author unknown

From njs at pobox.com  Tue Nov 24 14:37:41 2015
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 24 Nov 2015 11:37:41 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAP7+vJ+rPPzy-BX1q+oATgKJyBfTLaVv9m-1hwceQs4DH14_Tg@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
 <CAP7+vJ+rPPzy-BX1q+oATgKJyBfTLaVv9m-1hwceQs4DH14_Tg@mail.gmail.com>
Message-ID: <CAPJVwBn7A14=BtAK6BKGvt_eT8yMekont0uVjNnuOEUNPKmwuQ@mail.gmail.com>

On Nov 24, 2015 10:21 AM, "Guido van Rossum" <guido at python.org> wrote:
>
> To everyone claiming that you can't overload and/or because they are
> shortcut operators, please re-read PEP 335. It provides a clean
> solution -- it was rejected because it adds an extra byte code to all
> code using those operators (the majority of which don't need it).

The semantic objection that I raised -- short circuiting means that you
can't correctly overload 'True and numpy_array', because unlike all other
binops the overload must be defined on the left hand argument -- does apply
to PEP 335 AFAICT. This problem is IMHO serious enough that even if PEP 335
were accepted today I'm not entirely sure that numpy would actually
implement the overloads due to the headaches it would cause for teaching
and code review -- we'd have to have some debate about it at least.

(Possibly useful analogy: Having to always double check that the array
argument appears on the left rather than the right is kinda like how the
old 'a and b or c' idiom forces you to constantly keep an extra special
rule in the back of your head and always double check that b cannot be
falsey whenever you read or write it, which IIUC was one of the major
reasons why it was considered an insufficient substitute for a real ternary
operator.)

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/e7188569/attachment.html>

From guido at python.org  Tue Nov 24 14:51:46 2015
From: guido at python.org (Guido van Rossum)
Date: Tue, 24 Nov 2015 11:51:46 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAPJVwBn7A14=BtAK6BKGvt_eT8yMekont0uVjNnuOEUNPKmwuQ@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
 <CAP7+vJ+rPPzy-BX1q+oATgKJyBfTLaVv9m-1hwceQs4DH14_Tg@mail.gmail.com>
 <CAPJVwBn7A14=BtAK6BKGvt_eT8yMekont0uVjNnuOEUNPKmwuQ@mail.gmail.com>
Message-ID: <CAP7+vJL-Tm9Z00sfG3nH57T+AkQm3Uv-Z_eJ=nagzaRDBADq8A@mail.gmail.com>

On Tue, Nov 24, 2015 at 11:37 AM, Nathaniel Smith <njs at pobox.com> wrote:
> On Nov 24, 2015 10:21 AM, "Guido van Rossum" <guido at python.org> wrote:
>>
>> To everyone claiming that you can't overload and/or because they are
>> shortcut operators, please re-read PEP 335. It provides a clean
>> solution -- it was rejected because it adds an extra byte code to all
>> code using those operators (the majority of which don't need it).
>
> The semantic objection that I raised -- short circuiting means that you
> can't correctly overload 'True and numpy_array', because unlike all other
> binops the overload must be defined on the left hand argument -- does apply
> to PEP 335 AFAICT. This problem is IMHO serious enough that even if PEP 335
> were accepted today I'm not entirely sure that numpy would actually
> implement the overloads due to the headaches it would cause for teaching and
> code review -- we'd have to have some debate about it at least.

OK, that's useful feedback. Is NumPy interested in coming up with an
alternative that works, or are you fine with the status quo?

-- 
--Guido van Rossum (python.org/~guido)

From greg.ewing at canterbury.ac.nz  Tue Nov 24 16:24:07 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 25 Nov 2015 10:24:07 +1300
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <565406A9.90605@brenbarn.net>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <565406A9.90605@brenbarn.net>
Message-ID: <5654D577.6060601@canterbury.ac.nz>

Brendan Barnwell wrote:
> On 2015-11-23 22:34, Random832 wrote:
> 
 >> C# allows overloading the short-circuit operators
>>
>> a and b: a if a.__false__() else a & b
>> a or b: a if a.__true__() else a | b
>>
>     The problem is that this kind of overriding doesn't handle the main 
> use case, which is elementwise and-ing/or-ing.

Maybe we can hook into __bool__ somehow, though?

Suppose 'a and b' were treated as:

    try:
       result = bool(a)
    except IDoNotShortCircuit:
       result = a.__logical_and___(b)
    else:
       if not result:
          result = b

Since a __bool__ call is required anyway, this
shouldn't slow down the case where there is no
overriding.

-- 
Greg

From mike at selik.org  Tue Nov 24 16:42:24 2015
From: mike at selik.org (Michael Selik)
Date: Tue, 24 Nov 2015 21:42:24 +0000
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <5654D577.6060601@canterbury.ac.nz>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <565406A9.90605@brenbarn.net>
 <5654D577.6060601@canterbury.ac.nz>
Message-ID: <CADqi7962TBXXKfMwvEckkE98hBA4oepsuafFJJ1ZNyBf0jXguw@mail.gmail.com>

Interesting. To allow rich logical operation in this manner, we would need
to relax the requirement that ``__bool__`` returns a ``bool``. What
negative effects would that have?

On Tue, Nov 24, 2015 at 4:24 PM Greg Ewing <greg.ewing at canterbury.ac.nz>
wrote:

> Brendan Barnwell wrote:
> > On 2015-11-23 22:34, Random832 wrote:
> >
>  >> C# allows overloading the short-circuit operators
> >>
> >> a and b: a if a.__false__() else a & b
> >> a or b: a if a.__true__() else a | b
> >>
> >     The problem is that this kind of overriding doesn't handle the main
> > use case, which is elementwise and-ing/or-ing.
>
> Maybe we can hook into __bool__ somehow, though?
>
> Suppose 'a and b' were treated as:
>
>     try:
>        result = bool(a)
>     except IDoNotShortCircuit:
>        result = a.__logical_and___(b)
>     else:
>        if not result:
>           result = b
>
> Since a __bool__ call is required anyway, this
> shouldn't slow down the case where there is no
> overriding.
>
> --
> Greg
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151124/81a5236d/attachment.html>

From python at mrabarnett.plus.com  Tue Nov 24 16:47:07 2015
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 24 Nov 2015 21:47:07 +0000
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <5654D577.6060601@canterbury.ac.nz>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <565406A9.90605@brenbarn.net>
 <5654D577.6060601@canterbury.ac.nz>
Message-ID: <5654DADB.7010405@mrabarnett.plus.com>

On 2015-11-24 21:24, Greg Ewing wrote:
> Brendan Barnwell wrote:
>> On 2015-11-23 22:34, Random832 wrote:
>>
>   >> C# allows overloading the short-circuit operators
>>>
>>> a and b: a if a.__false__() else a & b
>>> a or b: a if a.__true__() else a | b
>>>
>>     The problem is that this kind of overriding doesn't handle the main
>> use case, which is elementwise and-ing/or-ing.
>
> Maybe we can hook into __bool__ somehow, though?
>
> Suppose 'a and b' were treated as:
>
>      try:
>         result = bool(a)
>      except IDoNotShortCircuit:
>         result = a.__logical_and___(b)
>      else:
>         if not result:
>            result = b
>
> Since a __bool__ call is required anyway, this
> shouldn't slow down the case where there is no
> overriding.
>
That still doesn't deal with the issue of what should happen if the
order is reversed, e.g. "numpy_array and simple_bool" vs "simple_bool
and numpy_array", where "numpy_array" has the non-shortcircuiting
behaviour but "simple_bool" hasn't.


From greg.ewing at canterbury.ac.nz  Tue Nov 24 17:29:57 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 25 Nov 2015 11:29:57 +1300
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAP7+vJ+rPPzy-BX1q+oATgKJyBfTLaVv9m-1hwceQs4DH14_Tg@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
 <CAP7+vJ+rPPzy-BX1q+oATgKJyBfTLaVv9m-1hwceQs4DH14_Tg@mail.gmail.com>
Message-ID: <5654E4E5.6070206@canterbury.ac.nz>

Guido van Rossum wrote:
> To everyone claiming that you can't overload and/or because they are
> shortcut operators, please re-read PEP 335. It provides a clean
> solution -- it was rejected because it adds an extra byte code to all
> code using those operators (the majority of which don't need it).

Was that the *only* reason for rejection? If that
problem could be solved somehow, would you reconsider?

-- 
Greg


From greg.ewing at canterbury.ac.nz  Tue Nov 24 17:45:45 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 25 Nov 2015 11:45:45 +1300
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <5654DADB.7010405@mrabarnett.plus.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <565406A9.90605@brenbarn.net>
 <5654D577.6060601@canterbury.ac.nz> <5654DADB.7010405@mrabarnett.plus.com>
Message-ID: <5654E899.9090502@canterbury.ac.nz>

MRAB wrote:

> That still doesn't deal with the issue of what should happen if the
> order is reversed, e.g. "numpy_array and simple_bool" vs "simple_bool
> and numpy_array", where "numpy_array" has the non-shortcircuiting
> behaviour but "simple_bool" hasn't.

That's true.

I guess the only solution that really works properly is
to have a second set of operators, or some way of flagging
that you're using non-short-circuiting semantics.

There's one language I've seen -- I think it was Eiffel,
or maybe Ada -- that had two sets of boolean operators.
But it was kind of the other way around: 'and' and 'or'
were non-short-circuiting, and to get short-circuiting
you had to say 'and then' or 'or else'.

-- 
Greg

From rymg19 at gmail.com  Tue Nov 24 18:00:34 2015
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 24 Nov 2015 17:00:34 -0600
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <5654E899.9090502@canterbury.ac.nz>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <565406A9.90605@brenbarn.net>
 <5654D577.6060601@canterbury.ac.nz> <5654DADB.7010405@mrabarnett.plus.com>
 <5654E899.9090502@canterbury.ac.nz>
Message-ID: <F28D6EA9-6715-4C95-A20A-AC7405E9F3B8@gmail.com>



On November 24, 2015 4:45:45 PM CST, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>MRAB wrote:
>
>> That still doesn't deal with the issue of what should happen if the
>> order is reversed, e.g. "numpy_array and simple_bool" vs "simple_bool
>> and numpy_array", where "numpy_array" has the non-shortcircuiting
>> behaviour but "simple_bool" hasn't.
>
>That's true.
>
>I guess the only solution that really works properly is
>to have a second set of operators, or some way of flagging
>that you're using non-short-circuiting semantics.
>
>There's one language I've seen -- I think it was Eiffel,
>or maybe Ada -- that had two sets of boolean operators.
>But it was kind of the other way around: 'and' and 'or'
>were non-short-circuiting, and to get short-circuiting
>you had to say 'and then' or 'or else'.

That was a lot of languages:
- Ada: and then, or else
- Algol: ANDTHEN, ORELSE
- Erlang: andalso, orelse
- Extended Pascal: and_then, or_else
- GNU Pascal: and then, or else
- Oz: andthen, orelse
- Standard ML: andalso, orelse
- Visual Basic: AndAlso, OrElse

Interestingly, Visual Basic allows you to overload non-short-circuiting And. To overload AndAlso, you need to overload And and IsFalse. It seems that AndAlso was roughly equivalent to And + IsFalse:

A AndAlso B

=

If Not (IsFalse A) Then A And B Else A Endif
-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.

From random832 at fastmail.com  Tue Nov 24 18:53:07 2015
From: random832 at fastmail.com (Random832)
Date: Tue, 24 Nov 2015 23:53:07 +0000 (UTC)
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
 <n32chn$2fq$2@ger.gmane.org> <5654BA37.8080208@brenbarn.net>
Message-ID: <n32t93$lun$1@ger.gmane.org>

On 2015-11-24, Brendan Barnwell wrote:
> 	I guess I'm not understanding what you mean by __true__ then.  What is 
> this __true__ for which ([0, 1]).__true__() is not true?

It'd be a new operator / magic-function, so whatever the
person writing the class wants it to be. I specifically
didn't use bool().

def __true__(self):
	# return True ony if you want "or" to short-circuit
	pass


From random832 at fastmail.com  Tue Nov 24 19:00:07 2015
From: random832 at fastmail.com (Random832)
Date: Wed, 25 Nov 2015 00:00:07 +0000 (UTC)
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
 <CAP7+vJ+rPPzy-BX1q+oATgKJyBfTLaVv9m-1hwceQs4DH14_Tg@mail.gmail.com>
 <CAPJVwBn7A14=BtAK6BKGvt_eT8yMekont0uVjNnuOEUNPKmwuQ@mail.gmail.com>
Message-ID: <n32tm6$lun$2@ger.gmane.org>

On 2015-11-24, Nathaniel Smith wrote:
> The semantic objection that I raised -- short circuiting means that you
> can't correctly overload 'True and numpy_array',

Well, you mean "True or" or "False and", but anyway... I've got
to confess, I don't really understand the semantic paradigm under
which it's appropriate to do this all, but it's not appropriate
for it to return True. If you can pass in True here, that implies
that the True you'll get out of this operation is an appropriate
value for using in further operations on similarly-shaped arrays.

What does it _matter_ that you get True (or 1, etc) instead of
[True, True, True, True, True, True]?


From ncoghlan at gmail.com  Wed Nov 25 02:06:04 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 25 Nov 2015 17:06:04 +1000
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <5653FE15.8080402@canterbury.ac.nz>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
Message-ID: <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>

On 24 November 2015 at 16:05, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>>
>> Something that *could* potentially be comprehensible is the idea of
>> allowing "elementwise" logical operators, with a suitable syntactic
>> spelling.
>
> -1 on tying any of this explicity to a notion of
> elementwise operations. The range of potential uses
> is wider than that.

While I'm still largely of the view that introducing additional
operators would make things more confusing rather than less, I'm also
convinced that if anything like this is going to be pursued without
being incredibly confusing for beginners there needs to be a fairly
concise answer to "What are these operators for?".

Take the "bitwise operators", for example. The notion of a "bitwise
operator" is conceptually dense for folks that have never worked with
binary numbers before. Despite that, if someone asks "What do the &
and | operators do in Python?" the semantics can still be conveyed
relatively quickly using some truth table examples like:

>>> bin(0b101 & 0b110)
'0b100'
>>> bin(0b101 & 0b101)
'0b101'
>>> bin(0b101 | 0b110)
'0b111'
>>> bin(0b101 | 0b101)
'0b101'

Explaining "~" fully is a bit trickier (since you would need to
explain why two's complement representations of binary numbers are
useful), but it's possible to avoid that explanation by using the
alternative arithmetic formulation for "~" given in
https://wiki.python.org/moin/BitwiseOperators : "~x == -x -1"

Matrix multiplication is another example of something that isn't
particularly easy to explain to folks that aren't already familiar
with the relevant domain, but also conveys clearly that you can ignore
it if you're not working with matrices.

It's then also useful to remember that the answers to "What is this
for?" and "How is this used?" for a language construct can diverge
over time. The original "What is this for?" use cases are the ones
that guide the design decisions towards concrete answers that define
how the construct works, and provide the underlying rationale for the
way the construct behaves. The "How is this used?" cases then arise
later when folks say "Yes, those existing semantics are suitable for
my current use case, so I can reuse the syntax".

Some specific examples:

"+" is used not only for addition, but also sequence concatenation.
"&" is not only "bitwise and", but also set intersection
"/" is not only division, but also pathlib path joining

NumPy repurposes most of the binary operators (including the bitwise
ones) as element-wise matrix operations. SQL Alchemy repurposes a
number of them for SQL query operations. SymPy changes them from
arithmetic operations to symbolic ones. Those use cases don't change
the answers to "What are these operators for?" from a language design
perspective, they only change the answers to "How are these operators
used?" from a practical perspective.

Getting back to the specific topic of this thread, this could actually
make an interesting usability study for a language design theorist, by
looking at the kinds of mistakes folks make trying to learn
elementwise logic operations in NumPy, and then seeing whether the
introduction of overridable elementwise logical operators reduces the
learning curve.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris.barker at noaa.gov  Wed Nov 25 11:00:37 2015
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Wed, 25 Nov 2015 08:00:37 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <n32tm6$lun$2@ger.gmane.org>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
 <CAP7+vJ+rPPzy-BX1q+oATgKJyBfTLaVv9m-1hwceQs4DH14_Tg@mail.gmail.com>
 <CAPJVwBn7A14=BtAK6BKGvt_eT8yMekont0uVjNnuOEUNPKmwuQ@mail.gmail.com>
 <n32tm6$lun$2@ger.gmane.org>
Message-ID: <-1350841247926098108@unknownmsgid>

> What does it _matter_ that you get True (or 1, etc) instead of
> [True, True, True, True, True, True]?

Sometimes you want to know that everything in an array is True ( which
is common enough that numpy.alltrue() exists)

But other times you want to know which items in an array a true, and
which are not -- most often to use as a Boolean mask -- in which case,
an array that happens to have all true values is a different beast
altogether from a single True. And remember that thee could be arrays
of any number of dimensions.

Also: Python Truthyness rules define an empty container as False, and
any non-empty container as True -- it's probably better not to make
arrays unique in that regard.

-CHB






> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From chris.barker at noaa.gov  Wed Nov 25 18:58:46 2015
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Wed, 25 Nov 2015 15:58:46 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
Message-ID: <3518005034548011247@unknownmsgid>

> While I'm still largely of the view that introducing additional
> operators would make things more confusing rather than less,

Well, almost by definition, more stuff to understand is more confusing
for beginners.

> there needs to be a fairly
> concise answer to "What are these operators for?".

I don't think "they are for doing logical operations on each of the
elements in a sequence, rather than the sequence as a whole", along
with an example or two is particularly challenging.

In fact, much less so than the Bitwise operators, or matrix
multiplication, which require a bit of domain knowledge, as you say.
But those aren't a big problem either: "if you don't know what it
means, you probably don't need it"

But as for general element-wise operators:

IIRC, this was discussed a lot back in the day -- and was driven by
experience with e.g. Matlab, where the regular math operators do
linear algebra by default, and there are alternative "element wise"
operators. Numpy, on the other hand, does element-wise by default, so
we wanted another set for linear algebra. However, we came to realize
that the only one really needed was matrix multiply -- and thus the
new @ operator.

This all worked because Numpy could overload the math operators to be
element wise, and once rich comparisons were implemented, that covered
almost everything. So all that's left is and-or.

Add the fact that use the Bitwise & and | in their place in most
cases, and we've done fine so far.

All that being said -- two more operators for "rich and" and "rich
or". Would nicely complete the picture.

I was just introducing my intro Python class to the magic methods last
night -- there are a LOT of them! Two more is pretty trivial Addison
of complexity.

As long as we can find a way to spell them that is not too confusing
or ugly -- I think it's a win-win.

Note: having worked with array-oriented languages/libraries for a long
time, I'd like element-wise operators that worked with all the
built-in types. But I suspect that Python is never going to go there.
So we only need these two.

-CHB

From ncoghlan at gmail.com  Thu Nov 26 00:44:17 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Nov 2015 15:44:17 +1000
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <3518005034548011247@unknownmsgid>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
Message-ID: <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>

On 26 November 2015 at 09:58, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
>> While I'm still largely of the view that introducing additional
>> operators would make things more confusing rather than less,
>
> Well, almost by definition, more stuff to understand is more confusing
> for beginners.
>
>> there needs to be a fairly
>> concise answer to "What are these operators for?".
>
> I don't think "they are for doing logical operations on each of the
> elements in a sequence, rather than the sequence as a whole", along
> with an example or two is particularly challenging.
>
> In fact, much less so than the Bitwise operators, or matrix
> multiplication, which require a bit of domain knowledge, as you say.
> But those aren't a big problem either: "if you don't know what it
> means, you probably don't need it"

Right, that's why I think "elementwise logical operators (which may
potentially be useful for other things)" is an idea that has some hope
of avoiding creating new barriers to learning:

* if "elementwise" doesn't mean anything to you, and no library you're
using mentions them in its documentation, you can ignore them
* for folks that do know what it means, "elementwise" is evocative of
the relevant semantics for at least the data analysis use case

> But as for general element-wise operators:

I wasn't suggesting those - just element-wise logical operators. In
terms of scoping the use case: bitwise and/or/not work fine for
manipulating existing boolean masks, and converting a single matrix to
a boolean mask involves apply an elementwise function, not elementwise
logical operations.

So elementwise logical operators would presumably be aimed at *data
merging* problems - creating a combined matrix where some values are
taken from matrix A and others from matrix B, based on the truthiness
of those values. I'm not enough of a data analyst to know how common
that problem is, or whether it might be better served by a higher
level "replace_elements" operation that accepts a base array, a
replacement array, and a boolean mask saying which values to replace.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From abarnert at yahoo.com  Thu Nov 26 01:27:19 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 25 Nov 2015 22:27:19 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
Message-ID: <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>

On Nov 25, 2015, at 21:44, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 26 November 2015 at 09:58, Chris Barker - NOAA Federal
> <chris.barker at noaa.gov> wrote:
>>> While I'm still largely of the view that introducing additional
>>> operators would make things more confusing rather than less,
>> 
>> Well, almost by definition, more stuff to understand is more confusing
>> for beginners.
>> 
>>> there needs to be a fairly
>>> concise answer to "What are these operators for?".
>> 
>> I don't think "they are for doing logical operations on each of the
>> elements in a sequence, rather than the sequence as a whole", along
>> with an example or two is particularly challenging.
>> 
>> In fact, much less so than the Bitwise operators, or matrix
>> multiplication, which require a bit of domain knowledge, as you say.
>> But those aren't a big problem either: "if you don't know what it
>> means, you probably don't need it"
> 
> Right, that's why I think "elementwise logical operators (which may
> potentially be useful for other things)" is an idea that has some hope
> of avoiding creating new barriers to learning:
> 
> * if "elementwise" doesn't mean anything to you, and no library you're
> using mentions them in its documentation, you can ignore them

But "elementwise" isn't what people doing symbolic computation or most other uses of DSL/expression-tree libraries are doing.

Even for ORMs and other query-based libraries like AppScript, where arguably it is what they're doing, they probably aren't thinking of it that way, and wouldn't recognize that it should mean something to them, much less that it's what they're looking for.

So I think this is effectively less general/useless than a solution that just allows overloading boolean operators somehow, without adding a distinction between elementwise (and overloadable) and objectwise (and not).

> * for folks that do know what it means, "elementwise" is evocative of
> the relevant semantics for at least the data analysis use case
> 
>> But as for general element-wise operators:
> 
> I wasn't suggesting those - just element-wise logical operators. In
> terms of scoping the use case: bitwise and/or/not work fine for
> manipulating existing boolean masks, and converting a single matrix to
> a boolean mask involves apply an elementwise function, not elementwise
> logical operations.
> 
> So elementwise logical operators would presumably be aimed at *data
> merging* problems - creating a combined matrix where some values are
> taken from matrix A and others from matrix B, based on the truthiness
> of those values.
> I'm not enough of a data analyst to know how common
> that problem is, or whether it might be better served by a higher
> level "replace_elements" operation that accepts a base array, a
> replacement array, and a boolean mask saying which values to replace.

When I use NumPy, sometimes I'm doing GPU-ish stream operations, which need things like compacting select, which aren't obviously expressible in boolean terms, so I end up looking for methods for everything rather than operators even when they might make sense.

But otherwise, when I'm doing more typical NumPy stuff (or at least what I think is more typical, but I could easily be wrong), I look for elementwise operators all over the place, including abusing the bitwise operators when it makes sense, so I probably would use real boolean operators if it were more obvious/readable.

From njs at pobox.com  Thu Nov 26 01:42:01 2015
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 25 Nov 2015 22:42:01 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAP7+vJL-Tm9Z00sfG3nH57T+AkQm3Uv-Z_eJ=nagzaRDBADq8A@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
 <CAP7+vJ+rPPzy-BX1q+oATgKJyBfTLaVv9m-1hwceQs4DH14_Tg@mail.gmail.com>
 <CAPJVwBn7A14=BtAK6BKGvt_eT8yMekont0uVjNnuOEUNPKmwuQ@mail.gmail.com>
 <CAP7+vJL-Tm9Z00sfG3nH57T+AkQm3Uv-Z_eJ=nagzaRDBADq8A@mail.gmail.com>
Message-ID: <CAPJVwB=W2z6wxZvOpwWtQKveOQUsAMJG=v+n8UAd2Ne8pmiZ5Q@mail.gmail.com>

On Nov 24, 2015 11:53 AM, "Guido van Rossum" <guido at python.org> wrote:
>
> On Tue, Nov 24, 2015 at 11:37 AM, Nathaniel Smith <njs at pobox.com> wrote:
> > On Nov 24, 2015 10:21 AM, "Guido van Rossum" <guido at python.org> wrote:
> >>
> >> To everyone claiming that you can't overload and/or because they are
> >> shortcut operators, please re-read PEP 335. It provides a clean
> >> solution -- it was rejected because it adds an extra byte code to all
> >> code using those operators (the majority of which don't need it).
> >
> > The semantic objection that I raised -- short circuiting means that you
> > can't correctly overload 'True and numpy_array', because unlike all other
> > binops the overload must be defined on the left hand argument -- does apply
> > to PEP 335 AFAICT. This problem is IMHO serious enough that even if PEP 335
> > were accepted today I'm not entirely sure that numpy would actually
> > implement the overloads due to the headaches it would cause for teaching and
> > code review -- we'd have to have some debate about it at least.
>
> OK, that's useful feedback. Is NumPy interested in coming up with an
> alternative that works, or are you fine with the status quo?

We'd certainly love it if there were a better alternative, but --
speaking just for myself here -- I've hesitated to wade in because I
don't have any brilliant ideas to contribute :-). The right-hand-side
overload problem seems like an inevitable consequence of
short-circuiting, and we certainly aren't going to switch 'and'/'or'
to become eagerly evaluating. OTOH none of the alternative proposals
mooted so far have struck me as very compelling or pythonic, if only
because all the proposed spellings are ugly, but, who knows, sometimes
something awesome appears deep in these threads.

Two thoughts on places where it might be easier to make some progress...

- the 'not' overloading proposed in PEP 335 doesn't seem to create any
horrible problems - it'd be a minor thing, but maybe it's worth
pulling out as a standalone change?

- the worst code expansion created by lack of overloading isn't
  a == 1 and b == 2
becoming
  (a == 1) & (b == 2)
but rather
  0 < complex expression < 1
becoming
  tmp = complex expression
  (0 < tmp) & (tmp < 1)
That is, the implicit 'and' inside chained comparisons is the biggest
pain point. And this issue is orthogonal to the proposals that involve
adding new operators, b/c they only help with explicit 'and'/'or', not
implicit 'and'. Which is why I was sounding you out about making
chained comparisons eagerly evaluated at the bar at pycon this year
;-). I have the suspicion that the short-circuiting semantics of
chained comparisons are more surprising and confusing than they are
useful and we should just make them eagerly evaluated, and I know you
have the opposite intuition, so I think the next step here would be to
collect some data (run a survey, scan some code, ...?) to figure out
which of us is right :-).

The latter idea in particular has been on my todo list for at least 6
months and still has not bubbled up near the top, so if anyone is
interested in pushing it forward then please feel free :-).

-n

From stephen at xemacs.org  Thu Nov 26 02:55:48 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 26 Nov 2015 16:55:48 +0900
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
Message-ID: <22102.47876.529469.808030@turnbull.sk.tsukuba.ac.jp>

Nick Coghlan writes:

 > So elementwise logical operators would presumably be aimed at *data
 > merging* problems - creating a combined matrix where some values are
 > taken from matrix A and others from matrix B, based on the truthiness
 > of those values. I'm not enough of a data analyst to know how
 > common

Well, for the social science data analysis I do, that would be
inappropriate.  Variables from different sources are different
variables, you wouldn't just "or" them into a single column of a data
frame.  You would want your data model to account for the fact that
even if they purport to measure the same factor, they're actually
different indicators.

But for a completely different kind of data, images, that sounds a lot
like (Duff's?) compositing operations.  But those are a lot more
flexible than just "and" and "or": color images are "fuzzy" logic, and
so admit many more logical operations (eg, "clamped sum",
"proportional combination", etc.  I'm not sure how that would fit
here, since Python has only a limited number of operator symbols,
fewer than there are compositing operations IIRC.

Steve




From oscar.j.benjamin at gmail.com  Thu Nov 26 06:56:36 2015
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Thu, 26 Nov 2015 11:56:36 +0000
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAPJVwB=W2z6wxZvOpwWtQKveOQUsAMJG=v+n8UAd2Ne8pmiZ5Q@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
 <CAP7+vJ+rPPzy-BX1q+oATgKJyBfTLaVv9m-1hwceQs4DH14_Tg@mail.gmail.com>
 <CAPJVwBn7A14=BtAK6BKGvt_eT8yMekont0uVjNnuOEUNPKmwuQ@mail.gmail.com>
 <CAP7+vJL-Tm9Z00sfG3nH57T+AkQm3Uv-Z_eJ=nagzaRDBADq8A@mail.gmail.com>
 <CAPJVwB=W2z6wxZvOpwWtQKveOQUsAMJG=v+n8UAd2Ne8pmiZ5Q@mail.gmail.com>
Message-ID: <CAHVvXxSYVSKKTJXk9kY1T5UNsVdwVOq3OD-mDSCJAGTiT-zzjw@mail.gmail.com>

On 26 November 2015 at 06:42, Nathaniel Smith <njs at pobox.com> wrote:
>
> - the worst code expansion created by lack of overloading isn't
>   a == 1 and b == 2
> becoming
>   (a == 1) & (b == 2)
> but rather
>   0 < complex expression < 1
> becoming
>   tmp = complex expression
>   (0 < tmp) & (tmp < 1)
> That is, the implicit 'and' inside chained comparisons is the biggest
> pain point. And this issue is orthogonal to the proposals that involve
> adding new operators, b/c they only help with explicit 'and'/'or', not
> implicit 'and'. Which is why I was sounding you out about making
> chained comparisons eagerly evaluated at the bar at pycon this year
> ;-). I have the suspicion that the short-circuiting semantics of
> chained comparisons are more surprising and confusing than they are
> useful and we should just make them eagerly evaluated, and I know you
> have the opposite intuition, so I think the next step here would be to
> collect some data (run a survey, scan some code, ...?) to figure out
> which of us is right :-).

Regardless of which is more useful it would be a very subtle backwards
compatibility break. I imagine that code that relies on the
short-circuit here is rare. I'm confident that it is much rarer than
numpy-ish code that works around this in the way you showed above or
just by evaluating the complex expression twice as in
     (0 < complex expression) & (complex expression < 1)
which is what I normally write when the expression isn't too long.

But there's guaranteed to be some breakage and a good migration path
to mitigate that is unclear. You could add a __future__ import and a
warning mode to detect when short-circuiting happens in chained
comparisons. Unfortunately the naive implementation of the warning
mode would simply trigger on 50% of chained comparisons (every time
the left hand relation is False).

I would definitely use chained comparisons for numpy arrays if it were
possible but at the same time I don't think the status quo on this is
that bad. It's a bit of a gotcha for new numpy users to learn but
numpy is good at giving the appropriate error messages:

    >>> from numpy import array
    >>> 1 < array([1, 2, 3]) < 2
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: The truth value of an array with more than one element
is ambiguous. Use a.any() or a.all()
    >>> 1 < array([1, 2, 3]) and 3
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: The truth value of an array with more than one element
is ambiguous. Use a.any() or a.all()

If you google that error message there's lots of SO posts etc. that
can explain in more detail.

It's also worth noting since we're comparing with Matlab that Matlab
actually has both short-circuit logical && and element-wise logical &
equivalent to Python's and/& respectively. And Matlab doesn't have
chained comparisons or rather they don't do anything nearly as useful
as Python's chained comparisons i.e. in Matlab -2 < -1 < 0 is
evaluated as:
    -2 < -1 < 0
    (-2 < -1) < 0
    1 < 0
    0   (i.e. False)
which is basically useless and doesn't even give a decent error
message like numpy does.

--
Oscar

From greg.ewing at canterbury.ac.nz  Thu Nov 26 15:54:44 2015
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 27 Nov 2015 09:54:44 +1300
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
 <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>
Message-ID: <56577194.5000003@canterbury.ac.nz>

Andrew Barnert via Python-ideas wrote:
> But "elementwise" isn't what people doing symbolic computation or most other
> uses of DSL/expression-tree libraries are doing.

Right. I think just describing them as "overloadable versions
of the boolean operators" would be best. What they mean is
up to the types concerned, just as with all the oher operators.

-- 
Greg

From python at lucidity.plus.com  Thu Nov 26 18:22:25 2015
From: python at lucidity.plus.com (Erik)
Date: Thu, 26 Nov 2015 23:22:25 +0000
Subject: [Python-ideas] PEP471 - (os.scandir())
Message-ID: <56579431.4010907@lucidity.plus.com>

PEP471 introduces a faster way of doing low-level directory traversal 
which is then used to implement and speed up the higher-level API 
os.walk() - which for me at least is the "go to API" for most directory 
scanning code I write.

However, when using os.walk() the first thing that one tends to do with 
the results is to analyse them in some way (look at file sizes, 
datestamps and other things that stat() returns) which is exactly the 
information that os.scandir() is caching and speeding up but which is 
then thrown away in order to emulate os.walk()'s original name-based API 
(well, name and type as the directory/file distinction is also there).

So, I'd like to suggest an os.walk()-like API that returns the 
os.scandir() DirEntry structures rather than names (*). I have my own 
local version that's just a copy of os.walk() that appends "entry" 
rather than "entry.name" to the returned lists, but that's a nasty way 
of achieving this.

How to do it -

os.walk() "direntries=True" keyword?
os.walkentries() function?
Something else better than those?

Regards, E.

(*) I have studied the PEP, followed a lot of the references and looked 
at the 3.5.0 implementation. I can't see that I've missed such a thing 
already existing, but it's possible. If so, perhaps this is instead a 
request to make that thing more obvious somehow!

From guido at python.org  Thu Nov 26 19:47:15 2015
From: guido at python.org (Guido van Rossum)
Date: Thu, 26 Nov 2015 16:47:15 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAPJVwB=W2z6wxZvOpwWtQKveOQUsAMJG=v+n8UAd2Ne8pmiZ5Q@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <n310dc$92q$1@ger.gmane.org> <5654A89B.40607@brenbarn.net>
 <CAP7+vJ+rPPzy-BX1q+oATgKJyBfTLaVv9m-1hwceQs4DH14_Tg@mail.gmail.com>
 <CAPJVwBn7A14=BtAK6BKGvt_eT8yMekont0uVjNnuOEUNPKmwuQ@mail.gmail.com>
 <CAP7+vJL-Tm9Z00sfG3nH57T+AkQm3Uv-Z_eJ=nagzaRDBADq8A@mail.gmail.com>
 <CAPJVwB=W2z6wxZvOpwWtQKveOQUsAMJG=v+n8UAd2Ne8pmiZ5Q@mail.gmail.com>
Message-ID: <CAP7+vJKw5xtSwSS0=yvAY_jtRBr7m=kQr-RBWOC17FHLnWS=XQ@mail.gmail.com>

On Wed, Nov 25, 2015 at 10:42 PM, Nathaniel Smith <njs at pobox.com> wrote:

> On Nov 24, 2015 11:53 AM, "Guido van Rossum" <guido at python.org> wrote:
> >
> > On Tue, Nov 24, 2015 at 11:37 AM, Nathaniel Smith <njs at pobox.com> wrote:
> > > On Nov 24, 2015 10:21 AM, "Guido van Rossum" <guido at python.org> wrote:
> > >>
> > >> To everyone claiming that you can't overload and/or because they are
> > >> shortcut operators, please re-read PEP 335. It provides a clean
> > >> solution -- it was rejected because it adds an extra byte code to all
> > >> code using those operators (the majority of which don't need it).
> > >
> > > The semantic objection that I raised -- short circuiting means that you
> > > can't correctly overload 'True and numpy_array', because unlike all
> other
> > > binops the overload must be defined on the left hand argument -- does
> apply
> > > to PEP 335 AFAICT. This problem is IMHO serious enough that even if
> PEP 335
> > > were accepted today I'm not entirely sure that numpy would actually
> > > implement the overloads due to the headaches it would cause for
> teaching and
> > > code review -- we'd have to have some debate about it at least.
> >
> > OK, that's useful feedback. Is NumPy interested in coming up with an
> > alternative that works, or are you fine with the status quo?
>
> We'd certainly love it if there were a better alternative, but --
> speaking just for myself here -- I've hesitated to wade in because I
> don't have any brilliant ideas to contribute :-). The right-hand-side
> overload problem seems like an inevitable consequence of
> short-circuiting, and we certainly aren't going to switch 'and'/'or'
> to become eagerly evaluating. OTOH none of the alternative proposals
> mooted so far have struck me as very compelling or pythonic, if only
> because all the proposed spellings are ugly, but, who knows, sometimes
> something awesome appears deep in these threads.
>
> Two thoughts on places where it might be easier to make some progress...
>
> - the 'not' overloading proposed in PEP 335 doesn't seem to create any
> horrible problems - it'd be a minor thing, but maybe it's worth
> pulling out as a standalone change?
>

This seems pretty harmless, and mostly orthogonal to the rest -- except
that overloadable 'not' is not very attractive or useful by itself if we
decide not to address the others.


> - the worst code expansion created by lack of overloading isn't
>   a == 1 and b == 2
> becoming
>   (a == 1) & (b == 2)
> but rather
>   0 < complex expression < 1
> becoming
>   tmp = complex expression
>   (0 < tmp) & (tmp < 1)
> That is, the implicit 'and' inside chained comparisons is the biggest
> pain point. And this issue is orthogonal to the proposals that involve
> adding new operators, b/c they only help with explicit 'and'/'or', not
> implicit 'and'. Which is why I was sounding you out about making
> chained comparisons eagerly evaluated at the bar at pycon this year
> ;-). I have the suspicion that the short-circuiting semantics of
> chained comparisons are more surprising and confusing than they are
> useful and we should just make them eagerly evaluated, and I know you
> have the opposite intuition, so I think the next step here would be to
> collect some data (run a survey, scan some code, ...?) to figure out
> which of us is right :-).
>
> The latter idea in particular has been on my todo list for at least 6
> months and still has not bubbled up near the top, so if anyone is
> interested in pushing it forward then please feel free :-).


What to do with chaining comparisons is a very good question, but changing
them to be non-short-circuiting would cause a world of backwards
incompatible pain. For example, I've definitely written code where I
carefully arranged the comparisons so that the most expensive one comes
last (in hopes of sometimes avoiding the time spent on it if the outcome is
already determined). This is particularly easy with chained ==, but you can
sometimes also change a < b < c into c > b > a, if a happens to be the
expensive one.

However, note that PEP 335 *does* address the problem of chained
comparisons head-on -- in a < b < c, if a < b returns a numpy array (or
some other special object that's not falsey), it will then proceed to
compute b < c and combine the two using the overloading of the default
'and'; because the first result is not a simple bool, the problem you
described with `True and numpy_array` does not apply.

So, maybe you and the numpy community can ponder PEP 335 some more?
Honestly, from the general language design POV (i.e., mine :-), PEP 335
feels more acceptable than introducing new non-short-circuit and/or
operators. How common would the `True and numpy_array` problem really be? I
suppose any real occurrences would not use the literal True; `True and x`
is just a wordy way to spell x, and doing this element-wise would just
return the array x unchanged. (Or would it cast the elements to bool? That
still feels like a unary operator to me that deserves a more direct
spelling.) I don't see a use case for a literal left operator in symbolic
algebra or SQL either. But let's assume we have some scalar expression that
evaluates to a bool. Even then, `x and numpy_array` feels like a clumsy way
to spell `numpy_array if x else <an array of the same shape filled with
False>`. But I suppose you've thought about this more than I have. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151126/7bfdd46f/attachment.html>

From ncoghlan at gmail.com  Thu Nov 26 21:11:32 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 27 Nov 2015 12:11:32 +1000
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <56577194.5000003@canterbury.ac.nz>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
 <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>
 <56577194.5000003@canterbury.ac.nz>
Message-ID: <CADiSq7dQ-Myk7k7shUFyf1fx5n0ww5NERWQ3s3Db2J9=pHqu4g@mail.gmail.com>

On 27 November 2015 at 06:54, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Andrew Barnert via Python-ideas wrote:
>>
>> But "elementwise" isn't what people doing symbolic computation or most
>> other
>> uses of DSL/expression-tree libraries are doing.
>
> Right. I think just describing them as "overloadable versions
> of the boolean operators" would be best. What they mean is
> up to the types concerned, just as with all the oher operators.

Except that *isn't* what we do with the other operators. "&" is the
bitwise and operator, for example - that's why the special method is
called "__and__". The fact you can use it for an elementwise bitwise
and operation on NumPy arrays, or for set intersection, isn't part of
the core design. If there isn't *at least one* specific motivating use
case, then "it might be useful for something" isn't a good reason to
add new syntax.

However, looking again at PEP 335, I'm not sure I see any reason it
needs to noticeably slower in the standard case than the status quo
(I'm not saying it would be *easy* to retain the speed, but the
complexity would be in the eval loop implementation and the code
generation process, not user code). We also have the richer benchmark
suite these days to actually quantify the impact of checking for the
new __and1__/__or1__ slots before falling back to __bool__, and
tracing JIT's would still be able to generate appropriate code for the
fast path at runtime.

So perhaps it might be worth dusting off that original idea and seeing
what the impact is on the performance benchmarks?

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ericfahlgren at gmail.com  Fri Nov 27 08:49:55 2015
From: ericfahlgren at gmail.com (Eric Fahlgren)
Date: Fri, 27 Nov 2015 05:49:55 -0800
Subject: [Python-ideas] PEP471 - (os.scandir())
In-Reply-To: <56579431.4010907@lucidity.plus.com>
References: <56579431.4010907@lucidity.plus.com>
Message-ID: <037b01d1291a$80123360$80369a20$@gmail.com>

> -----Original Message-----
> From: Erik [mailto:python at lucidity.plus.com] 
> Sent: Thursday, November 26, 2015 15:22
> To: python-ideas
> Subject: [Python-ideas] PEP471 - (os.scandir())
> 
> PEP471 introduces a faster way of doing low-level directory traversal which is then used to implement and speed up the higher-level API
> os.walk() - which for me at least is the "go to API" for most directory scanning code I write.
> 
> However, when using os.walk() the first thing that one tends to do with the results is to analyse them in some way (look at file sizes, datestamps and other things that stat() returns) which is exactly the information that os.scandir() is caching and speeding up but which is then thrown away in order to emulate os.walk()'s original name-based API (well, name and type as the directory/file distinction is also there).
> 
> So, I'd like to suggest an os.walk()-like API that returns the
> os.scandir() DirEntry structures rather than names (*). I have my own local version that's just a copy of os.walk() that appends "entry" 
> rather than "entry.name" to the returned lists, but that's a nasty way of achieving this.
> 
> How to do it -
> 
> os.walk() "direntries=True" keyword?
> os.walkentries() function?
> Something else better than those?

"walk" + "scandir" = "walkdir"???

I'm definitely +1 on this, as it is fresh on my mind, too.  I just converted our build tools over to use a homebrew walk as you did, and now use DirEntry instead of path names almost exclusively.

EricF


From abarnert at yahoo.com  Fri Nov 27 12:28:10 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 27 Nov 2015 09:28:10 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CADiSq7dQ-Myk7k7shUFyf1fx5n0ww5NERWQ3s3Db2J9=pHqu4g@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
 <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>
 <56577194.5000003@canterbury.ac.nz>
 <CADiSq7dQ-Myk7k7shUFyf1fx5n0ww5NERWQ3s3Db2J9=pHqu4g@mail.gmail.com>
Message-ID: <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com>

On Nov 26, 2015, at 18:11, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
>> On 27 November 2015 at 06:54, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>> Andrew Barnert via Python-ideas wrote:
>>> 
>>> But "elementwise" isn't what people doing symbolic computation or most
>>> other
>>> uses of DSL/expression-tree libraries are doing.
>> 
>> Right. I think just describing them as "overloadable versions
>> of the boolean operators" would be best. What they mean is
>> up to the types concerned, just as with all the oher operators.
> 
> Except that *isn't* what we do with the other operators. "&" is the
> bitwise and operator, for example - that's why the special method is
> called "__and__". The fact you can use it for an elementwise bitwise
> and operation on NumPy arrays, or for set intersection, isn't part of
> the core design. If there isn't *at least one* specific motivating use
> case, then "it might be useful for something" isn't a good reason to
> add new syntax.

Sure. But the sqlanywhere case was the very first motivating use the OP mentioned, before NumPy, not something that may come up in the future that we haven't imagined yet.

Also, the distinguishing thing about this new magic method vs. the existing __and__ isn't that it's elementwise, but that it's boolean/logical rather than bitwise/arithmetic, even for NumPy users. So, calling it "elementwise and", or giving it a name that implies elementwise, will confuse anyone who hasn't read this whole thread.

And finally, NumPy is one of the uses that doesn't require short circuiting, and the same is almost certainly true for other elementwise uses, and yet we seem to all be agreed that the new overload has to be short-circuitable.

> However, looking again at PEP 335, I'm not sure I see any reason it
> needs to noticeably slower in the standard case than the status quo
> (I'm not saying it would be *easy* to retain the speed, but the
> complexity would be in the eval loop implementation and the code
> generation process, not user code). We also have the richer benchmark
> suite these days to actually quantify the impact of checking for the
> new __and1__/__or1__ slots before falling back to __bool__, and
> tracing JIT's would still be able to generate appropriate code for the
> fast path at runtime.
> 
> So perhaps it might be worth dusting off that original idea and seeing
> what the impact is on the performance benchmarks?
> 
> Regards,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From wjun77 at gmail.com  Fri Nov 27 12:49:32 2015
From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=)
Date: Sat, 28 Nov 2015 01:49:32 +0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
Message-ID: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>

Hello everyone:

I'm suggesting a modification to the AttributeError/__getattr__ mechanism,
see issue25634:
http://bugs.python.org/issue25634

I used __getattr__ sometimes, and descriptor especially property is so
widely used. I wonder whether someone had encountered the same problem with
me.

However, this is a complicated problem including performance issues, and
backward compatibility.

Thanks for your attention,
Jun Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151128/fd5e58c3/attachment.html>

From python at mrabarnett.plus.com  Fri Nov 27 13:20:01 2015
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 27 Nov 2015 18:20:01 +0000
Subject: [Python-ideas] PEP471 - (os.scandir())
In-Reply-To: <037b01d1291a$80123360$80369a20$@gmail.com>
References: <56579431.4010907@lucidity.plus.com>
 <037b01d1291a$80123360$80369a20$@gmail.com>
Message-ID: <56589ED1.6030805@mrabarnett.plus.com>

On 2015-11-27 13:49, Eric Fahlgren wrote:
>> -----Original Message-----
>> From: Erik [mailto:python at lucidity.plus.com]
>> Sent: Thursday, November 26, 2015 15:22
>> To: python-ideas
>> Subject: [Python-ideas] PEP471 - (os.scandir())
>>
>> PEP471 introduces a faster way of doing low-level directory traversal which is then used to implement and speed up the higher-level API
>> os.walk() - which for me at least is the "go to API" for most directory scanning code I write.
>>
>> However, when using os.walk() the first thing that one tends to do with the results is to analyse them in some way (look at file sizes, datestamps and other things that stat() returns) which is exactly the information that os.scandir() is caching and speeding up but which is then thrown away in order to emulate os.walk()'s original name-based API (well, name and type as the directory/file distinction is also there).
>>
>> So, I'd like to suggest an os.walk()-like API that returns the
>> os.scandir() DirEntry structures rather than names (*). I have my own local version that's just a copy of os.walk() that appends "entry"
>> rather than "entry.name" to the returned lists, but that's a nasty way of achieving this.
>>
>> How to do it -
>>
>> os.walk() "direntries=True" keyword?
>> os.walkentries() function?
>> Something else better than those?
>
> "walk" + "scandir" = "walkdir"???
>
> I'm definitely +1 on this, as it is fresh on my mind, too.  I just converted our build tools over to use a homebrew walk as you did, and now use DirEntry instead of path names almost exclusively.
>
There's nothing in that name that suggests "scandir" rather that
"listdir". You could just as easily say "walk" + "listdir" = "walkdir".


From p.f.moore at gmail.com  Fri Nov 27 13:20:35 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 27 Nov 2015 18:20:35 +0000
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>
References: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>
Message-ID: <CACac1F-zE5PPPcDsY4cMyHVbEo_xP-bsRNDPKybpEhA40gdL9g@mail.gmail.com>

On 27 November 2015 at 17:49, ?? <wjun77 at gmail.com> wrote:
> I used __getattr__ sometimes, and descriptor especially property is so
> widely used. I wonder whether someone had encountered the same problem with
> me.
>
> However, this is a complicated problem including performance issues, and
> backward compatibility.

I understand the issue you are reporting (and thanks for stripping it
down to a simple example in the issue) but could you give a real world
example of how this might actually be an issue in practice? I don't
believe I've ever heard anyone report this as an issue before, which
implies to me that it's not a particularly common problem in real
code.

I'm not 100% clear on how bad any backward compatibility or
performance issues might be, but it's certainly true that we'd need
real-world use cases to justify the cost of the change.
Paul

From p.f.moore at gmail.com  Fri Nov 27 13:32:09 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 27 Nov 2015 18:32:09 +0000
Subject: [Python-ideas] PEP471 - (os.scandir())
In-Reply-To: <56579431.4010907@lucidity.plus.com>
References: <56579431.4010907@lucidity.plus.com>
Message-ID: <CACac1F8FN0NvtgJ2EBDXYFPfWcMjNDohB4W1RmLKubYt2dCGPA@mail.gmail.com>

On 26 November 2015 at 23:22, Erik <python at lucidity.plus.com> wrote:
> I have studied the PEP, followed a lot of the references and looked at the
> 3.5.0 implementation. I can't see that I've missed such a thing already
> existing, but it's possible. If so, perhaps this is instead a request to
> make that thing more obvious somehow!

Does pathlib use scandir? If so, then maybe you get the caching
benefits by using pathlib? And if pathlib doesn't use scandir, maybe
it should? [I just checked, it looks like pathlib doesn't use scandir
:-(]

Paul

From abarnert at yahoo.com  Fri Nov 27 13:34:24 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 27 Nov 2015 10:34:24 -0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>
References: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>
Message-ID: <CED82D87-A423-4874-82CB-97E3E1562A0F@yahoo.com>

It seems like you can get some of the benefits of this proposal without backward compat issues.

Instead of changing things so AttributeError from __getattribute__ or a descriptor no longer calls __getattr__, just add a new subclass that doesn't, and change the Descriptor HOWTO to suggest using that subclass (with a bit of discussion that says you can use AttributeError if you want to trigger __getattr__, but usually you won't).

That wouldn't fix any current code, but it also wouldn't break any code that intentionally uses the features as documented. And it would make it easy to write correct new code.

One more thing:
> Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection.

Is that really an imperfection? It says right there in the docs for __getattribute__ that you can delegate to __getattr__ by raising AttributeError, so if someone does that, presumably it's intentional. It's not like the case with descriptors, where you have to think through the interaction of multiple features to figure out that raising an AttributeError will call __getattr__, and therefore many such uses are probably bugs.

Sent from my iPhone

> On Nov 27, 2015, at 09:49, ?? <wjun77 at gmail.com> wrote:
> 
> Hello everyone:
> 
> I'm suggesting a modification to the AttributeError/__getattr__ mechanism, see issue25634:
> http://bugs.python.org/issue25634
> 
> I used __getattr__ sometimes, and descriptor especially property is so widely used. I wonder whether someone had encountered the same problem with me.
> 
> However, this is a complicated problem including performance issues, and backward compatibility.
> 
> Thanks for your attention,
> Jun Wang
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151127/c855fbbe/attachment-0001.html>

From p.f.moore at gmail.com  Fri Nov 27 13:34:45 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 27 Nov 2015 18:34:45 +0000
Subject: [Python-ideas] PEP471 - (os.scandir())
In-Reply-To: <CACac1F8FN0NvtgJ2EBDXYFPfWcMjNDohB4W1RmLKubYt2dCGPA@mail.gmail.com>
References: <56579431.4010907@lucidity.plus.com>
 <CACac1F8FN0NvtgJ2EBDXYFPfWcMjNDohB4W1RmLKubYt2dCGPA@mail.gmail.com>
Message-ID: <CACac1F9Bxb__h0mHSr9LWc7TKEZZWLS_kSPth__Z-53aLH1JAA@mail.gmail.com>

On 27 November 2015 at 18:32, Paul Moore <p.f.moore at gmail.com> wrote:
> On 26 November 2015 at 23:22, Erik <python at lucidity.plus.com> wrote:
>> I have studied the PEP, followed a lot of the references and looked at the
>> 3.5.0 implementation. I can't see that I've missed such a thing already
>> existing, but it's possible. If so, perhaps this is instead a request to
>> make that thing more obvious somehow!
>
> Does pathlib use scandir? If so, then maybe you get the caching
> benefits by using pathlib? And if pathlib doesn't use scandir, maybe
> it should? [I just checked, it looks like pathlib doesn't use scandir
> :-(]

Never mind - see
https://www.python.org/dev/peps/pep-0471/#return-values-being-pathlib-path-objects
Pathlib objects must not cache the results of stat calls, so they
cannot use scandir.

Paul

From abarnert at yahoo.com  Fri Nov 27 13:42:31 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 27 Nov 2015 10:42:31 -0800
Subject: [Python-ideas] PEP471 - (os.scandir())
In-Reply-To: <CACac1F8FN0NvtgJ2EBDXYFPfWcMjNDohB4W1RmLKubYt2dCGPA@mail.gmail.com>
References: <56579431.4010907@lucidity.plus.com>
 <CACac1F8FN0NvtgJ2EBDXYFPfWcMjNDohB4W1RmLKubYt2dCGPA@mail.gmail.com>
Message-ID: <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com>

On Nov 27, 2015, at 10:32, Paul Moore <p.f.moore at gmail.com> wrote:
> 
>> On 26 November 2015 at 23:22, Erik <python at lucidity.plus.com> wrote:
>> I have studied the PEP, followed a lot of the references and looked at the
>> 3.5.0 implementation. I can't see that I've missed such a thing already
>> existing, but it's possible. If so, perhaps this is instead a request to
>> make that thing more obvious somehow!
> 
> Does pathlib use scandir? If so, then maybe you get the caching
> benefits by using pathlib? And if pathlib doesn't use scandir, maybe
> it should? [I just checked, it looks like pathlib doesn't use scandir
> :-(]

Does pathlib even have a walk equivalent? (I know it has glob('**'), but that's not the same thing.)

Or are you suggesting that people should use path.iterdir with explicit recursion (or an explicit stack), and therefore just changing iterdir to use scandir (and prefill as many cached attribs as possible in each result) is what we want?

From cuthbert at mit.edu  Fri Nov 27 14:29:36 2015
From: cuthbert at mit.edu (Michael Scott Cuthbert)
Date: Fri, 27 Nov 2015 19:29:36 +0000
Subject: [Python-ideas] add a single __future__ for py3?
Message-ID: <0540E031-580B-4ED7-9E9D-7E37C84959E4@mit.edu>

(an old list topic that I accidentally sent to the Google Groups list)

One smaller request is to remove 'barry_as_FLUFL' from "__future__.all_feature_names".  It's a great practical joke, but it bit me hard on a project where I check that, for Py2/Py3 consistency, each module imports all relevant __future__ names, and ensures that doctests on Py2 are run with them on.  I figured that I could iterate through .all_feature_names and turn them all on except unicode_literals and be safe, but suddenly I was getting Syntax errors on all my != tests, and there was not anywhere to turn to for an explanation.  Barry might be pleased.  In any case it seems better to treat it like "braces" which isn?t listed in "all_feature_names"

Best,
Myke Cuthbert

(long-time-reader-first-time-writer intro: switched from Perl to Python in 2006, wrote music21 (music analysis toolkit) in Python, got tenure at MIT for it. Hugely thankful to the amazing community)

From brenbarn at brenbarn.net  Fri Nov 27 14:43:14 2015
From: brenbarn at brenbarn.net (Brendan Barnwell)
Date: Fri, 27 Nov 2015 11:43:14 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
 <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>
 <56577194.5000003@canterbury.ac.nz>
 <CADiSq7dQ-Myk7k7shUFyf1fx5n0ww5NERWQ3s3Db2J9=pHqu4g@mail.gmail.com>
 <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com>
Message-ID: <5658B252.1060902@brenbarn.net>

On 2015-11-27 09:28, Andrew Barnert via Python-ideas wrote:
> And finally, NumPy is one of the uses that doesn't require short
> circuiting, and the same is almost certainly true for other
> elementwise uses, and yet we seem to all be agreed that the new
> overload has to be short-circuitable.

	Do we?  I don't.  I agree that if we add a way to overload the existing 
and/or to support these new usages, then that has to be 
short-circuitable. because and/or currently are short-circuitable and we 
can't get rid of that.  But to me one of the attractive aspects of this 
new proposal is that the new operators need not be short-circuitable, 
which would avoid the various contortions required in a scheme like PEP 
335 and thus greatly simplify the overloading.  In other words the whole 
point of these new operators would be to do and-like and/or or-like 
operations that definitely do want both of their arguments all the time 
(such as elementwise operations or combining abstract query objects like 
in these SQL cases).

-- 
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no
path, and leave a trail."
    --author unknown

From guido at python.org  Fri Nov 27 15:15:42 2015
From: guido at python.org (Guido van Rossum)
Date: Fri, 27 Nov 2015 12:15:42 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <5658B252.1060902@brenbarn.net>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
 <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>
 <56577194.5000003@canterbury.ac.nz>
 <CADiSq7dQ-Myk7k7shUFyf1fx5n0ww5NERWQ3s3Db2J9=pHqu4g@mail.gmail.com>
 <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com>
 <5658B252.1060902@brenbarn.net>
Message-ID: <CAP7+vJLVWh--+b4qHkawMXVPQ+V8ZifpRUPocnfR+Jm55CVquQ@mail.gmail.com>

On Fri, Nov 27, 2015 at 11:43 AM, Brendan Barnwell <brenbarn at brenbarn.net>
wrote:

> On 2015-11-27 09:28, Andrew Barnert via Python-ideas wrote:
>
>> And finally, NumPy is one of the uses that doesn't require short
>> circuiting, and the same is almost certainly true for other
>> elementwise uses, and yet we seem to all be agreed that the new
>> overload has to be short-circuitable.
>>
>
>         Do we?  I don't.  I agree that if we add a way to overload the
> existing and/or to support these new usages, then that has to be
> short-circuitable. because and/or currently are short-circuitable and we
> can't get rid of that.  But to me one of the attractive aspects of this new
> proposal is that the new operators need not be short-circuitable, which
> would avoid the various contortions required in a scheme like PEP 335 and
> thus greatly simplify the overloading.  In other words the whole point of
> these new operators would be to do and-like and/or or-like operations that
> definitely do want both of their arguments all the time (such as
> elementwise operations or combining abstract query objects like in these
> SQL cases).
>

Agreed, that final clause from Andrew seems a non-sequitur.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151127/39ef7617/attachment-0001.html>

From wjun77 at gmail.com  Fri Nov 27 15:18:39 2015
From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=)
Date: Sat, 28 Nov 2015 04:18:39 +0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CED82D87-A423-4874-82CB-97E3E1562A0F@yahoo.com>
References: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>
 <CED82D87-A423-4874-82CB-97E3E1562A0F@yahoo.com>
Message-ID: <CAPM967Tq8DcgRN_ScyrAFRoAX6h6_J3JNcfNd20dUxcYwNi2Mg@mail.gmail.com>

I think the real world use case, as *property* is so common, is just when
__getattr__ needed. So anyone using __getattr__ in practice? In fact I'm
new tweaking these advanced features of python. Any suggestions are welcome
and appreciated.


In my case, I delegate failed attribute lookups to one member of the
instance, as the given simple example in issue25634. For example, there's a
class Point,

class Point():
    def __init__(self, x, y):
        self.x = x
        self.y = y

and class Circle

class Circle():
    def __init__(self, center, radius):
        self.center = center
        self.radius = radius

I don't think Circle should inherit Point. But I want to write Circle().x
instead of Circle().center.x, so I write something like
    def __getattr__(self, name):
        try:
            return getattr(self.center, name)
        except AttributeError:
            raise AttributeError("'{}' object has no attribute
'{}'".format(self.__class__.__name__, name)) from None


Another case is when I try to implement the design pattern of state. That
is, when calling window.rightClick(), the behavior differs according to the
state of the window, or window.__state. So

class ActiveState(State):
    @staticmethod
    def rightClick(self):
        print('right clicked')
class InactiveState(State):
    @staticmethod
    def rightClick(self):
        pass

and

class Window():
    def __init__(self):
        self.__state = ActiveState
    def __getattr__(self, name):
        try:
            return partial(getattr(self.__state, name), self)
        except AttributeError:
            raise AttributeError("'{}' object has no attribute
'{}'".format(self.__class__.__name__, name)) from None

(The real situation is more complicated. In fact I've written a state
module, and I think I can publish it in a week if anyone is interested.)


> Is that really an imperfection? It says right there in the docs for
__getattribute__ that you can delegate to __getattr__ by raising
AttributeError
Just like raising *StopIteration* in generator, as described in PEP479.
Although I'm not sure *StopIteration* has ever been documented as one way
to exit generator. And I admit that an unintentional AttributeError in
__getattribute__ is a rare case if ever.

2015-11-28 2:34 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:

> It seems like you can get some of the benefits of this proposal without
> backward compat issues.
>
> Instead of changing things so AttributeError from __getattribute__ or a
> descriptor no longer calls __getattr__, just add a new subclass that
> doesn't, and change the Descriptor HOWTO to suggest using that subclass
> (with a bit of discussion that says you can use AttributeError if you want
> to trigger __getattr__, but usually you won't).
>
> That wouldn't fix any current code, but it also wouldn't break any code
> that intentionally uses the features as documented. And it would make it
> easy to write correct new code.
>
> One more thing:
>
> Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection.
>
> Is that really an imperfection? It says right there in the docs for
> __getattribute__ that you can delegate to __getattr__ by raising
> AttributeError, so if someone does that, presumably it's intentional. It's
> not like the case with descriptors, where you have to think through the
> interaction of multiple features to figure out that raising an
> AttributeError will call __getattr__, and therefore many such uses are
> probably bugs.
>
> Sent from my iPhone
>
> On Nov 27, 2015, at 09:49, ?? <wjun77 at gmail.com> wrote:
>
> Hello everyone:
>
> I'm suggesting a modification to the AttributeError/__getattr__ mechanism,
> see issue25634:
> http://bugs.python.org/issue25634
>
> I used __getattr__ sometimes, and descriptor especially property is so
> widely used. I wonder whether someone had encountered the same problem with
> me.
>
> However, this is a complicated problem including performance issues, and
> backward compatibility.
>
> Thanks for your attention,
> Jun Wang
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151128/dbd40bc3/attachment.html>

From guido at python.org  Fri Nov 27 15:20:52 2015
From: guido at python.org (Guido van Rossum)
Date: Fri, 27 Nov 2015 12:20:52 -0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967Tq8DcgRN_ScyrAFRoAX6h6_J3JNcfNd20dUxcYwNi2Mg@mail.gmail.com>
References: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>
 <CED82D87-A423-4874-82CB-97E3E1562A0F@yahoo.com>
 <CAPM967Tq8DcgRN_ScyrAFRoAX6h6_J3JNcfNd20dUxcYwNi2Mg@mail.gmail.com>
Message-ID: <CAP7+vJLsVnC9tMKUVTeBiCQ7m0sYD9=CP+sagPQ3RbFeAnyPOQ@mail.gmail.com>

I'm confused about how the proposal would work. How would you prevent
self.x.t from also raising AttributeMissError (thereby defeating its
purpose)?

On Fri, Nov 27, 2015 at 12:18 PM, ?? <wjun77 at gmail.com> wrote:

> I think the real world use case, as *property* is so common, is just when
> __getattr__ needed. So anyone using __getattr__ in practice? In fact I'm
> new tweaking these advanced features of python. Any suggestions are welcome
> and appreciated.
>
>
> In my case, I delegate failed attribute lookups to one member of the
> instance, as the given simple example in issue25634. For example, there's a
> class Point,
>
> class Point():
>     def __init__(self, x, y):
>         self.x = x
>         self.y = y
>
> and class Circle
>
> class Circle():
>     def __init__(self, center, radius):
>         self.center = center
>         self.radius = radius
>
> I don't think Circle should inherit Point. But I want to write Circle().x
> instead of Circle().center.x, so I write something like
>     def __getattr__(self, name):
>         try:
>             return getattr(self.center, name)
>         except AttributeError:
>             raise AttributeError("'{}' object has no attribute
> '{}'".format(self.__class__.__name__, name)) from None
>
>
> Another case is when I try to implement the design pattern of state. That
> is, when calling window.rightClick(), the behavior differs according to the
> state of the window, or window.__state. So
>
> class ActiveState(State):
>     @staticmethod
>     def rightClick(self):
>         print('right clicked')
> class InactiveState(State):
>     @staticmethod
>     def rightClick(self):
>         pass
>
> and
>
> class Window():
>     def __init__(self):
>         self.__state = ActiveState
>     def __getattr__(self, name):
>         try:
>             return partial(getattr(self.__state, name), self)
>         except AttributeError:
>             raise AttributeError("'{}' object has no attribute
> '{}'".format(self.__class__.__name__, name)) from None
>
> (The real situation is more complicated. In fact I've written a state
> module, and I think I can publish it in a week if anyone is interested.)
>
>
> > Is that really an imperfection? It says right there in the docs for
> __getattribute__ that you can delegate to __getattr__ by raising
> AttributeError
> Just like raising *StopIteration* in generator, as described in PEP479.
> Although I'm not sure *StopIteration* has ever been documented as one way
> to exit generator. And I admit that an unintentional AttributeError in
> __getattribute__ is a rare case if ever.
>
> 2015-11-28 2:34 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:
>
>> It seems like you can get some of the benefits of this proposal without
>> backward compat issues.
>>
>> Instead of changing things so AttributeError from __getattribute__ or a
>> descriptor no longer calls __getattr__, just add a new subclass that
>> doesn't, and change the Descriptor HOWTO to suggest using that subclass
>> (with a bit of discussion that says you can use AttributeError if you want
>> to trigger __getattr__, but usually you won't).
>>
>> That wouldn't fix any current code, but it also wouldn't break any code
>> that intentionally uses the features as documented. And it would make it
>> easy to write correct new code.
>>
>> One more thing:
>>
>> Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection.
>>
>> Is that really an imperfection? It says right there in the docs for
>> __getattribute__ that you can delegate to __getattr__ by raising
>> AttributeError, so if someone does that, presumably it's intentional. It's
>> not like the case with descriptors, where you have to think through the
>> interaction of multiple features to figure out that raising an
>> AttributeError will call __getattr__, and therefore many such uses are
>> probably bugs.
>>
>> Sent from my iPhone
>>
>> On Nov 27, 2015, at 09:49, ?? <wjun77 at gmail.com> wrote:
>>
>> Hello everyone:
>>
>> I'm suggesting a modification to the AttributeError/__getattr__
>> mechanism, see issue25634:
>> http://bugs.python.org/issue25634
>>
>> I used __getattr__ sometimes, and descriptor especially property is so
>> widely used. I wonder whether someone had encountered the same problem with
>> me.
>>
>> However, this is a complicated problem including performance issues, and
>> backward compatibility.
>>
>> Thanks for your attention,
>> Jun Wang
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151127/7df7e383/attachment-0001.html>

From wjun77 at gmail.com  Fri Nov 27 15:32:18 2015
From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=)
Date: Sat, 28 Nov 2015 04:32:18 +0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAP7+vJLsVnC9tMKUVTeBiCQ7m0sYD9=CP+sagPQ3RbFeAnyPOQ@mail.gmail.com>
References: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>
 <CED82D87-A423-4874-82CB-97E3E1562A0F@yahoo.com>
 <CAPM967Tq8DcgRN_ScyrAFRoAX6h6_J3JNcfNd20dUxcYwNi2Mg@mail.gmail.com>
 <CAP7+vJLsVnC9tMKUVTeBiCQ7m0sYD9=CP+sagPQ3RbFeAnyPOQ@mail.gmail.com>
Message-ID: <CAPM967RV=gErPrBDHEuvMT_U26v0o6EiHW11HqUzxjwOVZ=2pA@mail.gmail.com>

At first I think of adding AttributeMissError. In that case,
object.__getattr__ (here self.x.__getattr__) could turn AttributeMissError
to AttributeError.
But after some consideration I think there's no need to add an extra
Exception. Just ignore the title of that issue.

2015-11-28 4:20 GMT+08:00 Guido van Rossum <guido at python.org>:

> I'm confused about how the proposal would work. How would you prevent
> self.x.t from also raising AttributeMissError (thereby defeating its
> purpose)?
>
> On Fri, Nov 27, 2015 at 12:18 PM, ?? <wjun77 at gmail.com> wrote:
>
>> I think the real world use case, as *property* is so common, is just
>> when __getattr__ needed. So anyone using __getattr__ in practice? In fact
>> I'm new tweaking these advanced features of python. Any suggestions are
>> welcome and appreciated.
>>
>>
>> In my case, I delegate failed attribute lookups to one member of the
>> instance, as the given simple example in issue25634. For example, there's a
>> class Point,
>>
>> class Point():
>>     def __init__(self, x, y):
>>         self.x = x
>>         self.y = y
>>
>> and class Circle
>>
>> class Circle():
>>     def __init__(self, center, radius):
>>         self.center = center
>>         self.radius = radius
>>
>> I don't think Circle should inherit Point. But I want to write Circle().x
>> instead of Circle().center.x, so I write something like
>>     def __getattr__(self, name):
>>         try:
>>             return getattr(self.center, name)
>>         except AttributeError:
>>             raise AttributeError("'{}' object has no attribute
>> '{}'".format(self.__class__.__name__, name)) from None
>>
>>
>> Another case is when I try to implement the design pattern of state. That
>> is, when calling window.rightClick(), the behavior differs according to the
>> state of the window, or window.__state. So
>>
>> class ActiveState(State):
>>     @staticmethod
>>     def rightClick(self):
>>         print('right clicked')
>> class InactiveState(State):
>>     @staticmethod
>>     def rightClick(self):
>>         pass
>>
>> and
>>
>> class Window():
>>     def __init__(self):
>>         self.__state = ActiveState
>>     def __getattr__(self, name):
>>         try:
>>             return partial(getattr(self.__state, name), self)
>>         except AttributeError:
>>             raise AttributeError("'{}' object has no attribute
>> '{}'".format(self.__class__.__name__, name)) from None
>>
>> (The real situation is more complicated. In fact I've written a state
>> module, and I think I can publish it in a week if anyone is interested.)
>>
>>
>> > Is that really an imperfection? It says right there in the docs for
>> __getattribute__ that you can delegate to __getattr__ by raising
>> AttributeError
>> Just like raising *StopIteration* in generator, as described in PEP479.
>> Although I'm not sure *StopIteration* has ever been documented as one
>> way to exit generator. And I admit that an unintentional AttributeError in
>> __getattribute__ is a rare case if ever.
>>
>> 2015-11-28 2:34 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:
>>
>>> It seems like you can get some of the benefits of this proposal without
>>> backward compat issues.
>>>
>>> Instead of changing things so AttributeError from __getattribute__ or a
>>> descriptor no longer calls __getattr__, just add a new subclass that
>>> doesn't, and change the Descriptor HOWTO to suggest using that subclass
>>> (with a bit of discussion that says you can use AttributeError if you want
>>> to trigger __getattr__, but usually you won't).
>>>
>>> That wouldn't fix any current code, but it also wouldn't break any code
>>> that intentionally uses the features as documented. And it would make it
>>> easy to write correct new code.
>>>
>>> One more thing:
>>>
>>> Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection.
>>>
>>> Is that really an imperfection? It says right there in the docs for
>>> __getattribute__ that you can delegate to __getattr__ by raising
>>> AttributeError, so if someone does that, presumably it's intentional. It's
>>> not like the case with descriptors, where you have to think through the
>>> interaction of multiple features to figure out that raising an
>>> AttributeError will call __getattr__, and therefore many such uses are
>>> probably bugs.
>>>
>>> Sent from my iPhone
>>>
>>> On Nov 27, 2015, at 09:49, ?? <wjun77 at gmail.com> wrote:
>>>
>>> Hello everyone:
>>>
>>> I'm suggesting a modification to the AttributeError/__getattr__
>>> mechanism, see issue25634:
>>> http://bugs.python.org/issue25634
>>>
>>> I used __getattr__ sometimes, and descriptor especially property is so
>>> widely used. I wonder whether someone had encountered the same problem with
>>> me.
>>>
>>> However, this is a complicated problem including performance issues, and
>>> backward compatibility.
>>>
>>> Thanks for your attention,
>>> Jun Wang
>>>
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151128/5e77bdeb/attachment.html>

From rosuav at gmail.com  Fri Nov 27 16:23:08 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 28 Nov 2015 08:23:08 +1100
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967RV=gErPrBDHEuvMT_U26v0o6EiHW11HqUzxjwOVZ=2pA@mail.gmail.com>
References: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>
 <CED82D87-A423-4874-82CB-97E3E1562A0F@yahoo.com>
 <CAPM967Tq8DcgRN_ScyrAFRoAX6h6_J3JNcfNd20dUxcYwNi2Mg@mail.gmail.com>
 <CAP7+vJLsVnC9tMKUVTeBiCQ7m0sYD9=CP+sagPQ3RbFeAnyPOQ@mail.gmail.com>
 <CAPM967RV=gErPrBDHEuvMT_U26v0o6EiHW11HqUzxjwOVZ=2pA@mail.gmail.com>
Message-ID: <CAPTjJmpFMWktFVNipyV7XmY7srwNHUZ-0bMwJ=LX_=XV8R31Ag@mail.gmail.com>

On Sat, Nov 28, 2015 at 7:32 AM, ?? <wjun77 at gmail.com> wrote:
> At first I think of adding AttributeMissError. In that case,
> object.__getattr__ (here self.x.__getattr__) could turn AttributeMissError
> to AttributeError.
> But after some consideration I think there's no need to add an extra
> Exception. Just ignore the title of that issue.

So... if I'm understanding the issue correctly, it's about the
interaction of @property and __getattr__? You're talking about this as
a boundary over which AttributeError changes. Borrowing your example
from the tracker issue:

class property(property):
    def __get__(self, *a):
        try:
            return super().__get__(*a)
        except AttributeError as e:
            raise RuntimeError("Property raised AttributeError") from e

class A():
    def __init__(self, x=None):
        self.x = x

    @property
    def t(self):
        return self.x.t

    def __getattr__(self, name):
        return 'default'

print(A().t)


Traceback (most recent call last):
  File "attrerr.py", line 4, in __get__
    return super().__get__(*a)
  File "attrerr.py", line 14, in t
    return self.x.t
AttributeError: 'NoneType' object has no attribute 't'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "attrerr.py", line 19, in <module>
    print(A().t)
  File "attrerr.py", line 6, in __get__
    raise RuntimeError("Property raised AttributeError") from e
RuntimeError: Property raised AttributeError


This would create a boundary, same as PEP 479 does for StopIteration,
across which AttributeError becomes RuntimeError.

This could be incorporated into the built-in property if desired, or
kept on a per-module basis with the above.

ChrisA

From abarnert at yahoo.com  Fri Nov 27 16:37:51 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 27 Nov 2015 13:37:51 -0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967Tq8DcgRN_ScyrAFRoAX6h6_J3JNcfNd20dUxcYwNi2Mg@mail.gmail.com>
References: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>
 <CED82D87-A423-4874-82CB-97E3E1562A0F@yahoo.com>
 <CAPM967Tq8DcgRN_ScyrAFRoAX6h6_J3JNcfNd20dUxcYwNi2Mg@mail.gmail.com>
Message-ID: <C7D754EA-DECC-452F-BEE5-78F8ADD6E1E3@yahoo.com>

On Nov 27, 2015, at 12:18, ?? <wjun77 at gmail.com> wrote:
> 
> I think the real world use case, as property is so common, is just when __getattr__ needed. So anyone using __getattr__ in practice?

Your own examples seem like reasonable uses of __getattr__ to me. But they don't demonstrate your problem, because you don't have any properties, other custom descriptors, or __getattribute__. Do you have any examples that actually do demonstrate the problem to be solved?

> > Is that really an imperfection? It says right there in the docs for __getattribute__ that you can delegate to __getattr__ by raising AttributeError
> Just like raising StopIteration in generator, as described in PEP479.

But the docs didn't recommend raising StopIteration to exit a generator. The docs implied it by omission, or at least we're ambiguous about what would happen, and you could test it and see that it did, which meant some people took advantage of it to do things like using a StopIteration-throwing function in a comprehension filter clause (with caveats about listcomp vs. genexpr and 2.x vs. 3.x and most readers having to guess why it works). But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades.

> Although I'm not sure StopIteration has ever been documented as one way to exit generator. And I admit that an unintentional AttributeError in __getattribute__ is a rare case if ever. 

That's my point: "fixing" __getattribute__ to eliminate a "rare case if ever" bug, while also introducing a possibly less-rare backward compatibility problem, seems like a terrible idea. Of course that isn't your intention; it's just a side effect of trying to eliminate a more common bug, with probably rarer intentional uses, in descriptors. But to me, that implies that any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better. And, as you imply in the bug report, this could be done by having either the descriptor mechanism itself, or the object.__getattribute__ implementation, handle AttributeError from descriptor lookup by reraising it as something else.

But still, this is all a minor side issue about choosing between your different variations. The big problem is that I think all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so. Maybe you don't want to do so very often, but is it really so rare that we can justify making it impossible? (Especially considering the backward-compat issues for any code that's been doing that for years...)

You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable?

> 2015-11-28 2:34 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:
>> It seems like you can get some of the benefits of this proposal without backward compat issues.
>> 
>> Instead of changing things so AttributeError from __getattribute__ or a descriptor no longer calls __getattr__, just add a new subclass that doesn't, and change the Descriptor HOWTO to suggest using that subclass (with a bit of discussion that says you can use AttributeError if you want to trigger __getattr__, but usually you won't).
>> 
>> That wouldn't fix any current code, but it also wouldn't break any code that intentionally uses the features as documented. And it would make it easy to write correct new code.
>> 
>> One more thing:
>>> Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection.
>> 
>> Is that really an imperfection? It says right there in the docs for __getattribute__ that you can delegate to __getattr__ by raising AttributeError, so if someone does that, presumably it's intentional. It's not like the case with descriptors, where you have to think through the interaction of multiple features to figure out that raising an AttributeError will call __getattr__, and therefore many such uses are probably bugs.
>> 
>> Sent from my iPhone
>> 
>>> On Nov 27, 2015, at 09:49, ?? <wjun77 at gmail.com> wrote:
>>> 
>>> Hello everyone:
>>> 
>>> I'm suggesting a modification to the AttributeError/__getattr__ mechanism, see issue25634:
>>> http://bugs.python.org/issue25634
>>> 
>>> I used __getattr__ sometimes, and descriptor especially property is so widely used. I wonder whether someone had encountered the same problem with me.
>>> 
>>> However, this is a complicated problem including performance issues, and backward compatibility.
>>> 
>>> Thanks for your attention,
>>> Jun Wang
>>> _______________________________________________
>>> Python-ideas mailing list
>>> Python-ideas at python.org
>>> https://mail.python.org/mailman/listinfo/python-ideas
>>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151127/c4df94e7/attachment-0001.html>

From wjun77 at gmail.com  Fri Nov 27 18:23:13 2015
From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=)
Date: Sat, 28 Nov 2015 07:23:13 +0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <C7D754EA-DECC-452F-BEE5-78F8ADD6E1E3@yahoo.com>
References: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>
 <CED82D87-A423-4874-82CB-97E3E1562A0F@yahoo.com>
 <CAPM967Tq8DcgRN_ScyrAFRoAX6h6_J3JNcfNd20dUxcYwNi2Mg@mail.gmail.com>
 <C7D754EA-DECC-452F-BEE5-78F8ADD6E1E3@yahoo.com>
Message-ID: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>

> This would create a boundary, same as PEP 479 does for StopIteration,
> across which AttributeError becomes RuntimeError.
Although my problem is about property as I don't use descriptor here, I
think descriptor other than property should also be considered, such as
classmethod/staticmethod, and normal descriptor. Raising "RuntimeError:
descriptor raised AttributeError" is acceptable for me, but maybe some c
code need modification, I don't know.

> Do you have any examples that actually do demonstrate the problem to be
solved?
So you want more details about AttributeError in property? I thought
property is widely used, and AttributeError occurs at all times. Maybe I've
used property too heavy.
In the window example, a simplified demonstration:

class Window():
    @property
    def backgroundImg(self):
        if self._backgroundImg is None: #need update, while the number of
items changes
            self.set_backgroundImg()
        return self._backgroundImg

    def set_backgroundImg(self):
        self._backgroundImg = loadImg('white.bmp')
        for widget in self.widgets:
            widget.show(self._backgroundImg)

Class Widget():
    def show(self, img):
        img.draw(self.item.img, self.pos)

However, widget.item may be None, while e.g. there are four widgets but
only three items in total. In this case I should fill the area with white.
But in this version of *show*, I just FORGET item can be None. So the
traceback infomation: 'Window' object has no attribute 'backgroundImg'. In
fact it takes a while before I find the cause is the
AttributeError/__getattr__ mechanism.

> But surely breaking that isn't the same as breaking code that's been
explicitly stated to work, and used as sample code, for decades.
I don't know this is such a severe problem. I used to think raising
AttributeError in __getattribute__ to trigger __getattr__ is rare.

> any solution that can fix descriptors without also "fixing"
__getattribute__ is a lot better
In practice I don't concern __getattribute__. But in my opinion it's better
to 'fix' this in python4.

> all of your solutions make it too hard to trigger __getattr__ from a
descriptor when you really _do_ want to do so
This is a big problem, OK.

> You didn't comment on the alternative I suggested; would it not satisfy
your needs, or have some other problem that makes it unacceptable?
I don't quite understand, you mean adding a subclass of *object* with the
only difference of this behavior?

2015-11-28 5:37 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:

> On Nov 27, 2015, at 12:18, ?? <wjun77 at gmail.com> wrote:
>
> I think the real world use case, as *property* is so common, is just when
> __getattr__ needed. So anyone using __getattr__ in practice?
>
>
> Your own examples seem like reasonable uses of __getattr__ to me. But they
> don't demonstrate your problem, because you don't have any properties,
> other custom descriptors, or __getattribute__. Do you have any examples
> that actually do demonstrate the problem to be solved?
>
> > Is that really an imperfection? It says right there in the docs for
> __getattribute__ that you can delegate to __getattr__ by raising
> AttributeError
> Just like raising *StopIteration* in generator, as described in PEP479.
>
>
> But the docs didn't recommend raising StopIteration to exit a generator.
> The docs implied it by omission, or at least we're ambiguous about what
> would happen, and you could test it and see that it did, which meant some
> people took advantage of it to do things like using a
> StopIteration-throwing function in a comprehension filter clause (with
> caveats about listcomp vs. genexpr and 2.x vs. 3.x and most readers having
> to guess why it works). But surely breaking that isn't the same as breaking
> code that's been explicitly stated to work, and used as sample code, for
> decades.
>
> Although I'm not sure *StopIteration* has ever been documented as one way
> to exit generator. And I admit that an unintentional AttributeError in
> __getattribute__ is a rare case if ever.
>
>
> That's my point: "fixing" __getattribute__ to eliminate a "rare case if
> ever" bug, while also introducing a possibly less-rare backward
> compatibility problem, seems like a terrible idea. Of course that isn't
> your intention; it's just a side effect of trying to eliminate a more
> common bug, with probably rarer intentional uses, in descriptors. But to
> me, that implies that any solution that can fix descriptors without also
> "fixing" __getattribute__ is a lot better. And, as you imply in the bug
> report, this could be done by having either the descriptor mechanism
> itself, or the object.__getattribute__ implementation, handle
> AttributeError from descriptor lookup by reraising it as something else.
>
> But still, this is all a minor side issue about choosing between your
> different variations. The big problem is that I think all of your solutions
> make it too hard to trigger __getattr__ from a descriptor when you really
> _do_ want to do so. Maybe you don't want to do so very often, but is it
> really so rare that we can justify making it impossible? (Especially
> considering the backward-compat issues for any code that's been doing that
> for years...)
>
> You didn't comment on the alternative I suggested; would it not satisfy
> your needs, or have some other problem that makes it unacceptable?
>
> 2015-11-28 2:34 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:
>
>> It seems like you can get some of the benefits of this proposal without
>> backward compat issues.
>>
>> Instead of changing things so AttributeError from __getattribute__ or a
>> descriptor no longer calls __getattr__, just add a new subclass that
>> doesn't, and change the Descriptor HOWTO to suggest using that subclass
>> (with a bit of discussion that says you can use AttributeError if you want
>> to trigger __getattr__, but usually you won't).
>>
>> That wouldn't fix any current code, but it also wouldn't break any code
>> that intentionally uses the features as documented. And it would make it
>> easy to write correct new code.
>>
>> One more thing:
>>
>> Without descriptor, unexpected AttributeError could only come from overriding __getattribute__, which is a rare case, although still an imperfection.
>>
>> Is that really an imperfection? It says right there in the docs for
>> __getattribute__ that you can delegate to __getattr__ by raising
>> AttributeError, so if someone does that, presumably it's intentional. It's
>> not like the case with descriptors, where you have to think through the
>> interaction of multiple features to figure out that raising an
>> AttributeError will call __getattr__, and therefore many such uses are
>> probably bugs.
>>
>> Sent from my iPhone
>>
>> On Nov 27, 2015, at 09:49, ?? <wjun77 at gmail.com> wrote:
>>
>> Hello everyone:
>>
>> I'm suggesting a modification to the AttributeError/__getattr__
>> mechanism, see issue25634:
>> http://bugs.python.org/issue25634
>>
>> I used __getattr__ sometimes, and descriptor especially property is so
>> widely used. I wonder whether someone had encountered the same problem with
>> me.
>>
>> However, this is a complicated problem including performance issues, and
>> backward compatibility.
>>
>> Thanks for your attention,
>> Jun Wang
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151128/5c93f2dd/attachment.html>

From rosuav at gmail.com  Fri Nov 27 18:27:57 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 28 Nov 2015 10:27:57 +1100
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
References: <CAPM967SG01N5L3un-tKozAg9N32kACsXYyW=rfeuZCN=XsC+9A@mail.gmail.com>
 <CED82D87-A423-4874-82CB-97E3E1562A0F@yahoo.com>
 <CAPM967Tq8DcgRN_ScyrAFRoAX6h6_J3JNcfNd20dUxcYwNi2Mg@mail.gmail.com>
 <C7D754EA-DECC-452F-BEE5-78F8ADD6E1E3@yahoo.com>
 <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
Message-ID: <CAPTjJmqCwfHqSCLe+Semc2RpD-eOOOmbdPPEeV_xgD9QC-OCpw@mail.gmail.com>

On Sat, Nov 28, 2015 at 10:23 AM, ?? <wjun77 at gmail.com> wrote:
> However, widget.item may be None, while e.g. there are four widgets but only
> three items in total. In this case I should fill the area with white. But in
> this version of show, I just FORGET item can be None. So the traceback
> infomation: 'Window' object has no attribute 'backgroundImg'. In fact it
> takes a while before I find the cause is the AttributeError/__getattr__
> mechanism.

Hmm. A possibly more general solution is to declare that a function
mustn't ever raise a certain exception.

from functools import wraps
def dontraise(exc):
    def wrapper(f):
        @wraps(f)
        def inner(*a,**kw):
            try:
                f(*a,**kw)
            except exc as e:
                raise RuntimeError from e
        return inner
    return wrapper

class A():
    def __init__(self, x=None):
        self.x = x

    @property
    @dontraise(AttributeError)
    def t(self):
        return self.x.t

    def __getattr__(self, name):
        return 'default'

print(A().t)


By guarding your function with dontraise(AttributeError), you declare
that any AttributeError it raises must not leak out. Same as the
property change, but not bound to the property class itself.

ChrisA

From steve at pearwood.info  Fri Nov 27 22:46:09 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 28 Nov 2015 14:46:09 +1100
Subject: [Python-ideas] add a single __future__ for py3?
In-Reply-To: <0540E031-580B-4ED7-9E9D-7E37C84959E4@mit.edu>
References: <0540E031-580B-4ED7-9E9D-7E37C84959E4@mit.edu>
Message-ID: <20151128034609.GU3821@ando.pearwood.info>

On Fri, Nov 27, 2015 at 07:29:36PM +0000, Michael Scott Cuthbert wrote:

> (an old list topic that I accidentally sent to the Google Groups list)

I'm confused -- your subject line talks about a single __future__ for 
Python 3, but you don't talk about that in the body of the post. I'm not 
sure what the comment about Google Groups is supposed to mean. If you 
posted something there, most of us aren't going to see it, you'll need 
to re-post it here, not just tell us you posted it on Google Groups.


As for your secondary point:

> One smaller request is to remove 'barry_as_FLUFL' from 
> "__future__.all_feature_names".  It's a great practical joke, but it 
> bit me hard on a project where I check that, for Py2/Py3 consistency, 
> each module imports all relevant __future__ names, and ensures that 
> doctests on Py2 are run with them on.  I figured that I could iterate 
> through .all_feature_names and turn them all on except 
> unicode_literals and be safe, 

Hmmm, well, no, that's not really safe (as you've found out), 
particularly if you intend your code to be forward-compatible. 
Barry_as_FLUFL is a joke, but it demonstrates a real problem with your 
approach. Should some future version of Python add a new __future__ 
feature, and you run your code under that new version, you will 
unintentionally turn the feature on when your code is not expecting it, 
and likely break.

To put it another way... suppose Guido had a change of heart about != 
and decided that <> really was better, and so Barry_as_FLUFL was spelled 
"proper_ne" and was intended as a real feature, not a joke. Your code 
would still have broken, but removing "proper_ne" from all_feature_names 
would certainly not be an option.

The problem isn't that Barry_as_FLUFL is listed, but that you blindly 
applied all features without knowing what they are. Don't do that, it 
isn't safe.


-- 
Steve

From abarnert at yahoo.com  Fri Nov 27 22:52:33 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 28 Nov 2015 03:52:33 +0000 (UTC)
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
References: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
Message-ID: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>

On Friday, November 27, 2015 3:23 PM, ?? <wjun77 at gmail.com> wrote:

>> Do you have any examples that actually do demonstrate the problem to be solved?

>So you want more details about AttributeError in property?

No. I'm assuming Paul wanted an example that demonstrates the problem (a @property or other descriptor that raises or passes AttributeError, and a __getattr__ that blindly returns a value for anything). Just like your toy example on the bug tracker does, but realistic code rather than a toy example.

What you've provided is an example that doesn't demonstrate the problem at all. The code below doesn't even have a __getattr__ in it, and it does exactly what you should expect it to do (assuming you fill in the missing bits in any reasonable way), so it can't possibly demonstrate why interactions with __getattr__ are a problem.

> I thought property is widely used, and AttributeError occurs at all times. Maybe I've used property too heavy.>In the window example, a simplified demonstration: 
>
>class Window():
>    @property
>    def backgroundImg(self): 
>        if self._backgroundImg is None: #need update, while the number of items changes
>            self.set_backgroundImg()
>        return self._backgroundImg
>
>    def set_backgroundImg(self):
>        self._backgroundImg = loadImg('white.bmp')
>        for widget in self.widgets:
>            widget.show(self._backgroundImg)
>
>Class Widget():
>    def show(self, img):
>        img.draw(self.item.img, self.pos)
>

>However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of show, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'.

No, that can't possibly be your problem. If that were the case, the AttributeError will say that 'NoneType' object has no attribute 'img'. And the traceback would run from the Widget.show method, where the self.item.img is, back up the chain through your @property method.

I'm guessing your actual problem is that you forgot to set self._backgroundImg = None somewhere (e.g., in the __init__ method). In that case, you would get an error that looks more like the one you're claiming to get (but the attribute mentioned is '_backgroundImg', not 'backgroundImg'), with only one level of traceback and everything.


Or maybe there's a typo in your actual code, and you really don't have a 'backgroundImg' at all on Window objects; that would give exactly the error you're describing.

No matter which case it is, the problem has nothing to do with @property or descriptors in general, or with __getattr__ (obviously, since there is no __getattr__ in the code), much less with the interaction between them, so any fix to that interaction couldn't possibly help this example.


> In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism.


Since that isn't the cause, it would be bad if Python pointed you to look in that direction sooner...
>> But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades.
>I don't know this is such a severe problem. I used to think raising AttributeError in __getattribute__ to trigger __getattr__ is rare.
>
>> any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better

>In practice I don't concern __getattribute__. But in my opinion it's better to 'fix' this in python4.

Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit.
>> all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so
>This is a big problem, OK.
>
>> You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable?
>I don't quite understand, you mean adding a subclass of object with the only difference of this behavior?


No, adding a subclass of AttributeError, much like the one you mentioned in the bug report, but with the opposite meaning: the existing AttributeError continues to trigger __getattr__, but the new subclass doesn't. This makes it trivial to write new code that doesn't accidentally trigger __getattr__, without breaking old code (or rare new code) that wants to trigger __getattr__.

The code currently does something like this pseudocode:

    try:
        val = obj.__getattribute__(name)
    except AttributeError:
        __getattr__ = getattr(type(obj), '__getattr__', None)
        if __getattr__: return __getattr__(name)

I'm cheating a bit, but you get the idea. The problem is that we have no idea whether __getattribute__ failed to find anything (in which case we definitely want __getattr__ called), or found a descriptor whose __get__ raised an AttributeError (in which case we may not--e.g., a write-only attribute should not all through to __getattr__).

My suggestion is to change it like this:


    try:
        val = obj.__getattribute__(name)
    except AttributeDynamicError:
        raise
    except AttributeError:
        __getattr__ = getattr(type(obj), '__getattr__', None)
        if __getattr__: return __getattr__(name)


Now, if __getattribute__ found a descriptor whose __get__ raised an AttributeDynamicError, that passes on to the user code. (And, since it's a subclass of AttributeError, the user code should have no problem handling it.) And the Descriptor HOWTO will be changed to suggest raising AttributeDynamicError, except when you explicitly want it to call __getattr__, which you usually don't. And examples like simulating a write-only attribute will raise AttributeDynamicError. And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part).

So, new code can easily be written to act the way you want, but existing code using descriptors that intentionally raise or pass an AttributeError continues to work the same way it always has, and new code that does the same can also be written easily.

Obviously, the downside of any backward-compat-friendly change is that someone who has old code with a hidden bug they didn't know about will still have that same bug in Python 3.7; they have to change their code to take advantage of the fix. But I don't think that's a serious problem. (Especially if we decide @property is buggy and should be changed--most people who are writing actual custom descriptors, not just using the ones in the stdlib, probably understand this stuff.)

From abarnert at yahoo.com  Fri Nov 27 23:03:25 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 28 Nov 2015 04:03:25 +0000 (UTC)
Subject: [Python-ideas] add a single __future__ for py3?
In-Reply-To: <20151128034609.GU3821@ando.pearwood.info>
References: <20151128034609.GU3821@ando.pearwood.info>
Message-ID: <992758577.9555647.1448683405221.JavaMail.yahoo@mail.yahoo.com>

On Friday, November 27, 2015 7:46 PM, Steven D'Aprano <steve at pearwood.info> wrote:

> > On Fri, Nov 27, 2015 at 07:29:36PM +0000, Michael Scott Cuthbert wrote:
> 
>>  (an old list topic that I accidentally sent to the Google Groups list)
> 
> I'm confused -- your subject line talks about a single __future__ for 
> Python 3, but you don't talk about that in the body of the post. I'm not 
> 
> sure what the comment about Google Groups is supposed to mean. If you 
> posted something there, most of us aren't going to see it, you'll need 
> to re-post it here, not just tell us you posted it on Google Groups.
> 
> 
> As for your secondary point:
> 
>>  One smaller request is to remove 'barry_as_FLUFL' from 
>>  "__future__.all_feature_names".  It's a great practical joke, 
> but it 
>>  bit me hard on a project where I check that, for Py2/Py3 consistency, 
>>  each module imports all relevant __future__ names, and ensures that 
>>  doctests on Py2 are run with them on.  I figured that I could iterate 
>>  through .all_feature_names and turn them all on except 
>>  unicode_literals and be safe, 
> 
> Hmmm, well, no, that's not really safe (as you've found out), 
> particularly if you intend your code to be forward-compatible. 
> Barry_as_FLUFL is a joke, but it demonstrates a real problem with your 
> approach. Should some future version of Python add a new __future__ 
> feature, and you run your code under that new version, you will 
> unintentionally turn the feature on when your code is not expecting it, 
> and likely break.
> 
> To put it another way... suppose Guido had a change of heart about != 
> and decided that <> really was better, and so Barry_as_FLUFL was spelled 
> "proper_ne" and was intended as a real feature, not a joke. Your code 
> would still have broken, but removing "proper_ne" from 
> all_feature_names 
> would certainly not be an option.
> 
> The problem isn't that Barry_as_FLUFL is listed, but that you blindly 
> applied all features without knowing what they are. Don't do that, it 
> isn't safe.

Assuming he's doing this in a test suite, rather than in real code, I think it is safe, and maybe even a good idea. His test suite passes in 3.6, but in 3.7, it fails because some of his code doesn't work correctly with "from futures import thing_nobody_expects". That implies that his code may fail in 3.8, so he'd better take a look at thing_nobody_expects. And it's good that he finds this out now, instead of 18 months from now when his first users upgrade to 3.8 and his code breaks.

That being said, whatever test system he uses had better let him mark a future as disabled or known-bad or whatever. Otherwise, he has to put off any commits at all until he fixes the thing_nobody_expects stuff. And realistically, he probably has a lot of higher-priority bugs to take care of in the intervening 18 months. And if he can do that temporarily for thing_nobody_expects, I don't see why he can't do that permanently for Barry_as_FLUFL. So, I don't think Python needs to change; his test tool does. (In fact, Barry_as_FLUFL serves as a good test for test tools' future testing.:))

Also, the fact that he's already got code to "turn them all on except unicode_literals" implies that he doesn't really have a problem in the first place; he just needs to change that code to turn them all on except unicode_literals and Barry_as_FLUFL.

From wjun77 at gmail.com  Fri Nov 27 23:25:44 2015
From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=)
Date: Sat, 28 Nov 2015 12:25:44 +0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>
References: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
 <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>

I don't have realistic code that blindly returns a value now, but showing
unhelpful and confusing traceback messages.

> The code below doesn't even have a __getattr__ in it
>> In *the* window example
I've already post the __getattr__-related code when discussing the use case
of __getattr__, so I omitted those code here. Sorry for the misleading
code. Here's the complete version:

class ActiveState(State):
    @staticmethod
    def rightClick(self):
        print('right clicked')
class InactiveState(State):
    @staticmethod
    def rightClick(self):
        pass

class Window():
    def __init__(self):
        self.__state = ActiveState
    def __getattr__(self, name):
        try:
            return partial(getattr(self.__state, name), self)
        except AttributeError:
            raise AttributeError("'{}' object has no attribute
'{}'".format(self.__class__.__name__, name)) from None

    @property
    def backgroundImg(self):
        if self._backgroundImg is None: #need update, while the number of
items changes
            self.set_backgroundImg()
        return self._backgroundImg

    def set_backgroundImg(self):
        self._backgroundImg = loadImg('white.bmp')
        for widget in self.widgets:
            widget.show(self._backgroundImg)

Class Widget():
    def show(self, img):
        img.draw(self.item.img, self.pos)


> Why? If you think that erroneous uses are rare or nonexistent, while
intentional uses are rare but not nonexistent, "fixing" it means breaking
code gratuitously for no benefit.
I mean if we don't consider any backward compatibility, maybe when creating
a new language other than python, I think it's a better choice to 'fix' it.
Only my personal opinion.

I have something urgent to do now, so I'll read the rest part of your post
carefully later. Anyway thanks for your attention.

2015-11-28 11:52 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:

> On Friday, November 27, 2015 3:23 PM, ?? <wjun77 at gmail.com> wrote:
>
> >> Do you have any examples that actually do demonstrate the problem to be
> solved?
>
> >So you want more details about AttributeError in property?
>
> No. I'm assuming Paul wanted an example that demonstrates the problem (a
> @property or other descriptor that raises or passes AttributeError, and a
> __getattr__ that blindly returns a value for anything). Just like your toy
> example on the bug tracker does, but realistic code rather than a toy
> example.
>
> What you've provided is an example that doesn't demonstrate the problem at
> all. The code below doesn't even have a __getattr__ in it, and it does
> exactly what you should expect it to do (assuming you fill in the missing
> bits in any reasonable way), so it can't possibly demonstrate why
> interactions with __getattr__ are a problem.
>
> > I thought property is widely used, and AttributeError occurs at all
> times. Maybe I've used property too heavy.>In the window example, a
> simplified demonstration:
> >
> >class Window():
> >    @property
> >    def backgroundImg(self):
> >        if self._backgroundImg is None: #need update, while the number of
> items changes
> >            self.set_backgroundImg()
> >        return self._backgroundImg
> >
> >    def set_backgroundImg(self):
> >        self._backgroundImg = loadImg('white.bmp')
> >        for widget in self.widgets:
> >            widget.show(self._backgroundImg)
> >
> >Class Widget():
> >    def show(self, img):
> >        img.draw(self.item.img, self.pos)
> >
>
> >However, widget.item may be None, while e.g. there are four widgets but
> only three items in total. In this case I should fill the area with white.
> But in this version of show, I just FORGET item can be None. So the
> traceback infomation: 'Window' object has no attribute 'backgroundImg'.
>
> No, that can't possibly be your problem. If that were the case, the
> AttributeError will say that 'NoneType' object has no attribute 'img'. And
> the traceback would run from the Widget.show method, where the
> self.item.img is, back up the chain through your @property method.
>
> I'm guessing your actual problem is that you forgot to set
> self._backgroundImg = None somewhere (e.g., in the __init__ method). In
> that case, you would get an error that looks more like the one you're
> claiming to get (but the attribute mentioned is '_backgroundImg', not
> 'backgroundImg'), with only one level of traceback and everything.
>
>
> Or maybe there's a typo in your actual code, and you really don't have a
> 'backgroundImg' at all on Window objects; that would give exactly the error
> you're describing.
>
> No matter which case it is, the problem has nothing to do with @property
> or descriptors in general, or with __getattr__ (obviously, since there is
> no __getattr__ in the code), much less with the interaction between them,
> so any fix to that interaction couldn't possibly help this example.
>
>
> > In fact it takes a while before I find the cause is the
> AttributeError/__getattr__ mechanism.
>
>
> Since that isn't the cause, it would be bad if Python pointed you to look
> in that direction sooner...
> >> But surely breaking that isn't the same as breaking code that's been
> explicitly stated to work, and used as sample code, for decades.
> >I don't know this is such a severe problem. I used to think raising
> AttributeError in __getattribute__ to trigger __getattr__ is rare.
> >
> >> any solution that can fix descriptors without also "fixing"
> __getattribute__ is a lot better
>
> >In practice I don't concern __getattribute__. But in my opinion it's
> better to 'fix' this in python4.
>
> Why? If you think that erroneous uses are rare or nonexistent, while
> intentional uses are rare but not nonexistent, "fixing" it means breaking
> code gratuitously for no benefit.
> >> all of your solutions make it too hard to trigger __getattr__ from a
> descriptor when you really _do_ want to do so
> >This is a big problem, OK.
> >
> >> You didn't comment on the alternative I suggested; would it not satisfy
> your needs, or have some other problem that makes it unacceptable?
> >I don't quite understand, you mean adding a subclass of object with the
> only difference of this behavior?
>
>
> No, adding a subclass of AttributeError, much like the one you mentioned
> in the bug report, but with the opposite meaning: the existing
> AttributeError continues to trigger __getattr__, but the new subclass
> doesn't. This makes it trivial to write new code that doesn't accidentally
> trigger __getattr__, without breaking old code (or rare new code) that
> wants to trigger __getattr__.
>
> The code currently does something like this pseudocode:
>
>     try:
>         val = obj.__getattribute__(name)
>     except AttributeError:
>         __getattr__ = getattr(type(obj), '__getattr__', None)
>         if __getattr__: return __getattr__(name)
>
> I'm cheating a bit, but you get the idea. The problem is that we have no
> idea whether __getattribute__ failed to find anything (in which case we
> definitely want __getattr__ called), or found a descriptor whose __get__
> raised an AttributeError (in which case we may not--e.g., a write-only
> attribute should not all through to __getattr__).
>
> My suggestion is to change it like this:
>
>
>     try:
>         val = obj.__getattribute__(name)
>     except AttributeDynamicError:
>         raise
>     except AttributeError:
>         __getattr__ = getattr(type(obj), '__getattr__', None)
>         if __getattr__: return __getattr__(name)
>
>
> Now, if __getattribute__ found a descriptor whose __get__ raised an
> AttributeDynamicError, that passes on to the user code. (And, since it's a
> subclass of AttributeError, the user code should have no problem handling
> it.) And the Descriptor HOWTO will be changed to suggest raising
> AttributeDynamicError, except when you explicitly want it to call
> __getattr__, which you usually don't. And examples like simulating a
> write-only attribute will raise AttributeDynamicError. And maybe @property
> will automatically convert any AttributeError to AttributeDynamicError (not
> sure about that part).
>
> So, new code can easily be written to act the way you want, but existing
> code using descriptors that intentionally raise or pass an AttributeError
> continues to work the same way it always has, and new code that does the
> same can also be written easily.
>
> Obviously, the downside of any backward-compat-friendly change is that
> someone who has old code with a hidden bug they didn't know about will
> still have that same bug in Python 3.7; they have to change their code to
> take advantage of the fix. But I don't think that's a serious problem.
> (Especially if we decide @property is buggy and should be changed--most
> people who are writing actual custom descriptors, not just using the ones
> in the stdlib, probably understand this stuff.)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151128/cc354dde/attachment-0001.html>

From abarnert at yahoo.com  Sat Nov 28 01:13:12 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 27 Nov 2015 22:13:12 -0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>
References: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
 <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>
 <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>
Message-ID: <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com>

On Nov 27, 2015, at 20:25, ?? <wjun77 at gmail.com> wrote:
> 
> I don't have realistic code that blindly returns a value now, but showing unhelpful and confusing traceback messages. 

Well, yes. If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing. But just don't do that, and you don't have that problem.

(That being said, the way you've written it, I think the new AttributeError will contain the old one, so you should still be able to debug it--unless you're using Python 2, but in that case, obviously you need to migrating to Python 3... But anyway, it would be simpler to just not write code that makes debugging harder and provides no benefits.)

As a side note: why are you writing @staticmethods that take a self parameter? The whole point of static methods is that they don't get passed self. If you need to pass in the "owner", calling it "self" is misleading; give it a name that makes it clear what's being passed. But it looks like you don't even need that, since you never use it. Is this code by chance an attempt to directly port some Java code to Python?

> > The code below doesn't even have a __getattr__ in it
> >> In the window example
> I've already post the __getattr__-related code when discussing the use case of __getattr__, so I omitted those code here.
> Sorry for the misleading code. Here's the complete version:
> 
> class ActiveState(State):
>     @staticmethod
>     def rightClick(self):
>         print('right clicked')
> class InactiveState(State):
>     @staticmethod
>     def rightClick(self):
>         pass
> 
> class Window():
>     def __init__(self):
>         self.__state = ActiveState
>     def __getattr__(self, name):
>         try:
>             return partial(getattr(self.__state, name), self)
>         except AttributeError:
>             raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None
> 
>     @property
>     def backgroundImg(self):
>         if self._backgroundImg is None: #need update, while the number of items changes
>             self.set_backgroundImg()
>         return self._backgroundImg
> 
>     def set_backgroundImg(self):
>         self._backgroundImg = loadImg('white.bmp')
>         for widget in self.widgets:
>             widget.show(self._backgroundImg)
> 
> Class Widget():
>     def show(self, img):
>         img.draw(self.item.img, self.pos)
> 
> 
> > Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit.
> I mean if we don't consider any backward compatibility, maybe when creating a new language other than python, I think it's a better choice to 'fix' it. Only my personal opinion. 
> 
> I have something urgent to do now, so I'll read the rest part of your post carefully later. Anyway thanks for your attention. 
> 
> 2015-11-28 11:52 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:
>> On Friday, November 27, 2015 3:23 PM, ?? <wjun77 at gmail.com> wrote:
>> 
>> >> Do you have any examples that actually do demonstrate the problem to be solved?
>> 
>> >So you want more details about AttributeError in property?
>> 
>> No. I'm assuming Paul wanted an example that demonstrates the problem (a @property or other descriptor that raises or passes AttributeError, and a __getattr__ that blindly returns a value for anything). Just like your toy example on the bug tracker does, but realistic code rather than a toy example.
>> 
>> What you've provided is an example that doesn't demonstrate the problem at all. The code below doesn't even have a __getattr__ in it, and it does exactly what you should expect it to do (assuming you fill in the missing bits in any reasonable way), so it can't possibly demonstrate why interactions with __getattr__ are a problem.
>> 
>> > I thought property is widely used, and AttributeError occurs at all times. Maybe I've used property too heavy.>In the window example, a simplified demonstration:
>> >
>> >class Window():
>> >    @property
>> >    def backgroundImg(self):
>> >        if self._backgroundImg is None: #need update, while the number of items changes
>> >            self.set_backgroundImg()
>> >        return self._backgroundImg
>> >
>> >    def set_backgroundImg(self):
>> >        self._backgroundImg = loadImg('white.bmp')
>> >        for widget in self.widgets:
>> >            widget.show(self._backgroundImg)
>> >
>> >Class Widget():
>> >    def show(self, img):
>> >        img.draw(self.item.img, self.pos)
>> >
>> 
>> >However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of show, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'.
>> 
>> No, that can't possibly be your problem. If that were the case, the AttributeError will say that 'NoneType' object has no attribute 'img'. And the traceback would run from the Widget.show method, where the self.item.img is, back up the chain through your @property method.
>> 
>> I'm guessing your actual problem is that you forgot to set self._backgroundImg = None somewhere (e.g., in the __init__ method). In that case, you would get an error that looks more like the one you're claiming to get (but the attribute mentioned is '_backgroundImg', not 'backgroundImg'), with only one level of traceback and everything.
>> 
>> 
>> Or maybe there's a typo in your actual code, and you really don't have a 'backgroundImg' at all on Window objects; that would give exactly the error you're describing.
>> 
>> No matter which case it is, the problem has nothing to do with @property or descriptors in general, or with __getattr__ (obviously, since there is no __getattr__ in the code), much less with the interaction between them, so any fix to that interaction couldn't possibly help this example.
>> 
>> 
>> > In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism.
>> 
>> 
>> Since that isn't the cause, it would be bad if Python pointed you to look in that direction sooner...
>> >> But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades.
>> >I don't know this is such a severe problem. I used to think raising AttributeError in __getattribute__ to trigger __getattr__ is rare.
>> >
>> >> any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better
>> 
>> >In practice I don't concern __getattribute__. But in my opinion it's better to 'fix' this in python4.
>> 
>> Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit.
>> >> all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so
>> >This is a big problem, OK.
>> >
>> >> You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable?
>> >I don't quite understand, you mean adding a subclass of object with the only difference of this behavior?
>> 
>> 
>> No, adding a subclass of AttributeError, much like the one you mentioned in the bug report, but with the opposite meaning: the existing AttributeError continues to trigger __getattr__, but the new subclass doesn't. This makes it trivial to write new code that doesn't accidentally trigger __getattr__, without breaking old code (or rare new code) that wants to trigger __getattr__.
>> 
>> The code currently does something like this pseudocode:
>> 
>>     try:
>>         val = obj.__getattribute__(name)
>>     except AttributeError:
>>         __getattr__ = getattr(type(obj), '__getattr__', None)
>>         if __getattr__: return __getattr__(name)
>> 
>> I'm cheating a bit, but you get the idea. The problem is that we have no idea whether __getattribute__ failed to find anything (in which case we definitely want __getattr__ called), or found a descriptor whose __get__ raised an AttributeError (in which case we may not--e.g., a write-only attribute should not all through to __getattr__).
>> 
>> My suggestion is to change it like this:
>> 
>> 
>>     try:
>>         val = obj.__getattribute__(name)
>>     except AttributeDynamicError:
>>         raise
>>     except AttributeError:
>>         __getattr__ = getattr(type(obj), '__getattr__', None)
>>         if __getattr__: return __getattr__(name)
>> 
>> 
>> Now, if __getattribute__ found a descriptor whose __get__ raised an AttributeDynamicError, that passes on to the user code. (And, since it's a subclass of AttributeError, the user code should have no problem handling it.) And the Descriptor HOWTO will be changed to suggest raising AttributeDynamicError, except when you explicitly want it to call __getattr__, which you usually don't. And examples like simulating a write-only attribute will raise AttributeDynamicError. And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part).
>> 
>> So, new code can easily be written to act the way you want, but existing code using descriptors that intentionally raise or pass an AttributeError continues to work the same way it always has, and new code that does the same can also be written easily.
>> 
>> Obviously, the downside of any backward-compat-friendly change is that someone who has old code with a hidden bug they didn't know about will still have that same bug in Python 3.7; they have to change their code to take advantage of the fix. But I don't think that's a serious problem. (Especially if we decide @property is buggy and should be changed--most people who are writing actual custom descriptors, not just using the ones in the stdlib, probably understand this stuff.)
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151127/41ff3ae1/attachment.html>

From wjun77 at gmail.com  Sun Nov 29 09:31:06 2015
From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=)
Date: Sun, 29 Nov 2015 22:31:06 +0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com>
References: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
 <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>
 <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>
 <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com>
Message-ID: <CAPM967SwVLH7BVD94s+16+oPcwS68JAPCD7NfH0Tg6Awbt6BPQ@mail.gmail.com>

>  If you catch an AttributeError and raise a different one that hides all
the relevant information, it will be unhelpful and confusing.
What matters is whether __getattr__ which hides all the relevant
information of any AttributeError is defined, not a default value returned
or any other behavior in __getattr__.

> all of your solutions make it too hard to trigger __getattr__ from a
descriptor when you really _do_ want to do so
If we do want to trigger __getattr__ from a descriptor, the
AttributeMissError solution seems feasible. Raising AttributeMissError in
descriptor triggers __getattr__.
(By the way, I used to think controlling the program flow by Exception is a
bad idea, and now I understand that it do be practical sometimes. But I
insist that AttributeError is too general to be used here.)

>  And maybe @property will automatically convert any AttributeError to
AttributeDynamicError (not sure about that part).
Then raising AttributeError in property intentionally 'when you really _do_
want to do so' to trigger __getattr__ will fail, right?
@property (or descriptor) converting AttributeError to
AttributeDynamicError, or __getattr__ converting AttributeMissError to
AttributeError. The former has the advantage of keeping __getattr__
triggered by raising AttributeError in __getattribute__ as documented for
decades. But I don't like this idea, because it's conceptually ugly. If you
don't agree or understand, my reply can only be 'we have different
aesthetic'. I personally prefer no change to this.


> why are you writing @staticmethods that take a self parameter?
'self' is more familiar to type, and I can change the class of the method
between the Window class and the ActiveState class without modifying the
name between 'self' and 'owner'. In real code there's no need to write
@staticmethod explicitly; the metaclass of State will automatically change
any normal method to staticmethod. And ActiveState is defined in Window.


Hmm, it seems that no one feels necessary to make change other than myself.
I guess it's because no one uses __getattr__ in practice at all.
Anyway, now that I know there is a 'pit' there, it won't bother me too much
in the future, either by using the *dontraise* decorator by Chris or any
other means.

2015-11-28 14:13 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:

> On Nov 27, 2015, at 20:25, ?? <wjun77 at gmail.com> wrote:
>
> I don't have realistic code that blindly returns a value now, but showing
> unhelpful and confusing traceback messages.
>
>
> Well, yes. If you catch an AttributeError and raise a different one that
> hides all the relevant information, it will be unhelpful and confusing. But
> just don't do that, and you don't have that problem.
>
> (That being said, the way you've written it, I think the new
> AttributeError will contain the old one, so you should still be able to
> debug it--unless you're using Python 2, but in that case, obviously you
> need to migrating to Python 3... But anyway, it would be simpler to just
> not write code that makes debugging harder and provides no benefits.)
>
> As a side note: why are you writing @staticmethods that take a self
> parameter? The whole point of static methods is that they don't get passed
> self. If you need to pass in the "owner", calling it "self" is misleading;
> give it a name that makes it clear what's being passed. But it looks like
> you don't even need that, since you never use it. Is this code by chance an
> attempt to directly port some Java code to Python?
>
> > The code below doesn't even have a __getattr__ in it
> >> In *the* window example
> I've already post the __getattr__-related code when discussing the use
> case of __getattr__, so I omitted those code here.
>
> Sorry for the misleading code. Here's the complete version:
>
> class ActiveState(State):
>     @staticmethod
>     def rightClick(self):
>         print('right clicked')
> class InactiveState(State):
>     @staticmethod
>     def rightClick(self):
>         pass
>
> class Window():
>     def __init__(self):
>         self.__state = ActiveState
>     def __getattr__(self, name):
>         try:
>             return partial(getattr(self.__state, name), self)
>         except AttributeError:
>             raise AttributeError("'{}' object has no attribute
> '{}'".format(self.__class__.__name__, name)) from None
>
>     @property
>     def backgroundImg(self):
>         if self._backgroundImg is None: #need update, while the number of
> items changes
>             self.set_backgroundImg()
>         return self._backgroundImg
>
>     def set_backgroundImg(self):
>         self._backgroundImg = loadImg('white.bmp')
>         for widget in self.widgets:
>             widget.show(self._backgroundImg)
>
> Class Widget():
>     def show(self, img):
>         img.draw(self.item.img, self.pos)
>
>
> > Why? If you think that erroneous uses are rare or nonexistent, while
> intentional uses are rare but not nonexistent, "fixing" it means breaking
> code gratuitously for no benefit.
> I mean if we don't consider any backward compatibility, maybe when
> creating a new language other than python, I think it's a better choice to
> 'fix' it. Only my personal opinion.
>
> I have something urgent to do now, so I'll read the rest part of your post
> carefully later. Anyway thanks for your attention.
>
> 2015-11-28 11:52 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:
>
>> On Friday, November 27, 2015 3:23 PM, ?? <wjun77 at gmail.com> wrote:
>>
>> >> Do you have any examples that actually do demonstrate the problem to
>> be solved?
>>
>> >So you want more details about AttributeError in property?
>>
>> No. I'm assuming Paul wanted an example that demonstrates the problem (a
>> @property or other descriptor that raises or passes AttributeError, and a
>> __getattr__ that blindly returns a value for anything). Just like your toy
>> example on the bug tracker does, but realistic code rather than a toy
>> example.
>>
>> What you've provided is an example that doesn't demonstrate the problem
>> at all. The code below doesn't even have a __getattr__ in it, and it does
>> exactly what you should expect it to do (assuming you fill in the missing
>> bits in any reasonable way), so it can't possibly demonstrate why
>> interactions with __getattr__ are a problem.
>>
>> > I thought property is widely used, and AttributeError occurs at all
>> times. Maybe I've used property too heavy.>In the window example, a
>> simplified demonstration:
>> >
>> >class Window():
>> >    @property
>> >    def backgroundImg(self):
>> >        if self._backgroundImg is None: #need update, while the number
>> of items changes
>> >            self.set_backgroundImg()
>> >        return self._backgroundImg
>> >
>> >    def set_backgroundImg(self):
>> >        self._backgroundImg = loadImg('white.bmp')
>> >        for widget in self.widgets:
>> >            widget.show(self._backgroundImg)
>> >
>> >Class Widget():
>> >    def show(self, img):
>> >        img.draw(self.item.img, self.pos)
>> >
>>
>> >However, widget.item may be None, while e.g. there are four widgets but
>> only three items in total. In this case I should fill the area with white.
>> But in this version of show, I just FORGET item can be None. So the
>> traceback infomation: 'Window' object has no attribute 'backgroundImg'.
>>
>> No, that can't possibly be your problem. If that were the case, the
>> AttributeError will say that 'NoneType' object has no attribute 'img'. And
>> the traceback would run from the Widget.show method, where the
>> self.item.img is, back up the chain through your @property method.
>>
>> I'm guessing your actual problem is that you forgot to set
>> self._backgroundImg = None somewhere (e.g., in the __init__ method). In
>> that case, you would get an error that looks more like the one you're
>> claiming to get (but the attribute mentioned is '_backgroundImg', not
>> 'backgroundImg'), with only one level of traceback and everything.
>>
>>
>> Or maybe there's a typo in your actual code, and you really don't have a
>> 'backgroundImg' at all on Window objects; that would give exactly the error
>> you're describing.
>>
>> No matter which case it is, the problem has nothing to do with @property
>> or descriptors in general, or with __getattr__ (obviously, since there is
>> no __getattr__ in the code), much less with the interaction between them,
>> so any fix to that interaction couldn't possibly help this example.
>>
>>
>> > In fact it takes a while before I find the cause is the
>> AttributeError/__getattr__ mechanism.
>>
>>
>> Since that isn't the cause, it would be bad if Python pointed you to look
>> in that direction sooner...
>> >> But surely breaking that isn't the same as breaking code that's been
>> explicitly stated to work, and used as sample code, for decades.
>> >I don't know this is such a severe problem. I used to think raising
>> AttributeError in __getattribute__ to trigger __getattr__ is rare.
>> >
>> >> any solution that can fix descriptors without also "fixing"
>> __getattribute__ is a lot better
>>
>> >In practice I don't concern __getattribute__. But in my opinion it's
>> better to 'fix' this in python4.
>>
>> Why? If you think that erroneous uses are rare or nonexistent, while
>> intentional uses are rare but not nonexistent, "fixing" it means breaking
>> code gratuitously for no benefit.
>> >> all of your solutions make it too hard to trigger __getattr__ from a
>> descriptor when you really _do_ want to do so
>> >This is a big problem, OK.
>> >
>> >> You didn't comment on the alternative I suggested; would it not
>> satisfy your needs, or have some other problem that makes it unacceptable?
>> >I don't quite understand, you mean adding a subclass of object with the
>> only difference of this behavior?
>>
>>
>> No, adding a subclass of AttributeError, much like the one you mentioned
>> in the bug report, but with the opposite meaning: the existing
>> AttributeError continues to trigger __getattr__, but the new subclass
>> doesn't. This makes it trivial to write new code that doesn't accidentally
>> trigger __getattr__, without breaking old code (or rare new code) that
>> wants to trigger __getattr__.
>>
>> The code currently does something like this pseudocode:
>>
>>     try:
>>         val = obj.__getattribute__(name)
>>     except AttributeError:
>>         __getattr__ = getattr(type(obj), '__getattr__', None)
>>         if __getattr__: return __getattr__(name)
>>
>> I'm cheating a bit, but you get the idea. The problem is that we have no
>> idea whether __getattribute__ failed to find anything (in which case we
>> definitely want __getattr__ called), or found a descriptor whose __get__
>> raised an AttributeError (in which case we may not--e.g., a write-only
>> attribute should not all through to __getattr__).
>>
>> My suggestion is to change it like this:
>>
>>
>>     try:
>>         val = obj.__getattribute__(name)
>>     except AttributeDynamicError:
>>         raise
>>     except AttributeError:
>>         __getattr__ = getattr(type(obj), '__getattr__', None)
>>         if __getattr__: return __getattr__(name)
>>
>>
>> Now, if __getattribute__ found a descriptor whose __get__ raised an
>> AttributeDynamicError, that passes on to the user code. (And, since it's a
>> subclass of AttributeError, the user code should have no problem handling
>> it.) And the Descriptor HOWTO will be changed to suggest raising
>> AttributeDynamicError, except when you explicitly want it to call
>> __getattr__, which you usually don't. And examples like simulating a
>> write-only attribute will raise AttributeDynamicError. And maybe @property
>> will automatically convert any AttributeError to AttributeDynamicError (not
>> sure about that part).
>>
>> So, new code can easily be written to act the way you want, but existing
>> code using descriptors that intentionally raise or pass an AttributeError
>> continues to work the same way it always has, and new code that does the
>> same can also be written easily.
>>
>> Obviously, the downside of any backward-compat-friendly change is that
>> someone who has old code with a hidden bug they didn't know about will
>> still have that same bug in Python 3.7; they have to change their code to
>> take advantage of the fix. But I don't think that's a serious problem.
>> (Especially if we decide @property is buggy and should be changed--most
>> people who are writing actual custom descriptors, not just using the ones
>> in the stdlib, probably understand this stuff.)
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151129/7afdf8c0/attachment-0001.html>

From abarnert at yahoo.com  Sun Nov 29 11:07:22 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 29 Nov 2015 08:07:22 -0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967SwVLH7BVD94s+16+oPcwS68JAPCD7NfH0Tg6Awbt6BPQ@mail.gmail.com>
References: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
 <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>
 <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>
 <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com>
 <CAPM967SwVLH7BVD94s+16+oPcwS68JAPCD7NfH0Tg6Awbt6BPQ@mail.gmail.com>
Message-ID: <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com>

On Nov 29, 2015, at 06:31, ?? <wjun77 at gmail.com> wrote:
> 
> >  If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing.
> What matters is whether __getattr__ which hides all the relevant information of any AttributeError is defined, not a default value returned or any other behavior in __getattr__.

Your problem is that it took you hours to hunt down where the AttributeError came from, because it gave useless information instead of useful information. The only reason that's happening is that you're handling any AttributeError by swallowing it and raising a new one with useless information. Just take that out, and your hours of debugging go away.

> > all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so
> If we do want to trigger __getattr__ from a descriptor, the AttributeMissError solution seems feasible. Raising AttributeMissError in descriptor triggers __getattr__. 

The difference, as I've already explained, is that your solution breaks backward compatibility, requiring anyone using such code to change it, while my variation leaves working code alone, only requiring new code to be written differently. All else being equal, the latter is obvious less disruptive a change and therefore better.

So, if you think your more-disruptive solution is better, that must mean all else is not equal. But you have to explain how. What's wrong with my version?

> (By the way, I used to think controlling the program flow by Exception is a bad idea, and now I understand that it do be practical sometimes. But I insist that AttributeError is too general to be used here.)

All the places exceptions are used for flow control in Python and its stdlib and pythonic code in general use general exceptions. Occasionally (as with generators raising StopIteration where they shouldn't be), this can lead to specific trouble, and that occasional specific trouble gets fixed. Maybe this is one of those cases. That would be two times in as many decades. That's not a good reason to change the general principle.

> >  And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part).
> Then raising AttributeError in property intentionally 'when you really _do_ want to do so' to trigger __getattr__ will fail, right?

The point is that @property is a high-level, simplified way to write a specific kind of descriptor. It may be perfectly reasonable that properties should never trigger __getattr__; after all, you can always fall back to writing your own descriptor (or even your own property-like descriptor factory--it's only a few lines of code) if you want to. (Again, I'm not sure one way or the other about this.)

> @property (or descriptor) converting

Nobody is suggestion descriptors do anything different here. The code would be explicit code in the implementation of @property (and in the Descriptor HOWTO section that shows how @property works).

> AttributeError to AttributeDynamicError, or __getattr__ converting AttributeMissError to AttributeError. The former has the advantage of keeping __getattr__ triggered by raising AttributeError in __getattribute__ as documented for decades. But I don't like this idea, because it's conceptually ugly. If you don't agree or understand, my reply can only be 'we have different aesthetic'. I personally prefer no change to this.
> 
> > why are you writing @staticmethods that take a self parameter?
> 'self' is more familiar to type, and I can change the class of the method between the Window class and the ActiveState class without modifying the name between 'self' and 'owner'. In real code there's no need to write @staticmethod explicitly; the metaclass of State will automatically change any normal method to staticmethod. And ActiveState is defined in Window.

"More familiar to type" is a bad reason to use something where it has the wrong meaning. You wouldn't try to write exponentiation as "*" because a single asterisk is more familiar than a double, so why would you use "self" instead of the right parameter name? The fact that in this case it only misleads human readers, instead of also misleading the interpreter, doesn't make it any less of a problem. (And having a metaclass that automatically changes methods to @staticmethod doesn't make a difference. Do you call the first parameter of a __new__ method "self" because you don't have to decorate it?)

> Hmm, it seems that no one feels necessary to make change other than myself. I guess it's because no one uses __getattr__ in practice at all.

First, I'm sure lots of people use __getattr__ all over the place. (Personally, I consider the ability to write simple dynamic proxy classes easier than any language except Smalltalk to be a major selling point for Python, and it's been a deciding factor over ObjC in at least one project.)

More importantly: you've had at least two people saying "I see the problem in principle, but I'd like to see real-life code where you're mixing properties that raise AttributeError and __getattr__", and another person trying to dig out details of your proposal and discuss alternatives. How do you interpret that as a lack of interest? If nobody else saw any point to your proposal, nobody would be responding at all.

> Anyway, now that I know there is a 'pit' there, it won't bother me too much in the future, either by using the dontraise decorator by Chris or any other means.
> 
> 2015-11-28 14:13 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:
>>> On Nov 27, 2015, at 20:25, ?? <wjun77 at gmail.com> wrote:
>>> 
>>> I don't have realistic code that blindly returns a value now, but showing unhelpful and confusing traceback messages.
>> 
>> Well, yes. If you catch an AttributeError and raise a different one that hides all the relevant information, it will be unhelpful and confusing. But just don't do that, and you don't have that problem.
>> 
>> (That being said, the way you've written it, I think the new AttributeError will contain the old one, so you should still be able to debug it--unless you're using Python 2, but in that case, obviously you need to migrating to Python 3... But anyway, it would be simpler to just not write code that makes debugging harder and provides no benefits.)
>> 
>> As a side note: why are you writing @staticmethods that take a self parameter? The whole point of static methods is that they don't get passed self. If you need to pass in the "owner", calling it "self" is misleading; give it a name that makes it clear what's being passed. But it looks like you don't even need that, since you never use it. Is this code by chance an attempt to directly port some Java code to Python?
>> 
>>> > The code below doesn't even have a __getattr__ in it
>>> >> In the window example
>>> I've already post the __getattr__-related code when discussing the use case of __getattr__, so I omitted those code here.
>>> Sorry for the misleading code. Here's the complete version:
>>> 
>>> class ActiveState(State):
>>>     @staticmethod
>>>     def rightClick(self):
>>>         print('right clicked')
>>> class InactiveState(State):
>>>     @staticmethod
>>>     def rightClick(self):
>>>         pass
>>> 
>>> class Window():
>>>     def __init__(self):
>>>         self.__state = ActiveState
>>>     def __getattr__(self, name):
>>>         try:
>>>             return partial(getattr(self.__state, name), self)
>>>         except AttributeError:
>>>             raise AttributeError("'{}' object has no attribute '{}'".format(self.__class__.__name__, name)) from None
>>> 
>>>     @property
>>>     def backgroundImg(self):
>>>         if self._backgroundImg is None: #need update, while the number of items changes
>>>             self.set_backgroundImg()
>>>         return self._backgroundImg
>>> 
>>>     def set_backgroundImg(self):
>>>         self._backgroundImg = loadImg('white.bmp')
>>>         for widget in self.widgets:
>>>             widget.show(self._backgroundImg)
>>> 
>>> Class Widget():
>>>     def show(self, img):
>>>         img.draw(self.item.img, self.pos)
>>> 
>>> 
>>> > Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit.
>>> I mean if we don't consider any backward compatibility, maybe when creating a new language other than python, I think it's a better choice to 'fix' it. Only my personal opinion. 
>>> 
>>> I have something urgent to do now, so I'll read the rest part of your post carefully later. Anyway thanks for your attention. 
>>> 
>>> 2015-11-28 11:52 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:
>>>> On Friday, November 27, 2015 3:23 PM, ?? <wjun77 at gmail.com> wrote:
>>>> 
>>>> >> Do you have any examples that actually do demonstrate the problem to be solved?
>>>> 
>>>> >So you want more details about AttributeError in property?
>>>> 
>>>> No. I'm assuming Paul wanted an example that demonstrates the problem (a @property or other descriptor that raises or passes AttributeError, and a __getattr__ that blindly returns a value for anything). Just like your toy example on the bug tracker does, but realistic code rather than a toy example.
>>>> 
>>>> What you've provided is an example that doesn't demonstrate the problem at all. The code below doesn't even have a __getattr__ in it, and it does exactly what you should expect it to do (assuming you fill in the missing bits in any reasonable way), so it can't possibly demonstrate why interactions with __getattr__ are a problem.
>>>> 
>>>> > I thought property is widely used, and AttributeError occurs at all times. Maybe I've used property too heavy.>In the window example, a simplified demonstration:
>>>> >
>>>> >class Window():
>>>> >    @property
>>>> >    def backgroundImg(self):
>>>> >        if self._backgroundImg is None: #need update, while the number of items changes
>>>> >            self.set_backgroundImg()
>>>> >        return self._backgroundImg
>>>> >
>>>> >    def set_backgroundImg(self):
>>>> >        self._backgroundImg = loadImg('white.bmp')
>>>> >        for widget in self.widgets:
>>>> >            widget.show(self._backgroundImg)
>>>> >
>>>> >Class Widget():
>>>> >    def show(self, img):
>>>> >        img.draw(self.item.img, self.pos)
>>>> >
>>>> 
>>>> >However, widget.item may be None, while e.g. there are four widgets but only three items in total. In this case I should fill the area with white. But in this version of show, I just FORGET item can be None. So the traceback infomation: 'Window' object has no attribute 'backgroundImg'.
>>>> 
>>>> No, that can't possibly be your problem. If that were the case, the AttributeError will say that 'NoneType' object has no attribute 'img'. And the traceback would run from the Widget.show method, where the self.item.img is, back up the chain through your @property method.
>>>> 
>>>> I'm guessing your actual problem is that you forgot to set self._backgroundImg = None somewhere (e.g., in the __init__ method). In that case, you would get an error that looks more like the one you're claiming to get (but the attribute mentioned is '_backgroundImg', not 'backgroundImg'), with only one level of traceback and everything.
>>>> 
>>>> 
>>>> Or maybe there's a typo in your actual code, and you really don't have a 'backgroundImg' at all on Window objects; that would give exactly the error you're describing.
>>>> 
>>>> No matter which case it is, the problem has nothing to do with @property or descriptors in general, or with __getattr__ (obviously, since there is no __getattr__ in the code), much less with the interaction between them, so any fix to that interaction couldn't possibly help this example.
>>>> 
>>>> 
>>>> > In fact it takes a while before I find the cause is the AttributeError/__getattr__ mechanism.
>>>> 
>>>> 
>>>> Since that isn't the cause, it would be bad if Python pointed you to look in that direction sooner...
>>>> >> But surely breaking that isn't the same as breaking code that's been explicitly stated to work, and used as sample code, for decades.
>>>> >I don't know this is such a severe problem. I used to think raising AttributeError in __getattribute__ to trigger __getattr__ is rare.
>>>> >
>>>> >> any solution that can fix descriptors without also "fixing" __getattribute__ is a lot better
>>>> 
>>>> >In practice I don't concern __getattribute__. But in my opinion it's better to 'fix' this in python4.
>>>> 
>>>> Why? If you think that erroneous uses are rare or nonexistent, while intentional uses are rare but not nonexistent, "fixing" it means breaking code gratuitously for no benefit.
>>>> >> all of your solutions make it too hard to trigger __getattr__ from a descriptor when you really _do_ want to do so
>>>> >This is a big problem, OK.
>>>> >
>>>> >> You didn't comment on the alternative I suggested; would it not satisfy your needs, or have some other problem that makes it unacceptable?
>>>> >I don't quite understand, you mean adding a subclass of object with the only difference of this behavior?
>>>> 
>>>> 
>>>> No, adding a subclass of AttributeError, much like the one you mentioned in the bug report, but with the opposite meaning: the existing AttributeError continues to trigger __getattr__, but the new subclass doesn't. This makes it trivial to write new code that doesn't accidentally trigger __getattr__, without breaking old code (or rare new code) that wants to trigger __getattr__.
>>>> 
>>>> The code currently does something like this pseudocode:
>>>> 
>>>>     try:
>>>>         val = obj.__getattribute__(name)
>>>>     except AttributeError:
>>>>         __getattr__ = getattr(type(obj), '__getattr__', None)
>>>>         if __getattr__: return __getattr__(name)
>>>> 
>>>> I'm cheating a bit, but you get the idea. The problem is that we have no idea whether __getattribute__ failed to find anything (in which case we definitely want __getattr__ called), or found a descriptor whose __get__ raised an AttributeError (in which case we may not--e.g., a write-only attribute should not all through to __getattr__).
>>>> 
>>>> My suggestion is to change it like this:
>>>> 
>>>> 
>>>>     try:
>>>>         val = obj.__getattribute__(name)
>>>>     except AttributeDynamicError:
>>>>         raise
>>>>     except AttributeError:
>>>>         __getattr__ = getattr(type(obj), '__getattr__', None)
>>>>         if __getattr__: return __getattr__(name)
>>>> 
>>>> 
>>>> Now, if __getattribute__ found a descriptor whose __get__ raised an AttributeDynamicError, that passes on to the user code. (And, since it's a subclass of AttributeError, the user code should have no problem handling it.) And the Descriptor HOWTO will be changed to suggest raising AttributeDynamicError, except when you explicitly want it to call __getattr__, which you usually don't. And examples like simulating a write-only attribute will raise AttributeDynamicError. And maybe @property will automatically convert any AttributeError to AttributeDynamicError (not sure about that part).
>>>> 
>>>> So, new code can easily be written to act the way you want, but existing code using descriptors that intentionally raise or pass an AttributeError continues to work the same way it always has, and new code that does the same can also be written easily.
>>>> 
>>>> Obviously, the downside of any backward-compat-friendly change is that someone who has old code with a hidden bug they didn't know about will still have that same bug in Python 3.7; they have to change their code to take advantage of the fix. But I don't think that's a serious problem. (Especially if we decide @property is buggy and should be changed--most people who are writing actual custom descriptors, not just using the ones in the stdlib, probably understand this stuff.)
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151129/260b8ece/attachment-0001.html>

From wjun77 at gmail.com  Sun Nov 29 13:10:12 2015
From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=)
Date: Mon, 30 Nov 2015 02:10:12 +0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967TJt5f-M=1izWQ-aHG9AEftLWUihOcOiWaZL-eQJiBXFA@mail.gmail.com>
References: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
 <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>
 <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>
 <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com>
 <CAPM967SwVLH7BVD94s+16+oPcwS68JAPCD7NfH0Tg6Awbt6BPQ@mail.gmail.com>
 <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com>
 <CAPM967S+NU=EnB=UFGwvKq2j8Dn3X7fsxbw1ub8a+E-rpKixQw@mail.gmail.com>
 <CAPM967TJt5f-M=1izWQ-aHG9AEftLWUihOcOiWaZL-eQJiBXFA@mail.gmail.com>
Message-ID: <CAPM967Qup3DB7PkSy6-gF6x1-QEbYno9pBS1Y3-azO4ZzAdOBQ@mail.gmail.com>

> A dedicated subclass of *Attr*ibuteError to trigger __get*attr*__ is more
natural.
I mean it's unnatural to convert AttributeError to AttributeDynamicError by
property or descriptor. Why not use RuntimeError as that of PEP479?

And there's no need to write custom property-like descriptor factory when
you want to trigger __getattr__, although it may be only a few lines of
code.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/a5f783cb/attachment.html>

From rosuav at gmail.com  Sun Nov 29 13:34:19 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 30 Nov 2015 05:34:19 +1100
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967Qup3DB7PkSy6-gF6x1-QEbYno9pBS1Y3-azO4ZzAdOBQ@mail.gmail.com>
References: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
 <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>
 <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>
 <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com>
 <CAPM967SwVLH7BVD94s+16+oPcwS68JAPCD7NfH0Tg6Awbt6BPQ@mail.gmail.com>
 <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com>
 <CAPM967S+NU=EnB=UFGwvKq2j8Dn3X7fsxbw1ub8a+E-rpKixQw@mail.gmail.com>
 <CAPM967TJt5f-M=1izWQ-aHG9AEftLWUihOcOiWaZL-eQJiBXFA@mail.gmail.com>
 <CAPM967Qup3DB7PkSy6-gF6x1-QEbYno9pBS1Y3-azO4ZzAdOBQ@mail.gmail.com>
Message-ID: <CAPTjJmrGxGEsUmq_LYYJeU8yJSAO4EeFrM50+=D8d2JwdzoA_w@mail.gmail.com>

On Mon, Nov 30, 2015 at 5:10 AM, ?? <wjun77 at gmail.com> wrote:
>> A dedicated subclass of AttributeError to trigger __getattr__ is more
>> natural.
> I mean it's unnatural to convert AttributeError to AttributeDynamicError by
> property or descriptor. Why not use RuntimeError as that of PEP479?
>

Earlier I posted some example code that would do exactly that. If an
AttributeError would escape a property function, it becomes a
RuntimeError. Does that do what you're looking for?

ChrisA

From abarnert at yahoo.com  Sun Nov 29 14:31:14 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 29 Nov 2015 11:31:14 -0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <CAPM967Qup3DB7PkSy6-gF6x1-QEbYno9pBS1Y3-azO4ZzAdOBQ@mail.gmail.com>
References: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
 <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>
 <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>
 <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com>
 <CAPM967SwVLH7BVD94s+16+oPcwS68JAPCD7NfH0Tg6Awbt6BPQ@mail.gmail.com>
 <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com>
 <CAPM967S+NU=EnB=UFGwvKq2j8Dn3X7fsxbw1ub8a+E-rpKixQw@mail.gmail.com>
 <CAPM967TJt5f-M=1izWQ-aHG9AEftLWUihOcOiWaZL-eQJiBXFA@mail.gmail.com>
 <CAPM967Qup3DB7PkSy6-gF6x1-QEbYno9pBS1Y3-azO4ZzAdOBQ@mail.gmail.com>
Message-ID: <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com>

On Nov 29, 2015, at 10:10, ?? <wjun77 at gmail.com> wrote:
> 
> > A dedicated subclass of AttributeError to trigger __getattr__ is more natural.
> I mean it's unnatural to convert AttributeError to AttributeDynamicError by property or descriptor.

Again, nobody has suggested that the descriptor protocol should do such a thing, so I don't know why you keep arguing about it.

I did suggest that maybe the property descriptor should do so. Or maybe not. If your only problem with my alternate proposal is something that was included as a "maybe" optional bit in parentheses, then you don't seem to have an actual problem with my alternate proposal.

Which brings us back to the point: My alternate proposal is less disruptive and more backward-compatible. Is there some actual problem with it, or some other reason to prefer your original version?

> Why not use RuntimeError as that of PEP479?

Three reasons:

1. There are thousands of lines of working code that handle AttributeError; forcing them to change to handle RuntimeError would be gratuitous backward incompatibility.

2. RuntimeError suggests that the code did something unexpected wrong, rather than a specific thing, possibly intentionally, that you may want to trap and handle. In PEP 479, this makes sense--it was decided that generators should not raise StopIteration instead of returning, and therefore it is a serious error to do so, not just a normal exception. But raising AttributeError from a property or other descriptor is a perfectly reasonable thing to do. It comes naturally from build-time proxies. It's recommended in the descriptor HOWTO. And so on. It makes perfect sense--and there's no other way to accomplish what it does, either. (How would you implement a write-only property that makes "print(spam.eggs)" raise an AttributeError except by raising it in the property function?)

3. In the vast majority of uses of properties and other descriptors, there is no __getattr__, and AttributeError is exactly what people want there. The only problem to be solved here is the rare but unexpected and annoying interaction of such dynamic AttributeErrors and __getattr__. Preventing descriptors from raising AttributeError even in the common cases to block that rare case is like removing a smudge from your table by disintegrating the table. Yes, the smudge is gone, but you haven't come out ahead.

> And there's no need to write custom property-like descriptor factory when you want to trigger __getattr__, although it may be only a few lines of  code.

Your original suggestion would make it _impossible_ for a descriptor to trigger __getattr__. And so would your suggestion here of converting AttributeError to RuntimeError. That doesn't remove the need; it leaves the need but makes it impossible to satisfy.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151129/5cf5b94f/attachment.html>

From wjun77 at gmail.com  Mon Nov 30 00:14:55 2015
From: wjun77 at gmail.com (=?UTF-8?B?546L54+6?=)
Date: Mon, 30 Nov 2015 13:14:55 +0800
Subject: [Python-ideas] The AttributeError/__getattr__ mechanism
In-Reply-To: <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com>
References: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
 <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>
 <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>
 <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com>
 <CAPM967SwVLH7BVD94s+16+oPcwS68JAPCD7NfH0Tg6Awbt6BPQ@mail.gmail.com>
 <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com>
 <CAPM967S+NU=EnB=UFGwvKq2j8Dn3X7fsxbw1ub8a+E-rpKixQw@mail.gmail.com>
 <CAPM967TJt5f-M=1izWQ-aHG9AEftLWUihOcOiWaZL-eQJiBXFA@mail.gmail.com>
 <CAPM967Qup3DB7PkSy6-gF6x1-QEbYno9pBS1Y3-azO4ZzAdOBQ@mail.gmail.com>
 <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com>
Message-ID: <CAPM967SDkV9Zy7o5JXweBp-zObRgk4_XJ0Mop2px+TtQZrCxNg@mail.gmail.com>

>
> Earlier I posted some example code that would do exactly that. If an
> AttributeError would escape a property function, it becomes a
> RuntimeError. Does that do what you're looking for?
>
Yes, while I need to decorate every property with *dontraise*. In python
3.5 or older it do be a handy tool and thanks for your suggestion. However,
similar scheme could be used to solve the problem of PEP479, but a change
in the interpreter core is made. I would like similar change in interpreter
core or stdlib in python 3.6, so no one will encounter the same problem
with future versions of python, knowing this discussion or not.

> Again, nobody has suggested that the descriptor protocol should do such a
thing, so I don't know why you keep arguing about it.
I myself suggested it as an option if nobody else, that's all.

> Is there some actual problem with it, or some other reason to prefer your
original version?
> Your original suggestion would make it _impossible_ for a descriptor to
trigger __getattr__.
I proposed 3 versions: AttributeMissError, RuntimeError, or
'object.__getattribute__ calls __getattr__ explicitly; __getattr__ not
triggered by AttributeError anymore' in the issue tracker. In fact I think
I am not qualified to talk about the detailed implementation because I know
little about the related CPython code. I said I personally prefer no change
to your version in aesthetic, which means that I don't have any persuasive
reasons. I think expressing opinion is more important than persuading
others.

> Three reasons:
These looks reasonable to me and thanks for your explanation. Now I think
your version is better than RuntimeError.
So if compatibility do count, your version seems the only choice other than
no change.

2015-11-30 3:31 GMT+08:00 Andrew Barnert <abarnert at yahoo.com>:

> On Nov 29, 2015, at 10:10, ?? <wjun77 at gmail.com> wrote:
>
> > A dedicated subclass of *Attr*ibuteError to trigger __get*attr*__ is
> more natural.
> I mean it's unnatural to convert AttributeError to AttributeDynamicError
> by property or descriptor.
>
>
> Again, nobody has suggested that the descriptor protocol should do such a
> thing, so I don't know why you keep arguing about it.
>
> I did suggest that maybe the property descriptor should do so. Or maybe
> not. If your only problem with my alternate proposal is something that was
> included as a "maybe" optional bit in parentheses, then you don't seem to
> have an actual problem with my alternate proposal.
>
> Which brings us back to the point: My alternate proposal is less
> disruptive and more backward-compatible. Is there some actual problem with
> it, or some other reason to prefer your original version?
>
> Why not use RuntimeError as that of PEP479?
>
>
> Three reasons:
>
> 1. There are thousands of lines of working code that handle
> AttributeError; forcing them to change to handle RuntimeError would be
> gratuitous backward incompatibility.
>
> 2. RuntimeError suggests that the code did something unexpected wrong,
> rather than a specific thing, possibly intentionally, that you may want to
> trap and handle. In PEP 479, this makes sense--it was decided that
> generators should not raise StopIteration instead of returning, and
> therefore it is a serious error to do so, not just a normal exception. But
> raising AttributeError from a property or other descriptor is a perfectly
> reasonable thing to do. It comes naturally from build-time proxies. It's
> recommended in the descriptor HOWTO. And so on. It makes perfect sense--and
> there's no other way to accomplish what it does, either. (How would you
> implement a write-only property that makes "print(spam.eggs)" raise an
> AttributeError except by raising it in the property function?)
>
> 3. In the vast majority of uses of properties and other descriptors, there
> is no __getattr__, and AttributeError is exactly what people want there.
> The only problem to be solved here is the rare but unexpected and annoying
> interaction of such dynamic AttributeErrors and __getattr__. Preventing
> descriptors from raising AttributeError even in the common cases to block
> that rare case is like removing a smudge from your table by disintegrating
> the table. Yes, the smudge is gone, but you haven't come out ahead.
>
> And there's no need to write custom property-like descriptor factory when
> you want to trigger __getattr__, although it may be only a few lines of
> code.
>
>
> Your original suggestion would make it _impossible_ for a descriptor to
> trigger __getattr__. And so would your suggestion here of converting
> AttributeError to RuntimeError. That doesn't remove the need; it leaves the
> need but makes it impossible to satisfy.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/9f729279/attachment-0001.html>

From ncoghlan at gmail.com  Mon Nov 30 01:45:25 2015
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 30 Nov 2015 16:45:25 +1000
Subject: [Python-ideas] PEP471 - (os.scandir())
In-Reply-To: <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com>
References: <56579431.4010907@lucidity.plus.com>
 <CACac1F8FN0NvtgJ2EBDXYFPfWcMjNDohB4W1RmLKubYt2dCGPA@mail.gmail.com>
 <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com>
Message-ID: <CADiSq7cDRCpOK2UKxA9hpcK3Au+2T05M-26QrXZnVBRqwhHdAQ@mail.gmail.com>

On 28 November 2015 at 04:42, Andrew Barnert via Python-ideas
<python-ideas at python.org> wrote:
> On Nov 27, 2015, at 10:32, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>>> On 26 November 2015 at 23:22, Erik <python at lucidity.plus.com> wrote:
>>> I have studied the PEP, followed a lot of the references and looked at the
>>> 3.5.0 implementation. I can't see that I've missed such a thing already
>>> existing, but it's possible. If so, perhaps this is instead a request to
>>> make that thing more obvious somehow!
>>
>> Does pathlib use scandir? If so, then maybe you get the caching
>> benefits by using pathlib? And if pathlib doesn't use scandir, maybe
>> it should? [I just checked, it looks like pathlib doesn't use scandir
>> :-(]
>
> Does pathlib even have a walk equivalent? (I know it has glob('**'), but that's not the same thing.)
>
> Or are you suggesting that people should use path.iterdir with explicit recursion (or an explicit stack), and therefore just changing iterdir to use scandir (and prefill as many cached attribs as possible in each result) is what we want?

The main problem with having pathlib do any caching at all is that
caching the results of stat calls implicitly in any context is a
recipe for significant confusion, since you're at the mercy of race
conditions as the filesystem changes out from underneath you. There
also isn't an obviously "right" answer in the general case for cache
invalidation, as in some cases you're interested in the file as it was
when you originally opened it, and don't care if it got swapped out
from underneath you, while in others you're interested in the file
path, and want the filesystem details for right now, not the details
from a few seconds ago.

For os.scandir(), we just delegate the behaviour to the underlying
filesystem APIs - how readdir() and FindNextFile react to the
directory contents changing during iteration is OS defined, and Python
will inherit that variation (and may miss newly added files as a
result).

The current os.walk() implementation constrains the scope of the
scandir() filesystem state caching, since it doesn't let the DirEntry
objects escape outside the generator - there's no need to ask yourself
"What's the risk of stale filesystem data here?", since you're not
getting access to the cached info in the first place, and hence always
need to go query the filesystem directly.

This is a fairly universal pattern: for a given *application* you can
likely figure out what to cache and when to invalidate it, even though
those are unanswerable questions in the general case. Another example
of that would be the stat caches in the current implementation of the
import system, together with the corresponding need to call
importlib.invalidate_caches() if you want to make sure the import
system can see a module that was only just written to disk.

That's not to say that a general purpose directory walking utility
producing DirEntry objects isn't an interesting prospect. Rather, it's
an attempt to highlight that this is an area where there may be a
significant gulf between "works for my use case" and "is a suitable
addition to the standard library", particularly since this can now be
a pure Python recipe atop os.scandir.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From abarnert at yahoo.com  Mon Nov 30 04:12:23 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 30 Nov 2015 01:12:23 -0800
Subject: [Python-ideas] PEP471 - (os.scandir())
In-Reply-To: <CADiSq7cDRCpOK2UKxA9hpcK3Au+2T05M-26QrXZnVBRqwhHdAQ@mail.gmail.com>
References: <56579431.4010907@lucidity.plus.com>
 <CACac1F8FN0NvtgJ2EBDXYFPfWcMjNDohB4W1RmLKubYt2dCGPA@mail.gmail.com>
 <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com>
 <CADiSq7cDRCpOK2UKxA9hpcK3Au+2T05M-26QrXZnVBRqwhHdAQ@mail.gmail.com>
Message-ID: <9B8778E8-0133-42A9-A116-2A0E931B6ACE@yahoo.com>

On Nov 29, 2015, at 22:45, Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> On 28 November 2015 at 04:42, Andrew Barnert via Python-ideas
> <python-ideas at python.org> wrote:
>> On Nov 27, 2015, at 10:32, Paul Moore <p.f.moore at gmail.com> wrote:
>>> 
>>>> On 26 November 2015 at 23:22, Erik <python at lucidity.plus.com> wrote:
>>>> I have studied the PEP, followed a lot of the references and looked at the
>>>> 3.5.0 implementation. I can't see that I've missed such a thing already
>>>> existing, but it's possible. If so, perhaps this is instead a request to
>>>> make that thing more obvious somehow!
>>> 
>>> Does pathlib use scandir? If so, then maybe you get the caching
>>> benefits by using pathlib? And if pathlib doesn't use scandir, maybe
>>> it should? [I just checked, it looks like pathlib doesn't use scandir
>>> :-(]
>> 
>> Does pathlib even have a walk equivalent? (I know it has glob('**'), but that's not the same thing.)
>> 
>> Or are you suggesting that people should use path.iterdir with explicit recursion (or an explicit stack), and therefore just changing iterdir to use scandir (and prefill as many cached attribs as possible in each result) is what we want?
> 
> The main problem with having pathlib do any caching at all is that
> caching the results of stat calls implicitly in any context is a
> recipe for significant confusion, since you're at the mercy of race
> conditions as the filesystem changes out from underneath you. There
> also isn't an obviously "right" answer in the general case for cache
> invalidation, as in some cases you're interested in the file as it was
> when you originally opened it, and don't care if it got swapped out
> from underneath you, while in others you're interested in the file
> path, and want the filesystem details for right now, not the details
> from a few seconds ago.
> 
> For os.scandir(), we just delegate the behaviour to the underlying
> filesystem APIs - how readdir() and FindNextFile react to the
> directory contents changing during iteration is OS defined, and Python
> will inherit that variation (and may miss newly added files as a
> result).
> 
> The current os.walk() implementation constrains the scope of the
> scandir() filesystem state caching, since it doesn't let the DirEntry
> objects escape outside the generator

That's a good point. The fts functions that both BSD and GNU use to replace the ftw and the various other old *nix filesystem walk functions deal with this carefully; the short version is that any information you want to keep around from the current file after looking at the next file, you have to explicitly copy it, which makes it hard to confuse yourself about how up-to-date it is. (You can also go a directory at a time, more like os.walk, but with the same basic restriction: once you go to the next directory, the previous list of file entries is invalid.)

> - there's no need to ask yourself
> "What's the risk of stale filesystem data here?", since you're not
> getting access to the cached info in the first place, and hence always
> need to go query the filesystem directly.
> 
> This is a fairly universal pattern: for a given *application* you can
> likely figure out what to cache and when to invalidate it, even though
> those are unanswerable questions in the general case. Another example
> of that would be the stat caches in the current implementation of the
> import system, together with the corresponding need to call
> importlib.invalidate_caches() if you want to make sure the import
> system can see a module that was only just written to disk.
> 
> That's not to say that a general purpose directory walking utility
> producing DirEntry objects isn't an interesting prospect. Rather, it's
> an attempt to highlight that this is an area where there may be a
> significant gulf between "works for my use case" and "is a suitable
> addition to the standard library", particularly since this can now be
> a pure Python recipe atop os.scandir.

I still think providing an fts-like API instead of os.walk would be the clearest way to provide cached data (especially since people could look up nice generic documentation on fts). I don't think it would be too hard to emulate it (or a large enough subset--you can ask fts to return anything from just names to full stat structs, with well-defined performance characteristics for each combination of flags, and that probably can't be exactly the same) on top of scandir (or directly on Windows FindFirst/etc.), but then I said that when the PEP was being discussed and then never had time to actually try it... The bigger problem is that "you have to copy it, which is painful enough that you can't confuse yourself" is a much lower barrier to confusion in Python than in C, so it might not be as effective.


From p.f.moore at gmail.com  Mon Nov 30 04:40:21 2015
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 30 Nov 2015 09:40:21 +0000
Subject: [Python-ideas] PEP471 - (os.scandir())
In-Reply-To: <9B8778E8-0133-42A9-A116-2A0E931B6ACE@yahoo.com>
References: <56579431.4010907@lucidity.plus.com>
 <CACac1F8FN0NvtgJ2EBDXYFPfWcMjNDohB4W1RmLKubYt2dCGPA@mail.gmail.com>
 <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com>
 <CADiSq7cDRCpOK2UKxA9hpcK3Au+2T05M-26QrXZnVBRqwhHdAQ@mail.gmail.com>
 <9B8778E8-0133-42A9-A116-2A0E931B6ACE@yahoo.com>
Message-ID: <CACac1F9szRGomHwbLdRaEojR10cHd+6XoW0wMmF7Ash4=vS3ew@mail.gmail.com>

On 30 November 2015 at 09:12, Andrew Barnert <abarnert at yahoo.com> wrote:
> I still think providing an fts-like API instead of os.walk would be the clearest way to provide cached data (especially since people could look up nice generic documentation on fts).

As a Windows user I'm not familiar with fts (and Google didn't come up
with anything obvious). So I'm not sure how true "people could look up
generic docuimentation" would be in practice. But from your
description it may be useful - I presume it's something that could be
built as a 3rd party library based on os.scandir, at least as an
initial proof of concept?

Paul

From steve at pearwood.info  Mon Nov 30 05:32:54 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 30 Nov 2015 21:32:54 +1100
Subject: [Python-ideas] Write-only property [was Re: The
 AttributeError/__getattr__ mechanism]
In-Reply-To: <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com>
References: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
 <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>
 <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>
 <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com>
 <CAPM967SwVLH7BVD94s+16+oPcwS68JAPCD7NfH0Tg6Awbt6BPQ@mail.gmail.com>
 <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com>
 <CAPM967S+NU=EnB=UFGwvKq2j8Dn3X7fsxbw1ub8a+E-rpKixQw@mail.gmail.com>
 <CAPM967TJt5f-M=1izWQ-aHG9AEftLWUihOcOiWaZL-eQJiBXFA@mail.gmail.com>
 <CAPM967Qup3DB7PkSy6-gF6x1-QEbYno9pBS1Y3-azO4ZzAdOBQ@mail.gmail.com>
 <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com>
Message-ID: <20151130103254.GW3821@ando.pearwood.info>

On Sun, Nov 29, 2015 at 11:31:14AM -0800, Andrew Barnert via Python-ideas wrote:

> (How would you implement a write-only property 
> that makes "print(spam.eggs)" raise an AttributeError except by 
> raising it in the property function?)

eggs = property(None, setter, None, "")

makes eggs a write-only property. print(spam.eggs) gives 

AttributeError: unreadable attribute



-- 
Steve

From abarnert at yahoo.com  Mon Nov 30 11:07:14 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 30 Nov 2015 08:07:14 -0800
Subject: [Python-ideas] Write-only property [was Re: The
 AttributeError/__getattr__ mechanism]
In-Reply-To: <20151130103254.GW3821@ando.pearwood.info>
References: <CAPM967TRBYjSjy_fO_O6XS7v5GyLSmraXTMDqnDo-z=Re7QEmA@mail.gmail.com>
 <1302699507.9503487.1448682753277.JavaMail.yahoo@mail.yahoo.com>
 <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>
 <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com>
 <CAPM967SwVLH7BVD94s+16+oPcwS68JAPCD7NfH0Tg6Awbt6BPQ@mail.gmail.com>
 <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com>
 <CAPM967S+NU=EnB=UFGwvKq2j8Dn3X7fsxbw1ub8a+E-rpKixQw@mail.gmail.com>
 <CAPM967TJt5f-M=1izWQ-aHG9AEftLWUihOcOiWaZL-eQJiBXFA@mail.gmail.com>
 <CAPM967Qup3DB7PkSy6-gF6x1-QEbYno9pBS1Y3-azO4ZzAdOBQ@mail.gmail.com>
 <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com>
 <20151130103254.GW3821@ando.pearwood.info>
Message-ID: <B6AB1F04-573C-4360-B24B-D784EEFB7705@yahoo.com>

On Nov 30, 2015, at 02:32, Steven D'Aprano <steve at pearwood.info> wrote:
> 
>> On Sun, Nov 29, 2015 at 11:31:14AM -0800, Andrew Barnert via Python-ideas wrote:
>> 
>> (How would you implement a write-only property 
>> that makes "print(spam.eggs)" raise an AttributeError except by 
>> raising it in the property function?)
> 
> eggs = property(None, setter, None, "")
> 
> makes eggs a write-only property. print(spam.eggs) gives 
> 
> AttributeError: unreadable attribute

And how do you think property implements that? The Descriptor HOWTO shows a pure-Python equivalent to property, which shows exactly how it works: by raising it in the property __get__ function, of course.

Also notice that it looks exactly the same as if you'd provided an fget argument to property and that function raised (except that way you can control the text of the error).

I think this is a good design. And I think that, except for often-unwanted (but usually-irrelevant) interaction with __getattr__, it's exactly what users would want and expect. But even if that weren't true, it's what Python had done for decades. Which implies that if we are going to gratuitously break it, we need to make it relatively cheap for people whose code depends on the existing behavior to restore it.

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        if self.fget is None:
            raise AttributeError("unreadable attribute")
        return self.fget(obj)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/87bd777f/attachment-0001.html>

From abarnert at yahoo.com  Mon Nov 30 11:18:25 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 30 Nov 2015 08:18:25 -0800
Subject: [Python-ideas] PEP471 - (os.scandir())
In-Reply-To: <CACac1F9szRGomHwbLdRaEojR10cHd+6XoW0wMmF7Ash4=vS3ew@mail.gmail.com>
References: <56579431.4010907@lucidity.plus.com>
 <CACac1F8FN0NvtgJ2EBDXYFPfWcMjNDohB4W1RmLKubYt2dCGPA@mail.gmail.com>
 <3E188123-D4E5-4913-AA56-53A9CEF85BDD@yahoo.com>
 <CADiSq7cDRCpOK2UKxA9hpcK3Au+2T05M-26QrXZnVBRqwhHdAQ@mail.gmail.com>
 <9B8778E8-0133-42A9-A116-2A0E931B6ACE@yahoo.com>
 <CACac1F9szRGomHwbLdRaEojR10cHd+6XoW0wMmF7Ash4=vS3ew@mail.gmail.com>
Message-ID: <85EF61C2-3448-4A5B-B315-CB63E0B93E1F@yahoo.com>

On Nov 30, 2015, at 01:40, Paul Moore <p.f.moore at gmail.com> wrote:
> 
>> On 30 November 2015 at 09:12, Andrew Barnert <abarnert at yahoo.com> wrote:
>> I still think providing an fts-like API instead of os.walk would be the clearest way to provide cached data (especially since people could look up nice generic documentation on fts).
> 
> As a Windows user I'm not familiar with fts (and Google didn't come up
> with anything obvious).

The perils of acronym-based naming; it's very easy to go from one of two meaningful search results to way down the list just because UrbanDictionary popularized some txt speak slang and wikipedia started covering every government agency in the world with its name translated to English...

So you're right, that benefit no longer applies. You can still find "man fts" very easily, but that isn't what people would be looking for, and doesn't find any of the user-friendly tutorials, just the manpage.

> So I'm not sure how true "people could look up
> generic docuimentation" would be in practice. But from your
> description it may be useful - I presume it's something that could be
> built as a 3rd party library based on os.scandir, at least as an
> initial proof of concept?

A complete implementation that supported all of the flags and maintained the appropriate performance guarantees might be hard. But a partial implementation that supports just the most common flags and falls back to "stat everything, sometimes twice" as a proof of concept should be doable, once I get some free time.

From henryschreineriii at gmail.com  Mon Nov 30 11:32:49 2015
From: henryschreineriii at gmail.com (Henry Schreiner)
Date: Mon, 30 Nov 2015 10:32:49 -0600
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
Message-ID: <38974CAF-ECC7-499F-819E-61D46C713FF3@gmail.com>

It seems to me there are two ideas being discussed. One is adding 2 new operators, something like && and ||, and providing them for some sort of "element wise" or other boolean context, and giving them a better order of evaluations than bitwise ops. This would have the advantage of being usable on the built in python lists/tuples if needed, but otherwise doesn't seem to solve all the issues above, including expressions like -1 < mat < 1, which work for python variables but not for things like Numpy arrays. (for example, selecting values from an array works like this: mat[(-1<mat) & (mat<1)] but not like this: mat[-1 < mat < 1] or this: mat[-1<mat & mat<1] ).

The common "general purpose infix notation" pops up here, of course, but I'd consider it a variant of the first idea.

The other idea is that overloads could be added to the existing and and or operators. This, I believe, would solve all the main problems, and would not add more to the language for a newcomer to learn. (In fact, most people could forget about the current bitwise and and or, since that's not what most people need, at least those using databases and Numpy). The downside is that the current and and or operators are short-circuiting, and adding a way to overload a short-circuiting operator is tricky/ugly/expensive (though the last one might not be that bad anymore). But, as I think Andrew was mentioning, what about a mixed solution? One that provides a non-short-circuiting overload, but the original short-circuiting behavior is default? It would look like this:

Two new magic methods would be added, for now called __booland__ and __boolor__, along with __rbooland__ and __rboolor__.
 
Evaluation would work like this, using and for an example:
When x and y is written, if x has __booland__, then x.__booland__(y) is returned. If not, then return x if x==False else if x==True, then if y has y.__rbooland__(x), that is returned. If not, then return y.
 
This would allow object that want custom and/or behavior to implement these magic functions. They would lose (explicit) short-circuiting, but they would gain custom behavior. Most of these special objects do not need or expect the short circuit behavior (Only if an array was all True or False would it even make sense in numpy , and this would allow expressions like -1 < mat < 1 to work as expected in Python. This would be easy to teach to newcomers, too, since only objects that have a definite True/False should short circuit, others have the special methods. The old behavior could be forced on overriding objects by wrapping them in bool().

Would something like that be possible and better than PEP 335?

PS:
Either type of solution would also be useful for the Plumbum library, by the way, as it would be useful to add and-ing and or-ing to pipelines, but the bitwise operators have a different meaning for Plumbum. Due to the lazy evaluation of Plumbum, even the above solution (or any solution) would still allow short-circuiting in the pipeline.

PPS: Hopefully I?m not double posting, apparently the google group and the mailing list are separate subscriptions.

Henry Schreiner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/f0aad8a8/attachment.html>

From henryschreineriii at gmail.com  Mon Nov 30 11:22:31 2015
From: henryschreineriii at gmail.com (Henry Schreiner)
Date: Mon, 30 Nov 2015 08:22:31 -0800 (PST)
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
 <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>
 <56577194.5000003@canterbury.ac.nz>
 <CADiSq7dQ-Myk7k7shUFyf1fx5n0ww5NERWQ3s3Db2J9=pHqu4g@mail.gmail.com>
 <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com>
 <5658B252.1060902@brenbarn.net>
 <CAP7+vJLVWh--+b4qHkawMXVPQ+V8ZifpRUPocnfR+Jm55CVquQ@mail.gmail.com>
 <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com>
Message-ID: <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com>

Sorry, the order was incorrect in that last message. It should be:

When x and y is written, if x has __booland__, then x.__booland__(y) is 
returned. If not, then return x if x==False else if x==True, then if y has 
y.__rbooland__(x), that is returned. If not, then return y.

On Monday, November 30, 2015 at 10:11:03 AM UTC-6, Henry Schreiner wrote:
>
> It seems to me there are two ideas being discussed. One is adding 2 new 
> operators, something like && and ||, and providing them for some sort of 
> "element wise" or other boolean context, and giving them a better order of 
> evaluations than bitwise ops. This would have the advantage of being usable 
> on the built in python lists/tuples if needed, but otherwise doesn't seem 
> to solve all the issues above, including expressions like -1 < mat < 1, 
> which work for python variables but not for things like Numpy arrays. (for 
> example, selecting values from an array works like this: mat[(-1<mat) & 
> (mat<1)] but not like this: mat[-1 < mat < 1] or this: mat[-1<mat & 
> mat<1] ).
>
> The common "general purpose infix notation" pops up here, of course, but 
> I'd consider it a variant of the first idea.
>
> The other idea is that overloads could be added to the existing and and or 
> operators. This, I believe, would solve all the main problems, and would 
> not add more to the language for a newcomer to learn. (In fact, most people 
> could forget about the current bitwise and and or, since that's not what 
> most people need, at least those using databases and Numpy). The downside 
> is that the current and and or operators are short-circuiting, and adding 
> a way to overload a short-circuiting operator is tricky/ugly/expensive 
> (though the last one might not be that bad anymore). But, as I think Andrew 
> was mentioning, what about a mixed solution? One that provides a 
> non-short-circuiting overload, but the original short-circuiting behavior 
> is default? It would look like this:
>
> Two new magic methods would be added, for now called __booland__ and 
> __boolor__, along with __rbooland__ and __rboolor__.
>
>  
>
> Evaluation would work like this, using and for an example:
> When x and y is written, if x has __booland__, then y.__booland__(y) is 
> called. If it does not, then if y exists and has y.__rbooland__(x), that 
> is called. If neither, then x and y is performed in a normal, 
> short circuiting manner, calling bool(x) then calling bool(y) only if x 
> == True. (The second line here might be the hard part due to the way 
> short circuiting works)
>
>  
>
> This would allow object that want custom and/or behavior to implement 
> these magic functions. They would lose (explicit) short-circuiting, but 
> they would gain custom behavior. Most of these special objects do not need 
> or expect the short circuit behavior (Only if an array was all True or 
> False would it even make sense in numpy , and this would allow 
> expressions like -1 < mat < 1 to work as expected in Python. This would 
> be easy to teach to newcomers, too, since only objects that have a definite 
> True/False should short circuit, others have the special methods. The 
> old behavior could be forced on overriding objects by wrapping them in 
> bool().
>
>
> Would something like that be possible and better than PEP 335?
>
> PS:
> Either type of solution would also be useful for the Plumbum library, by 
> the way, as it would be useful to add and-ing and or-ing to pipelines, but 
> the bitwise operators have a different meaning for Plumbum. Due to the lazy 
> evaluation of Plumbum, even the above solution (or any solution) would 
> still allow short-circuiting in the pipeline.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/3a22d0ec/attachment-0001.html>

From steve at pearwood.info  Mon Nov 30 12:08:12 2015
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 1 Dec 2015 04:08:12 +1100
Subject: [Python-ideas] Write-only property [was Re: The
 AttributeError/__getattr__ mechanism]
In-Reply-To: <B6AB1F04-573C-4360-B24B-D784EEFB7705@yahoo.com>
References: <CAPM967QoEddzR7NH4nH-G9v2=LtuOjNmZ+QWs6Uss2_yZr_Vow@mail.gmail.com>
 <2F3F5D81-3109-442E-BFDB-B166D40C415A@yahoo.com>
 <CAPM967SwVLH7BVD94s+16+oPcwS68JAPCD7NfH0Tg6Awbt6BPQ@mail.gmail.com>
 <84C3BBEA-3D17-4B50-9571-6405EABE03F3@yahoo.com>
 <CAPM967S+NU=EnB=UFGwvKq2j8Dn3X7fsxbw1ub8a+E-rpKixQw@mail.gmail.com>
 <CAPM967TJt5f-M=1izWQ-aHG9AEftLWUihOcOiWaZL-eQJiBXFA@mail.gmail.com>
 <CAPM967Qup3DB7PkSy6-gF6x1-QEbYno9pBS1Y3-azO4ZzAdOBQ@mail.gmail.com>
 <6EA6FD1E-AB22-431C-9A68-88C47975B667@yahoo.com>
 <20151130103254.GW3821@ando.pearwood.info>
 <B6AB1F04-573C-4360-B24B-D784EEFB7705@yahoo.com>
Message-ID: <20151130170807.GY3821@ando.pearwood.info>

On Mon, Nov 30, 2015 at 08:07:14AM -0800, Andrew Barnert wrote:
> On Nov 30, 2015, at 02:32, Steven D'Aprano <steve at pearwood.info> wrote:
> > 
> >> On Sun, Nov 29, 2015 at 11:31:14AM -0800, Andrew Barnert via Python-ideas wrote:
> >> 
> >> (How would you implement a write-only property 
> >> that makes "print(spam.eggs)" raise an AttributeError except by 
> >> raising it in the property function?)
> > 
> > eggs = property(None, setter, None, "")
> > 
> > makes eggs a write-only property. print(spam.eggs) gives 
> > 
> > AttributeError: unreadable attribute
> 
> And how do you think property implements that? 

Sorry, I misunderstood your (rhetorical?) question -- I read it as 
implying one should explicitly write a getter which raised 
AttributeError.



-- 
Steve

From guido at python.org  Mon Nov 30 12:23:52 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 30 Nov 2015 09:23:52 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
 <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>
 <56577194.5000003@canterbury.ac.nz>
 <CADiSq7dQ-Myk7k7shUFyf1fx5n0ww5NERWQ3s3Db2J9=pHqu4g@mail.gmail.com>
 <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com>
 <5658B252.1060902@brenbarn.net>
 <CAP7+vJLVWh--+b4qHkawMXVPQ+V8ZifpRUPocnfR+Jm55CVquQ@mail.gmail.com>
 <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com>
 <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com>
Message-ID: <CAP7+vJ+9gXPct0aK0PLwzYwPeZuF-L+BYYDJATQY8_NLrs0pkg@mail.gmail.com>

On Mon, Nov 30, 2015 at 8:22 AM, Henry Schreiner <
henryschreineriii at gmail.com> wrote:

> Sorry, the order was incorrect in that last message. It should be:
>
> When x and y is written, if x has __booland__, then x.__booland__(y) is
> returned. If not, then return x if x==False else if x==True, then if y
>  has y.__rbooland__(x), that is returned. If not, then return y
>

This proposal isn't significantly different from PEP 335. In PEP 335, first
you ask the left operand whether to overload 'and' or not (by calling its
__and1__); if it says yes, you evaluate the second arg and ask the first to
evaluate it. In your proposal, the first question is answered by inspecting
whether x defines __booland__. I don't think this makes much practical
difference. The second part of your proposal is just a renaming of
__and2__/__rand2__ to __booland__/__rbooland__. I'm not sure I like this
much; the return type of these methods is almost never going to be an
actual bool, so the 'bool' in the name is somewhat misleading. (The names
chosen in PEP 335 are intentionally free of semantic hints, since the
intuition of someone who encounters these for the first time isn't likely
to be useful.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/05643c65/attachment.html>

From henryschreineriii at gmail.com  Mon Nov 30 13:08:24 2015
From: henryschreineriii at gmail.com (Henry Schreiner)
Date: Mon, 30 Nov 2015 10:08:24 -0800 (PST)
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAP7+vJ+9gXPct0aK0PLwzYwPeZuF-L+BYYDJATQY8_NLrs0pkg@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
 <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>
 <56577194.5000003@canterbury.ac.nz>
 <CADiSq7dQ-Myk7k7shUFyf1fx5n0ww5NERWQ3s3Db2J9=pHqu4g@mail.gmail.com>
 <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com>
 <5658B252.1060902@brenbarn.net>
 <CAP7+vJLVWh--+b4qHkawMXVPQ+V8ZifpRUPocnfR+Jm55CVquQ@mail.gmail.com>
 <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com>
 <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com>
 <CAP7+vJ+9gXPct0aK0PLwzYwPeZuF-L+BYYDJATQY8_NLrs0pkg@mail.gmail.com>
Message-ID: <7df3290f-c021-4e8b-b71b-f49be3f1554b@googlegroups.com>

I didn't realize the similarity of the proposals. I do think that testing 
for existence is better than running a method (that might not exist), at 
least from a high level perspective.

So PEP 335 might be dusted off again and run through benchmarks? There 
seems to be at least a test implementation for Python 2.3, are there 
benchmarks documented somewhere? It sound like the version that adds new 
byte codes instead of an additional byte code per op was never included.

I do like the cleanliness of not adding extra operators with nearly 
identical meanings.

On Monday, November 30, 2015 at 11:24:12 AM UTC-6, Guido van Rossum wrote:
>
> On Mon, Nov 30, 2015 at 8:22 AM, Henry Schreiner <henrysch... at gmail.com 
> <javascript:>> wrote:
>
>> Sorry, the order was incorrect in that last message. It should be:
>>
>> When x and y is written, if x has __booland__, then x.__booland__(y) is 
>> returned. If not, then return x if x==False else if x==True, then if y
>>  has y.__rbooland__(x), that is returned. If not, then return y
>>
>
> This proposal isn't significantly different from PEP 335. In PEP 335, 
> first you ask the left operand whether to overload 'and' or not (by calling 
> its __and1__); if it says yes, you evaluate the second arg and ask the 
> first to evaluate it. In your proposal, the first question is answered by 
> inspecting whether x defines __booland__. I don't think this makes much 
> practical difference. The second part of your proposal is just a renaming 
> of __and2__/__rand2__ to __booland__/__rbooland__. I'm not sure I like this 
> much; the return type of these methods is almost never going to be an 
> actual bool, so the 'bool' in the name is somewhat misleading. (The names 
> chosen in PEP 335 are intentionally free of semantic hints, since the 
> intuition of someone who encounters these for the first time isn't likely 
> to be useful.)
>  
> -- 
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/88a44e85/attachment.html>

From guido at python.org  Mon Nov 30 13:23:26 2015
From: guido at python.org (Guido van Rossum)
Date: Mon, 30 Nov 2015 10:23:26 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <7df3290f-c021-4e8b-b71b-f49be3f1554b@googlegroups.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
 <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>
 <56577194.5000003@canterbury.ac.nz>
 <CADiSq7dQ-Myk7k7shUFyf1fx5n0ww5NERWQ3s3Db2J9=pHqu4g@mail.gmail.com>
 <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com>
 <5658B252.1060902@brenbarn.net>
 <CAP7+vJLVWh--+b4qHkawMXVPQ+V8ZifpRUPocnfR+Jm55CVquQ@mail.gmail.com>
 <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com>
 <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com>
 <CAP7+vJ+9gXPct0aK0PLwzYwPeZuF-L+BYYDJATQY8_NLrs0pkg@mail.gmail.com>
 <7df3290f-c021-4e8b-b71b-f49be3f1554b@googlegroups.com>
Message-ID: <CAP7+vJJrMZfM4LDsmy525hPjL8g1m45Qi9EmjFw5MA10ReJJFQ@mail.gmail.com>

Well, there is still the issue of whether numpy would adopt PEP 335 even if
it was implemented. I'm still waiting for a response to my message
questioning how common `bool_value and/or numpy_array` really is.

Also, there may still be a significant difference between PEP 335 and your
proposal -- IIUC in your proposal if the value of the first operand
requires the second operand to be evaluated (e.g. `True or y`) then in your
proposal if y defined the 'r' special method it will be used; I don't think
PEP 335 does that. I like your version better.

On Mon, Nov 30, 2015 at 10:08 AM, Henry Schreiner <
henryschreineriii at gmail.com> wrote:

> I didn't realize the similarity of the proposals. I do think that testing
> for existence is better than running a method (that might not exist), at
> least from a high level perspective.
>
> So PEP 335 might be dusted off again and run through benchmarks? There
> seems to be at least a test implementation for Python 2.3, are there
> benchmarks documented somewhere? It sound like the version that adds new
> byte codes instead of an additional byte code per op was never included.
>
> I do like the cleanliness of not adding extra operators with nearly
> identical meanings.
>
> On Monday, November 30, 2015 at 11:24:12 AM UTC-6, Guido van Rossum wrote:
>>
>> On Mon, Nov 30, 2015 at 8:22 AM, Henry Schreiner <henrysch... at gmail.com>
>> wrote:
>>
>>> Sorry, the order was incorrect in that last message. It should be:
>>>
>>> When x and y is written, if x has __booland__, then x.__booland__(y) is
>>> returned. If not, then return x if x==False else if x==True, then if y
>>>  has y.__rbooland__(x), that is returned. If not, then return y
>>>
>>
>> This proposal isn't significantly different from PEP 335. In PEP 335,
>> first you ask the left operand whether to overload 'and' or not (by calling
>> its __and1__); if it says yes, you evaluate the second arg and ask the
>> first to evaluate it. In your proposal, the first question is answered by
>> inspecting whether x defines __booland__. I don't think this makes much
>> practical difference. The second part of your proposal is just a renaming
>> of __and2__/__rand2__ to __booland__/__rbooland__. I'm not sure I like this
>> much; the return type of these methods is almost never going to be an
>> actual bool, so the 'bool' in the name is somewhat misleading. (The names
>> chosen in PEP 335 are intentionally free of semantic hints, since the
>> intuition of someone who encounters these for the first time isn't likely
>> to be useful.)
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/0381c4ea/attachment-0001.html>

From henryschreineriii at gmail.com  Mon Nov 30 15:56:49 2015
From: henryschreineriii at gmail.com (Henry Schreiner)
Date: Mon, 30 Nov 2015 14:56:49 -0600
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <712322608.10180212.1448915321340.JavaMail.yahoo@mail.yahoo.com>
References: <CAP7+vJJrMZfM4LDsmy525hPjL8g1m45Qi9EmjFw5MA10ReJJFQ@mail.gmail.com>
 <712322608.10180212.1448915321340.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <F19E8A3C-DE82-4E62-BCF9-08DDB5721E81@gmail.com>

My proposal, I believe, is exactly the same as the ?Reduced Special Methods set? of PEP 335; that?s the phase two only part. I believe the __r*__ methods there are the same as my proposal. A portion of the complexity of PEP 335 is used in the ?phase 1? special methods, which are only helpful if you want to overload the shortcutting behavior itself on a per object basis, which I don?t think there is much need for. Assuming your methods are part of the class, as Python usually requires anyway, this would allow a class to either be short circuiting, boolean only (not containing __and2__ or __sand__ or __xand__ or whatever it would be called), or would have them and therefore would have customized non-short-circuiting behavior. (But, due to the fact that the class controls how much calculation happens, this generally can still limit computation, as in my Plumbum example; the only requirement is that the variable exist so it can be passed to the function).

If it looks like this would be acceptable to the NumPy folks, then maybe PEP 335 could be changed to make the Reduced Special Methods the main suggestion, with extending to the two phase method as a side note?


Henry Schreiner

> On Nov 30, 2015, at 2:28 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> 
> PEP 335 uses the standard rules for 'r' methods, so y.__rand2__ will only be used if issubclass(type(y), type(x)) or if x.__and2__ doesn't exist. Wouldn't having different rules for 'r' methods in this case than for all other operators be confusing? (Maybe not; the comparison operators are the closest analogs, and they have special rules too...)
> 
> Meanwhile, looking at PEP 335 again, it seems like we could very easily add just the "phase-2" part today (which would allow non-short-circuiting overloads, while preserving short-circuiting for types that don't overload), and then add the "phase-1" part (which would add optional short-circuiting overloads) later if needed, or just never add it if not.
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/6812886c/attachment.html>

From vgr255 at live.ca  Mon Nov 30 20:07:29 2015
From: vgr255 at live.ca (Emanuel Barry)
Date: Mon, 30 Nov 2015 20:07:29 -0500
Subject: [Python-ideas] Multiple arguments for decorators
Message-ID: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>

An idea that I had a couple of times was to have a syntactic way to define multiple arguments for a @decorator. This would be good if you want to, for example, define a property with a getter and a setter. I had a few ideas, but none of them seem to be "the obvious way to do it". I'm including them anyway, but I'd rather see if people are interested in the idea or if it's just something that's not worth it. I'll gladly write a PEP if it gains a lot of interest.
Method 1: Add parens around all arguments
class Foo:    def __init__(self):        self._x = 42    @property(    def x(self): # getter        return self._x    def x(self, value): # setter        self._x = value    def x(self): # deleter        del self._x    )
This one was my first idea, but with long methods you might very well miss the closing paren, or even wonder "how does the code even run?"
Good for small properties like this, bad with longer methods. -1 from me
Method 2: Specify how many arguments in the decorator
class Foo:    def __init__(self):        self._x = 42    @3:property    def x(self): # getter        return self._x    def x(self, value): # setter        self._x = value    def x(self): # deleter        del self._x
This one would get all three following methods defined (or should it get any value defined -- for example, a None in place of the setter there, would set the setter to None). Implementation-wise, I believe the order is kept while the class is first evaluated, and then often thrown away when it's inserted into the class' __dict__ (I could be wrong, though).
This has the advantage of not breaking code already written, as it cannot possibly conflict with an existing name. I'm neutral here.
Method 3: Specify arguments using the parameters' names
class Foo:    def __init__(self):        self._x = 42    @property    def x:fget(self):        return self._x    def x:fset(self, value):        self._x = value    def x:fdel(self):        del self._x
This has the advantage of being explicit (and the arguments can be swapped since they're named), but the disadvantage that some builtins don't accept kwargs (this would be a good occasion to fix that, though, but that is besides my point).
I've got mixed feelings on this one. I'm neutral as well here.
What do you guys think? Is this a good idea, or is it not?
Thank you for your time!-Emanuel 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/7a436aa0/attachment.html>

From rosuav at gmail.com  Mon Nov 30 20:43:28 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 1 Dec 2015 12:43:28 +1100
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>
Message-ID: <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>

On Tue, Dec 1, 2015 at 12:07 PM, Emanuel Barry <vgr255 at live.ca> wrote:
> An idea that I had a couple of times was to have a syntactic way to define
> multiple arguments for a @decorator. This would be good if you want to, for
> example, define a property with a getter and a setter. I had a few ideas,
> but none of them seem to be "the obvious way to do it". I'm including them
> anyway, but I'd rather see if people are interested in the idea or if it's
> just something that's not worth it. I'll gladly write a PEP if it gains a
> lot of interest.

Other than @property, are there any use-cases you know of?

> Method 1: Add parens around all arguments

Definitely -1 on this syntax - it'd make incomplete lines of code hard
to diagnose.

> Method 2: Specify how many arguments in the decorator
>
> class Foo:
>     def __init__(self):
>         self._x = 42
>     @3:property
>     def x(self): # getter
>         return self._x
>     def x(self, value): # setter
>         self._x = value
>     def x(self): # deleter
>         del self._x

-0.5 on this syntax. It's too much action-at-a-distance; if the three
functions are really trivial, then it wouldn't be too bad, but
otherwise how do you know that "def x(self):" is making the deleter?

> Method 3: Specify arguments using the parameters' names
>
> class Foo:
>     def __init__(self):
>         self._x = 42
>     @property
>     def x:fget(self):
>         return self._x
>     def x:fset(self, value):
>         self._x = value
>     def x:fdel(self):
>         del self._x
>
> This has the advantage of being explicit (and the arguments can be swapped
> since they're named), but the disadvantage that some builtins don't accept
> kwargs (this would be a good occasion to fix that, though, but that is
> besides my point).

This is very close to the existing syntax for @property, and it better
adorns the individual functions. +0.5. What happens if you rename the
property, though? How about this:

class Foo:
    def __init__(self):
        self._x = 42
    @property
    def x(self): # First positional parameter
        return self._x
    def :fset(self, value): # Named parameter
        self._x = value
    def :fdel(self): # Another named parameter
        del self._x

Remove the repetition of the name "x", and then there's no chance of
getting it wrong.

But it's sounding here more like you're creating a block of code. And
that, to my mind, suggests that it should be indented. Something like:

class Foo:
    def __init__(self):
        self._x = 42
    with @property as x:
        def fget(self):
            return self._x
        def fset(self, value):
            self._x = value
        def fdel(self):
            del self._x

This groups the three functions, and their names would be available to
use as keyword arguments. It would be rather different from current
with-block semantics, though. Effectively, it'd be something like
this:

1) Evaluate the decorator itself (in this case, the simple name
'property'), but don't call it.
2) Create a new scope, nested inside the current scope. (Similar to a
list comp.)
3) Execute the indented block in that scope.
4) Call the decorator, passing all names bound in this scope as
keyword arguments.
5) Bind the return value of the decorator to the given name.

Thoughts?

ChrisA

From cfkaran2 at gmail.com  Mon Nov 30 20:54:45 2015
From: cfkaran2 at gmail.com (Cem Karan)
Date: Mon, 30 Nov 2015 20:54:45 -0500
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>
 <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
Message-ID: <48DFBAB2-974D-44EE-BB0A-A4F673B0A908@gmail.com>


On Nov 30, 2015, at 8:43 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Tue, Dec 1, 2015 at 12:07 PM, Emanuel Barry <vgr255 at live.ca> wrote:

<<SNIP>>

> But it's sounding here more like you're creating a block of code. And
> that, to my mind, suggests that it should be indented. Something like:
> 
> class Foo:
>    def __init__(self):
>        self._x = 42
>    with @property as x:
>        def fget(self):
>            return self._x
>        def fset(self, value):
>            self._x = value
>        def fdel(self):
>            del self._x
> 
> This groups the three functions, and their names would be available to
> use as keyword arguments. It would be rather different from current
> with-block semantics, though. Effectively, it'd be something like
> this:
> 
> 1) Evaluate the decorator itself (in this case, the simple name
> 'property'), but don't call it.
> 2) Create a new scope, nested inside the current scope. (Similar to a
> list comp.)
> 3) Execute the indented block in that scope.
> 4) Call the decorator, passing all names bound in this scope as
> keyword arguments.
> 5) Bind the return value of the decorator to the given name.

I like this; it really SHOULD be indented.  But I also agree that 'with' is probably not the best keyword here, it makes it a little difficult to quickly read and see what's going on.  Since this is for decorators, could we just drop 'with' altogether?  E.g.:

class Foo:
   def __init__(self):
       self._x = 42
   @property as x:
       def fget(self):
           return self._x
       def fset(self, value):
           self._x = value
       def fdel(self):
           del self._x

Thanks,
Cem Karan

From vgr255 at live.ca  Mon Nov 30 21:01:39 2015
From: vgr255 at live.ca (Emanuel Barry)
Date: Mon, 30 Nov 2015 21:01:39 -0500
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>,
 <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
Message-ID: <BLU172-W165CEEA1D87AF12B2C564E910F0@phx.gbl>

> Date: Tue, 1 Dec 2015 12:43:28 +1100
> From: rosuav at gmail.com
> CC: python-ideas at python.org
> Subject: Re: [Python-ideas] Multiple arguments for decorators
> 
> 
> Other than @property, are there any use-cases you know of?

Not in the builtins/stdlib, but I do have decorators that I'd probably rewrite to support the new syntax. In one of my projects I work around this by some other means, but it looks ugly.
> > Method 1: Add parens around all arguments
> 
> Definitely -1 on this syntax - it'd make incomplete lines of code hard
> to diagnose.

Yep, not surprised -- I only included it because that was the first idea I had, but I myself don't like it :)

> > Method 2: Specify how many arguments in the decorator
> 
> -0.5 on this syntax. It's too much action-at-a-distance; if the three
> functions are really trivial, then it wouldn't be too bad, but
> otherwise how do you know that "def x(self):" is making the deleter?
I guess it does make it a bit of a hassle to maintain, maybe if it was combined with another one it would make sense. We'll see.
> > Method 3: Specify arguments using the parameters' names
> 
> This is very close to the existing syntax for @property, and it better
> adorns the individual functions. +0.5. What happens if you rename the
> property, though? How about this:
> 
> class Foo:
>     def __init__(self):
>         self._x = 42
>     @property
>     def x(self): # First positional parameter
>         return self._x
>     def :fset(self, value): # Named parameter
>         self._x = value
>     def :fdel(self): # Another named parameter
>         del self._x
> 
> Remove the repetition of the name "x", and then there's no chance of
> getting it wrong.
I personally don't like this, the colon there looks weird to me. And you're just ignoring the first method's name and passing it positional, unlike the others which are named. I like the general idea you're bringing though, but it could use a tweak or three imo.
> But it's sounding here more like you're creating a block of code. And
> that, to my mind, suggests that it should be indented. Something like:
> 
> class Foo:
>     def __init__(self):
>         self._x = 42
>     with @property as x:
>         def fget(self):
>             return self._x
>         def fset(self, value):
>             self._x = value
>         def fdel(self):
>             del self._x
> 
> This groups the three functions, and their names would be available to
> use as keyword arguments. It would be rather different from current
> with-block semantics, though. Effectively, it'd be something like
> this:
> 
> 1) Evaluate the decorator itself (in this case, the simple name
> 'property'), but don't call it.
> 2) Create a new scope, nested inside the current scope. (Similar to a
> list comp.)
> 3) Execute the indented block in that scope.
> 4) Call the decorator, passing all names bound in this scope as
> keyword arguments.
> 5) Bind the return value of the decorator to the given name.
> 
> Thoughts?

Glancing at this, it seems to me that "property" is having a unary @ operator applied to it, but I guess that since the possibility to introduce a unary @ operator shrank down to exactly 0 when the decorator syntax was added, that's not really an issue. I'm also not sure about overloading the semantics of the 'with' statement.
Nevertheless, I like this approach. I wonder if something similar (using a with statement) can be achieved right now. Probably, with the use of vars and sys._getframe (I never said it would be clean!)
> ChrisA

Thanks for your input! 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/eca11616/attachment.html>

From rosuav at gmail.com  Mon Nov 30 21:03:52 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 1 Dec 2015 13:03:52 +1100
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <48DFBAB2-974D-44EE-BB0A-A4F673B0A908@gmail.com>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>
 <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
 <48DFBAB2-974D-44EE-BB0A-A4F673B0A908@gmail.com>
Message-ID: <CAPTjJmrd8kGC9PRFbOhTGcFkpMk5QWVPJE4hKO50PzfyChYYqg@mail.gmail.com>

On Tue, Dec 1, 2015 at 12:54 PM, Cem Karan <cfkaran2 at gmail.com> wrote:
> I like this; it really SHOULD be indented.  But I also agree that 'with' is probably not the best keyword here, it makes it a little difficult to quickly read and see what's going on.  Since this is for decorators, could we just drop 'with' altogether?  E.g.:
>
> class Foo:
>    def __init__(self):
>        self._x = 42
>    @property as x:
>        def fget(self):
>            return self._x
>        def fset(self, value):
>            self._x = value
>        def fdel(self):
>            del self._x
>
> Thanks,
> Cem Karan

Either way works for me. My first try was some kind of hack that
actually used a context manager as a context manager, but without
compiler support, it wouldn't really work properly. So the syntax can
be varied away from 'with'.

ChrisA

From rosuav at gmail.com  Mon Nov 30 21:14:48 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 1 Dec 2015 13:14:48 +1100
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <BLU172-W165CEEA1D87AF12B2C564E910F0@phx.gbl>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>
 <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
 <BLU172-W165CEEA1D87AF12B2C564E910F0@phx.gbl>
Message-ID: <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>

On Tue, Dec 1, 2015 at 1:01 PM, Emanuel Barry <vgr255 at live.ca> wrote:
>> Date: Tue, 1 Dec 2015 12:43:28 +1100
>> From: rosuav at gmail.com
>> CC: python-ideas at python.org
>> Subject: Re: [Python-ideas] Multiple arguments for decorators
>>
>>
>> Other than @property, are there any use-cases you know of?
>
> Not in the builtins/stdlib, but I do have decorators that I'd probably
> rewrite to support the new syntax. In one of my projects I work around this
> by some other means, but it looks ugly.

The proposal would be strengthened by more examples. Currently,
@property can do something very similar to what your proposal offers,
so this is only a small improvement.

>> This is very close to the existing syntax for @property, and it better
>> adorns the individual functions. +0.5. What happens if you rename the
>> property, though? How about this:
>>
>> class Foo:
>> def __init__(self):
>> self._x = 42
>> @property
>> def x(self): # First positional parameter
>> return self._x
>> def :fset(self, value): # Named parameter
>> self._x = value
>> def :fdel(self): # Another named parameter
>> del self._x
>>
>> Remove the repetition of the name "x", and then there's no chance of
>> getting it wrong.
>
> I personally don't like this, the colon there looks weird to me. And you're
> just ignoring the first method's name and passing it positional, unlike the
> others which are named. I like the general idea you're bringing though, but
> it could use a tweak or three imo.

Agreed that the colon looks weird, but I don't know of any better way
to spell it. This was just a half-way house to the thought that
followed, though.

>> But it's sounding here more like you're creating a block of code. And
>> that, to my mind, suggests that it should be indented. Something like:
>>
>> class Foo:
>> def __init__(self):
>> self._x = 42
>> with @property as x:
>> def fget(self):
>> return self._x
>> def fset(self, value):
>> self._x = value
>> def fdel(self):
>> del self._x
>>
>> This groups the three functions, and their names would be available to
>> use as keyword arguments. It would be rather different from current
>> with-block semantics, though. Effectively, it'd be something like
>> this:
>>
>> 1) Evaluate the decorator itself (in this case, the simple name
>> 'property'), but don't call it.
>> 2) Create a new scope, nested inside the current scope. (Similar to a
>> list comp.)
>> 3) Execute the indented block in that scope.
>> 4) Call the decorator, passing all names bound in this scope as
>> keyword arguments.
>> 5) Bind the return value of the decorator to the given name.
>>
>> Thoughts?
>
> Glancing at this, it seems to me that "property" is having a unary @
> operator applied to it, but I guess that since the possibility to introduce
> a unary @ operator shrank down to exactly 0 when the decorator syntax was
> added, that's not really an issue. I'm also not sure about overloading the
> semantics of the 'with' statement.
>
> Nevertheless, I like this approach. I wonder if something similar (using a
> with statement) can be achieved right now. Probably, with the use of vars
> and sys._getframe (I never said it would be clean!)

I don't think it can, because there's no way for a context manager to
say "tell me about all name bindings in this block". The nearest I can
think of is a nested class definition, which I can make work, but it's
definitely ugly:

def call(func):
    def inner(cls):
        return func(**{k:v for k,v in cls.__dict__.items() if not
k.startswith('_')})
    return inner

class Foo:
    def __init__(self):
        self._x = 42
    @call(property)
    class x:
        def fget(self):
            return self._x
        def fset(self, value):
            self._x = value
        def fdel(self):
            del self._x


foo = Foo()
print(foo.x)
foo.x = 28
print(foo._x)

ChrisA

From mertz at gnosis.cx  Mon Nov 30 21:41:15 2015
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 30 Nov 2015 18:41:15 -0800
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>
 <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
 <BLU172-W165CEEA1D87AF12B2C564E910F0@phx.gbl>
 <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
Message-ID: <CAEbHw4ZsBZYXeZupn3jCUQ-musNozBRVvPDD5RLwww7hfDdZxg@mail.gmail.com>

On Mon, Nov 30, 2015 at 6:14 PM, Chris Angelico <rosuav at gmail.com> wrote:

> def call(func):
>     def inner(cls):
>         return func(**{k:v for k,v in cls.__dict__.items() if not
> k.startswith('_')})
>     return inner
>
> class Foo:
>     def __init__(self):
>         self._x = 42
>     @call(property)
>     class x:
>         def fget(self):
>             return self._x
>         def fset(self, value):
>             self._x = value
>         def fdel(self):
>             del self._x
>

I think this looks perfectly nice, actually.  I was just trying to work out
almost the same thing but using a `def x()` rather than `class f` as the
nesting construct.  I think Chris' is better though.  I think I might want
to define something like:

make_property = call(property)

class Foo:
    def __init__(self):
        self._x = 42
    @make_property
    class x:
        def fget(self):
            return self._x
        def fset(self, value):
            self._x = value
        def fdel(self):
            del self._x



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/27ceff0e/attachment.html>

From vgr255 at live.ca  Mon Nov 30 21:43:14 2015
From: vgr255 at live.ca (Emanuel Barry)
Date: Mon, 30 Nov 2015 21:43:14 -0500
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>,
 <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>,
 <BLU172-W165CEEA1D87AF12B2C564E910F0@phx.gbl>,
 <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
Message-ID: <BLU172-W2764C914ADA6B13A8262CC910F0@phx.gbl>

> The proposal would be strengthened by more examples. Currently,
> @property can do something very similar to what your proposal offers,
> so this is only a small improvement.

While searching for more examples, it struck me that I wouldn't want to allow only methods to be passed, but I would want any value (for the name and/or docstring, mainly). For example, consider the following code:
class Foo:    @attribute    def bar(self):        return 42    bar.__doc__ = "The meaning of life, the universe, and everything."

I would benefit from being able to pass in the doc at the same time, as well. That is a poor example though, here is a better one:
@total_decorate(handler=custom_handler, name="special_cased")class Foo:    <...>
Or, as I do sometimes:
@total_decorate()class Foo:    <...>
This would benefit from being able to make that a single call (given some changes to the code) and avoid the ugly parens, that I see quite often. This example could be extended to reprlib.recursive_repr too.
I must admit that I currently lack in examples, I'll try to provide more tomorrow. I took property because it's simple and everyone knows about it (pretty much).
> > I personally don't like this, the colon there looks weird to me. And you're
> > just ignoring the first method's name and passing it positional, unlike the
> > others which are named. I like the general idea you're bringing though, but
> > it could use a tweak or three imo.
> 
> Agreed that the colon looks weird, but I don't know of any better way
> to spell it. This was just a half-way house to the thought that
> followed, though.

I'm not particularly attached to any syntax, all my suggestions are only that - suggestions.

> I don't think it can, because there's no way for a context manager to
> say "tell me about all name bindings in this block". The nearest I can
> think of is a nested class definition, which I can make work, but it's
> definitely ugly:

Well, you store all the names in the previous frame in __enter__, and then do your stuff in __exit__. It *is* hacky though.
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/4e685a5f/attachment.html>

From mertz at gnosis.cx  Mon Nov 30 21:49:48 2015
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 30 Nov 2015 18:49:48 -0800
Subject: [Python-ideas] [Python-ideos] Dedicated overloadable boolean
 operators
In-Reply-To: <CAP7+vJJrMZfM4LDsmy525hPjL8g1m45Qi9EmjFw5MA10ReJJFQ@mail.gmail.com>
References: <CAGECzQT2RGCyVdozLjNW6zSiPgSfLSteFyCcDp9sm02NtYePxw@mail.gmail.com>
 <BLU172-W44AE63DC970FDB54BC7F0191070@phx.gbl>
 <cc286f69-0da6-4244-8e65-87aa484c8943@googlegroups.com>
 <CAPTjJmowv5F=3nXwBjGjVWcSBqW2G8mOqpm_P=ZQeFHoSA3VeA@mail.gmail.com>
 <CAP7+vJLqQL8QF-5xkY+18J_oBTb9hBE1oG3qni12XhJB95y2pA@mail.gmail.com>
 <-334649383431326154@unknownmsgid>
 <CADiSq7d9=_36AfCD7itMEOtRyr8mHsM7Zck395F62nM6qrR_zQ@mail.gmail.com>
 <5653FE15.8080402@canterbury.ac.nz>
 <CADiSq7eY3ApTB3Y8D68A=i8KLsqFDprPVRKK5iqxf5ojMSEe3A@mail.gmail.com>
 <3518005034548011247@unknownmsgid>
 <CADiSq7fQGEUeFMD5xwRkhdYB0PduUdy_=NUv3_NWCSrB==Yc8Q@mail.gmail.com>
 <7EC853F0-CAA3-4746-AD0F-D5B7B0B3A955@yahoo.com>
 <56577194.5000003@canterbury.ac.nz>
 <CADiSq7dQ-Myk7k7shUFyf1fx5n0ww5NERWQ3s3Db2J9=pHqu4g@mail.gmail.com>
 <4A719937-E966-4D95-BFFE-1BEAEC8E7797@yahoo.com>
 <5658B252.1060902@brenbarn.net>
 <CAP7+vJLVWh--+b4qHkawMXVPQ+V8ZifpRUPocnfR+Jm55CVquQ@mail.gmail.com>
 <23d4f7d4-2cf0-49d5-9c18-71b1edce216f@googlegroups.com>
 <67d34a7d-d634-4a0f-bdf9-5a79015523f6@googlegroups.com>
 <CAP7+vJ+9gXPct0aK0PLwzYwPeZuF-L+BYYDJATQY8_NLrs0pkg@mail.gmail.com>
 <7df3290f-c021-4e8b-b71b-f49be3f1554b@googlegroups.com>
 <CAP7+vJJrMZfM4LDsmy525hPjL8g1m45Qi9EmjFw5MA10ReJJFQ@mail.gmail.com>
Message-ID: <CAEbHw4abCwf_CRwobzZ-7iKsfGU08pV_ji_i-Je5fZDNdgT4ZQ@mail.gmail.com>

I find the need to use np.logical_or quite often in NumPy.  I'm sure the
NumPy community would much prefer to be able to write:

    new = arr[arr<10 or arr>100]

Instead of the current:

    new = arr[np.logical_or(arr<10, arr>100)]

Likewise for np.logical_and().


On Mon, Nov 30, 2015 at 10:23 AM, Guido van Rossum <guido at python.org> wrote:

> Well, there is still the issue of whether numpy would adopt PEP 335 even
> if it was implemented. I'm still waiting for a response to my message
> questioning how common `bool_value and/or numpy_array` really is.
>
> Also, there may still be a significant difference between PEP 335 and your
> proposal -- IIUC in your proposal if the value of the first operand
> requires the second operand to be evaluated (e.g. `True or y`) then in your
> proposal if y defined the 'r' special method it will be used; I don't think
> PEP 335 does that. I like your version better.
>
> On Mon, Nov 30, 2015 at 10:08 AM, Henry Schreiner <
> henryschreineriii at gmail.com> wrote:
>
>> I didn't realize the similarity of the proposals. I do think that testing
>> for existence is better than running a method (that might not exist), at
>> least from a high level perspective.
>>
>> So PEP 335 might be dusted off again and run through benchmarks? There
>> seems to be at least a test implementation for Python 2.3, are there
>> benchmarks documented somewhere? It sound like the version that adds new
>> byte codes instead of an additional byte code per op was never included.
>>
>> I do like the cleanliness of not adding extra operators with nearly
>> identical meanings.
>>
>> On Monday, November 30, 2015 at 11:24:12 AM UTC-6, Guido van Rossum wrote:
>>>
>>> On Mon, Nov 30, 2015 at 8:22 AM, Henry Schreiner <henrysch... at gmail.com>
>>> wrote:
>>>
>>>> Sorry, the order was incorrect in that last message. It should be:
>>>>
>>>> When x and y is written, if x has __booland__, then x.__booland__(y) is
>>>> returned. If not, then return x if x==False else if x==True, then if y
>>>>  has y.__rbooland__(x), that is returned. If not, then return y
>>>>
>>>
>>> This proposal isn't significantly different from PEP 335. In PEP 335,
>>> first you ask the left operand whether to overload 'and' or not (by calling
>>> its __and1__); if it says yes, you evaluate the second arg and ask the
>>> first to evaluate it. In your proposal, the first question is answered by
>>> inspecting whether x defines __booland__. I don't think this makes much
>>> practical difference. The second part of your proposal is just a renaming
>>> of __and2__/__rand2__ to __booland__/__rbooland__. I'm not sure I like this
>>> much; the return type of these methods is almost never going to be an
>>> actual bool, so the 'bool' in the name is somewhat misleading. (The names
>>> chosen in PEP 335 are intentionally free of semantic hints, since the
>>> intuition of someone who encounters these for the first time isn't likely
>>> to be useful.)
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/5495b077/attachment-0001.html>

From abarnert at yahoo.com  Mon Nov 30 21:52:11 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 1 Dec 2015 02:52:11 +0000 (UTC)
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
References: <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
Message-ID: <1745702014.10551218.1448938331185.JavaMail.yahoo@mail.yahoo.com>

On Monday, November 30, 2015 6:14 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Tue, Dec 1, 2015 at 1:01 PM, Emanuel Barry <vgr255 at live.ca> wrote:


> The proposal would be strengthened by more examples. Currently,
> @property can do something very similar to what your proposal offers,
> so this is only a small improvement.

Agreed. And it's not clear why other decorators can't just do something similar to what @property does.


>>>  But it's sounding here more like you're creating a block of 
> code. And
>>>  that, to my mind, suggests that it should be indented. 

Agreed. The way to group things in Python is with an indented suite; trying to read three tiny things at the same level as being controlled by something above all of them isn't too terrible, but if there are three big things, or eight tiny things?


>>>  class Foo:

>>>  def __init__(self):
>>>  self._x = 42
>>>  with @property as x:
>>>  def fget(self):
>>>  return self._x
>>>  def fset(self, value):
>>>  self._x = value
>>>  def fdel(self):
>>>  del self._x
>>> 
>>>  This groups the three functions, and their names would be available to
>>>  use as keyword arguments. It would be rather different from current
>>>  with-block semantics, though.

I think this is a bad idea from the start. Only functions and classes have scopes; normal suite do not. If you change that to add "... except the suite of a with statement whose context manager is a decorator", that's no longer a simple rule you can hold in your head.

> The nearest I can
> think of is a nested class definition, which I can make work, but it's
> definitely ugly:

It seems a lot cleaner to just pass a class to the decorator:

class Property:
    def __init__(self, cls):
        self.fget = getattr(cls, 'fget', None)
        self.fset = getattr(cls, 'fset', None)
        self.fdel = getattr(cls, 'fdel', None)
        self.doc = getattr(cls, '__doc__', None)
        # everything below this point is exactly the same as the 
        # existing implementation in the descriptor HOWTO (or
        # the C implementation in descrobject.c).

class Foo:
    def __init__(self):
        self._x = 42
    @Property
    class x:
        def fget(self):
            return self._x
        def fset(self, value):
            self._x = value
        def fdel(self):
            del self._x


Sure, @call slightly simplifies those 4 lines of boilerplate at the start of Property.__init__, and does the same for every other decorator that you define to be used with @call--but at the cost of making every _use_ of every such decorator uglier, and making things more complex to think through. Is it really worth it? Actually, that's not a rhetorical question; it's hard to guess without seeing more examples beyond @property...

From rosuav at gmail.com  Mon Nov 30 21:58:37 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 1 Dec 2015 13:58:37 +1100
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <1745702014.10551218.1448938331185.JavaMail.yahoo@mail.yahoo.com>
References: <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
 <1745702014.10551218.1448938331185.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <CAPTjJmpy36OrY_yX1oR-=WGE1vDBG3yHi_6_VVnF7vv1T5nE1A@mail.gmail.com>

On Tue, Dec 1, 2015 at 1:52 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> I think this is a bad idea from the start. Only functions and classes have scopes; normal suite do not. If you change that to add "... except the suite of a with statement whose context manager is a decorator", that's no longer a simple rule you can hold in your head.
>

And list comprehensions. They have a new nested scope, even though you
can't see the function call. Everywhere else that you see a 'for'
loop, it's exactly the same semantics as any other assignment - but if
it's inside square brackets, it's a special form of local name that
doesn't extend past the brackets. This would be the same - a hidden
function call that creates a nested scope.

ChrisA

From stephen at xemacs.org  Mon Nov 30 22:16:30 2015
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 1 Dec 2015 12:16:30 +0900
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <CAEbHw4ZsBZYXeZupn3jCUQ-musNozBRVvPDD5RLwww7hfDdZxg@mail.gmail.com>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>
 <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
 <BLU172-W165CEEA1D87AF12B2C564E910F0@phx.gbl>
 <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
 <CAEbHw4ZsBZYXeZupn3jCUQ-musNozBRVvPDD5RLwww7hfDdZxg@mail.gmail.com>
Message-ID: <22109.4366.757509.63946@turnbull.sk.tsukuba.ac.jp>

David Mertz writes:
 > On Mon, Nov 30, 2015 at 6:14 PM, Chris Angelico <rosuav at gmail.com> wrote:

 >> def call(func):
 >>     def inner(cls):
 >>         return func(**{k:v for k,v in cls.__dict__.items() if not
 >> k.startswith('_')})
 >>     return inner
 >>
 >> class Foo:
 >>     def __init__(self):
 >>         self._x = 42
 >>     @call(property)
 >>     class x:
 >>         def fget(self):
 >>             return self._x
 >>         def fset(self, value):
 >>             self._x = value
 >>         def fdel(self):
 >>             del self._x
 >>

 > I think this looks perfectly nice, actually.

That was my first thought too.  The use of the "class" keyword was a
bit of a shock at first, but it grew on me, especially in this context
of @property (ie, if I think "@property defines a class with appropriate
behavior and installs instances on objects of the enclosing class").

I'm having trouble interpreting the function name "call", though.  Am
I missing something obvious ("Angelico" doesn't sound Dutch to me,
though :-), or maybe there's a better (longer) name for it?

Also, can call be expected to DTRT for anything *but* property?


From kmod at dropbox.com  Mon Nov 30 22:21:57 2015
From: kmod at dropbox.com (Kevin Modzelewski)
Date: Mon, 30 Nov 2015 19:21:57 -0800
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <CAEbHw4ZsBZYXeZupn3jCUQ-musNozBRVvPDD5RLwww7hfDdZxg@mail.gmail.com>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>
 <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
 <BLU172-W165CEEA1D87AF12B2C564E910F0@phx.gbl>
 <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
 <CAEbHw4ZsBZYXeZupn3jCUQ-musNozBRVvPDD5RLwww7hfDdZxg@mail.gmail.com>
Message-ID: <CAO=oM6ue=P0LmbBAzZsGOH2cJtckWkEnHmfSxf8ypR0BCJQv1w@mail.gmail.com>

Class scopes definitely feel like a good match -- they are a way of saying
"evaluate all of these expression, pass the resulting locals to a custom
function, and bind the result of that function to the classname".  Usually
the function is type(), which constructs a new class, but by setting a
custom metaclass we can avoid creating a class just to wrap the scope:

class PropertyMetaclass(type):
    def __new__(cls, name, bases, attrs):
        return property(attrs.get('get'), attrs.get('set'),
attrs.get('del'), attrs.get('__doc__'))

class Foo(object):
    class myprop(metaclass=PropertyMetaclass):
        def get(self):
            return 1
        def set(self, v):
            pass
        __doc__ = 1

f = Foo()
print(f.myprop)


The "class myprop(metaclass=PropertyClass)" line is pretty ugly though.

On Mon, Nov 30, 2015 at 6:41 PM, David Mertz <mertz at gnosis.cx> wrote:

> On Mon, Nov 30, 2015 at 6:14 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
>> def call(func):
>>     def inner(cls):
>>         return func(**{k:v for k,v in cls.__dict__.items() if not
>> k.startswith('_')})
>>     return inner
>>
>> class Foo:
>>     def __init__(self):
>>         self._x = 42
>>     @call(property)
>>     class x:
>>         def fget(self):
>>             return self._x
>>         def fset(self, value):
>>             self._x = value
>>         def fdel(self):
>>             del self._x
>>
>
> I think this looks perfectly nice, actually.  I was just trying to work
> out almost the same thing but using a `def x()` rather than `class f` as
> the nesting construct.  I think Chris' is better though.  I think I might
> want to define something like:
>
> make_property = call(property)
>
> class Foo:
>     def __init__(self):
>         self._x = 42
>     @make_property
>     class x:
>         def fget(self):
>             return self._x
>         def fset(self, value):
>             self._x = value
>         def fdel(self):
>             del self._x
>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20151130/cef752ab/attachment-0001.html>

From rosuav at gmail.com  Mon Nov 30 22:32:13 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 1 Dec 2015 14:32:13 +1100
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <22109.4366.757509.63946@turnbull.sk.tsukuba.ac.jp>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>
 <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
 <BLU172-W165CEEA1D87AF12B2C564E910F0@phx.gbl>
 <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
 <CAEbHw4ZsBZYXeZupn3jCUQ-musNozBRVvPDD5RLwww7hfDdZxg@mail.gmail.com>
 <22109.4366.757509.63946@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CAPTjJmrXUY7-ne3niABLjGzv5G6jn6HaT76HcKcU3=O=F9r+EA@mail.gmail.com>

On Tue, Dec 1, 2015 at 2:16 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> I'm having trouble interpreting the function name "call", though.  Am
> I missing something obvious ("Angelico" doesn't sound Dutch to me,
> though :-), or maybe there's a better (longer) name for it?
>
> Also, can call be expected to DTRT for anything *but* property?

No, it's not Dutch... but my maternal grandparents were Dutch, if that
helps? (My parents were both born here in Australia, but my
grandparents are split two-and-two Dutch and Sicilian. So I get to eat
garlic like nobody's business, and then pretend to understand Python.
Maybe.)

The semantics of the function I've given are simple: Pass it a
callable and a block of assignments (implemented as a class body), and
it calls the callable with the assignments as keyword arguments.

So it'll work for anything that accepts keyword arguments - which in
Python is nearly anything. It doesn't necessarily have to carry
callables:

import json
@call(json.loads)
class config:
    with open("config.json") as _f: # Leading underscore for non-arguments
        s = _f.read()
    encoding = "UTF-8"
    def object_hook(d):
        if "imag" in d and "real" in d and len(d)==2:
            return complex(d["real"], d["imag"])
        return d
    parse_int = float

import pprint
pprint.pprint(config)

Incidentally, this works in Python 2 as well as Python 3. I found out
by typoing the command and seeing a bunch of u'...' strings in the
pprint output - no other difference.

ChrisA

From abarnert at yahoo.com  Mon Nov 30 23:30:04 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 30 Nov 2015 20:30:04 -0800
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <CAPTjJmpy36OrY_yX1oR-=WGE1vDBG3yHi_6_VVnF7vv1T5nE1A@mail.gmail.com>
References: <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
 <1745702014.10551218.1448938331185.JavaMail.yahoo@mail.yahoo.com>
 <CAPTjJmpy36OrY_yX1oR-=WGE1vDBG3yHi_6_VVnF7vv1T5nE1A@mail.gmail.com>
Message-ID: <650A2120-A263-4568-8C4D-D1B4241B9382@yahoo.com>

On Nov 30, 2015, at 18:58, Chris Angelico <rosuav at gmail.com> wrote:
> 
>> On Tue, Dec 1, 2015 at 1:52 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
>> I think this is a bad idea from the start. Only functions and classes have scopes; normal suite do not. If you change that to add "... except the suite of a with statement whose context manager is a decorator", that's no longer a simple rule you can hold in your head.
> 
> And list comprehensions.

Comprehensions define functions, so it's the same rule; you just have to know that functions can be defined three ways (def statement, lambdas expression, or comprehension) rather than just two. Sure, that's not _ideally_ simple, but that hardly seems a reason to make it even _less_ simple.

Also, comprehensions don't have a suite--you can't define arbitrary new variables or use global/nonlocal statements or anything else that would make you have to think carefully about scoping.

Most importantly, comprehensions don't have a suite that looks exactly the same as another kind of suite introduced by the same keyword that doesn't define a scope, except on this one special case where it does.


From abarnert at yahoo.com  Mon Nov 30 23:37:17 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 30 Nov 2015 20:37:17 -0800
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <CAO=oM6ue=P0LmbBAzZsGOH2cJtckWkEnHmfSxf8ypR0BCJQv1w@mail.gmail.com>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>
 <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
 <BLU172-W165CEEA1D87AF12B2C564E910F0@phx.gbl>
 <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
 <CAEbHw4ZsBZYXeZupn3jCUQ-musNozBRVvPDD5RLwww7hfDdZxg@mail.gmail.com>
 <CAO=oM6ue=P0LmbBAzZsGOH2cJtckWkEnHmfSxf8ypR0BCJQv1w@mail.gmail.com>
Message-ID: <C1DC0AA6-A02A-44B9-BC06-678050375C07@yahoo.com>

On Nov 30, 2015, at 19:21, Kevin Modzelewski via Python-ideas <python-ideas at python.org> wrote:
> 
> Class scopes definitely feel like a good match -- they are a way of saying "evaluate all of these expression, pass the resulting locals to a custom function, and bind the result of that function to the classname".  Usually the function is type(), which constructs a new class, but by setting a custom metaclass we can avoid creating a class just to wrap the scope:

Is there really a harm in creating a class?

A property is a type, and the obvious way to simulate it in Python rather than C (as shown by the sample code in the HOWTO) is with a class statement.

Besides, if you're creating many thousands of properties in a loop, the time and space cost of property creation is probably the least of your worries.

Again, maybe that isn't true for other types of decorators this feature might be useful for, but without having any examples to think about, it's hard to guess...

> class PropertyMetaclass(type):
>     def __new__(cls, name, bases, attrs):
>         return property(attrs.get('get'), attrs.get('set'), attrs.get('del'), attrs.get('__doc__'))

I still don't get the benefit of having a metaclass or constructor function or wrapper function or anything else, instead of just making property take a class instead of four functions. The latter is significantly nicer on the user side, and only a tiny bit more verbose in the implementation of property, and easier to understand. Unless there are other decorators where they wouldn't be true, or so many potentially useful one-shot decorators that defining them all a little more succinctly is worth the cost, why add the extra layer?

From rosuav at gmail.com  Mon Nov 30 23:38:26 2015
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 1 Dec 2015 15:38:26 +1100
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <650A2120-A263-4568-8C4D-D1B4241B9382@yahoo.com>
References: <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
 <1745702014.10551218.1448938331185.JavaMail.yahoo@mail.yahoo.com>
 <CAPTjJmpy36OrY_yX1oR-=WGE1vDBG3yHi_6_VVnF7vv1T5nE1A@mail.gmail.com>
 <650A2120-A263-4568-8C4D-D1B4241B9382@yahoo.com>
Message-ID: <CAPTjJmr-arw5+J=V1fb5HvWOG_rtQetVV6VTTKEwAFcWEKqiCA@mail.gmail.com>

On Tue, Dec 1, 2015 at 3:30 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> Comprehensions define functions, so it's the same rule; you just have to know that functions can be defined three ways (def statement, lambdas expression, or comprehension) rather than just two. Sure, that's not _ideally_ simple, but that hardly seems a reason to make it even _less_ simple.
>

Which I don't really like. It makes sense for generator expressions,
but list comprehensions don't *look* like they introduce a new
sub-scope. A 'with' block does look like it creates a new scope, but
its binding leaks out (usually giving you a closed file object or
something). Sure, I can understand why things are the way they are,
but it's not intuitive. You have to dig into things a bit to grok it.

> Most importantly, comprehensions don't have a suite that looks exactly the same as another kind of suite introduced by the same keyword that doesn't define a scope, except on this one special case where it does.
>

Agreed. And while I called the class-based system "ugly" to start
with, I'm coming around to it more and more - especially since it
works in current versions of Python, rather than demanding core
interpreter changes. It's not the most intuitive use of syntax either
(the 'class' block isn't really creating a class at all - it creates a
function parameter list), but it isn't as bad as I thought it was.

ChrisA

From abarnert at yahoo.com  Mon Nov 30 23:47:09 2015
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 30 Nov 2015 20:47:09 -0800
Subject: [Python-ideas] Multiple arguments for decorators
In-Reply-To: <C1DC0AA6-A02A-44B9-BC06-678050375C07@yahoo.com>
References: <BLU172-W49BD4295306EC89D13651D910F0@phx.gbl>
 <CAPTjJmp4Pu0rYyU2eVe8kboGCPVH_fM51JuK4eDY_JdEff7_zQ@mail.gmail.com>
 <BLU172-W165CEEA1D87AF12B2C564E910F0@phx.gbl>
 <CAPTjJmqXdG-TUbGLTHLHSxxw7pAW9FBs-nWrb8z_r2oa6U3=Jw@mail.gmail.com>
 <CAEbHw4ZsBZYXeZupn3jCUQ-musNozBRVvPDD5RLwww7hfDdZxg@mail.gmail.com>
 <CAO=oM6ue=P0LmbBAzZsGOH2cJtckWkEnHmfSxf8ypR0BCJQv1w@mail.gmail.com>
 <C1DC0AA6-A02A-44B9-BC06-678050375C07@yahoo.com>
Message-ID: <A7BB1B6E-A342-4A32-AA1F-53E917CC220B@yahoo.com>

Tangent here:

Is this potentially a good use case for Nick Coghlan's PEP 403 (@in clauses) and/or PEP 3150 (statement-local namespaces with "given:")?

At first glance, the obvious way to use PEP 3150 doesn't look beautiful:

    class Spam:
        x = property(**?.__dict__) given:
            """x"""
            def fget(self): return self._x
            def fset(self, value): self._x = value
            def fdel(self): del self._x

But maybe there's a way to make this nicer?

> On Nov 30, 2015, at 20:37, Andrew Barnert via Python-ideas <python-ideas at python.org> wrote:
> 
>> On Nov 30, 2015, at 19:21, Kevin Modzelewski via Python-ideas <python-ideas at python.org> wrote:
>> 
>> Class scopes definitely feel like a good match -- they are a way of saying "evaluate all of these expression, pass the resulting locals to a custom function, and bind the result of that function to the classname".  Usually the function is type(), which constructs a new class, but by setting a custom metaclass we can avoid creating a class just to wrap the scope:
> 
> Is there really a harm in creating a class?
> 
> A property is a type, and the obvious way to simulate it in Python rather than C (as shown by the sample code in the HOWTO) is with a class statement.
> 
> Besides, if you're creating many thousands of properties in a loop, the time and space cost of property creation is probably the least of your worries.
> 
> Again, maybe that isn't true for other types of decorators this feature might be useful for, but without having any examples to think about, it's hard to guess...
> 
>> class PropertyMetaclass(type):
>>    def __new__(cls, name, bases, attrs):
>>        return property(attrs.get('get'), attrs.get('set'), attrs.get('del'), attrs.get('__doc__'))
> 
> I still don't get the benefit of having a metaclass or constructor function or wrapper function or anything else, instead of just making property take a class instead of four functions. The latter is significantly nicer on the user side, and only a tiny bit more verbose in the implementation of property, and easier to understand. Unless there are other decorators where they wouldn't be true, or so many potentially useful one-shot decorators that defining them all a little more succinctly is worth the cost, why add the extra layer?
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/