From castironpi at gmail.com  Sun Aug  3 19:46:08 2014
From: castironpi at gmail.com (Aaron Brady)
Date: Sun, 3 Aug 2014 12:46:08 -0500
Subject: [Python-ideas] Mutating while iterating
In-Reply-To: <CADiSq7cG0Cfp0AVn2xeVx-rRyqzeGUPV3XWmQQcXSeMa=02LPA@mail.gmail.com>
References: <cd149d5b-5b57-4fbe-a639-a7f13fae64cf@googlegroups.com>
 <CADiSq7cG0Cfp0AVn2xeVx-rRyqzeGUPV3XWmQQcXSeMa=02LPA@mail.gmail.com>
Message-ID: <CAEX1CPOxUiqcC860zxb9kyNTM6Dc6jVmqe0FE0G151GUF==vhQ@mail.gmail.com>

On Sat, Jul 26, 2014 at 10:39 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 27 July 2014 06:51, Aaron Brady <castironpi at gmail.com> wrote:
>> Hi, I asked about the inconsistency of the "RuntimeError" being raised when
>> mutating a container while iterating over it here [1], "set and dict
>> iteration" on Aug 16, 2012.
>
> Hi,
>
> This is clearly an issue of grave concern to you, but as Raymond
> pointed out previously, you appear to have misunderstood the purpose
> of those exceptions. They're there to prevent catastrophic failure of
> the interpreter itself (i.e. segmentation faults), not to help find
> bugs in user code. If users want to mutate containers while they're
> iterating over them, they're generally free to do so. The only time
> we'll actively disallow it is when such mutation will outright *break*
> the iterator, rather than merely producing potentially surprising
> results.
>
> I have closed the new issue and added a longer reply (with examples)
> that will hopefully better explain why we have no intention of
> changing this behaviour: http://bugs.python.org/issue22084#msg224100


Python is replete with examples of prohibiting structures which are
likely bugs but aren't segfaults.  There are also reciprocal-- though
not necessarily inverse-- limitations of the unordered collections
themselves ("set" and "dict").  The new behavior is transparent for
the programmer; no possible programs can /rely/ on the existing
behavior.  The new behavior introduces no new objects, possibly except
"IterationError", no new syntax, and no new costs.

I propose we leave this discussion thread open for the time being.  I
also take the issue of /re/-assigning to keys during iteration to be
settled as permitted.

From ram.rachum at gmail.com  Mon Aug  4 18:17:14 2014
From: ram.rachum at gmail.com (Ram Rachum)
Date: Mon, 4 Aug 2014 09:17:14 -0700 (PDT)
Subject: [Python-ideas] strptime without second argument as an inverse to
	__str__
Message-ID: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>

What do you think about having `datetime.strptime`, when called without a 
`format` for the second argument, be a precise inverse of 
`datetime.__str__`? This is because I don't currently see an obvious way to 
get an inverse of `datetime.__str__`, and this seems like an okay place to 
put it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140804/0887383a/attachment.html>

From steve at pearwood.info  Mon Aug  4 20:15:28 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 5 Aug 2014 04:15:28 +1000
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
Message-ID: <20140804181528.GP4525@ando>

On Mon, Aug 04, 2014 at 09:17:14AM -0700, Ram Rachum wrote:
> What do you think about having `datetime.strptime`, when called without a 
> `format` for the second argument, be a precise inverse of 
> `datetime.__str__`? This is because I don't currently see an obvious way to 
> get an inverse of `datetime.__str__`, and this seems like an okay place to 
> put it.

Is str(datetime) guaranteed to use a specific format, or is that an 
implementation detail?


-- 
Steven

From alexander.belopolsky at gmail.com  Mon Aug  4 20:40:57 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 4 Aug 2014 14:40:57 -0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <20140804181528.GP4525@ando>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
Message-ID: <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>

On Mon, Aug 4, 2014 at 2:15 PM, Steven D'Aprano <steve at pearwood.info> wrote:

> > What do you think about having `datetime.strptime`, when called without a
> > `format` for the second argument, be a precise inverse of
> > `datetime.__str__`? This is because I don't currently see an obvious way
> to
> > get an inverse of `datetime.__str__`, and this seems like an okay place
> to
> > put it.
>
> Is str(datetime) guaranteed to use a specific format, or is that an
> implementation detail?


Why is this question relevant for Ram's proposal?  As long as str(datetime)
is guaranteed to be different for different datetimes, one should be able
to implement an inverse.  The inverse function should accept ISO format
(with either ' ' or 'T' separator) and str(datetime) if it is different in
the implementation.

I agree that datetime type should provide a simple way to construct
 instances from well-formatted strings, but I don't think
datetime.strptime() is a good choice of name.  I would much rather have
date(str), time(str) and datetime(str) constructors.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140804/1968fb75/attachment.html>

From skip at pobox.com  Mon Aug  4 21:00:17 2014
From: skip at pobox.com (Skip Montanaro)
Date: Mon, 4 Aug 2014 14:00:17 -0500
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
Message-ID: <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>

On Mon, Aug 4, 2014 at 1:40 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
> Why is this question relevant for Ram's proposal?

It would seem to have some impact on how hard it is to create a
general inverse. Will one format work for all platforms ("one and
done"), or will the inverse implementation potentially have to be
updated as new platforms come into (or go out of) existence?

Also, would the creation of such an inverse lock the implementation
into existing format(s)? For example, when fed a datetime object, the
CSV module will stringify it for output. If I create a CSV file with
one version of Python, then read it into another version of Python (or
on a different platform), it's not unreasonable that I would expect
one-argument strptime() to parse it. That would lock you into a
specific format. If only one format exists today, no big deal, bless
it and move on.

Skip

From alexander.belopolsky at gmail.com  Mon Aug  4 21:23:08 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 4 Aug 2014 15:23:08 -0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
Message-ID: <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>

On Mon, Aug 4, 2014 at 3:00 PM, Skip Montanaro <skip at pobox.com> wrote:

> > Why is this question relevant for Ram's proposal?
>
> It would seem to have some impact on how hard it is to create a
> general inverse. Will one format work for all platforms ("one and
> done"), or will the inverse implementation potentially have to be
> updated as new platforms come into (or go out of) existence?


I think str(datetime) format is an implementation detail to the same extent
as str(int) or str(float) is.  In the past, these variations did not
prevent providing (sometimes imperfect) inverse.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140804/56c8995c/attachment.html>

From skip at pobox.com  Mon Aug  4 22:14:11 2014
From: skip at pobox.com (Skip Montanaro)
Date: Mon, 4 Aug 2014 15:14:11 -0500
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
Message-ID: <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>

On Mon, Aug 4, 2014 at 2:23 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
>
> On Mon, Aug 4, 2014 at 3:00 PM, Skip Montanaro <skip at pobox.com> wrote:
>>
>> > Why is this question relevant for Ram's proposal?
>>
>> It would seem to have some impact on how hard it is to create a
>> general inverse. Will one format work for all platforms ("one and
>> done"), or will the inverse implementation potentially have to be
>> updated as new platforms come into (or go out of) existence?
>
>
> I think str(datetime) format is an implementation detail to the same extent
> as str(int) or str(float) is.  In the past, these variations did not prevent
> providing (sometimes imperfect) inverse.

I took a look at whatever version of CPython I have laying about (some
variant of 2.7). str(datetime) seems to be well-defined as calling
isoformat with " " as the separator. The only caveat is that if the
microsecond field is zero, it's omitted.

If that behavior holds true in 3.x, only two cases require consideration:

%Y-%m-%d %H:%M:%S
%Y-%m-%d %H:%M:%S.%f

Skip

From wolfgang.maier at biologie.uni-freiburg.de  Mon Aug  4 22:56:56 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Mon, 04 Aug 2014 22:56:56 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
Message-ID: <lros2p$kdd$1@ger.gmane.org>

On 04.08.2014 22:14, Skip Montanaro wrote:
> On Mon, Aug 4, 2014 at 2:23 PM, Alexander Belopolsky
> <alexander.belopolsky at gmail.com> wrote:
>>
>> On Mon, Aug 4, 2014 at 3:00 PM, Skip Montanaro <skip at pobox.com> wrote:
>>>
>>>> Why is this question relevant for Ram's proposal?
>>>
>>> It would seem to have some impact on how hard it is to create a
>>> general inverse. Will one format work for all platforms ("one and
>>> done"), or will the inverse implementation potentially have to be
>>> updated as new platforms come into (or go out of) existence?
>>
>>
>> I think str(datetime) format is an implementation detail to the same extent
>> as str(int) or str(float) is.  In the past, these variations did not prevent
>> providing (sometimes imperfect) inverse.
>
> I took a look at whatever version of CPython I have laying about (some
> variant of 2.7). str(datetime) seems to be well-defined as calling
> isoformat with " " as the separator. The only caveat is that if the
> microsecond field is zero, it's omitted.
>
> If that behavior holds true in 3.x, only two cases require consideration:
>

it does hold true in 3.x, but the documented behavior is slightly more 
complex (I assume also in 2.x):

datetime.__str__()
     For a datetime instance d, str(d) is equivalent to d.isoformat(' ').

datetime.isoformat(sep='T')

     Return a string representing the date and time in ISO 8601 format, 
YYYY-MM-DDTHH:MM:SS.mmmmmm or, if microsecond is 0, YYYY-MM-DDTHH:MM:SS

     If utcoffset() does not return None, a 6-character string is 
appended, giving the UTC offset in (signed) hours and minutes: 
YYYY-MM-DDTHH:MM:SS.mmmmmm+HH:MM or, if microsecond is 0 
YYYY-MM-DDTHH:MM:SS+HH:MM

     The optional argument sep (default 'T') is a one-character 
separator, placed between the date and time portions of the result.

> %Y-%m-%d %H:%M:%S
> %Y-%m-%d %H:%M:%S.%f
>

=> plus timezone versions of the above.

Wolfgang


From steve at pearwood.info  Tue Aug  5 03:39:44 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 5 Aug 2014 11:39:44 +1000
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <lros2p$kdd$1@ger.gmane.org>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org>
Message-ID: <20140805013944.GQ4525@ando>

On Mon, Aug 04, 2014 at 10:56:56PM +0200, Wolfgang Maier wrote:
[...]
> it does hold true in 3.x, but the documented behavior is slightly more 
> complex (I assume also in 2.x):
> 
> datetime.__str__()
>     For a datetime instance d, str(d) is equivalent to d.isoformat(' ').

Since str(d) is documented to use a well-defined format, then I agree 
that it makes sense to make the second argument to d.strptime optional, 
and default to that same format. The concern I had was the sort of 
scenario Skip suggested: I might write out a datetime object as a string 
on one machine, where the format is X, and read it back elsewhere, where 
the format is Y, leading to at best an exception and at worse incorrect 
data.

+1 on the suggestion.


-- 
Steven

From wolfgang.maier at biologie.uni-freiburg.de  Tue Aug  5 23:22:11 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Tue, 05 Aug 2014 23:22:11 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <20140805013944.GQ4525@ando>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
Message-ID: <lrrhu3$p2l$1@ger.gmane.org>

On 05.08.2014 03:39, Steven D'Aprano wrote:
>
> Since str(d) is documented to use a well-defined format, then I agree
> that it makes sense to make the second argument to d.strptime optional,
> and default to that same format. The concern I had was the sort of
> scenario Skip suggested: I might write out a datetime object as a string
> on one machine, where the format is X, and read it back elsewhere, where
> the format is Y, leading to at best an exception and at worse incorrect
> data.
>
> +1 on the suggestion.
>

After looking a bit into the code of the datetime module, I am not 
convinced anymore that strptime() is the right place for the 
functionality for the following reasons:

1) strptime already has a clear counterpart and that's strftime.

2) strftime/strptime use explicit format strings, not any more 
sophisticated parsing (as would be required to parse the different 
formats that datetime.__str__ can produce) and they try, intentionally, 
to mimick the behavior of their C equivalents.

In other words, strftime/strptime have a very clear underlying concept, 
which IMO should not be given up just because we are trying to stuff 
some extra-functionality into them.

That said, I still think that the basic idea - being able to 
reverse-parse the output of datetime.__str__ - is right.

I would suggest that a better place for this is an additional 
classmethod constructor (the datetime class already has quite a number 
of them). Maybe fromisostring() could be a suitable name ?
With this you could even pass an extra-argument for the date-time 
separator just like with the current isoformat.
This constructor would then be more like a counterpart to 
datetime.isoformat(), but it could simply be documented that calling it 
with fromisostring(datestring, sep=" ") can be used to parse strings 
written with datetime.str().

-1 on the specifics of the proposal,
+1 on the general idea.


From j.wielicki at sotecware.net  Tue Aug  5 23:28:22 2014
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Tue, 05 Aug 2014 23:28:22 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
 to __str__
In-Reply-To: <lrrhu3$p2l$1@ger.gmane.org>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org>
Message-ID: <53E14C76.5010603@sotecware.net>

On 05.08.2014 23:22, Wolfgang Maier wrote:
> On 05.08.2014 03:39, Steven D'Aprano wrote:
>>
>> Since str(d) is documented to use a well-defined format, then I agree
>> that it makes sense to make the second argument to d.strptime optional,
>> and default to that same format. The concern I had was the sort of
>> scenario Skip suggested: I might write out a datetime object as a string
>> on one machine, where the format is X, and read it back elsewhere, where
>> the format is Y, leading to at best an exception and at worse incorrect
>> data.
>>
>> +1 on the suggestion.
>>
> 
> After looking a bit into the code of the datetime module, I am not
> convinced anymore that strptime() is the right place for the
> functionality for the following reasons:
> 
> 1) strptime already has a clear counterpart and that's strftime.
> 
> 2) strftime/strptime use explicit format strings, not any more
> sophisticated parsing (as would be required to parse the different
> formats that datetime.__str__ can produce) and they try, intentionally,
> to mimick the behavior of their C equivalents.
> 
> In other words, strftime/strptime have a very clear underlying concept,
> which IMO should not be given up just because we are trying to stuff
> some extra-functionality into them.
> 
> That said, I still think that the basic idea - being able to
> reverse-parse the output of datetime.__str__ - is right.
> 
> I would suggest that a better place for this is an additional
> classmethod constructor (the datetime class already has quite a number
> of them). Maybe fromisostring() could be a suitable name ?

Maybe rather fromisoformat(), to stay analogous with the formatting method?

> With this you could even pass an extra-argument for the date-time
> separator just like with the current isoformat.
> This constructor would then be more like a counterpart to
> datetime.isoformat(), but it could simply be documented that calling it
> with fromisostring(datestring, sep=" ") can be used to parse strings
> written with datetime.str().
> 
> -1 on the specifics of the proposal,
> +1 on the general idea.

+1 for this rating.

> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


From abarnert at yahoo.com  Tue Aug  5 23:35:06 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 5 Aug 2014 14:35:06 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <lrrhu3$p2l$1@ger.gmane.org>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org>
Message-ID: <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>

On Aug 5, 2014, at 14:22, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote:

> On 05.08.2014 03:39, Steven D'Aprano wrote:
>> 
>> Since str(d) is documented to use a well-defined format, then I agree
>> that it makes sense to make the second argument to d.strptime optional,
>> and default to that same format. The concern I had was the sort of
>> scenario Skip suggested: I might write out a datetime object as a string
>> on one machine, where the format is X, and read it back elsewhere, where
>> the format is Y, leading to at best an exception and at worse incorrect
>> data.
>> 
>> +1 on the suggestion.
> 
> After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons:
> 
> 1) strptime already has a clear counterpart and that's strftime.
> 
> 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents.
> 
> In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them.

What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses.

> That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right.
> 
> I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ?
> With this you could even pass an extra-argument for the date-time separator just like with the current isoformat.
> This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str().

Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months?

What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name.

> 
> -1 on the specifics of the proposal,
> +1 on the general idea.
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From encukou at gmail.com  Tue Aug  5 23:46:10 2014
From: encukou at gmail.com (Petr Viktorin)
Date: Tue, 5 Aug 2014 23:46:10 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org>
 <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
Message-ID: <CA+=+wqDFFiu-1F9Gvd0JRQAFP_y17vBOo20pGXF1DEsnt-GfCQ@mail.gmail.com>

On Tue, Aug 5, 2014 at 11:35 PM, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
> On Aug 5, 2014, at 14:22, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote:
>
>> On 05.08.2014 03:39, Steven D'Aprano wrote:
>>>
>>> Since str(d) is documented to use a well-defined format, then I agree
>>> that it makes sense to make the second argument to d.strptime optional,
>>> and default to that same format. The concern I had was the sort of
>>> scenario Skip suggested: I might write out a datetime object as a string
>>> on one machine, where the format is X, and read it back elsewhere, where
>>> the format is Y, leading to at best an exception and at worse incorrect
>>> data.
>>>
>>> +1 on the suggestion.
>>
>> After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons:
>>
>> 1) strptime already has a clear counterpart and that's strftime.
>>
>> 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents.
>>
>> In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them.
>
> What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses.

+1

>
>> That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right.
>>
>> I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ?
>> With this you could even pass an extra-argument for the date-time separator just like with the current isoformat.
>> This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str().
>
> Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months?
>
> What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name.

Let me just note this, since it hasn't been said here yet:

When people say "iso" in the context of datestimes, they usually mean RFC 3339.

As Wikipedia can tell you, ISO 8601 is a big complicated non-public
specification under which today can be written as:
- 2014-08-05
- 2014-W32-2
- 2014-217
... and by now I can see why there's no ISO 8601 parser in the stdlib.

RFC 3339, on the other hand, specifies one specific variant of ISO
8601: the one we're all used to, and which datetime's isoformat and
__str__ return. (Just about the only exception is that to be
compatible with ISO 8601, it still specifies "T"/"t" for the separator
and graciously lets people agree on space.)

From tjreedy at udel.edu  Wed Aug  6 01:12:47 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 05 Aug 2014 19:12:47 -0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
Message-ID: <lrrodr$5ur$1@ger.gmane.org>

On 8/5/2014 5:35 PM, Andrew Barnert wrote:
> On Aug 5, 2014, at 14:22, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote:
>
>> On 05.08.2014 03:39, Steven D'Aprano wrote:
>>>
>>> Since str(d) is documented to use a well-defined format, then I agree
>>> that it makes sense to make the second argument to d.strptime optional,
>>> and default to that same format. The concern I had was the sort of
>>> scenario Skip suggested: I might write out a datetime object as a string
>>> on one machine, where the format is X, and read it back elsewhere, where
>>> the format is Y, leading to at best an exception and at worse incorrect
>>> data.
>>>
>>> +1 on the suggestion.
>>
>> After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons:
>>
>> 1) strptime already has a clear counterpart and that's strftime.
>>
>> 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents.
>>
>> In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them.
>
> What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses.
>
>> That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right.
>>
>> I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ?
>> With this you could even pass an extra-argument for the date-time separator just like with the current isoformat.
>> This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str().
>
> Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months?

Probably yes

> What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name.

Given that str(dti) (datetime instance) is conceptually dt.tostr(dit), 
name the inverse as dti = dt.fromstr(s).

-- 
Terry Jan Reedy


From wolfgang.maier at biologie.uni-freiburg.de  Wed Aug  6 10:35:57 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Wed, 06 Aug 2014 10:35:57 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
Message-ID: <53E1E8ED.6040403@biologie.uni-freiburg.de>

On 05.08.2014 23:35, Andrew Barnert wrote:
> On Aug 5, 2014, at 14:22, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote:
>
>> On 05.08.2014 03:39, Steven D'Aprano wrote:
>>>
>>> Since str(d) is documented to use a well-defined format, then I agree
>>> that it makes sense to make the second argument to d.strptime optional,
>>> and default to that same format. The concern I had was the sort of
>>> scenario Skip suggested: I might write out a datetime object as a string
>>> on one machine, where the format is X, and read it back elsewhere, where
>>> the format is Y, leading to at best an exception and at worse incorrect
>>> data.
>>>
>>> +1 on the suggestion.
>>
>> After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons:
>>
>> 1) strptime already has a clear counterpart and that's strftime.
>>
>> 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents.
>>
>> In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them.
>
> What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses.
>

Yes, but strftime without format string would then be completely 
redundant with __str__ and isoformat with " " separator, which is really 
quite against the one and only one way of doing things idea.

Plus again, right now strftime takes an explicit format string and then 
generates a datetime string with exactly this and only this format.
In the optional format string scenario, it would have to generate 
slightly differently formatted output depending on whether there is 
microseconds and/or timezone information. So, like for strptime, this 
would change the very clearly defined current behavior into a mix of 
things, unnecessarily.

>> That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right.
>>
>> I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ?
>> With this you could even pass an extra-argument for the date-time separator just like with the current isoformat.
>> This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str().
>
> Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months?
>
> What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name.
>

Fair enough, it was just the first half-reasonable thing that came to my 
mind :)
Being able to parse any valid ISO string would be another nice feature, 
but it's really a different story.

Wolfgang


From ethan at stoneleaf.us  Wed Aug  6 12:34:49 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 06 Aug 2014 03:34:49 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
 to __str__
In-Reply-To: <53E1E8ED.6040403@biologie.uni-freiburg.de>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <53E1E8ED.6040403@biologie.uni-freiburg.de>
Message-ID: <53E204C9.30307@stoneleaf.us>

On 08/06/2014 01:35 AM, Wolfgang Maier wrote:
>
> [...] which is really quite against the one and only one way of doing things idea.

It's "One Obvious Way" not "Only One Way".

--
~Ethan~

From wolfgang.maier at biologie.uni-freiburg.de  Wed Aug  6 12:40:33 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Wed, 06 Aug 2014 12:40:33 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <53E204C9.30307@stoneleaf.us>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <53E1E8ED.6040403@biologie.uni-freiburg.de> <53E204C9.30307@stoneleaf.us>
Message-ID: <lrt0n2$4qr$1@ger.gmane.org>

On 06.08.2014 12:34, Ethan Furman wrote:
> On 08/06/2014 01:35 AM, Wolfgang Maier wrote:
>>
>> [...] which is really quite against the one and only one way of doing
>> things idea.
>
> It's "One Obvious Way" not "Only One Way".
>

I wasn't quoting, just paraphrasing.


From abarnert at yahoo.com  Wed Aug  6 15:48:44 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 6 Aug 2014 06:48:44 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <53E1E8ED.6040403@biologie.uni-freiburg.de>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <53E1E8ED.6040403@biologie.uni-freiburg.de>
Message-ID: <EF46264C-A4DB-4E1C-9C6D-5E4B9B1FDB60@yahoo.com>

On Aug 6, 2014, at 1:35, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote:

> On 05.08.2014 23:35, Andrew Barnert wrote:
>> On Aug 5, 2014, at 14:22, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote:
>> 
>>> On 05.08.2014 03:39, Steven D'Aprano wrote:
>>>> 
>>>> Since str(d) is documented to use a well-defined format, then I agree
>>>> that it makes sense to make the second argument to d.strptime optional,
>>>> and default to that same format. The concern I had was the sort of
>>>> scenario Skip suggested: I might write out a datetime object as a string
>>>> on one machine, where the format is X, and read it back elsewhere, where
>>>> the format is Y, leading to at best an exception and at worse incorrect
>>>> data.
>>>> 
>>>> +1 on the suggestion.
>>> 
>>> After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons:
>>> 
>>> 1) strptime already has a clear counterpart and that's strftime.
>>> 
>>> 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents.
>>> 
>>> In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them.
>> 
>> What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses.
> 
> Yes, but strftime without format string would then be completely redundant with __str__ and isoformat with " " separator, which is really quite against the one and only one way of doing things idea.

They're not redundant. str provides gives you some human-readable, ideally but not necessarily parseable, representation. isoformat gives you a specific format that you know is parseable by many other libraries and languages, and sorts in date order. strftime lets you specify a format to be parsed by specific code or problem-specific human expectations. Is the fact that they happen to overlap (which is already true, since you can always specify the same format explicitly if you want) any worse than the fact that str(3) and format(3, 'd') give you the same result?

> Plus again, right now strftime takes an explicit format string and then generates a datetime string with exactly this and only this format.
> In the optional format string scenario, it would have to generate slightly differently formatted output depending on whether there is microseconds and/or timezone information. So, like for strptime, this would change the very clearly defined current behavior into a mix of things, unnecessarily.

The purpose of strftime and strptime is to be inverses of each other--to generate and parse datetime strings in a specified way. If one of those ways is "the default Python string representation" for one function, it should be true for the other. (Doesn't gnu strf/ptime have an extension that gives you % codes for "default" date and time representations, which don't guarantee anything other than that they be reasonable for the locale and reversible?)

>>> That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right.
>>> 
>>> I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ?
>>> With this you could even pass an extra-argument for the date-time separator just like with the current isoformat.
>>> This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str().
>> 
>> Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months?
>> 
>> What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name.
> 
> Fair enough, it was just the first half-reasonable thing that came to my mind :)
> Being able to parse any valid ISO string would be another nice feature, but it's really a different story.
> 
> Wolfgang
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From abarnert at yahoo.com  Wed Aug  6 16:02:20 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 6 Aug 2014 07:02:20 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CA+=+wqDFFiu-1F9Gvd0JRQAFP_y17vBOo20pGXF1DEsnt-GfCQ@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <CA+=+wqDFFiu-1F9Gvd0JRQAFP_y17vBOo20pGXF1DEsnt-GfCQ@mail.gmail.com>
Message-ID: <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com>

On Aug 5, 2014, at 14:46, Petr Viktorin <encukou at gmail.com> wrote:

> On Tue, Aug 5, 2014 at 11:35 PM, Andrew Barnert
> <abarnert at yahoo.com.dmarc.invalid> wrote:
>> On Aug 5, 2014, at 14:22, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote:
>> 
>>> On 05.08.2014 03:39, Steven D'Aprano wrote:
>>>> 
>>>> Since str(d) is documented to use a well-defined format, then I agree
>>>> that it makes sense to make the second argument to d.strptime optional,
>>>> and default to that same format. The concern I had was the sort of
>>>> scenario Skip suggested: I might write out a datetime object as a string
>>>> on one machine, where the format is X, and read it back elsewhere, where
>>>> the format is Y, leading to at best an exception and at worse incorrect
>>>> data.
>>>> 
>>>> +1 on the suggestion.
>>> 
>>> After looking a bit into the code of the datetime module, I am not convinced anymore that strptime() is the right place for the functionality for the following reasons:
>>> 
>>> 1) strptime already has a clear counterpart and that's strftime.
>>> 
>>> 2) strftime/strptime use explicit format strings, not any more sophisticated parsing (as would be required to parse the different formats that datetime.__str__ can produce) and they try, intentionally, to mimick the behavior of their C equivalents.
>>> 
>>> In other words, strftime/strptime have a very clear underlying concept, which IMO should not be given up just because we are trying to stuff some extra-functionality into them.
>> 
>> What if strftime _also_ allowed the format string to be omitted, in which case it would produce the same format as str? Then they would remain perfect inverses.
> 
> +1
> 
>> 
>>> That said, I still think that the basic idea - being able to reverse-parse the output of datetime.__str__ - is right.
>>> 
>>> I would suggest that a better place for this is an additional classmethod constructor (the datetime class already has quite a number of them). Maybe fromisostring() could be a suitable name ?
>>> With this you could even pass an extra-argument for the date-time separator just like with the current isoformat.
>>> This constructor would then be more like a counterpart to datetime.isoformat(), but it could simply be documented that calling it with fromisostring(datestring, sep=" ") can be used to parse strings written with datetime.str().
>> 
>> Wouldn't you expect a method called fromisostring to be able to parse any valid ISO string, especially given that there are third-party libs with functions named fromisoformat that do exactly that, and people suggest adding one of them to the stdlib every few months?
>> 
>> What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name.
> 
> Let me just note this, since it hasn't been said here yet:
> 
> When people say "iso" in the context of datestimes, they usually mean RFC 3339.

RFC 3339 is still more complicated than just reversing Python's str or isoformat. IIRC (it's hard to check on my phone), it mandates that parsers should accept 2-digit years (including 3-digit or semicolon-and-two-digit years), lowercase T and Z, missing "-", and other things that you shouldn't generate but some code might.

That being said, it's still obviously easier to write an RFC 3339 parser than a full ISO 8601 parser, and as long as someone is willing to write it (with sufficient tests) I don't see any problem with the stdlib having one. But I don't know that it should be called "fromisostring". 

"fromisoformat" isn't quite as bad, since at least it implies that it's the inverse of the same type's "isoformat", but it still seems misleading.


From ethan at stoneleaf.us  Wed Aug  6 16:05:48 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 06 Aug 2014 07:05:48 -0700
Subject: [Python-ideas] oow vs. oow [was: strptime without second argument
 as an inverse to __str__]
In-Reply-To: <lrt0n2$4qr$1@ger.gmane.org>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <53E1E8ED.6040403@biologie.uni-freiburg.de> <53E204C9.30307@stoneleaf.us>
 <lrt0n2$4qr$1@ger.gmane.org>
Message-ID: <53E2363C.50405@stoneleaf.us>

On 08/06/2014 03:40 AM, Wolfgang Maier wrote:
> On 06.08.2014 12:34, Ethan Furman wrote:
>> On 08/06/2014 01:35 AM, Wolfgang Maier wrote:
>>>
>>> [...] which is really quite against the one and only one way of doing
>>> things idea.
>>
>> It's "One Obvious Way" not "Only One Way".
>>
>
> I wasn't quoting, just paraphrasing.

It's a bad paraphrase as the two have nearly completely different meanings.

If you wish to pursue this sub-thread further I'll have to turn my portion over to D'Aprano (assuming he's willing) as 
he is much better at long explanations than I am.

--
~Ethan~

From abarnert at yahoo.com  Wed Aug  6 16:04:52 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 6 Aug 2014 07:04:52 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <lrrodr$5ur$1@ger.gmane.org>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <lrrodr$5ur$1@ger.gmane.org>
Message-ID: <79E4DDD2-7F63-4297-8D7C-566496C1BF05@yahoo.com>

On Aug 5, 2014, at 16:12, Terry Reedy <tjreedy at udel.edu> wrote:

> On 8/5/2014 5:35 PM, Andrew Barnert wrote:
>> What you want to get across is that this function parses the default Python representation of datetimes; the fact that it happens to be a subset of ISO format doesn't seem as relevant here. I like the idea of a new alternate constructor, I'm just not crazy about the name.
> 
> Given that str(dti) (datetime instance) is conceptually dt.tostr(dit), name the inverse as dti = dt.fromstr(s).

Wow, now I feel stupid for not thinking of this one.

+00:00:01


From ethan at stoneleaf.us  Wed Aug  6 16:14:29 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 06 Aug 2014 07:14:29 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
 to __str__
In-Reply-To: <20140805013944.GQ4525@ando>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
Message-ID: <53E23845.3030100@stoneleaf.us>

On 08/04/2014 06:39 PM, Steven D'Aprano wrote:
> On Mon, Aug 04, 2014 at 10:56:56PM +0200, Wolfgang Maier wrote:
> [...]
>> it does hold true in 3.x, but the documented behavior is slightly more
>> complex (I assume also in 2.x):
>>
>> datetime.__str__()
>>      For a datetime instance d, str(d) is equivalent to d.isoformat(' ').
>
> Since str(d) is documented to use a well-defined format, then I agree
> that it makes sense to make the second argument to d.strptime optional,
> and default to that same format. The concern I had was the sort of
> scenario Skip suggested: I might write out a datetime object as a string
> on one machine, where the format is X, and read it back elsewhere, where
> the format is Y, leading to at best an exception and at worse incorrect
> data.

What are the downsides of:

dt = datetime.datetime.now()	# assuming this works ;)
sdt = str(dt)
ndt = datetime.datetime(std)
print(dt == ndt)
#True

--
~Ethan~

From wolfgang.maier at biologie.uni-freiburg.de  Wed Aug  6 17:19:30 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Wed, 06 Aug 2014 17:19:30 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <53E23845.3030100@stoneleaf.us>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us>
Message-ID: <lrth22$sbi$1@ger.gmane.org>

On 06.08.2014 16:14, Ethan Furman wrote:
> On 08/04/2014 06:39 PM, Steven D'Aprano wrote:
>> On Mon, Aug 04, 2014 at 10:56:56PM +0200, Wolfgang Maier wrote:
>> [...]
>>> it does hold true in 3.x, but the documented behavior is slightly more
>>> complex (I assume also in 2.x):
>>>
>>> datetime.__str__()
>>>      For a datetime instance d, str(d) is equivalent to d.isoformat('
>>> ').
>>
>> Since str(d) is documented to use a well-defined format, then I agree
>> that it makes sense to make the second argument to d.strptime optional,
>> and default to that same format. The concern I had was the sort of
>> scenario Skip suggested: I might write out a datetime object as a string
>> on one machine, where the format is X, and read it back elsewhere, where
>> the format is Y, leading to at best an exception and at worse incorrect
>> data.
>
> What are the downsides of:
>
> dt = datetime.datetime.now()    # assuming this works ;)
> sdt = str(dt)
> ndt = datetime.datetime(std)
> print(dt == ndt)
> #True
>

I'll refrain from mentioning "explicit is better than implicit" ;)

It's just that it seems to be a design pattern of the datetime class to 
provide alternative constructors as classmethods instead of doing magic 
things in __new__ . There are fromtimestamp and utcfromtimestamp 
already, and you can think of datetime.now() the same way. After all, 
you could decide to have this called when datetime() is called without 
an argument.
I guess there are just too many different things that *could* make sense 
to pass to __new__ and to be treated implicitly.

Wolfgang


From wolfgang.maier at biologie.uni-freiburg.de  Wed Aug  6 17:22:47 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Wed, 06 Aug 2014 17:22:47 +0200
Subject: [Python-ideas] oow vs. oow [was: strptime without second
 argument as an inverse to __str__]
In-Reply-To: <53E2363C.50405@stoneleaf.us>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <53E1E8ED.6040403@biologie.uni-freiburg.de> <53E204C9.30307@stoneleaf.us>
 <lrt0n2$4qr$1@ger.gmane.org> <53E2363C.50405@stoneleaf.us>
Message-ID: <lrth88$vvg$1@ger.gmane.org>

On 06.08.2014 16:05, Ethan Furman wrote:
> On 08/06/2014 03:40 AM, Wolfgang Maier wrote:
>> On 06.08.2014 12:34, Ethan Furman wrote:
>>> On 08/06/2014 01:35 AM, Wolfgang Maier wrote:
>>>>
>>>> [...] which is really quite against the one and only one way of doing
>>>> things idea.
>>>
>>> It's "One Obvious Way" not "Only One Way".
>>>
>>
>> I wasn't quoting, just paraphrasing.
>
> It's a bad paraphrase as the two have nearly completely different meanings.
>
> If you wish to pursue this sub-thread further I'll have to turn my
> portion over to D'Aprano (assuming he's willing) as he is much better at
> long explanations than I am.
>

I don't think that's necessary :)
I like focused threads !


From alexander.belopolsky at gmail.com  Wed Aug  6 17:42:55 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 6 Aug 2014 11:42:55 -0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <lrth22$sbi$1@ger.gmane.org>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
Message-ID: <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>

On Wed, Aug 6, 2014 at 11:19 AM, Wolfgang Maier <
wolfgang.maier at biologie.uni-freiburg.de> wrote:

> What are the downsides of:
>>
>> dt = datetime.datetime.now()    # assuming this works ;)
>> sdt = str(dt)
>> ndt = datetime.datetime(std)
>> print(dt == ndt)
>> #True
>>
>>
> I'll refrain from mentioning "explicit is better than implicit" ;)
>
> It's just that it seems to be a design pattern of the datetime class to
> provide alternative constructors as classmethods instead of doing magic
> things in __new__ .


I don't think this is a "design pattern".  In Python 2, having date(str)
constructor was blocked by some magic that is there to support unpickling:

>>> from datetime import date
>>> date('\x07\xd0\x01\x01')
datetime.date(2000, 1, 1)

This is no longer an issue in Python 3.

Note that if we allow date('2000-01-01'), this may become a more readable
and efficient alternative to date(2001, 1, 1).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140806/50e3ccf0/attachment-0001.html>

From wolfgang.maier at biologie.uni-freiburg.de  Wed Aug  6 18:20:37 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Wed, 06 Aug 2014 18:20:37 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
Message-ID: <lrtkkm$dap$1@ger.gmane.org>

On 06.08.2014 17:42, Alexander Belopolsky wrote:
>
> On Wed, Aug 6, 2014 at 11:19 AM, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de
> <mailto:wolfgang.maier at biologie.uni-freiburg.de>>
> wrote:
>
>         What are the downsides of:
>
>         dt = datetime.datetime.now()    # assuming this works ;)
>         sdt = str(dt)
>         ndt = datetime.datetime(std)
>         print(dt == ndt)
>         #True
>
>
>     I'll refrain from mentioning "explicit is better than implicit" ;)
>
>     It's just that it seems to be a design pattern of the datetime class
>     to provide alternative constructors as classmethods instead of doing
>     magic things in __new__ .
>
>
> I don't think this is a "design pattern".  In Python 2, having date(str)
> constructor was blocked by some magic that is there to support unpickling:
>
>  >>> from datetime import date
>  >>> date('\x07\xd0\x01\x01')
> datetime.date(2000, 1, 1)
>

I see. None of my examples (fromtimestamp, utcfromtimestamp and now) 
involves string parsing though.

> This is no longer an issue in Python 3.
>
> Note that if we allow date('2000-01-01'), this may become a more
> readable and efficient alternative to date(2001, 1, 1).
>

One problem with this is the very first concern raised by Steven and 
Skip in this thread: choosing a string format that __new__ can deal with 
*would* lock you into this format.
If later, for example, full-blown ISO 8601 or even just RFC 3339 parsing 
makes it into the module, wouldn't you rather want this to be done by 
__new__ when it sees a string ?
Implementing the current proposal as a classmethod with its own name 
(once a good one is accepted) is a much more cautious approach.

BTW, Terry's suggestion of datetime.fromstr(s) sounds very reasonable.



From ethan at stoneleaf.us  Wed Aug  6 18:45:11 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 06 Aug 2014 09:45:11 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
 to __str__
In-Reply-To: <lrtkkm$dap$1@ger.gmane.org>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <lrtkkm$dap$1@ger.gmane.org>
Message-ID: <53E25B97.4010004@stoneleaf.us>

On 08/06/2014 09:20 AM, Wolfgang Maier wrote:
> On 06.08.2014 17:42, Alexander Belopolsky wrote:
>>
>> On Wed, Aug 6, 2014 at 11:19 AM, Wolfgang Maier
>> <wolfgang.maier at biologie.uni-freiburg.de
>> <mailto:wolfgang.maier at biologie.uni-freiburg.de>>
>> wrote:
>>
>>         What are the downsides of:
>>
>>         dt = datetime.datetime.now()    # assuming this works ;)
>>         sdt = str(dt)
>>         ndt = datetime.datetime(std)
>>         print(dt == ndt)
>>         #True
>>
>>
>>     I'll refrain from mentioning "explicit is better than implicit" ;)
>>
>>     It's just that it seems to be a design pattern of the datetime class
>>     to provide alternative constructors as classmethods instead of doing
>>     magic things in __new__ .
>>
>>
>> I don't think this is a "design pattern".  In Python 2, having date(str)
>> constructor was blocked by some magic that is there to support unpickling:
>>
>>  >>> from datetime import date
>>  >>> date('\x07\xd0\x01\x01')
>> datetime.date(2000, 1, 1)
>>
>
> I see. None of my examples (fromtimestamp, utcfromtimestamp and now) involves string parsing though.
>
>> This is no longer an issue in Python 3.
>>
>> Note that if we allow date('2000-01-01'), this may become a more
>> readable and efficient alternative to date(2001, 1, 1).
>>
>
> One problem with this is the very first concern raised by Steven and Skip in this thread: choosing a string format that
> __new__ can deal with *would* lock you into this format.
> If later, for example, full-blown ISO 8601 or even just RFC 3339 parsing makes it into the module, wouldn't you rather
> want this to be done by __new__ when it sees a string ?
> Implementing the current proposal as a classmethod with its own name (once a good one is accepted) is a much more
> cautious approach.
>
> BTW, Terry's suggestion of datetime.fromstr(s) sounds very reasonable.

For my own classes I accept both year, month, day, ..., or a single string in __new__.

But for the stdlib I agree that .fromstr() is the better approach.

+1 for .fromstr()

--
~Ethan~

From alexander.belopolsky at gmail.com  Wed Aug  6 18:47:04 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 6 Aug 2014 12:47:04 -0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <lrtkkm$dap$1@ger.gmane.org>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <lrtkkm$dap$1@ger.gmane.org>
Message-ID: <CAP7h-xa0pV-JRCL9+yGXo=syRisPYx-bhRWzOMM4XUpPGZm9Lg@mail.gmail.com>

On Wed, Aug 6, 2014 at 12:20 PM, Wolfgang Maier <
wolfgang.maier at biologie.uni-freiburg.de> wrote:

> BTW, Terry's suggestion of datetime.fromstr(s) sounds very reasonable.


We don't have int.fromstr, float.fromstr, or in fact a .fromstr constructor
for any other type.  IMO, date(str) is "the obvious way to do it."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140806/94b965c5/attachment.html>

From alexander.belopolsky at gmail.com  Wed Aug  6 18:53:22 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 6 Aug 2014 12:53:22 -0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <53E25B97.4010004@stoneleaf.us>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <lrtkkm$dap$1@ger.gmane.org> <53E25B97.4010004@stoneleaf.us>
Message-ID: <CAP7h-xZfH+JK1FUcDcW_UmzrRcwrwrCfX_XnZPTOZ6H=_20mPw@mail.gmail.com>

On Wed, Aug 6, 2014 at 12:45 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> For my own classes I accept both year, month, day, ..., or a single string
> in __new__.
>
> But for the stdlib I agree that .fromstr() is the better approach.
>

Can you explain why what is good for your own classes is not good for
stdlib?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140806/ce7137e6/attachment.html>

From ethan at stoneleaf.us  Wed Aug  6 19:01:45 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 06 Aug 2014 10:01:45 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
 to __str__
In-Reply-To: <CAP7h-xa0pV-JRCL9+yGXo=syRisPYx-bhRWzOMM4XUpPGZm9Lg@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <lrtkkm$dap$1@ger.gmane.org>
 <CAP7h-xa0pV-JRCL9+yGXo=syRisPYx-bhRWzOMM4XUpPGZm9Lg@mail.gmail.com>
Message-ID: <53E25F79.1070507@stoneleaf.us>

On 08/06/2014 09:47 AM, Alexander Belopolsky wrote:
> On Wed, Aug 6, 2014 at 12:20 PM, Wolfgang Maier wrote:
>>
>> BTW, Terry's suggestion of datetime.fromstr(s) sounds very reasonable.
>
> We don't have int.fromstr, float.fromstr, or in fact a .fromstr
> constructor for any other type.  IMO, date(str) is "the obvious
>  way to do it."

int, float, and, I suspect, all the core data types, have the same __str__ as __repr__, so there's really no difference. 
  datetime objects, on the other hand, definitely have different __str__ and __repr__, and the desire is to be able no 
eval(obj.__repr__()), not of __str__.

--
~Ethan~

From ethan at stoneleaf.us  Wed Aug  6 19:02:19 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 06 Aug 2014 10:02:19 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
 to __str__
In-Reply-To: <CAP7h-xZfH+JK1FUcDcW_UmzrRcwrwrCfX_XnZPTOZ6H=_20mPw@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <lrtkkm$dap$1@ger.gmane.org> <53E25B97.4010004@stoneleaf.us>
 <CAP7h-xZfH+JK1FUcDcW_UmzrRcwrwrCfX_XnZPTOZ6H=_20mPw@mail.gmail.com>
Message-ID: <53E25F9B.4080801@stoneleaf.us>

On 08/06/2014 09:53 AM, Alexander Belopolsky wrote:
> On Wed, Aug 6, 2014 at 12:45 PM, Ethan Furman wrote:
>>
>> For my own classes I accept both year, month, day, ..., or a single string in __new__.
>>
>> But for the stdlib I agree that .fromstr() is the better approach.
>
> Can you explain why what is good for your own classes is not good for stdlib?

I tolerate a higher level of risk in my own work, but the stdlib should be more stable.

--
~Ethan~

From tjreedy at udel.edu  Wed Aug  6 21:26:16 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 06 Aug 2014 15:26:16 -0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CAP7h-xa0pV-JRCL9+yGXo=syRisPYx-bhRWzOMM4XUpPGZm9Lg@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <lrtkkm$dap$1@ger.gmane.org>
 <CAP7h-xa0pV-JRCL9+yGXo=syRisPYx-bhRWzOMM4XUpPGZm9Lg@mail.gmail.com>
Message-ID: <lrtvh4$v51$1@ger.gmane.org>

On 8/6/2014 12:47 PM, Alexander Belopolsky wrote:
>
> On Wed, Aug 6, 2014 at 12:20 PM, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de
> <mailto:wolfgang.maier at biologie.uni-freiburg.de>>
> wrote:
>
>     BTW, Terry's suggestion of datetime.fromstr(s) sounds very reasonable.
>
>
> We don't have int.fromstr, float.fromstr, or in fact a .fromstr
> constructor for any other type.  IMO, date(str) is "the obvious way to
> do it."

There are two parts to my suggestion. The first is to focus on the 
original goal of the thread, 'an inverse to __str__', and only that, not 
on automatically parsing, without providing a format, larger classes of 
possible strings.

The second is to suggest a better spelling for that original goal than 
the original proposal (strptime without second argument) or alternate 
proposals like .fromisostring based on an expanded goal. I think 
'.fromstr' is the best possible spelling among '.from...' choices.

If instead using the standard constructor works, fine with me.  I am not 
sure of the criterion for adding alternative to .__init__ versus an 
alterntive method.

-- 
Terry Jan Reedy


From alexander.belopolsky at gmail.com  Wed Aug  6 22:11:24 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 6 Aug 2014 16:11:24 -0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <lrtvh4$v51$1@ger.gmane.org>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <lrtkkm$dap$1@ger.gmane.org>
 <CAP7h-xa0pV-JRCL9+yGXo=syRisPYx-bhRWzOMM4XUpPGZm9Lg@mail.gmail.com>
 <lrtvh4$v51$1@ger.gmane.org>
Message-ID: <CAP7h-xbEPuBuPe5q-5QzEJdLy_DmvnE1JKxNxVgF21GxhJK1JA@mail.gmail.com>

On Wed, Aug 6, 2014 at 3:26 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> I think '.fromstr' is the best possible spelling among '.from...' choices.


With this I agree.  Since the path is already paved with float.fromhex, I
am casting my +0 for date/datetime.fromstr.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140806/49f3d428/attachment.html>

From abarnert at yahoo.com  Thu Aug  7 01:55:25 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 6 Aug 2014 16:55:25 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
Message-ID: <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>

On Aug 6, 2014, at 8:42, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote:

> On Wed, Aug 6, 2014 at 11:19 AM, Wolfgang Maier <wolfgang.maier at biologie.uni-freiburg.de> wrote:
>>> What are the downsides of:
>>> 
>>> dt = datetime.datetime.now()    # assuming this works ;)
>>> sdt = str(dt)
>>> ndt = datetime.datetime(std)
>>> print(dt == ndt)
>>> #True
>> 
>> I'll refrain from mentioning "explicit is better than implicit" ;)
>> 
>> It's just that it seems to be a design pattern of the datetime class to provide alternative constructors as classmethods instead of doing magic things in __new__ .
> 
> I don't think this is a "design pattern".  In Python 2, having date(str) constructor was blocked by some magic that is there to support unpickling:
> 
> >>> from datetime import date
> >>> date('\x07\xd0\x01\x01')
> datetime.date(2000, 1, 1)
> 
> This is no longer an issue in Python 3.
> 
> Note that if we allow date('2000-01-01'), this may become a more readable and efficient alternative to date(2001, 1, 1).

More readable maybe, but more efficient? You're doing the same work, plus string parsing; you're eliminating two parameters (but only if you use *args) at the cost of three locals; by any measure it's less efficient.

But more readable is the important part. In isolation it's readable, the question is whether the added complexity (in the docs and in people's heads) of an effectively-overloaded constructor is worth the cost.

Given that, unlike all the obvious parallel cases (int, float, etc.) this constructor will not accept the repr, I'm not sure the answer comes out the same. But maybe it does. 

I'm -0 on this, +1 on Terry's fromstr, +0 on strptime and strftime both accepting no arguments, -1 on only strptime, -0.5 on fromisostring/fromisoformat, and -1 on remembering any of the other ideas in this thread well enough to comment.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140806/64b980cc/attachment.html>

From steve at pearwood.info  Thu Aug  7 02:10:36 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 7 Aug 2014 10:10:36 +1000
Subject: [Python-ideas] oow vs. oow [was: strptime without second
	argument as an inverse to __str__]
In-Reply-To: <53E2363C.50405@stoneleaf.us>
References: <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <53E1E8ED.6040403@biologie.uni-freiburg.de> <53E204C9.30307@stoneleaf.us>
 <lrt0n2$4qr$1@ger.gmane.org> <53E2363C.50405@stoneleaf.us>
Message-ID: <20140807001036.GU4525@ando>

On Wed, Aug 06, 2014 at 07:05:48AM -0700, Ethan Furman wrote:
> On 08/06/2014 03:40 AM, Wolfgang Maier wrote:
> >On 06.08.2014 12:34, Ethan Furman wrote:
> >>On 08/06/2014 01:35 AM, Wolfgang Maier wrote:
> >>>
> >>>[...] which is really quite against the one and only one way of doing
> >>>things idea.
> >>
> >>It's "One Obvious Way" not "Only One Way".
> >>
> >
> >I wasn't quoting, just paraphrasing.
> 
> It's a bad paraphrase as the two have nearly completely different meanings.
> 
> If you wish to pursue this sub-thread further I'll have to turn my portion 
> over to D'Aprano (assuming he's willing) as he is much better at long 
> explanations than I am.

I can make it a short explanation :-)

"Only One Way" implies:

    assert len(collection_of_ways) == 1


"One Obvious Way" implies:

    assert any(way.is_obvious() for way in collection_of_ways)


Now I suppose I'll have to go back and read the rest of the thread to 
understand what this is about *wink*

-- 
Steven

From alexander.belopolsky at gmail.com  Thu Aug  7 02:19:40 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 6 Aug 2014 20:19:40 -0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
Message-ID: <CAP7h-xaJjb89ay7ecF6ap465MFseBj-W4nQ7_t9T48T4d+mwpw@mail.gmail.com>

On Wed, Aug 6, 2014 at 7:55 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> Given that, unlike all the obvious parallel cases (int, float, etc.) this
> constructor will not accept the repr, I'm not sure the answer comes out the
> same.


The parallel is in accepting str, not repr.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140806/6c13e015/attachment-0001.html>

From ryan at ryanhiebert.com  Thu Aug  7 02:27:35 2014
From: ryan at ryanhiebert.com (Ryan Hiebert)
Date: Wed, 6 Aug 2014 19:27:35 -0500
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CAP7h-xaJjb89ay7ecF6ap465MFseBj-W4nQ7_t9T48T4d+mwpw@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
 <CAP7h-xaJjb89ay7ecF6ap465MFseBj-W4nQ7_t9T48T4d+mwpw@mail.gmail.com>
Message-ID: <29D6F3B3-0297-4FB3-9315-B9C18F78A82F@ryanhiebert.com>


> On Aug 6, 2014, at 7:19 PM, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote:
> 
> 
> On Wed, Aug 6, 2014 at 7:55 PM, Andrew Barnert <abarnert at yahoo.com <mailto:abarnert at yahoo.com>> wrote:
> Given that, unlike all the obvious parallel cases (int, float, etc.) this constructor will not accept the repr, I'm not sure the answer comes out the same.
> 
> The parallel is in accepting str, not repr.   

Indeed. The precedent for repr is that may be eval-able, not that the repr string can be passed into the constructor.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140806/627b4b23/attachment.html>

From alexander.belopolsky at gmail.com  Thu Aug  7 02:29:12 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 6 Aug 2014 20:29:12 -0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
Message-ID: <CAP7h-xbQ=aUx8F8_EiiZi5hdoNN1W7pXA1rqQa5_U+nJXLi=TA@mail.gmail.com>

On Wed, Aug 6, 2014 at 7:55 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> More readable maybe, but more efficient? You're doing the same work, plus
> string parsing; you're eliminating two parameters (but only if you use
> *args) at the cost of three locals; by any measure it's less efficient.


>>> dis(lambda: date(2001, 1, 1))
  1           0 LOAD_GLOBAL              0 (date)
              3 LOAD_CONST               1 (2001)
              6 LOAD_CONST               2 (1)
              9 LOAD_CONST               2 (1)
             12 CALL_FUNCTION            3
             15 RETURN_VALUE
>>> dis(lambda: date('2001-01-01'))
  1           0 LOAD_GLOBAL              0 (date)
              3 LOAD_CONST               1 ('2001-01-01')
              6 CALL_FUNCTION            1
              9 RETURN_VALUE

Since parsing will be done in C, it's cost can be made negligible.  In
implementations other than CPython, YMMV.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140806/9717dc0a/attachment.html>

From steve at pearwood.info  Thu Aug  7 02:32:28 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 7 Aug 2014 10:32:28 +1000
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <EF46264C-A4DB-4E1C-9C6D-5E4B9B1FDB60@yahoo.com>
References: <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <53E1E8ED.6040403@biologie.uni-freiburg.de>
 <EF46264C-A4DB-4E1C-9C6D-5E4B9B1FDB60@yahoo.com>
Message-ID: <20140807003228.GV4525@ando>

On Wed, Aug 06, 2014 at 06:48:44AM -0700, Andrew Barnert wrote:

> The purpose of strftime and strptime is to be inverses of each 
> other--to generate and parse datetime strings in a specified way. If 
> one of those ways is "the default Python string representation" for 
> one function, it should be true for the other. 

As of Python 3.3, neither strftime nor strptime take a default format. 
It's only __str__ which has an implicit default format.


> (Doesn't gnu strf/ptime 
> have an extension that gives you % codes for "default" date and time 
> representations, which don't guarantee anything other than that they 
> be reasonable for the locale and reversible?)

I worry about something like that. Unless the default is guaranteed to 
be a particular format, what counts as "reasonable" when the string is 
written out and when read back in may not be the same.


-- 
Steven

From skip at pobox.com  Thu Aug  7 03:06:25 2014
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 6 Aug 2014 20:06:25 -0500
Subject: [Python-ideas] oow vs. oow [was: strptime without second
 argument as an inverse to __str__]
In-Reply-To: <20140807001036.GU4525@ando>
References: <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org>
 <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <53E1E8ED.6040403@biologie.uni-freiburg.de>
 <53E204C9.30307@stoneleaf.us> <lrt0n2$4qr$1@ger.gmane.org>
 <53E2363C.50405@stoneleaf.us> <20140807001036.GU4525@ando>
Message-ID: <CANc-5Uyq9j9BY0igDpL0ir2FvUCPLVJnhv+2vtVBD+RzFPDY7Q@mail.gmail.com>

On Wed, Aug 6, 2014 at 7:10 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Now I suppose I'll have to go back and read the rest of the thread to
> understand what this is about *wink*

Might be more fun to recast the Zen of Python into Python itself. :-)

Skip

From abarnert at yahoo.com  Thu Aug  7 04:10:49 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 6 Aug 2014 19:10:49 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <20140807003228.GV4525@ando>
References: <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <53E1E8ED.6040403@biologie.uni-freiburg.de>
 <EF46264C-A4DB-4E1C-9C6D-5E4B9B1FDB60@yahoo.com> <20140807003228.GV4525@ando>
Message-ID: <595C7047-90AF-4538-9756-8679E84C9AC2@yahoo.com>

On Aug 6, 2014, at 17:32, Steven D'Aprano <steve at pearwood.info> wrote:

> On Wed, Aug 06, 2014 at 06:48:44AM -0700, Andrew Barnert wrote:
> 
>> The purpose of strftime and strptime is to be inverses of each 
>> other--to generate and parse datetime strings in a specified way. If 
>> one of those ways is "the default Python string representation" for 
>> one function, it should be true for the other.
> 
> As of Python 3.3, neither strftime nor strptime take a default format. 
> It's only __str__ which has an implicit default format.

Exactly my point. They're currently balanced. Adding a default format to strptime only would mean they can no longer be used as inverses of each other.

One solution is to also add a default format to strftime.

The other solution is to recognize that if the desire is an inverse for str, strptime is not the best way to write that. (I didn't have a good alternative suggestion, but Terry's later fromstr seems perfect to me.)

>> (Doesn't gnu strf/ptime 
>> have an extension that gives you % codes for "default" date and time 
>> representations, which don't guarantee anything other than that they 
>> be reasonable for the locale and reversible?)
> 
> I worry about something like that. Unless the default is guaranteed to 
> be a particular format, what counts as "reasonable" when the string is 
> written out and when read back in may not be the same.

I share that worry. Is it guaranteed that str on any Python anywhere can be parsed back to the same value on a different Python somewhere else?

The docs for str (and the presumed docs got the new function) can make that guarantee, but most of the people in this thread didn't know that, or weren't confident in it. That's why I don't really like the idea of the inverse functions being __str__ and the constructor--it isn't obvious or explicit to the reader, and it's not quite as easy to look up. (That wasn't the initial proposal, but it came up later in the thread, so it was worth responding to.)


From abarnert at yahoo.com  Thu Aug  7 04:24:33 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 6 Aug 2014 19:24:33 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CAP7h-xaJjb89ay7ecF6ap465MFseBj-W4nQ7_t9T48T4d+mwpw@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
 <CAP7h-xaJjb89ay7ecF6ap465MFseBj-W4nQ7_t9T48T4d+mwpw@mail.gmail.com>
Message-ID: <F691AE0D-5DB3-4A54-B422-427868F40216@yahoo.com>

On Aug 6, 2014, at 17:19, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote:

> 
> On Wed, Aug 6, 2014 at 7:55 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
>> Given that, unlike all the obvious parallel cases (int, float, etc.) this constructor will not accept the repr, I'm not sure the answer comes out the same.
> 
> The parallel is in accepting str, not repr.   


Is it? Sure, I'll accept that's the parallel you have in mind, but is it a good one?

The only way I can think to distinguish is this: For bytes, str, tuple, etc., there is no constructor from either string representation. For int, the two representations are identical. For float, they're different--and it's float(repr(f)) that gives you back the same value you started with. (Of course in the other direction, neither one is guaranteed to do so.) Also note that int can accept hex, etc. Python literal values as strings, the same ones eval can.

Is there some documentation that implies that the int, float, etc. constructors are meant to take "human-readable" strings rather than "Python-evaluable" strings? Or some other case I'm missing?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140806/54cb523c/attachment.html>

From wolfgang.maier at biologie.uni-freiburg.de  Thu Aug  7 10:17:13 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Thu, 07 Aug 2014 10:17:13 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CAP7h-xbQ=aUx8F8_EiiZi5hdoNN1W7pXA1rqQa5_U+nJXLi=TA@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
 <CAP7h-xbQ=aUx8F8_EiiZi5hdoNN1W7pXA1rqQa5_U+nJXLi=TA@mail.gmail.com>
Message-ID: <lrvcm9$m5f$1@ger.gmane.org>

On 07.08.2014 02:29, Alexander Belopolsky wrote:
>
> On Wed, Aug 6, 2014 at 7:55 PM, Andrew Barnert
> <abarnert at yahoo.com
> <mailto:abarnert at yahoo.com>> wrote:
>
>     More readable maybe, but more efficient? You're doing the same work,
>     plus string parsing; you're eliminating two parameters (but only if
>     you use *args) at the cost of three locals; by any measure it's less
>     efficient.
>
> Since parsing will be done in C, it's cost can be made negligible.  In
> implementations other than CPython, YMMV.
>

Why would parsing occur in C ? The datetime module is implemented in 
pure Python.


From wolfgang.maier at biologie.uni-freiburg.de  Thu Aug  7 11:57:47 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Thu, 07 Aug 2014 11:57:47 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
Message-ID: <53E34D9B.5040004@biologie.uni-freiburg.de>

On 07.08.2014 01:55, Andrew Barnert wrote:
> On Aug 6, 2014, at 8:42, Alexander Belopolsky
> <alexander.belopolsky at gmail.com
> <mailto:alexander.belopolsky at gmail.com>>
> wrote:
>
>> Note that if we allow date('2000-01-01'), this may become a more
>> readable and efficient alternative to date(2001, 1, 1).
>
> I'm -0 on this, +1 on Terry's fromstr, +0 on strptime and strftime both
> accept ing no arguments, -1 on only strptime, -0.5 on
> fromisostring/fromisoformat, and -1 on remembering any of the other
> ideas in this thread well enough to comment.
>

So to summarize, the three currently discussed options (in order of 
current votes for) are:

- datetime.fromstr(string)
     classmethod to be used as an alternative constructor

- datetime (string)
     add limited string parsing to the default constructor

- datetime.strptime (string)
     parse datetime.__str__ format when format argument is not specified
     (the OP's original proposal)

A point that has not been discussed yet is that the first two options 
could easily be implemented for datetime.date and datetime.time objects 
as well providing counterparts for date.__str_ and time.__str__ .
With strptime this is not possible since datetime.date and datetime.time 
don't have such a method.


In addition, the first two options, in particular, raise a scope 
question. I can see the following options:

- string has to be of exactly the format generated by datetime.__str__, 
i.e. YYYY-MM-DD HH:MM:SS with optional microseconds and timezone 
information (the original proposal)

- string has to be of a format that can be generated by 
datetime.isoformat, of which the datetime.__str__ format is a subset.
The difference is that with datetime.isoformat the separator between the 
date and time portions of the string can be specified, while with 
datetime.__str__ this is fixed to " ".
Accordingly with this option, you would have either:

     datetime.fromstr(string, sep = " ") or
     datetime(string, sep = " ")

to be able to pass an optional separator. When absent the 
datetime.__str__ format is expected.

- (potentially, but I don't think anyone opted for it:
more powerful parsing of a wider range of formats)


Personally, I'm in favor of the second option here for the following 
reason: the string format returned by datetime.__str__ is not fully ISO 
8601 compatible because it uses " " as the separator instead of "T", 
i.e. if an application has to produce ISO 8601 compliant output it has 
to use datetime.isoformat not __str__, even though the two formats 
differ by just a single character. Hence, allowing the separator to be 
specified makes the new functionality a lot more useful at the expense 
of only a moderate increase in complexity.
Note that this is still fundamentally different from asking for any full 
parsing for ISO 8601 and that it would have no impact on potential 
date.fromstr and time.fromstr methods or their default constructor 
versions since they do not have to deal with a separator.

So my preferred complete version would be something like:

datetime.fromstr (string, sep = ' ')
     Return a datetime object from a string as generated by
     datetime.isoformat(sep).
     With the default value of sep this is also the format generated by
     datetime.__str__().

date.fromstr (string)
     Return a date object from a string as generated by
     date.__str__() and date.isoformat().

time.fromstr (string)
     Return a time object from a string as generated by
     time.__str__() and time.isoformat().

Wolfgang


From encukou at gmail.com  Thu Aug  7 12:30:37 2014
From: encukou at gmail.com (Petr Viktorin)
Date: Thu, 7 Aug 2014 12:30:37 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <53E34D9B.5040004@biologie.uni-freiburg.de>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
 <53E34D9B.5040004@biologie.uni-freiburg.de>
Message-ID: <CA+=+wqDEpX4OJB11U-Gk8oLBm18BTiRXqWrZmsT-wUwWYm9oAQ@mail.gmail.com>

On Thu, Aug 7, 2014 at 11:57 AM, Wolfgang Maier
<wolfgang.maier at biologie.uni-freiburg.de> wrote:
> On 07.08.2014 01:55, Andrew Barnert wrote:
>>
>> On Aug 6, 2014, at 8:42, Alexander Belopolsky
>> <alexander.belopolsky at gmail.com
>> <mailto:alexander.belopolsky at gmail.com>>
>>
>> wrote:
>>
>>> Note that if we allow date('2000-01-01'), this may become a more
>>> readable and efficient alternative to date(2001, 1, 1).
>>
>>
>> I'm -0 on this, +1 on Terry's fromstr, +0 on strptime and strftime both
>> accept ing no arguments, -1 on only strptime, -0.5 on
>>
>> fromisostring/fromisoformat, and -1 on remembering any of the other
>> ideas in this thread well enough to comment.
>>
>
> So to summarize, the three currently discussed options (in order of current
> votes for) are:
>
> - datetime.fromstr(string)
>     classmethod to be used as an alternative constructor
>
> - datetime (string)
>     add limited string parsing to the default constructor
>
> - datetime.strptime (string)
>     parse datetime.__str__ format when format argument is not specified
>     (the OP's original proposal)
>
> A point that has not been discussed yet is that the first two options could
> easily be implemented for datetime.date and datetime.time objects as well
> providing counterparts for date.__str_ and time.__str__ .
> With strptime this is not possible since datetime.date and datetime.time
> don't have such a method.
>
>
> In addition, the first two options, in particular, raise a scope question. I
> can see the following options:
>
> - string has to be of exactly the format generated by datetime.__str__, i.e.
> YYYY-MM-DD HH:MM:SS with optional microseconds and timezone information (the
> original proposal)
>
> - string has to be of a format that can be generated by datetime.isoformat,
> of which the datetime.__str__ format is a subset.
> The difference is that with datetime.isoformat the separator between the
> date and time portions of the string can be specified, while with
> datetime.__str__ this is fixed to " ".
> Accordingly with this option, you would have either:
>
>     datetime.fromstr(string, sep = " ") or
>     datetime(string, sep = " ")
>
> to be able to pass an optional separator. When absent the datetime.__str__
> format is expected.
>
> - (potentially, but I don't think anyone opted for it:
> more powerful parsing of a wider range of formats)
>
>
> Personally, I'm in favor of the second option here for the following reason:
> the string format returned by datetime.__str__ is not fully ISO 8601
> compatible because it uses " " as the separator instead of "T", i.e. if an
> application has to produce ISO 8601 compliant output it has to use
> datetime.isoformat not __str__, even though the two formats differ by just a
> single character. Hence, allowing the separator to be specified makes the
> new functionality a lot more useful at the expense of only a moderate
> increase in complexity.
> Note that this is still fundamentally different from asking for any full
> parsing for ISO 8601 and that it would have no impact on potential
> date.fromstr and time.fromstr methods or their default constructor versions
> since they do not have to deal with a separator.

According to Wikipedia's quote of the standard: "the character [T] may
be omitted in applications where there is no risk of confusing a date
and time of day representation with others defined in this
International Standard."

If you give the string to a deatetime constructor, there is obviously
no such confusion.
I'm not sure if replacing by space counts as omitting, but it's the
de-facto standard :)

Anyway, i'd prefer not taking sep as an argument, but accepting both '
' and 'T', since other separators aren't widely used.

Also, note that we while a full ISO parser may not be feasible, we can
(and IMO should) implement a complete RFC 3339 parser (with optional
space separator), and document that both __str__ and fromstr (or
whatever it ends up to be) is indeed is compatible with RFC 3339.

From 4kir4.1i at gmail.com  Thu Aug  7 14:35:09 2014
From: 4kir4.1i at gmail.com (Akira Li)
Date: Thu, 07 Aug 2014 16:35:09 +0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org>
 <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <CA+=+wqDFFiu-1F9Gvd0JRQAFP_y17vBOo20pGXF1DEsnt-GfCQ@mail.gmail.com>
 <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com>
Message-ID: <87vbq4a35u.fsf@gmail.com>

Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> writes:

> On Aug 5, 2014, at 14:46, Petr Viktorin
> <encukou at gmail.com> wrote:
>> When people say "iso" in the context of datestimes, they usually mean RFC 3339.
>
> RFC 3339 is still more complicated than just reversing Python's str or
> isoformat. IIRC (it's hard to check on my phone), it mandates that
> parsers should accept 2-digit years (including 3-digit or
> semicolon-and-two-digit years), lowercase T and Z, missing "-", and
> other things that you shouldn't generate but some code might.

Please, don't spread misinformation.

Among the explicit rfc 3339 design goals are simplicity and human
readability. 

Just read http://tools.ietf.org/html/rfc3339 (for an rfc it is
relatively short and readable). Here's full ABNF:

  date-fullyear   = 4DIGIT
  date-month      = 2DIGIT  ; 01-12
  date-mday       = 2DIGIT  ; 01-28, 01-29, 01-30, 01-31 based on
                            ; month/year
  time-hour       = 2DIGIT  ; 00-23
  time-minute     = 2DIGIT  ; 00-59
  time-second     = 2DIGIT  ; 00-58, 00-59, 00-60 based on leap second
                            ; rules
  time-secfrac    = "." 1*DIGIT
  time-numoffset  = ("+" / "-") time-hour ":" time-minute
  time-offset     = "Z" / time-numoffset

  partial-time    = time-hour ":" time-minute ":" time-second
                    [time-secfrac]
  full-date       = date-fullyear "-" date-month "-" date-mday
  full-time       = partial-time time-offset

  date-time       = full-date "T" full-time

Example:

  1937-01-01T12:00:27.87+00:20


The format is so simple that people just write adhoc parsers using
strptime() without installing any formal rfc3339 module (if it even
exists). 

> That being said, it's still obviously easier to write an RFC 3339
> parser than a full ISO 8601 parser, and as long as someone is willing
> to write it (with sufficient tests) I don't see any problem with the
> stdlib having one. But I don't know that it should be called
> "fromisostring".
>
> "fromisoformat" isn't quite as bad, since at least it implies that
> it's the inverse of the same type's "isoformat", but it still seems
> misleading.


--
Akira


From abarnert at yahoo.com  Thu Aug  7 16:52:03 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 7 Aug 2014 07:52:03 -0700
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <87vbq4a35u.fsf@gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <CA+=+wqDFFiu-1F9Gvd0JRQAFP_y17vBOo20pGXF1DEsnt-GfCQ@mail.gmail.com>
 <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com> <87vbq4a35u.fsf@gmail.com>
Message-ID: <F4725A9B-1329-42B7-B1BC-66BB2488145B@yahoo.com>

On Aug 7, 2014, at 5:35, Akira Li <4kir4.1i at gmail.com> wrote:

> Andrew Barnert
> <abarnert at yahoo.com.dmarc.invalid> writes:
> 
>> On Aug 5, 2014, at 14:46, Petr Viktorin
>> <encukou at gmail.com> wrote:
>>> When people say "iso" in the context of datestimes, they usually mean RFC 3339.
>> 
>> RFC 3339 is still more complicated than just reversing Python's str or
>> isoformat. IIRC (it's hard to check on my phone), it mandates that
>> parsers should accept 2-digit years (including 3-digit or
>> semicolon-and-two-digit years), lowercase T and Z, missing "-", and
>> other things that you shouldn't generate but some code might.
> 
> Please, don't spread misinformation.
> 
> Among the explicit rfc 3339 design goals are simplicity and human
> readability. 
> 
> Just read http://tools.ietf.org/html/rfc3339 (for an rfc it is
> relatively short and readable).

OK, I just read it. Among other things:

> NOTE: Per [ABNF] and ISO8601, the "T" and "Z" characters in this syntax may alternatively be lower case "t" or "z" respectively. This date/time format may be used in some environments or contexts that distinguish between the upper- and lower-case letters 'A'-'Z' and 'a'-'z' (e.g. XML). Specifications that use this format in such environments MAY further limit the date/time syntax so that the letters 'T' and 'Z' used in the date/time syntax must always be upper case. Applications that generate this format SHOULD use upper case letters. NOTE: ISO 8601 defines date and time separated by "T". Applications using this syntax may choose, for the sake of readability, to specify a full-date and full-time separated by (say) a space character. Klyne, et. al. Standards Track [Page 8]

(And there's also a whole section of interpreting "legacy"/"deprecated" 2-digit years and how you should handle them.)

So, is the RFC "spreading misinformation" about itself?

> The format is so simple that people just write adhoc parsers using
strptime() without installing any formal rfc3339 module (if it even
exists). 

Sure, and people also do that and call it an ISO parser. 

If it can't interoperable with everything compliant applications may generate (much less deprecated formats the standard doesn't allow you to generate but mandates how you parse and interpret), it's not accurate to call it an RFC 3339 parser (at least not in a general-purpose library).

As I said before, it's still certainly much easier to write an RFC 3339 parser than an ISO 8601 parser, but it's not as trivial as you're implying.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140807/bb1ddf82/attachment-0001.html>

From alexander.belopolsky at gmail.com  Thu Aug  7 19:27:06 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Thu, 7 Aug 2014 13:27:06 -0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <lrvcm9$m5f$1@ger.gmane.org>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
 <CAP7h-xbQ=aUx8F8_EiiZi5hdoNN1W7pXA1rqQa5_U+nJXLi=TA@mail.gmail.com>
 <lrvcm9$m5f$1@ger.gmane.org>
Message-ID: <CAP7h-xasW83gbMExnbLgtFnB-3U5NxL6cQ3Prqwbqe0LMr8Xpw@mail.gmail.com>

On Thu, Aug 7, 2014 at 4:17 AM, Wolfgang Maier <
wolfgang.maier at biologie.uni-freiburg.de> wrote:

>
>> Since parsing will be done in C, it's cost can be made negligible.  In
>> implementations other than CPython, YMMV.
>>
>>
> Why would parsing occur in C ? The datetime module is implemented in pure
> Python.


No.  In CPython, datetime module is implemented in C.

http://hg.python.org/cpython/file/default/Modules/_datetimemodule.c
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140807/26495fb9/attachment.html>

From ncoghlan at gmail.com  Thu Aug  7 19:48:58 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 8 Aug 2014 03:48:58 +1000
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CAP7h-xasW83gbMExnbLgtFnB-3U5NxL6cQ3Prqwbqe0LMr8Xpw@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
 <CAP7h-xbQ=aUx8F8_EiiZi5hdoNN1W7pXA1rqQa5_U+nJXLi=TA@mail.gmail.com>
 <lrvcm9$m5f$1@ger.gmane.org>
 <CAP7h-xasW83gbMExnbLgtFnB-3U5NxL6cQ3Prqwbqe0LMr8Xpw@mail.gmail.com>
Message-ID: <CADiSq7e_A8yvFJ=xKeUZjGhHrEh6x+XtMDC11e5RNW5H8V5Djg@mail.gmail.com>

On 8 Aug 2014 03:27, "Alexander Belopolsky" <alexander.belopolsky at gmail.com>
wrote:
>
>
> On Thu, Aug 7, 2014 at 4:17 AM, Wolfgang Maier <
wolfgang.maier at biologie.uni-freiburg.de> wrote:
>>>
>>>
>>> Since parsing will be done in C, it's cost can be made negligible.  In
>>> implementations other than CPython, YMMV.
>>>
>>
>> Why would parsing occur in C ? The datetime module is implemented in
pure Python.
>
>
> No.  In CPython, datetime module is implemented in C.
>
> http://hg.python.org/cpython/file/default/Modules/_datetimemodule.c

Don't we have both these days? (C accelerator with pure Python fallback)

Anyway, I'm +1 for Wolfgang's trio of "fromstr" alternative constructors,
but the suggested variation that allows both " " and "T" as the separator,
rather than accepting a parameter. Anyone wanting more flexibility can use
strptime, or else switch to something like dateutil.

Cheers,
Nick.

>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140808/b422679a/attachment.html>

From wolfgang.maier at biologie.uni-freiburg.de  Thu Aug  7 22:40:56 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Thu, 07 Aug 2014 22:40:56 +0200
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
In-Reply-To: <CAP7h-xasW83gbMExnbLgtFnB-3U5NxL6cQ3Prqwbqe0LMr8Xpw@mail.gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <53E23845.3030100@stoneleaf.us> <lrth22$sbi$1@ger.gmane.org>
 <CAP7h-xbqr7GW41Bi0M08i14YQzKaR5Wc1aufH9+Sxzocu-P14Q@mail.gmail.com>
 <235D283E-302F-40B1-AB24-F0CE3BA73486@yahoo.com>
 <CAP7h-xbQ=aUx8F8_EiiZi5hdoNN1W7pXA1rqQa5_U+nJXLi=TA@mail.gmail.com>
 <lrvcm9$m5f$1@ger.gmane.org>
 <CAP7h-xasW83gbMExnbLgtFnB-3U5NxL6cQ3Prqwbqe0LMr8Xpw@mail.gmail.com>
Message-ID: <53E3E458.3060302@biologie.uni-freiburg.de>

On 07.08.2014 19:27, Alexander Belopolsky wrote:
>
> On Thu, Aug 7, 2014 at 4:17 AM, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de
> <mailto:wolfgang.maier at biologie.uni-freiburg.de>>
> wrote:
>
>
>         Since parsing will be done in C, it's cost can be made
>         negligible.  In
>         implementations other than CPython, YMMV.
>
>
>     Why would parsing occur in C ? The datetime module is implemented in
>     pure Python.
>
>
> No.  In CPython, datetime module is implemented in C.
>
> http://hg.python.org/cpython/file/default/Modules/_datetimemodule.c
>

Just to make sure I understood things right: new additions to the 
datetime module would normally be implemented in the Python version.

But then I was forgetting that your suggestion was about changing the 
default constructor, which would have to happen in the C version.

Correct ?


From 4kir4.1i at gmail.com  Fri Aug  8 07:43:32 2014
From: 4kir4.1i at gmail.com (Akira Li)
Date: Fri, 08 Aug 2014 09:43:32 +0400
Subject: [Python-ideas] strptime without second argument as an inverse
	to __str__
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org>
 <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <CA+=+wqDFFiu-1F9Gvd0JRQAFP_y17vBOo20pGXF1DEsnt-GfCQ@mail.gmail.com>
 <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com>
 <87vbq4a35u.fsf@gmail.com>
 <F4725A9B-1329-42B7-B1BC-66BB2488145B@yahoo.com>
Message-ID: <87lhqza64b.fsf@gmail.com>

Andrew Barnert <abarnert at yahoo.com> writes:

> On Aug 7, 2014, at 5:35, Akira Li <4kir4.1i at gmail.com> wrote:
>
>> Andrew Barnert
>> <abarnert at yahoo.com.dmarc.invalid> writes:
>> 
>>> On Aug 5, 2014, at 14:46, Petr Viktorin
>>> <encukou at gmail.com> wrote:
>>>> When people say "iso" in the context of datestimes, they usually mean RFC 3339.
>>> 
>>> RFC 3339 is still more complicated than just reversing Python's str or
>>> isoformat. IIRC (it's hard to check on my phone), it mandates that
>>> parsers should accept 2-digit years (including 3-digit or
>>> semicolon-and-two-digit years), lowercase T and Z, missing "-", and
>>> other things that you shouldn't generate but some code might.
>> 
>> Please, don't spread misinformation.
>> 
>> Among the explicit rfc 3339 design goals are simplicity and human
>> readability. 
>> 
>> Just read http://tools.ietf.org/html/rfc3339 (for an rfc it is
>> relatively short and readable).
...
> (And there's also a whole section of interpreting
> "legacy"/"deprecated" 2-digit years and how you should handle them.)
>
> So, is the RFC "spreading misinformation" about itself?

You are *obviously* wrong for the rfc 3339 Internet Date/Time Format
itself (used by __str__, isoformat -- relevant to the current topic).
http://tools.ietf.org/html/rfc3339

You can be *subtly* wrong for the rfc as a whole. 

The full ABNF from my previous message contains date-fullyear
definition. "The following profile of ISO 8601 [ISO8601] dates SHOULD be
used in new protocols on the Internet" [rfc3339]:

   date-fullyear   = 4DIGIT

It means that the year SHOULD be *exactly* 4 digits i.e., the rfc 3339
Internet Date/Time Format uses only 4-digit years.

I don't know what

  "mandates that parsers should accept 2-digit years" [Andrew Barnert]

means (does "mandates" mean SHOULD or MUST here?) but it seems inspired
by:

  "Internet Protocols MUST generate four digit years in dates. The use
  of 2-digit years is deprecated.  If a 2-digit year is received, it
  should be accepted ONLY if an incorrect interpretation will not cause
  a protocol or processing failure" [rfc3339]

and the words MUST and SHOULD are well-defined [rfc2119].

Do you see that 2-digit year MUST or SHOULD be accepted? (I don't see
it). 

  'missing "-"' [Andrew Barnert]

seems like an obvious mistake. All punctuaction except optional
time-secfrac (fraction of a second) is mandatory.

  "and other things that you shouldn't generate but some code might."
  [Andrew Barnert] 

What things? Could you be more specific? There is not much room in the
format. Redundant information is not included.

>> The format is so simple that people just write adhoc parsers using
> strptime() without installing any formal rfc3339 module (if it even
> exists). 
>
> Sure, and people also do that and call it an ISO parser. 
>
> If it can't interoperable with everything compliant applications may
> generate (much less deprecated formats the standard doesn't allow you
> to generate but mandates how you parse and interpret), it's not
> accurate to call it an RFC 3339 parser (at least not in a
> general-purpose library).
>
> As I said before, it's still certainly much easier to write an RFC
> 3339 parser than an ISO 8601 parser, but it's not as trivial as you're
> implying.

Compared to the full ISO 8601 format (I don't know whether it can be
parsed unambiguously), rfc 3339 (a conformant subset of the ISO 8601
extended format) is simple by design.

It *is* accurate to call it (with only 4-digit year support) an rfc 3339
parser in a general purpose library: compliant software MUST generate
4-digit year, Internet Date/Time Format SHOULD contain ONLY 4-digit year
(show quotes from the rfc that contain MUST, SHOULD, etc that say
otherwise if you disagree).

To be fair, the rfc *does not forbid* 2-digit year outright in *all
possible cases*.

Regardless the rfc language nuances, 2-digit year is harmful in practice
-- different software may interpret it differently: software that is
aware of rfc 3339 MUST generate 4-digit year, software that is not aware
of rfc 3339 can interpret 2-digit year differently. If an error is
possible when 2-digit year should not be used: "it should be accepted
ONLY if an incorrect interpretation will not cause a protocol or
processing failure" [rfc3339]


P.S.
...
>
> OK, I just read it. Among other things:
>
>> NOTE: Per [ABNF] and ISO8601, the "T" and "Z" characters in this
>> syntax may alternatively be lower case "t" or "z" respectively. This
>> date/time format may be used in some environments or contexts that
>> distinguish between the upper- and lower-case letters 'A'-'Z' and
>> a'-'z' (e.g. XML). Specifications that use this format in such
>> environments MAY further limit the date/time syntax so that the
>> letters 'T' and 'Z' used in the date/time syntax must always be
>> upper case. Applications that generate this format SHOULD use upper
>> case letters. NOTE: ISO 8601 defines date and time separated by
>> "T". Applications using this syntax may choose, for the sake of
>> readability, to specify a full-date and full-time separated by (say)
>> a space character. Klyne, et. al. Standards Track [Page 8]

What is the point of the copy-paste? Does handling lowercase "t", "z",
and a space complicates the parsing in a meaningful manner?

Imagine a parser that supports sep='T' (default for isoformat()) and
sep=' ' (space for __str__). How hard do you think to extend such parser
to support 't' as a separator?


--
Akira


From abarnert at yahoo.com  Fri Aug  8 08:10:57 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 7 Aug 2014 23:10:57 -0700
Subject: [Python-ideas] strptime without second argument as an
	inverse	to __str__
In-Reply-To: <87lhqza64b.fsf@gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <CA+=+wqDFFiu-1F9Gvd0JRQAFP_y17vBOo20pGXF1DEsnt-GfCQ@mail.gmail.com>
 <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com> <87vbq4a35u.fsf@gmail.com>
 <F4725A9B-1329-42B7-B1BC-66BB2488145B@yahoo.com> <87lhqza64b.fsf@gmail.com>
Message-ID: <1407478257.58223.YahooMailNeo@web181006.mail.ne1.yahoo.com>

Mixing up responses to an previous email you already responded to with the new one makes it harder to reply, but I'll try.


On Thursday, August 7, 2014 10:43 PM, Akira Li <4kir4.1i at gmail.com> wrote:
> > Andrew Barnert <abarnert at yahoo.com> writes:
> 
>>  On Aug 7, 2014, at 5:35, Akira Li <4kir4.1i at gmail.com> wrote:
>> 
>>>  Please, don't spread misinformation.
>>> 
>>>  Among the explicit rfc 3339 design goals are simplicity and human
>>>  readability. 
>>> 
>>>  Just read http://tools.ietf.org/html/rfc3339 (for an rfc it is
>>>  relatively short and readable).
> ...
>>  (And there's also a whole section of interpreting
>>  "legacy"/"deprecated" 2-digit years and how you should 
> handle them.)
>> 
>>  So, is the RFC "spreading misinformation" about itself?
> 
> You are *obviously* wrong for the rfc 3339 Internet Date/Time Format
> itself (used by __str__, isoformat -- relevant to the current topic).


You accused me of "spreading misinformation" by saying that RFC 3339 is more complicated than what Python's str generates. You also suggested that people often parse RFC 3339 with a simple strptime.

I pointed out that the RFC itself clearly defines something more complicated than what Python's str generates, and that it can't be fully parsed by a simple strptime, and then added a parenthetical remark about support for 2-year dates.

You cut out everything but the parenthetical remark, and replied to that as if it was the whole point of my message. And you even responded to nothing but the parenthetical 2-year comment in replies to completely separate sections of the message. I have no idea how to respond to that except to say: read it again.

The fact that a compliant parser should accept any of "T", "t", or " " as a separator already makes it impossible to parse with strptime, and more complicated than parsing Python's str output. I don't see how you can dispute that, or how it's "spreading misinformation" to point that out.

> ? "and other things that you shouldn't generate but some code?

> might."
> ? [Andrew Barnert] 
> 
> What things? Could you be more specific?

I was specific, and you chopped it out of my message, or moved it to a different part of the message.

>>>  The format is so simple that people just write adhoc parsers using
>>  strptime() without installing any formal rfc3339 module (if it even
>>  exists). 
>> 
>>  Sure, and people also do that and call it an ISO parser. 
>> 
>>  If it can't interoperable with everything compliant applications may
>>  generate (much less deprecated formats the standard doesn't allow you
>>  to generate but mandates how you parse and interpret), it's not
>>  accurate to call it an RFC 3339 parser (at least not in a
>>  general-purpose library).
>> 
>>  As I said before, it's still certainly much easier to write an RFC
>>  3339 parser than an ISO 8601 parser, but it's not as trivial as 
> you're
>>  implying.
> 
> Compared to the full ISO 8601 format (I don't know whether it can be
> parsed unambiguously), rfc 3339 (a conformant subset of the ISO 8601
> extended format) is simple by design.


Since you're arguing here exactly what I said, in the very paragraph you just quoted?"it's certainly much easier to write an RFC 3339 parser than an ISO 8601 parser"?I don't know why you think you have to convince me of that fact.

But again, that doesn't mean that a trivial strptime call is sufficient for a general-purpose library that claims to be a compliant RFC 3339 parser.

> Regardless the rfc language nuances, 2-digit year is harmful in practice


Of course. So what?

If my point actually were that RFC 3339 was complicated because of 2-digit years, and if that point were true, then this rebuttal might be relevant?but in that case it would only be proving the opposite point you're trying to make, that RFC 3339 parsing is _hard_. Fortunately, that isn't my point, and your statement is just irrelevant rather than contradictory to your whole argument.

> P.S.

> ...
> What is the point of the copy-paste? Does handling lowercase "t", 
> "z",
> and a space complicates the parsing in a meaningful manner?


Sure. Even ignoring the fact that you're claiming that people can parse it with strptime, the very fact that many people have written code that claims to be able to parse RFC 3339, and yet doesn't handle lowercase "t" and "z", shows that there is something you can get wrong; therefore, it is not trivial.

> Imagine a parser that supports sep='T' (default for isoformat()) and
> sep=' ' (space for __str__). How hard do you think to extend such parser
> to support 't' as a separator?


One again: "it's certainly much easier to write an RFC 3339 parser than an ISO 8601 parser," but it's not completely trivial.

You seem to think that this is a black-or-white issue: either RFC 3339 is a complete failure and it's actually as hard to parse as ISO 8601, or RFC 3339 is trivially parseable and no one should bother to write a parser for a subset of the language because it's just as easy to parse the whole thing. In reality, neither one of those is true. The latter may be closer to the truth than the former, but it's still wrong.

In other words, it makes perfect sense to write a parser for exactly what Python generates, and not claim it as RFC 3339 compliant.

From 4kir4.1i at gmail.com  Fri Aug  8 17:17:36 2014
From: 4kir4.1i at gmail.com (Akira Li)
Date: Fri, 08 Aug 2014 19:17:36 +0400
Subject: [Python-ideas] strptime without second argument as an
	inverse	to __str__
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org>
 <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <CA+=+wqDFFiu-1F9Gvd0JRQAFP_y17vBOo20pGXF1DEsnt-GfCQ@mail.gmail.com>
 <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com>
 <87vbq4a35u.fsf@gmail.com>
 <F4725A9B-1329-42B7-B1BC-66BB2488145B@yahoo.com>
 <87lhqza64b.fsf@gmail.com>
 <1407478257.58223.YahooMailNeo@web181006.mail.ne1.yahoo.com>
Message-ID: <87a97f9fjj.fsf@gmail.com>


Please, don't put words in my mouth. To avoid Straw-man, quote me
directly in context. As I do with you in my messages.

Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> writes:

> Mixing up responses to an previous email you already responded to with
> the new one makes it harder to reply, but I'll try.
>
>
> On Thursday, August 7, 2014 10:43 PM, Akira Li <4kir4.1i at gmail.com> wrote:
>> > Andrew Barnert <abarnert at yahoo.com> writes:
>>
>>>  On Aug 7, 2014, at 5:35, Akira Li <4kir4.1i at gmail.com> wrote:
>>>
>>>>  Please, don't spread misinformation.
>>>>
>>>>  Among the explicit rfc 3339 design goals are simplicity and human
>>>>  readability.
>>>>
>>>>  Just read http://tools.ietf.org/html/rfc3339 (for an rfc it is
>>>>  relatively short and readable).
>> ...
>>>  (And there's also a whole section of interpreting
>>>  "legacy"/"deprecated" 2-digit years and how you should
>> handle them.)
>>>
>>>  So, is the RFC "spreading misinformation" about itself?
>>
>> You are *obviously* wrong for the rfc 3339 Internet Date/Time Format
>> itself (used by __str__, isoformat -- relevant to the current topic).
>
>
> You accused me of "spreading misinformation" by saying that RFC 3339
> is more complicated than what Python's str generates. You also
> suggested that people often parse RFC 3339 with a simple strptime.

You said [1]:

  "RFC 3339 is still more complicated than just reversing Python's str
  or isoformat. IIRC (it's hard to check on my phone), it mandates that
  parsers should accept 2-digit years (including 3-digit or
  semicolon-and-two-digit years), lowercase T and Z, missing "-", and
  other things that you shouldn't generate but some code might."

[1]
https://mail.python.org/pipermail/python-ideas/2014-August/028509.html

What I consider misinformation:

  "it mandates that parsers should accept 2-digit years (including
  3-digit or semicolon-and-two-digit years)" [Andrew Barnert]

  '"missing "-"' [Andrew Barnert]

   "and other things that you shouldn't generate but some code might."
   [Andrew Barnert]

I've already described it quote by quote in [2] with all the gory
details including a careful usage of MUST, SHOULD words from [rfc
2119]. 

[2]
https://mail.python.org/pipermail/python-ideas/2014-August/028541.html
[rfc 2119] http://tools.ietf.org/html/rfc2119

> I pointed out that the RFC itself clearly defines something more
> complicated than what Python's str generates, and that it can't be
> fully parsed by a simple strptime, and then added a parenthetical
> remark about support for 2-year dates.

It *is* obvious that the rfc is more complex than the str method e.g.,
the rfc supports 'T', 'Z' (default str doesn't generate them as far as I
know). 

But all specific details (except "lowercase T and Z") you provided in
the quote from [1] are wrong as described in [2]. What conclusions can
you draw about the whole statement after that?

> You cut out everything but the parenthetical remark, and replied to
> that as if it was the whole point of my message. And you even
> responded to nothing but the parenthetical 2-year comment in replies
> to completely separate sections of the message. I have no idea how to
> respond to that except to say: read it again.

I've read and reread [1]. I don't see what "other things" you are referring
to. To avoid ambiguity, could you use a direct quote and a link (as I've
demonstrated above)?


--
Akira


From abarnert at yahoo.com  Fri Aug  8 18:46:40 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 8 Aug 2014 09:46:40 -0700
Subject: [Python-ideas] strptime without second argument as an
	inverse	to __str__
In-Reply-To: <87a97f9fjj.fsf@gmail.com>
References: <ed6cab97-c2de-4015-a239-6111d4b402fe@googlegroups.com>
 <20140804181528.GP4525@ando>
 <CAP7h-xb7sVn_A=qrQ4dCK1Phi-C5zWy+ZTvnHR5Q4bGgjwat9A@mail.gmail.com>
 <CANc-5Uyoue2qKy8b26oe+8f90GhpVnJt14M9JzVfove-OJJabA@mail.gmail.com>
 <CAP7h-xZX06O5NFzNHwvW7zMqhY3x_uVFSZu8WAgDcRZhA2uU2Q@mail.gmail.com>
 <CANc-5Uwth6Cmq64=L1zH2o2LgttDU2rOUrTCsNw+AT74qRb5MQ@mail.gmail.com>
 <lros2p$kdd$1@ger.gmane.org> <20140805013944.GQ4525@ando>
 <lrrhu3$p2l$1@ger.gmane.org> <FD7246C5-3BC1-478C-93C9-1A01DFE38058@yahoo.com>
 <CA+=+wqDFFiu-1F9Gvd0JRQAFP_y17vBOo20pGXF1DEsnt-GfCQ@mail.gmail.com>
 <4C035B91-0520-412C-8E47-19E4341FABF2@yahoo.com> <87vbq4a35u.fsf@gmail.com>
 <F4725A9B-1329-42B7-B1BC-66BB2488145B@yahoo.com> <87lhqza64b.fsf@gmail.com>
 <1407478257.58223.YahooMailNeo@web181006.mail.ne1.yahoo.com>
 <87a97f9fjj.fsf@gmail.com>
Message-ID: <2B496614-E2B8-48CD-8B95-537B5A589CCC@yahoo.com>

On Aug 8, 2014, at 8:17, Akira Li <4kir4.1i at gmail.com> wrote:

> It *is* obvious that the rfc is more complex than the str method e.g.,
> the rfc supports 'T', 'Z' (default str doesn't generate them as far as I
> know). 

This is the only point that matters here to me. If you're no longer disagreeing with it, I have no interest in continuing to argue, and I doubt anyone else has any interest in reading it.

So, let me summarize and see if you have anything substantive to disagree with:

People want a function to reverse __str__, possibly also handling space as a separator. Such a function is trivial to write. If it's not a fully compliant RFC 3339 parser, that's fine, and it will be useful for it's intended goal, as long as it isn't called fromrfc3339string (which no one has suggested anyway, although someone did suggest fromisostring, which is what started the whole RFC 3339 side track--obviously that name shouldn't be used either).

From ncoghlan at gmail.com  Sun Aug 10 14:33:49 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 10 Aug 2014 22:33:49 +1000
Subject: [Python-ideas] The non-obvious nature of str.join (was Re: sum(...)
	limitation)
Message-ID: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>

(switching from python-dev to python-ideas)

FWIW, I don't consider str.join *or* sum with an empty string as the
starting point to be particularly intuitive ways of joining iterables of
strings.

str.join was invented before we had keyword-only arguments as a common
construct, and before print became an ordinary function that accepted a
"sep" keyword-only argument.

I'd be interested in seeing a concrete proposal for a "concat" builtin that
accepted a "sep" keyword only argument. Even if such a PEP ends up being
rejected, it would hopefully help cut short the *next* potentially
interminable thread on the topic by gathering the arguments for and against
in a more readily accessible place.

Regards,
Nick.
 On 10 Aug 2014 18:26, "Stephen J. Turnbull" <stephen at xemacs.org> wrote:

> Alexander Belopolsky writes:
>  > On Sat, Aug 9, 2014 at 3:08 AM, Stephen J. Turnbull <stephen at xemacs.org
> >
>  > wrote:
>  >
>  > > All the suggestions
>  > > I've seen so far are (IMHO, YMMV) just as ugly as the present
>  > > situation.
>  > >
>  >
>  > What is ugly about allowing strings?  CPython certainly has a way to to
>  > make sum(x, '')
>
> sum(it, '') itself is ugly.  As I say, YMMV, but in general last I
> heard arguments that are usually constants drawn from a small set of
> constants are considered un-Pythonic; a separate function to express
> that case is preferred.  I like the separate function style.
>
> And that's the current situation, except that in the case of strings
> it turns out to be useful to allow for "sums" that have "glue" at the
> joints, so it's spelled as a string method rather than a builtin: eg,
> ", ".join(paramlist).
>
> Actually ... if I were a fan of the "".join() idiom, I'd seriously
> propose 0.sum(numeric_iterable) as the RightThang{tm].  Then we could
> deprecate "".join(string_iterable) in favor of "".sum(string_iterable)
> (with the same efficient semantics).
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140810/9d71f3d8/attachment.html>

From tjreedy at udel.edu  Sun Aug 10 20:25:08 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 10 Aug 2014 14:25:08 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
Message-ID: <ls8den$d7t$1@ger.gmane.org>

On 8/10/2014 8:33 AM, Nick Coghlan wrote:
> (switching from python-dev to python-ideas)
>
> FWIW, I don't consider str.join *or* sum with an empty string as the
> starting point to be particularly intuitive ways of joining iterables of
> strings.

I think sum(iofs, '') is obvious and I do not understand the 
determination to not allow it and implement it as ''.join(iofs). 
Perhaps that is because I think '+' is the *right* choice for string 
concatenation.  Addition of tallies (base 1 numbers) is concatenation. 
For tallies, or strings, r and s, len(r+s) == len(r) + len(s).  (That 
this is not true of sets is a reason to not use '+ for set union, which 
Python doesn't.)

(I also think sum(it_of_lists, somelist), which is allowed, should be 
implemented as somelist.extend(it_of_lists), but that is another issue.)

> str.join was invented before we had keyword-only arguments as a common
> construct, and before print became an ordinary function that accepted a
> "sep" keyword-only argument.

The reason sep in print has to be keyword-only is because print takes an 
indefinite of main arguments rather than an iterable of items to print. 
This is appropriate because we generally print a few items that are not 
already in a collection, because collections of strings can be .joined, 
and because collections can have, and builtin collections do have, 
methods to join the string representations of their members. Print, str 
and .__str__ methods, and str.join works together wonderfully.

The start paramenter of sum does not need to be keyword only because sum 
takes one iterable of items to sum. This is appropriate because a few 
items to sum, not in a collection, can be handled by '+', and sum is 
needed for large collections of items already in a collection.

> I'd be interested in seeing a concrete proposal for a "concat" builtin
> that accepted a "sep" keyword only argument. Even if such a PEP ends up
> being rejected, it would hopefully help cut short the *next* potentially
> interminable thread on the topic by gathering the arguments for and
> against in a more readily accessible place.

For the against list: New builtins should add something more than a pure 
synonym;  "concat(iofs, sep=joiner)" is harder (6 more letters), not 
easier, to write than "joiner.join(iofs)".

-- 
Terry Jan Reedy


From abarnert at yahoo.com  Sun Aug 10 21:48:08 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 10 Aug 2014 12:48:08 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was
	Re:	sum(...) limitation)
In-Reply-To: <ls8den$d7t$1@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
Message-ID: <1407700088.46613.YahooMailNeo@web181002.mail.ne1.yahoo.com>

On Sunday, August 10, 2014 11:26 AM, Terry Reedy <tjreedy at udel.edu> wrote:

?
> For the against list: New builtins should add something more than a pure 
> synonym;? "concat(iofs, sep=joiner)" is harder (6 more letters), not 
> easier, to write than "joiner.join(iofs)".


If concat instead meant "joiner.join(map(type(joiner), iofs))", that might be more useful than just a synonym. I've seen many novices try to figure out how to join up their list of ints.

Also, it could conceivably be faster for user string-like classes, especially mutable ones (because then, as long as they had slice replacement, everything else would happen in C, whereas if they have to write a custom join, it will be looping in Python).

Also, being a new function, it might be reasonable for concat to handle iterators differently from sequences (exponential expansion rather than preallocating). I've seen a couple people asking why ''.join(genexpr) uses twice as much memory as they expected (although far fewer than the joining-up-ints questions).

Of course any (or all) of those also might well be a bad idea in its own right. But a new function gives us a chance to think through how it should work?and then, if it turns out it should work exactly like join, the answer may be "just a synonym, reject it", but if not, it might be reasonable to add it and gradually phase out join.

From toddrjen at gmail.com  Sun Aug 10 22:43:02 2014
From: toddrjen at gmail.com (Todd)
Date: Sun, 10 Aug 2014 22:43:02 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <1407700088.46613.YahooMailNeo@web181002.mail.ne1.yahoo.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <1407700088.46613.YahooMailNeo@web181002.mail.ne1.yahoo.com>
Message-ID: <CAFpSVp+-orY50Kpe4KqjJnHw4Kazht+3gJPNC9A4KLpkZ1YXkA@mail.gmail.com>

On Aug 10, 2014 9:51 PM, "Andrew Barnert" <abarnert at yahoo.com.dmarc.invalid>
wrote:
>
> On Sunday, August 10, 2014 11:26 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>
>
> > For the against list: New builtins should add something more than a pure
> > synonym;  "concat(iofs, sep=joiner)" is harder (6 more letters), not
> > easier, to write than "joiner.join(iofs)".
>
>
> If concat instead meant "joiner.join(map(type(joiner), iofs))", that
might be more useful than just a synonym. I've seen many novices try to
figure out how to join up their list of ints.

Couldn't this also be handled with a keyword argument to join?  For example
"joiner.join(iofs, convert=True)".

> Also, it could conceivably be faster for user string-like classes,
especially mutable ones (because then, as long as they had slice
replacement, everything else would happen in C, whereas if they have to
write a custom join, it will be looping in Python).

This seems like something an abstract base class could take care of.

> Also, being a new function, it might be reasonable for concat to handle
iterators differently from sequences (exponential expansion rather than
preallocating). I've seen a couple people asking why ''.join(genexpr) uses
twice as much memory as they expected (although far fewer than the
joining-up-ints questions).

Isn't this an implementation detail that could be fixed?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140810/1823a8ab/attachment-0001.html>

From stephen at xemacs.org  Mon Aug 11 02:20:53 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 11 Aug 2014 09:20:53 +0900
Subject: [Python-ideas] The non-obvious nature of str.join (was
	Re:	sum(...) limitation)
In-Reply-To: <ls8den$d7t$1@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
Message-ID: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>

Terry Reedy writes:

 > Addition of tallies (base 1 numbers) is concatenation. 

Except that nobody actually tallies that way.  In practice everybody
I've ever seen count to ten by tallying does something special for
fives (Americans tally four times then cross the four vertical strokes
with a fifth, Japanese build the character sei (? == true)).  So what
you're actually arguing is circular: if you represent a tally as a
string with a singleton alphabet, it behaves like a string.  But in
Japanese it's actually a 5 element alphabet, and your homomorphism:

 > For tallies, or strings, r and s, len(r+s) == len(r) + len(s).

fails.

I think that rather than focus on formal properties, we should look at
how sum is used in Python.  For example, numerical sum is in fact an
attractive nuisance as long as floats are considered numbers.  So
unless and until we deprecate floats, "formal sum" will always be
splintered into multiple functions.

As far as concat goes, I don't really see what advantage it has over
joiner.join, except in teaching.  That might be enough for a builtin
in this case, since concatenating iterables of strings is a pretty
frequent operation in my experience.


From wolfgang.maier at biologie.uni-freiburg.de  Mon Aug 11 08:56:28 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Mon, 11 Aug 2014 08:56:28 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <ls9pcc$vqs$1@ger.gmane.org>

On 08/11/2014 02:20 AM, Stephen J. Turnbull wrote:
>
> As far as concat goes, I don't really see what advantage it has over
> joiner.join, except in teaching.  That might be enough for a builtin
> in this case, since concatenating iterables of strings is a pretty
> frequent operation in my experience.
>

I am not so sure about the benefit for teaching. I am using Python for 
teaching programming to absolute beginners at university and, in my 
experience, joiner.join is never a big hurdle. You have to explain it 
once, of course, but after that people, especially people not biased by 
other languages, do not have problems with it.

Wolfgang


From alexander.belopolsky at gmail.com  Mon Aug 11 15:56:38 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 11 Aug 2014 09:56:38 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <ls9pcc$vqs$1@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
Message-ID: <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>

On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier <
wolfgang.maier at biologie.uni-freiburg.de> wrote:

> I am using Python for teaching programming to absolute beginners at
> university and, in my experience, joiner.join is never a big hurdle.


In my experience, it is the asymmetry between x.join(y) and x.split(y)
which causes most of the confusion.  In x.join(y), x is the separator and y
is the data being joined, but in x.split(y), it is the other way around.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/ad4f4728/attachment.html>

From wolfgang.maier at biologie.uni-freiburg.de  Mon Aug 11 16:12:29 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Mon, 11 Aug 2014 16:12:29 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
Message-ID: <lsaits$mje$1@ger.gmane.org>

On 08/11/2014 03:56 PM, Alexander Belopolsky wrote:
>
> On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de
> <mailto:wolfgang.maier at biologie.uni-freiburg.de>>
> wrote:
>
>     I am using Python for teaching programming to absolute beginners at
>     university and, in my experience, joiner.join is never a big hurdle.
>
>
> In my experience, it is the asymmetry between x.join(y) and x.split(y)
> which causes most of the confusion.  In x.join(y), x is the separator
> and y is the data being joined, but in x.split(y), it is the other way
> around.
>

Once you have explained the beauty of

s == sep.join(s.split(sep))

to anyone, they will not be confused again, but find it perfectly logical.


From bborcic at gmail.com  Mon Aug 11 16:46:37 2014
From: bborcic at gmail.com (Boris Borcic)
Date: Mon, 11 Aug 2014 16:46:37 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <lsaits$mje$1@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsaits$mje$1@ger.gmane.org>
Message-ID: <lsal0e$iuj$1@ger.gmane.org>

Wolfgang Maier wrote:

>
> s == sep.join(s.split(sep))
>
> to anyone, they will not be confused again, but find it perfectly logical.


You are kidding, right? What will prevent them from using logic to recall the invariant mistakenly as
s == sep.join(sep.split(s)) ? If only because there are natural use cases for passing mysep.join around as a 
callable that will join with a fixed separator, and there would similarly exist use cases for mysep.split, 
while there are really none for passing around mystring.split to split a fixed string by varying separators.

This whole corner of python is a most ugly system of inconsistent acne scars in the 2.x series (dunno about 
3.x). What with sum(...) pedantically telling you to use str.join - if it knows what you want, it should just 
to it.

---
Ce courrier ?lectronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active.
http://www.avast.com



From breamoreboy at yahoo.co.uk  Mon Aug 11 16:59:12 2014
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Mon, 11 Aug 2014 15:59:12 +0100
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
Message-ID: <lsalo0$rkd$2@ger.gmane.org>

On 11/08/2014 14:56, Alexander Belopolsky wrote:
>
> On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de
> <mailto:wolfgang.maier at biologie.uni-freiburg.de>>
> wrote:
>
>     I am using Python for teaching programming to absolute beginners at
>     university and, in my experience, joiner.join is never a big hurdle.
>
>
> In my experience, it is the asymmetry between x.join(y) and x.split(y)
> which causes most of the confusion.  In x.join(y), x is the separator
> and y is the data being joined, but in x.split(y), it is the other way
> around.
>

Could you try something which to my understanding is unprecedented in 
the world of computing, as in point them to the docs?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From alexander.belopolsky at gmail.com  Mon Aug 11 17:47:29 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 11 Aug 2014 11:47:29 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <lsalo0$rkd$2@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
Message-ID: <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>

On Mon, Aug 11, 2014 at 10:59 AM, Mark Lawrence <breamoreboy at yahoo.co.uk>
wrote:

> Could you try something which to my understanding is unprecedented in the
> world of computing, as in point them to the docs?


Sure - this is the universal remedy to any design mistake, but how many
people would need to look at the docs to understand something like
join(list, sep=',') or split(string, sep=',')?  And how many of those
seeing x.join(y) and x.split(y) for the first time will guess which
argument is data and which is separator?

Beautiful is better than ugly.
Explicit is better than implicit.
..
Readability counts.
..
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
..

(The Zen of Python, by Tim Peters, AKA python -m this)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/26c431be/attachment.html>

From toddrjen at gmail.com  Mon Aug 11 17:55:25 2014
From: toddrjen at gmail.com (Todd)
Date: Mon, 11 Aug 2014 17:55:25 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
Message-ID: <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>

On Mon, Aug 11, 2014 at 3:56 PM, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
> On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier <
> wolfgang.maier at biologie.uni-freiburg.de> wrote:
>
>> I am using Python for teaching programming to absolute beginners at
>> university and, in my experience, joiner.join is never a big hurdle.
>
>
> In my experience, it is the asymmetry between x.join(y) and x.split(y)
> which causes most of the confusion.  In x.join(y), x is the separator and y
> is the data being joined, but in x.split(y), it is the other way around.
>

What would be the solution to this?  Making join a list method, or
reversing the behavior of split which means it no longer acts on x?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/9d491a4f/attachment-0001.html>

From me+python at ixokai.io  Mon Aug 11 18:21:41 2014
From: me+python at ixokai.io (Stephen Hansen)
Date: Mon, 11 Aug 2014 09:21:41 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
Message-ID: <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>

On Mon, Aug 11, 2014 at 8:47 AM, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
> On Mon, Aug 11, 2014 at 10:59 AM, Mark Lawrence <breamoreboy at yahoo.co.uk>
> wrote:
>
>> Could you try something which to my understanding is unprecedented in the
>> world of computing, as in point them to the docs?
>
>
> Sure - this is the universal remedy to any design mistake, but how many
> people would need to look at the docs to understand something like
> join(list, sep=',') or split(string, sep=',')?  And how many of those
> seeing x.join(y) and x.split(y) for the first time will guess which
> argument is data and which is separator?
>

Two new builtins now?

"x.split(y)" is ambiguous because its terrible code, not because of the
order. Names matter and exist to provide clarity. Don't name variables 'x'
and 'y'. If it were as simple as "x.split(sep)" or "x.split('\t')" then I
bet almost no one will need to consult the documentation to know which is
which; similarly with "sep.join(list_of_strings)". Now for someone wanting
to know how to join a list of strings and doesn't know how,
sep.join(strings) is not per se obviously discoverable, but it isn't that
bad and people will have to go to the docs anyways. They are likely to look
in the string section I think, and find it. They might look in the list
section, and may look in the bulitins section, but its still not that bad.

As for how many would need to go look it up with your proposed builtins,
anyone who wanted to set the separator, that's how many. And considering
"," is not even kind of a sensible default for either, that'd be a lot.

I remember this discussion very clearly when sum went in, and again during
the py3k period, and here the zombie rises again.

 Beautiful is better than ugly.
> Explicit is better than implicit.
> ..
> Readability counts.
> ..
> In the face of ambiguity, refuse the temptation to guess.
> There should be one-- and preferably only one --obvious way to do it.
> ..
>
> (The Zen of Python, by Tim Peters, AKA python -m this)
>

The Zen are nice words and good pieces of advice that are not a tool to
wield to win an debate, especially since most of them are fundamentally
subjective and require you to think in Python for their full meaning to
have an impact. They were also written after the fact by a long margin as a
joke, even if like most jokes they have a measure of truth in them.

Adding builtins defies 'Explicit is better then implicit', and though
",".join(stuff) is arguably less then beautiful, the axiom says *better*
not *best*, and the order of things to almost any function at all could be
called ambiguous until you look it up. Finally, at least ",".join(stuff)
directly (and explicitly!) refuses any temptation to guess.

Zen aside, I'm not sure if sum(list_of_strings) is that pancea of
readability and beauty and who-needs-documentation-now that its sort of
implied in this thread, that it doesn't work for strings is for reasons
already specified (CPython's immutable strings implies it could lead to
*BROKEN* behavior in a reasonable implementation, even if CPython currently
has a limited optimization). I ever so much don't believe we need "join"
and "split" builtins, I much rather think your arguments for forcing people
to use str.join make more sense, but I hope we don't see that either.

This isn't a situation that causes a huge amount of confusion. Teach the
students that sep.join(list_of_strings) make sense because who would know
how to combine strings, but a string itself? Of course list_of_strings
wouldn't know this. Watch the awareness blossom. Sometimes I think people
overstate how big a deal things are in teaching because we all care about
making Python teachable.

--S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/3400fce7/attachment.html>

From alexander.belopolsky at gmail.com  Mon Aug 11 18:22:59 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 11 Aug 2014 12:22:59 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
Message-ID: <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>

On Mon, Aug 11, 2014 at 11:55 AM, Todd <toddrjen at gmail.com> wrote:

> In my experience, it is the asymmetry between x.join(y) and x.split(y)
>> which causes most of the confusion.  In x.join(y), x is the separator and y
>> is the data being joined, but in x.split(y), it is the other way around.
>>
>
> What would be the solution to this?
>

Allow sum(list_of_strings, '') and stop mocking people who prefer it to
''.join(..).  This will not solve all the issues with join/split, but at
least a simple task of concatenating a list of strings will have a more or
less obvious solution.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/49003a10/attachment.html>

From daviesk24 at yahoo.com  Mon Aug 11 18:25:31 2014
From: daviesk24 at yahoo.com (Kevin Davies)
Date: Mon, 11 Aug 2014 06:25:31 -1000
Subject: [Python-ideas] float comparison in doctest
Message-ID: <53E8EE7B.50408@yahoo.com>

Erik Bray has created a nifty addition to doctest to compare floating 
point numbers using a `+FLOAT_CMP` flag. It is implemented within 
Astropy (http://www.astropy.org/) and the related issue is 
https://github.com/astropy/astropy/issues/2662. According to Erik, it 
may need some tweaks, but I think it would be a really useful feature to 
make available in the doctest package itself.

Kevin

From toddrjen at gmail.com  Mon Aug 11 18:24:35 2014
From: toddrjen at gmail.com (Todd)
Date: Mon, 11 Aug 2014 18:24:35 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
Message-ID: <CAFpSVpKEt3tGr=EYXbiAvjSVUR_k0Zt3ghr3PgQ32djf4Wn7HA@mail.gmail.com>

On Mon, Aug 11, 2014 at 6:22 PM, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
> On Mon, Aug 11, 2014 at 11:55 AM, Todd <toddrjen at gmail.com> wrote:
>
>> In my experience, it is the asymmetry between x.join(y) and x.split(y)
>>> which causes most of the confusion.  In x.join(y), x is the separator and y
>>> is the data being joined, but in x.split(y), it is the other way around.
>>>
>>
>> What would be the solution to this?
>>
>
> Allow sum(list_of_strings, '') and stop mocking people who prefer it to
> ''.join(..).  This will not solve all the issues with join/split, but at
> least a simple task of concatenating a list of strings will have a more or
> less obvious solution.
>

I am confused, if it won't solve the problem, how is it relevant to my
question?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/ba62cc1e/attachment-0001.html>

From alexander.belopolsky at gmail.com  Mon Aug 11 18:38:10 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 11 Aug 2014 12:38:10 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
Message-ID: <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>

On Mon, Aug 11, 2014 at 12:21 PM, Stephen Hansen <me+python at ixokai.io>
wrote:

> Don't name variables 'x' and 'y'. If it were as simple as "x.split(sep)"
> or "x.split('\t')" then I bet almost no one will need to consult the
> documentation to know which is which


Sure, but on the same token if someone writes sep.split(x), how likely will
this error be caught on a quick review?  This is not theoretical.  I've
seen people make this mistake and asking for help in debugging rather
non-obvious behaviors.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/b0ee91dc/attachment.html>

From me+python at ixokai.io  Mon Aug 11 18:37:39 2014
From: me+python at ixokai.io (Stephen Hansen)
Date: Mon, 11 Aug 2014 09:37:39 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
Message-ID: <CAM1gar5c=DbZuoP2h628hirdi8v0sa22iyPp0uy73LJep22USw@mail.gmail.com>

On Mon, Aug 11, 2014 at 9:22 AM, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
> On Mon, Aug 11, 2014 at 11:55 AM, Todd <toddrjen at gmail.com> wrote:
>
>> In my experience, it is the asymmetry between x.join(y) and x.split(y)
>>> which causes most of the confusion.  In x.join(y), x is the separator and y
>>> is the data being joined, but in x.split(y), it is the other way around.
>>>
>>
>> What would be the solution to this?
>>
>
> Allow sum(list_of_strings, '') and stop mocking people who prefer it to
> ''.join(..).  This will not solve all the issues with join/split, but at
> least a simple task of concatenating a list of strings will have a more or
> less obvious solution.
>

Then we'll have two obvious solutions since "".join() and the huge body of
code using it won't go away. One which is only even vaguely workable
because of an implementation-specific optimization that isn't a promise of
the language, which makes it at best obvious* if not obvious-ish.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/694b9902/attachment.html>

From alexander.belopolsky at gmail.com  Mon Aug 11 18:46:43 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 11 Aug 2014 12:46:43 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAM1gar5c=DbZuoP2h628hirdi8v0sa22iyPp0uy73LJep22USw@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAM1gar5c=DbZuoP2h628hirdi8v0sa22iyPp0uy73LJep22USw@mail.gmail.com>
Message-ID: <CAP7h-xaGriZJ4=KEYeDMwcXRWPLwauXi5GXx5phDQNWv4xuimA@mail.gmail.com>

On Mon, Aug 11, 2014 at 12:37 PM, Stephen Hansen <me+python at ixokai.io>
wrote:

> Then we'll have two obvious solutions since "".join() and the huge body of
> code using it won't go away.


''.join() is not an obvious solution to concatenation.  It is a fringe case
of a solution to a completely different problem - building a delimited
string.  Buy the same token, we have "two obvious solutions" to negation:
-x and 0-x.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/ec5ebf02/attachment.html>

From me+python at ixokai.io  Mon Aug 11 18:47:09 2014
From: me+python at ixokai.io (Stephen Hansen)
Date: Mon, 11 Aug 2014 09:47:09 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
Message-ID: <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>

On Mon, Aug 11, 2014 at 9:38 AM, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
> On Mon, Aug 11, 2014 at 12:21 PM, Stephen Hansen <me+python at ixokai.io>
> wrote:
>
>> Don't name variables 'x' and 'y'. If it were as simple as "x.split(sep)"
>> or "x.split('\t')" then I bet almost no one will need to consult the
>> documentation to know which is which
>
>
> Sure, but on the same token if someone writes sep.split(x), how likely
> will this error be caught on a quick review?  This is not theoretical.
>  I've seen people make this mistake and asking for help in debugging rather
> non-obvious behaviors.
>

I've seen people make innumerable mistakes in the past, I've seen certain
classes of mistakes repeated -- this isn't an argument for change by
itself. People make mistakes. Its going to happen.

That said, I find the idea that "x.split(sep)" as non-obvious to be...
weird, to say the least. But, obvious is subjective. Your obvious may not
be my obvious nor most people's obvious.

Yes, sep.join(list) is a bit of a weird construct, but its one thing to
learn, and its not a hard one to teach at that. In fact, it makes very
logical sense once you explain it and makes people think of things more
Pythonically after. I say from experience, not in theory. But,
string.split(sep) is very natural. You seem to think that they need to be
in the same order to be obvious but I don't see why nor do I think any of
the alternatives are not without problems that are bigger issues.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/e410cb2b/attachment.html>

From alexander.belopolsky at gmail.com  Mon Aug 11 19:14:25 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 11 Aug 2014 13:14:25 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
Message-ID: <CAP7h-xbsKBJEhuwM+=udj8H-grXOFrdVgtzUnBy7zDKbahP-OA@mail.gmail.com>

On Mon, Aug 11, 2014 at 12:47 PM, Stephen Hansen <me+python at ixokai.io>
wrote:

> Yes, sep.join(list) is a bit of a weird construct, but its one thing to
> learn, and its not a hard one to teach at that. In fact, it makes very
> logical sense once you explain it and makes people think of things more
> Pythonically after. I say from experience, not in theory. But,
> string.split(sep) is very natural. You seem to think that they need to be
> in the same order to be obvious but I don't see why nor do I think any of
> the alternatives are not without problems that are bigger issues.


I am not suggesting any changes to str.join or str.split methods.  I am
just arguing that sum(list_of_strings, '') should be allowed by the
language and its performance is a matter of the quality of implementation.
 Once you learn that string addition is concatenation in Python, it is
natural to "sum" lists of Python strings regardless of whether  it makes
sense in your native language.

sep.join(list) is not such a weird construct when sep is non-empty - it is
the sep='' case which is weird and non-obvious.  (Note that someone in this
thread suggested demonstrating s == sep.join(s.split(sep)) invariant as a
teaching tool, but this invariant fails when sep is empty.)  When you are
tasked with finding s1 + s2 + ... + sN given [s1, s2, ..., sN], it is sum
that first comes to mind, not join.  The situation is different when you
have a separator to begin with, but when you don't using an empty separator
feels like a performance hack in the absence of an efficient natural
solution.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/c59c33b6/attachment-0001.html>

From haoyi.sg at gmail.com  Mon Aug 11 19:10:20 2014
From: haoyi.sg at gmail.com (Haoyi Li)
Date: Mon, 11 Aug 2014 10:10:20 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
Message-ID: <CALruUQL1dA5EcMd0_rfPjM1Y7Y8+hD8QUPzYrKvJbZz0jq0b5w@mail.gmail.com>

> In fact, it makes very logical sense once you explain it and makes people
think of things more Pythonically after. I say from experience, not in
theory.

Could you elaborate about "making people think more pythonically after"
bit? I can see how explaining the API makes people understand the API, but
I'm curious how it makes people behave differently after.
On Mon, Aug 11, 2014 at 9:38 AM, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
> On Mon, Aug 11, 2014 at 12:21 PM, Stephen Hansen <me+python at ixokai.io>
> wrote:
>
>> Don't name variables 'x' and 'y'. If it were as simple as "x.split(sep)"
>> or "x.split('\t')" then I bet almost no one will need to consult the
>> documentation to know which is which
>
>
> Sure, but on the same token if someone writes sep.split(x), how likely
> will this error be caught on a quick review?  This is not theoretical.
>  I've seen people make this mistake and asking for help in debugging rather
> non-obvious behaviors.
>

I've seen people make innumerable mistakes in the past, I've seen certain
classes of mistakes repeated -- this isn't an argument for change by
itself. People make mistakes. Its going to happen.

That said, I find the idea that "x.split(sep)" as non-obvious to be...
weird, to say the least. But, obvious is subjective. Your obvious may not
be my obvious nor most people's obvious.

Yes, sep.join(list) is a bit of a weird construct, but its one thing to
learn, and its not a hard one to teach at that. In fact, it makes very
logical sense once you explain it and makes people think of things more
Pythonically after. I say from experience, not in theory. But,
string.split(sep) is very natural. You seem to think that they need to be
in the same order to be obvious but I don't see why nor do I think any of
the alternatives are not without problems that are bigger issues.


_______________________________________________
Python-ideas mailing list
Python-ideas at python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/c6e1a298/attachment.html>

From me+python at ixokai.io  Mon Aug 11 19:45:34 2014
From: me+python at ixokai.io (Stephen Hansen)
Date: Mon, 11 Aug 2014 10:45:34 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CALruUQL1dA5EcMd0_rfPjM1Y7Y8+hD8QUPzYrKvJbZz0jq0b5w@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
 <CALruUQL1dA5EcMd0_rfPjM1Y7Y8+hD8QUPzYrKvJbZz0jq0b5w@mail.gmail.com>
Message-ID: <CAM1gar6A=-h6qx4PXS8y90+XXOe7G77BCjoNjPK41uN+2V6YOQ@mail.gmail.com>

On Mon, Aug 11, 2014 at 10:10 AM, Haoyi Li <haoyi.sg at gmail.com> wrote:

> > In fact, it makes very logical sense once you explain it and makes
> people think of things more Pythonically after. I say from experience, not
> in theory.
>
> Could you elaborate about "making people think more pythonically after"
> bit? I can see how explaining the API makes people understand the API, but
> I'm curious how it makes people behave differently after.
>

Well, its an opinion of course, so it may be as useful as the thought that
the sky is high. It doesn't make people behave differently, but it does
lead them to writing idiomatic code. There's naive Python code, which
ideally should be clear and easy still, and then there's idiomatic Python
code which has a beauty in its expressiveness while also being (relatively)
efficient.

But, if the opposite of "string.split(sep)" were
"list_of_strings.join(sep)" then that would mean that List  would have to
know about how to combine strings. That's not very Pythonic. Also, Strings
are immutable, that means:

str1 = str1 + str2

logically leads one to assume that it is constructing an entirely new
string, copying the contents of str1 and str2 into it, and discarding the
original str1. In fact, it did that for a long time. You can't change
strings, after all. That is an fundamental tenet, and when people learn
that they will get a new tool and think closer to idiomatic Python. Now,
CPython has an optimization for this case which is the only reason the idea
of sum(list_of_strings) is not largely fundamentally broken, but that's not
a promise and not a feature of the language.

IMHO.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/4b37a01b/attachment.html>

From ron3200 at gmail.com  Mon Aug 11 19:51:31 2014
From: ron3200 at gmail.com (Ron Adam)
Date: Mon, 11 Aug 2014 12:51:31 -0500
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
Message-ID: <lsavr4$t8u$1@ger.gmane.org>



On 08/10/2014 07:33 AM, Nick Coghlan wrote:
> FWIW, I don't consider str.join *or* sum with an empty string as the
> starting point to be particularly intuitive ways of joining iterables of
> strings.
>
> str.join was invented before we had keyword-only arguments as a common
> construct, and before print became an ordinary function that accepted a
> "sep" keyword-only argument.
>
> I'd be interested in seeing a concrete proposal for a "concat" builtin that
> accepted a "sep" keyword only argument. Even if such a PEP ends up being
> rejected, it would hopefully help cut short the *next* potentially
> interminable thread on the topic by gathering the arguments for and against
> in a more readily accessible place.

I think the contrast between the built in function "sum", and the string 
method "join", is very interesting from a language design point of view, 
but I'm finding it hard to describe just why in only a few words.  Others 
have pointed out some of the more detailed aspects of these issues, so here 
are some of the more general wider views I think come into play.


* Generality / Speciality

Their are advantages to both ends of this scale.  A more general function 
is very convenient, while a more specialised function can offer a greater 
degree of control.

* Complexity / Quantity

A function with a more complex signature can be equivalent to several 
functions with simpler signatures.  But as they become more complex, they 
also become more difficult to use.


It's not obvious how the "sum" function fits into these scales, and I 
believe that may be why it tends to come up as something that needs to be 
fixed frequently.  (how depends on the viewpoint of the fixer)  If "sum" 
was a method on numbers, it would clearly be more specialised, or if it was 
defined to call a method of what ever objects it was given, it would 
clearly be more general.

It is what it is, and I don't think there was any conscious consideration 
of these concepts when it was created.  Probably practicality over purity 
was more of a factor at the time.

I believe these concepts are more likely to be intuitively considered as 
the developers experiences increase instead of being consciously 
considered.  So they aren't formally defined in any documentation.



How a "concat" built in would relate to these concepts. Would "concat" be 
very general and delegate it's work to methods so it works on a variety of 
objects, or would it be limited to just strings?


I'm also concerned we may add a new function just to compliment the "sum" 
function, so that both of them look better.  I think any new function needs 
to fit into the language as a whole on it's own grounds with a clear 
propose and design.


Food for thought,
   Ron












































From me+python at ixokai.io  Mon Aug 11 19:52:31 2014
From: me+python at ixokai.io (Stephen Hansen)
Date: Mon, 11 Aug 2014 10:52:31 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAP7h-xbsKBJEhuwM+=udj8H-grXOFrdVgtzUnBy7zDKbahP-OA@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
 <CAP7h-xbsKBJEhuwM+=udj8H-grXOFrdVgtzUnBy7zDKbahP-OA@mail.gmail.com>
Message-ID: <CAM1gar6VM7xRgTNvS-br66P+RM5DrdX=dqE2v4zjHfDP4Mjddg@mail.gmail.com>

On Mon, Aug 11, 2014 at 10:14 AM, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:
>
>  it makes sense in your native language.
>

No, it makes no sense at all in my native language. sum is, fundamentally,
the total of numbers in my native language.


> When you are tasked with finding s1 + s2 + ... + sN given [s1, s2, ...,
> sN], it is sum that first comes to mind, not join.
>

It would have never occurred to me in a million years to address such a
problem with sum. The first thing that'd likely have come to my mind a
million years ago when I first started learning Python would be to look for
some concat function, but failing that I'd run across join in the docs.


>  The situation is different when you have a separator to begin with, but
> when you don't using an empty separator feels like a performance hack in
> the absence of an efficient natural solution.
>

I'm pretty much going to just bow out at this point because the idea that
"".join(list_of_strings) is some weird and non-obvious thing but when it
has a separator, its clear and obvious, doesn't make any sense at all to
me. That seems to fly directly in the face of "there should be one...."
which you previously quoted.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/028ea25a/attachment-0001.html>

From njs at pobox.com  Mon Aug 11 19:53:29 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 11 Aug 2014 18:53:29 +0100
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
Message-ID: <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>

On Mon, Aug 11, 2014 at 5:22 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
>
> On Mon, Aug 11, 2014 at 11:55 AM, Todd <toddrjen at gmail.com> wrote:
>>>
>>> In my experience, it is the asymmetry between x.join(y) and x.split(y)
>>> which causes most of the confusion.  In x.join(y), x is the separator and y
>>> is the data being joined, but in x.split(y), it is the other way around.
>>
>>
>> What would be the solution to this?
>
> Allow sum(list_of_strings, '') and stop mocking people who prefer it to
> ''.join(..).  This will not solve all the issues with join/split, but at
> least a simple task of concatenating a list of strings will have a more or
> less obvious solution.

I don't have any data here, but I bet people who know about str.join
(even for its natural use cases like ", ".join(...)) outnumber the
people who know that sum() takes a second argument by a very large
factor.

Of course this also means that sum()'s special error message is
probably pretty ineffective at reaching the people it's trying to
educate -- to do that we'd need to warn on str += str or something,
which is clearly not happening. So I can see the argument for just
making sum(iterable_of_strings, "") fast.

But practically speaking, how would this work? In general str.join and
sum have different semantics. What happens if we descend deep into the
iterable and then discover a non-string (that might nonetheless still
have a + operator)?

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

From haoyi.sg at gmail.com  Mon Aug 11 19:51:00 2014
From: haoyi.sg at gmail.com (Haoyi Li)
Date: Mon, 11 Aug 2014 10:51:00 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAM1gar6A=-h6qx4PXS8y90+XXOe7G77BCjoNjPK41uN+2V6YOQ@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
 <CALruUQL1dA5EcMd0_rfPjM1Y7Y8+hD8QUPzYrKvJbZz0jq0b5w@mail.gmail.com>
 <CAM1gar6A=-h6qx4PXS8y90+XXOe7G77BCjoNjPK41uN+2V6YOQ@mail.gmail.com>
Message-ID: <CALruUQL5oz7QjYKqzq3=eoUaeFUpuQku5BC9WjdT97ooz_eYTw@mail.gmail.com>

One reason I think this is confusing is that "list.join(sep)" is how
basically everyone else does it, including: Ruby, Javascript, Scala,
Guava-Java (normal java doesn't have anything). People who don't do it this
way include C# (static method), Boost-C++ (static method), PHP (static
method)

Nobody does it the Python way AFAICT.


On Mon, Aug 11, 2014 at 10:45 AM, Stephen Hansen <me+python at ixokai.io>
wrote:

> On Mon, Aug 11, 2014 at 10:10 AM, Haoyi Li <haoyi.sg at gmail.com> wrote:
>
>> > In fact, it makes very logical sense once you explain it and makes
>> people think of things more Pythonically after. I say from experience, not
>> in theory.
>>
>> Could you elaborate about "making people think more pythonically after"
>> bit? I can see how explaining the API makes people understand the API, but
>> I'm curious how it makes people behave differently after.
>>
>
> Well, its an opinion of course, so it may be as useful as the thought that
> the sky is high. It doesn't make people behave differently, but it does
> lead them to writing idiomatic code. There's naive Python code, which
> ideally should be clear and easy still, and then there's idiomatic Python
> code which has a beauty in its expressiveness while also being (relatively)
> efficient.
>
> But, if the opposite of "string.split(sep)" were
> "list_of_strings.join(sep)" then that would mean that List  would have to
> know about how to combine strings. That's not very Pythonic. Also, Strings
> are immutable, that means:
>
> str1 = str1 + str2
>
> logically leads one to assume that it is constructing an entirely new
> string, copying the contents of str1 and str2 into it, and discarding the
> original str1. In fact, it did that for a long time. You can't change
> strings, after all. That is an fundamental tenet, and when people learn
> that they will get a new tool and think closer to idiomatic Python. Now,
> CPython has an optimization for this case which is the only reason the idea
> of sum(list_of_strings) is not largely fundamentally broken, but that's not
> a promise and not a feature of the language.
>
> IMHO.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/39f299c4/attachment.html>

From tjreedy at udel.edu  Mon Aug 11 20:15:08 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 11 Aug 2014 14:15:08 -0400
Subject: [Python-ideas] float comparison in doctest
In-Reply-To: <53E8EE7B.50408@yahoo.com>
References: <53E8EE7B.50408@yahoo.com>
Message-ID: <lsb183$f7f$1@ger.gmane.org>

On 8/11/2014 12:25 PM, Kevin Davies wrote:
> Erik Bray has created a nifty addition to doctest to compare floating
> point numbers using a `+FLOAT_CMP` flag.

The problem with a simple flag is that almost_equal should sometimes be 
absolute and sometimes relative and in both cases a parameter in needed. 
I think there have been vague proposals for a float method, but
"a - b < delta" is shorter than "a.almost_equat(b, abs=delta)" and 
probably easier to understand.

 > It is implemented within
> Astropy (http://www.astropy.org/) and the related issue is
> https://github.com/astropy/astropy/issues/2662.

The related issue is applying flags to a block of test with a Sphinx 
directive.

> According to Erik, it
> may need some tweaks, but I think it would be a really useful feature to
> make available in the doctest package itself.

Erik would have to offer his code.

-- 
Terry Jan Reedy


From breamoreboy at yahoo.co.uk  Mon Aug 11 20:57:23 2014
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Mon, 11 Aug 2014 19:57:23 +0100
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CALruUQL5oz7QjYKqzq3=eoUaeFUpuQku5BC9WjdT97ooz_eYTw@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
 <CALruUQL1dA5EcMd0_rfPjM1Y7Y8+hD8QUPzYrKvJbZz0jq0b5w@mail.gmail.com>
 <CAM1gar6A=-h6qx4PXS8y90+XXOe7G77BCjoNjPK41uN+2V6YOQ@mail.gmail.com>
 <CALruUQL5oz7QjYKqzq3=eoUaeFUpuQku5BC9WjdT97ooz_eYTw@mail.gmail.com>
Message-ID: <lsb3mj$eo8$1@ger.gmane.org>

On 11/08/2014 18:51, Haoyi Li wrote:
> One reason I think this is confusing is that "list.join(sep)" is how
> basically everyone else does it, including: Ruby, Javascript, Scala,
> Guava-Java (normal java doesn't have anything). People who don't do it
> this way include C# (static method), Boost-C++ (static method), PHP
> (static method)
>
> Nobody does it the Python way AFAICT.
>

So one and only one language got it right and the rest got it wrong, 
what about it?

Now is it possible to leave this vitally important issue and move on to 
something trivial, e.g. helping people port Python 2 code to Python 3?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From tjreedy at udel.edu  Mon Aug 11 21:00:26 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 11 Aug 2014 15:00:26 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CAP7h-xbsKBJEhuwM+=udj8H-grXOFrdVgtzUnBy7zDKbahP-OA@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
 <CAP7h-xbsKBJEhuwM+=udj8H-grXOFrdVgtzUnBy7zDKbahP-OA@mail.gmail.com>
Message-ID: <lsb3t0$h3t$1@ger.gmane.org>

On 8/11/2014 1:14 PM, Alexander Belopolsky wrote:

> sep.join(list) is not such a weird construct when sep is non-empty - it
> is the sep='' case which is weird and non-obvious.  (Note that someone
> in this thread suggested demonstrating s == sep.join(s.split(sep))
> invariant as a teaching tool, but this invariant fails when sep is
> empty.)

Because s.split('') raises "ValueError: empty separator".  I expected 
the result should be the same as list(s), making the invariant above 
true.  But perhaps Guido thought splitting on '' might be a bug. 
re.split('', s) returns s, which seems wrong. The doc talks about 
'occurences of the pattern'.  I *see* an occurance of '' at every slice 
point.  In a sense, Python slicing does too.
 >>> 'abc'[1:1]
''

When I first learned Python, I knew to test edge case behavior rather 
than depend on my interpretation of docs, or worse, my expectations from 
previous experience and knowledge.  The interactive interpreter makes 
little tests like 'ab'split('') and re.split('', 'ab') trivial and 
faster than reading (or debugging).

-- 
Terry Jan Reedy


From abarnert at yahoo.com  Mon Aug 11 21:13:16 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 11 Aug 2014 12:13:16 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CAM1gar6A=-h6qx4PXS8y90+XXOe7G77BCjoNjPK41uN+2V6YOQ@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
 <CALruUQL1dA5EcMd0_rfPjM1Y7Y8+hD8QUPzYrKvJbZz0jq0b5w@mail.gmail.com>
 <CAM1gar6A=-h6qx4PXS8y90+XXOe7G77BCjoNjPK41uN+2V6YOQ@mail.gmail.com>
Message-ID: <75E96E84-7406-44B6-9436-1069F27DAF6A@yahoo.com>

On Aug 11, 2014, at 10:45, Stephen Hansen <me+python at ixokai.io> wrote:

> ... if the opposite of "string.split(sep)" were "list_of_strings.join(sep)" then that would mean that List  would have to know about how to combine strings. That's not very Pythonic.

I think the real problem here is that almost every OO language does it this way. Beginning programmers don't even know to look for a join function, so it doesn't matter where we put it. But someone coming to Python looking for the equivalent of -[NSArray componentsJoinedBySeparator:] or Enumerable#join or Sequence.joinStrings or whatever expects to find it as list.join. The fact that all of these other languages agreed on a stupid solution and Python on a more sensible one (in most of those languages, just as in Python, the enumeration/iteration/etc. protocol is a universal thing that every type already has to understand, while concatenating strings is something specific to strings) doesn't change the fact that they all agreed.

The question is whether this is more of a positive, as an opportunity to get people thinking Pythonically early instead of trying to write Ruby or C# code in Python, or a negative, as a stumbling block to them using the language.

Of course the obvious solution is to spell it like C++:

    stringstream ss;
    copy(a.begin(), a.end(), ostream_iterator(ss));
    s = ss.str();

Then nobody will find it at all, no matter what language they come from; problem solved. :)

> Also, Strings are immutable, that means:
> 
> str1 = str1 + str2
> 
> logically leads one to assume that it is constructing an entirely new string, copying the contents of str1 and str2 into it, and discarding the original str1. In fact, it did that for a long time. You can't change strings, after all. That is an fundamental tenet, and when people learn that they will get a new tool and think closer to idiomatic Python. Now, CPython has an optimization for this case which is the only reason the idea of sum(list_of_strings) is not largely fundamentally broken, but that's not a promise and not a feature of the language.
> 
> IMHO.
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From masklinn at masklinn.net  Mon Aug 11 21:29:01 2014
From: masklinn at masklinn.net (Masklinn)
Date: Mon, 11 Aug 2014 21:29:01 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CALruUQL5oz7QjYKqzq3=eoUaeFUpuQku5BC9WjdT97ooz_eYTw@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
 <CALruUQL1dA5EcMd0_rfPjM1Y7Y8+hD8QUPzYrKvJbZz0jq0b5w@mail.gmail.com>
 <CAM1gar6A=-h6qx4PXS8y90+XXOe7G77BCjoNjPK41uN+2V6YOQ@mail.gmail.com>
 <CALruUQL5oz7QjYKqzq3=eoUaeFUpuQku5BC9WjdT97ooz_eYTw@mail.gmail.com>
Message-ID: <BAD793C6-1078-451C-95E8-C6820DC74476@masklinn.net>

On 2014-08-11, at 19:51 , Haoyi Li <haoyi.sg at gmail.com> wrote:

> One reason I think this is confusing is that "list.join(sep)" is how basically everyone else does it, including: Ruby, Javascript, Scala, Guava-Java (normal java doesn't have anything). People who don't do it this way include C# (static method), Boost-C++ (static method), PHP (static method)
> 
> Nobody does it the Python way AFAICT. 

One could easily enough argue that MLs and Haskell do: although they
don't generally have objects (and thus no method), their signature is
(function name may vary)

   join :: String -> [String] -> String
   join separator list

which ? in haskell ? can be infixed to

    separator `intercalate` list

and which in both languages nicely lends itself to partial application
(much like Python's `sep.join` expression)

From wolfgang.maier at biologie.uni-freiburg.de  Mon Aug 11 21:38:51 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Mon, 11 Aug 2014 21:38:51 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CAP7h-xbsKBJEhuwM+=udj8H-grXOFrdVgtzUnBy7zDKbahP-OA@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
 <CAP7h-xbsKBJEhuwM+=udj8H-grXOFrdVgtzUnBy7zDKbahP-OA@mail.gmail.com>
Message-ID: <lsb64b$bqi$1@ger.gmane.org>

On 11.08.2014 19:14, Alexander Belopolsky wrote:
>
> sep.join(list) is not such a weird construct when sep is non-empty - it
> is the sep='' case which is weird and non-obvious.  (Note that someone
> in this thread suggested demonstrating s == sep.join(s.split(sep))
> invariant as a teaching tool, but this invariant fails when sep is
> empty.)

For the record, this doesn't fail because of any weirdness about 
''.join(s). It's just that s.split() does not take an empty string as 
separator. So, ok, I should have said:

s == sep.join(s.split(sep)) for any allowed sep (which should be an 
obvious requirement)

but this has nothing to do with the rest of the discussion other that it 
is a bit peculiar that join and sep do not act perfectly symmetrical.
On the other hand, a builtin function sum and a string method split 
would be alot more asymmetric.

> When you are tasked with finding s1 + s2 + ... + sN given [s1,
> s2, ..., sN], it is sum that first comes to mind, not join.

Not *my* first thought when it comes to strings, but if it is yours, 
then you try it once and you get an appropriate error message pointing 
you to the correct solution. Ok for me.

>  The situation is different when you have a separator to begin with, but when
> you don't using an empty separator feels like a performance hack in the
> absence of an efficient natural solution.
>



From tjreedy at udel.edu  Mon Aug 11 21:55:35 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 11 Aug 2014 15:55:35 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
Message-ID: <lsb74d$os7$1@ger.gmane.org>

On 8/11/2014 9:56 AM, Alexander Belopolsky wrote:
>
> On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier
> <wolfgang.maier at biologie.uni-freiburg.de
> <mailto:wolfgang.maier at biologie.uni-freiburg.de>>
> wrote:
>
>     I am using Python for teaching programming to absolute beginners at
>     university and, in my experience, joiner.join is never a big hurdle.
>
>
> In my experience, it is the asymmetry between x.join(y) and x.split(y)
> which causes most of the confusion.   In x.join(y), x is the separator

Given that the two parameters of join are a concrete string and the 
abstraction 'iterable of strings', join can only be a method of the joiner.

I would first teach ' '.join(it_of_strings) as the 'normal' case of 
joining 'words', along with print with the default sep = ' '.


> and y is the data being joined, but in x.split(y), it is the other way
> around.

*If* sep is present, then sep.split(string) would be possible. But when 
sep is *not* not present, split cannot be a method of something that is 
not there.  So I think I would teach s.split() first and then add 
.split(sep) and .splitlines().

I would also teach join and split together since they are, at their 
cores (excluding special cases), inverses.

-- 
Terry Jan Reedy


From wolfgang.maier at biologie.uni-freiburg.de  Mon Aug 11 22:17:50 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Mon, 11 Aug 2014 22:17:50 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <lsb74d$os7$1@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsb74d$os7$1@ger.gmane.org>
Message-ID: <53E924EE.40704@biologie.uni-freiburg.de>

On 11.08.2014 21:55, Terry Reedy wrote:
> On 8/11/2014 9:56 AM, Alexander Belopolsky wrote:
>>
>> On Mon, Aug 11, 2014 at 2:56 AM, Wolfgang Maier
>> <wolfgang.maier at biologie.uni-freiburg.de
>> <mailto:wolfgang.maier at biologie.uni-freiburg.de>>
>>
>> wrote:
>>
>>     I am using Python for teaching programming to absolute beginners at
>>     university and, in my experience, joiner.join is never a big hurdle.
>>
>>
>> In my experience, it is the asymmetry between x.join(y) and x.split(y)
>> which causes most of the confusion.   In x.join(y), x is the separator
>
> Given that the two parameters of join are a concrete string and the
> abstraction 'iterable of strings', join can only be a method of the joiner.
>
> I would first teach ' '.join(it_of_strings) as the 'normal' case of
> joining 'words', along with print with the default sep = ' '.
>
>
>> and y is the data being joined, but in x.split(y), it is the other way
>> around.
>
> *If* sep is present, then sep.split(string) would be possible. But when
> sep is *not* not present, split cannot be a method of something that is
> not there.  So I think I would teach s.split() first and then add
> .split(sep) and .splitlines().
>
> I would also teach join and split together since they are, at their
> cores (excluding special cases), inverses.
>

I like to show students early on that with Python they can do things 
very quickly that would be very hard to achieve manually. Most of my 
students are biologists so they do not think initially about theoretical 
aspects of programming much, but want to know whether they can do 
something with it fast.

For join/split, I typically use problems like:

- you'd like to use program xy to work on your data, but it expects 
input elements to be separated by semicolons when all you have is a 
tab-delimited format

or

- you have input consisting of numbers with ',' as the decimal separator 
(the default in German-speaking countries), but downstream software 
expects '.'

,i.e., they can all be solved with the general pattern:

s = new_sep.join(s.split(old_sep))

Seeing that, in Python, you can solve these problems (in principle) with 
one line of quite understandable code is a very convincing argument for 
starting to learn the language.

Wolfgang


From alexander.belopolsky at gmail.com  Mon Aug 11 22:34:38 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 11 Aug 2014 16:34:38 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
Message-ID: <CAP7h-xbwM3cYZT_vj5340TZdzEPmaqW4S1-iC7DZEt06iNLeQA@mail.gmail.com>

Getting back to the topic of this thread ...

On Sun, Aug 10, 2014 at 8:33 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> I'd be interested in seeing a concrete proposal for a "concat" builtin
> that accepted a "sep" keyword only argument.


I don't think concat() is the same operation as sep.join().  I view concat
as an operation that is naturally defined on a pair of sequences and can be
naturally (via reduce) extended to apply to an iterable of sequences:

z = concat(x, y) <=> len(z) = len(x) + len(y), z[i] = x[i] if i < len(x),
z[i] = y[i - len(x)] otherwise.

(Note that the definition works for any abstract sequence.)

I find it odd that although CPython defines concatenation as a part of the
sequence protocol at the implementation level, the abstract base class
Sequence does not.  This omission may present an opportunity to design
Sequence.__concat__ and builtin concat() so that concatenation of iterables
of sequences can be implemented efficiently in concrete types.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/6f072ced/attachment.html>

From abarnert at yahoo.com  Mon Aug 11 22:48:23 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 11 Aug 2014 13:48:23 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was
	Re:	sum(...) limitation)
In-Reply-To: <lsb74d$os7$1@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsb74d$os7$1@ger.gmane.org>
Message-ID: <1407790103.79044.YahooMailNeo@web181001.mail.ne1.yahoo.com>

On Monday, August 11, 2014 12:56 PM, Terry Reedy <tjreedy at udel.edu> wrote:



> Given that the two parameters of join are a concrete string and the 
> abstraction 'iterable of strings', join can only be a method of the 
> joiner.


In many languages (those with generic types), the notion of Sequence<String>, or S:Sequence where S.ElementType==String, or similar, makes sense.

The problem is that you can't define methods on specializations of types, only on the generic types. And Sequence.join makes no sense, because that implies the nonsensical Sequence<int>.join. (Neither of those applies to C++, where you can add methods to specializations, or add methods to the template and they just don't appear on specializations where they make no sense because of SFINAE. But I don't know of any other language that does things that way.)

But since most of those languages have leaky type systems despite most of them supposedly being statically strongly typed, it "works".

It's been fun to watch people try to do the same thing in Swift, which really is strongly typed, and will not let you define a Sequence.join method unless it compiles for any ElementType. Of course there's no such problem defining a String.join method whose parameter is S:Sequence where S.ElementType== String, so String.join works just fine. So anyone coming to Swift from Python can add a join method without even thinking twice, while anyone coming from any other language will fight with the compiler for a while, then give up and either copy to an ObjC NSArray types to escape the type system, or copy and paste their join loop all over their code.

But anyway, pointing out that every OO language gets this wrong may be entertaining, but it doesn't help the fact that everyone coming to Python from those other languages looks for it in the wrong place.


From ethan at stoneleaf.us  Mon Aug 11 23:10:31 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 11 Aug 2014 14:10:31 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
Message-ID: <53E93147.4050504@stoneleaf.us>

On 08/11/2014 10:53 AM, Nathaniel Smith wrote:
>
> But practically speaking, how would this work? In general str.join and
> sum have different semantics. What happens if we descend deep into the
> iterable and then discover a non-string (that might nonetheless still
> have a + operator)?

The same thing that happens now if you pass a list to join with a non-string entry:

--> ' '.join(['some', 'list', 'of', 'words', 'and', 10, 'as', 'a', 'number'])
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: sequence item 5: expected string, int found

--
~Ethan~

From tjreedy at udel.edu  Mon Aug 11 23:17:29 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 11 Aug 2014 17:17:29 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <75E96E84-7406-44B6-9436-1069F27DAF6A@yahoo.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
 <CALruUQL1dA5EcMd0_rfPjM1Y7Y8+hD8QUPzYrKvJbZz0jq0b5w@mail.gmail.com>
 <CAM1gar6A=-h6qx4PXS8y90+XXOe7G77BCjoNjPK41uN+2V6YOQ@mail.gmail.com>
 <75E96E84-7406-44B6-9436-1069F27DAF6A@yahoo.com>
Message-ID: <lsbbu0$kpd$1@ger.gmane.org>

On 8/11/2014 3:13 PM, Andrew Barnert wrote:

> Beginning programmers don't even know to look for a join function,

The rather complete index, where they will find it as a 
bytearray/bytes/str functions, as well as variations of thread.join.

> But someone coming to
> Python looking for the equivalent of -[NSArray
> componentsJoinedBySeparator:] or Enumerable#join or
> Sequence.joinStrings or whatever expects to find it as list.join.

When they don't find it, there is the index.

The first thing people should learn is that the Python doc set includes 
a tutorial for beginners, a Reference manual for syntax, a Library 
manual for objects, and a collective Index for all of these.  The Index 
starts with a Symbol page for looking up thing like '*' or '|=' that do 
not work in search engines.

-- 
Terry Jan Reedy


From ethan at stoneleaf.us  Mon Aug 11 23:32:10 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 11 Aug 2014 14:32:10 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <lsbbu0$kpd$1@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsalo0$rkd$2@ger.gmane.org>
 <CAP7h-xbGecRcGxxnLBGcqAxaaWjoA4fr3+A0hrs9_PE53FRbqA@mail.gmail.com>
 <CAM1gar4nPyRQJzVSHMfzroUBxph31L2HYuR7E1k_1cz_JLO3wg@mail.gmail.com>
 <CAP7h-xbQhMm0bxbsZr3RRNmbY=3yyt_UCjwLzLNU0nc9zLACSA@mail.gmail.com>
 <CAM1gar5uu-yOvhZuGRmyeNgtpV05ABxAZ6zbD_u9fAUv197AEg@mail.gmail.com>
 <CALruUQL1dA5EcMd0_rfPjM1Y7Y8+hD8QUPzYrKvJbZz0jq0b5w@mail.gmail.com>
 <CAM1gar6A=-h6qx4PXS8y90+XXOe7G77BCjoNjPK41uN+2V6YOQ@mail.gmail.com>
 <75E96E84-7406-44B6-9436-1069F27DAF6A@yahoo.com> <lsbbu0$kpd$1@ger.gmane.org>
Message-ID: <53E9365A.60207@stoneleaf.us>

On 08/11/2014 02:17 PM, Terry Reedy wrote:
>
> The first thing people should learn is that the Python doc set includes a tutorial for beginners, a Reference manual for
> syntax, a Library manual for objects, and a collective Index for all of these.  The Index starts with a Symbol page for
> looking up thing like '*' or '|=' that do not work in search engines.

+1

Before using any language one should familiarize oneself with it, and what better way than the tutorial that comes with 
the language?

(And no, I've never read it -- but I have read others, and several books, and a couple classes to fill in the chinks. ;)

--
~Ethan~

From ethan at stoneleaf.us  Mon Aug 11 23:35:50 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 11 Aug 2014 14:35:50 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
Message-ID: <53E93736.8030105@stoneleaf.us>

On 08/11/2014 02:25 PM, Nathaniel Smith wrote:
> On Mon, Aug 11, 2014 at 10:10 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
>> On 08/11/2014 10:53 AM, Nathaniel Smith wrote:
>>>
>>>
>>> But practically speaking, how would this work? In general str.join and
>>> sum have different semantics. What happens if we descend deep into the
>>> iterable and then discover a non-string (that might nonetheless still
>>> have a + operator)?
>>
>>
>> The same thing that happens now if you pass a list to join with a non-string
>> entry:
>>
>> --> ' '.join(['some', 'list', 'of', 'words', 'and', 10, 'as', 'a',
>> 'number'])
>> Traceback (most recent call last):
>>    File "<stdin>", line 1, in <module>
>> TypeError: sequence item 5: expected string, int found
>
> class Nasty:
>      def __radd__(self, other):
>          return other + "foo"
>
> "".join(["some", "strings", "and", "one", Nasty()])
> sum(["some", "strings", "and", "one", Nasty()], "")

Quite frankly, I regard this as a point in sum's favor.  We have, effectively, a string-subclass and join chokes on it.

--
~Ethan~

From njs at pobox.com  Mon Aug 11 23:25:04 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 11 Aug 2014 22:25:04 +0100
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <53E93147.4050504@stoneleaf.us>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
Message-ID: <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>

On Mon, Aug 11, 2014 at 10:10 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 08/11/2014 10:53 AM, Nathaniel Smith wrote:
>>
>>
>> But practically speaking, how would this work? In general str.join and
>> sum have different semantics. What happens if we descend deep into the
>> iterable and then discover a non-string (that might nonetheless still
>> have a + operator)?
>
>
> The same thing that happens now if you pass a list to join with a non-string
> entry:
>
> --> ' '.join(['some', 'list', 'of', 'words', 'and', 10, 'as', 'a',
> 'number'])
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: sequence item 5: expected string, int found

class Nasty:
    def __radd__(self, other):
        return other + "foo"

"".join(["some", "strings", "and", "one", Nasty()])
sum(["some", "strings", "and", "one", Nasty()], "")

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

From wolfgang.maier at biologie.uni-freiburg.de  Tue Aug 12 00:12:50 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Tue, 12 Aug 2014 00:12:50 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <53E93736.8030105@stoneleaf.us>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us>
Message-ID: <lsbf52$s9q$1@ger.gmane.org>

On 11.08.2014 23:35, Ethan Furman wrote:
>
>> class Nasty:
>>      def __radd__(self, other):
>>          return other + "foo"
>>
>> "".join(["some", "strings", "and", "one", Nasty()])
>> sum(["some", "strings", "and", "one", Nasty()], "")
>

Interesting. So a slight variation enables sum-based string 
concatenation *right now*:

class ZeroAdditionProperty:
     def __add__(self, other):
         return other

nullstr = ZeroAdditionProperty()

sum(["some", "strings", "and", "one"], nullstr)

Wolfgang


From ethan at stoneleaf.us  Tue Aug 12 00:13:40 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 11 Aug 2014 15:13:40 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAPJVwB=WGvkJ-mO-iv4krgkLCVKgVX=H1xBR4DAN-DYuiLR5kg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us>
 <CAPJVwB=WGvkJ-mO-iv4krgkLCVKgVX=H1xBR4DAN-DYuiLR5kg@mail.gmail.com>
Message-ID: <53E94014.6090008@stoneleaf.us>

On 08/11/2014 02:53 PM, Nathaniel Smith wrote:
>>
>> Quite frankly, I regard this as a point in sum's favor.  We have, effectively, a string-subclass and join chokes on it.
>
> Yes, but the proposal I was responding to was to do something like
>
> def sum(it, start=0):
>      if start == "":
>          return "".join(it)
>      ... regular sum impl here ...
>
> And the point is that this is not trivially a transparent optimization.

Ah, gotcha.

Even without an optimization in sum, I'd still rather it did what it said:  add things together.

--
~Ethan~

From tjreedy at udel.edu  Tue Aug 12 00:15:18 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 11 Aug 2014 18:15:18 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <53E93736.8030105@stoneleaf.us>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us>
Message-ID: <lsbfad$u8o$1@ger.gmane.org>

On 8/11/2014 5:35 PM, Ethan Furman wrote:
> On 08/11/2014 02:25 PM, Nathaniel Smith wrote:

>> class Nasty:
>>      def __radd__(self, other):
>>          return other + "foo"
>>
>> "".join(["some", "strings", "and", "one", Nasty()])
>> sum(["some", "strings", "and", "one", Nasty()], "")

I don't understand the point of this.

> Quite frankly, I regard this as a point in sum's favor.  We have,
> effectively, a string-subclass and join chokes on it.

Nasty is a subclass of object, with no default value.  Make it a real 
str subclass and join works fine.

class Nasty(str):
     def __radd__(self, other):
         return other + "foo"

print("".join(["some", "strings", "and", "one", Nasty()]))
 >>>
somestringsandone



-- 
Terry Jan Reedy


From wolfgang.maier at biologie.uni-freiburg.de  Tue Aug 12 00:23:11 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Tue, 12 Aug 2014 00:23:11 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <lsbfad$u8o$1@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us> <lsbfad$u8o$1@ger.gmane.org>
Message-ID: <lsbfof$4b3$1@ger.gmane.org>

On 12.08.2014 00:15, Terry Reedy wrote:
> On 8/11/2014 5:35 PM, Ethan Furman wrote:
>> On 08/11/2014 02:25 PM, Nathaniel Smith wrote:
>
>>> class Nasty:
>>>      def __radd__(self, other):
>>>          return other + "foo"
>>>
>>> "".join(["some", "strings", "and", "one", Nasty()])
>>> sum(["some", "strings", "and", "one", Nasty()], "")
>
> I don't understand the point of this.
>
>> Quite frankly, I regard this as a point in sum's favor.  We have,
>> effectively, a string-subclass and join chokes on it.
>
> Nasty is a subclass of object, with no default value.  Make it a real
> str subclass and join works fine.
>
> class Nasty(str):
>      def __radd__(self, other):
>          return other + "foo"
>
> print("".join(["some", "strings", "and", "one", Nasty()]))
>  >>>
> somestringsandone
>

No, it's not, at least not as intended or the result would be

somestringsandonefoo

The point about Ethan's example is that join only works with str and 
subclasses thereof, but not with proxy classes wrapping a str object.


From njs at pobox.com  Mon Aug 11 23:53:12 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 11 Aug 2014 22:53:12 +0100
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <53E93736.8030105@stoneleaf.us>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us>
Message-ID: <CAPJVwB=WGvkJ-mO-iv4krgkLCVKgVX=H1xBR4DAN-DYuiLR5kg@mail.gmail.com>

On 11 Aug 2014 22:36, "Ethan Furman" <ethan at stoneleaf.us> wrote:
>
> On 08/11/2014 02:25 PM, Nathaniel Smith wrote:
>>
>> On Mon, Aug 11, 2014 at 10:10 PM, Ethan Furman <ethan at stoneleaf.us>
wrote:
>>>
>>> On 08/11/2014 10:53 AM, Nathaniel Smith wrote:
>>>>
>>>>
>>>>
>>>> But practically speaking, how would this work? In general str.join and
>>>> sum have different semantics. What happens if we descend deep into the
>>>> iterable and then discover a non-string (that might nonetheless still
>>>> have a + operator)?
>>>
>>>
>>>
>>> The same thing that happens now if you pass a list to join with a
non-string
>>> entry:
>>>
>>> --> ' '.join(['some', 'list', 'of', 'words', 'and', 10, 'as', 'a',
>>> 'number'])
>>> Traceback (most recent call last):
>>>    File "<stdin>", line 1, in <module>
>>> TypeError: sequence item 5: expected string, int found
>>
>>
>> class Nasty:
>>      def __radd__(self, other):
>>          return other + "foo"
>>
>> "".join(["some", "strings", "and", "one", Nasty()])
>> sum(["some", "strings", "and", "one", Nasty()], "")
>
>
> Quite frankly, I regard this as a point in sum's favor.  We have,
effectively, a string-subclass and join chokes on it.

Yes, but the proposal I was responding to was to do something like

def sum(it, start=0):
    if start == "":
        return "".join(it)
    ... regular sum impl here ...

And the point is that this is not trivially a transparent optimization.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/e8e371c9/attachment-0001.html>

From python at 2sn.net  Tue Aug 12 00:56:27 2014
From: python at 2sn.net (Alexander Heger)
Date: Tue, 12 Aug 2014 08:56:27 +1000
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <lsal0e$iuj$1@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsaits$mje$1@ger.gmane.org> <lsal0e$iuj$1@ger.gmane.org>
Message-ID: <CAN3CYHwgOUaqqJjhw313uL5q42kKRsy8QcoWApT-Wz=eFAJmNg@mail.gmail.com>

> You are kidding, right? What will prevent them from using logic to recall
> the invariant mistakenly as
> s == sep.join(sep.split(s)) ? If only because there are natural use cases
> for passing mysep.join around as a callable that will join with a fixed
> separator, and there would similarly exist use cases for mysep.split, while
> there are really none for passing around mystring.split to split a fixed
> string by varying separators.
>
> This whole corner of python is a most ugly system of inconsistent acne scars
> in the 2.x series (dunno about 3.x). What with sum(...) pedantically telling
> you to use str.join - if it knows what you want, it should just to it.

I tend to agree...

From python at 2sn.net  Tue Aug 12 01:16:24 2014
From: python at 2sn.net (Alexander Heger)
Date: Tue, 12 Aug 2014 09:16:24 +1000
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <53E924EE.40704@biologie.uni-freiburg.de>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <lsb74d$os7$1@ger.gmane.org>
 <53E924EE.40704@biologie.uni-freiburg.de>
Message-ID: <CAN3CYHzVhH3LJrbo9Wx-sefLW7kToewYvw334MwFGWtv1JB9gQ@mail.gmail.com>

> - you have input consisting of numbers with ',' as the decimal separator
> (the default in German-speaking countries), but downstream software expects
> '.'
>
> ,i.e., they can all be solved with the general pattern:
>
> s = new_sep.join(s.split(old_sep))

In [1]: 'a,b'.replace(',','.')
Out[1]: 'a.b'

From tjreedy at udel.edu  Tue Aug 12 02:15:12 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 11 Aug 2014 20:15:12 -0400
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <lsbfof$4b3$1@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us> <lsbfad$u8o$1@ger.gmane.org>
 <lsbfof$4b3$1@ger.gmane.org>
Message-ID: <lsbmb7$9eq$1@ger.gmane.org>

On 8/11/2014 6:23 PM, Wolfgang Maier wrote:
> On 12.08.2014 00:15, Terry Reedy wrote:

>> Nasty is a subclass of object, with no default value.  Make it a real
>> str subclass and join works fine.
>>
>> class Nasty(str):
>>      def __radd__(self, other):
>>          return other + "foo"
>>
>> print("".join(["some", "strings", "and", "one", Nasty()]))

If one runs a program with a print statement from an Idle editor,

>>  >>>
>> somestringsandone

the shell prints the >>> prompt and then the output. I copy the prompt 
as a separator.

> No, it's not,

I have been copy and pasting python code and output for over 17 years. 
You could have verified that I did so accurately here in less than a 
minute.  I keep the Idle icon pinned to my task bar and usually have a 
scratch file for experiments in its recent files list, if not already 
open, to make this sort of thing easy.

 > at least not as intended

I was talking about reality, not intentions that were not clear to me.

-- 
Terry Jan Reedy


From ethan at stoneleaf.us  Tue Aug 12 05:03:21 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 11 Aug 2014 20:03:21 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <lsbmb7$9eq$1@ger.gmane.org>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us> <lsbfad$u8o$1@ger.gmane.org>
 <lsbfof$4b3$1@ger.gmane.org> <lsbmb7$9eq$1@ger.gmane.org>
Message-ID: <53E983F9.1090802@stoneleaf.us>

On 08/11/2014 05:15 PM, Terry Reedy wrote:
> On 8/11/2014 6:23 PM, Wolfgang Maier wrote:
>> On 12.08.2014 00:15, Terry Reedy wrote:
>
>>> Nasty is a subclass of object, with no default value.  Make it a real
>>> str subclass and join works fine.
>>>
>>> class Nasty(str):
>>>      def __radd__(self, other):
>>>          return other + "foo"
>>>
>>> print("".join(["some", "strings", "and", "one", Nasty()]))
>
> If one runs a program with a print statement from an Idle editor,
>
>>>  >>>
>>> somestringsandone
>
> the shell prints the >>> prompt and then the output. I copy the prompt as a separator.
>
>> No, it's not,
>
> I have been copy and pasting python code and output for over 17 years. You could have verified that I did so accurately
> here in less than a minute.  I keep the Idle icon pinned to my task bar and usually have a scratch file for experiments
> in its recent files list, if not already open, to make this sort of thing easy.
>
>> at least not as intended
>
> I was talking about reality, not intentions that were not clear to me.

His remark was pointed at the fact that your output is missing the final "foo".  Remove the 'r' from __radd__, though, 
and you would have what you were trying to demonstrate.

--
~Ethan~

From stephen at xemacs.org  Tue Aug 12 06:45:05 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 12 Aug 2014 13:45:05 +0900
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <53E94014.6090008@stoneleaf.us>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us>
 <CAPJVwB=WGvkJ-mO-iv4krgkLCVKgVX=H1xBR4DAN-DYuiLR5kg@mail.gmail.com>
 <53E94014.6090008@stoneleaf.us>
Message-ID: <87oavqiaem.fsf@uwakimon.sk.tsukuba.ac.jp>

Ethan Furman writes:

 > Even without an optimization in sum, I'd still rather it did what
 > it said: add things together.

But strings aren't additive, they're multiplicative:

>>> "x" "y"
'xy'
>>> 

This is a convention deeply embedded in mathematics notation
(multiplicative operators are noncommutative and frequently omitted in
notation, additive operators are commutative and explicit notation
required).

The point is not that it's *wrong* to think that "strings are
additive, so sum *should* apply to iterables of strings".  Rather,
it's that it's *not wrong* to think that "strings are non-additive and
therefore not sum()-able".  (I'm somewhere in between.)

Why choose '+' as the string multiplication operator, then?  Because
programming languages aren't as flexible as mathematics notation,
compromises are often made.  Among other things, '*' also has a
meaning for strings (when the other operand is an integer it means
repetition -- oddly enough, it's commutative!)  I suppose Guido could
have been a pure algebraist about it and chosen '*' and '^' for
concatenation and repetition, respectively, but that looks odd to me.

The lesson in the end is that although there are many functions that
are convenient to express in Python using operator notation, the
choice of operators should not be taken too seriously in deciding how
they will compose.


From stephen at xemacs.org  Tue Aug 12 07:51:19 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 12 Aug 2014 14:51:19 +0900
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
Message-ID: <87mwbai7c8.fsf@uwakimon.sk.tsukuba.ac.jp>

Nathaniel Smith writes:

 > I don't have any data here, but I bet people who know about str.join
 > (even for its natural use cases like ", ".join(...)) outnumber the
 > people who know that sum() takes a second argument by a very large
 > factor.

This is easy to fix, it now occurs to me.  Allow types with __add__ to
provide an optional __sum__ method, and give the numeric ABC a default
__sum__ implementation.  (It would be nice if it could check for
floats and restart with fsum if one is encountered.  And of course,
there may be other ABCs with __add__ that could get a default __sum__.)

Then sum could be just

    def sum(itr, start=0):
        if start = 0:
            itr = iter(itr)
            start = next(itr)
        return start.__sum__(itr)

and

    class str(...):

        def __sum__(self, itr):
            return self + ''.join(itr)    # probably can be optimized

Is it really it worth it, though?

 > But practically speaking, how would this work? In general str.join and
 > sum have different semantics.

sum(iter_of_str) currently doesn't have semantics.  The semantics
proponents of sum() seem to expect is precisely ''.join(iter_of_str).
Where's the problem?

 > What happens if we descend deep into the iterable and then discover
 > a non-string (that might nonetheless still have a + operator)?

We lose, er, an exception is raised.  Why is that a problem?  I think
most people who want a polymorphic sum() expect it to accept a
homogeneous iterable as the first argument.  I don't think they have
expectations that sum will be equivalent to

def new_sum(it, start=0):        # compatible signature ;-)
    it = iter(it)
    result = result or next(it)
    for x in it:
        result = result + next(it)
    return result

for heterogeneous iterables.  Among other things, how do you decide
the appropriate return type?  start's?  That of next(iter(it))?  The
"most important" of the types in it?  Ask for a BDFL pronouncement at
each invocation?

I suppose you could ask that functions that operate on iterables be
partially applicable in the sense that if they *do* raise on the
"wrong" type, the exception should provide a partial result, the
oddball operand, and an iterable containing the unconsumed operands
as attributes.  Then the __sum__ method could handle heterogeneous
operands if it wants to.

Note that partial_sum + oddball may have a different type from the
expected one even if it works.  This seems like a recipe for bugs to
me.  Are there use cases for such heterogenous sums?

The only exception that might be pretty safe would be a case where you
can coerce the oddball to the partial result's type.  But in the
salient case of str, pretty much every x has a str(x).  I don't think
that an optimized version of:

    def new_sum(iter, start):
        expected_type = type(start)
        result = start
        for x in iter:
            try:
                result = result + x
            except TypeError:
                result = result + expected_type(x)
        return result

is really what we want when type(start) == str, so it probably
shouldn't be default, and probably not when type(start) is numeric,
either.


From vito.detullio at gmail.com  Tue Aug 12 08:15:26 2014
From: vito.detullio at gmail.com (Vito De Tullio)
Date: Tue, 12 Aug 2014 08:15:26 +0200
Subject: [Python-ideas] float comparison in doctest
References: <53E8EE7B.50408@yahoo.com> <lsb183$f7f$1@ger.gmane.org>
Message-ID: <lscbe0$ba4$1@ger.gmane.org>

Terry Reedy wrote:

> I think there have been vague proposals for a float method, but
> "a - b < delta" is shorter than "a.almost_equat(b, abs=delta)" and
> probably easier to understand.

what if a < b? or if a or b (or both) are NaN/+Inf/-Inf? Or...

There are many pitfalls comparing floats, isn't better to delegate these 
checks in a function?

-- 
By ZeD


From stephen at xemacs.org  Tue Aug 12 08:29:20 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 12 Aug 2014 15:29:20 +0900
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <53E983F9.1090802@stoneleaf.us>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us> <lsbfad$u8o$1@ger.gmane.org>
 <lsbfof$4b3$1@ger.gmane.org> <lsbmb7$9eq$1@ger.gmane.org>
 <53E983F9.1090802@stoneleaf.us>
Message-ID: <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp>

Ethan Furman writes:

 > His remark was pointed at the fact that your output is missing the
 > final "foo".  Remove the 'r' from __radd__, though, and you would
 > have what you were trying to demonstrate.

Believe someone when he says he copy-pasted. ;-)  Unless you've
actually run the code and got a different result, and even then you
should probably include version information etc.

The presence of __radd__ (or __add__, for that matter, although it's
reasonably difficult to create a str derivative without it) is
irrelevant to why "".join works when Nasty is derived from str.
You're confusing the actual semantics of str.join (repeated copying at
appropriate offsets into a sufficiently large buffer) with the naive
implementation of sum (an iterated application of '+').


From daviesk24 at yahoo.com  Tue Aug 12 08:49:56 2014
From: daviesk24 at yahoo.com (Kevin Davies)
Date: Mon, 11 Aug 2014 20:49:56 -1000
Subject: [Python-ideas] float comparison in doctest
In-Reply-To: <lscbe0$ba4$1@ger.gmane.org>
References: <53E8EE7B.50408@yahoo.com> <lsb183$f7f$1@ger.gmane.org>
 <lscbe0$ba4$1@ger.gmane.org>
Message-ID: <53E9B914.1000702@yahoo.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140811/725cb403/attachment.html>

From wolfgang.maier at biologie.uni-freiburg.de  Tue Aug 12 09:16:09 2014
From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier)
Date: Tue, 12 Aug 2014 09:16:09 +0200
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us> <lsbfad$u8o$1@ger.gmane.org>
 <lsbfof$4b3$1@ger.gmane.org> <lsbmb7$9eq$1@ger.gmane.org>
 <53E983F9.1090802@stoneleaf.us> <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <53E9BF39.6060601@biologie.uni-freiburg.de>

On 08/12/2014 08:29 AM, Stephen J. Turnbull wrote:
> Ethan Furman writes:
>
>   > His remark was pointed at the fact that your output is missing the
>   > final "foo".  Remove the 'r' from __radd__, though, and you would
>   > have what you were trying to demonstrate.
>
> Believe someone when he says he copy-pasted. ;-)  Unless you've
> actually run the code and got a different result, and even then you
> should probably include version information etc.
>
> The presence of __radd__ (or __add__, for that matter, although it's
> reasonably difficult to create a str derivative without it) is
> irrelevant to why "".join works when Nasty is derived from str.
> You're confusing the actual semantics of str.join (repeated copying at
> appropriate offsets into a sufficiently large buffer) with the naive
> implementation of sum (an iterated application of '+').
>

Exactly. So my point was that when you don't subclass str, but instead 
use a wrapper around it, you can give it a as str-like interface as you 
want so the thing looks and feels like a string to users, it will still 
not work as part of an iterable passed to .join (because .join is C code 
moving around arrays of chars and is completely ignorant of all the nice 
methods you added to your object). Sum on the other hand knows how to 
use .__add__ and .__radd__ .
Not that I think this is an argument against str.join() - it is a 
specialized method to join strings efficiently and it's good at that (in 
fact, I think, that's why it makes complete sense to have it implemented 
as a str method because this and nothing else is the data type it can 
handle). I just found Ethan's Nasty() example interesting.

Wolfgang


From ncoghlan at gmail.com  Tue Aug 12 09:35:04 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 12 Aug 2014 17:35:04 +1000
Subject: [Python-ideas] float comparison in doctest
In-Reply-To: <lsb183$f7f$1@ger.gmane.org>
References: <53E8EE7B.50408@yahoo.com>
	<lsb183$f7f$1@ger.gmane.org>
Message-ID: <CADiSq7drM1jD=GLki_1H7f_BCg-4Tkknt5HLB_QYccSyMBd2uw@mail.gmail.com>

On 12 Aug 2014 04:16, "Terry Reedy" <tjreedy at udel.edu> wrote:
>
> On 8/11/2014 12:25 PM, Kevin Davies wrote:
>>
>> Erik Bray has created a nifty addition to doctest to compare floating
>> point numbers using a `+FLOAT_CMP` flag.
>
>
> The problem with a simple flag is that almost_equal should sometimes be
absolute and sometimes relative and in both cases a parameter in needed. I
think there have been vague proposals for a float method, but
> "a - b < delta" is shorter than "a.almost_equat(b, abs=delta)" and
probably easier to understand.

For more formal tests, we already have unittest.TestCase.assertAlmostEqual.

This sounds like a reasonable flag for doctest's original purpose of
checking that examples in docs are showing sensible answers.

Cheers,
Nick.

>
>
> > It is implemented within
>>
>> Astropy (http://www.astropy.org/) and the related issue is
>> https://github.com/astropy/astropy/issues/2662.
>
>
> The related issue is applying flags to a block of test with a Sphinx
directive.
>
>
>> According to Erik, it
>> may need some tweaks, but I think it would be a really useful feature to
>> make available in the doctest package itself.
>
>
> Erik would have to offer his code.
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140812/7719a4d9/attachment.html>

From ethan at stoneleaf.us  Tue Aug 12 17:24:27 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Aug 2014 08:24:27 -0700
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us> <lsbfad$u8o$1@ger.gmane.org>
 <lsbfof$4b3$1@ger.gmane.org> <lsbmb7$9eq$1@ger.gmane.org>
 <53E983F9.1090802@stoneleaf.us> <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <53EA31AB.7030200@stoneleaf.us>

On 08/11/2014 11:29 PM, Stephen J. Turnbull wrote:
> Ethan Furman writes:
>
>   > His remark was pointed at the fact that your output is missing the
>   > final "foo".  Remove the 'r' from __radd__, though, and you would
>   > have what you were trying to demonstrate.
>
> Believe someone when he says he copy-pasted. ;-)  Unless you've
> actually run the code and got a different result, and even then you
> should probably include version information etc.

I did believe -- I just also forgot that Python would use __radd__ in a subclass, and so thought that was a bug.  But 
you're right, I should have tried it myself before posting -- my apologies.

--
~Ethan~

From stephen at xemacs.org  Wed Aug 13 06:38:25 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 13 Aug 2014 13:38:25 +0900
Subject: [Python-ideas] The non-obvious nature of str.join (was
	Re:	sum(...) limitation)
In-Reply-To: <53E9BF39.6060601@biologie.uni-freiburg.de>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us> <lsbfad$u8o$1@ger.gmane.org>
 <lsbfof$4b3$1@ger.gmane.org> <lsbmb7$9eq$1@ger.gmane.org>
 <53E983F9.1090802@stoneleaf.us>
 <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E9BF39.6060601@biologie.uni-freiburg.de>
Message-ID: <87ha1hhum6.fsf@uwakimon.sk.tsukuba.ac.jp>

Wolfgang Maier writes:

 > Exactly. So my point was that when you don't subclass str, but instead 
 > use a wrapper around it, you can give it a as str-like interface as you 
 > want so the thing looks and feels like a string to users, it will still 
 > not work as part of an iterable passed to .join

You mean this behavior?

wideload:~ 12:42$ python3.2           
>>> 
... 
>>> class N:
...  def __init__(self, s=''):           
...   self.s = s
...  def __str__(self):
...   return self.s
... 
>>> " ".join(['a', N('b')])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: sequence item 1: expected str instance, N found
>>> ' '.join(str(x) for x in ['a', N('b')])
'a b'
>>> 

Given the fact that every object is str-able, I don't think we want to
give "str(x) for x in" semantics to str.join.  So I think the answer
is "if you want Nasty to automatically acquire all the behaviors of
str, make it a subclass of str".

I can't think of a use case where subclassing would be problematic.

 > Sum on the other hand knows how to use .__add__ and .__radd__ .

It seems to me that that's a strong argument against "summing strings"
with the current implementation of sum(), given the ease with which
you can construct types where the "sum" of an iterable can be
implemented efficiently and gives the same answer as the generic
algorithm based on '+', but the generic algorithm is inefficient (just
make it immutable).

I suppose most Sequence types are arrays of pointers at the C level,
or otherwise implement O(1) '+=', so either the join-style "just
memmove the arrays into a sufficiently large buffer", or iterated
'+=', does the trick for an efficient generic sum.

This just guesswork, though.



From ncoghlan at gmail.com  Wed Aug 13 07:50:13 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 13 Aug 2014 15:50:13 +1000
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
 sum(...) limitation)
In-Reply-To: <87ha1hhum6.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org>
 <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us> <lsbfad$u8o$1@ger.gmane.org>
 <lsbfof$4b3$1@ger.gmane.org> <lsbmb7$9eq$1@ger.gmane.org>
 <53E983F9.1090802@stoneleaf.us>
 <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E9BF39.6060601@biologie.uni-freiburg.de>
 <87ha1hhum6.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7daaYAhb9Cx777YKCT0enKw-inWBKWrOf9nyCTpF0Yb0g@mail.gmail.com>

On 13 August 2014 14:38, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Wolfgang Maier writes:
>
>  > Exactly. So my point was that when you don't subclass str, but instead
>  > use a wrapper around it, you can give it a as str-like interface as you
>  > want so the thing looks and feels like a string to users, it will still
>  > not work as part of an iterable passed to .join
>
> You mean this behavior?
>
> wideload:~ 12:42$ python3.2
>>>>
> ...
>>>> class N:
> ...  def __init__(self, s=''):
> ...   self.s = s
> ...  def __str__(self):
> ...   return self.s
> ...
>>>> " ".join(['a', N('b')])
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: sequence item 1: expected str instance, N found
>>>> ' '.join(str(x) for x in ['a', N('b')])
> 'a b'
>>>>
>
> Given the fact that every object is str-able, I don't think we want to
> give "str(x) for x in" semantics to str.join.  So I think the answer
> is "if you want Nasty to automatically acquire all the behaviors of
> str, make it a subclass of str".

Note that this is a general problem - it is quite common to use
explicit type checks against str rather than relying on ducktyping. In
theory, a suitable ABC could be defined (using collections.UserString
as a starting point), but nobody has ever found it a pressing enough
problem to take the time to do so - it's generally easier to just
inherit from str.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stephen at xemacs.org  Wed Aug 13 08:21:42 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 13 Aug 2014 15:21:42 +0900
Subject: [Python-ideas] [Python-Dev] sum(...) limitation
In-Reply-To: <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
References: <CALy7ODvEQOkhMR2TF8D4a8PJqZnnuWtHP-tLwRd4Yuh-wunedg@mail.gmail.com>
 <20140802203513.GA10447@k2>
 <CALGmxELhrqZVVdohueWp-EFk8W_BgB1oOXvKVwfbetzqaj3=8Q@mail.gmail.com>
 <20140804181013.GO4525@ando>
 <CALGmxE+Oi6JxYv0dLnpRPfiZ88vSzg9vF9xFFKa2XO0Hfv_A5Q@mail.gmail.com>
 <53E4055D.2040305@stoneleaf.us>
 <CALGmxE+mmNCYdPXS1DDwzUdKiwmUk0KfPfyfWfTA6eZ=NmFC7A@mail.gmail.com>
 <53E51269.5030209@stoneleaf.us>
 <7414D373-F598-4805-9DE8-F9779D08FEE8@gmail.com>
 <53E571B8.7030103@stoneleaf.us>
 <CAP7h-xa0-TDT-QqiC1ogjpkqrNBmY0Kn4g+afRJ17AHV3fPjbg@mail.gmail.com>
 <87fvh6jg1y.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAP7h-xZfJprehO18CJw1iwLSWy01zzgb=1dfywb4b8BhAyVK7Q@mail.gmail.com>
 <874mxkkb0f.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E7CBA4.40105@g.nevcal.com>
 <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp>
 <-2448384566377912251@unknownmsgid>
 <CADiSq7c8hur9Tggk7U-mn=hz5swrOdCHu+BYvzNC0Z74UVd4Hw@mail.gmail.com>
 <2076096455819154683@unknownmsgid>
 <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E991C9.7020404@stoneleaf.us>
 <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
Message-ID: <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp>

Redirecting to python-ideas, so trimming less than I might.

Chris Barker writes:
 > On Mon, Aug 11, 2014 at 11:07 PM, Stephen J. Turnbull <stephen at xemacs.org>
 > wrote:
 > 
 > > I'm referring to removing the unnecessary information that there's a
 > >  better way to do it, and simply raising an error (as in Python 3.2,
 > > say) which is all a RealProgrammer[tm] should ever need!
 > >
 > 
 > I can't imagine anyone is suggesting that -- disallow it, but don't tell
 > anyone why?

As I said, it's a regression.  That's exactly the behavior in Python 3.2.

 > The only thing that is remotely on the table here is:
 > 
 > 1) remove the special case for strings -- buyer beware -- but consistent
 > and less "ugly"

It's only consistent if you believe that Python has strict rules for
use of various operators.  It doesn't, except as far as they are
constrained by precedence.  For example, I have an application where I
add bytestrings bytewise modulo N <= 256, and concatenate them.  In
fact I use function call syntax, but the obvious operator syntax is
'+' for the bytewise addition, and '*' for the concatenation.

It's not in the Zen, but I believe in the maxim "If it's worth doing,
it's worth doing well."  So for me, 1) is out anyway.

 > 2) add a special case for strings that is fast and efficient -- may be as
 > simple as calling "".join() under the hood --no more code than the
 > exception check.

Sure, but what about all the other immutable containers with __add__
methods?  What about mappings with key-wise __add__ methods whose
values might be immutable but have __add__ methods?  Where do you stop
with the special-casing?  I consider this far more complex and ugly
than the simple "sum() is for numbers" rule (and even that is way too
complex considering accuracy of summing floats).

 > And I doubt anyone really is pushing for anything but (2)

I know that, but I think it's the wrong solution to the problem (which
is genuine IMO).  The right solution is something generic, possibly a
__sum__ method.  The question is whether that leads to too much work
to be worth it (eg, "homogeneous_iterable").

 > > Because obviously we'd want the attractive nuisance of "if you
 > > have __add__, there's a default definition of __sum__"
 > 
 > now I'm confused -- isn't that exactly what we have now?

Yes and my feeling (backed up by arguments that I admit may persuade
nobody but myself) is that what we have now kinda sucks[tm].  It
seemed like a good idea when I first saw it, but then, my apps don't
scale to where the pain starts in my own usage.

 > > It's possible that Python could provide some kind of feature that
 > > would allow an optimized sum function for every type that has
 > > __add__, but I think this will take a lot of thinking.
 > 
 > does it need to be every type? As it is the common ones work fine already
 > except for strings -- so if we add an optimized string sum() then we're
 > done.

I didn't say provide an optimized sum(), I said provide a feature
enabling people who want to optimize sum() to do so.  So yes, it needs
to be every type (the optional __sum__ method is a proof of concept,
modulo it actually being implementable ;-).

 > > *Somebody* will do it (I don't think anybody is +1 on restricting
 > > sum() to a subset of types with __add__).
 > 
 > uhm, that's exactly what we have now

Exactly.  Who's arguing that the sum() we have now is a ticket to
Paradise?  I'm just saying that there's probably somebody out there
negative enough on the current situation to come up with an answer
that I think is general enough (and I suspect that python-dev
consensus is that demanding, too).

 > sum() can be used for any type that has an __add__ defined.

I'd like to see that be mutable types with __iadd__.

 > What I fail to see is why it's better to raise an exception and
 > point users to a better way, than to simply provide an optimization
 > so that it's a mute issue.

Because inefficient sum() is an attractive nuisance, easy to overlook,
and likely to bite users other than the author.

 > The only justification offered here is that will teach people that summing
 > strings (and some other objects?)

Summing tuples works (with appropriate start=tuple()).  Haven't
benchmarked, but I bet that's O(N^2).

 > is order(N^2) and a bad idea. But:
 > 
 > a) Python's primary purpose is practical, not pedagogical (not that it
 > isn't great for that)

My argument is that in practical use sum() is a bad idea, period,
until you book up on the types and applications where it *does* work.
N.B. It doesn't even work properly for numbers (inaccurate for floats).

 > b) I doubt any naive users learn anything other than "I can't use sum() for
 > strings, I should use "".join()".

For people who think that special-casing strings is a good idea, I
think this is about as much benefit as you can expect.  Why go
farther?<0.5 wink/>

 > I submit that no naive user is going to get any closer to a proper
 > understanding of algorithmic Order behavior from this small hint. Which
 > leaves no reason to prefer an Exception to an optimization.

TOOWTDI.  str.join is in pretty much every code base by now, and
tutorials and FAQs recommending its user and severely deprecating sum
for strings are legion.

 > One other point: perhaps this will lead a naive user into thinking --
 > "sum() raises an exception if I try to use it inefficiently, so it must be
 > OK to use for anything that doesn't raise an exception" -- that would be a
 > bad lesson to mis-learn....

That assumes they know about the start argument.  I think most naive
users will just try to sum a bunch of tuples, and get the "can't add
0, tuple" Exception and write a loop.  I suspect that many of the
users who get the "use str.join" warning along with the Exception are
unaware of the start argument, too.  They expect sum(iter_of_str) to
magically add the strings.  Ie, when in 3.2 they got the
uninformative "can't add 0, str" message, they did not immediately go
"d'oh" and insert ", start=''" in the call to sum, they wrote a loop.

 > while we are at it, having the default sum() for floats be fsum()
 > would be nice

How do you propose to implement that, given math.fsum is perfectly
happy to sum integers?  You can't just check one or a few leading
elements for floatiness.  I think you have to dispatch on type(start),
but then sum(iter_of_floats) DTWT.  So I would suggest changing the
signature to sum(it, start=0.0).  This would probably be acceptable to
most users with iterables of ints, but does imply some performance hit.

 > This does turn sum() into a function that does type-based dispatch,
 > but isn't python full of those already? do something special for
 > the types you know about, call the generic dunder method for the
 > rest.

AFAIK Python is moving in the opposite direction: if there's a common
need for dispatching to type-specific implementations of a method,
define a standard (not "generic") dunder for the purpose, and have the
builtin (or operator, or whatever) look up (not "call") the
appropriate instance in the usual way, then call it.  If there's a
useful generic implementation, define an ABC to inherit from that
provides that generic implementation.


From castironpi at gmail.com  Wed Aug 13 14:21:16 2014
From: castironpi at gmail.com (Aaron Brady)
Date: Wed, 13 Aug 2014 07:21:16 -0500
Subject: [Python-ideas] Mutating while iterating
In-Reply-To: <CAEX1CPOxUiqcC860zxb9kyNTM6Dc6jVmqe0FE0G151GUF==vhQ@mail.gmail.com>
References: <cd149d5b-5b57-4fbe-a639-a7f13fae64cf@googlegroups.com>
 <CADiSq7cG0Cfp0AVn2xeVx-rRyqzeGUPV3XWmQQcXSeMa=02LPA@mail.gmail.com>
 <CAEX1CPOxUiqcC860zxb9kyNTM6Dc6jVmqe0FE0G151GUF==vhQ@mail.gmail.com>
Message-ID: <CAEX1CPNJBXnQn8UUe_ZObC2bNGb3_rFDpeuy-xSaOZtAA2W9fw@mail.gmail.com>

On Sun, Aug 3, 2014 at 12:46 PM, Aaron Brady <castironpi at gmail.com> wrote:
> On Sat, Jul 26, 2014 at 10:39 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
[snip]
>> Hi,
>>
>> This is clearly an issue of grave concern to you, but as Raymond
>> pointed out previously, you appear to have misunderstood the purpose
>> of those exceptions. They're there to prevent catastrophic failure of
>> the interpreter itself (i.e. segmentation faults), not to help find
>> bugs in user code. If users want to mutate containers while they're
>> iterating over them, they're generally free to do so. The only time
>> we'll actively disallow it is when such mutation will outright *break*
>> the iterator, rather than merely producing potentially surprising
>> results.
>>
>> I have closed the new issue and added a longer reply (with examples)
>> that will hopefully better explain why we have no intention of
>> changing this behaviour: http://bugs.python.org/issue22084#msg224100
>
>
> Python is replete with examples of prohibiting structures which are
> likely bugs but aren't segfaults.  There are also reciprocal-- though
> not necessarily inverse-- limitations of the unordered collections
> themselves ("set" and "dict").  The new behavior is transparent for
> the programmer; no possible programs can /rely/ on the existing
> behavior.  The new behavior introduces no new objects, possibly except
> "IterationError", no new syntax, and no new costs.
>
> I propose we leave this discussion thread open for the time being.  I
> also take the issue of /re/-assigning to keys during iteration to be
> settled as permitted.

Nick, It works by comparing the state of the container, to its state
when the iterator was opened.  We're ensuring it will always have a
unique state, up to comparison.  A state can be reused once no
iterators refer to it, hence needing the reference count.  A full
"object" is not needed for a memo, only the reference count, but the
"object" is easier and only twice the size, as "PyBaseObject Type" is
allocated anyway.

I'll point out that among the additional costs that there aren't,
garbage collection isn't any slower, as both "tp traverse" and "tp
clear" are empty in the "PyBaseObject Type" definition, on line
3511-3512 in "typeobject.c" at time of writing [1].

[1] http://svn.python.org/view/python/trunk/Objects/typeobject.c?revision=81744&view=markup#l3511

From erik.m.bray at gmail.com  Wed Aug 13 17:30:05 2014
From: erik.m.bray at gmail.com (Erik Bray)
Date: Wed, 13 Aug 2014 11:30:05 -0400
Subject: [Python-ideas] float comparison in doctest
Message-ID: <CAOTD34Y-bOXmn+NeSMesF5LXD7nbiqCMWqzup81wcTN4MiQybg@mail.gmail.com>

[Sorry for breaking the threading--turns out I didn't have a
python-ideas subscription from this address]

Just a few followup points I wanted to make to Kevin's post about a
proposed +FLOAT_CMP flag for doctest.  A better link for the
implementation is this PR in Astropy:

https://github.com/astropy/astropy/pull/2087

I can't take full credit either--most of the existing code was
borrowed (with some small improvements) from the SymPy project.  I had
started on a similar feature independently, but then borrowed their
implementation upon seeing that it was further along than mine.

On 12 Aug 2014 04:16, "Terry Reedy" <tjreedy at udel.edu> wrote:
> The problem with a simple flag is that almost_equal should sometimes be
> absolute and sometimes relative and in both cases a parameter in needed. I
> think there have been vague proposals for a float method, but "a - b < delta"
> is shorter than "a.almost_equat(b, abs=delta)" and probably easier to
> understand.

Indeed, that's what I had in mind when I told Kevin that there were
still issues with this.  There's no obvious way (I can think of at
least) to parameterize doctest flags.

However, the original immediate purpose of this feature was to handle
very small differences in representation of the same value between
different platforms, and in that respect it has worked very well.
Astropy has a lot of doctests, and many of which have floating point
outputs.  This +FLOAT_CMP flag enabled removing tons of ellipses from
the test outputs, and restoring the full outputs which certainly read
better in the docs.

For more complete unit tests of course we use assert_almost_equal type
functions.

That said, if anyone has any ideas for allowing tweaking the
tolerances for a doctest flag that would be great.  If this otherwise
seems like a good idea in general to include in the doctest module I
will offer a patch.

Erik

From steve at pearwood.info  Wed Aug 13 18:37:30 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 14 Aug 2014 02:37:30 +1000
Subject: [Python-ideas] sum(...) limitation
In-Reply-To: <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <53E7CBA4.40105@g.nevcal.com>
 <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp>
 <-2448384566377912251@unknownmsgid>
 <CADiSq7c8hur9Tggk7U-mn=hz5swrOdCHu+BYvzNC0Z74UVd4Hw@mail.gmail.com>
 <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
 <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20140813163729.GI4525@ando>

I'm going to cut straight to the chase here because this thread, and its 
related ones, on Python-Dev are giving me a headache and overloading my 
inbox. So I'm going to make a probably futile :-( attempt to cut off 
yet another huge thread before it starts by explaining why I think sum() 
ought to stay exactly as it is.

Built-in sum() is already quite complex. It has a fast path and a slow 
path, and that's just for numbers. While it's tempting to imagine sum() 
being even more clever and able to handle more cases, that increases the 
complexity and makes it more likely to end up slower rather than faster, 
or buggy, or both. Better to let the caller choose a specialist function 
(like numpy.array.sum, or math.fsum, or str.join) that handles the 
caller's specialist needs, than to try to make sum() master of 
everything. The more special cases sum() has, the more the pressure to 
add even more.

In the statistics module, I have a private _sum() function which tried 
really hard to deal with high-accuracy sums of mixed arbitrary numeric 
types without compromising too badly on speed, and it's much harder than 
it seems. Trying to handle non-numeric types too increases the 
complexity significantly. If you're smarter than me (I expect that many 
of you probably are) and believe that you can write a version of sum() 
which:

(1) is fast
(2) has better than O(N**2) performance
(3) is correct
(4) is accurate
(5) handles INFs and NANs (for those types which have them)
(6) handles mixed types (for those types which allow mixing)
(7) honours subclasses with custom __add__ and __radd__ methods
(8) and keeps the expected semantics that sum() is like repeated 
    addition (or concatenation)

and does so for *both* numeric and non-numeric cases (like strings, 
bytes, tuples, lists), then PLEASE write some code and publish it. I for 
one would love to see it or use it for the statistics module. But until 
you have tried writing such a thing, whether in C or Python, I think 
you're probably underestimating how hard it is and how fragile the 
result will be.

So, a plea: please stop trying to overloaded poor ol' built-in sum. 
sum() is *not* the One Obvious Way to add arbitrary objects in every 
domain. sum() is intended for simple cases of adding numbers, it is not 
intended as a specialist summation function for everything under the sun 
that can be added or concatenated.

A bit of history, as I remember it: sum() exists because for half of 
Python's lifetime, people were regularly defining this:

    def sum(numbers):
        return reduce(lambda a, b: a+b, numbers)

so they could easy add up a bunch of numbers:

    num_pages = sum([ch.pages() for ch in self.chapters])

sort of thing. Since this was a common need, it was decided to add it to 
the built-ins. But sum() was never intended to replace str.join or
list.extend, let alone even more exotic cases.

Built-in sum is aimed at sequences of numbers, not strings, lists, 
tuples, or Widgets for that matter. Perhaps giving it a start parameter 
was a mistake, but it is there and backwards compatibility says it isn't 
going to be removed. But that doesn't mean that the use of sum() on 
arbitrary types ought to be *encouraged*, even if it is *allowed*.

Conceptually, sum() is intended to behave like:

for value in sequence:
    start = start + value

That means calling custom __add__ or __radd__ methods if they exist. It 
also means that sum() cannot delegate to (say) str.join() without 
changing the semantics. Given:

class Special(str):
    def __radd__(self, other):
        print("I'm special!")
        return other + str(self)

s = Special('y')

the sum 'x' + s is *not* the same as ''.join(['x', s]). A similar 
argument applies to list.extend, and the source code in bltinmodule.c 
already makes that point.


Replying to a couple of points from Stephen:

On Wed, Aug 13, 2014 at 03:21:42PM +0900, Stephen J. Turnbull wrote:

>  > sum() can be used for any type that has an __add__ defined.
> 
> I'd like to see that be mutable types with __iadd__.

Surely you don't mean that. That would mean that sum([1, 2, 3]) would no 
longer work, since ints are not mutable types with __iadd__.


[...]
> Summing tuples works (with appropriate start=tuple()).  Haven't
> benchmarked, but I bet that's O(N^2).

Correct: increasing the number of tuples being added by a factor of 10 
requires almost a factor of 100 more time:

py> t = tuple([(i,) for i in range(1000)])
py> with Stopwatch():
...     _ = sum(t, ())
...
time taken: 0.003805 seconds
py> t *= 10
py> with Stopwatch():
...     _ = sum(t, ())
...
time taken: 0.230225 seconds
py> t *= 10
py> with Stopwatch():
...     _ = sum(t, ())
...
time taken: 32.206471 seconds


> My argument is that in practical use sum() is a bad idea, period,
> until you book up on the types and applications where it *does* work.
> N.B. It doesn't even work properly for numbers (inaccurate for floats).

sum() works fine for its intended uses, especially:

- summing ints exactly
- "low precision" sums of floats

I put "low precision" in scare quotes because, for many purposes, that 
precision is plenty high enough. For the use-case of adding together a 
dozen or a hundred positive floats of similar magnitude, sum() is fine. 
It's only advanced and high-precision uses where it falls short.

To put it another way: if you want to add the mass of Jupiter to the 
mass of a flea, you probably want math.fsum(). If you want to add the 
weight of an apple to the weight of a banana, both measured on a typical 
kitchen scale, sum() will do the job perfectly adequately.


>  > while we are at it, having the default sum() for floats be fsum()
>  > would be nice
> 
> How do you propose to implement that, given math.fsum is perfectly
> happy to sum integers?

And you really don't want sum(integers) to convert to float by default:

py> from math import fsum
py> fsum([10**30, 1234])
1e+30
py> sum([10**30, 1234])
1000000000000000000000000001234


Insisting that there ought to be one and only one way to sum up is, in 
my opinion, foolhardy, no matter how attractive it might seem. I believe 
that the right way is what Python already has: a simple sum() for simple 
cases, a few specialist sums (including ''.join) for the most common or 
important specialist cases, and leave the rest to third-party libraries 
or code you write yourself. That way the caller can then decide exactly 
what trade-offs between time, memory, convenience, accuracy and 
behaviour they wish, instead of invariably being surprised or 
disappointed by whatever trade-offs the one ?ber-sum() made.



-- 
Steven

From ckaynor at zindagigames.com  Wed Aug 13 18:59:52 2014
From: ckaynor at zindagigames.com (Chris Kaynor)
Date: Wed, 13 Aug 2014 09:59:52 -0700
Subject: [Python-ideas] sum(...) limitation
In-Reply-To: <20140813163729.GI4525@ando>
References: <53E7CBA4.40105@g.nevcal.com>
 <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp>
 <-2448384566377912251@unknownmsgid>
 <CADiSq7c8hur9Tggk7U-mn=hz5swrOdCHu+BYvzNC0Z74UVd4Hw@mail.gmail.com>
 <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
 <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando>
Message-ID: <CALvWhxs+MDW4Wa8HPZ1dCsxO_FkwKxhL6HmbSdNKzuYJFsG4Gw@mail.gmail.com>

On Wed, Aug 13, 2014 at 9:37 AM, Steven D'Aprano <steve at pearwood.info>
wrote:

> In the statistics module, I have a private _sum() function which tried
> really hard to deal with high-accuracy sums of mixed arbitrary numeric
> types without compromising too badly on speed, and it's much harder than
> it seems. Trying to handle non-numeric types too increases the
> complexity significantly. If you're smarter than me (I expect that many
> of you probably are) and believe that you can write a version of sum()
> which:
>
> (1) is fast
> (2) has better than O(N**2) performance
> (3) is correct
> (4) is accurate
> (5) handles INFs and NANs (for those types which have them)
> (6) handles mixed types (for those types which allow mixing)
> (7) honours subclasses with custom __add__ and __radd__ methods
> (8) and keeps the expected semantics that sum() is like repeated
>     addition (or concatenation)
>
> and does so for *both* numeric and non-numeric cases (like strings,
> bytes, tuples, lists), then PLEASE write some code and publish it. I for
> one would love to see it or use it for the statistics module. But until
> you have tried writing such a thing, whether in C or Python, I think
> you're probably underestimating how hard it is and how fragile the
> result will be.
>

The basic form I would use for an updated sum, which would often, but not
always, perform better would be something like (preferably, written in C,
not pure Python):

def sum(items, start=0):
    value = copy.copy(start) # Make sure that start is not mutated.
    for item in items:
        value += item
    return value

To avoid needing to access copy.copy, something like the following would
also work, but with a bit more local complexity:

def sum(items, start=0):
    count = len(items)
    if count > 0:
        value = start + items[0] # Make sure not to mutate start.
    else
        return start

    for i in range(1, count):
        value += items[i]
    return value

Either of these would likely perform better when summing items which
support an optimized __iadd__, which allows many types to perform better.
For those which do not support __iadd__, it would fallback to the slower
__add__ but still function. Note that the first version may be slower than
existing some in some cases, namely if copy.copy(start) is very slow for
the type. The second case should never be slower than the existing sum,
presuming __iadd__ is not implemented in a slower manner than __add__.

Due to the string add optimization CPython, this should make strings sum
better than O(N**2), many custom types will also sum faster, and, for all
well-behaved types, should produce the same result as the existing sum.

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/4ea2f96d/attachment-0001.html>

From steve at pearwood.info  Wed Aug 13 19:46:30 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 14 Aug 2014 03:46:30 +1000
Subject: [Python-ideas] sum(...) limitation
In-Reply-To: <CALvWhxs+MDW4Wa8HPZ1dCsxO_FkwKxhL6HmbSdNKzuYJFsG4Gw@mail.gmail.com>
References: <-2448384566377912251@unknownmsgid>
 <CADiSq7c8hur9Tggk7U-mn=hz5swrOdCHu+BYvzNC0Z74UVd4Hw@mail.gmail.com>
 <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
 <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando>
 <CALvWhxs+MDW4Wa8HPZ1dCsxO_FkwKxhL6HmbSdNKzuYJFsG4Gw@mail.gmail.com>
Message-ID: <20140813174630.GM4525@ando>

On Wed, Aug 13, 2014 at 09:59:52AM -0700, Chris Kaynor wrote:

> The basic form I would use for an updated sum, which would often, but not
> always, perform better would be something like (preferably, written in C,
> not pure Python):
> 
> def sum(items, start=0):
>     value = copy.copy(start) # Make sure that start is not mutated.
>     for item in items:
>         value += item
>     return value

That's a semantic change from the existing sum: start does not have 
to be copyable. Currently I can write this:

py> class A:
...     def __copy__(self):
...             raise TypeError("I am unique!")
...
py> a = A()
py> class B:
...     def __radd__(self, other):
...             return 23
...
py> b = B()
py> sum([b, 1000, 2000], a)
3023


By changing the semantics of sum() to require start to be copyable, 
your revision has just broken my application.

 
> To avoid needing to access copy.copy, something like the following would
> also work, but with a bit more local complexity:
> 
> def sum(items, start=0):
>     count = len(items)

That's another semantic change: sum() currently accepts iterators 
(although not *infinite* iterators). This revision has now broken tens 
of thousands of applications.



-- 
Steven

From antoine at python.org  Wed Aug 13 20:03:17 2014
From: antoine at python.org (Antoine Pitrou)
Date: Wed, 13 Aug 2014 14:03:17 -0400
Subject: [Python-ideas] sum(...) limitation
In-Reply-To: <20140813163729.GI4525@ando>
References: <53E7CBA4.40105@g.nevcal.com>
 <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp>
 <-2448384566377912251@unknownmsgid>
 <CADiSq7c8hur9Tggk7U-mn=hz5swrOdCHu+BYvzNC0Z74UVd4Hw@mail.gmail.com>
 <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
 <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando>
Message-ID: <lsg996$tua$1@ger.gmane.org>

Le 13/08/2014 12:37, Steven D'Aprano a ?crit :
>
> sum() is *not* the One Obvious Way to add arbitrary objects in every
> domain.

Well, it is. It's a builtin and it's called "sum", which makes it pretty 
clear it adds objects together. That could hardly be any more obvious, 
actually: all competitors are *less* obvious.

Indeed it cannot *practically* be the most efficient or accurate to sum 
arbitrary objects, unless we manage to devise a clever protocol 
(__sum__? how would that work?) that can delegate to arbitrary 
third-party code. But people are bound to think, legitimately, that 
built-in sum() is the logical answer when wanting to sum a sequence of 
objects.

(of course, whether or not "sum" is a reasonable notion when applied to 
strings - rather than lists or floats - is a separate question)

Regards

Antoine.



From ckaynor at zindagigames.com  Wed Aug 13 20:05:52 2014
From: ckaynor at zindagigames.com (Chris Kaynor)
Date: Wed, 13 Aug 2014 11:05:52 -0700
Subject: [Python-ideas] sum(...) limitation
In-Reply-To: <20140813174630.GM4525@ando>
References: <-2448384566377912251@unknownmsgid>
 <CADiSq7c8hur9Tggk7U-mn=hz5swrOdCHu+BYvzNC0Z74UVd4Hw@mail.gmail.com>
 <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
 <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando>
 <CALvWhxs+MDW4Wa8HPZ1dCsxO_FkwKxhL6HmbSdNKzuYJFsG4Gw@mail.gmail.com>
 <20140813174630.GM4525@ando>
Message-ID: <CALvWhxsas8cK20n4eK32aoSSEbh2ZaaJB8A+uFJ3za7JTdpB2g@mail.gmail.com>

On Wed, Aug 13, 2014 at 10:46 AM, Steven D'Aprano <steve at pearwood.info>
wrote:

> That's another semantic change: sum() currently accepts iterators
> (although not *infinite* iterators). This revision has now broken tens
> of thousands of applications.
>

okay, how about:

def sum(items, start=0):
    first = True
    for item in items:
        if first:
            start = start + item
        else:
            start += item
    return start

I believe that has the same effect as my other two, but only adds the
requirement that "value += item" behaves the same as "value = value + item".

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/f6cabd0c/attachment.html>

From alexander.belopolsky at gmail.com  Wed Aug 13 20:29:29 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 13 Aug 2014 14:29:29 -0400
Subject: [Python-ideas] sum(...) limitation
In-Reply-To: <lsg996$tua$1@ger.gmane.org>
References: <53E7CBA4.40105@g.nevcal.com>
 <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp>
 <-2448384566377912251@unknownmsgid>
 <CADiSq7c8hur9Tggk7U-mn=hz5swrOdCHu+BYvzNC0Z74UVd4Hw@mail.gmail.com>
 <2076096455819154683@unknownmsgid>
 <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E991C9.7020404@stoneleaf.us>
 <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
 <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20140813163729.GI4525@ando> <lsg996$tua$1@ger.gmane.org>
Message-ID: <CAP7h-xZuSB6ZGi2ubf=cxonRWfbDa0VgMUZ2m=jdzJg9Snvh-g@mail.gmail.com>

On Wed, Aug 13, 2014 at 2:03 PM, Antoine Pitrou <antoine at python.org> wrote:

> Le 13/08/2014 12:37, Steven D'Aprano a ?crit :
>
>
>> sum() is *not* the One Obvious Way to add arbitrary objects in every
>> domain.
>>
>
> Well, it is. It's a builtin and it's called "sum", which makes it pretty
> clear it adds objects together. That could hardly be any more obvious,
> actually: all competitors are *less* obvious.
>

When it comes to "adding" sequences or containers, "concat" would be a
strong competitor.  After all, this is exactly what + does on sequences
under the hood. [1]

While there are technical reasons for reusing the same infix operator for
adding and concatenation, we are not so restricted when it comes to
builtins.

[1] https://docs.python.org/2/c-api/sequence.html#PySequence_Concat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/f2c6dd3c/attachment.html>

From mertz at gnosis.cx  Wed Aug 13 21:04:25 2014
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 13 Aug 2014 12:04:25 -0700
Subject: [Python-ideas] sum(...) limitation
In-Reply-To: <CALvWhxsas8cK20n4eK32aoSSEbh2ZaaJB8A+uFJ3za7JTdpB2g@mail.gmail.com>
References: <-2448384566377912251@unknownmsgid>
 <CADiSq7c8hur9Tggk7U-mn=hz5swrOdCHu+BYvzNC0Z74UVd4Hw@mail.gmail.com>
 <2076096455819154683@unknownmsgid>
 <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E991C9.7020404@stoneleaf.us>
 <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
 <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20140813163729.GI4525@ando>
 <CALvWhxs+MDW4Wa8HPZ1dCsxO_FkwKxhL6HmbSdNKzuYJFsG4Gw@mail.gmail.com>
 <20140813174630.GM4525@ando>
 <CALvWhxsas8cK20n4eK32aoSSEbh2ZaaJB8A+uFJ3za7JTdpB2g@mail.gmail.com>
Message-ID: <CAEbHw4b8cpC7JPMRDvaP3o2CNOUTQQsfgMFbS=zZV8-1j3jghg@mail.gmail.com>

This isn't the first time around in this discussion of (abusing) 'sum()'.
 Actually, I discussed the last round in my keynote at PyCon UK in 2013.
 In particular, I actually discussed there already how Chris' latest
version of 'sum()' breaks *existing* and *widespread* code.

My slides are at:

     http://gnosis.cx/pycon-uk-2013/Keynote-Ideas.pdf

Look at slides 24-26.  The TL;DR version: numpy arrays give different
meanings to __add__() and __iadd__().  Chris' version breaks every program
ever written that uses numpy.

On Wed, Aug 13, 2014 at 11:05 AM, Chris Kaynor <ckaynor at zindagigames.com>
wrote:

>
> On Wed, Aug 13, 2014 at 10:46 AM, Steven D'Aprano <steve at pearwood.info>
> wrote:
>
>> That's another semantic change: sum() currently accepts iterators
>> (although not *infinite* iterators). This revision has now broken tens
>> of thousands of applications.
>>
>
> okay, how about:
>
> def sum(items, start=0):
>     first = True
>     for item in items:
>         if first:
>             start = start + item
>         else:
>             start += item
>     return start
>
> I believe that has the same effect as my other two, but only adds the
> requirement that "value += item" behaves the same as "value = value + item".
>
> Chris
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/00721765/attachment-0001.html>

From abarnert at yahoo.com  Wed Aug 13 21:12:50 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 13 Aug 2014 12:12:50 -0700
Subject: [Python-ideas] sum(...) limitation
In-Reply-To: <lsg996$tua$1@ger.gmane.org>
References: <53E7CBA4.40105@g.nevcal.com>
 <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp>
 <-2448384566377912251@unknownmsgid>
 <CADiSq7c8hur9Tggk7U-mn=hz5swrOdCHu+BYvzNC0Z74UVd4Hw@mail.gmail.com>
 <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
 <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando>
 <lsg996$tua$1@ger.gmane.org>
Message-ID: <7FDFE116-9CDB-4EA5-842E-4356BF8DF217@yahoo.com>

On Aug 13, 2014, at 11:03, Antoine Pitrou <antoine at python.org> wrote:

> Well, it is. It's a builtin and it's called "sum", which makes it pretty clear it adds objects together. That could hardly be any more obvious, actually: all competitors are *less* obvious.

Have you ever heard of anyone wanting to "sum" a bunch of pieces of text? Or a bunch of lists?

Actually, now that I think about it, although I was being somewhat flippant, I've heard both of those things plenty of times, and never once has it meant concatenation. It always means literal or figurative element-wise summing, or some other kind of aggregation. Which makes it even _less_ obvious as a one true way to concatenate a bunch of things.

Also, how often do you really want to concatenate a bunch of lists into a bigger list, as opposed to just chaining them together into a new iterable? It's not _never_, but I think it's less common. So if itertools.chain.from_iterable doesn't deserve to be a builtin (or even a top-level module function), is concat, a modified sum, or some other way to spell the same idea really necessary?

From marky1991 at gmail.com  Wed Aug 13 21:28:02 2014
From: marky1991 at gmail.com (Mark Young)
Date: Wed, 13 Aug 2014 15:28:02 -0400
Subject: [Python-ideas] sum(...) limitation
In-Reply-To: <7FDFE116-9CDB-4EA5-842E-4356BF8DF217@yahoo.com>
References: <53E7CBA4.40105@g.nevcal.com>
 <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp>
 <-2448384566377912251@unknownmsgid>
 <CADiSq7c8hur9Tggk7U-mn=hz5swrOdCHu+BYvzNC0Z74UVd4Hw@mail.gmail.com>
 <2076096455819154683@unknownmsgid> <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E991C9.7020404@stoneleaf.us> <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
 <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp> <20140813163729.GI4525@ando>
 <lsg996$tua$1@ger.gmane.org> <7FDFE116-9CDB-4EA5-842E-4356BF8DF217@yahoo.com>
Message-ID: <CAG3cHaZdyUZ2r0ig0ACfjP7iineW=QVPrnDk74JLCWgWh1CQBg@mail.gmail.com>

Have you ever heard someone wanting to "add" a bunch of pieces of text?
Outside of programming, of course not. In our domain, adding strings makes
perfect sense (it's concatenation), thus the idea of summing strings makes
perfect sense, given that summation is defined as repeated addition. The
same argument applies for lists.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/e2b50146/attachment.html>

From guido at python.org  Wed Aug 13 21:44:21 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 12:44:21 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function annotations
Message-ID: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>

[There is no TL;DR other than the subject line. Please read the whole thing
before replying. I do have an appendix with some motivations for adding
type annotations at the end.]

Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man
of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy:
http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can
learn from Haskell (and other languages); yesterday he gave the same talk
at Dropbox. The talk is online (
https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad
strokes comes down to three suggestions:

  (a) Python should adopt mypy's syntax for function annotations
  (b) Python's use of mutabe containers by default is wrong
  (c) Python should adopt some kind of Abstract Data Types

Proposals (b) and (c) don't feel particularly actionable (if you disagree
please start a new thread, I'd be happy to discuss these further if there's
interest) but proposal (a) feels right to me.

So what is mypy?  It is a static type checker for Python written by Jukka
for his Ph.D. thesis. The basic idea is that you add type annotations to
your program using some custom syntax, and when running your program using
the mypy interpreter, type errors will be found during compilation (i.e.,
before the program starts running).

The clever thing here is that the custom syntax is actually valid Python 3,
using (mostly) function annotations: your annotated program will still run
with the regular Python 3 interpreter. In the latter case there will be no
type checking, and no runtime overhead, except to evaluate the function
annotations (which are evaluated at function definition time but don't have
any effect when the function is called).

In fact, it is probably more useful to think of mypy as a heavy-duty linter
than as a compiler or interpreter; leave the type checking to mypy, and the
execution to Python. It is easy to integrate mypy into a continuous
integration setup, for example.

To read up on mypy's annotation syntax, please see the mypy-lang.org
website. Here's just one complete example, to give a flavor:

  from typing import List, Dict

  def word_count(input: List[str]) -> Dict[str, int]:
      result = {}  #type: Dict[str, int]
      for line in input:
          for word in line.split():
              result[word] = result.get(word, 0) + 1
      return result

Note that the #type: comment is part of the mypy syntax; mypy uses comments
to declare types in situations where no syntax is available -- although
this particular line could also be written as follows:

    result = Dict[str, int]()

Either way the entire function is syntactically valid Python 3, and a
suitable implementation of typing.py (containing class definitions for List
and Dict, for example) can be written to make the program run correctly.
One is provided as part of the mypy project.

I should add that many of mypy's syntactic choices aren't actually new. The
basis of many of its ideas go back at least a decade: I blogged about this
topic in 2004 (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 --
see also the two followup posts linked from the top there).

I'll emphasize once more that mypy's type checking happens in a separate
pass: no type checking happens at run time (other than what the interpreter
already does, like raising TypeError on expressions like 1+"1").

There's a lot to this proposal, but I think it's possible to get a PEP
written, accepted and implemented in time for Python 3.5, if people are
supportive. I'll go briefly over some of the action items.

*(1) A change of direction for function annotations*

PEP 3107 <http://legacy.python.org/dev/peps/pep-3107/>, which introduced
function annotations, is intentional non-committal about how function
annotations should be used. It lists a number of use cases, including but
not limited to type checking. It also mentions some rejected proposals that
would have standardized either a syntax for indicating types and/or a way
for multiple frameworks to attach different annotations to the same
function. AFAIK in practice there is little use of function annotations in
mainstream code, and I propose a conscious change of course here by stating
that annotations should be used to indicate types and to propose a standard
notation for them.

(We may have to have some backwards compatibility provision to avoid
breaking code that currently uses annotations for some other purpose.
Fortunately the only issue, at least initially, will be that when running
mypy to type check such code it will produce complaints about the
annotations; it will not affect how such code is executed by the Python
interpreter. Nevertheless, it would be good to deprecate such alternative
uses of annotations.)

*(2) A specification for what to add to Python 3.5*

There needs to be at least a rough consensus on the syntax for annotations,
and the syntax must cover a large enough set of use cases to be useful.
Mypy is still under development, and some of its features are still
evolving (e.g. unions were only added a few weeks ago). It would be
possible to argue endlessly about details of the notation, e.g. whether to
use 'list' or 'List', what either of those means (is a duck-typed list-like
type acceptable?) or how to declare and use type variables, and what to do
with functions that have no annotations at all (mypy currently skips those
completely).

I am proposing that we adopt whatever mypy uses here, keeping discussion of
the details (mostly) out of the PEP. The goal is to make it possible to add
type checking annotations to 3rd party modules (and even to the stdlib)
while allowing unaltered execution of the program by the (unmodified)
Python 3.5 interpreter. The actual type checker will not be integrated with
the Python interpreter, and it will not be checked into the CPython
repository. The only thing that needs to be added to the stdlib is a copy
of mypy's typing.py module. This module defines several dozen new classes
(and a few decorators and other helpers) that can be used in expressing
argument types. If you want to type-check your code you have to download
and install mypy and run it separately.

The curious thing here is that while standardizing a syntax for type
annotations, we technically still won't be adopting standard rules for type
checking. This is intentional. First of all, fully specifying all the type
checking rules would make for a really long and boring PEP (a much better
specification would probably be the mypy source code). Second, I think it's
fine if the type checking algorithm evolves over time, or if variations
emerge. The worst that can happen is that you consider your code correct
but mypy disagrees; your code will still run.

That said, I don't want to *completely* leave out any specification. I want
the contents of the typing.py module to be specified in the PEP, so that it
can be used with confidence. But whether mypy will complain about your
particular form of duck typing doesn't have to be specified by the PEP.
Perhaps as mypy evolves it will take options to tell it how to handle
certain edge cases. Forks of mypy (or entirely different implementations of
type checking based on the same annotation syntax) are also a possibility.
Maybe in the distant future a version of Python will take a different
stance, once we have more experience with how this works out in practice,
but for Python 3.5 I want to restrict the scope of the upheaval.


*Appendix -- Why Add Type Annotations?*
The argument between proponents of static typing and dynamic typing has
been going on for many decades. Neither side is all wrong or all right.
Python has traditionally fallen in the camp of extremely dynamic typing,
and this has worked well for most users, but there are definitely some
areas where adding type annotations would help.

- Editors (IDEs) can benefit from type annotations; they can call out
obvious mistakes (like misspelled method names or inapplicable operations)
and suggest possible method names. Anyone who has used IntelliJ or Xcode
will recognize how powerful these features are, and type annotations will
make such features more useful when editing Python source code.

- Linters are an important tool for teams developing software. A linter
doesn't replace a unittest, but can find certain types of errors better or
quicker. The kind of type checking offered by mypy works much like a
linter, and has similar benefits; but it can find problems that are beyond
the capabilities of most linters.

- Type annotations are useful for the human reader as well! Take the above
word_count() example. How long would it have taken you to figure out the
types of the argument and return value without annotations? Currently most
people put the types in their docstrings; developing a standard notation
for type annotations will reduce the amount of documentation that needs to
be written, and running the type checker might find bugs in the
documentation, too. Once a standard type annotation syntax is introduced,
it should be simple to add support for this notation to documentation
generators like Sphinx.

- Refactoring. Bob's talk has a convincing example of how type annotations
help in (manually) refactoring code. I also expect that certain automatic
refactorings will benefit from type annotations -- imagine a tool like 2to3
(but used for some other transformation) augmented by type annotations, so
it will know whether e.g. x.keys() is referring to the keys of a dictionary
or not.

- Optimizers. I believe this is actually the least important application,
certainly initially. Optimizers like PyPy or Pyston
<https://github.com/dropbox/pyston> wouldn't be able to fully trust the
type annotations, and they are better off using their current strategy of
optimizing code based on the types actually observed at run time. But it's
certainly feasible to imagine a future optimizer also taking type
annotations into account.

-- 
--Guido "I need a new hobby" van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/2b249ded/attachment-0001.html>

From ethan at stoneleaf.us  Wed Aug 13 21:59:54 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 13 Aug 2014 12:59:54 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <53EBC3BA.10808@stoneleaf.us>

On 08/13/2014 12:44 PM, Guido van Rossum wrote:
>
> [There is no TL;DR other than the subject line. Please read the whole thing before replying. I do have an appendix with
> some motivations for adding type annotations at the end.]

+0 on the proposal as a whole.  It is not something I'm likely to use, but I'm not opposed to it, so long as it stays 
optional.


> Nevertheless, it would be good to deprecate such alternative uses of annotations.

-1 on deprecating alternative uses of annotations.

--
~Ethan~

From raymond.hettinger at gmail.com  Wed Aug 13 22:18:18 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 13 Aug 2014 13:18:18 -0700
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
In-Reply-To: <mailman.72122.1407959084.18129.python-ideas@python.org>
References: <mailman.72122.1407959084.18129.python-ideas@python.org>
Message-ID: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com>


On Aug 13, 2014, at 12:44 PM, python-ideas-request at python.org wrote:

>  The goal is to make it possible to add
> type checking annotations to 3rd party modules (and even to the stdlib)
> while allowing unaltered execution of the program by the (unmodified)
> Python 3.5 interpreter.

Is the goal to "make it possible" or would it quickly become required
(i.e. any time you write normal, readable Python, it would break 
someone's optimizer, refactorer, linter, etc.?)

Are these annotations going to be pushed into every nook and cranny
in the standard library and wend their way into the C code?


Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/6f664cd8/attachment.html>

From guido at python.org  Wed Aug 13 22:19:01 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 13:19:01 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53EBC3BA.10808@stoneleaf.us>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBC3BA.10808@stoneleaf.us>
Message-ID: <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>

On Wed, Aug 13, 2014 at 12:59 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 08/13/2014 12:44 PM, Guido van Rossum wrote:
>
>>
>> [There is no TL;DR other than the subject line. Please read the whole
>> thing before replying. I do have an appendix with
>> some motivations for adding type annotations at the end.]
>>
>
> +0 on the proposal as a whole.  It is not something I'm likely to use, but
> I'm not opposed to it, so long as it stays optional.
>
>
>
>  Nevertheless, it would be good to deprecate such alternative uses of
>> annotations.
>>
>
> -1 on deprecating alternative uses of annotations.
>

Do you have a favorite alternative annotation use that you actually use (or
are likely to)?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/bd407687/attachment.html>

From ron3200 at gmail.com  Wed Aug 13 22:19:48 2014
From: ron3200 at gmail.com (Ron Adam)
Date: Wed, 13 Aug 2014 15:19:48 -0500
Subject: [Python-ideas] The non-obvious nature of str.join (was Re:
	sum(...) limitation)
In-Reply-To: <CADiSq7daaYAhb9Cx777YKCT0enKw-inWBKWrOf9nyCTpF0Yb0g@mail.gmail.com>
References: <CADiSq7f5Ay0XGMy5F752yyr_Rj-y8iXJUs8FbKkq=sjAWf7nfg@mail.gmail.com>
 <ls8den$d7t$1@ger.gmane.org> <87vbpzj2qi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <ls9pcc$vqs$1@ger.gmane.org>
 <CAP7h-xZ+=sE9GAiQGLw4BtNu_WSWJ9csHuCbCoWokJqqgt_K5Q@mail.gmail.com>
 <CAFpSVpK_vYvQRJDb3o7PPUm-0pAYaSgABKth8-3y8JtaKLptBA@mail.gmail.com>
 <CAP7h-xYnZSR36Bbvd496402LZQujMicehsCiqFpFKCJMp45nqg@mail.gmail.com>
 <CAPJVwB==VcpZb057P8HmP3sK5qh0W0s4V6iWMeCKbS3UQH6QUg@mail.gmail.com>
 <53E93147.4050504@stoneleaf.us>
 <CAPJVwBneeaibd3Pxq3vmoskvUfsHo-8txdMq5RMuv91BnP15gA@mail.gmail.com>
 <53E93736.8030105@stoneleaf.us> <lsbfad$u8o$1@ger.gmane.org>
 <lsbfof$4b3$1@ger.gmane.org> <lsbmb7$9eq$1@ger.gmane.org>
 <53E983F9.1090802@stoneleaf.us> <87k36ei5kv.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E9BF39.6060601@biologie.uni-freiburg.de>
 <87ha1hhum6.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CADiSq7daaYAhb9Cx777YKCT0enKw-inWBKWrOf9nyCTpF0Yb0g@mail.gmail.com>
Message-ID: <lsgh95$4vs$1@ger.gmane.org>



On 08/13/2014 12:50 AM, Nick Coghlan wrote:
> On 13 August 2014 14:38, Stephen J. Turnbull<stephen at xemacs.org>  wrote:
>> >Wolfgang Maier writes:
>> >
>> >  > Exactly. So my point was that when you don't subclass str, but instead
>> >  > use a wrapper around it, you can give it a as str-like interface as you
>> >  > want so the thing looks and feels like a string to users, it will still
>> >  > not work as part of an iterable passed to .join
>> >
>> >You mean this behavior?
>> >
>> >wideload:~ 12:42$ python3.2
>>>>> >>>>
>> >...
>>>>> >>>>class N:
>> >...  def __init__(self, s=''):
>> >...   self.s = s
>> >...  def __str__(self):
>> >...   return self.s
>> >...
>>>>> >>>>" ".join(['a', N('b')])
>> >Traceback (most recent call last):
>> >   File "<stdin>", line 1, in <module>
>> >TypeError: sequence item 1: expected str instance, N found
>>>>> >>>>' '.join(str(x) for x in ['a', N('b')])
>> >'a b'
>>>>> >>>>
>> >
>> >Given the fact that every object is str-able, I don't think we want to
>> >give "str(x) for x in" semantics to str.join.  So I think the answer
>> >is "if you want Nasty to automatically acquire all the behaviors of
>> >str, make it a subclass of str".
> Note that this is a general problem - it is quite common to use
> explicit type checks against str rather than relying on ducktyping. In
> theory, a suitable ABC could be defined (using collections.UserString
> as a starting point), but nobody has ever found it a pressing enough
> problem to take the time to do so - it's generally easier to just
> inherit from str.

Is there a way to select a method more specifically on it's mixin?

       thing.method            # any like named method
       thing.method|mixin      # Only if it's from mixin

Where method is spelled the same on differnt types, but it's actual 
operation may be different.

Obviously that spelling won't work, but the idea is to allow a more fine 
grained method selection.

       thing.__add__|number(other)
       thing.__add__|sequence(other)
       thing.__add__|container(other)
       thing.__add__|str(other)

So if I wanted to join strings but not containers or numbers, I could use 
the __add__|str method.  And conversely if I wanted to iterate nested 
containers without iterating strings too, I could use __iter__|container 
method.


Now that I think about it a bit more, it probably would be spelled..

     mixin.method(thing, other)

But maybe for the same reason we don't normally call a class method 
directly applies?

     class.method(thing, other)


I think in most cases, the difference might be getting an attribute error 
early, vs a type error a bit later.  But it seems to me there may be other 
differences/gotcha's in the case of calling a mixin or class method directly.

Currently I'd add a type check before calling the method, but I'd like the 
finer grained method resolution over the type check.

Cheers,
    Ron


From alex.gaynor at gmail.com  Wed Aug 13 22:29:38 2014
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 13 Aug 2014 20:29:38 +0000 (UTC)
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <loom.20140813T222929-838@post.gmane.org>

I'm strongly opposed this, for a few reasons.

First, I think that standardizing on a syntax, without a semantics is
incredibly confusing, and I can't imagine how having *multiple* competing
implementations would be a boon for anyone.

This proposal seems to be built around the idea that we should have a syntax,
and then people can write third party tools, but Python itself won't really do
anything with them.

Fundamentally, this seems like a very confusing approach. How we write a type,
and what we do with that information are fundamentally connected. Can I cast a
``List[str]`` to a ``List[object]`` in any way? If yes, what happens when I go
to put an ``int`` in it? There's no runtime checking, so the type system is
unsound, on the other hand, disallowing this prevents many types of successes.

Both solutions have merit, but the idea of some implementations of the type
checker having covariance and some contravariance is fairly disturbing.

Another concern I have is that analysis based on these types is making some
pretty strong assumptions about static-ness of Python programs that aren't
valid. While existing checkers like ``flake8`` also do this, their assumptions
are basically constrained to the symbol table, while this is far deeper. For
example, can I annotate somethign as ``six.text_type``? What about
``django.db.models.sql.Query`` (keep in mind that this class is redefined based
on what database you're using (not actually true, but it used to be))?

Python's type system isn't very good. It lacks many features of more powerful
systems such as algebraic data types, interfaces, and parametric polymorphism.
Despite this, it works pretty well because of Python's dynamic typing. I
strongly believe that attempting to enforce the existing type system would be a
real shame.

Alex

PS: You're right. None of this would provide *any* value for PyPy.


From christian at python.org  Wed Aug 13 22:29:48 2014
From: christian at python.org (Christian Heimes)
Date: Wed, 13 Aug 2014 22:29:48 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <53EBCABC.7010801@python.org>

On 13.08.2014 21:44, Guido van Rossum wrote:
> Yesterday afternoon I had an inspiring conversation with Bob Ippolito
> (man of many trades, author of simplejson) and Jukka Lehtosalo (author
> of mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about
> what Python can learn from Haskell (and other languages); yesterday he
> gave the same talk at Dropbox. The talk is online
> (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad
> strokes comes down to three suggestions:
> 
>   (a) Python should adopt mypy's syntax for function annotations
>   (b) Python's use of mutabe containers by default is wrong
>   (c) Python should adopt some kind of Abstract Data Types

I was at Bob's talk during EP14 and really liked the idea. A couple of
colleagues and other attendees also said it's a good and useful
proposal. I also like your proposal to standardize the type annotations
first without a full integration of mypy.

In general I'm +1 but I like to discuss two aspects:

1) I'm not keen with the naming of mypy's typing classes. The visual
distinction between e.g. dict() and Dict() is too small and IMHO
confusing for newcomers. How about an additional 'T' prefix to make
clear that the objects are referring to typing objects?

  from typing import TList, TDict

  def word_count(input: TList[str]) -> TDict[str, int]:
      ...

2) PEP 3107 only specifies arguments and return values but not
exceptions that can be raised by a function. Java has the "throws"
syntax to list possible exceptions:

 public void readFile() throws IOException {}

May I suggest that we also standardize a way to annotate the exceptions
that can be raised by a function? It's a very useful piece of
information and commonly requested information on the Python user
mailing list. It doesn't have to be a new syntax element, a decorator in
the typing module would suffice, too. For example:

  from typing import TList, TDict, raises

  @raises(RuntimeError, (ValueError, "is raised when input is empty"))
  def word_count(input: TList[str]) -> TDict[str, int]:
      ...

Regards,
Christian


From ethan at stoneleaf.us  Wed Aug 13 22:50:06 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 13 Aug 2014 13:50:06 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBC3BA.10808@stoneleaf.us>
 <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
Message-ID: <53EBCF7E.2050909@stoneleaf.us>

On 08/13/2014 01:19 PM, Guido van Rossum wrote:
> On Wed, Aug 13, 2014 at 12:59 PM, Ethan Furman wrote:
>>
>> -1 on deprecating alternative uses of annotations.
>
> Do you have a favorite alternative annotation use that you actually use (or are likely to)?

My script argument parser [1] uses annotations to figure out how to parse the cli parameters and cast them to 
appropriate values (copied the idea from one of Michele Simionato's projects... plac [2], I believe).

I could store the info in some other structure besides 'annotations', but it's there and it fits the bill conceptually. 
  Amusingly, it's a form of type info, but instead of saying what it has to already be, says what it will become.

--
~Ethan~


[1] https://pypi.python.org/pypi/scription  (due for an overhaul now I've used it for awhile ;)
[2] https://pypi.python.org/pypi/plac/0.9.1

From donald at stufft.io  Wed Aug 13 22:53:35 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 13 Aug 2014 16:53:35 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <loom.20140813T222929-838@post.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
Message-ID: <CD06C3D0-4BFE-4B8A-9FC2-00125E87F4ED@stufft.io>


> On Aug 13, 2014, at 4:29 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> 
> I'm strongly opposed this, for a few reasons.
> 
> First, I think that standardizing on a syntax, without a semantics is
> incredibly confusing, and I can't imagine how having *multiple* competing
> implementations would be a boon for anyone.
> 
> This proposal seems to be built around the idea that we should have a syntax,
> and then people can write third party tools, but Python itself won't really do
> anything with them.
> 
> Fundamentally, this seems like a very confusing approach. How we write a type,
> and what we do with that information are fundamentally connected. Can I cast a
> ``List[str]`` to a ``List[object]`` in any way? If yes, what happens when I go
> to put an ``int`` in it? There's no runtime checking, so the type system is
> unsound, on the other hand, disallowing this prevents many types of successes.
> 
> Both solutions have merit, but the idea of some implementations of the type
> checker having covariance and some contravariance is fairly disturbing.
> 
> Another concern I have is that analysis based on these types is making some
> pretty strong assumptions about static-ness of Python programs that aren't
> valid. While existing checkers like ``flake8`` also do this, their assumptions
> are basically constrained to the symbol table, while this is far deeper. For
> example, can I annotate somethign as ``six.text_type``? What about
> ``django.db.models.sql.Query`` (keep in mind that this class is redefined based
> on what database you're using (not actually true, but it used to be))?
> 
> Python's type system isn't very good. It lacks many features of more powerful
> systems such as algebraic data types, interfaces, and parametric polymorphism.
> Despite this, it works pretty well because of Python's dynamic typing. I
> strongly believe that attempting to enforce the existing type system would be a
> real shame.
> 
> Alex
> 
> PS: You're right. None of this would provide *any* value for PyPy.
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/


I agree with Alex that I think leaving the actual semantics of what these things
mean up to a third party, which can possibly be swapped out by individual end
users, is terribly confusing. I don?t think I agree though that this is a bad
idea in general, I think that we should just add it for real and skip the
indirection.

IOW I'm not sure I see the benefit of defining the syntax but not the semantics
when it seems this is already completely possible given the fact that mypy
exists.

The only real benefits I can see from doing it are that the stdlib can use it,
and the ``import typing`` aspect. I don't believe that the stdlib benefits are
great enough to get the possible confusion of multiple different implementations
and I think that the typing import could easily be provided as a project on PyPI
that people can depend on if they want to use this in their code.

So my vote would be to add mypy semantics to the language itself.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/76121969/attachment.html>

From andrey.vlasovskikh at gmail.com  Wed Aug 13 23:08:33 2014
From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh)
Date: Thu, 14 Aug 2014 01:08:33 +0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBC3BA.10808@stoneleaf.us>
 <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
Message-ID: <DCF0B3C2-6067-4856-864A-319CEAA7230B@gmail.com>

2014-08-14, 0:19, Guido van Rossum <guido at python.org> wrote:

> Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can learn from Haskell (and other languages); yesterday he gave the same talk at Dropbox. The talk is online (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad strokes comes down to three suggestions:
> 
>  (a) Python should adopt mypy's syntax for function annotations


+1. I'm a developer of the code analysis engine of PyCharm. I have discussed this idea with Jukka Lehtosalo and recently with Dave Halter, the author of Jedi code completion library. Standardized type annotations would be very useful for code analysis tools and IDEs such as PyCharm, Jedi and pylint. Type annotations would be especially great for third-party libraries. The idea is that most Python programmers don't have to write annotations in order to benefit from them. Annotated libraries are often enough for good code analysis.

We (PyCharm) and Jukka have made some initial steps in this direction, including thoughts on semantics of annotations (https://github.com/pytypes/pytypes). Feedback is welcome.

Here are slides from my talk about optional typing in Python, that show how Mypy types can be used in both static and dynamic type checking (http://blog.pirx.ru/media/files/2013/python-optional-typing/), Mypy-related part starts from slide 14.

We are interested in getting type annotations standardized and we would like to help developing and testing type annotations proposals.

-- 
Andrey Vlasovskikh
Web: http://pirx.ru/


From guido at python.org  Wed Aug 13 23:09:55 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 14:09:55 -0700
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
In-Reply-To: <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com>
References: <mailman.72122.1407959084.18129.python-ideas@python.org>
 <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com>
Message-ID: <CAP7+vJJKP=1QfQrUaQ0Rq5Et-o81+MmzWJaeUT8MRkGvvhwRtQ@mail.gmail.com>

On Wed, Aug 13, 2014 at 1:18 PM, Raymond Hettinger <
raymond.hettinger at gmail.com> wrote:

>
> On Aug 13, 2014, at 12:44 PM, python-ideas-request at python.org wrote:
>
>  The goal is to make it possible to add
> type checking annotations to 3rd party modules (and even to the stdlib)
> while allowing unaltered execution of the program by the (unmodified)
> Python 3.5 interpreter.
>
>
> Is the goal to "make it possible" or would it quickly become required
> (i.e. any time you write normal, readable Python, it would break
> someone's optimizer, refactorer, linter, etc.?)
>

Whoa, whoa. That's not at all the idea. Currently *nobody* uses type
annotations because there's no standard notation. My goal is to enable
their use by proposing a standard, nothing more.


> Are these annotations going to be pushed into every nook and cranny
> in the standard library and wend their way into the C code?
>

There's no need for that. Mypy lets you create stub modules that shadow
stdlib (and other) modules during typechecking, and it comes with a modest
set of standard stubs for some of the most popular stdlib modules. In most
cases it's actually much more effective to create a stub than to add the
annotations to the stdlib source code -- the stdlib code itself often is
difficult to type check due to various optimizations or backwards
compatibility concerns, but writing stubs is relatively straightforward,
and the stubs will give useful guidance to users of the stdlib who care to
run mypy over their own code.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/62437a89/attachment.html>

From antoine at python.org  Wed Aug 13 23:12:26 2014
From: antoine at python.org (Antoine Pitrou)
Date: Wed, 13 Aug 2014 17:12:26 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <lsgkbq$bht$1@ger.gmane.org>


Hello,

First, as a disclaimer, I am currently working on Numba for Continuum 
Analytics.  Numba has its own type inference system which it applies to 
functions decorated with the @jit decorator. Due to Numba's objectives, 
the type inference system is heavily geared towards numerical computing, 
but it is conceptually (and a bit concretely) able to represent more 
generic information, such as "an enumerate() over an iterator of a 
complex128 numpy array".

There are two sides to type inference:

1) first the (optional) annotations
(I'm saying "optional" because in the most basic usage, a JIT compiler 
is normally able to defer compilation until the first function or method 
call, and to deduce input types from that)

2) second the inference engine properly, which walks the code (in 
whatever form the tool's developer has chosen: bytecode, AST, IR) and 
deduces types for any intermediate values

Now only #1 is implied by this PEP proposal, but it also sounds like we 
should take into account the desired properties of #2 (for example, 
being able to express "an iterator of three-tuples" can be important for 
a JIT compiler - or not, perhaps, depending on the JIT compiler :-)). 
What #2 wants to do will differ depending on the use case: e.g. a code 
checker may need less type granularity than a JIT compiler.


Therefore, regardless of mypy's typesystem's completeness and 
granularity, one requirement is for it to be easily extensible. By 
extensible I mean not only being able to define new type descriptions, 
but being able to do so for existing third-party libraries you don't 
want to modify.

I'm saying that because I'm looking at 
http://mypy-lang.org/tutorial.html#genericclasses , and it's not clear 
from this example whether the typing code has to be interwoven with the 
collection's implementation, or can be written as a separate code module 
entirely (*). Ideally both should probably be possible (in the same vein 
as being able to subclass an ABC, or register an existing class with 
it). This also includes being to type-declare functions and types from C 
extension modules.

In Numba, this would be typically required to write typing descriptions 
for Numpy arrays and functions; but also to derive descriptions for 
fixed-width integers, single-precision floats, etc. (this also means 
some form of subclassing for type descriptions themselves).

(*) (actually, I'm a bit worried when I see that "List[int]()" 
instantiates an actual list; calling a type description class should 
give you a parametered type description, not an object; the [] notation 
is in general not powerful enough if you want several type parameters, 
possibly keyword-only)


At some point, it will be even better if the typing system is powerful 
enough to remember properties of the *values* (for example not only "a 
string", but "a one-character string, or even "one of the 'Y', 'M', 'D' 
strings"). Think about type-checking / type-infering calls to the struct 
module.


I may come back with more comments once I've read the mypy docs and/or 
code in detail.

Regards

Antoine.



From guido at python.org  Wed Aug 13 23:46:08 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 14:46:08 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <loom.20140813T222929-838@post.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
Message-ID: <CAP7+vJLFP4BaRtp0Mhg3d2A56YVLj=odUbY94XtyFKtpyUoOFA@mail.gmail.com>

On Wed, Aug 13, 2014 at 1:29 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:

> I'm strongly opposed this, for a few reasons.
>
> First, I think that standardizing on a syntax, without a semantics is
> incredibly confusing, and I can't imagine how having *multiple* competing
> implementations would be a boon for anyone.
>

That part was probably overly vague in my original message. I actually do
want to standardize on semantics, but I think the semantics will prove
controversial (they already have :-) and I think it's better to standardize
the syntax and *some* semantics first rather than having to wait another
decade for the debate over the semantics to settle. I mostly want to leave
the door open for mypy to become smarter. But it might make sense to have a
"weaker" interpretation in some cases too (e.g. an IDE might use a weaker
type system in order to avoid overwhelming users with warnings).


> This proposal seems to be built around the idea that we should have a
> syntax,
> and then people can write third party tools, but Python itself won't
> really do
> anything with them.
>

Right.


> Fundamentally, this seems like a very confusing approach. How we write a
> type,
> and what we do with that information are fundamentally connected. Can I
> cast a
> ``List[str]`` to a ``List[object]`` in any way? If yes, what happens when
> I go
> to put an ``int`` in it? There's no runtime checking, so the type system is
> unsound, on the other hand, disallowing this prevents many types of
> successes.
>

Mypy has a cast() operator that you can use to shut it up when you (think
you) know the conversion is safe.


> Both solutions have merit, but the idea of some implementations of the type
> checker having covariance and some contravariance is fairly disturbing.
>

Yeah, that wouldn't be good. ;-)


> Another concern I have is that analysis based on these types is making some
> pretty strong assumptions about static-ness of Python programs that aren't
> valid. While existing checkers like ``flake8`` also do this, their
> assumptions
> are basically constrained to the symbol table, while this is far deeper.
> For
> example, can I annotate something as ``six.text_type``? What about
> ``django.db.models.sql.Query`` (keep in mind that this class is redefined
> based
> on what database you're using (not actually true, but it used to be))?
>

Time will have to tell. Stubs can help. I encourage you to try annotating a
medium-sized module. It's likely that you'll find a few things: maybe a bug
in mypy, maybe a missing mypy feature, maybe a bug in your code, maybe a
shady coding practice in your code or a poorly documented function (I know
I found several of each during my own experiments so far).


> Python's type system isn't very good. It lacks many features of more
> powerful
> systems such as algebraic data types, interfaces, and parametric
> polymorphism.
> Despite this, it works pretty well because of Python's dynamic typing. I
> strongly believe that attempting to enforce the existing type system would
> be a
> real shame.
>

Mypy shines in those areas of Python programs that are mostly statically
typed. There are many such areas in most large systems. There are usually
also some areas where mypy's type system is inadequate. It's easy to shut
it up for those cases (in fact, mypy is silent unless you use at least one
annotation for a function). But that's the case with most type systems.
Even Haskell sometimes calls out to C.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/124ee438/attachment-0001.html>

From guido at python.org  Thu Aug 14 00:00:20 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 15:00:20 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53EBCF7E.2050909@stoneleaf.us>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBC3BA.10808@stoneleaf.us>
 <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
 <53EBCF7E.2050909@stoneleaf.us>
Message-ID: <CAP7+vJ+MYtiK1B2F=MrN=yPY_dObaeSn3gtFps98xG7AJ4mWJw@mail.gmail.com>

On Wed, Aug 13, 2014 at 1:50 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 08/13/2014 01:19 PM, Guido van Rossum wrote:
>
>  On Wed, Aug 13, 2014 at 12:59 PM, Ethan Furman wrote:
>>
>>>
>>> -1 on deprecating alternative uses of annotations.
>>>
>>
>> Do you have a favorite alternative annotation use that you actually use
>> (or are likely to)?
>>
>
> My script argument parser [1] uses annotations to figure out how to parse
> the cli parameters and cast them to appropriate values (copied the idea
> from one of Michele Simionato's projects... plac [2], I believe).
>
> I could store the info in some other structure besides 'annotations', but
> it's there and it fits the bill conceptually.  Amusingly, it's a form of
> type info, but instead of saying what it has to already be, says what it
> will become.
>

I couldn't find any docs for scription (the tarball contains just the
source code, not even an example), although I did find some for plac. I
expect using type annotations to the source of scription.py might actually
make it easier to grok what it does. :-)

But really, I'm sure that in Python 3.5, scription and mypy can coexist. If
the mypy idea takes off you might eventually be convinced to use a
different convention. But you'd get plenty of warning.


> [1] https://pypi.python.org/pypi/scription  (due for an overhaul now I've
> used it for awhile ;)
> [2] https://pypi.python.org/pypi/plac/0.9.1
>

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/6588363b/attachment.html>

From guido at python.org  Thu Aug 14 00:05:53 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 15:05:53 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CD06C3D0-4BFE-4B8A-9FC2-00125E87F4ED@stufft.io>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CD06C3D0-4BFE-4B8A-9FC2-00125E87F4ED@stufft.io>
Message-ID: <CAP7+vJJd4EBcmR9iETAgk5H1qriPEKsCeaz3keWZ36PxmKNHgQ@mail.gmail.com>

On Wed, Aug 13, 2014 at 1:53 PM, Donald Stufft <donald at stufft.io> wrote:

> I agree with Alex that I think leaving the actual semantics of what these
> things
> mean up to a third party, which can possibly be swapped out by individual
> end
> users, is terribly confusing. I don?t think I agree though that this is a
> bad
> idea in general, I think that we should just add it for real and skip the
> indirection.
>

Yeah, I probably overstated the option of alternative interpretations. I
just don't want to have to write a PEP that specifies every little detail
of mypy's type checking algorithm, and I don't think anyone would want to
have to read such a PEP either. But maybe we can compromise on something
that sketches broad strokes and leaves the details up to the team that
maintains mypy (after all that tactic has worked pretty well for Python
itself :-).


> IOW I'm not sure I see the benefit of defining the syntax but not the
> semantics
> when it seems this is already completely possible given the fact that mypy
> exists.
>
> The only real benefits I can see from doing it are that the stdlib can use
> it,
> and the ``import typing`` aspect. I don't believe that the stdlib benefits
> are
> great enough to get the possible confusion of multiple different
> implementations
> and I think that the typing import could easily be provided as a project
> on PyPI
> that people can depend on if they want to use this in their code.
>
> So my vote would be to add mypy semantics to the language itself.
>

What exactly would that mean? I don't think the Python interpreter should
reject programs that fail the type check -- in fact, separating the type
check from run time is the most crucial point of my proposal.

I'm fine to have a discussion on things like covariance vs. contravariance,
or what form of duck typing are acceptable, etc.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/c3dde5bd/attachment.html>

From guido at python.org  Thu Aug 14 00:07:19 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 15:07:19 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <DCF0B3C2-6067-4856-864A-319CEAA7230B@gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBC3BA.10808@stoneleaf.us>
 <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
 <DCF0B3C2-6067-4856-864A-319CEAA7230B@gmail.com>
Message-ID: <CAP7+vJK4PE2W2nFAEG27c+vP-HZzcGm+DzZ4L5G8HNMA-gGS9Q@mail.gmail.com>

Wow. Awesome. I will make time to study what you have already done!


On Wed, Aug 13, 2014 at 2:08 PM, Andrey Vlasovskikh <
andrey.vlasovskikh at gmail.com> wrote:

> 2014-08-14, 0:19, Guido van Rossum <guido at python.org> wrote:
>
> > Yesterday afternoon I had an inspiring conversation with Bob Ippolito
> (man of many trades, author of simplejson) and Jukka Lehtosalo (author of
> mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about what
> Python can learn from Haskell (and other languages); yesterday he gave the
> same talk at Dropbox. The talk is online (
> https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad
> strokes comes down to three suggestions:
> >
> >  (a) Python should adopt mypy's syntax for function annotations
>
>
> +1. I'm a developer of the code analysis engine of PyCharm. I have
> discussed this idea with Jukka Lehtosalo and recently with Dave Halter, the
> author of Jedi code completion library. Standardized type annotations would
> be very useful for code analysis tools and IDEs such as PyCharm, Jedi and
> pylint. Type annotations would be especially great for third-party
> libraries. The idea is that most Python programmers don't have to write
> annotations in order to benefit from them. Annotated libraries are often
> enough for good code analysis.
>
> We (PyCharm) and Jukka have made some initial steps in this direction,
> including thoughts on semantics of annotations (
> https://github.com/pytypes/pytypes). Feedback is welcome.
>
> Here are slides from my talk about optional typing in Python, that show
> how Mypy types can be used in both static and dynamic type checking (
> http://blog.pirx.ru/media/files/2013/python-optional-typing/),
> Mypy-related part starts from slide 14.
>
> We are interested in getting type annotations standardized and we would
> like to help developing and testing type annotations proposals.
>
> --
> Andrey Vlasovskikh
> Web: http://pirx.ru/
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/f49ca84b/attachment.html>

From apalala at gmail.com  Thu Aug 14 00:21:40 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Wed, 13 Aug 2014 17:51:40 -0430
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <CAN1YFWtvpmRauVr6qH+Xv1HuDBuJdZppRY6FZ+JFBViNvH-7_A@mail.gmail.com>

On Wed, Aug 13, 2014 at 3:14 PM, Guido van Rossum <guido at python.org> wrote:

> I am proposing that we adopt whatever mypy uses here, keeping discussion
> of the details (mostly) out of the PEP. The goal is to make it possible to
> add type checking annotations to 3rd party modules (and even to the stdlib)
> while allowing unaltered execution of the program by the (unmodified)
> Python 3.5 interpreter.


I'll comment later on the core subject.

For now, I think this deserves some thought:

Function annotations are not available in Python 2.7, so promoting
widespread use of annotations in 3.5 would be promoting code that is
compatible only with 3.x, when the current situation is that much effort is
being spent on writing code that works on both 2.7 and 3.4 (most
libraries?).

Independently of its core merits, this proposal should fail unless
annotations are added to Python 2.8.

Cheers,

-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/a43b8896/attachment-0001.html>

From toddrjen at gmail.com  Thu Aug 14 00:28:15 2014
From: toddrjen at gmail.com (Todd)
Date: Thu, 14 Aug 2014 00:28:15 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>

On Aug 13, 2014 9:45 PM, "Guido van Rossum" <guido at python.org> wrote:
> (1) A change of direction for function annotations
>
> PEP 3107, which introduced function annotations, is intentional
non-committal about how function annotations should be used. It lists a
number of use cases, including but not limited to type checking. It also
mentions some rejected proposals that would have standardized either a
syntax for indicating types and/or a way for multiple frameworks to attach
different annotations to the same function. AFAIK in practice there is
little use of function annotations in mainstream code, and I propose a
conscious change of course here by stating that annotations should be used
to indicate types and to propose a standard notation for them.
>
> (We may have to have some backwards compatibility provision to avoid
breaking code that currently uses annotations for some other purpose.
Fortunately the only issue, at least initially, will be that when running
mypy to type check such code it will produce complaints about the
annotations; it will not affect how such code is executed by the Python
interpreter. Nevertheless, it would be good to deprecate such alternative
uses of annotations.)

I watched the original talk and read your proposal.  I think type
annotations could very very useful in certain contexts.

However, I still don't get this bit. Why would allowing type annotations
automatically imply that no other annotations would be possible?  Couldn't
we formalize what would be considered a type annotation while still
allowing annotations that don't fit this criteria to be used for other
things?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/a9d6d6fb/attachment.html>

From ceronman at gmail.com  Thu Aug 14 00:27:54 2014
From: ceronman at gmail.com (=?UTF-8?Q?Manuel_Cer=C3=B3n?=)
Date: Thu, 14 Aug 2014 00:27:54 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <CA+VECoWD7pvN1Ex0O=XrvAq1ZCOSKKrCoDATgk_5i=q_Hmrg4Q@mail.gmail.com>

On Wed, Aug 13, 2014 at 9:44 PM, Guido van Rossum <guido at python.org> wrote:

> [There is no TL;DR other than the subject line. Please read the whole
> thing before replying. I do have an appendix with some motivations for
> adding type annotations at the end.]
>

This is a very interesting idea. I played a bit with function annotations (
https://github.com/ceronman/typeannotations) and I gave a talk about them
at EuroPython 2013. Certainly static type analysis is probably the best use
case.

The curious thing here is that while standardizing a syntax for type
> annotations, we technically still won't be adopting standard rules for type
> checking. This is intentional. First of all, fully specifying all the type
> checking rules would make for a really long and boring PEP (a much better
> specification would probably be the mypy source code). Second, I think it's
> fine if the type checking algorithm evolves over time, or if variations
> emerge. The worst that can happen is that you consider your code correct
> but mypy disagrees; your code will still run.
>
> That said, I don't want to *completely* leave out any specification. I
> want the contents of the typing.py module to be specified in the PEP, so
> that it can be used with confidence. But whether mypy will complain about
> your particular form of duck typing doesn't have to be specified by the
> PEP. Perhaps as mypy evolves it will take options to tell it how to handle
> certain edge cases. Forks of mypy (or entirely different implementations of
> type checking based on the same annotation syntax) are also a possibility.
> Maybe in the distant future a version of Python will take a different
> stance, once we have more experience with how this works out in practice,
> but for Python 3.5 I want to restrict the scope of the upheaval.
>

The type checking algorithm might evolve over the time, but by including
typing.py in the stdlib, the syntax for annotations would be almost frozen
and that will be a limitation. In other projects such as TypeScript (
http://www.typescriptlang.org/), that the syntax usually evolves alongside
the algorithms.

Is the syntax specifyed in typing.py mature enough to put it in the stdlib
and expect users to start annotating their projects without worrying too
much about future changes?

Is there enough feedback from users using mypy in their projects?

I think that rushing typing.py into 3.5 is not a good idea. However, It'd
be nice to add some notes in PEP8, encourage it's use as an external
library, let some projects and tools (e.g. PyCharm) use it. It's not that
bad if mypy lives 100% outside the Python distribution for a while. Just
like TypeScript to JavaScript. After getting some user base, part of it
(typing.py) could be moved to the stdlib.

Manuel.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/eb99215a/attachment.html>

From rymg19 at gmail.com  Thu Aug 14 00:34:07 2014
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 13 Aug 2014 17:34:07 -0500
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53EBCABC.7010801@python.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org>
Message-ID: <CAO41-mMvSJSJ0ji4RrMLOS-OD=XtuFO9aieCWJ_P=k1FS=_5EA@mail.gmail.com>

On Wed, Aug 13, 2014 at 3:29 PM, Christian Heimes <christian at python.org>
wrote:

> On 13.08.2014 21:44, Guido van Rossum wrote:
> > Yesterday afternoon I had an inspiring conversation with Bob Ippolito
> > (man of many trades, author of simplejson) and Jukka Lehtosalo (author
> > of mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about
> > what Python can learn from Haskell (and other languages); yesterday he
> > gave the same talk at Dropbox. The talk is online
> > (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad
> > strokes comes down to three suggestions:
> >
> >   (a) Python should adopt mypy's syntax for function annotations
> >   (b) Python's use of mutabe containers by default is wrong
> >   (c) Python should adopt some kind of Abstract Data Types
>
> I was at Bob's talk during EP14 and really liked the idea. A couple of
> colleagues and other attendees also said it's a good and useful
> proposal. I also like your proposal to standardize the type annotations
> first without a full integration of mypy.
>
> In general I'm +1 but I like to discuss two aspects:
>
> 1) I'm not keen with the naming of mypy's typing classes. The visual
> distinction between e.g. dict() and Dict() is too small and IMHO
> confusing for newcomers. How about an additional 'T' prefix to make
> clear that the objects are referring to typing objects?
>
>   from typing import TList, TDict
>
>   def word_count(input: TList[str]) -> TDict[str, int]:
>       ...
>

Eeewwwww. That's way too Pascal-ish.


> 2) PEP 3107 only specifies arguments and return values but not
> exceptions that can be raised by a function. Java has the "throws"
> syntax to list possible exceptions:
>
>  public void readFile() throws IOException {}
>
> May I suggest that we also standardize a way to annotate the exceptions
> that can be raised by a function? It's a very useful piece of
> information and commonly requested information on the Python user
> mailing list. It doesn't have to be a new syntax element, a decorator in
> the typing module would suffice, too. For example:
>
>   from typing import TList, TDict, raises
>
>   @raises(RuntimeError, (ValueError, "is raised when input is empty"))
>   def word_count(input: TList[str]) -> TDict[str, int]:
>       ...
>

That was a disaster in C++. It's confusing, especially since Python uses
exceptions more than most other languages do.


>
> Regards,
> Christian
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/9862a70c/attachment-0001.html>

From donald at stufft.io  Thu Aug 14 00:44:18 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 13 Aug 2014 18:44:18 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJJd4EBcmR9iETAgk5H1qriPEKsCeaz3keWZ36PxmKNHgQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CD06C3D0-4BFE-4B8A-9FC2-00125E87F4ED@stufft.io>
 <CAP7+vJJd4EBcmR9iETAgk5H1qriPEKsCeaz3keWZ36PxmKNHgQ@mail.gmail.com>
Message-ID: <DA35096A-4FCA-4D06-8DE4-E79F5475889B@stufft.io>


> On Aug 13, 2014, at 6:05 PM, Guido van Rossum <guido at python.org> wrote:
> 
> On Wed, Aug 13, 2014 at 1:53 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
> I agree with Alex that I think leaving the actual semantics of what these things
> mean up to a third party, which can possibly be swapped out by individual end
> users, is terribly confusing. I don?t think I agree though that this is a bad
> idea in general, I think that we should just add it for real and skip the
> indirection.
> 
> Yeah, I probably overstated the option of alternative interpretations. I just don't want to have to write a PEP that specifies every little detail of mypy's type checking algorithm, and I don't think anyone would want to have to read such a PEP either. But maybe we can compromise on something that sketches broad strokes and leaves the details up to the team that maintains mypy (after all that tactic has worked pretty well for Python itself :-).
>  
> IOW I'm not sure I see the benefit of defining the syntax but not the semantics
> when it seems this is already completely possible given the fact that mypy
> exists.
> 
> The only real benefits I can see from doing it are that the stdlib can use it,
> and the ``import typing`` aspect. I don't believe that the stdlib benefits are
> great enough to get the possible confusion of multiple different implementations
> and I think that the typing import could easily be provided as a project on PyPI
> that people can depend on if they want to use this in their code.
> 
> So my vote would be to add mypy semantics to the language itself.
> 
> What exactly would that mean? I don't think the Python interpreter should reject programs that fail the type check -- in fact, separating the type check from run time is the most crucial point of my proposal.

I don?t know exactly :)

Some ideas:

1) Raise a warning when the type check fails, but allow it happen. This would
   have the benefit of possibly catching bugs, but it's still opt in in the
   sense that you have to write the annotations for anything to happen. This
   would also enable people to turn on enforced type checking by raising the
   warning level to an exception.

   Even if this was off by default it would make it easy to enable it during
   test runs and also enable easier/better quickcheck like functionality.

2) Simply add a flag to the interpreter that turns on type checking.

3) Add a stdlib module that would run the program under type checking, like
   ``python -m typing myprog`` instead of ``python -m myprog``.

Really I think a lot of the benefit is likely to come in the form of linting
and during test runs. However if I have to run a seperate Python interpreter
to actually do the run then I risk getting bad results through varying things
like interpreter differences, language level differences, etc.

Although I wouldn't complain if it meant that Python had actual type checking
at the run time if a function had type annotations :)

> 
> I'm fine to have a discussion on things like covariance vs. contravariance, or what form of duck typing are acceptable, etc.

I?m not particularly knowledgable about the actual workings of a type system and
covariance vs contravariance and the like. My main concern there is having a
single reality. The meaning of something shouldn't change because I used a
different interpreter/linter/whatever. Beyond that I don't know enough to have
an opinion on the actual semantics.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/e9198cde/attachment.html>

From guido at python.org  Thu Aug 14 01:11:46 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 16:11:46 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAN1YFWtvpmRauVr6qH+Xv1HuDBuJdZppRY6FZ+JFBViNvH-7_A@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAN1YFWtvpmRauVr6qH+Xv1HuDBuJdZppRY6FZ+JFBViNvH-7_A@mail.gmail.com>
Message-ID: <CAP7+vJ+AV0VtXf4jeSFokkVF4EqUQG0mQT3cfKuttrQDwkchPA@mail.gmail.com>

On Wed, Aug 13, 2014 at 3:21 PM, Juancarlo A?ez <apalala at gmail.com> wrote:

>
>
> On Wed, Aug 13, 2014 at 3:14 PM, Guido van Rossum <guido at python.org>
> wrote:
>
>> I am proposing that we adopt whatever mypy uses here, keeping discussion
>> of the details (mostly) out of the PEP. The goal is to make it possible to
>> add type checking annotations to 3rd party modules (and even to the stdlib)
>> while allowing unaltered execution of the program by the (unmodified)
>> Python 3.5 interpreter.
>
>
> I'll comment later on the core subject.
>
> For now, I think this deserves some thought:
>
> Function annotations are not available in Python 2.7, so promoting
> widespread use of annotations in 3.5 would be promoting code that is
> compatible only with 3.x, when the current situation is that much effort is
> being spent on writing code that works on both 2.7 and 3.4 (most
> libraries?).
>
> Independently of its core merits, this proposal should fail unless
> annotations are added to Python 2.8.
>

Actually, mypy already has a solution. There's a codec (
https://github.com/JukkaL/mypy/tree/master/mypy/codec) that you can use
which transforms Python-2-with-annotations into vanilla Python 2. It's not
an ideal solution, but it can work in cases where you absolutely have to
have state of the art Python 3.5 type checking *and* backwards
compatibility with Python 2.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/5ad86b97/attachment.html>

From guido at python.org  Thu Aug 14 01:24:43 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 16:24:43 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CA+VECoVO0COba4u3o5YD0shE0r-P8-q99sWMQtpRkisQ8wa8UQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CA+VECoVO0COba4u3o5YD0shE0r-P8-q99sWMQtpRkisQ8wa8UQ@mail.gmail.com>
Message-ID: <CAP7+vJLKNydsdLuc8jRjhfYr_R8P7u1Jg1aZ4Yb1LA1c-t3w1g@mail.gmail.com>

On Wed, Aug 13, 2014 at 3:26 PM, Manuel Cer?n <ceronman at gmail.com> wrote:

> The type checking algorithm might evolve over the time, but by including
> typing.py in the stdlib, the syntax for annotations would be almost frozen
> and that will be a limitation. In other projects such as TypeScript (
> http://www.typescriptlang.org/), that the syntax usually evolves
> alongside the algorithms.
>

What kind of evolution did TypeScript experience?


> Is the syntax specifyed in typing.py mature enough to put it in the stdlib
> and expect users to start annotating their projects without worrying too
> much about future changes?
>

This is a good question. I do think it is good enough as a starting point
for future evolution. Perhaps the biggest question is how fast will the
annotation syntax need to evolve? If it needs to evolve significantly
faster than Python 3 feature releases come out (every 18 months,
approximately) then it may be better to hold off and aim for inclusion in
the 3.6 standard library. That would allow more time to reach agreement
(though I'm not sure that's a good thing :-), and in the mean time
typing.py could be distributed as a 3rd party module on PyPI.


> Is there enough feedback from users using mypy in their projects?
>
> I think that rushing typing.py into 3.5 is not a good idea. However, It'd
> be nice to add some notes in PEP8, encourage it's use as an external
> library, let some projects and tools (e.g. PyCharm) use it. It's not that
> bad if mypy lives 100% outside the Python distribution for a while. Just
> like TypeScript to JavaScript.
>

Well, JavaScript's evolution is tied up forever in a standards body, so
TypeScript realistically had no choice in the matter. But are there
actually people writing TypeScript? I haven't heard from them yet (people
at Dropbox seem to rather like CoffeeScript). Anyway, the situation isn't
quite the same -- you wouldn't make any friends in the Python world if you
wrote your code in an incompatible dialect that could only be executed
after a translation step, but in the JavaScript world that's how all
alternative languages work (and they even manage to interoperate).


> After getting some user base, part of it (typing.py) could be moved to the
> stdlib.
>

I'm still hopeful that we can get a sufficient user base and agreement on
mypy's features for inclusion in 3.5 (extrapolating the 3.4 release
schedule by 18 months, 3.5 alpha 1 would go out around February 2015; the
feature freeze cut-off date, beta 1, would around May thereafter).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/6e237ac1/attachment-0001.html>

From ben+python at benfinney.id.au  Thu Aug 14 01:27:20 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 14 Aug 2014 09:27:20 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org>
Message-ID: <85vbpw6kdj.fsf@benfinney.id.au>

Christian Heimes <christian at python.org>
writes:

> 1) I'm not keen with the naming of mypy's typing classes. The visual
> distinction between e.g. dict() and Dict() is too small and IMHO
> confusing for newcomers. How about an additional 'T' prefix to make
> clear that the objects are referring to typing objects?

To this reader, ?dict? and ?list? *are* ?typing objects? ? they are
objects that are types. Seeing code that referred to something else as
?typing objects? would be an infitation to confusion, IMO.

You could argue ?that's because you don't know the special meaning of
?typing object? being discussed here?. To which my response would be,
for a proposal to add something else as meaningful Python syntax, the
jargon is poorly chosen and needlessly confusing with established terms
in Python.

If there's going to be a distinction between the types (?dict?, ?list?,
etc.) and something else, I'd prefer it to be based on a clearer
terminology distinction.

-- 
 \         ?Simplicity and elegance are unpopular because they require |
  `\           hard work and discipline to achieve and education to be |
_o__)                                appreciated.? ?Edsger W. Dijkstra |
Ben Finney


From guido at python.org  Thu Aug 14 01:30:10 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 16:30:10 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
Message-ID: <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>

On Wed, Aug 13, 2014 at 3:28 PM, Todd <toddrjen at gmail.com> wrote:

> However, I still don't get this bit. Why would allowing type annotations
> automatically imply that no other annotations would be possible?  Couldn't
> we formalize what would be considered a type annotation while still
> allowing annotations that don't fit this criteria to be used for other
> things?
>

We certainly *could* do that. However, I haven't seen sufficient other uses
of annotations. If there is only one use for annotations (going forward),
annotations would be unambiguous. If we allow different types of
annotations, there would have to be a way to tell whether a particular
annotation is intended as a type annotation or not. Currently mypy ignores
all modules that don't import typing.py (using any form of import
statement), and we could continue this convention. But it would mean that
something like this would still require the typing import in order to be
checked by mypy:

import typing

def gcd(int a, int b) -> int:
    <tralala>

The (necessary) import would be flagged as unused by every linter in the
world... :-(

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/93bb632d/attachment.html>

From guido at python.org  Thu Aug 14 01:43:56 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 16:43:56 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <DA35096A-4FCA-4D06-8DE4-E79F5475889B@stufft.io>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CD06C3D0-4BFE-4B8A-9FC2-00125E87F4ED@stufft.io>
 <CAP7+vJJd4EBcmR9iETAgk5H1qriPEKsCeaz3keWZ36PxmKNHgQ@mail.gmail.com>
 <DA35096A-4FCA-4D06-8DE4-E79F5475889B@stufft.io>
Message-ID: <CAP7+vJJvTZ5KZbT0vgG6Jn7x4pUMTmuq6pF259PLQkDnZzpZJQ@mail.gmail.com>

On Wed, Aug 13, 2014 at 3:44 PM, Donald Stufft <donald at stufft.io> wrote:

> On Aug 13, 2014, at 6:05 PM, Guido van Rossum <guido at python.org> wrote:
>
> On Wed, Aug 13, 2014 at 1:53 PM, Donald Stufft <donald at stufft.io> wrote:
>>
>>
>> So my vote would be to add mypy semantics to the language itself.
>>
>
> What exactly would that mean? I don't think the Python interpreter should
> reject programs that fail the type check -- in fact, separating the type
> check from run time is the most crucial point of my proposal.
>
>
> I don?t know exactly :)
>
> Some ideas:
>
> 1) Raise a warning when the type check fails, but allow it happen. This
> would
>    have the benefit of possibly catching bugs, but it's still opt in in the
>    sense that you have to write the annotations for anything to happen.
> This
>    would also enable people to turn on enforced type checking by raising
> the
>    warning level to an exception.
>

I don't think that's going to happen. It would require the entire mypy
implementation to be checked into the stdlib. It would also require all
sorts of hacks in that implementation to deal with dynamic (or just
delayed) imports. Mypy currently doesn't handle any of that -- it must be
able to find all imported modules before it starts executing even one line
of code.


>    Even if this was off by default it would make it easy to enable it
> during
>    test runs and also enable easier/better quickcheck like functionality.
>

It would *have* to be off by default -- it's way too slow to be on by
default (note that some people are already fretting out today about a 25
msec process start-up time).


> 2) Simply add a flag to the interpreter that turns on type checking.
>
> 3) Add a stdlib module that would run the program under type checking, like
>    ``python -m typing myprog`` instead of ``python -m myprog``.
>
> Really I think a lot of the benefit is likely to come in the form of
> linting
> and during test runs. However if I have to run a separate Python
> interpreter
> to actually do the run then I risk getting bad results through varying
> things
> like interpreter differences, language level differences, etc.
>

Yeah, but I just don't think it's realistic to do anything about that for
3.5 (or 3.6 for that matter). In a decade... Who knows! :-)


> Although I wouldn't complain if it meant that Python had actual type
> checking
> at the run time if a function had type annotations :)
>

It's probably possibly to write a decorator that translates annotations
into assertions that are invoked when a function is called. But in most
cases it would be way too slow to turn on everywhere.

> I'm fine to have a discussion on things like covariance vs.
> contravariance, or what form of duck typing are acceptable, etc.
>
> I?m not particularly knowledgable about the actual workings of a type
> system and
> covariance vs contravariance and the like. My main concern there is having
> a
> single reality. The meaning of something shouldn't change because I used a
> different interpreter/linter/whatever. Beyond that I don't know enough to
> have
> an opinion on the actual semantics.
>

Yeah, I regret writing it so vaguely already. Having Alex Gaynor open with
"I'm strongly opposed [to] this" is a great joy killer. :-)

I just really don't want to have to redundantly write up a specification
for all the details of mypy's type checking rules in PEP-worthy English.
But I'm fine with discussing whether List[str] is a subclass or a
superclass of List[object] and how to tell the difference.

Still, different linters exist and I don't hear people complain about that.
I would also be okay if PyCharm's interpretation of the finer points of the
type checking syntax was subtly different from mypy's. In fact I would be
surprised if they weren't sometimes in disagreement. Heck, PyPy doesn't
give *every* Python program the same meaning as CPython, and that's a
feature. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/94e4cfaa/attachment.html>

From donald at stufft.io  Thu Aug 14 01:58:59 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 13 Aug 2014 19:58:59 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJJvTZ5KZbT0vgG6Jn7x4pUMTmuq6pF259PLQkDnZzpZJQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CD06C3D0-4BFE-4B8A-9FC2-00125E87F4ED@stufft.io>
 <CAP7+vJJd4EBcmR9iETAgk5H1qriPEKsCeaz3keWZ36PxmKNHgQ@mail.gmail.com>
 <DA35096A-4FCA-4D06-8DE4-E79F5475889B@stufft.io>
 <CAP7+vJJvTZ5KZbT0vgG6Jn7x4pUMTmuq6pF259PLQkDnZzpZJQ@mail.gmail.com>
Message-ID: <378F9047-5A37-49C8-A229-55904F8B2240@stufft.io>


> On Aug 13, 2014, at 7:43 PM, Guido van Rossum <guido at python.org> wrote:
> 
> On Wed, Aug 13, 2014 at 3:44 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
>> On Aug 13, 2014, at 6:05 PM, Guido van Rossum <guido at python.org <mailto:guido at python.org>> wrote:
>> 
>> On Wed, Aug 13, 2014 at 1:53 PM, Donald Stufft <donald at stufft.io <mailto:donald at stufft.io>> wrote:
>> 
>> So my vote would be to add mypy semantics to the language itself.
>> 
>> What exactly would that mean? I don't think the Python interpreter should reject programs that fail the type check -- in fact, separating the type check from run time is the most crucial point of my proposal.
> 
> I don?t know exactly :)
> 
> Some ideas:
> 
> 1) Raise a warning when the type check fails, but allow it happen. This would
>    have the benefit of possibly catching bugs, but it's still opt in in the
>    sense that you have to write the annotations for anything to happen. This
>    would also enable people to turn on enforced type checking by raising the
>    warning level to an exception.
> 
> I don't think that's going to happen. It would require the entire mypy implementation to be checked into the stdlib. It would also require all sorts of hacks in that implementation to deal with dynamic (or just delayed) imports. Mypy currently doesn't handle any of that -- it must be able to find all imported modules before it starts executing even one line of code.
>  
>    Even if this was off by default it would make it easy to enable it during
>    test runs and also enable easier/better quickcheck like functionality.
> 
> It would *have* to be off by default -- it's way too slow to be on by default (note that some people are already fretting out today about a 25 msec process start-up time).
>  
> 2) Simply add a flag to the interpreter that turns on type checking.
> 
> 3) Add a stdlib module that would run the program under type checking, like
>    ``python -m typing myprog`` instead of ``python -m myprog``.
> 
> Really I think a lot of the benefit is likely to come in the form of linting
> and during test runs. However if I have to run a separate Python interpreter
> to actually do the run then I risk getting bad results through varying things
> like interpreter differences, language level differences, etc.
> 
> Yeah, but I just don't think it's realistic to do anything about that for 3.5 (or 3.6 for that matter). In a decade... Who knows! :-)
>  
> Although I wouldn't complain if it meant that Python had actual type checking
> at the run time if a function had type annotations :)
> 
> It's probably possibly to write a decorator that translates annotations into assertions that are invoked when a function is called. But in most cases it would be way too slow to turn on everywhere.
>> I'm fine to have a discussion on things like covariance vs. contravariance, or what form of duck typing are acceptable, etc.
> 
> I?m not particularly knowledgable about the actual workings of a type system and
> covariance vs contravariance and the like. My main concern there is having a
> single reality. The meaning of something shouldn't change because I used a
> different interpreter/linter/whatever. Beyond that I don't know enough to have
> an opinion on the actual semantics.
> 
> Yeah, I regret writing it so vaguely already. Having Alex Gaynor open with "I'm strongly opposed [to] this" is a great joy killer. :-)
> 
> I just really don't want to have to redundantly write up a specification for all the details of mypy's type checking rules in PEP-worthy English. But I'm fine with discussing whether List[str] is a subclass or a superclass of List[object] and how to tell the difference.

Understood! And really the most important thing I'm worried about isn?t that
there is some sort of code in the stdlib or in the interpreter just that there
is an authoritative source of what stuff means.

> 
> Still, different linters exist and I don't hear people complain about that. I would also be okay if PyCharm's interpretation of the finer points of the type checking syntax was subtly different from mypy's. In fact I would be surprised if they weren't sometimes in disagreement. Heck, PyPy doesn't give *every* Python program the same meaning as CPython, and that's a feature. :-)
> 

Depends on what is meant by "meaning" I suppose. Generally in those linters or
PyPy itself if there is a different *meaningful* result (for instance if
print was defaulting to sys.stderr) then CPython (incl docs) acts as the
authoritative source of what ``print()`` means (in this case writing to
sys.stdout).

I'm also generally OK with deferring possible code/interpreter changes to add
actual type checking until a later point in time. If there's a defined semantics
to what those annotations mean than third parties can experiment and do things
with it and those different things can be looked at adding/incorporating into
Python proper in 3.6 (or 3.7, or whatever).

Honestly I think that probably the things I was worried about is sufficiently
allayed given that it appears I was reading more into the vaguness and the
optionally different interpretations than what was meant and I don't want to
keep harping on it :) As long as there's some single source of what List[str]
or what have you means than I'm pretty OK with it all.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/0746dec6/attachment-0001.html>

From rosuav at gmail.com  Thu Aug 14 02:32:04 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 14 Aug 2014 10:32:04 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <CAPTjJmraSZrAt7hhsoq5otCLJ_KgVZj9Dy9jP-r2U+0oub2w-Q@mail.gmail.com>

On Thu, Aug 14, 2014 at 5:44 AM, Guido van Rossum <guido at python.org> wrote:
>   from typing import List, Dict
>
>   def word_count(input: List[str]) -> Dict[str, int]:
>       result = {}  #type: Dict[str, int]
>       for line in input:
>           for word in line.split():
>               result[word] = result.get(word, 0) + 1
>       return result

I strongly support the concept of standardized typing information.
There'll be endless bikeshedding on names, though - personally, I
don't like the idea of "from typing import ..." as there's already a
"types" module and I think it'd be confusing. (Also, "mypy" sounds
like someone's toy reimplementation of Python, which it does seem to
be :) but that's not really well named for "type checker using stdlib
annotations".) But I think the idea is excellent, and it deserves
stdlib support.

The cast notation sounds to me like it's what Pike calls a "soft cast"
- it doesn't actually *change* anything (contrast a C or C++ type
cast, where (float)42 is 42.0), it just says to the copmiler/type
checker "this thing is actually now this type". If the naming is clear
on this point, it leaves open the possibility of actual recursive
casting - where casting a List[str] to List[int] is equivalent to
[int(x) for x in lst]. Whether or not that's a feature worth adding
can be decided in the distant future :)

+1 on the broad proposal. +0.5 on defining the notation while leaving
the actual type checking to an external program.

ChrisA

From haoyi.sg at gmail.com  Thu Aug 14 02:53:26 2014
From: haoyi.sg at gmail.com (Haoyi Li)
Date: Wed, 13 Aug 2014 17:53:26 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAPTjJmraSZrAt7hhsoq5otCLJ_KgVZj9Dy9jP-r2U+0oub2w-Q@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAPTjJmraSZrAt7hhsoq5otCLJ_KgVZj9Dy9jP-r2U+0oub2w-Q@mail.gmail.com>
Message-ID: <CALruUQKn8ApU83twA4ADperXA6QcmF=eBE4e_v4ByyO72KM89Q@mail.gmail.com>

> Both solutions have merit, but the idea of some implementations of the
type checker having covariance and some contravariance is fairly disturbing.

Why can't we have both? That's the only way to properly type things, since
immutable-get-style APIs are always going to be convariant, set-only style
APIs (e.g. a function that takes 1 arg and returns None) are going to be
contravariant and mutable get-set APIs (like most python collections)
should really be invariant.


On Wed, Aug 13, 2014 at 5:32 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Thu, Aug 14, 2014 at 5:44 AM, Guido van Rossum <guido at python.org>
> wrote:
> >   from typing import List, Dict
> >
> >   def word_count(input: List[str]) -> Dict[str, int]:
> >       result = {}  #type: Dict[str, int]
> >       for line in input:
> >           for word in line.split():
> >               result[word] = result.get(word, 0) + 1
> >       return result
>
> I strongly support the concept of standardized typing information.
> There'll be endless bikeshedding on names, though - personally, I
> don't like the idea of "from typing import ..." as there's already a
> "types" module and I think it'd be confusing. (Also, "mypy" sounds
> like someone's toy reimplementation of Python, which it does seem to
> be :) but that's not really well named for "type checker using stdlib
> annotations".) But I think the idea is excellent, and it deserves
> stdlib support.
>
> The cast notation sounds to me like it's what Pike calls a "soft cast"
> - it doesn't actually *change* anything (contrast a C or C++ type
> cast, where (float)42 is 42.0), it just says to the copmiler/type
> checker "this thing is actually now this type". If the naming is clear
> on this point, it leaves open the possibility of actual recursive
> casting - where casting a List[str] to List[int] is equivalent to
> [int(x) for x in lst]. Whether or not that's a feature worth adding
> can be decided in the distant future :)
>
> +1 on the broad proposal. +0.5 on defining the notation while leaving
> the actual type checking to an external program.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/a1d1c34c/attachment.html>

From raymond.hettinger at gmail.com  Thu Aug 14 02:59:28 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Wed, 13 Aug 2014 17:59:28 -0700
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
In-Reply-To: <CAP7+vJJKP=1QfQrUaQ0Rq5Et-o81+MmzWJaeUT8MRkGvvhwRtQ@mail.gmail.com>
References: <mailman.72122.1407959084.18129.python-ideas@python.org>
 <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com>
 <CAP7+vJJKP=1QfQrUaQ0Rq5Et-o81+MmzWJaeUT8MRkGvvhwRtQ@mail.gmail.com>
Message-ID: <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com>


On Aug 13, 2014, at 2:09 PM, Guido van Rossum <guido at python.org> wrote:

>>  The goal is to make it possible to add
>> type checking annotations to 3rd party modules (and even to the stdlib)
>> while allowing unaltered execution of the program by the (unmodified)
>> Python 3.5 interpreter.
> 
> Is the goal to "make it possible" or would it quickly become required
> (i.e. any time you write normal, readable Python, it would break 
> someone's optimizer, refactorer, linter, etc.?)
> 
> Whoa, whoa. That's not at all the idea. Currently *nobody* uses type annotations because there's no standard notation. My goal is to enable their use by proposing a standard, nothing more.
>  ...
> Mypy lets you create stub modules that shadow stdlib (and other) modules during typechecking, and it comes with a modest set of standard stubs for some of the most popular stdlib modules. In most cases it's actually much more effective to create a stub than to add the annotations to the stdlib source code -- the stdlib code itself often is difficult to type check due to various optimizations or backwards compatibility concerns, but writing stubs is relatively straightforward, and the stubs will give useful guidance to users of the stdlib who care to run mypy over their own code.

+1 from me :-)

In the PEP, please make it clear that you don't want any
overly enthusiastic coredevs injecting the annotations into
the standard library (have no doubt, some one would do it)
and demanding changes to existing APIs where annotations didn't
fit neatly (like Larry's proposal to change the long-standing APIs
to use his nullable ints by adding None to optional int signatures).


Raymond


P.S.  I would really like for the annotations to grow some way
to communicate exceptions as well as return types (i.e. that
list.index can raise a ValueError and list.pop can raise an IndexError).
This would be only for the exceptions directly added by a function or method,
not ones raised by the data (which is something the function can't control).


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/d6ad64e4/attachment.html>

From lukasz at langa.pl  Thu Aug 14 03:00:53 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Wed, 13 Aug 2014 18:00:53 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <75CA8559-0A4C-4BA0-94AA-A69C0CD56AE7@langa.pl>

It?s great to see this finally happening!
I did some research on existing optional-typing approaches [1]. What I learned in the process was that linting is the most important use case for optional typing; runtime checks is too little, too late.

That being said, having optional runtime checks available *is* also important. Used in staging environments and during unit testing, this case is able to cover cases obscured by meta-programming. Implementations like ?obiwan? and ?pytypedecl? show that providing a runtime type checker is absolutely feasible.

The function annotation syntax currently supported in Python 3.4 is not well-suited for typing. This is because users expect to be able to operate on the types they know. This is currently not feasible because:
1. forward references are impossible
2. generics are impossible without custom syntax (which is the reason Mypy?s Dict exists)
3. optional types are clumsy to express (Optional[int] is very verbose for a use case this common)
4. union types are clumsy to express

All those problems are elegantly solved by Google?s pytypedecl via moving type information to a separate file. Because for our use case that would not be an acceptable approach, my intuition would be to:

1. Provide support for generics (understood as an answer to the question: ?what does this collection contain??) in Abstract Base Classes. That would be a PEP in itself.
2. Change the function annotation syntax so that it?s not executed at import time but rather treated as strings. This solves forward references and enables us to?
3. Extend the function annotation syntax with first-class generics support (most languages like "list<str>?)
4. Extend the function annotation syntax with first-class union type support. pytypedecl simply uses ?int or None?, which I find very elegant.
5. Speaking of None, possibly further extend the function annotation syntax with first-class optionality support. In the Facebook codebase in Hack we have tens of thousands of optional ints (nevermind other optional types!), this is a case that?s going to be used all the time. Hack uses ?int, that?s the most succinct style you can get. Yes, it?s special but None is a special type, too.

All in all, I believe Mypy has the highest chance of becoming our typing linter, which is great! I just hope we can improve on the syntax, which is currently lacking. Also, reusing our existing ABCs where applicable would be nice. With Mypy?s typing module I feel like we?re going to get a new, orthogonal set of ABCs, which will confuse users to no end. Finally, the runtime type checker would make the ecosystem complete.

This is just the beginning of the open issues I was juggling with and the reason my own try at the PEP was coming up slower than I?d like.

[1] You can find a summary of examples I looked at here: http://lukasz.langa.pl/typehinting/

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

On Aug 13, 2014, at 12:44 PM, Guido van Rossum <guido at python.org> wrote:

> [There is no TL;DR other than the subject line. Please read the whole thing before replying. I do have an appendix with some motivations for adding type annotations at the end.]
> 
> Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy: http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can learn from Haskell (and other languages); yesterday he gave the same talk at Dropbox. The talk is online (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad strokes comes down to three suggestions:
> 
>   (a) Python should adopt mypy's syntax for function annotations
>   (b) Python's use of mutabe containers by default is wrong
>   (c) Python should adopt some kind of Abstract Data Types
> 
> Proposals (b) and (c) don't feel particularly actionable (if you disagree please start a new thread, I'd be happy to discuss these further if there's interest) but proposal (a) feels right to me.
> 
> So what is mypy?  It is a static type checker for Python written by Jukka for his Ph.D. thesis. The basic idea is that you add type annotations to your program using some custom syntax, and when running your program using the mypy interpreter, type errors will be found during compilation (i.e., before the program starts running).
> 
> The clever thing here is that the custom syntax is actually valid Python 3, using (mostly) function annotations: your annotated program will still run with the regular Python 3 interpreter. In the latter case there will be no type checking, and no runtime overhead, except to evaluate the function annotations (which are evaluated at function definition time but don't have any effect when the function is called).
> 
> In fact, it is probably more useful to think of mypy as a heavy-duty linter than as a compiler or interpreter; leave the type checking to mypy, and the execution to Python. It is easy to integrate mypy into a continuous integration setup, for example.
> 
> To read up on mypy's annotation syntax, please see the mypy-lang.org website. Here's just one complete example, to give a flavor:
> 
>   from typing import List, Dict
> 
>   def word_count(input: List[str]) -> Dict[str, int]:
>       result = {}  #type: Dict[str, int]
>       for line in input:
>           for word in line.split():
>               result[word] = result.get(word, 0) + 1
>       return result
> 
> Note that the #type: comment is part of the mypy syntax; mypy uses comments to declare types in situations where no syntax is available -- although this particular line could also be written as follows:
> 
>     result = Dict[str, int]()
> 
> Either way the entire function is syntactically valid Python 3, and a suitable implementation of typing.py (containing class definitions for List and Dict, for example) can be written to make the program run correctly. One is provided as part of the mypy project.
> 
> I should add that many of mypy's syntactic choices aren't actually new. The basis of many of its ideas go back at least a decade: I blogged about this topic in 2004 (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- see also the two followup posts linked from the top there).
> 
> I'll emphasize once more that mypy's type checking happens in a separate pass: no type checking happens at run time (other than what the interpreter already does, like raising TypeError on expressions like 1+"1").
> 
> There's a lot to this proposal, but I think it's possible to get a PEP written, accepted and implemented in time for Python 3.5, if people are supportive. I'll go briefly over some of the action items.
> 
> (1) A change of direction for function annotations
> 
> PEP 3107, which introduced function annotations, is intentional non-committal about how function annotations should be used. It lists a number of use cases, including but not limited to type checking. It also mentions some rejected proposals that would have standardized either a syntax for indicating types and/or a way for multiple frameworks to attach different annotations to the same function. AFAIK in practice there is little use of function annotations in mainstream code, and I propose a conscious change of course here by stating that annotations should be used to indicate types and to propose a standard notation for them.
> 
> (We may have to have some backwards compatibility provision to avoid breaking code that currently uses annotations for some other purpose. Fortunately the only issue, at least initially, will be that when running mypy to type check such code it will produce complaints about the annotations; it will not affect how such code is executed by the Python interpreter. Nevertheless, it would be good to deprecate such alternative uses of annotations.)
> 
> (2) A specification for what to add to Python 3.5
> 
> There needs to be at least a rough consensus on the syntax for annotations, and the syntax must cover a large enough set of use cases to be useful. Mypy is still under development, and some of its features are still evolving (e.g. unions were only added a few weeks ago). It would be possible to argue endlessly about details of the notation, e.g. whether to use 'list' or 'List', what either of those means (is a duck-typed list-like type acceptable?) or how to declare and use type variables, and what to do with functions that have no annotations at all (mypy currently skips those completely).
> 
> I am proposing that we adopt whatever mypy uses here, keeping discussion of the details (mostly) out of the PEP. The goal is to make it possible to add type checking annotations to 3rd party modules (and even to the stdlib) while allowing unaltered execution of the program by the (unmodified) Python 3.5 interpreter. The actual type checker will not be integrated with the Python interpreter, and it will not be checked into the CPython repository. The only thing that needs to be added to the stdlib is a copy of mypy's typing.py module. This module defines several dozen new classes (and a few decorators and other helpers) that can be used in expressing argument types. If you want to type-check your code you have to download and install mypy and run it separately.
> 
> The curious thing here is that while standardizing a syntax for type annotations, we technically still won't be adopting standard rules for type checking. This is intentional. First of all, fully specifying all the type checking rules would make for a really long and boring PEP (a much better specification would probably be the mypy source code). Second, I think it's fine if the type checking algorithm evolves over time, or if variations emerge. The worst that can happen is that you consider your code correct but mypy disagrees; your code will still run.
> 
> That said, I don't want to completely leave out any specification. I want the contents of the typing.py module to be specified in the PEP, so that it can be used with confidence. But whether mypy will complain about your particular form of duck typing doesn't have to be specified by the PEP. Perhaps as mypy evolves it will take options to tell it how to handle certain edge cases. Forks of mypy (or entirely different implementations of type checking based on the same annotation syntax) are also a possibility. Maybe in the distant future a version of Python will take a different stance, once we have more experience with how this works out in practice, but for Python 3.5 I want to restrict the scope of the upheaval.
> 
> Appendix -- Why Add Type Annotations?
> 
> The argument between proponents of static typing and dynamic typing has been going on for many decades. Neither side is all wrong or all right. Python has traditionally fallen in the camp of extremely dynamic typing, and this has worked well for most users, but there are definitely some areas where adding type annotations would help.
> 
> - Editors (IDEs) can benefit from type annotations; they can call out obvious mistakes (like misspelled method names or inapplicable operations) and suggest possible method names. Anyone who has used IntelliJ or Xcode will recognize how powerful these features are, and type annotations will make such features more useful when editing Python source code.
> 
> - Linters are an important tool for teams developing software. A linter doesn't replace a unittest, but can find certain types of errors better or quicker. The kind of type checking offered by mypy works much like a linter, and has similar benefits; but it can find problems that are beyond the capabilities of most linters.
> 
> - Type annotations are useful for the human reader as well! Take the above word_count() example. How long would it have taken you to figure out the types of the argument and return value without annotations? Currently most people put the types in their docstrings; developing a standard notation for type annotations will reduce the amount of documentation that needs to be written, and running the type checker might find bugs in the documentation, too. Once a standard type annotation syntax is introduced, it should be simple to add support for this notation to documentation generators like Sphinx.
> 
> - Refactoring. Bob's talk has a convincing example of how type annotations help in (manually) refactoring code. I also expect that certain automatic refactorings will benefit from type annotations -- imagine a tool like 2to3 (but used for some other transformation) augmented by type annotations, so it will know whether e.g. x.keys() is referring to the keys of a dictionary or not.
> 
> - Optimizers. I believe this is actually the least important application, certainly initially. Optimizers like PyPy or Pyston wouldn't be able to fully trust the type annotations, and they are better off using their current strategy of optimizing code based on the types actually observed at run time. But it's certainly feasible to imagine a future optimizer also taking type annotations into account.
> 
> -- 
> --Guido "I need a new hobby" van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/74d7c9fe/attachment-0001.html>

From greg at krypto.org  Thu Aug 14 03:09:23 2014
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 13 Aug 2014 18:09:23 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <CAGE7PN+TyBsUfE_yjZR5VLsJ4oNkJ6zgr_gR_KkGZGvSGYSkTQ@mail.gmail.com>

On Wed, Aug 13, 2014 at 12:44 PM, Guido van Rossum <guido at python.org> wrote:

> [There is no TL;DR other than the subject line. Please read the whole
> thing before replying. I do have an appendix with some motivations for
> adding type annotations at the end.]
>
> Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man
> of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy:
> http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python
> can learn from Haskell (and other languages); yesterday he gave the same
> talk at Dropbox. The talk is online (
> https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad
> strokes comes down to three suggestions:
>
>   (a) Python should adopt mypy's syntax for function annotations
>   (b) Python's use of mutabe containers by default is wrong
>   (c) Python should adopt some kind of Abstract Data Types
>
> Proposals (b) and (c) don't feel particularly actionable (if you disagree
> please start a new thread, I'd be happy to discuss these further if there's
> interest) but proposal (a) feels right to me.
>
> So what is mypy?  It is a static type checker for Python written by Jukka
> for his Ph.D. thesis. The basic idea is that you add type annotations to
> your program using some custom syntax, and when running your program using
> the mypy interpreter, type errors will be found during compilation (i.e.,
> before the program starts running).
>
> The clever thing here is that the custom syntax is actually valid Python
> 3, using (mostly) function annotations: your annotated program will still
> run with the regular Python 3 interpreter. In the latter case there will be
> no type checking, and no runtime overhead, except to evaluate the function
> annotations (which are evaluated at function definition time but don't have
> any effect when the function is called).
>
> In fact, it is probably more useful to think of mypy as a heavy-duty
> linter than as a compiler or interpreter; leave the type checking to mypy,
> and the execution to Python. It is easy to integrate mypy into a continuous
> integration setup, for example.
>
> To read up on mypy's annotation syntax, please see the mypy-lang.org
> website. Here's just one complete example, to give a flavor:
>
>   from typing import List, Dict
>
>   def word_count(input: List[str]) -> Dict[str, int]:
>       result = {}  #type: Dict[str, int]
>       for line in input:
>           for word in line.split():
>               result[word] = result.get(word, 0) + 1
>       return result
>
> Note that the #type: comment is part of the mypy syntax; mypy uses
> comments to declare types in situations where no syntax is available --
> although this particular line could also be written as follows:
>
>     result = Dict[str, int]()
>
> Either way the entire function is syntactically valid Python 3, and a
> suitable implementation of typing.py (containing class definitions for List
> and Dict, for example) can be written to make the program run correctly.
> One is provided as part of the mypy project.
>
> I should add that many of mypy's syntactic choices aren't actually new.
> The basis of many of its ideas go back at least a decade: I blogged about
> this topic in 2004 (
> http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- see also the
> two followup posts linked from the top there).
>
> I'll emphasize once more that mypy's type checking happens in a separate
> pass: no type checking happens at run time (other than what the interpreter
> already does, like raising TypeError on expressions like 1+"1").
>
> There's a lot to this proposal, but I think it's possible to get a PEP
> written, accepted and implemented in time for Python 3.5, if people are
> supportive. I'll go briefly over some of the action items.
>
> *(1) A change of direction for function annotations*
>
> PEP 3107 <http://legacy.python.org/dev/peps/pep-3107/>, which introduced
> function annotations, is intentional non-committal about how function
> annotations should be used. It lists a number of use cases, including but
> not limited to type checking. It also mentions some rejected proposals that
> would have standardized either a syntax for indicating types and/or a way
> for multiple frameworks to attach different annotations to the same
> function. AFAIK in practice there is little use of function annotations in
> mainstream code, and I propose a conscious change of course here by stating
> that annotations should be used to indicate types and to propose a standard
> notation for them.
>
> (We may have to have some backwards compatibility provision to avoid
> breaking code that currently uses annotations for some other purpose.
> Fortunately the only issue, at least initially, will be that when running
> mypy to type check such code it will produce complaints about the
> annotations; it will not affect how such code is executed by the Python
> interpreter. Nevertheless, it would be good to deprecate such alternative
> uses of annotations.)
>
> *(2) A specification for what to add to Python 3.5*
>
> There needs to be at least a rough consensus on the syntax for
> annotations, and the syntax must cover a large enough set of use cases to
> be useful. Mypy is still under development, and some of its features are
> still evolving (e.g. unions were only added a few weeks ago). It would be
> possible to argue endlessly about details of the notation, e.g. whether to
> use 'list' or 'List', what either of those means (is a duck-typed list-like
> type acceptable?) or how to declare and use type variables, and what to do
> with functions that have no annotations at all (mypy currently skips those
> completely).
>
> I am proposing that we adopt whatever mypy uses here, keeping discussion
> of the details (mostly) out of the PEP. The goal is to make it possible to
> add type checking annotations to 3rd party modules (and even to the stdlib)
> while allowing unaltered execution of the program by the (unmodified)
> Python 3.5 interpreter. The actual type checker will not be integrated with
> the Python interpreter, and it will not be checked into the CPython
> repository. The only thing that needs to be added to the stdlib is a copy
> of mypy's typing.py module. This module defines several dozen new classes
> (and a few decorators and other helpers) that can be used in expressing
> argument types. If you want to type-check your code you have to download
> and install mypy and run it separately.
>
> The curious thing here is that while standardizing a syntax for type
> annotations, we technically still won't be adopting standard rules for type
> checking. This is intentional. First of all, fully specifying all the type
> checking rules would make for a really long and boring PEP (a much better
> specification would probably be the mypy source code). Second, I think it's
> fine if the type checking algorithm evolves over time, or if variations
> emerge. The worst that can happen is that you consider your code correct
> but mypy disagrees; your code will still run.
>
> That said, I don't want to *completely* leave out any specification. I
> want the contents of the typing.py module to be specified in the PEP, so
> that it can be used with confidence. But whether mypy will complain about
> your particular form of duck typing doesn't have to be specified by the
> PEP. Perhaps as mypy evolves it will take options to tell it how to handle
> certain edge cases. Forks of mypy (or entirely different implementations of
> type checking based on the same annotation syntax) are also a possibility.
> Maybe in the distant future a version of Python will take a different
> stance, once we have more experience with how this works out in practice,
> but for Python 3.5 I want to restrict the scope of the upheaval.
>
>
> *Appendix -- Why Add Type Annotations?*
> The argument between proponents of static typing and dynamic typing has
> been going on for many decades. Neither side is all wrong or all right.
> Python has traditionally fallen in the camp of extremely dynamic typing,
> and this has worked well for most users, but there are definitely some
> areas where adding type annotations would help.
>
> - Editors (IDEs) can benefit from type annotations; they can call out
> obvious mistakes (like misspelled method names or inapplicable operations)
> and suggest possible method names. Anyone who has used IntelliJ or Xcode
> will recognize how powerful these features are, and type annotations will
> make such features more useful when editing Python source code.
>
> - Linters are an important tool for teams developing software. A linter
> doesn't replace a unittest, but can find certain types of errors better or
> quicker. The kind of type checking offered by mypy works much like a
> linter, and has similar benefits; but it can find problems that are beyond
> the capabilities of most linters.
>
> - Type annotations are useful for the human reader as well! Take the above
> word_count() example. How long would it have taken you to figure out the
> types of the argument and return value without annotations? Currently most
> people put the types in their docstrings; developing a standard notation
> for type annotations will reduce the amount of documentation that needs to
> be written, and running the type checker might find bugs in the
> documentation, too. Once a standard type annotation syntax is introduced,
> it should be simple to add support for this notation to documentation
> generators like Sphinx.
>
> - Refactoring. Bob's talk has a convincing example of how type annotations
> help in (manually) refactoring code. I also expect that certain automatic
> refactorings will benefit from type annotations -- imagine a tool like 2to3
> (but used for some other transformation) augmented by type annotations, so
> it will know whether e.g. x.keys() is referring to the keys of a dictionary
> or not.
>
> - Optimizers. I believe this is actually the least important application,
> certainly initially. Optimizers like PyPy or Pyston
> <https://github.com/dropbox/pyston> wouldn't be able to fully trust the
> type annotations, and they are better off using their current strategy of
> optimizing code based on the types actually observed at run time. But it's
> certainly feasible to imagine a future optimizer also taking type
> annotations into account.
>
> --
> --Guido "I need a new hobby" van Rossum (python.org/~guido)
>
>
First, I am really happy that you are interested in this and that your
point (2) of what you want to see done is very limited and acknowledges
that it isn't going to specify everything!  Because that isn't possible. :)

Unfortunately I feel that adding syntax like this to the language itself is
not useful without enforcement because it that leads to code being written
with unintentionally incorrect annotations that winds up deployed in
libraries that later become a problem as soon as an actual analysis tool
attempts to run over something that uses that unknowingly incorrectly
specified code in a place where it cannot be easily updated (like the
standard library).

At the summit in Montreal earlier this year ?ukasz Langa (cc'd) volunteered
to lead writing the PEP on Python type hinting based on the many existing
implementations of such things (including mypy, cython, numba and pytypedecl
<https://github.com/google/pytypedecl>). I believe he has an initial draft
he intends to send out soon. I'll let him speak to that.

Looks like ?ukasz already responded, I'll stop writing now and go read
that. :)

Personal opinion from experience trying: You can't express the depth of
types for an interface within the Python language syntax itself (assuming
hacks such as specially formatted comments, strings or docstrings do not
count). Forward references to things that haven't even been defined yet are
common. You often want an ability to specify a duck type interface rather
than a specific type.  I think he has those points covered better than I do.

-gps

PS If anyone want to see a run time type checker make code run at half
speed, look at the one pytypedecl
<https://github.com/google/pytypedecl> offers.
I'm sure it could be sped up, but run-time checkers in an interpreter are
always likely to be slow.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/f917e72f/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Thu Aug 14 03:28:16 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 14 Aug 2014 13:28:16 +1200
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
In-Reply-To: <CAPTjJmraSZrAt7hhsoq5otCLJ_KgVZj9Dy9jP-r2U+0oub2w-Q@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAPTjJmraSZrAt7hhsoq5otCLJ_KgVZj9Dy9jP-r2U+0oub2w-Q@mail.gmail.com>
Message-ID: <53EC10B0.9020105@canterbury.ac.nz>

On 08/14/2014 12:32 PM, Chris Angelico wrote:
> I don't like the idea of "from typing import ..." as there's already a
> "types" module and I think it'd be confusing.

Maybe

    from __statictyping__ import ...

More explicit, and being a dunder name suggests that it's
something special that linters should ignore if they don't
understand it.

-- 
Greg


From abarnert at yahoo.com  Thu Aug 14 03:26:56 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 13 Aug 2014 18:26:56 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
In-Reply-To: <loom.20140813T222929-838@post.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org> 
Message-ID: <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com>

On Wednesday, August 13, 2014 1:30 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:




>I'm strongly opposed this, for a few reasons.


[...]

>Python's type system isn't very good. It lacks many features of more powerful
>systems such as algebraic data types, interfaces, and parametric polymorphism.
>Despite this, it works pretty well because of Python's dynamic typing. I
>strongly believe that attempting to enforce the existing type system would be a
>real shame.

This is my main concern, but I'd phrase it very differently.


First, Python's type system _is_ powerful, but only because it's dynamic. Duck typing simulates parametric polymorphism perfectly, disjunction types as long as they don't include themselves recursively,?algebraic data types in some but not all cases, etc. Simple (Java-style) generics, of the kind that Guido seems to be proposing, are not nearly as flexible. That's the problem.

On the other hand, even though these types only cover a small portion of the space of Python's implicit type system, a lot of useful functions fall within that small portion. As long as you can just leave the rest of the program untyped, and there are no boundary problems, there's no real risk.

On the third hand, what worries me is this:

> Mypy has a cast() operator that you can use to shut it up when you (think you) know the conversion is safe.

Why do we need casts? You shouldn't be trying to enforce static typing in a part of the program whose static type isn't sound. Languages like Java and C++ have no choice; Python does, so why not take advantage of it?

The standard JSON example seems appropriate here. What's the return type of?json.loads? In Haskell, you write a pretty trivial JSONThing ADT, and you return a JSONThing that's an Object (which means its value maps String to JSONThing). In Python today, you return a dict, and use it exactly the same as in Haskell, except that you can't verify its soundness at compile time. In Java or C++, it's? what??The sound option is a?special JSONThing that has separate getObjectMemberString and getArrayMemberString and getObjectMemberInt, which is incredibly painful to use. A plain old Dict[String, Object] looks simple, but it means you have to downcast all over the place to do anything, making it completely unsound, and still unpleasant.?The official Java json.org library gives you a hybrid between the two that manages to be neither sound nor user-friendly. And of course there are libraries for many poor static languages (especially C++) that try to fake duck
typing as far as possible for their JSON objects, which is of course nowhere near as far as Python gets for free.

From greg.ewing at canterbury.ac.nz  Thu Aug 14 03:33:16 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 14 Aug 2014 13:33:16 +1200
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
In-Reply-To: <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com>
References: <mailman.72122.1407959084.18129.python-ideas@python.org>
 <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com>
 <CAP7+vJJKP=1QfQrUaQ0Rq5Et-o81+MmzWJaeUT8MRkGvvhwRtQ@mail.gmail.com>
 <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com>
Message-ID: <53EC11DC.7070007@canterbury.ac.nz>

On 08/14/2014 12:59 PM, Raymond Hettinger wrote:
>
> I would really like for the annotations to grow some way
> to communicate exceptions as well as return types

I'd hate for exception typing to be compulsory, though,
since that turned out to be extremely annoying in Java.

It would be even worse in Python, since there's no
branch of the exception hierarchy corresponding to
Java's RuntimeError for things that can be raised
from anywhere.

-- 
Greg


From abarnert at yahoo.com  Thu Aug 14 03:39:15 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 13 Aug 2014 18:39:15 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>

On Wednesday, August 13, 2014 12:45 PM, Guido van Rossum <guido at python.org> wrote:


>? def word_count(input: List[str]) -> Dict[str, int]:
>????? result = {}? #type: Dict[str, int]
>????? for line in input:
>????????? for word in line.split():
>????????????? result[word] = result.get(word, 0) + 1
>????? return result


I just realized why this bothers me.

This function really, really ought to be taking an Iterable[String] (except that we don't have a String ABC). If you hadn't statically typed it, it would work just fine with, say, a text file?or, for that matter, a binary file. By restricting it to List[str], you've made it a lot less usable, for no visible benefit.

And, while this is less serious, I don't think it should be guaranteeing that the result is a Dict rather than just some kind of Mapping. If you want to change the implementation tomorrow to return some kind of proxy or a tree-based sorted mapping, you can't do so without breaking all the code that uses your function.

And if even Guido, in the motivating example for this feature, is needlessly restricting the usability and future flexibility of a function, I suspect it may be a much bigger problem in practice.


This example also shows exactly what's wrong with simple generics: if this function takes an Iterable[String], it doesn't just return a Mapping[String, int], it returns a Mapping of _the same String type_. If your annotations can't express that, any value that passes through this function loses type information.?

And not being able to tell whether the keys in word_count(f) are str or bytes *even if you know that f was a text file* seems like a pretty major loss.

From guido at python.org  Thu Aug 14 03:42:38 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 18:42:38 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CALruUQKn8ApU83twA4ADperXA6QcmF=eBE4e_v4ByyO72KM89Q@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAPTjJmraSZrAt7hhsoq5otCLJ_KgVZj9Dy9jP-r2U+0oub2w-Q@mail.gmail.com>
 <CALruUQKn8ApU83twA4ADperXA6QcmF=eBE4e_v4ByyO72KM89Q@mail.gmail.com>
Message-ID: <CAP7+vJK38Y+M0+ghqk1mPHS4s4Q_vhgqhBAGk26R=YHFKt6jbg@mail.gmail.com>

On Wed, Aug 13, 2014 at 5:53 PM, Haoyi Li <haoyi.sg at gmail.com> wrote:

> > Both solutions have merit, but the idea of some implementations of the
> type checker having covariance and some contravariance is fairly
> disturbing.
>
> Why can't we have both? That's the only way to properly type things, since
> immutable-get-style APIs are always going to be convariant, set-only style
> APIs (e.g. a function that takes 1 arg and returns None) are going to be
> contravariant and mutable get-set APIs (like most python collections)
> should really be invariant.
>

That makes sense. Can you put something in the mypy tracker about this? (Or
send a pull request. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/19aba365/attachment-0001.html>

From apalala at gmail.com  Thu Aug 14 03:43:54 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Wed, 13 Aug 2014 21:13:54 -0430
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+AV0VtXf4jeSFokkVF4EqUQG0mQT3cfKuttrQDwkchPA@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAN1YFWtvpmRauVr6qH+Xv1HuDBuJdZppRY6FZ+JFBViNvH-7_A@mail.gmail.com>
 <CAP7+vJ+AV0VtXf4jeSFokkVF4EqUQG0mQT3cfKuttrQDwkchPA@mail.gmail.com>
Message-ID: <CAN1YFWtXET9OrZV1ssohieo5hF00K90ihb6nKkeX=dw=Ad11GA@mail.gmail.com>

On Wed, Aug 13, 2014 at 6:41 PM, Guido van Rossum <guido at python.org> wrote:

> Actually, mypy already has a solution. There's a codec (
> https://github.com/JukkaL/mypy/tree/master/mypy/codec) that you can use
> which transforms Python-2-with-annotations into vanilla Python 2. It's not
> an ideal solution, but it can work in cases where you absolutely have to
> have state of the art Python 3.5 type checking *and* backwards
> compatibility with Python 2.
>

It can't be a solution because it's a hack...

Cheers,

-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/c5b134c1/attachment.html>

From greg.ewing at canterbury.ac.nz  Thu Aug 14 03:44:13 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 14 Aug 2014 13:44:13 +1200
Subject: [Python-ideas] Proposal: Use mypy syntax
	for	function	annotations
In-Reply-To: <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com>
Message-ID: <53EC146D.3090402@canterbury.ac.nz>

On 08/14/2014 01:26 PM, Andrew Barnert wrote:

> In Java or C++, it's? what? The sound option is a special JSONThing that
> has separate getObjectMemberString and getArrayMemberString and
> getObjectMemberInt, which is incredibly painful to use.

That's mainly because Java doesn't let you define your own
types that use convenient syntax such as [] for indexing.

Python doesn't have that problem, so a decent static type
system for Python should let you define a JSONThing class
that's fully type-safe while having a standard mapping
interface.

-- 
Greg

From lukasz at langa.pl  Thu Aug 14 03:51:37 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Wed, 13 Aug 2014 18:51:37 -0700
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
In-Reply-To: <53EC11DC.7070007@canterbury.ac.nz>
References: <mailman.72122.1407959084.18129.python-ideas@python.org>
 <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com>
 <CAP7+vJJKP=1QfQrUaQ0Rq5Et-o81+MmzWJaeUT8MRkGvvhwRtQ@mail.gmail.com>
 <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com>
 <53EC11DC.7070007@canterbury.ac.nz>
Message-ID: <E2B424EB-3716-4657-B157-639E2031E162@langa.pl>

On Aug 13, 2014, at 6:33 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> On 08/14/2014 12:59 PM, Raymond Hettinger wrote:
>> 
>> I would really like for the annotations to grow some way
>> to communicate exceptions as well as return types
> 
> I'd hate for exception typing to be compulsory, though,
> since that turned out to be extremely annoying in Java.
> 
> It would be even worse in Python, since there's no
> branch of the exception hierarchy corresponding to
> Java's RuntimeError for things that can be raised
> from anywhere.

As I understand it, that would serve mostly documentational purpose. The only analysis we could do on that would be to check if an exception is ever caught by the caller somewhere up the stack. This could already be done by finding ?raise? statements and going from there.

But even with well annotated types that would be often hard to infer, limiting the usefulness of that analysis. Moreover, it wouldn?t surface problems when the code is using any type of framework that swallows exceptions to process them for the user.

So that leaves documentation. Now this information already lands in docstrings, which lacks the ?one-obvious-way-to-do-it? feel. However, thanks to the hackiness of that approach, users don?t expect the provided list of exceptions to be comprehensive. That would change with a first-class syntax for them.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/8192d2c2/attachment.html>

From lukasz at langa.pl  Thu Aug 14 03:56:54 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Wed, 13 Aug 2014 18:56:54 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
Message-ID: <55F05150-98D7-42FF-9C13-84D99B921853@langa.pl>

On Aug 13, 2014, at 6:39 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote:

> On Wednesday, August 13, 2014 12:45 PM, Guido van Rossum <guido at python.org> wrote:
> 
>>   def word_count(input: List[str]) -> Dict[str, int]:
>>       result = {}  #type: Dict[str, int]
>>       for line in input:
>>           for word in line.split():
>>               result[word] = result.get(word, 0) + 1
>>       return result
> 
> I just realized why this bothers me.
> 
> This function really, really ought to be taking an Iterable[String]

You do realize String also happens to be an Iterable[String], right? One of my big dreams about Python is that one day we'll drop support for strings being iterable. Nothing of value would be lost and that would enable us to use isinstance(x, Iterable) and more importantly isinstance(x, Sequence). Funny that this surfaces now, too.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/c71b6f91/attachment.html>

From tjreedy at udel.edu  Thu Aug 14 04:27:25 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 13 Aug 2014 22:27:25 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <lsh6r6$rkn$1@ger.gmane.org>

Guido, as requesting, I read your whole post before replying. Please to 
the same. This response is both critical and supportive.

On 8/13/2014 3:44 PM, Guido van Rossum wrote:

> Yesterday afternoon I had an inspiring conversation with Bob Ippolito
> (man of many trades, author of simplejson) and Jukka Lehtosalo (author
> of mypy: http://mypy-lang.org/).

My main concern with static typing is that it tends to be 
anti-duck-typing, while I consider duck-typing to be a major *feature* 
of Python.  The example in the page above is "def fib(n: int):". Fib 
should get an count (non-negative integer) value, but it need not be an 
int, and 'half' the ints do not qualify. Reading the tutorial, I could 
not tell if it supports numbers.Number (which should approximate the 
domain from above.)

Now consider an extended version (after Lucas).

def fib(n, a, b):
     i = 0
     while i <= n:
         print(i,a)
         i += 1
         a, b = b, a+b

The only requirement of a, b is that they be addable. Any numbers should 
be allowed, as in fib(10, 1, 1+1j), but so should fib(5, '0', '1'). 
Addable would be approximated from below by Union(Number, str).

> Bob gave a talk at EuroPython about
> what Python can learn from Haskell (and other languages); yesterday he
> gave the same talk at Dropbox. The talk is online
> (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad
> strokes comes down to three suggestions:
>
>    (a) Python should adopt mypy's syntax for function annotations

-+ Syntax with no meaning is a bit strange. On the other hand, syntax 
not bound to semantics, or at least not bound to just one meaning is 
quite pythonic. '+' has two standard meanings, plus custom meanings 
embodied in .__add__ methods.

+ The current semantics of annotations is that they are added to 
functions objects as .__annotations__ (for whatever use) *and* used as 
part of inspect.signature and included in help(ob) responses.  In other 
words, annotations are already used in the stdlib.

 >>> def f(i:int) -> float: pass

 >>> from inspect import signature as sig
 >>> str(sig(f))
'(i:int) -> float'
 >>> help(f)
Help on function f in module __main__:

f(i:int) -> float

Idle calltips include them also. A appropriately flexible standardized 
notation would enhance this usage and many others.

+-+ I see the point of "The goal is to make it possible to add type 
checking annotations to 3rd party modules (and even to the stdlib) while 
allowing unaltered execution of the program by the  (unmodified) Python 
3.5 interpreter." On the other hand, "pip install mypytyping" is not a 
huge burden. On the third hand, in the stdlib allows use in the stdlib.

>    (b) Python's use of mutabe [mutable] containers by default is wrong

The premise of this is partly wrong and partly obsolete. As far as I can 
remember, Python *syntax* only use tuples, not lists: "except (ex1, 
ex2):", "s % (val1, val2)", etc. The use of lists as the common format 
for data interchange between functions has largely been replaced by 
iterators.  This fact makes Python code much more generic, and 
anti-generic static typing more wrong.

In remaining cases, 'wrong' is as much a philosophical opinion as a fact.

>    (c) Python should adopt some kind of Abstract Data Types

I would have to look at the talk to know what Jukka means.

> Proposals (b) and (c) don't feel particularly actionable (if you
> disagree please start a new thread, I'd be happy to discuss these
> further if there's interest) but proposal (a) feels right to me.

> So what is mypy?  It is a static type checker for Python written by
> Jukka for his Ph.D. thesis. The basic idea is that you add type
> annotations to your program using some custom syntax, and when running
> your program using the mypy interpreter, type errors will be found
> during compilation (i.e., before the program starts running).
>
> The clever thing here is that the custom syntax is actually valid Python
> 3, using (mostly) function annotations: your annotated program will
> still run with the regular Python 3 interpreter. In the latter case
> there will be no type checking, and no runtime overhead, except to
> evaluate the function annotations (which are evaluated at function
> definition time but don't have any effect when the function is called).
>
> In fact, it is probably more useful to think of mypy as a heavy-duty
> linter than as a compiler or interpreter; leave the type checking to
> mypy, and the execution to Python. It is easy to integrate mypy into a
> continuous integration setup, for example.
>
> To read up on mypy's annotation syntax, please see the mypy-lang.org
> <http://mypy-lang.org> website.

I did not see a 'reference' page, but the tutorial comes pretty close.
http://mypy-lang.org/tutorial.html
Beyond that, typings.py would be definitive,
https://github.com/JukkaL/mypy/blob/master/lib-typing/3.2/typing.py

 > Here's just one complete example, to give a flavor:

>    from typing import List, Dict
>
>    def word_count(input: List[str]) -> Dict[str, int]:

The input annotation should be Iterable[str], which mypy does have.

>        result = {}  #type: Dict[str, int]
>        for line in input:
>            for word in line.split():
>                result[word] = result.get(word, 0) + 1
>        return result

The information that input is an Iterable[str] can be used either within 
the definition of word_count or at places where word_count is called.  A 
type aware checker, either in the editor or compiler, could check that 
the only uses of 'input' within the function is as input to functions 
declared to accept an Iterable or in for statements.

Checking that the input to word_count is specifically Iterable[str] as 
opposed to any other Iterable may not be possible.  But I think what can 
be done, including enhancing help information, might be worth it.

For instance, the parameter to s.join is named 'iterable'. Something 
more specific, either 'iterable_of_strings' or 'strings: Iterable[str]' 
would be more helpful. Indeed, there have been people posting on python 
list who thought that 'iterable' means iterable and that .join would 
call str() on each object. I think there are other cases where a 
parameter is given a bland under-informative type name instead of a 
context-specific semantic name just because there was no type annotation 
available. There are places where the opposite problem occurs, too 
specific instead of too general, where iterable parameters are still 
called 'list'.

> Note that the #type: comment is part of the mypy syntax; mypy uses
> comments to declare types in situations where no syntax is available --
> although this particular line could also be written as follows:
>
>      result = Dict[str, int]()
>
> Either way the entire function is syntactically valid Python 3, and a
> suitable implementation of typing.py (containing class definitions for
> List and Dict, for example) can be written to make the program run
> correctly. One is provided as part of the mypy project.
>
> I should add that many of mypy's syntactic choices aren't actually new.
> The basis of many of its ideas go back at least a decade: I blogged
> about this topic in 2004
> (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- see also the
> two followup posts linked from the top there).
>
> I'll emphasize once more that mypy's type checking happens in a separate
> pass: no type checking happens at run time (other than what the
> interpreter already does, like raising TypeError on expressions like 1+"1").
>
> There's a lot to this proposal, but I think it's possible to get a PEP
> written, accepted and implemented in time for Python 3.5, if people are
> supportive. I'll go briefly over some of the action items.
>
> *(1) A change of direction for function annotations*
>
> PEP 3107 <http://legacy.python.org/dev/peps/pep-3107/>, which introduced
> function annotations, is intentional non-committal about how function
> annotations should be used. It lists a number of use cases, including
> but not limited to type checking. It also mentions some rejected
> proposals that would have standardized either a syntax for indicating
> types and/or a way for multiple frameworks to attach different
> annotations to the same function. AFAIK in practice there is little use
> of function annotations in mainstream code, and I propose a conscious
> change of course here by stating that annotations should be used to
> indicate types and to propose a standard notation for them.

There are many uses for type information and I think Python should 
remain neutral among them.

> (We may have to have some backwards compatibility provision to avoid
> breaking code that currently uses annotations for some other purpose.
> Fortunately the only issue, at least initially, will be that when
> running mypy to type check such code it will produce complaints about
> the annotations; it will not affect how such code is executed by the
> Python interpreter. Nevertheless, it would be good to deprecate such
> alternative uses of annotations.)

I can imagine that people who have used annotations might feel a bit 
betrayed by deprecation of a new-in-py3 feature.  But I do not think it 
necessary to do so.  Tools that work with mypy annotations, including 
mypy itself, should only assume mypy typing if typing is imported.  No 
'import typing', no 'Warning: annotation does not follow typing rules." 
  If 'typing' were a package with a 'mypy' module, the door would be 
left open to other 'blessed' typing modules.

> *(2) A specification for what to add to Python 3.5*
>
> There needs to be at least a rough consensus on the syntax for
> annotations, and the syntax must cover a large enough set of use cases
> to be useful. Mypy is still under development, and some of its features
> are still evolving (e.g. unions were only added a few weeks ago). It
> would be possible to argue endlessly about details of the notation, e.g.
> whether to use 'list' or 'List', what either of those means (is a
> duck-typed list-like type acceptable?) or how to declare and use type
> variables, and what to do with functions that have no annotations at all
> (mypy currently skips those completely).
>
> I am proposing that we adopt whatever mypy uses here, keeping discussion
> of the details (mostly) out of the PEP. The goal is to make it possible
> to add type checking annotations to 3rd party modules (and even to the
> stdlib) while allowing unaltered execution of the program by the
> (unmodified) Python 3.5 interpreter. The actual type checker will not be
> integrated with the Python interpreter, and it will not be checked into
> the CPython repository. The only thing that needs to be added to the
> stdlib is a copy of mypy's typing.py module. This module defines several
> dozen new classes (and a few decorators and other helpers) that can be
> used in expressing argument types. If you want to type-check your code
> you have to download and install mypy and run it separately.
>
> The curious thing here is that while standardizing a syntax for type
> annotations, we technically still won't be adopting standard rules for
> type checking.

Fine with me, as that is not the only use.  And even for type checking, 
there is the choice between accept unless clearly wrong, versus reject 
unless clearly right.

 > This is intentional. First of all, fully specifying all
> the type checking rules would make for a really long and boring PEP (a
> much better specification would probably be the mypy source code).
> Second, I think it's fine if the type checking algorithm evolves over
> time, or if variations emerge.

As in the choice between accept unless clearly wrong, versus reject 
unless clearly right.

 > The worst that can happen is that you
> consider your code correct but mypy disagrees; your code will still run.
>
> That said, I don't want to /completely/ leave out any specification. I
> want the contents of the typing.py module to be specified in the PEP, so
> that it can be used with confidence. But whether mypy will complain
> about your particular form of duck typing doesn't have to be specified
> by the PEP. Perhaps as mypy evolves it will take options to tell it how
> to handle certain edge cases. Forks of mypy (or entirely different
> implementations of type checking based on the same annotation syntax)
> are also a possibility. Maybe in the distant future a version of Python
> will take a different stance, once we have more experience with how this
> works out in practice, but for Python 3.5 I want to restrict the scope
> of the upheaval.

As usual, we should review the code before acceptance. It is not clear 
to me how much of the tutorial is implemented, as it says "Some of these 
features might never see the light of day. " ???

> *Appendix -- Why Add Type Annotations?
> *
> The argument between proponents of static typing and dynamic typing has
> been going on for many decades. Neither side is all wrong or all right.
> Python has traditionally fallen in the camp of extremely dynamic typing,
> and this has worked well for most users, but there are definitely some
> areas where adding type annotations would help.

The answer to why on the mypy page is 'easier to find bugs', 'easier 
maintenance'.  I find this under-convincing as sufficient justification 
in itself. I don't think there are many bugs on the tracker due to 
calling functions with the wrong type of object. Logic errors, ignored 
corner cases, and system idiosyncrasies are much more of a problem.

Your broader list is more convincing.

> - Editors (IDEs) can benefit from type annotations; they can call out
> obvious mistakes (like misspelled method names or inapplicable
> operations) and suggest possible method names. Anyone who has used
> IntelliJ or Xcode will recognize how powerful these features are, and
> type annotations will make such features more useful when editing Python
> source code.
>
> - Linters are an important tool for teams developing software. A linter
> doesn't replace a unittest, but can find certain types of errors better
> or quicker. The kind of type checking offered by mypy works much like a
> linter, and has similar benefits; but it can find problems that are
> beyond the capabilities of most linters.

Currently, Python linters do not have standard type annotations to work 
with. I suspect that programs other than mypy would use them if available.

> - Type annotations are useful for the human reader as well! Take the
> above word_count() example. How long would it have taken you to figure
> out the types of the argument and return value without annotations?

Under a minute, including the fact the the annotation was overly 
restrictive.  But then I already know that only a mutation method can 
require a list.

> Currently most people put the types in their docstrings; developing a
> standard notation for type annotations will reduce the amount of
> documentation that needs to be written, and running the type checker
> might find bugs in the documentation, too. Once a standard type
> annotation syntax is introduced, it should be simple to add support for
> this notation to documentation generators like Sphinx.
>
> - Refactoring. Bob's talk has a convincing example of how type
> annotations help in (manually) refactoring code. I also expect that
> certain automatic refactorings will benefit from type annotations --
> imagine a tool like 2to3 (but used for some other transformation)
> augmented by type annotations, so it will know whether e.g. x.keys() is
> referring to the keys of a dictionary or not.
>
> - Optimizers. I believe this is actually the least important
> application, certainly initially. Optimizers like PyPy or Pyston
> <https://github.com/dropbox/pyston> wouldn't be able to fully trust the
> type annotations, and they are better off using their current strategy
> of optimizing code based on the types actually observed at run time. But
> it's certainly feasible to imagine a future optimizer also taking type
> annotations into account.

-- 
Terry Jan Reedy


From abarnert at yahoo.com  Thu Aug 14 04:58:21 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 13 Aug 2014 19:58:21 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax
	for	function	annotations
In-Reply-To: <53EC146D.3090402@canterbury.ac.nz>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <53EC146D.3090402@canterbury.ac.nz>
Message-ID: <3C196232-D06D-4EFC-81BB-58FCBBF86DC2@yahoo.com>

On Aug 13, 2014, at 18:44, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> On 08/14/2014 01:26 PM, Andrew Barnert wrote:
> 
>> In Java or C++, it's? what? The sound option is a special JSONThing that
>> has separate getObjectMemberString and getArrayMemberString and
>> getObjectMemberInt, which is incredibly painful to use.
> 
> That's mainly because Java doesn't let you define your own
> types that use convenient syntax such as [] for indexing.

No it's not, or other languages like C++ (which has operator methods and overloading) wouldn't have the exact same problem, but they do. Look at JsonCpp, for example:

https://github.com/open-source-parsers/jsoncpp/

> Python doesn't have that problem, so a decent static type
> system for Python should let you define a JSONThing class
> that's fully type-safe while having a standard mapping
> interface.

How?

If you go with a single JSONThing type that represents an object, array, number, bool, string, or null, then it can't have a standard mapping interface, because it also needs to have a standard sequence interface, and they conflict. Likewise for number vs. string. The only fully type-safe interface it can have is as_string, as_number, etc. methods (which of course can only check at runtime, so it's no better than using isinstance from Python, and you're forced to do it for every single access.)

What if you go the other way and have separate JSONObject, JSONArray, etc. types? Then all of those problems go away; you can define an unambiguous __getitem__. But what is its return value? The only possibility is a union of all the various types mentioned above, and such a union type has no interface at all. It's only useful if people subvert the type safety by casting.  (I guess you could argue that returning a union type makes your JSON library type safe, it's only every program that ever uses it for anything that's unsafe. But where does that get you?) The only usable type safe interface is separate get_string, get_number, etc. methods in place of __getitem__.

Or you can merge the two together and have a single JSONThing that has both as methods and, for convenience, combined as_object+get, or even as_object+get+as_str.

Also, look at the mutation interfaces for these libraries. They're only marginally tolerable because all variable have obligatory types that you can overload on, which wouldn't be the case in Python.

The alternative is, of course, to come up with a way to avoid type safety. In Swift, you parse a JSON object into a Cocoa NSDictionary, which is a dynamically-typed heterogeneous collection just like a Python dict. There are C++ libraries with a bunch of types that effectively act like Python dict, list, float, str, etc. and try to magically cast when they come into contact with native types. That's the best solution anyone has to dealing with even a dead-simple algebraic data type like JSON in a static language whose type system isn't powerful enough: to try to fake being a duck typed language.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/53ef0e09/attachment.html>

From carlopires at gmail.com  Thu Aug 14 05:06:56 2014
From: carlopires at gmail.com (Carlo Pires)
Date: Thu, 14 Aug 2014 00:06:56 -0300
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lsh6r6$rkn$1@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsh6r6$rkn$1@ger.gmane.org>
Message-ID: <CAO6hKovWpwG2XD9SsvuhQ_Wxf_6DMxdy9CLzh8_Kq0_=_vB5Xg@mail.gmail.com>

I'm very happy to see this happening. "Optional" type checking for python
would be a great addition to the language. For large codebases, use of type
checking can really help.

I also like the idea of using annotations instead of decorators (or
something like that). I'm already using this for python3 [1] in a non
intrusive way, by using assert to disable it on production.

[1] https://pypi.python.org/pypi/optypecheck
-- 
  Carlo Pires
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/cd806c76/attachment.html>

From abarnert at yahoo.com  Thu Aug 14 05:08:12 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 13 Aug 2014 20:08:12 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <55F05150-98D7-42FF-9C13-84D99B921853@langa.pl>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
 <55F05150-98D7-42FF-9C13-84D99B921853@langa.pl>
Message-ID: <BF97718F-EB3A-491B-93A8-66D1C7587D7C@yahoo.com>

On Aug 13, 2014, at 18:56, ?ukasz Langa <lukasz at langa.pl> wrote:

> On Aug 13, 2014, at 6:39 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote:
> 
>> On Wednesday, August 13, 2014 12:45 PM, Guido van Rossum <guido at python.org> wrote:
>> 
>>>   def word_count(input: List[str]) -> Dict[str, int]:
>>>       result = {}  #type: Dict[str, int]
>>>       for line in input:
>>>           for word in line.split():
>>>               result[word] = result.get(word, 0) + 1
>>>       return result
>> 
>> I just realized why this bothers me.
>> 
>> This function really, really ought to be taking an Iterable[String]
> 
> You do realize String also happens to be an Iterable[String], right?

Of course, but that's not a new problem, so I didn't want to bring it up. The fact that the static type checker couldn't reject word_count(f.read()) is annoying, but that's not the fault of the static type checking proposal.

> One of my big dreams about Python is that one day we'll drop support for strings being iterable. Nothing of value would be lost and that would enable us to use isinstance(x, Iterable) and more importantly isinstance(x, Sequence). Funny that this surfaces now, too.

IIRC, str doesn't implement Container, and therefore doesn't implement Sequence, because its __contains__ method is substring match instead of containment. So if you really want to treat sequences of strings separately from strings, you can. If only that really _were_ more important than Iterable, but I think the opposite is true.

But anyway, this is probably off topic, so I'll stop here.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/314f9871/attachment-0001.html>

From lukasz at langa.pl  Thu Aug 14 05:16:04 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Wed, 13 Aug 2014 20:16:04 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <BF97718F-EB3A-491B-93A8-66D1C7587D7C@yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
 <55F05150-98D7-42FF-9C13-84D99B921853@langa.pl>
 <BF97718F-EB3A-491B-93A8-66D1C7587D7C@yahoo.com>
Message-ID: <2016271B-67D2-447D-8E51-369806CBA13A@langa.pl>

On Aug 13, 2014, at 8:08 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Aug 13, 2014, at 18:56, ?ukasz Langa <lukasz at langa.pl> wrote:
> 
>> One of my big dreams about Python is that one day we'll drop support for strings being iterable. Nothing of value would be lost and that would enable us to use isinstance(x, Iterable) and more importantly isinstance(x, Sequence). Funny that this surfaces now, too.
> 
> IIRC, str doesn't implement Container, and therefore doesn't implement Sequence, because its __contains__ method is substring match instead of containment. So if you really want to treat sequences of strings separately from strings, you can.

str and bytes objects respond True to both isinstance(x, Container) and isinstance(x, Sequence).

But you?re right, off topic.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/7dad0859/attachment.html>

From mertz at gnosis.cx  Thu Aug 14 05:28:19 2014
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 13 Aug 2014 20:28:19 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <2016271B-67D2-447D-8E51-369806CBA13A@langa.pl>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
 <55F05150-98D7-42FF-9C13-84D99B921853@langa.pl>
 <BF97718F-EB3A-491B-93A8-66D1C7587D7C@yahoo.com>
 <2016271B-67D2-447D-8E51-369806CBA13A@langa.pl>
Message-ID: <CAEbHw4YhdLMVCApSnnP7ro=vmQ+=09BCSWh4=t_GwB7ztdRL4A@mail.gmail.com>

A long while back I posted a recipe for using annotations for type
checking.  I'm certainly not the first person to do this, and what I did
was deliberately simple:


https://code.activestate.com/recipes/578528-type-checking-using-python-3x-annotations/?in=user-4173018

The approach I used was to use per-function decorators to say that a given
function should be type checked.  The type system I enforce in that recipe
is much less than what mypy allows, but I can't see a real reason that it
couldn't be extended to cover exactly the same range of type specifiers.

The advantage I perceive in this approach is that it is purely optional,
per module and per function.  As well, it doesn't actually require making
ANY change to Python 3.5 to implement it.  Or as a minimal change, an extra
decorator could simply be available in functools or elsewhere in the
standard library, which implemented the full semantics of mypy.

Now admittedly, this would be type checking, but not *static* type
checking.  There may not be an easy way to make a pre-runtime "lint" tool
do the checking there.  On the other hand, as a number of posters have
noted, there's also no way to enforce, e.g. 'Iterable[String]' either
statically.

I'm not the BDFL of course, but I do not really get what advantage there is
to the pre-runtime check that can catch a fairly small subset of type
constraints rather than check at runtime everything that is available then
(as the decorator approach could get you).


On Wed, Aug 13, 2014 at 8:16 PM, ?ukasz Langa <lukasz at langa.pl> wrote:

> On Aug 13, 2014, at 8:08 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
>
> On Aug 13, 2014, at 18:56, ?ukasz Langa <lukasz at langa.pl> wrote:
>
> One of my big dreams about Python is that one day we'll drop support for
> strings being iterable. Nothing of value would be lost and that would
> enable us to use isinstance(x, Iterable) and more importantly isinstance(x,
> Sequence). Funny that this surfaces now, too.
>
>
> IIRC, str doesn't implement Container, and therefore doesn't implement
> Sequence, because its __contains__ method is substring match instead of
> containment. So if you really want to treat sequences of strings separately
> from strings, you can.
>
>
> str and bytes objects respond True to both isinstance(x, Container) and
> isinstance(x, Sequence).
>
> But you?re right, off topic.
>
> --
> Best regards,
> ?ukasz Langa
>
> WWW: http://lukasz.langa.pl/
> Twitter: @llanga
> IRC: ambv on #python-dev
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/0397a52b/attachment.html>

From guido at python.org  Thu Aug 14 05:41:30 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 20:41:30 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
Message-ID: <CAP7+vJ+QR7ajXkMqQ3065VZE-mOMrUqaR_jY2P1EFbqqpPpseA@mail.gmail.com>

On Wed, Aug 13, 2014 at 6:39 PM, Andrew Barnert <
abarnert at yahoo.com.dmarc.invalid> wrote:

> On Wednesday, August 13, 2014 12:45 PM, Guido van Rossum <guido at python.org>
> wrote:
>
> >  def word_count(input: List[str]) -> Dict[str, int]:
> >      result = {}  #type: Dict[str, int]
> >      for line in input:
> >          for word in line.split():
> >              result[word] = result.get(word, 0) + 1
> >      return result
>
> I just realized why this bothers me.
>
> This function really, really ought to be taking an Iterable[String]
> (except that we don't have a String ABC). If you hadn't statically typed
> it, it would work just fine with, say, a text file?or, for that matter, a
> binary file. By restricting it to List[str], you've made it a lot less
> usable, for no visible benefit.
>

Heh. :-) I had wanted to write an additional paragraph explaining that it's
easy to change this to use typing.Iterable instead of typing.List, but I
forgot to add that.


> And, while this is less serious, I don't think it should be guaranteeing
> that the result is a Dict rather than just some kind of Mapping. If you
> want to change the implementation tomorrow to return some kind of proxy or
> a tree-based sorted mapping, you can't do so without breaking all the code
> that uses your function.
>

Yeah, there's a typing.Mapping for that.


> And if even Guido, in the motivating example for this feature, is
> needlessly restricting the usability and future flexibility of a function,
> I suspect it may be a much bigger problem in practice.
>

Well, so it was actually semi-intentional. :-)


> This example also shows exactly what's wrong with simple generics: if this
> function takes an Iterable[String], it doesn't just return a
> Mapping[String, int], it returns a Mapping of _the same String type_. If
> your annotations can't express that, any value that passes through this
> function loses type information.
>

In most cases it really doesn't matter though -- some types are better left
concrete, especially strings and numbers. If you read the mypy docs you'll
find that there are generic types, so that it's possible to define a
function as taking an Iterable[T] and returning a Mapping[T, int]. What's
not currently possible is expressing additional constraints on T such as
that it must be a String. When I last talked to Jukka he explained that he
was going to add something for that too (@Jukka: structured types?).


> And not being able to tell whether the keys in word_count(f) are str or
> bytes *even if you know that f was a text file* seems like a pretty major
> loss.
>

On this point one of us must be confused. Let's assume it's me. :-) Mypy
has a few different IO types that can express the difference between text
and binary files. I think there's some work that needs to be done (and of
course the built-in open() function has a terribly ambiguous return type
:-( ), but it should be possible to say that a text file is an
Interable[str] and a binary file is an Iterable[bytes]. So together with
the structured (?) types it should be possible to specify the signature of
word_count() just as you want it. However, in most cases it's overkill, and
you wouldn't want to do that for most code.

Also, it probably wouldn't work for more realistic examples -- as soon as
you replace the split() method call with something that takes punctuation
into account, you're probably going to write it in a way that works only
for text strings anyway, and very few people will want or need to write the
polymorphic version. (But if they do, mypy has a handy @overload decorator
that they can use. :-)

Anyway, I agree it would be good to make sure that some of these more
advanced things can actually be spelled before we freeze our commitment to
a specific syntax, but let's not assume that just because you can't spell
every possible generic use case it's no good.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/63c1d94e/attachment-0001.html>

From jlehtosalo at gmail.com  Thu Aug 14 06:06:41 2014
From: jlehtosalo at gmail.com (Jukka Lehtosalo)
Date: Wed, 13 Aug 2014 21:06:41 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
Message-ID: <CAA_f+LxPbisNd949SmiKhkQNM_N12e0qvQ+aO9uhn1qQ0ji6+g@mail.gmail.com>

On Wed, Aug 13, 2014 at 6:39 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Wednesday, August 13, 2014 12:45 PM, Guido van Rossum <guido at python.org>
> wrote:
>
>
> >  def word_count(input: List[str]) -> Dict[str, int]:
> >      result = {}  #type: Dict[str, int]
> >      for line in input:
> >          for word in line.split():
> >              result[word] = result.get(word, 0) + 1
> >      return result
>
>
> I just realized why this bothers me.
>
> This function really, really ought to be taking an Iterable[String]
> (except that we don't have a String ABC). If you hadn't statically typed
> it, it would work just fine with, say, a text file?or, for that matter, a
> binary file. By restricting it to List[str], you've made it a lot less
> usable, for no visible benefit.
>
> And, while this is less serious, I don't think it should be guaranteeing
> that the result is a Dict rather than just some kind of Mapping. If you
> want to change the implementation tomorrow to return some kind of proxy or
> a tree-based sorted mapping, you can't do so without breaking all the code
> that uses your function.
>

I see this is a matter of programming style. In a library module, I'd
usually use about as general types as feasible (without making them overly
complex). However, if we have just a simple utility function that's only
used within a single program, declaring everything using abstract types
buys you little, IMHO, but may make things much more complicated. You can
always refactor the code to use more general types if the need arises.
Using simple, concrete types seems to decrease the cognitive load, but
that's just my experience.

Also, programmers don't always read documentation/annotations and can abuse
the knowledge of the concrete return type of any function (they can figure
this out easily by using repr()/type()). In general, as long as dynamically
typed programs may call your function, changing the concrete return type of
a library function risks breaking code that makes too many assumptions.
Thus I'd rather use concrete types for function return types -- but of
course everybody is free to not follow this convention.


> And if even Guido, in the motivating example for this feature, is
> needlessly restricting the usability and future flexibility of a function,
> I suspect it may be a much bigger problem in practice.
>
>
> This example also shows exactly what's wrong with simple generics: if this
> function takes an Iterable[String], it doesn't just return a
> Mapping[String, int], it returns a Mapping of _the same String type_. If
> your annotations can't express that, any value that passes through this
> function loses type information.
>

If I define a subclass X of str, split() still returns a List[str] rather
than List[X], unless I override something, so this wouldn't work with the
above example:

>>> class X(str): pass
...
>>> type(X('x y').split()[0])
<class 'str'>


> And not being able to tell whether the keys in word_count(f) are str or
> bytes *even if you know that f was a text file* seems like a pretty major
> loss.
>

Mypy considers bytes incompatible with str, and vice versa. The annotation
Iterable[str] says that Iterable[bytes] (such as a binary file) would not
be a valid argument. Text files and binary files have different types,
though the return type of open(...) is not inferred correctly right now. It
would be easy to fix this for the most common cases, though.

You could use AnyStr to make the example work with bytes as well:

  def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]:
      result = {}  #type: Dict[AnyStr, int]
      for line in input:
          for word in line.split():
              result[word] = result.get(word, 0) + 1
      return result

Again, if this is just a simple utility function that you use once or
twice, I see no reason to spend a lot of effort in coming up with the most
general signature. Types are an abstraction and they can't express
everything precisely -- there will always be a lot of cases where you can't
express the most general type. However, I think that relatively simple
types work well enough most of the time, and give the most bang for the
buck.

Jukka
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/a243f94a/attachment.html>

From tjreedy at udel.edu  Thu Aug 14 06:21:22 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 14 Aug 2014 00:21:22 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <DCF0B3C2-6067-4856-864A-319CEAA7230B@gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBC3BA.10808@stoneleaf.us>
 <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
 <DCF0B3C2-6067-4856-864A-319CEAA7230B@gmail.com>
Message-ID: <lshdgr$1ol$1@ger.gmane.org>

On 8/13/2014 5:08 PM, Andrey Vlasovskikh wrote:

> Here are slides from my talk about optional typing in Python, that
> show how Mypy types can be used in both static and dynamic type
> checking
> (http://blog.pirx.ru/media/files/2013/python-optional-typing/),
I tried this on Windows 7in both Firefox and Internet Explorer and I 
cannot find any way to advance other than changing the page number on 
the url bar.


-- 
Terry Jan Reedy


From jlehtosalo at gmail.com  Thu Aug 14 06:28:48 2014
From: jlehtosalo at gmail.com (Jukka Lehtosalo)
Date: Wed, 13 Aug 2014 21:28:48 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+QR7ajXkMqQ3065VZE-mOMrUqaR_jY2P1EFbqqpPpseA@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
 <CAP7+vJ+QR7ajXkMqQ3065VZE-mOMrUqaR_jY2P1EFbqqpPpseA@mail.gmail.com>
Message-ID: <CAA_f+LwgeMX=sAg_fH=hA+ShbWjjjYya8MmDO57R1BnOHWkQPQ@mail.gmail.com>

On Wed, Aug 13, 2014 at 8:41 PM, Guido van Rossum <guido at python.org> wrote:

> On Wed, Aug 13, 2014 at 6:39 PM, Andrew Barnert <
> abarnert at yahoo.com.dmarc.invalid> wrote:
>
>> This example also shows exactly what's wrong with simple generics: if
>> this function takes an Iterable[String], it doesn't just return a
>> Mapping[String, int], it returns a Mapping of _the same String type_. If
>> your annotations can't express that, any value that passes through this
>> function loses type information.
>>
>
> In most cases it really doesn't matter though -- some types are better
> left concrete, especially strings and numbers. If you read the mypy docs
> you'll find that there are generic types, so that it's possible to define a
> function as taking an Iterable[T] and returning a Mapping[T, int]. What's
> not currently possible is expressing additional constraints on T such as
> that it must be a String. When I last talked to Jukka he explained that he
> was going to add something for that too (@Jukka: structured types?).
>

I wrote another message where I touched this. Mypy is likely to support
something like this in the future, but I doubt it's usually worth the
complexity. If a type signature is very general, at some point it describes
the implementation in sufficient detail that you can't modify the code
without changing the type. For example, we could plausibly allow anything
that just supports split(), but if we change the implementation to use
something other than split(), the signature would have to change. If we use
more specific types (such as str), we leave us the freedom to modify the
implementation within the bounds of the str interface. Standard library
functions often only accept concrete str objects, so the moment you start
using an abstract string type you lose access to much of the stdlib.


>
>> And not being able to tell whether the keys in word_count(f) are str or
>> bytes *even if you know that f was a text file* seems like a pretty major
>> loss.
>>
>
> On this point one of us must be confused. Let's assume it's me. :-) Mypy
> has a few different IO types that can express the difference between text
> and binary files. I think there's some work that needs to be done (and of
> course the built-in open() function has a terribly ambiguous return type
> :-( ), but it should be possible to say that a text file is an
> Interable[str] and a binary file is an Iterable[bytes]. So together with
> the structured (?) types it should be possible to specify the signature of
> word_count() just as you want it. However, in most cases it's overkill, and
> you wouldn't want to do that for most code.
>

See my other message where I show that you can do this right now, except
for the problem with open().


>
> Also, it probably wouldn't work for more realistic examples -- as soon as
> you replace the split() method call with something that takes punctuation
> into account, you're probably going to write it in a way that works only
> for text strings anyway, and very few people will want or need to write the
> polymorphic version. (But if they do, mypy has a handy @overload decorator
> that they can use. :-)
>
> Anyway, I agree it would be good to make sure that some of these more
> advanced things can actually be spelled before we freeze our commitment to
> a specific syntax, but let's not assume that just because you can't spell
> every possible generic use case it's no good.
>

It's always easy to come up with interesting corner cases where a type
system would break down, but luckily, these are often almost non-existent
in the wild :-) I've learned that examples should be motivated by patterns
in existing, 'real' code, as otherwise you'll waste your time on things
that happen maybe once a million lines (or maybe only in code that *you*
write).

Jukka


> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/89d92ebb/attachment-0001.html>

From guido at python.org  Thu Aug 14 06:55:11 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 21:55:11 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <75CA8559-0A4C-4BA0-94AA-A69C0CD56AE7@langa.pl>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <75CA8559-0A4C-4BA0-94AA-A69C0CD56AE7@langa.pl>
Message-ID: <CAP7+vJL_Lfmn1iLsx2FSsK2qKHRXJEYRjtosJuMzA_jX0cKHxg@mail.gmail.com>

On Wed, Aug 13, 2014 at 6:00 PM, ?ukasz Langa <lukasz at langa.pl> wrote:

> It?s great to see this finally happening!
>

Yes. :-)


> I did some research on existing optional-typing approaches [1]. What I
> learned in the process was that linting is the most important use case for
> optional typing; runtime checks is too little, too late.
>
> That being said, having optional runtime checks available *is* also
> important. Used in staging environments and during unit testing, this case
> is able to cover cases obscured by meta-programming. Implementations like
> ?obiwan? and ?pytypedecl? show that providing a runtime type checker is
> absolutely feasible.
>

Yes. And the proposal here might well enable such applications (by
providing a standard way to spell complex types). But I think it's going to
be less important than good support for linting, so that's what I want to
focus on first.


> The function annotation syntax currently supported in Python 3.4 is not
> well-suited for typing. This is because users expect to be able to operate
> on the types they know. This is currently not feasible because:
> 1. forward references are impossible
>

(Mypy's hack for this is that a string literal can be used as a forward
reference.)

2. generics are impossible without custom syntax (which is the reason
> Mypy?s Dict exists)
> 3. optional types are clumsy to express (Optional[int] is very verbose for
> a use case this common)
>

So define an alias 'oint'. :-)


> 4. union types are clumsy to express
>

Aliasing can help.


> All those problems are elegantly solved by Google?s pytypedecl via moving
> type information to a separate file.
>

Mypy supports this too using stub files, but I think it is actually a
strength that it doesn't require new syntax (although if the idea becomes
popular we could certainly add syntax to support those things where mypy
currently requires magic comments).

Honestly I'm not sure what to do about mypy vs. pytypedecl. Should they
compete, collaborate, converge? Do we need a bake-off or a joint hackathon?
Food for thought.


> Because for our use case that would not be an acceptable approach, my
> intuition would be to:
>
> 1. Provide support for generics (understood as an answer to the question:
> ?what does this collection contain??) in Abstract Base Classes. That would
> be a PEP in itself.
>
2. Change the function annotation syntax so that it?s not executed at
> import time but rather treated as strings. This solves forward references
> and enables us to?
>
3. Extend the function annotation syntax with first-class generics support
> (most languages like "list<str>?)
>
4. Extend the function annotation syntax with first-class union type
> support. pytypedecl simply uses ?int or None?, which I find very elegant.
>
5. Speaking of None, possibly further extend the function annotation syntax
> with first-class optionality support. In the Facebook codebase in Hack we
> have tens of thousands of optional ints (nevermind other optional types!),
> this is a case that?s going to be used all the time. Hack uses ?int, that?s
> the most succinct style you can get. Yes, it?s special but None is a
> special type, too.
>

Hm. I think that selling such (IMO) substantial changes to Python's syntax
is going to be much harder than just the idea of a standard typing syntax
implemented as a new stdlib module. While mypy's syntax is perhaps not as
concise or elegant as would be possible if we were to design the syntax
from the ground up, it's actually pretty darn readable, and it is
compatible with Python 3.2. It has decent ways to spell generics, forward
references, unions and optional types already. And while I want to
eventually phase out other uses of function annotations, your change #2
would break all existing packages that use them for other purposes (like
Ethan Furman's scription).


> All in all, I believe Mypy has the highest chance of becoming our typing
> linter, which is great! I just hope we can improve on the syntax, which is
> currently lacking. Also, reusing our existing ABCs where applicable would
> be nice. With Mypy?s typing module I feel like we?re going to get a new,
> orthogonal set of ABCs, which will confuse users to no end. Finally, the
> runtime type checker would make the ecosystem complete.
>

We can discuss these things separately. Language evolution is an exercise
in compromise. We may be able to reuse the existing ABCs, and mypy could
still support Python 3.2 (or, with the codeck hack, 2.7) by having the
typing module export aliases to those ABCs. I won't stop you from
implementing a runtime type checker, but I think it should be a separate
project.


> This is just the beginning of the open issues I was juggling with and the
> reason my own try at the PEP was coming up slower than I?d like.
>

Hopefully I've motivated you to speed up!


> [1] You can find a summary of examples I looked at here:
> http://lukasz.langa.pl/typehinting/
>

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/4e4336a0/attachment.html>

From guido at python.org  Thu Aug 14 07:11:17 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 22:11:17 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAGE7PN+TyBsUfE_yjZR5VLsJ4oNkJ6zgr_gR_KkGZGvSGYSkTQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAGE7PN+TyBsUfE_yjZR5VLsJ4oNkJ6zgr_gR_KkGZGvSGYSkTQ@mail.gmail.com>
Message-ID: <CAP7+vJJwsR0cWPhwuPcX6gk1m-CNaOySY2BaLLb_pUZkAGqpLg@mail.gmail.com>

On Wed, Aug 13, 2014 at 6:09 PM, Gregory P. Smith <greg at krypto.org> wrote:

> First, I am really happy that you are interested in this and that your
> point (2) of what you want to see done is very limited and acknowledges
> that it isn't going to specify everything!  Because that isn't possible. :)
>

What a shame. :-)


> Unfortunately I feel that adding syntax like this to the language itself
> is not useful without enforcement because it that leads to code being
> written with unintentionally incorrect annotations that winds up deployed
> in libraries that later become a problem as soon as an actual analysis tool
> attempts to run over something that uses that unknowingly incorrectly
> specified code in a place where it cannot be easily updated (like the
> standard library).
>

We could refrain from using type annotations in the stdlib (similar to how
we refrain from using Unicode identifiers). Mypy's stubs mechanism makes it
possible to ship the type declarations for stdlib modules with mypy instead
of baking them into the stdlib.


> At the summit in Montreal earlier this year ?ukasz Langa (cc'd)
> volunteered to lead writing the PEP on Python type hinting based on the
> many existing implementations of such things (including mypy, cython, numba
> and pytypedecl <https://github.com/google/pytypedecl>). I believe he has
> an initial draft he intends to send out soon. I'll let him speak to that.
>

Mypy has a lot more than an initial draft. Don't be mistaken by its status
as "one person's Ph.D. project" -- Jukka has been thinking about this topic
for a decade, and mypy works remarkably well already. It also has some very
active contributors already.


> Looks like ?ukasz already responded, I'll stop writing now and go read
> that. :)
>
> Personal opinion from experience trying: You can't express the depth of
> types for an interface within the Python language syntax itself (assuming
> hacks such as specially formatted comments, strings or docstrings do not
> count). Forward references to things that haven't even been defined yet are
> common. You often want an ability to specify a duck type interface rather
> than a specific type.  I think he has those points covered better than I do.
>

I think mypy has solutions for the syntactic issues, and the rest can be
addressed by introducing a few more magic helper functions. It's remarkably
readable.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/957a6f27/attachment-0001.html>

From guido at python.org  Thu Aug 14 07:24:12 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 22:24:12 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAA_f+LxPbisNd949SmiKhkQNM_N12e0qvQ+aO9uhn1qQ0ji6+g@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
 <CAA_f+LxPbisNd949SmiKhkQNM_N12e0qvQ+aO9uhn1qQ0ji6+g@mail.gmail.com>
Message-ID: <CAP7+vJK6rOQTG37P9z6DfSAKBMx-wpCGTQdzG12G4bK_fatO+w@mail.gmail.com>

On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo <jlehtosalo at gmail.com>
wrote:
>
>
> You could use AnyStr to make the example work with bytes as well:
>
>   def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]:
>       result = {}  #type: Dict[AnyStr, int]
>
>       for line in input:
>           for word in line.split():
>               result[word] = result.get(word, 0) + 1
>       return result
>
> Again, if this is just a simple utility function that you use once or
> twice, I see no reason to spend a lot of effort in coming up with the most
> general signature. Types are an abstraction and they can't express
> everything precisely -- there will always be a lot of cases where you can't
> express the most general type. However, I think that relatively simple
> types work well enough most of the time, and give the most bang for the
> buck.
>

I heartily agree. But just for the type theorists amongst us, if I really
wanted to write the most general type, how would I express that the AnyStr
in the return type matches the one in the argument? (I think pytypedecl
would use something like T <= AnyStr.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/021fb680/attachment.html>

From haoyi.sg at gmail.com  Thu Aug 14 07:39:14 2014
From: haoyi.sg at gmail.com (Haoyi Li)
Date: Wed, 13 Aug 2014 22:39:14 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJK6rOQTG37P9z6DfSAKBMx-wpCGTQdzG12G4bK_fatO+w@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
 <CAA_f+LxPbisNd949SmiKhkQNM_N12e0qvQ+aO9uhn1qQ0ji6+g@mail.gmail.com>
 <CAP7+vJK6rOQTG37P9z6DfSAKBMx-wpCGTQdzG12G4bK_fatO+w@mail.gmail.com>
Message-ID: <CALruUQKprK-iKRC5eaZpFMZXwpoePggCxXYXu_gouUAu20be2Q@mail.gmail.com>

> I heartily agree. But just for the type theorists amongst us, if I really
wanted to write the most general type, how would I express that the AnyStr
in the return type matches the one in the argument? (I think pytypedecl
would use something like T <= AnyStr.)

To borrow Scala syntax, it would look something like

def word_count[AnyStr <: String](input: Iterable[AnyStr]): Dict[AnyStr, Int]

Where *word_count* is a generic function on the type *AnyStr*, which is not
just *any* type but a type bound by the restriction it is a subclass of
*String*. Thus you can force that the *AnyStr* going in and the *AnyStr*
going out are the same one.

I'm not sure if mypy allows for type-bounds, but that's the way you achieve
what you want if it does, or will, in future =P


On Wed, Aug 13, 2014 at 10:24 PM, Guido van Rossum <guido at python.org> wrote:

> On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo <jlehtosalo at gmail.com>
> wrote:
>>
>>
>> You could use AnyStr to make the example work with bytes as well:
>>
>>   def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]:
>>       result = {}  #type: Dict[AnyStr, int]
>>
>>       for line in input:
>>           for word in line.split():
>>               result[word] = result.get(word, 0) + 1
>>       return result
>>
>> Again, if this is just a simple utility function that you use once or
>> twice, I see no reason to spend a lot of effort in coming up with the most
>> general signature. Types are an abstraction and they can't express
>> everything precisely -- there will always be a lot of cases where you can't
>> express the most general type. However, I think that relatively simple
>> types work well enough most of the time, and give the most bang for the
>> buck.
>>
>
> I heartily agree. But just for the type theorists amongst us, if I really
> wanted to write the most general type, how would I express that the AnyStr
> in the return type matches the one in the argument? (I think pytypedecl
> would use something like T <= AnyStr.)
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/dd1db775/attachment.html>

From guido at python.org  Thu Aug 14 07:44:13 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 13 Aug 2014 22:44:13 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CALruUQKprK-iKRC5eaZpFMZXwpoePggCxXYXu_gouUAu20be2Q@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
 <CAA_f+LxPbisNd949SmiKhkQNM_N12e0qvQ+aO9uhn1qQ0ji6+g@mail.gmail.com>
 <CAP7+vJK6rOQTG37P9z6DfSAKBMx-wpCGTQdzG12G4bK_fatO+w@mail.gmail.com>
 <CALruUQKprK-iKRC5eaZpFMZXwpoePggCxXYXu_gouUAu20be2Q@mail.gmail.com>
Message-ID: <CAP7+vJ++p620YVwHsT8XziW96bt3MqhbWrjL_UKGJSn=q07qPQ@mail.gmail.com>

I'd be more interested in Jukka's specific proposal. (Note that in mypy,
AnyStr is the type that you call String here -- it's either bytes or str;
and it seems that you're introducing AnyStr here as a type variable -- mypy
conventionally uses T for this, but you have to define it first.)


On Wed, Aug 13, 2014 at 10:39 PM, Haoyi Li <haoyi.sg at gmail.com> wrote:

> > I heartily agree. But just for the type theorists amongst us, if I
> really wanted to write the most general type, how would I express that the
> AnyStr in the return type matches the one in the argument? (I think
> pytypedecl would use something like T <= AnyStr.)
>
> To borrow Scala syntax, it would look something like
>
>
> def word_count[AnyStr <: String](input: Iterable[AnyStr]): Dict[AnyStr, Int]
>
> Where *word_count* is a generic function on the type *AnyStr*, which is
> not just *any* type but a type bound by the restriction it is a subclass of
> *String*. Thus you can force that the *AnyStr* going in and the *AnyStr*
> going out are the same one.
>
> I'm not sure if mypy allows for type-bounds, but that's the way you
> achieve what you want if it does, or will, in future =P
>
>
>  On Wed, Aug 13, 2014 at 10:24 PM, Guido van Rossum <guido at python.org>
> wrote:
>
>>  On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo <jlehtosalo at gmail.com>
>> wrote:
>>>
>>>
>>> You could use AnyStr to make the example work with bytes as well:
>>>
>>>   def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]:
>>>       result = {}  #type: Dict[AnyStr, int]
>>>
>>>       for line in input:
>>>           for word in line.split():
>>>               result[word] = result.get(word, 0) + 1
>>>       return result
>>>
>>> Again, if this is just a simple utility function that you use once or
>>> twice, I see no reason to spend a lot of effort in coming up with the most
>>> general signature. Types are an abstraction and they can't express
>>> everything precisely -- there will always be a lot of cases where you can't
>>> express the most general type. However, I think that relatively simple
>>> types work well enough most of the time, and give the most bang for the
>>> buck.
>>>
>>
>> I heartily agree. But just for the type theorists amongst us, if I really
>> wanted to write the most general type, how would I express that the AnyStr
>> in the return type matches the one in the argument? (I think pytypedecl
>> would use something like T <= AnyStr.)
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140813/166402e5/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Thu Aug 14 08:48:07 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 14 Aug 2014 18:48:07 +1200
Subject: [Python-ideas] Proposal: Use mypy syntax
	for	function	annotations
In-Reply-To: <3C196232-D06D-4EFC-81BB-58FCBBF86DC2@yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <53EC146D.3090402@canterbury.ac.nz>
 <3C196232-D06D-4EFC-81BB-58FCBBF86DC2@yahoo.com>
Message-ID: <53EC5BA7.10904@canterbury.ac.nz>

Andrew Barnert wrote:
> If you go with a single JSONThing type that represents an object, array, 
> number, bool, string, or null, then it can't have a standard mapping 
> interface, because it also needs to have a standard sequence interface,

I didn't mean that. The most Pythonic way would probably be to
have something like

JSONThing = Union(JSONDict, JSONList, Int, Float, Str, Bool)
JSONDict = Mapping[str, JSONThing]
JSONList = Sequence[JSONThing]

If you're concerned about type safety, at some point you
need to introspect on what you've got to figure out what
to do with it. This is inevitable, since a JSON object
is inherently a dynamically-typed thing. This is true
even in Haskell, you just don't notice it so much because
you do it with pattern matching.

> The only 
> possibility is a union of all the various types mentioned above, and 
> such a union type has no interface at all. It's only useful if people 
> subvert the type safety by casting.

I don't know what mypy does with union types, but if I were
designing a type system like this I would say that, e.g. if
i is know to be of type Int and d of type JSONDict, then

    i = d['foo']

should be allowed, on the grounds that the return value
*could* be an Int, with a run-time type check to make sure
that it actually is. An implicit cast, in other words.

As long as mypy is just a linter it's obviously not in
a position to insert the runtime check, but it could be
argued that it should allow the assignment anyway,
since Python will end up raising a TypeError at some
point if it's wrong.

I've forgotten what the original point of all this was.
If the point was that there's no benefit in trying to
make JSON type-safe in Python, and you should just leave
it all dynamically typed, maybe you're right.

-- 
Greg

From abarnert at yahoo.com  Thu Aug 14 08:45:02 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 13 Aug 2014 23:45:02 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAA_f+LxPbisNd949SmiKhkQNM_N12e0qvQ+aO9uhn1qQ0ji6+g@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
 <CAA_f+LxPbisNd949SmiKhkQNM_N12e0qvQ+aO9uhn1qQ0ji6+g@mail.gmail.com>
Message-ID: <6CB4D512-E476-4E3B-AB2B-501C5E03FD28@yahoo.com>

On Aug 13, 2014, at 21:06, Jukka Lehtosalo <jlehtosalo at gmail.com> wrote:

> You could use AnyStr to make the example work with bytes as well:
> 
>   def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]:
>       result = {}  #type: Dict[AnyStr, int]
>       for line in input:
>           for word in line.split():
>               result[word] = result.get(word, 0) + 1
>       return result

Defining a function as taking an Iterable[AnyStr] and returning a Dict[AnyStr, int], without any way to declare that the two AnyStr are the same type, is exactly what I meant by losing critical information. I pass in something that I know is a text file, and I get out something that are either strings or bytes and I don't know which; what am I going to do with that? If you can't propagate types, static typing doesn't help.

My point is that the depth you seem to be reaching for is not actually a sweet spot for power vs. simplicity. You could make it simpler by just dropping generics in favor of a handful of special-purpose modifiers (optional, iterable-of), or you could make it more useful by having real parametric types, or at least the equivalent of C++ template templates or Swift arbitrary constraints. What makes Java-style simple generics with subclass constraints the right level of complexity? 

From abarnert at yahoo.com  Thu Aug 14 09:37:14 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 14 Aug 2014 00:37:14 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax
	for	function	annotations
In-Reply-To: <53EC5BA7.10904@canterbury.ac.nz>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <1407979616.59487.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <53EC146D.3090402@canterbury.ac.nz>
 <3C196232-D06D-4EFC-81BB-58FCBBF86DC2@yahoo.com>
 <53EC5BA7.10904@canterbury.ac.nz>
Message-ID: <13C23919-EF10-4A7F-870F-8561B70C4E99@yahoo.com>

On Aug 13, 2014, at 23:48, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> I've forgotten what the original point of all this was.
> If the point was that there's no benefit in trying to
> make JSON type-safe in Python, and you should just leave
> it all dynamically typed, maybe you're right.

Well, that was my starting point, which I assumed people would take for granted, so I could use it to make my real point, which is that "just downcast it", which came up a few times in the first couple hours of this discussion, is a bad answer in general.

If you start from the assumption that everything can and should be statically typed, but don't design a type system powerful enough to do that, you end up putting downcasts from void* or Object or whatever all over the place, and it's impossible to tell what parts of the program are actually safe, which defeats the entire purpose of static typing.

If you start from the assumption that there will be some self-contained regions of your code that can be typed by your type system, and other parts just have to be dynamically typed and that's OK, you don't subvert the type system all over the place, and you know which parts of the code are or aren't known to be safe.

Anyway, I think this is basically what you just said--you started explaining how we could effectively add C++ style implicit casts to hide the ugliness and unsafety of dealing with JSON, but then said that maybe it would be better to just leave the values dynamically typed. As long as the type checker can (not necessarily in the initial version, but at least reasonably plausibly) make it easy to tell which areas of your code have been "infected" by dynamic types, I think defaulting to "too hard, leave it dynamic" is the simplest, and probably right, thing to do. (Unless we actually want to design a sufficiently powerful type system, which I don't think we do.)

There's a great paper on how this works in practice in an ML variant, but unfortunately googling any subset of the only keywords I can remember (ML type duck infect dynamic) just brings up papers about avian virology (even if I leave our "duck"). I think there's also a (much more recent and less technical) blog post by one of the Swift guys about a similar idea--making it easy to stay dynamic means you end up with cleanly walled off regions of safe and unsafe code. (Or you would, if Swift had a halfway-decent stdlib instead of forcing you to bridge to ObjC for almost anything, but that's not important here.)

From ncoghlan at gmail.com  Thu Aug 14 10:13:37 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Aug 2014 18:13:37 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJJwsR0cWPhwuPcX6gk1m-CNaOySY2BaLLb_pUZkAGqpLg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAGE7PN+TyBsUfE_yjZR5VLsJ4oNkJ6zgr_gR_KkGZGvSGYSkTQ@mail.gmail.com>
 <CAP7+vJJwsR0cWPhwuPcX6gk1m-CNaOySY2BaLLb_pUZkAGqpLg@mail.gmail.com>
Message-ID: <CADiSq7d1brUcwx9GzvTS10m8TyzFidEQxLEVN1wOKAB49ZPhow@mail.gmail.com>

On 14 August 2014 15:11, Guido van Rossum <guido at python.org> wrote:
> On Wed, Aug 13, 2014 at 6:09 PM, Gregory P. Smith <greg at krypto.org> wrote:
>>
>> At the summit in Montreal earlier this year ?ukasz Langa (cc'd)
>> volunteered to lead writing the PEP on Python type hinting based on the many
>> existing implementations of such things (including mypy, cython, numba and
>> pytypedecl). I believe he has an initial draft he intends to send out soon.
>> I'll let him speak to that.
>
>
> Mypy has a lot more than an initial draft. Don't be mistaken by its status
> as "one person's Ph.D. project" -- Jukka has been thinking about this topic
> for a decade, and mypy works remarkably well already. It also has some very
> active contributors already.

FWIW, I was strongly encouraging folks at SciPy that were interested
in static typing to look at mypy as an example of a potentially
acceptable syntax for standardised optional static typing.

Aside from being +1 on the general idea of picking *something* as
"good enough" and iterating from there, I don't have a strong personal
opinion, though.

(And yes, my main interest is in improving the ability to do effective
linting for larger projects. "pylint -E" is a lot better than nothing,
but it has its limits in the absence of additional static hints)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Aug 14 10:16:01 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Aug 2014 18:16:01 +1000
Subject: [Python-ideas] float comparison in doctest
In-Reply-To: <CAOTD34Y-bOXmn+NeSMesF5LXD7nbiqCMWqzup81wcTN4MiQybg@mail.gmail.com>
References: <CAOTD34Y-bOXmn+NeSMesF5LXD7nbiqCMWqzup81wcTN4MiQybg@mail.gmail.com>
Message-ID: <CADiSq7c63Uumnh+2mRiYgQ4w3pt2ah0m=Y1FE9WN5AMYb5WT9w@mail.gmail.com>

On 14 August 2014 01:30, Erik Bray <erik.m.bray at gmail.com> wrote:
>
> That said, if anyone has any ideas for allowing tweaking the
> tolerances for a doctest flag that would be great.  If this otherwise
> seems like a good idea in general to include in the doctest module I
> will offer a patch.

I think it sounds like a reasonable idea, even with the "can't be
tuned" limitation. It does become a possible use case *for* a doctest
flag tuning system, but that doesn't need to be a requirement.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Aug 14 10:37:55 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Aug 2014 18:37:55 +1000
Subject: [Python-ideas] sum(...) limitation
In-Reply-To: <20140813163729.GI4525@ando>
References: <53E7CBA4.40105@g.nevcal.com>
 <87wqafj3tb.fsf@uwakimon.sk.tsukuba.ac.jp>
 <-2448384566377912251@unknownmsgid>
 <CADiSq7c8hur9Tggk7U-mn=hz5swrOdCHu+BYvzNC0Z74UVd4Hw@mail.gmail.com>
 <2076096455819154683@unknownmsgid>
 <87ppg6icxu.fsf@uwakimon.sk.tsukuba.ac.jp>
 <53E991C9.7020404@stoneleaf.us>
 <87lhqui6la.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CALGmxEJH-9zptJgs6k_n9TSwXOSmkj-UUSGWbv5=0Y38uC8dmg@mail.gmail.com>
 <87egwkj4eh.fsf@uwakimon.sk.tsukuba.ac.jp>
 <20140813163729.GI4525@ando>
Message-ID: <CADiSq7c8tzjFnTMggjmT+A9DkYRHnRZ6V=VH4oggy9SPp34j+g@mail.gmail.com>

On 14 August 2014 02:37, Steven D'Aprano <steve at pearwood.info> wrote:
> A bit of history, as I remember it: sum() exists because for half of
> Python's lifetime, people were regularly defining this:
>
>     def sum(numbers):
>         return reduce(lambda a, b: a+b, numbers)
>
> so they could easy add up a bunch of numbers:
>
>     num_pages = sum([ch.pages() for ch in self.chapters])
>
> sort of thing. Since this was a common need, it was decided to add it to
> the built-ins. But sum() was never intended to replace str.join or
> list.extend, let alone even more exotic cases.
>
> Built-in sum is aimed at sequences of numbers, not strings, lists,
> tuples, or Widgets for that matter. Perhaps giving it a start parameter
> was a mistake, but it is there and backwards compatibility says it isn't
> going to be removed. But that doesn't mean that the use of sum() on
> arbitrary types ought to be *encouraged*, even if it is *allowed*.

Interesting point of history: Adding the sum() builtin is when Alex
Martelli was given commit privileges (see
http://bugs.python.org/issue724936 - found by putting "builtin sum"
into the search field on bugs.python.org)

Reviewing that issues, in one of the draft patches, all sequences were
banned, but that was changed in order to allow sum() to handle
sequences that define __add__ as an element-wise operation (since
those won't exhibit the quadratic behaviour).

The date of the issue also made it possible to find the thread Alex
mentions in the list archives:
https://mail.python.org/pipermail/python-dev/2003-April/034767.html

And there we find that the original patch *did* automatically delegate
to str.join, and after an intial review that expressed some
reservations (https://mail.python.org/pipermail/python-dev/2003-April/034825.html),
Guido specifically requested changing it to disallow strings:
https://mail.python.org/pipermail/python-dev/2003-April/034853.html
And Alex agreed:
https://mail.python.org/pipermail/python-dev/2003-April/034855.html

The idea of a join() builtin was also discussed and rejected in that thread.

Regards,
Nick.

P.S. Note that this kind of thread (which I contributed to myself) is
why I'm inclined to insist on a PEP for *any* new builtin these days,
even non-controversial ones: so the rationale for the associated
design decision is less likely to be rehashed at length more than a
decade later. Remembering "there was a PEP for that feature, and it
listed this as a rejected design alternative" is relatively easy.
Remembering "that idea came up in that thread back in 2003" is much,
much, harder :P

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Aug 14 10:44:53 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Aug 2014 18:44:53 +1000
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
In-Reply-To: <CAP7+vJJKP=1QfQrUaQ0Rq5Et-o81+MmzWJaeUT8MRkGvvhwRtQ@mail.gmail.com>
References: <mailman.72122.1407959084.18129.python-ideas@python.org>
 <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com>
 <CAP7+vJJKP=1QfQrUaQ0Rq5Et-o81+MmzWJaeUT8MRkGvvhwRtQ@mail.gmail.com>
Message-ID: <CADiSq7eK7vXgrmrrHCu=2jkH4EWS7nzQtV51k0OZ1213fznOqA@mail.gmail.com>

On 14 August 2014 07:09, Guido van Rossum <guido at python.org> wrote:
> On Wed, Aug 13, 2014 at 1:18 PM, Raymond Hettinger
> <raymond.hettinger at gmail.com> wrote:
>>
>>
>> On Aug 13, 2014, at 12:44 PM, python-ideas-request at python.org wrote:
>>
>>  The goal is to make it possible to add
>> type checking annotations to 3rd party modules (and even to the stdlib)
>> while allowing unaltered execution of the program by the (unmodified)
>> Python 3.5 interpreter.
>>
>>
>> Is the goal to "make it possible" or would it quickly become required
>> (i.e. any time you write normal, readable Python, it would break
>> someone's optimizer, refactorer, linter, etc.?)
>
>
> Whoa, whoa. That's not at all the idea. Currently *nobody* uses type
> annotations because there's no standard notation. My goal is to enable their
> use by proposing a standard, nothing more.

Given the general lack of familiarity with ABCs, I don't have any
great fear of optional static typing becoming standard practice
either. Just like ABCs, it's a useful way to formalise some checks
when scaling up a code base to larger development teams. Most code
bases aren't going to be that large, so it often isn't going to be
worth the hassle.

On the other hand, it's going to make it easier to detect certain
kinds of structural errors more easily than even extensive unit
testing, as well as helping out IDEs to provide better prompts to
users, so I'm personally a fan of the idea.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From christian at python.org  Thu Aug 14 11:08:43 2014
From: christian at python.org (Christian Heimes)
Date: Thu, 14 Aug 2014 11:08:43 +0200
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
In-Reply-To: <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com>
References: <mailman.72122.1407959084.18129.python-ideas@python.org>
 <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com>
 <CAP7+vJJKP=1QfQrUaQ0Rq5Et-o81+MmzWJaeUT8MRkGvvhwRtQ@mail.gmail.com>
 <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com>
Message-ID: <53EC7C9B.1000806@python.org>

On 14.08.2014 02:59, Raymond Hettinger wrote:
> P.S.  I would really like for the annotations to grow some way
> to communicate exceptions as well as return types (i.e. that
> list.index can raise a ValueError and list.pop can raise an IndexError).
> This would be only for the exceptions directly added by a function or
> method,
> not ones raised by the data (which is something the function can't control).

+1

I made the same suggestion a couple of hours earlier. A standardized and
introspectable way to add exception annotation is a common request. It's
especially useful for C code because it's very hard to impossible to
deduce exceptions from C code.

For now there is no need for a new syntax. IMHO a decorator and standard
location for exception annotations are sufficient. Perhaps somebody is
able to come up with a syntax later.


Such a decorator should also include the reason for an exception, too.
For example:

class list:

    @raises(IndexError, "when list is empty")
    def pop(self, index=None):
        pass

    @raises(ValueError)
    def index(self, value):
        pass

Christian


From ceronman at gmail.com  Thu Aug 14 12:28:26 2014
From: ceronman at gmail.com (=?UTF-8?Q?Manuel_Cer=C3=B3n?=)
Date: Thu, 14 Aug 2014 12:28:26 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJLKNydsdLuc8jRjhfYr_R8P7u1Jg1aZ4Yb1LA1c-t3w1g@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CA+VECoVO0COba4u3o5YD0shE0r-P8-q99sWMQtpRkisQ8wa8UQ@mail.gmail.com>
 <CAP7+vJLKNydsdLuc8jRjhfYr_R8P7u1Jg1aZ4Yb1LA1c-t3w1g@mail.gmail.com>
Message-ID: <CA+VECoXkb+=UeSQbckY+prOgKx7AY4yUwFRV9ogPTTQ+mR=DOA@mail.gmail.com>

On Thu, Aug 14, 2014 at 1:24 AM, Guido van Rossum <guido at python.org> wrote:

> On Wed, Aug 13, 2014 at 3:26 PM, Manuel Cer?n <ceronman at gmail.com> wrote:
>
>> The type checking algorithm might evolve over the time, but by including
>> typing.py in the stdlib, the syntax for annotations would be almost frozen
>> and that will be a limitation. In other projects such as TypeScript (
>> http://www.typescriptlang.org/), that the syntax usually evolves
>> alongside the algorithms.
>>
>
> What kind of evolution did TypeScript experience?
>

There is a nice summary of breaking changes here:
https://typescript.codeplex.com/wikipage?title=Known%20breaking%20changes%20between%200.8%20and%200.9

Most of these are in the way that the type checking engine works, but there
are some syntax changes. While the basic types have remained stable, some
special things have not. For example: generics, optional and default
arguments, interfaces.

I think it'd be valuable to learn from TypeScript as much as possible, it's
the only project that I know that It's trying to bring static type analysis
to a widely used dynamically typed language.

One interesting feature of TypeScript is that it allows you to annotate
existing code without modifying it, by using external definition files. In
the JavaScript world, many people have contributed TypeScript annotation
files for popular JS libraries (http://definitelytyped.org/).

I think this is possible in Python as well doing something like this:

@annotate('math.ciel')
def ciel(x: float) -> int:
    pass

I think this should be the way to go for annotating the stdlib. It has the
advantage that if the type syntax changes, it's possible to provide new
type annotations without changing the libraries at all, and even supporting
older versions. In this way the code and type annotations can evolve
separately.

Does mypy support something like this?

Manuel.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/a41bfbfc/attachment.html>

From apalala at gmail.com  Thu Aug 14 12:27:20 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Thu, 14 Aug 2014 05:57:20 -0430
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
In-Reply-To: <53EC7C9B.1000806@python.org>
References: <mailman.72122.1407959084.18129.python-ideas@python.org>
 <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com>
 <CAP7+vJJKP=1QfQrUaQ0Rq5Et-o81+MmzWJaeUT8MRkGvvhwRtQ@mail.gmail.com>
 <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com> <53EC7C9B.1000806@python.org>
Message-ID: <CAN1YFWv323V2dggktk8QHRaj-RkOFdrdW2KRkKwJMcRtFR1eHA@mail.gmail.com>

On Thu, Aug 14, 2014 at 4:38 AM, Christian Heimes <christian at python.org>
wrote:

> I made the same suggestion a couple of hours earlier. A standardized and
> introspectable way to add exception annotation is a common request. It's
> especially useful for C code because it's very hard to impossible to
> deduce exceptions from C code.
>
> For now there is no need for a new syntax. IMHO a decorator and standard
> location for exception annotations are sufficient. Perhaps somebody is
> able to come up with a syntax later.
>

The problem with declaring raisable exceptions is that it invites static
verification of callers doing "the right thing" with them, as in Java, and
that introduces noise in the software development process that is almost
impossible to clean up.

    @raises(ValueError)
    def index(self, value):
        pass

    def found(self, value):
        return self.index(value) >= 0

Is found() correct? If not, I assure you the most common (and probably
incorrect) solution will be:

    def found(self, value):
        try:
           return self.index(value) >= 0
        except ValueError:
           return False

Or worse:

    def found(self, value):
        try:
           return self.index(value) >= 0
        except:
           return False


The right thing to do with an exception is to let it through, unless you
know exactly what you have to do about it.

In this interview, Anders Hejlsberg talks about why checked exceptions were
left out of the design of C#:

http://www.artima.com/intv/handcuffs.html

Cheers,

-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/ea9d6d16/attachment.html>

From willvarfar at gmail.com  Thu Aug 14 13:16:11 2014
From: willvarfar at gmail.com (willvarfar at gmail.com)
Date: Thu, 14 Aug 2014 13:16:11 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
Message-ID: <CAKkef2yzJZ5m6B1wLOW2bxXsPCPCF2sZa9ctsrHYckOYHLf4yw@mail.gmail.com>

I fully support formalizing Python 3's annotations for type checking.

I wrote - and use daily - my own type checker called obiwan
https://pypi.python.org/pypi/obiwan

Its a runtime type checker, and if enabled will check and enforce
types on every call.

I support the wider adoption and standardization of static type
checkers, but runtime checkers are still wanted for very dynamic code
and for checking external data e.g. I use obiwan for validating JSON.

One small detail is that I feel the obiwan annotations are more
pythonic than the mypy examples given.

E.g. instead of:

  from typing import List, Dict

  def word_count(input: List[str]) -> Dict[str, int]:
      ...

It would be:

  def word_count(input: [str]) -> {str, int}:
      ...

Obiwan does not check types within functions; I was unwilling to try
and overload comments!  You can invoke obiwan to check things
explicitly, but these are more as assertions.

Anyway, when I look at the mypy in-function annotations (where
comments are overloaded) I am cautious.  It would be far nicer if we
had annotations as part of the language instead, e.g. instead of:

  result = {}  #type: Dict[str, int]

It would be:

  result = {} -> {str, int}

where we use the -> arrow again.  I can see pros and cons for any
implementation (as we'd want the annotation to be both to declare a
type and to check a type, and want the annotation to be attached to
the variable forever etc) so this would need a full PEP treatment and
possibly be configurable as asserts are.

But proper annotation support in the language rather than overloading
comments would definitely be my preference.

/Will

/Will

From andrey.vlasovskikh at gmail.com  Thu Aug 14 13:36:26 2014
From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh)
Date: Thu, 14 Aug 2014 15:36:26 +0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CA+VECoXkb+=UeSQbckY+prOgKx7AY4yUwFRV9ogPTTQ+mR=DOA@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CA+VECoVO0COba4u3o5YD0shE0r-P8-q99sWMQtpRkisQ8wa8UQ@mail.gmail.com>
 <CAP7+vJLKNydsdLuc8jRjhfYr_R8P7u1Jg1aZ4Yb1LA1c-t3w1g@mail.gmail.com>
 <CA+VECoXkb+=UeSQbckY+prOgKx7AY4yUwFRV9ogPTTQ+mR=DOA@mail.gmail.com>
Message-ID: <CA+qPcwsyRW418k6LuwOQoEnBb+NAfD9ZkgqQhq9hJcvfv7nWNw@mail.gmail.com>

On Thu, Aug 14, 2014 at 2:28 PM, Manuel Cer?n <ceronman at gmail.com> wrote:

> One interesting feature of TypeScript is that it allows you to annotate
> existing code without modifying it, by using external definition files. In
> the JavaScript world, many people have contributed TypeScript annotation
> files for popular JS libraries (http://definitelytyped.org/).
>
> I think this is possible in Python as well doing something like this:
>
> @annotate('math.ciel')
> def ciel(x: float) -> int:
>     pass
>
> I think this should be the way to go for annotating the stdlib. It has the
> advantage that if the type syntax changes, it's possible to provide new type
> annotations without changing the libraries at all, and even supporting older
> versions. In this way the code and type annotations can evolve separately.
>
> Does mypy support something like this?

We use something quite similar to TypeScript's repository of
annotations in PyCharm. Here is our external annotations proposal and
stubs for some stdlib modules
(https://github.com/JetBrains/python-skeletons) that uses a custom
type syntax in docstrings due to lack of a better standard, see
README. We state in our proposal that we would like the standard to
emerge. The idea being discussed here about using Mypy's type system
is not new to us. As I've mentioned in the original thread, we have
discussed it with Jukka Lehtosalo, the author of Mypy. Some initial
ideas are listed here (https://github.com/pytypes/pytypes).

-- 
Andrey Vlasovskikh
Web: http://pirx.ru/

From christian at python.org  Thu Aug 14 13:38:45 2014
From: christian at python.org (Christian Heimes)
Date: Thu, 14 Aug 2014 13:38:45 +0200
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
In-Reply-To: <CAN1YFWv323V2dggktk8QHRaj-RkOFdrdW2KRkKwJMcRtFR1eHA@mail.gmail.com>
References: <mailman.72122.1407959084.18129.python-ideas@python.org>
 <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com>
 <CAP7+vJJKP=1QfQrUaQ0Rq5Et-o81+MmzWJaeUT8MRkGvvhwRtQ@mail.gmail.com>
 <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com>
 <53EC7C9B.1000806@python.org>
 <CAN1YFWv323V2dggktk8QHRaj-RkOFdrdW2KRkKwJMcRtFR1eHA@mail.gmail.com>
Message-ID: <53EC9FC5.2010701@python.org>

On 14.08.2014 12:27, Juancarlo A?ez wrote:
> The problem with declaring raisable exceptions is that it invites
> static verification of callers doing "the right thing" with them,
> as in Java, and that introduces noise in the software development
> process that is almost impossible to clean up.
[snip]
> The right thing to do with an exception is to let it through,
> unless you know exactly what you have to do about it.
> 
> In this interview, Anders Hejlsberg talks about why checked
> exceptions were left out of the design of C#:

Thanks for your input, Juancarlo.

I don't think that anybody wants to add checked exceptions to Python
at this point. It's one of my pain points with Java, too. As far as I
can judge Raymond we both want to archive the same goal: improve
introspection and self-documentation of code with a strong focus on
bindings (e.g. C code).

As I tried to explain earlier neither users nor tools are able to
analyze code like builtin types and functions. Exception annotations
could help IDEs to figure out what kind of exception could be raised.
Does it sound better to you?

The general idea is remotely related to mypy and should be addressed
by a different PEP that accompanies the mypy PEP.

Regards,
Christian


From kaiser.yann at gmail.com  Thu Aug 14 15:59:33 2014
From: kaiser.yann at gmail.com (Yann Kaiser)
Date: Thu, 14 Aug 2014 15:59:33 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CA+qPcwsyRW418k6LuwOQoEnBb+NAfD9ZkgqQhq9hJcvfv7nWNw@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CA+VECoVO0COba4u3o5YD0shE0r-P8-q99sWMQtpRkisQ8wa8UQ@mail.gmail.com>
 <CAP7+vJLKNydsdLuc8jRjhfYr_R8P7u1Jg1aZ4Yb1LA1c-t3w1g@mail.gmail.com>
 <CA+VECoXkb+=UeSQbckY+prOgKx7AY4yUwFRV9ogPTTQ+mR=DOA@mail.gmail.com>
 <CA+qPcwsyRW418k6LuwOQoEnBb+NAfD9ZkgqQhq9hJcvfv7nWNw@mail.gmail.com>
Message-ID: <CANUJvPVgQOTiGfCB=LSaBvj=5uG0RrZah9G8xPZGF+GZw4N26w@mail.gmail.com>

I'd like to offer my perspective(and objections) on this:

I author and maintain a reflective argument parser for python, clize[1],
possibly in the same vein as Ethan's. While it originally did not support
any Python 3 features, it has come to accept keyword-only parameters and
annotations for alternate parameter names and conversion functions(which
may look like type annotations to a reader looking for them). The upcoming
version will completely deprecate the old "pass a bunch of unsightly
arguments to a decorators[2]" API, and use annotations exclusively[3],
using backports[4][5] to make them and keyword-only parameters available to
Python 2 users. Clize has a handful of users, with 100 stars on GitHub, and
~75 downloads per day according to PyPI[1].

Needless to say, the idea of deprecating uses of parameter and return
annotations that aren't this particular way of typechecking is upsetting to
me.

On to the most formal of formal complaints: PEP 3107[6] rejected
standardizing typing-related annotations. What has changed since then, or
what has otherwise invalidated that conclusion, that it should be
backtracked on and changed? GvR says he knows of little practical use of
function annotations in mainstream code. Here's the (unfortunate) reason:
Mainstream code is either incompatible with Python 3 or maintaining
compatibility between both Python 2 and Python 3. Standard library code
which had no such concern for the most part stuck with what stood in PEP
3107, which was not to teach annotations as typing info. Short of resorting
to tools like sigtools.modifiers.annotate[4], *mainstream code can't use
function annotations*.

You will argue that people are already coming up with type-checking
libraries en masse, and that little alternative uses have come up. That's
because type-checking is an idea that people have seen and had in mind well
before now, PEP 3107, or Python itself. It is even one of the two examples
in the PEP, the other being fairly irrelevant given docstrings, IMO.
Mainstream code can't use annotations. New brains are mostly taught to use
python 2[7] or keep compatibility with it, so no new ideas there(save for
the endless stream of "I've used Python for a day and it needs static
typing" coming from statically-typed languages.) IMO there simply hasn't
been enough time for new uses to show up. There's the argument parsers me
and a couple others have mentioned, and although the idea can be ported to
other interfaces(web forms, GUI forms?), this is only one use yet, but I
think it is a good one, one that is also fairly unique to Python. Do we
want to close this window of opportunity by trying to imitate the 80's?

Even if you don't deprecate other uses, putting this in the standard
library sends a message to the mob: "Annotations are meant to express types
and other uses should conform." It's already slightly against-current to
propose reflective tools because "reflection is weird and unreliable," I'd
hate to see where this would take us.

Worse than shutting down a potential area of innovation, promoting
type-checking in such a way would alienate what many experienced Python
programmers view as a core tenet of Python: duck-typing.

Ironically, GvR's example illustrates it perfectly:

  from typing import List, Dict

  def word_count(input: List[str]) -> Dict[str, int]:
      result = {}  #type: Dict[str, int]
      for line in input:
          for word in line.split():
              result[word] = result.get(word, 0) + 1
      return result

First, there's the obvious mistake others have noted: this thing does not
need a list, it needs an iterable, and that iterable merely needs to yield
objects that have a split method, which yields... hashable objects. That's
it. This function could work just as well if line.split yielded integers.
It's what's awesome about things like collections.Counter. You can give it
whatever iterable you like, whether it yields characters, strings,
integers, the types of objects tracked by the garbage collector, you name
it. This is awesome. Declaring and casting between type templates is
confusing, verbose, scary and not awesome in general. It just makes me
think of C++ (sorry).

Let's say that this function needs to operate on strings, because at some
point in the future it will replace line.split() with something that
accounts for punctuation and whatnot, and ignore that this would really
mean it actually became a different function. (Do you expect str.split to
begin handling punctuation overnight?) Now the only mistake is that it
pretends it is specific to lists. This mistake is easily realized, so what
happens when you write your fancy library that fully declares its typing,
declaring generic interfaces where appropriate, and one of your
dependencies finally makes the switch over and gets it wrong? Now your
typechecker says your functions are making incorrect calls to that API
because you passed the result of a generator function to something that
expects a list. How often do you think this will happen? How many people
eager to type-check everything will not know how duck-typing matters in
their library or program and specify overly restrictive interfaces for
their functions? How many will assume checking declared types is a
substitute for unit testing? This is speculative, but being a regular
participant of #python on FreeNode, I already dread this.

Finally, if I abstract this proposal, it boils down to: The proposal wants
to take what has been so far a fully free-for-all attribute of functions,
and make it free-for-all-but-only-for-this-purpose. That's a bit... weird?
Maybe the mypy project could split its declaration and checking components,
sort of like I split clize(CLI argument parsing) and sigtools(high-level
signature operations and reflection improvements)? Isn't the way those
libraries declare types-to-be-checked the main way they will distinguish
each other for developers? Is there a problem preventing IDEs from
implementing a runner for mypy, much like eg. the way you can use pyflakes
as your compiler in vim?

Sorry for the long rant.

[1] Clize on PyPI: https://pypi.python.org/pypi/clize
[2] Old clize API:
https://github.com/epsy/clize/blob/e84637a631574e793719114ad7e40d0b36df1a78/README.rst#using-clize-in-your-programs
[3] New clize API involving annotations:
http://clize.readthedocs.org/en/latest/basics.html#converting-arguments
[4] Inspect.signature-compatible backport of annotations:
http://sigtools.readthedocs.org/en/latest/#sigtools.modifiers.annotate
[5] Backport of inspect.signature: https://pypi.python.org/pypi/funcsigs/0.4
[6] Rejected proposals from Function Annotations PEP:
http://legacy.python.org/dev/peps/pep-3107/#rejected-proposals
[7] Warnings for Beginners - Learn Python the Hard Way:
http://learnpythonthehardway.org/book/ex0.html#warnings-for-beginners

-Yann Kaiser


On 14 August 2014 13:36, Andrey Vlasovskikh <andrey.vlasovskikh at gmail.com>
wrote:

> On Thu, Aug 14, 2014 at 2:28 PM, Manuel Cer?n <ceronman at gmail.com> wrote:
>
> > One interesting feature of TypeScript is that it allows you to annotate
> > existing code without modifying it, by using external definition files.
> In
> > the JavaScript world, many people have contributed TypeScript annotation
> > files for popular JS libraries (http://definitelytyped.org/).
> >
> > I think this is possible in Python as well doing something like this:
> >
> > @annotate('math.ciel')
> > def ciel(x: float) -> int:
> >     pass
> >
> > I think this should be the way to go for annotating the stdlib. It has
> the
> > advantage that if the type syntax changes, it's possible to provide new
> type
> > annotations without changing the libraries at all, and even supporting
> older
> > versions. In this way the code and type annotations can evolve
> separately.
> >
> > Does mypy support something like this?
>
> We use something quite similar to TypeScript's repository of
> annotations in PyCharm. Here is our external annotations proposal and
> stubs for some stdlib modules
> (https://github.com/JetBrains/python-skeletons) that uses a custom
> type syntax in docstrings due to lack of a better standard, see
> README. We state in our proposal that we would like the standard to
> emerge. The idea being discussed here about using Mypy's type system
> is not new to us. As I've mentioned in the original thread, we have
> discussed it with Jukka Lehtosalo, the author of Mypy. Some initial
> ideas are listed here (https://github.com/pytypes/pytypes).
>
> --
> Andrey Vlasovskikh
> Web: http://pirx.ru/
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/d4637c92/attachment.html>

From apalala at gmail.com  Thu Aug 14 16:10:51 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Thu, 14 Aug 2014 09:40:51 -0430
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
In-Reply-To: <53EC9FC5.2010701@python.org>
References: <mailman.72122.1407959084.18129.python-ideas@python.org>
 <7DEBAD94-1B93-474A-9F2E-01009B665918@gmail.com>
 <CAP7+vJJKP=1QfQrUaQ0Rq5Et-o81+MmzWJaeUT8MRkGvvhwRtQ@mail.gmail.com>
 <7A31B791-B785-4858-9544-DDE2506B1E2C@gmail.com> <53EC7C9B.1000806@python.org>
 <CAN1YFWv323V2dggktk8QHRaj-RkOFdrdW2KRkKwJMcRtFR1eHA@mail.gmail.com>
 <53EC9FC5.2010701@python.org>
Message-ID: <CAN1YFWtM7uRgeV5LbJp2_LOWgOfL7GFuen2LujUMe9DGqgkFPg@mail.gmail.com>

On Thu, Aug 14, 2014 at 7:08 AM, Christian Heimes <christian at python.org>
wrote:

> As I tried to explain earlier neither users nor tools are able to
> analyze code like builtin types and functions. Exception annotations
> could help IDEs to figure out what kind of exception could be raised.
> Does it sound better to you?
>

It does, but it remains that annotations about exceptions will probably be
inconsistent or incomplete:

   1. If the function is changed, the annotation will remain valid even if
   the mentioned exceptions are not raised (this is the typical doc-comment
   problem).
   2. The annotation cannot be validated unless the raisables of called
   functions (objects, actually) are analysed.

Dealing with #2 means going the Java way and promoting incorrect exception
handlers throughout.

Not dealing with #2 means that the annotations are for documentation
purposes only, so the same information can go in the doc-comment.

This situation is unlike that for the proposed type signatures, because the
intent for those is that they are strictly verified (with mypy or some
other).

In short, an exceptions signature would amount to:

"This function may or may not raise the listed exceptions, and it may also
raise other exceptions not in the list.".

Yet, #2 can be dealt with by not being strict, and issuing warnings when
things seem broken, taking note that the Python callables that can raise
exceptions are not only functions and methods, but anything for which
callable(o)==True.

Cheers,

-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/7637427b/attachment-0001.html>

From antoine at python.org  Thu Aug 14 16:20:44 2014
From: antoine at python.org (Antoine Pitrou)
Date: Thu, 14 Aug 2014 10:20:44 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <lsigjt$k35$1@ger.gmane.org>


Hi,

So a couple more comments:

- there seems to be no detailed description of mypy's typing.py.  There 
are some docstrings in the module but they don't really explain the 
intended semantics of using those types (and, most importantly, 
combining and extending them). Or is the tutorial the description?

- mypy is a young module; is it actually proven on real-world codebases? 
if a PEP gets written, typing.py should be proposed on the "provisional" 
basis as described in PEP 411

- for the two reasons above I would expect a PEP to spell out and 
somehow motivate the typing semantics (by "typing" I do not mean the 
actual algorithm of a type checker, but the semantics of combining types 
and the mechanisms for extension), rather than defer to third-party docs 
and source code

- many discussion topics seems to revolve around the syntax for 
declarations; besides syntactic sugar for declarations, though, there 
should probably be a hook to describe typing in a more programmatic way, 
especially when parametric types ("generic"?) are involved. I can't 
think of an example, so I realize this comment may not be very helpful 
(!), but I'm sure we'll hit the limits of what declarative syntax can 
give us, especially once people try it out on non-trivial code bases. Do 
such hooks already exist?

- the typing module apparently has its own brand of generic functions 
(with multiple dispatch?). Are they actually necessary for typing 
declarations and type inference? If so, it would be nice if this could 
be unified with functools.singledispatch, or at least put in the 
functools namespace, and sufficiently similar API-wise that it doesn't 
stand out.

- it would be nice if the PEP could discuss at least a little bit the 
performance expectations of using the type system, and if there are 
design decisions that can have a (positive or negative) performance 
impact on the algorithms using it. I realize this is not top-priority 
for the linting use case, but it's still important (especially for 
linting large code bases, which is one of the proclaimed use cases).

(this reminds me how annoyed I am at the slowness of Sphinx on non-small 
reST docs)

Regards

Antoine.



From rymg19 at gmail.com  Thu Aug 14 16:37:56 2014
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 14 Aug 2014 09:37:56 -0500
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lsh6r6$rkn$1@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsh6r6$rkn$1@ger.gmane.org>
Message-ID: <CAO41-mPOE7TnJsmoNH38XKW7KjSCwpi0RkwuHwwwDR=zSwcRAQ@mail.gmail.com>

On Wed, Aug 13, 2014 at 9:27 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> Guido, as requesting, I read your whole post before replying. Please to
> the same. This response is both critical and supportive.
>
>
> On 8/13/2014 3:44 PM, Guido van Rossum wrote:
>
>  Yesterday afternoon I had an inspiring conversation with Bob Ippolito
>> (man of many trades, author of simplejson) and Jukka Lehtosalo (author
>> of mypy: http://mypy-lang.org/).
>>
>
> My main concern with static typing is that it tends to be
> anti-duck-typing, while I consider duck-typing to be a major *feature* of
> Python.  The example in the page above is "def fib(n: int):". Fib should
> get an count (non-negative integer) value, but it need not be an int, and
> 'half' the ints do not qualify. Reading the tutorial, I could not tell if
> it supports numbers.Number (which should approximate the domain from above.)
>

Very true; this is one of my fears. There are plenty of people who adore
static typing and will make everything in all their libraries static with
static this and static that. Maybe there could be a way to disable type
checks?


>
> Now consider an extended version (after Lucas).
>
> def fib(n, a, b):
>     i = 0
>     while i <= n:
>         print(i,a)
>         i += 1
>         a, b = b, a+b
>
> The only requirement of a, b is that they be addable. Any numbers should
> be allowed, as in fib(10, 1, 1+1j), but so should fib(5, '0', '1'). Addable
> would be approximated from below by Union(Number, str).
>
>
Unless MyPy added some sort of type classes...


>
>  Bob gave a talk at EuroPython about
>> what Python can learn from Haskell (and other languages); yesterday he
>> gave the same talk at Dropbox. The talk is online
>> (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad
>> strokes comes down to three suggestions:
>>
>>    (a) Python should adopt mypy's syntax for function annotations
>>
>
> -+ Syntax with no meaning is a bit strange. On the other hand, syntax not
> bound to semantics, or at least not bound to just one meaning is quite
> pythonic. '+' has two standard meanings, plus custom meanings embodied in
> .__add__ methods.
>
> + The current semantics of annotations is that they are added to functions
> objects as .__annotations__ (for whatever use) *and* used as part of
> inspect.signature and included in help(ob) responses.  In other words,
> annotations are already used in the stdlib.
>
> >>> def f(i:int) -> float: pass
>
> >>> from inspect import signature as sig
> >>> str(sig(f))
> '(i:int) -> float'
> >>> help(f)
> Help on function f in module __main__:
>
> f(i:int) -> float
>
> Idle calltips include them also. A appropriately flexible standardized
> notation would enhance this usage and many others.
>
> +-+ I see the point of "The goal is to make it possible to add type
> checking annotations to 3rd party modules (and even to the stdlib) while
> allowing unaltered execution of the program by the  (unmodified) Python 3.5
> interpreter." On the other hand, "pip install mypytyping" is not a huge
> burden. On the third hand, in the stdlib allows use in the stdlib.
>
>     (b) Python's use of mutabe [mutable] containers by default is wrong
>>
>
> The premise of this is partly wrong and partly obsolete. As far as I can
> remember, Python *syntax* only use tuples, not lists: "except (ex1, ex2):",
> "s % (val1, val2)", etc. The use of lists as the common format for data
> interchange between functions has largely been replaced by iterators.  This
> fact makes Python code much more generic, and anti-generic static typing
> more wrong.
>
> In remaining cases, 'wrong' is as much a philosophical opinion as a fact.
>
>
>     (c) Python should adopt some kind of Abstract Data Types
>>
>
> I would have to look at the talk to know what Jukka means.
>
>
>  Proposals (b) and (c) don't feel particularly actionable (if you
>> disagree please start a new thread, I'd be happy to discuss these
>> further if there's interest) but proposal (a) feels right to me.
>>
>
>  So what is mypy?  It is a static type checker for Python written by
>> Jukka for his Ph.D. thesis. The basic idea is that you add type
>> annotations to your program using some custom syntax, and when running
>> your program using the mypy interpreter, type errors will be found
>> during compilation (i.e., before the program starts running).
>>
>> The clever thing here is that the custom syntax is actually valid Python
>> 3, using (mostly) function annotations: your annotated program will
>> still run with the regular Python 3 interpreter. In the latter case
>> there will be no type checking, and no runtime overhead, except to
>> evaluate the function annotations (which are evaluated at function
>> definition time but don't have any effect when the function is called).
>>
>> In fact, it is probably more useful to think of mypy as a heavy-duty
>> linter than as a compiler or interpreter; leave the type checking to
>> mypy, and the execution to Python. It is easy to integrate mypy into a
>> continuous integration setup, for example.
>>
>> To read up on mypy's annotation syntax, please see the mypy-lang.org
>> <http://mypy-lang.org> website.
>>
>
> I did not see a 'reference' page, but the tutorial comes pretty close.
> http://mypy-lang.org/tutorial.html
> Beyond that, typings.py would be definitive,
> https://github.com/JukkaL/mypy/blob/master/lib-typing/3.2/typing.py
>
>
> > Here's just one complete example, to give a flavor:
>
>     from typing import List, Dict
>>
>>    def word_count(input: List[str]) -> Dict[str, int]:
>>
>
> The input annotation should be Iterable[str], which mypy does have.
>
>
>         result = {}  #type: Dict[str, int]
>>        for line in input:
>>            for word in line.split():
>>                result[word] = result.get(word, 0) + 1
>>        return result
>>
>
> The information that input is an Iterable[str] can be used either within
> the definition of word_count or at places where word_count is called.  A
> type aware checker, either in the editor or compiler, could check that the
> only uses of 'input' within the function is as input to functions declared
> to accept an Iterable or in for statements.
>
> Checking that the input to word_count is specifically Iterable[str] as
> opposed to any other Iterable may not be possible.  But I think what can be
> done, including enhancing help information, might be worth it.
>
> For instance, the parameter to s.join is named 'iterable'. Something more
> specific, either 'iterable_of_strings' or 'strings: Iterable[str]' would be
> more helpful. Indeed, there have been people posting on python list who
> thought that 'iterable' means iterable and that .join would call str() on
> each object. I think there are other cases where a parameter is given a
> bland under-informative type name instead of a context-specific semantic
> name just because there was no type annotation available. There are places
> where the opposite problem occurs, too specific instead of too general,
> where iterable parameters are still called 'list'.
>
>  Note that the #type: comment is part of the mypy syntax; mypy uses
>> comments to declare types in situations where no syntax is available --
>> although this particular line could also be written as follows:
>>
>>      result = Dict[str, int]()
>>
>> Either way the entire function is syntactically valid Python 3, and a
>> suitable implementation of typing.py (containing class definitions for
>> List and Dict, for example) can be written to make the program run
>> correctly. One is provided as part of the mypy project.
>>
>> I should add that many of mypy's syntactic choices aren't actually new.
>> The basis of many of its ideas go back at least a decade: I blogged
>> about this topic in 2004
>> (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 -- see also the
>> two followup posts linked from the top there).
>>
>> I'll emphasize once more that mypy's type checking happens in a separate
>> pass: no type checking happens at run time (other than what the
>> interpreter already does, like raising TypeError on expressions like
>> 1+"1").
>>
>> There's a lot to this proposal, but I think it's possible to get a PEP
>> written, accepted and implemented in time for Python 3.5, if people are
>> supportive. I'll go briefly over some of the action items.
>>
>> *(1) A change of direction for function annotations*
>>
>> PEP 3107 <http://legacy.python.org/dev/peps/pep-3107/>, which introduced
>>
>> function annotations, is intentional non-committal about how function
>> annotations should be used. It lists a number of use cases, including
>> but not limited to type checking. It also mentions some rejected
>> proposals that would have standardized either a syntax for indicating
>> types and/or a way for multiple frameworks to attach different
>> annotations to the same function. AFAIK in practice there is little use
>> of function annotations in mainstream code, and I propose a conscious
>> change of course here by stating that annotations should be used to
>> indicate types and to propose a standard notation for them.
>>
>
> There are many uses for type information and I think Python should remain
> neutral among them.
>
>
>  (We may have to have some backwards compatibility provision to avoid
>> breaking code that currently uses annotations for some other purpose.
>> Fortunately the only issue, at least initially, will be that when
>> running mypy to type check such code it will produce complaints about
>> the annotations; it will not affect how such code is executed by the
>> Python interpreter. Nevertheless, it would be good to deprecate such
>> alternative uses of annotations.)
>>
>
> I can imagine that people who have used annotations might feel a bit
> betrayed by deprecation of a new-in-py3 feature.  But I do not think it
> necessary to do so.  Tools that work with mypy annotations, including mypy
> itself, should only assume mypy typing if typing is imported.  No 'import
> typing', no 'Warning: annotation does not follow typing rules."  If
> 'typing' were a package with a 'mypy' module, the door would be left open
> to other 'blessed' typing modules.
>
>  *(2) A specification for what to add to Python 3.5*
>>
>>
>> There needs to be at least a rough consensus on the syntax for
>> annotations, and the syntax must cover a large enough set of use cases
>> to be useful. Mypy is still under development, and some of its features
>> are still evolving (e.g. unions were only added a few weeks ago). It
>> would be possible to argue endlessly about details of the notation, e.g.
>> whether to use 'list' or 'List', what either of those means (is a
>> duck-typed list-like type acceptable?) or how to declare and use type
>> variables, and what to do with functions that have no annotations at all
>> (mypy currently skips those completely).
>>
>> I am proposing that we adopt whatever mypy uses here, keeping discussion
>> of the details (mostly) out of the PEP. The goal is to make it possible
>> to add type checking annotations to 3rd party modules (and even to the
>> stdlib) while allowing unaltered execution of the program by the
>> (unmodified) Python 3.5 interpreter. The actual type checker will not be
>> integrated with the Python interpreter, and it will not be checked into
>> the CPython repository. The only thing that needs to be added to the
>> stdlib is a copy of mypy's typing.py module. This module defines several
>> dozen new classes (and a few decorators and other helpers) that can be
>> used in expressing argument types. If you want to type-check your code
>> you have to download and install mypy and run it separately.
>>
>> The curious thing here is that while standardizing a syntax for type
>> annotations, we technically still won't be adopting standard rules for
>> type checking.
>>
>
> Fine with me, as that is not the only use.  And even for type checking,
> there is the choice between accept unless clearly wrong, versus reject
> unless clearly right.
>
>
> > This is intentional. First of all, fully specifying all
>
>> the type checking rules would make for a really long and boring PEP (a
>> much better specification would probably be the mypy source code).
>> Second, I think it's fine if the type checking algorithm evolves over
>> time, or if variations emerge.
>>
>
> As in the choice between accept unless clearly wrong, versus reject unless
> clearly right.
>
>
> > The worst that can happen is that you
>
>> consider your code correct but mypy disagrees; your code will still run.
>>
>> That said, I don't want to /completely/ leave out any specification. I
>>
>> want the contents of the typing.py module to be specified in the PEP, so
>> that it can be used with confidence. But whether mypy will complain
>> about your particular form of duck typing doesn't have to be specified
>> by the PEP. Perhaps as mypy evolves it will take options to tell it how
>> to handle certain edge cases. Forks of mypy (or entirely different
>> implementations of type checking based on the same annotation syntax)
>> are also a possibility. Maybe in the distant future a version of Python
>> will take a different stance, once we have more experience with how this
>> works out in practice, but for Python 3.5 I want to restrict the scope
>> of the upheaval.
>>
>
> As usual, we should review the code before acceptance. It is not clear to
> me how much of the tutorial is implemented, as it says "Some of these
> features might never see the light of day. " ???
>
>  *Appendix -- Why Add Type Annotations?
>>
>> *
>> The argument between proponents of static typing and dynamic typing has
>> been going on for many decades. Neither side is all wrong or all right.
>> Python has traditionally fallen in the camp of extremely dynamic typing,
>> and this has worked well for most users, but there are definitely some
>> areas where adding type annotations would help.
>>
>
> The answer to why on the mypy page is 'easier to find bugs', 'easier
> maintenance'.  I find this under-convincing as sufficient justification in
> itself. I don't think there are many bugs on the tracker due to calling
> functions with the wrong type of object. Logic errors, ignored corner
> cases, and system idiosyncrasies are much more of a problem.
>
> Your broader list is more convincing.
>
>
>  - Editors (IDEs) can benefit from type annotations; they can call out
>> obvious mistakes (like misspelled method names or inapplicable
>> operations) and suggest possible method names. Anyone who has used
>> IntelliJ or Xcode will recognize how powerful these features are, and
>> type annotations will make such features more useful when editing Python
>> source code.
>>
>> - Linters are an important tool for teams developing software. A linter
>> doesn't replace a unittest, but can find certain types of errors better
>> or quicker. The kind of type checking offered by mypy works much like a
>> linter, and has similar benefits; but it can find problems that are
>> beyond the capabilities of most linters.
>>
>
> Currently, Python linters do not have standard type annotations to work
> with. I suspect that programs other than mypy would use them if available.
>
>
>  - Type annotations are useful for the human reader as well! Take the
>> above word_count() example. How long would it have taken you to figure
>> out the types of the argument and return value without annotations?
>>
>
> Under a minute, including the fact the the annotation was overly
> restrictive.  But then I already know that only a mutation method can
> require a list.
>
>  Currently most people put the types in their docstrings; developing a
>> standard notation for type annotations will reduce the amount of
>> documentation that needs to be written, and running the type checker
>> might find bugs in the documentation, too. Once a standard type
>> annotation syntax is introduced, it should be simple to add support for
>> this notation to documentation generators like Sphinx.
>>
>> - Refactoring. Bob's talk has a convincing example of how type
>> annotations help in (manually) refactoring code. I also expect that
>> certain automatic refactorings will benefit from type annotations --
>> imagine a tool like 2to3 (but used for some other transformation)
>> augmented by type annotations, so it will know whether e.g. x.keys() is
>> referring to the keys of a dictionary or not.
>>
>> - Optimizers. I believe this is actually the least important
>> application, certainly initially. Optimizers like PyPy or Pyston
>> <https://github.com/dropbox/pyston> wouldn't be able to fully trust the
>>
>> type annotations, and they are better off using their current strategy
>> of optimizing code based on the types actually observed at run time. But
>> it's certainly feasible to imagine a future optimizer also taking type
>> annotations into account.
>>
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/2109b477/attachment-0001.html>

From breamoreboy at yahoo.co.uk  Thu Aug 14 16:46:52 2014
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Thu, 14 Aug 2014 15:46:52 +0100
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CANUJvPVgQOTiGfCB=LSaBvj=5uG0RrZah9G8xPZGF+GZw4N26w@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CA+VECoVO0COba4u3o5YD0shE0r-P8-q99sWMQtpRkisQ8wa8UQ@mail.gmail.com>
 <CAP7+vJLKNydsdLuc8jRjhfYr_R8P7u1Jg1aZ4Yb1LA1c-t3w1g@mail.gmail.com>
 <CA+VECoXkb+=UeSQbckY+prOgKx7AY4yUwFRV9ogPTTQ+mR=DOA@mail.gmail.com>
 <CA+qPcwsyRW418k6LuwOQoEnBb+NAfD9ZkgqQhq9hJcvfv7nWNw@mail.gmail.com>
 <CANUJvPVgQOTiGfCB=LSaBvj=5uG0RrZah9G8xPZGF+GZw4N26w@mail.gmail.com>
Message-ID: <lsii4t$agg$1@ger.gmane.org>

On 14/08/2014 14:59, Yann Kaiser wrote:

> New brains are mostly taught to use python 2[7] or keep compatibility with it,

> [7] Warnings for Beginners - Learn Python the Hard Way:
> http://learnpythonthehardway.org/book/ex0.html#warnings-for-beginners
>
> -Yann Kaiser

This is the second time in a few days that I've seen a reference to that 
and IMHO it's simply plain wrong and should be ignored.

Just my ?0.02p worth.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From encukou at gmail.com  Thu Aug 14 16:52:33 2014
From: encukou at gmail.com (Petr Viktorin)
Date: Thu, 14 Aug 2014 16:52:33 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lsigjt$k35$1@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsigjt$k35$1@ger.gmane.org>
Message-ID: <CA+=+wqCJ7eA0f2K1kgytb7wFz44eYzxfKmN4Pg7Vc1d+UDk6_A@mail.gmail.com>

It seems to me that rather than adding a module which is only used by
one project so far to the standard library is a bit premature.

I support optional typing, but why push this to stdlib now? Wouldn't
it be better to wait until most IDEs/linters all agree on this syntax,
until freezing it in stdlib? So far typing seems to be a part of mypy,
shouldn't it spend some time on PyPI first?

I'm also sure about there not being other uses of annotations -- clize
aside, there are not many widely used Python3-only 3rd party
libraries, so it's no surprise that nothing big is built around Python
3 features.

Maybe the way from PEP 3107's "here's a feature, use it for whatever
you like" to "annotations are for typing declarations, using
mypy/typing syntax" should include a step of "if you use annotations
for typing, use mypy/typing syntax for it". (And perhaps it should end
there.)

From apalala at gmail.com  Thu Aug 14 16:59:54 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Thu, 14 Aug 2014 10:29:54 -0430
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <CAN1YFWsjrBfijwnuwfuVqgSHODdAGyBOy6JMV3QuwgNxJ2O9qg@mail.gmail.com>

On Wed, Aug 13, 2014 at 3:14 PM, Guido van Rossum <guido at python.org> wrote:

> I am proposing that we adopt whatever mypy uses here, keeping discussion
> of the details (mostly) out of the PEP. The goal is to make it possible to
> add type checking annotations to 3rd party modules (and even to the stdlib)
> while allowing unaltered execution of the program by the (unmodified)
> Python 3.5 interpreter.


To the bottom of things...

About the second time I wrote about Python ("Why not Python", 2007) I
dismissed it as serious software development environment because the lack
of static type checking hindered the creation of proper software
development environments.

http://blog.neogeny.org/why-not-python.html

So, Why do I now have doubts about adding support for static type checking?

I've been programming in almost-only Python for several years now, and this
discussion had me think hard about "Why?".

The answer is simple: I never was as productive as I've been since I've
centered on Python.

But, again Why?

Despite what my '07 article says, the IDE I use is pythonized-VIM and the
command line. Where does the productivity come from?

   1. Readability with the right amount of succinctness. Python programs
   are very small, but understandable.
   2. The breadth and design consistency of the standard library. Some 70%?
   of what I need is there, and the design consistency makes it easy
   (intiutive) to use.
   3. PyPi covers another 28%.
   4. The Zen of Python (import this) permeates all of the above, including
   most third-party packages. The ecosystem is consistent too. It's a culture.

What do I fear? I think it is that Python be transformed into a programming
language different from the one that now makes me so productive.

I studied Ruby, and I don't like it. I've been studying Go, and I don't
like it. One must like the concepts and the power, sure, but the syntax
required for some day-to-day stuff stinks like trouble;  simple stuff is so
complicated to express and so easy to get wrong...

I hate "List[str]" and "Dict[str:int]". Where did those come from?
Shouldn't they (as others have proposed) be "[str]" and "{str:int}"? What
about tuples?
Why not write a similar, but different programming language that targets
the Cython runtime and includes all the desired features?

All said, this is my proposal.

The PSF could support (even fund) MyPy and similar projects, promoting
their maturity and their convergence. The changes in 3.5 would be limited
but enough to enable those efforts, and those of the several IDE
tool-smiths (changes in annotations, and maybe in ABCs). Basically, treat
MyPy as PyPy or NumPy (which got '::'). It's in Python's history to enable
third-party developments and then adopt what's mature or become the
de-facto standard.
Then, on a separate line of work, it would be good to think about how to
enable different programming languages to target the CPython environment
(because of #2, #3, and #4 above), maybe by improving AST creation and
AST-to-bytecode? There could be other languages targeting the CPython
runtime, which is the relationship that Scala, Jython, IronPython, and
others have to their own runtimes.

-1 for standardizing static type checking in 3.5

Cheers,

-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/5f6dce8f/attachment.html>

From abarnert at yahoo.com  Thu Aug 14 17:21:40 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 14 Aug 2014 08:21:40 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAO41-mPOE7TnJsmoNH38XKW7KjSCwpi0RkwuHwwwDR=zSwcRAQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsh6r6$rkn$1@ger.gmane.org>
 <CAO41-mPOE7TnJsmoNH38XKW7KjSCwpi0RkwuHwwwDR=zSwcRAQ@mail.gmail.com>
Message-ID: <E200320E-014F-46DC-A2C8-E7F221CD6976@yahoo.com>

On Aug 14, 2014, at 7:37, Ryan Gonzalez <rymg19 at gmail.com> wrote:

>> On 8/13/2014 3:44 PM, Guido van Rossum wrote:

>> Now consider an extended version (after Lucas).
>> 
>> def fib(n, a, b):
>>     i = 0
>>     while i <= n:
>>         print(i,a)
>>         i += 1
>>         a, b = b, a+b
>> 
>> The only requirement of a, b is that they be addable. Any numbers should be allowed, as in fib(10, 1, 1+1j), but so should fib(5, '0', '1'). Addable would be approximated from below by Union(Number, str).
> 
> Unless MyPy added some sort of type classes...

By "type classes", do you mean this in the Haskell sense, or do you mean classes used just for typing--whether more granular ABCs (like an Addable which both Number and AnyStr and probably inherit) or typing.py type specifiers (like an Addable defined as Union(Number, AnyStr)?

It's also worth noting that the idea that this function should take a Number or a str seems way off. It's questionable whether it should accept str, but if it does, shouldn't it also accept bytes, bytearray, and other string-like types? What about sequences? And meanwhile, whether or not it accepts str, it should probably accept np.ndarray and other types of element-wise adding types. If you create an Addable type, it has to define, globally, which of those counts as addable, but different functions will have different definitions that make sense.

In fact, look at the other discussion going on. People want to ensure that sum only works on numbers or number-like types (and does that include NumPy arrays or not?), while others want to change it to work on all sequences, or only mutable sequences with += plus str because it effectively has magical += handling under the covers, etc. If we can't even decide what Addable means for one specific function that everyone has experience with...

On the other hand, if sum could have been annotated to tell us the author's intention (or, rather, the consensus of the dev list), then all of these recurring arguments about summing str would go away. Until someone defined a number-like class (maybe even one that meets the ABC, but by calling register on it) and sum won't work for him.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/90648cbe/attachment-0001.html>

From brett at python.org  Thu Aug 14 18:01:08 2014
From: brett at python.org (Brett Cannon)
Date: Thu, 14 Aug 2014 16:01:08 +0000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsigjt$k35$1@ger.gmane.org>
 <CA+=+wqCJ7eA0f2K1kgytb7wFz44eYzxfKmN4Pg7Vc1d+UDk6_A@mail.gmail.com>
Message-ID: <CAP1=2W6veDE_VEb_YU3-DTYpZAF1Lfe+8pXkqg91x=hyjxWN_g@mail.gmail.com>

On Thu Aug 14 2014 at 10:53:37 AM Petr Viktorin <encukou at gmail.com> wrote:

> It seems to me that rather than adding a module which is only used by
> one project so far to the standard library is a bit premature.
>
> I support optional typing, but why push this to stdlib now? Wouldn't
> it be better to wait until most IDEs/linters all agree on this syntax,
> until freezing it in stdlib? So far typing seems to be a part of mypy,
> shouldn't it spend some time on PyPI first?
>

Because as you have noticed in this thread there are already a ton of
competing solutions and no consensus has been reached. Sometimes Guido
and/or python-dev have to step in and simply say "there is obvious need and
the community is not reaching consensus, so we will make the decision
ourselves".


>
> I'm also sure about there not being other uses of annotations -- clize
> aside, there are not many widely used Python3-only 3rd party
> libraries, so it's no surprise that nothing big is built around Python
> 3 features.
>
> Maybe the way from PEP 3107's "here's a feature, use it for whatever
> you like" to "annotations are for typing declarations, using
> mypy/typing syntax" should include a step of "if you use annotations
> for typing, use mypy/typing syntax for it". (And perhaps it should end
> there.)
>

That's a possibility. Another thing to support this approach is that if
something like List[str] is used over `[str]`  then the returned object can
subclass some common superclass which can be typechecked for to know that
the annotation is from typing.py and not clize/scription and continue to
function. That way you can avoid any decorators adding some attribute on
functions to know that types have been specified while allowing function
annotations to be used for anything. Otherwise a @typing.ignore decorator
could also exist for alternative formats to use (since typing has been the
most common use case and decorating your single main() function with
@typing.ignore is not exactly heavy-handed).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/288aede0/attachment.html>

From varma.sunjay at gmail.com  Thu Aug 14 18:01:37 2014
From: varma.sunjay at gmail.com (Sunjay Varma)
Date: Thu, 14 Aug 2014 12:01:37 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
Message-ID: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>

I am strongly opposed to this entire proposal. As Juancarlo points out,
Python programs are small, but very understandable. I think this syntax
detracts from that. I'll suggest an alternative further down in my reply.

One benefit of Python that makes it so attractive for new programmers and
even old programmers alike is that you can usually pick out any piece of
Python code and begin to understand it immediately. Even if you come from a
different programming language, Python is written in
english explicitly using words like "and" and "or". Those constructs, as
opposed to "&&" or "||" make the language less scary for new developers and
in general easier to read as well. It's also easier to type regular english
words (no need to use the shift key). Using the annotation syntax this
heavily will detract very much from the readability of Python and from the
overall usability as well. Programs are read more times than they are
written.

Several years ago, before I had any programming experience in any language
at all, I needed to edit some Python code to make something I was doing
work. Without any experience at all, I was able to look through the (small)
program I was editing and figure out exactly what I needed to adjust.
Without Python being such a clean, almost English language, that would have
been impossible.

Though the annotation syntax is already present in Python 3, I would argue
that using this for type annotations will get very messy very quickly. If
I'm understanding the syntax correctly, writing any function using a large
library with many nested subpackages could result in code like this:

    import twisted.protocols.mice.mouseman

    def
process_mouseman(inputMouseMan: twisted.protocols.mice.mouseman.MouseMan)
-> twisted.protocols.mice.mouseman.MouseMan:
        pass

That function definition is 122 characters long. Far more than what PEP8
recommends. Though this example was crafted to illustrate my point (I don't
think most people would really write code like this), it is easy to see
that this kind of code is possible and may sometimes be written by some
less experienced programmers. It demonstrates how messy things can get even
with just one parameter.

It is also easy to see that it is very difficult to parse out what is going
on in that function. Adding type annotations inline makes it very difficult
to quickly get an idea of what arguments a function takes and in what
order. It detracts from the overall readability of a program and can also
lead to very poorly formatted programs that break the guidelines in PEP8.
Though I have only demonstrated this for function declarations, the example
could also be extended to inline statement comments as well. Things get too
messy too quickly.

My Alternative Proposal:
As an alternative, I would like to propose a syntax that Pycharm already
supports:
http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.html

Since this type information isn't going to be used at runtime in the
regular Python interpreter anyway, why not have it in the function
docstring instead? This provides both readability and type checking.
Standardizing that syntax or at least adding it as an optional way to check
your program would in my opinion be a much better addition to the language.
This approach needs no new syntax, keeps readability and allows the
programmer to add additional documentation without going over the 80
character limit.

Additionally, this approach can be used by documentation generators as well
and removes any duplication from the function declaration and the docstring.

Here's a taste of what that looks like:
    class SimpleEquation(object):

        def demo(self, a, b, c):
            """
            This function returns the product of a, b and c
            @type self: SimpleEquation
            :param a: int - The first number
            :param b: int
            :param c: int - The third number should not be zero and should
also
                only be -1 if you enjoy carrots (this comment spans 2 lines)
            :return: int
            """
            return a * b * c

Overall, I think overloading function declarations and inline comments is a
bad idea. It promotes writing code with poor readability and in general
adds a lot of extra bits to the language that (from the sounds of your
proposal) aren't even going to be used by the main interpreter.

On the original proposal:
These changes really do seem to be overestimating the staticness of Python
programs as well. What about functions that don't care about the type? What
about functions that only want you to pass in an object that implements
__iter__? Python should not become a language where developers are required
to add hundreds of odd cast() calls every time they choose to pass a
different, but still compatible type, to a function. This syntax makes too
many assumptions about what developers know about their code. What if I
develop a similar, but different class that is compatible with an existing
function? If that function doesn't specify that my class can be used, my
perfectly valid code will be rejected.

-1 to adding mypy annotations to Python 3.

Sunjay

On Thu, Aug 14, 2014 at 10:59 AM, Juancarlo A?ez <apalala at gmail.com> wrote:

>
> On Wed, Aug 13, 2014 at 3:14 PM, Guido van Rossum <guido at python.org>
> wrote:
>
>> I am proposing that we adopt whatever mypy uses here, keeping discussion
>> of the details (mostly) out of the PEP. The goal is to make it possible to
>> add type checking annotations to 3rd party modules (and even to the stdlib)
>> while allowing unaltered execution of the program by the (unmodified)
>> Python 3.5 interpreter.
>
>
> To the bottom of things...
>
> About the second time I wrote about Python ("Why not Python", 2007) I
> dismissed it as serious software development environment because the lack
> of static type checking hindered the creation of proper software
> development environments.
>
> http://blog.neogeny.org/why-not-python.html
>
> So, Why do I now have doubts about adding support for static type checking?
>
> I've been programming in almost-only Python for several years now, and
> this discussion had me think hard about "Why?".
>
> The answer is simple: I never was as productive as I've been since I've
> centered on Python.
>
> But, again Why?
>
> Despite what my '07 article says, the IDE I use is pythonized-VIM and the
> command line. Where does the productivity come from?
>
>    1. Readability with the right amount of succinctness. Python programs
>    are very small, but understandable.
>    2. The breadth and design consistency of the standard library. Some
>    70%? of what I need is there, and the design consistency makes it easy
>    (intiutive) to use.
>    3. PyPi covers another 28%.
>    4. The Zen of Python (import this) permeates all of the above,
>    including most third-party packages. The ecosystem is consistent too. It's
>    a culture.
>
> What do I fear? I think it is that Python be transformed into a
> programming language different from the one that now makes me so productive.
>
> I studied Ruby, and I don't like it. I've been studying Go, and I don't
> like it. One must like the concepts and the power, sure, but the syntax
> required for some day-to-day stuff stinks like trouble;  simple stuff is so
> complicated to express and so easy to get wrong...
>
> I hate "List[str]" and "Dict[str:int]". Where did those come from?
> Shouldn't they (as others have proposed) be "[str]" and "{str:int}"? What
> about tuples?
> Why not write a similar, but different programming language that targets
> the Cython runtime and includes all the desired features?
>
> All said, this is my proposal.
>
> The PSF could support (even fund) MyPy and similar projects, promoting
> their maturity and their convergence. The changes in 3.5 would be limited
> but enough to enable those efforts, and those of the several IDE
> tool-smiths (changes in annotations, and maybe in ABCs). Basically, treat
> MyPy as PyPy or NumPy (which got '::'). It's in Python's history to enable
> third-party developments and then adopt what's mature or become the
> de-facto standard.
> Then, on a separate line of work, it would be good to think about how to
> enable different programming languages to target the CPython environment
> (because of #2, #3, and #4 above), maybe by improving AST creation and
> AST-to-bytecode? There could be other languages targeting the CPython
> runtime, which is the relationship that Scala, Jython, IronPython, and
> others have to their own runtimes.
>
> -1 for standardizing static type checking in 3.5
>
> Cheers,
>
> --
> Juancarlo *A?ez*
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Sunjay Varma
Python Programmer & Web Developer
www.sunjay.ca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/334b83f1/attachment-0001.html>

From ceronman at gmail.com  Thu Aug 14 18:34:43 2014
From: ceronman at gmail.com (=?UTF-8?Q?Manuel_Cer=C3=B3n?=)
Date: Thu, 14 Aug 2014 18:34:43 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lsh6r6$rkn$1@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsh6r6$rkn$1@ger.gmane.org>
Message-ID: <CA+VECoUo_Asn+JTt9dw-FECD9z10KEy6E+FhdQwhN-jTt2j=+Q@mail.gmail.com>

On Thu, Aug 14, 2014 at 4:27 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>
> My main concern with static typing is that it tends to be
> anti-duck-typing, while I consider duck-typing to be a major *feature* of
> Python.  The example in the page above is "def fib(n: int):". Fib should
> get an count (non-negative integer) value, but it need not be an int, and
> 'half' the ints do not qualify. Reading the tutorial, I could not tell if
> it supports numbers.Number (which should approximate the domain from above.)


This is a good point. But I think that static typing and duck typing are
not mutually exclusive. TypeScript does this very nicely by defining
structural interfaces (http://www.typescriptlang.org/Handbook#interfaces).
With them is possible to define a given behaviour and any object capable of
providing that behaviour is accepted, without having to be part of any
specific type hierarchy or having to explicitly register as implementation
of certain specification. That's basically what duck typing means.

For example:

interface Named {
    name: string;
say(): string;
}

function doSomething(x: Named) {
console.log(x.name);
}

doSomething({name: "hello", say: function() { return this.name }}); // OK
doSomething({something: "hello"}); // ERROR

I think something like this is a must have for mypy. In Python, I've been
playing with something similar (https://github.com/ceronman/typeannotations)
but for runtime checking only:

>>> class Person(Interface):
    ...     name = str
    ...     age = int
    ...     def say_hello(name: str) -> str:
    ...             pass

Any object defining those the name, age and say_hello() members is a valid
implementation of that interface. For example:

>>> class Developer:
...     def __init__(self, name, age):
...             self.name = name
...             self.age = age
...     def say_hello(self, name: str) -> str:
...             return 'hello ' + name
...
>>> isinstance(Developer('bill', 20), Person)
True

Are there any plans for adding something like this to mypy?

Manuel.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/b276c741/attachment.html>

From apalala at gmail.com  Thu Aug 14 19:17:48 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Thu, 14 Aug 2014 12:47:48 -0430
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CA+VECoUo_Asn+JTt9dw-FECD9z10KEy6E+FhdQwhN-jTt2j=+Q@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsh6r6$rkn$1@ger.gmane.org>
 <CA+VECoUo_Asn+JTt9dw-FECD9z10KEy6E+FhdQwhN-jTt2j=+Q@mail.gmail.com>
Message-ID: <CAN1YFWtzKBFhovpagbZ_BaeVyCKXzVkr7jeOM7JyV7YnAq3bkg@mail.gmail.com>

On Thu, Aug 14, 2014 at 12:04 PM, Manuel Cer?n <ceronman at gmail.com> wrote:

> For example:
>
> interface Named {
>     name: string;
> say(): string;
> }
>
> function doSomething(x: Named) {
> console.log(x.name);
> }
>
> doSomething({name: "hello", say: function() { return this.name }}); // OK
> doSomething({something: "hello"}); // ERROR
>

That is so Java....!


-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/db3c76fd/attachment.html>

From ethan at stoneleaf.us  Thu Aug 14 19:29:01 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 14 Aug 2014 10:29:01 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
Message-ID: <53ECF1DD.8040808@stoneleaf.us>

On 08/14/2014 09:01 AM, Sunjay Varma wrote:
>
> Additionally, this approach can be used by documentation generators as well and removes any duplication from the
> function declaration and the docstring.
>
> Here's a taste of what that looks like:
>      class SimpleEquation(object):
>          def demo(self, a, b, c):
>              """
>              This function returns the product of a, b and c
>              @type self: SimpleEquation
>              :param a: int - The first number
>              :param b: int
>              :param c: int - The third number should not be zero and should also
>                  only be -1 if you enjoy carrots (this comment spans 2 lines)
>              :return: int
>              """
>              return a * b * c


+1  I like this much more.

--
~Ethan~

From steve at pearwood.info  Thu Aug 14 19:31:03 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Aug 2014 03:31:03 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <20140814173103.GO4525@ando>

As requested, I've read the whole post, and the whole thread, before 
responding :-)

On Wed, Aug 13, 2014 at 12:44:21PM -0700, Guido van Rossum wrote:

>   (a) Python should adopt mypy's syntax for function annotations
[...]

I'm very excited to see functional annotations being treated seriously, 
I think the introduction of static typing, even optional, has the 
potential to radically change the nature of Python language and I'm not 
sure if that will be good or bad :-) but it is reassuring to hear that 
the intention is that it will be treated more like an optional linter 
than as a core part of the language.

On the other hand, are you aware of Cobra, which explicitly was modelled 
on Python but with optional static typing?

http://cobra-language.com/


[...]
> *(1) A change of direction for function annotations*
> [...] I propose a conscious change of course here by stating
> that annotations should be used to indicate types and to propose a standard
> notation for them.

And in a later email, Guido also stated:

> I want to eventually phase out other uses of function annotations

That disappoints me and I hope you will reconsider.

I've spent some time thinking about using annotations for purposes other 
than type checking, but because most of my code has to run on Python 2, 
there's nothing concrete. One example is that I started exploring ways 
to use annotations as documentation for the statistics module in 3.4, 
except that annotations are banned from the standard library. (Naturally 
I haven't spent a lot of time on something that I knew was going to be 
rejected.) I came up with ideas like this:

def mean(data) -> '? = ?(x)/n':

def pvariance(data) -> '?? = ?(x - ?)? ? n':

which might have been a solution to this request:

http://bugs.python.org/issue21046

had annotations been allowed in the stdlib. Regardless of whether this 
specific idea is a good one or not, I will be disappointed if 
annotations are limited to one and only one use. I don't mind if there 
is a standard, default, set of semantics so long as there is a way to 
opt-out and use something else:

@use_spam_annotations
def frobnicate(x: spam, y: eggs)->breakfast:
    ...

for example. Whatever the mechanism, I think Python should not prohibit 
or deprecate other annotation semantics.


> *(2) A specification for what to add to Python 3.5*
> 
> There needs to be at least a rough consensus on the syntax for annotations,
> and the syntax must cover a large enough set of use cases to be useful.
> Mypy is still under development, and some of its features are still
> evolving (e.g. unions were only added a few weeks ago). It would be
> possible to argue endlessly about details of the notation, e.g. whether to
> use 'list' or 'List', what either of those means (is a duck-typed list-like
> type acceptable?) or how to declare and use type variables, and what to do
> with functions that have no annotations at all (mypy currently skips those
> completely).

It doesn't sound to me like the mypy syntax is mature enough to bless, 
let alone to start using it in the standard library.


> I am proposing that we adopt whatever mypy uses here, keeping discussion of
> the details (mostly) out of the PEP. The goal is to make it possible to add
> type checking annotations to 3rd party modules (and even to the stdlib)
> while allowing unaltered execution of the program by the (unmodified)
> Python 3.5 interpreter. The actual type checker will not be integrated with
> the Python interpreter, and it will not be checked into the CPython
> repository. The only thing that needs to be added to the stdlib is a copy
> of mypy's typing.py module. 

What happens when the typing.py module in the standard library gets out 
of sync with the typing.py module in mypy?


[...]
> *Appendix -- Why Add Type Annotations?*
> The argument between proponents of static typing and dynamic typing has
> been going on for many decades. Neither side is all wrong or all right.
> Python has traditionally fallen in the camp of extremely dynamic typing,
> and this has worked well for most users, but there are definitely some
> areas where adding type annotations would help.

Some people have probably already seen this, but I have found this 
article to be very useful for understanding why static and dynamic type 
checking can be complementary rather than opposed:

http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/



-- 
Steven

From ndbecker2 at gmail.com  Thu Aug 14 19:33:47 2014
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 14 Aug 2014 13:33:47 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <lsirtr$gri$1@ger.gmane.org>

Does mypy support annotation of functions implemented in C code?

If I extend cpython via C-API, does mypy provide a mechanism to annotate those 
functions?


From steve at pearwood.info  Thu Aug 14 19:35:19 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Aug 2014 03:35:19 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CA+VECoUo_Asn+JTt9dw-FECD9z10KEy6E+FhdQwhN-jTt2j=+Q@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsh6r6$rkn$1@ger.gmane.org>
 <CA+VECoUo_Asn+JTt9dw-FECD9z10KEy6E+FhdQwhN-jTt2j=+Q@mail.gmail.com>
Message-ID: <20140814173519.GP4525@ando>

On Thu, Aug 14, 2014 at 06:34:43PM +0200, Manuel Cer?n wrote:
> On Thu, Aug 14, 2014 at 4:27 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> >
> > My main concern with static typing is that it tends to be
> > anti-duck-typing, while I consider duck-typing to be a major *feature* of
> > Python.  The example in the page above is "def fib(n: int):". Fib should
> > get an count (non-negative integer) value, but it need not be an int, and
> > 'half' the ints do not qualify. Reading the tutorial, I could not tell if
> > it supports numbers.Number (which should approximate the domain from above.)
> 
> 
> This is a good point. But I think that static typing and duck typing are
> not mutually exclusive.
[...]
> Are there any plans for adding something like this to mypy?

The mypy FAQs claim to be focusing on nominative typing, but haven't 
ruled out structural typing in the future:

http://www.mypy-lang.org/faq.html



-- 
Steven

From stefan_ml at behnel.de  Thu Aug 14 19:37:55 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 14 Aug 2014 19:37:55 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <lsis5j$l94$1@ger.gmane.org>

Guido van Rossum schrieb am 13.08.2014 um 21:44:
> Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man
> of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy:
> http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can
> learn from Haskell (and other languages); yesterday he gave the same talk
> at Dropbox. The talk is online (
> https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad
> strokes comes down to three suggestions:
> 
>   (a) Python should adopt mypy's syntax for function annotations
> [...] proposal (a) feels right to me.

FWIW, Cython currently understands these forms of function argument
annotations in Python code:

    x: dict
    x: {"type": dict}
    x: {"type": "dict"}
    x: {"ctype": "long double"}

The "ctype" values that are usable here obviously only include those that
can be converted from and to Python types, e.g. no arbitrary pointers.

It'd be nice to have a way to declare container item types, but that's
never really been a priority in the Cython project so far. Declaring
protocols, OTOH, is pretty useless for a compiler, as it's obvious from the
code which protocols are being used on a given value (iteration, item
access, etc.).

It's clear that restricting everything to one kind of annotation isn't
enough, as there are use cases for a mix of different type systems, Python
itself plus at least C/C++ in CPython and Cython, Java in Jython, C# in
IronPython, plus others that people might want to interface with. C/C++ are
generally interesting, for example, also for .NET or JVM users.

Although I wasn't very impressed by Bob Ippolito's talk at EuroPython, I'm
generally not opposed to type annotations to provide 1) additional contract
information for libraries, 2) documentation, 3) better static analysis or
4) type hints for compilers. But these are actually quite different use
cases that may each suggest a different strictness in the type
declarations. For 3) and 4), function signatures aren't enough and should
be accompanied by declarations for local variables. 4) should also support
additional type systems for language integration. But 1) and 2) aren't
completely overlapping either. 1) would need declarations that can be used
for hard type checking, whereas 2) can be much more relaxed, generic and
incomplete. Trying to get all of these under one umbrella might not be a
good idea, but letting people add three different annotations for each
function argument definitely isn't either.

Stefan



From steve at pearwood.info  Thu Aug 14 19:45:12 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Aug 2014 03:45:12 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAN1YFWtvpmRauVr6qH+Xv1HuDBuJdZppRY6FZ+JFBViNvH-7_A@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAN1YFWtvpmRauVr6qH+Xv1HuDBuJdZppRY6FZ+JFBViNvH-7_A@mail.gmail.com>
Message-ID: <20140814174512.GQ4525@ando>

On Wed, Aug 13, 2014 at 05:51:40PM -0430, Juancarlo A?ez wrote:

> Function annotations are not available in Python 2.7, so promoting
> widespread use of annotations in 3.5 would be promoting code that is
> compatible only with 3.x, 

Yes. You say that as if it were a bad thing. It is not. Python 3 is 
here to stay and we should be promoting Python 3 only code. There is 
absolutely no need to apologise for that fact. If people are happy with 
Python the way it is in 2.7, or 1.5 for that matter, that's great, they 
can stay on it for ever, but all new features are aimed at 3.x and not 
2.x or 1.x.


> when the current situation is that much effort is
> being spent on writing code that works on both 2.7 and 3.4 (most
> libraries?).

There's no reason why all new code should be aimed at 2.x and 3.x. But 
even for code which is, the nice thing about this proposal is that it's 
optional, so you can run your type-check using mypy under Python 3.x and 
still get the benefit of it when running under 2.x.


> Independently of its core merits, this proposal should fail unless
> annotations are added to Python 2.8.

There will be no Python 2.8, and no Python 2.9 either. New features go 
into 3.x.



-- 
Steven

From greg at krypto.org  Thu Aug 14 19:46:43 2014
From: greg at krypto.org (Gregory P. Smith)
Date: Thu, 14 Aug 2014 10:46:43 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP1=2W6veDE_VEb_YU3-DTYpZAF1Lfe+8pXkqg91x=hyjxWN_g@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsigjt$k35$1@ger.gmane.org>
 <CA+=+wqCJ7eA0f2K1kgytb7wFz44eYzxfKmN4Pg7Vc1d+UDk6_A@mail.gmail.com>
 <CAP1=2W6veDE_VEb_YU3-DTYpZAF1Lfe+8pXkqg91x=hyjxWN_g@mail.gmail.com>
Message-ID: <CAGE7PNJs6iLJDFWegOov6KZ=vXc7TeHSVM2zd4HFf6mtS96m5A@mail.gmail.com>

On Thu, Aug 14, 2014 at 9:01 AM, Brett Cannon <brett at python.org> wrote:

>
> On Thu Aug 14 2014 at 10:53:37 AM Petr Viktorin <encukou at gmail.com> wrote:
>
>> It seems to me that rather than adding a module which is only used by
>> one project so far to the standard library is a bit premature.
>>
>> I support optional typing, but why push this to stdlib now? Wouldn't
>> it be better to wait until most IDEs/linters all agree on this syntax,
>> until freezing it in stdlib? So far typing seems to be a part of mypy,
>> shouldn't it spend some time on PyPI first?
>>
>
> Because as you have noticed in this thread there are already a ton of
> competing solutions and no consensus has been reached. Sometimes Guido
> and/or python-dev have to step in and simply say "there is obvious need and
> the community is not reaching consensus, so we will make the decision
> ourselves".
>

My overarching concern with the entire proposal is that adding this would
just be yet more syntax added to the language with not much use that
doesn't go far enough.

We'd ultimately need pytd or something else regardless when it comes to
full scale Python static analysis.

But that *isn't necessarily* a bad thing. Specifying an actual basic
annotation syntax that can do some subset of what you want to annotate in
the language should in theory still be useful to a real code analysis tool.
If it isn't, it will simply ignore it. If it is, it can use it and build on
it even though it needs the ability to specify on the side.

If you do add a module for this, at least consider hiding it behind a "from
__future__ import some_module_full_of_annotation_related_things" instead of
making it a new no-op top level module.

-gps


>
>
>>
>> I'm also sure about there not being other uses of annotations -- clize
>> aside, there are not many widely used Python3-only 3rd party
>> libraries, so it's no surprise that nothing big is built around Python
>> 3 features.
>>
>> Maybe the way from PEP 3107's "here's a feature, use it for whatever
>> you like" to "annotations are for typing declarations, using
>> mypy/typing syntax" should include a step of "if you use annotations
>> for typing, use mypy/typing syntax for it". (And perhaps it should end
>> there.)
>>
>
> That's a possibility. Another thing to support this approach is that if
> something like List[str] is used over `[str]`  then the returned object can
> subclass some common superclass which can be typechecked for to know that
> the annotation is from typing.py and not clize/scription and continue to
> function. That way you can avoid any decorators adding some attribute on
> functions to know that types have been specified while allowing function
> annotations to be used for anything. Otherwise a @typing.ignore decorator
> could also exist for alternative formats to use (since typing has been the
> most common use case and decorating your single main() function with
> @typing.ignore is not exactly heavy-handed).
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/6b8d471e/attachment.html>

From stefan_ml at behnel.de  Thu Aug 14 19:53:44 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 14 Aug 2014 19:53:44 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lsirtr$gri$1@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsirtr$gri$1@ger.gmane.org>
Message-ID: <lsit38$vf7$1@ger.gmane.org>

Neal Becker schrieb am 14.08.2014 um 19:33:
> Does mypy support annotation of functions implemented in C code?
> 
> If I extend cpython via C-API, does mypy provide a mechanism to annotate those 
> functions?

No, mypy isn't about C or even CPython.

However, you can already do that, although not easily, I think. What you'd
need is an __annotations__ dict on the function object and a bit of
trickery to make CPython believe it's a function. Cython gives you that for
free (by simply providing the normal Python semantics), but you can get the
same thing with some additional work when writing your own C code. There's
also the argument clinic, but IIRC it doesn't support signature annotations
for some reason, guess it wasn't considered relevant (yet).

Stefan



From rymg19 at gmail.com  Thu Aug 14 20:05:58 2014
From: rymg19 at gmail.com (Ryan)
Date: Thu, 14 Aug 2014 13:05:58 -0500
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <E200320E-014F-46DC-A2C8-E7F221CD6976@yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsh6r6$rkn$1@ger.gmane.org>
 <CAO41-mPOE7TnJsmoNH38XKW7KjSCwpi0RkwuHwwwDR=zSwcRAQ@mail.gmail.com>
 <E200320E-014F-46DC-A2C8-E7F221CD6976@yahoo.com>
Message-ID: <87d5d44f-f48e-4df0-a4c7-2a8d7c6f4996@email.android.com>



Andrew Barnert <abarnert at yahoo.com> wrote:
>On Aug 14, 2014, at 7:37, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>
>>> On 8/13/2014 3:44 PM, Guido van Rossum wrote:
>
>>> Now consider an extended version (after Lucas).
>>> 
>>> def fib(n, a, b):
>>>     i = 0
>>>     while i <= n:
>>>         print(i,a)
>>>         i += 1
>>>         a, b = b, a+b
>>> 
>>> The only requirement of a, b is that they be addable. Any numbers
>should be allowed, as in fib(10, 1, 1+1j), but so should fib(5, '0',
>'1'). Addable would be approximated from below by Union(Number, str).
>> 
>> Unless MyPy added some sort of type classes...
>
>By "type classes", do you mean this in the Haskell sense, or do you
>mean classes used just for typing--whether more granular ABCs (like an
>Addable which both Number and AnyStr and probably inherit) or typing.py
>type specifiers (like an Addable defined as Union(Number, AnyStr)?

The Haskell way. Having ABCs and type classes can get confusing, but, when I can, I use type classes for the more unrelated concepts(such as Addable) and ABCs for the more parent-child concepts(such as Node in an AST or Generator in a set of generators).

The fine line might actually make it a bad choice to add to Python/mypy, though.

>
>It's also worth noting that the idea that this function should take a
>Number or a str seems way off. It's questionable whether it should
>accept str, but if it does, shouldn't it also accept bytes, bytearray,
>and other string-like types? What about sequences? And meanwhile,
>whether or not it accepts str, it should probably accept np.ndarray and
>other types of element-wise adding types. If you create an Addable
>type, it has to define, globally, which of those counts as addable, but
>different functions will have different definitions that make sense.
>
>In fact, look at the other discussion going on. People want to ensure
>that sum only works on numbers or number-like types (and does that
>include NumPy arrays or not?), while others want to change it to work
>on all sequences, or only mutable sequences with += plus str because it
>effectively has magical += handling under the covers, etc. If we can't
>even decide what Addable means for one specific function that everyone
>has experience with...
>
>On the other hand, if sum could have been annotated to tell us the
>author's intention (or, rather, the consensus of the dev list), then
>all of these recurring arguments about summing str would go away. Until
>someone defined a number-like class (maybe even one that meets the ABC,
>but by calling register on it) and sum won't work for him.

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

From steve at pearwood.info  Thu Aug 14 20:15:54 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Aug 2014 04:15:54 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
Message-ID: <20140814181554.GR4525@ando>

On Thu, Aug 14, 2014 at 12:01:37PM -0400, Sunjay Varma wrote:

> Though the annotation syntax is already present in Python 3, I would argue
> that using this for type annotations will get very messy very quickly. If
> I'm understanding the syntax correctly, writing any function using a large
> library with many nested subpackages could result in code like this:
> 
>     import twisted.protocols.mice.mouseman
> 
>     def
> process_mouseman(inputMouseMan: twisted.protocols.mice.mouseman.MouseMan)
> -> twisted.protocols.mice.mouseman.MouseMan:
>         pass

I would write that like this:

    from twisted.protocols.mice.mouseman import MouseMan
 
    def process_mouseman(inputMouseMan: MouseMan) -> MouseMan:
        pass


> That function definition is 122 characters long.

Or 58.


> It is also easy to see that it is very difficult to parse out what is going
> on in that function.

Only because I have no idea what MouseMan means :-)


> As an alternative, I would like to propose a syntax that Pycharm already
> supports:
> http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.html
[...]
> Here's a taste of what that looks like:
>     class SimpleEquation(object):
> 
>         def demo(self, a, b, c):
>             """
>             This function returns the product of a, b and c
>             @type self: SimpleEquation
>             :param a: int - The first number
>             :param b: int
>             :param c: int - The third number should not be zero and should
> also
>                 only be -1 if you enjoy carrots (this comment spans 2 lines)
>             :return: int
>             """
>             return a * b * c

I really dislike that syntax. I dislike adding cruft like "@type" and 
":param" into docstrings, which should be written for human readers, not 
linters. I dislike that you have documented that self is a 
SimpleEquation. (What else could it be?) I dislike that the syntax 
clashes with ReST syntax. I dislike that it isn't obvious to me why the 
first parameter uses @type while the second parameter uses :param.


> Overall, I think overloading function declarations and inline comments is a
> bad idea. It promotes writing code with poor readability

I like the annotation syntax. I'm not completely convinced that the mypy 
syntax is mature enough to bless, but the basic idea of type annotations 
is pretty common in dozens of languages. I think you are in a tiny 
minority if you think that putting the type declaration right next to 
the parameter make it *less* clear that putting the type declaration in 
a completely different part of the code.

# the type is together with the parameter
def frobinate(x: Spam, y: Egg)->Breakfast:

# the type declaration and parameter are distantly apart
def frobinate(x, y): 
    """Return the frobinated x and y.

    Some more text goes here. Perhaps lots of text.

    :param x: Spam
    :param y: Eggs
    :return: Breakfast
    """


> On the original proposal:
> These changes really do seem to be overestimating the staticness of Python
> programs as well. What about functions that don't care about the type?

They can declare that they are object. Or not declare a type at all.

> What about functions that only want you to pass in an object that implements
> __iter__?


I would expect this should work:

from typing import Iter
def func(it:Iter):
    ...


> Python should not become a language where developers are required
> to add hundreds of odd cast() calls every time they choose to pass a
> different, but still compatible type, to a function.

I'm not sure how you go from *optional* static typing to developers 
being *required* to cast values.

As I see it, one HUGE advantage of this proposal is that people who want 
strict static typing currently might write code like this:

def make_sandwich(filling):
    if not isinstance(filling, Ham):
        raise TypeError
    ...


With the new proposal, they will probably write this:

def make_sandwich(filling: Ham):
    ...

and allow the static type check to occur at compile time. That means 
that if I want to pass a Spam instance instead of a Ham instance, all I 
need do is disable the compile-time type check, and make_sandwich will 
happily accept anything that has the same duck-type interface as Ham, 
like Spam. If I pass an int instead, I'll get the same run-time error 
that I would have got if make_sandwich did not include an explicit type 
check.

So, I think this proposal might actually lead to *more* duck typing 
rather than less, since you can always turn off the type checking.


-- 
Steven

From steve at pearwood.info  Thu Aug 14 20:30:14 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Aug 2014 04:30:14 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAGE7PNJs6iLJDFWegOov6KZ=vXc7TeHSVM2zd4HFf6mtS96m5A@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsigjt$k35$1@ger.gmane.org>
 <CA+=+wqCJ7eA0f2K1kgytb7wFz44eYzxfKmN4Pg7Vc1d+UDk6_A@mail.gmail.com>
 <CAP1=2W6veDE_VEb_YU3-DTYpZAF1Lfe+8pXkqg91x=hyjxWN_g@mail.gmail.com>
 <CAGE7PNJs6iLJDFWegOov6KZ=vXc7TeHSVM2zd4HFf6mtS96m5A@mail.gmail.com>
Message-ID: <20140814183014.GS4525@ando>

On Thu, Aug 14, 2014 at 10:46:43AM -0700, Gregory P. Smith wrote:

> My overarching concern with the entire proposal is that adding this would
> just be yet more syntax added to the language with not much use that
> doesn't go far enough.

This isn't new syntax. Functional annotations have been in Python since 
Python 3.0. What's new here is blessing one specific meaning for that 
syntax as the One Offical use for annotations.

I'd rather:

- bless function annotations for static type checking as the default
  meaning, but allowing code to opt-out and use annotations for
  something else;

- encourage the various type checkers and linters to come up with a
  standard syntax for type annotations.

I'm not convinced that mypy syntax is yet mature enough to be that 
standard. But, perhaps if the typing module is given provisional status, 
maybe it could be a good start.


> If you do add a module for this, at least consider hiding it behind a "from
> __future__ import some_module_full_of_annotation_related_things" instead of
> making it a new no-op top level module.

I'm not sure what this comment means. Did you read Guido's first post in 
this thread? I thought he was clear that to get type checking, you would 
do this:

from typing import Dict

def function(d: Dict[str, int])-> int:
    ...

I bet you can guess what that does, but in case you can't, it declares 
that argument d is a Dict with str keys and int values, and the return 
result is an int. I'm not sure where you get the idea of a no-op top 
level module from.

"from __future__ import ..." is inappropriate too, since that is 
intended for changes to compiler semantics (e.g. new syntax). This is 
existing syntax, and the actual type checking itself will be delegated 
to a separate product, mypy, which as Guido stated will behave as a 
linter. That means that the type checks won't do anything unless you 
have mypy installed, and you can still run your code under any Python 
compiler you like. As I understand it, CPython itself requires no 
changes to make this work, just the addition of typing.py from the mypy 
project and a policy change to the use of annotations.



-- 
Steven

From cory at lukasa.co.uk  Thu Aug 14 20:25:04 2014
From: cory at lukasa.co.uk (Cory Benfield)
Date: Thu, 14 Aug 2014 19:25:04 +0100
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140814181554.GR4525@ando>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
Message-ID: <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>

On 14 August 2014 19:15, Steven D'Aprano <steve at pearwood.info> wrote:
> I really dislike that syntax. I dislike adding cruft like "@type" and
> ":param" into docstrings, which should be written for human readers, not
> linters.

That ship has long-since sailed. Sphinx uses exactly this :param: and
:return: syntax for its docstring parsing. It is by now a common
convention (at least, I see it all over the place in open source
code), and should not be considered a surprise. I appreciate that it
doesn't lead to clean docstrings, but I've found it leads to
docstrings that are genuinely written to be read (because they're part
of your documentation).

> So, I think this proposal might actually lead to *more* duck typing
> rather than less, since you can always turn off the type checking.

I found this conclusion impossible to understand: have I missed
something Steven? To my eyes, the fact that when run by a user who
knows nothing about the static type checker much duck typing will fail
will clearly not lead to more duck typing. It will lead either to a)
less duck typing because of all the bug reports (your code breaks
whenever I try to run it!), or b) everyone turning the static type
checker off.

That objection assumes the static checker would be on by default. If
it were off by default but available, both of these problems go away
but we're back in the situation we're in right now. In that case, I
don't see why we'd add this to CPython.

From ethan at stoneleaf.us  Thu Aug 14 20:33:45 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 14 Aug 2014 11:33:45 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140814181554.GR4525@ando>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
Message-ID: <53ED0109.6070207@stoneleaf.us>

On 08/14/2014 11:15 AM, Steven D'Aprano wrote:
>
> I like the annotation syntax. I'm not completely convinced that the mypy
> syntax is mature enough to bless, but the basic idea of type annotations
> is pretty common in dozens of languages. I think you are in a tiny
> minority if you think that putting the type declaration right next to
> the parameter make it *less* clear that putting the type declaration in
> a completely different part of the code.
>
> # the type is together with the parameter
> def frobinate(x: Spam, y: Egg)->Breakfast:
>
> # the type declaration and parameter are distantly apart
> def frobinate(x, y):
>      """Return the frobinated x and y.
>
>      Some more text goes here. Perhaps lots of text.
>
>      :param x: Spam
>      :param y: Eggs
>      :return: Breakfast
>      """

Sure, keeping that info in the annotations makes more sense, but I'd rather see it in the doc string instead of ruling 
out all other possible uses of annotations -- particularly for something that's supposed to be /optional/.

--
~Ethan~

From steve at pearwood.info  Thu Aug 14 20:52:45 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Aug 2014 04:52:45 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
 <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>
Message-ID: <20140814185245.GT4525@ando>

On Thu, Aug 14, 2014 at 07:25:04PM +0100, Cory Benfield wrote:
> On 14 August 2014 19:15, Steven D'Aprano <steve at pearwood.info> wrote:
> > I really dislike that syntax. I dislike adding cruft like "@type" and
> > ":param" into docstrings, which should be written for human readers, not
> > linters.
> 
> That ship has long-since sailed. Sphinx uses exactly this :param: and
> :return: syntax for its docstring parsing. It is by now a common
> convention (at least, I see it all over the place in open source
> code), and should not be considered a surprise.

I've seen it too, but not in docstrings written in vanilla ReST. That's 
a disappointment to hear that Sphinx uses it, because I think it is 
hideously ugly :-(


> > So, I think this proposal might actually lead to *more* duck typing
> > rather than less, since you can always turn off the type checking.
> 
> I found this conclusion impossible to understand: have I missed
> something Steven? To my eyes, the fact that when run by a user who
> knows nothing about the static type checker much duck typing will fail
> will clearly not lead to more duck typing. It will lead either to a)
> less duck typing because of all the bug reports (your code breaks
> whenever I try to run it!), or b) everyone turning the static type
> checker off.

Let me explain my reasoning.

Back in the Old Days, before Python 2.2, there was no isinstance(). We 
were strongly discouraged from doing type checks, instead we were 
encouraged to rely on duck-typing and that functions would fail loudly 
if passed the wrong argument. With the introduction of isinstance, 
Python code has slowly, gradually, begun using more and more run-time 
explicit type checks with isinstance. Some people do this more than 
others. Let's consider Fred, who is a Java programmer at heart and so 
writes code like this:

def foo(x):
    if not instance(x, float): 
        raise TypeError("Why doesn't python check this for me?")
    return (x+1)/2

I want to pass a Decimal to foo(), but can't, because of the explicit 
type check. I am sad.

But with this proposal, Fred may write his function like this:

def foo(x:float)->float:
    return (x+1)/2

and rely on mypy to check the types at compile time. Fred is happy: he 
has static type checks, Python does it automatically for him (once he 
has set up his build system to call mypy), and he is now convinced that 
foo() is type-safe and an isinstance check at run-time would be a waste 
of cycles.

I want to pass a Decimal to foo(). All I have to do is *not* install 
mypy, or disable it, and lo and behold, like magic, the type checking 
doesn't happen, and foo() operates by duck-typing just like in the glory 
days of Python 1.5. Both Fred and I are now happy, and with the explicit 
isinstance check removed, the only type checking that occurs when I run 
Fred's library are the run-time duck-typing checks.

-- 
Steven

From steve at pearwood.info  Thu Aug 14 20:55:55 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Aug 2014 04:55:55 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CA+VECoXkb+=UeSQbckY+prOgKx7AY4yUwFRV9ogPTTQ+mR=DOA@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CA+VECoVO0COba4u3o5YD0shE0r-P8-q99sWMQtpRkisQ8wa8UQ@mail.gmail.com>
 <CAP7+vJLKNydsdLuc8jRjhfYr_R8P7u1Jg1aZ4Yb1LA1c-t3w1g@mail.gmail.com>
 <CA+VECoXkb+=UeSQbckY+prOgKx7AY4yUwFRV9ogPTTQ+mR=DOA@mail.gmail.com>
Message-ID: <20140814185555.GU4525@ando>

On Thu, Aug 14, 2014 at 12:28:26PM +0200, Manuel Cer?n wrote:

> One interesting feature of TypeScript is that it allows you to annotate
> existing code without modifying it, by using external definition files. In
> the JavaScript world, many people have contributed TypeScript annotation
> files for popular JS libraries (http://definitelytyped.org/).
> 
> I think this is possible in Python as well doing something like this:
> 
> @annotate('math.ciel')
> def ciel(x: float) -> int:
>     pass

I'm afraid I don't understand what the annotate decorator is doing here. 
Can you explain please?




-- 
Steven

From steve at pearwood.info  Thu Aug 14 21:02:38 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Aug 2014 05:02:38 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53EBCABC.7010801@python.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org>
Message-ID: <20140814190237.GV4525@ando>

On Wed, Aug 13, 2014 at 10:29:48PM +0200, Christian Heimes wrote:

> 1) I'm not keen with the naming of mypy's typing classes. The visual
> distinction between e.g. dict() and Dict() is too small and IMHO
> confusing for newcomers. How about an additional 'T' prefix to make
> clear that the objects are referring to typing objects?
> 
>   from typing import TList, TDict
> 
>   def word_count(input: TList[str]) -> TDict[str, int]:
>       ...

Would it be possible, and desirable, to modify the built-in types so 
that we could re-use them in the type annotations?

    def word_count(input: list[str]) -> dict[str, int]:


Since types are otherwise unlikely to be indexable like that, I think 
that might work.


 
> 2) PEP 3107 only specifies arguments and return values but not
> exceptions that can be raised by a function. Java has the "throws"
> syntax to list possible exceptions:
> 
>  public void readFile() throws IOException {}

I understand that this is called a "checked exception" in Java. I also 
understand that they are hated and derided as useless or even 
counter-productive:

http://literatejava.com/exceptions/checked-exceptions-javas-biggest-mistake/

 
> May I suggest that we also standardize a way to annotate the exceptions
> that can be raised by a function? It's a very useful piece of
> information and commonly requested information on the Python user
> mailing list. 

And as frequently explained on the python-list, it's almost impossible 
to answer. Or rather, the answer is usually no more specific than 
"raises Exception".

There are very few guarantees you can reliably make about what 
exceptions *cannot* be raised by a function. To put it simply, given 
almost any operation in your function, say, x+1, there's no limit on 
what x.__add__ might raise. Even if you know x is a subclass of int, it 
could do anything in its __add__ method. Only if you know x is *exactly* 
a builtin int can you be confident that it won't raise (say) 
ImportError.

Perhaps with a few years of experience, we might be able to extend this 
to exceptions without making the same mistakes as Java's checked 
exceptions, but I wouldn't rush into it.


-- 
Steven

From varma.sunjay at gmail.com  Thu Aug 14 21:21:59 2014
From: varma.sunjay at gmail.com (Sunjay Varma)
Date: Thu, 14 Aug 2014 15:21:59 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAPJVwBmHuG6beqBk80cFeyFGnbBNQJfa8wOfNe9w4Xypb-SHEQ@mail.gmail.com>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <CAPJVwBmHuG6beqBk80cFeyFGnbBNQJfa8wOfNe9w4Xypb-SHEQ@mail.gmail.com>
Message-ID: <CAJaQC303RUtSCVmvw2gqBq0sQ_V6QEPLZTz3XS0XSV3NDonLcg@mail.gmail.com>

Frankly, I'd just really like to get all of this noise out of the function
declaration. Any reasonable, readable and consistent documentation format
is fine with me. I chose the sphinx format because it is already well
supported in pycharm and that was mentioned in the first few responses.

I actually don't like the colon syntax very much (it's awkward and
unnatural to type), so if anyone has a different suggestion I'd be very
open to that.

Mainly I want to ensure that Python doesn't sacrifice readability and line
length (which is part of readability) just because annotations are already
built in.

I suggest we decide on a standard format that can be used in documentation
strings and also used with type checking.

Let's enhance our documentation with types, not obfuscate function
declarations.

Sunjay
On Aug 14, 2014 3:14 PM, "Nathaniel Smith" <njs at pobox.com> wrote:

> On 14 Aug 2014 17:02, "Sunjay Varma" <varma.sunjay at gmail.com> wrote:
> > Here's a taste of what that looks like:
> >     class SimpleEquation(object):
> >
> >         def demo(self, a, b, c):
> >             """
> >             This function returns the product of a, b and c
> >             @type self: SimpleEquation
> >             :param a: int - The first number
> >             :param b: int
> >             :param c: int - The third number should not be zero and
> should also
> >                 only be -1 if you enjoy carrots (this comment spans 2
> lines)
> >             :return: int
> >             """
> >             return a * b * c
>
> There are at least three existing, popular, standardized syntaxes for
> these kinds of docstring annotations in use: plain ReST, Google's docstring
> standard, and numpy's docstring standard. All are supported by Sphinx out
> of the box. (The latter two require enabling the "napolean" extension, but
> this is literally a one line config file switch.)
>
> Would you suggest that python-dev should pick one of these and declare it
> to be the official standard, or...?
>
> -n
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/d7b8e013/attachment.html>

From njs at pobox.com  Thu Aug 14 21:14:27 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 14 Aug 2014 20:14:27 +0100
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
Message-ID: <CAPJVwBmHuG6beqBk80cFeyFGnbBNQJfa8wOfNe9w4Xypb-SHEQ@mail.gmail.com>

On 14 Aug 2014 17:02, "Sunjay Varma" <varma.sunjay at gmail.com> wrote:
> Here's a taste of what that looks like:
>     class SimpleEquation(object):
>
>         def demo(self, a, b, c):
>             """
>             This function returns the product of a, b and c
>             @type self: SimpleEquation
>             :param a: int - The first number
>             :param b: int
>             :param c: int - The third number should not be zero and
should also
>                 only be -1 if you enjoy carrots (this comment spans 2
lines)
>             :return: int
>             """
>             return a * b * c

There are at least three existing, popular, standardized syntaxes for these
kinds of docstring annotations in use: plain ReST, Google's docstring
standard, and numpy's docstring standard. All are supported by Sphinx out
of the box. (The latter two require enabling the "napolean" extension, but
this is literally a one line config file switch.)

Would you suggest that python-dev should pick one of these and declare it
to be the official standard, or...?

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/190b006b/attachment-0001.html>

From brakhane at googlemail.com  Thu Aug 14 21:28:11 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Thu, 14 Aug 2014 21:28:11 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBC3BA.10808@stoneleaf.us>
 <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
Message-ID: <53ED0DCB.2000306@googlemail.com>

Allow me to chime in.

Am 13.08.2014 22:19, schrieb Guido van Rossum:
> On Wed, Aug 13, 2014 at 12:59 PM, Ethan Furman <ethan at stoneleaf.us
> <mailto:ethan at stoneleaf.us>> wrote:
>
> -1 on deprecating alternative uses of annotations.
>
>
> Do you have a favorite alternative annotation use that you actually
> use (or are likely to)?
>
I would be very sad to see annotations being limited to convey type
information. But I think I have a solution that
will make all happy (see end of mail)


I've programmed Java web application for many years now (hoping to
finally switch to Python), and "method parameter
annotations" as they are called in Java would be one thing I'd really
miss, as they can be very useful.

Let me give you two examples:

1. Annotations can be used to communicate additional restrictions on
values that must be checked on run time

Let's assume a simple Web service that is called via HTTP to register a
user, and the underlying framework decodes
the request and finally calls a simple controller function, it could
look like this

(Java code, @ signifies an annotation)

  public Response register(@Range(18,100) int age, @ValidEmail String
email) { ... }

The framework would check the range of the age parameter and the
validity of the email and if there are validation errors,
refusing the request with a suitable error message without ever calling
our function.

Even if we assume that mypy's type system will be incredibly complex and
allowing to specify all kinds of restrictions on a type,
it won't help because those checks have to be done at run time, and are
not optional.

Of course those checks could be hard coded into the function, but using
annotation also provides a simple and immediate
documentation about the allowed values, and avoids boilerplate (I do not
have to write
"if (emailNotvalie(email) throw ValidationError("Field email is not a
valid email")"  in every method that uses an email)


2. They can give meta information to a framework

An admittedly contrieved example, let's expand our register function:

public Response register( int age, @Inject @Scope("session") UserService
userService, @Authenticated User user) ....

Here I can tell the dependency injection framework that I want an
instance of the UserService injected, but one instance
that has session scope instead of the normal "singleton" scope.
I also ask the framework to inject me the currently authenticated user
object (let's assume if I'd write
"@Authenticated String user" I could get the login name as string etc.)


The flexibility annotations give in Java makes programming much less
frustrating. It also took quite some time before
annotations were widly used (they were only available starting in Java
5) and people started finding more uses for them.
I think this will also be true for Python, it will take time before
people find useful ways for them. Redefining them now to
be "not much more" than static type information feels wrong to me.


My proposed solution:

If an annotation is a tuple, mypy will take a look at each item and do
it's usual thing. If it doesn't recognise an item, it is skipped.

Every framework that uses annotations should also iterate over entries
of a tuple to find the ones it is interested in.

This also allows more than one annotation at a time, and is completely
backwards compatible (as far as Python itself is concerned)


for example, my first example could be written as

  def register(age: (int, range(18,100)), email: (str, ValidEmail))

also, it will allow me to add annotations to existing "typed" functions,

  def foo(bar: int) -> int

could become

  def foo(bar: (int, MyAnnotation)) -> (int, AnotherOfMyAnnotations)

I'm not sure what should happen if two conflicting types are given, like
(int, str); I think it should be treated as a union
type (either int or str).

-- 
Dennis


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/0d7d284f/attachment.sig>

From skip at pobox.com  Thu Aug 14 21:34:51 2014
From: skip at pobox.com (Skip Montanaro)
Date: Thu, 14 Aug 2014 14:34:51 -0500
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53ED0DCB.2000306@googlemail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBC3BA.10808@stoneleaf.us>
 <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
 <53ED0DCB.2000306@googlemail.com>
Message-ID: <CANc-5UwTy0b6Jo_ZYBMy1AUVMxTFj3dV_H05xK700XvgmyGG9A@mail.gmail.com>

On Thu, Aug 14, 2014 at 2:28 PM, Dennis Brakhane
<brakhane at googlemail.com> wrote:
> public Response register(@Range(18,100) int age, @ValidEmail String
> email) { ... }
>
> The framework would check the range of the age parameter and the
> validity of the email and if there are validation errors,
> refusing the request with a suitable error message without ever calling
> our function.

Couldn't you do that today in Python with a suitably sophisticated
function decorator? The range/type checking would happen before the
user's actual function is called.

Skip

From lukasz at langa.pl  Thu Aug 14 21:35:38 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Thu, 14 Aug 2014 12:35:38 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140814190237.GV4525@ando>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
Message-ID: <E41C1565-370A-410B-92A2-8E4D6BECFAF8@langa.pl>

On Aug 14, 2014, at 12:02 PM, Steven D'Aprano <steve at pearwood.info> wrote:

> Would it be possible, and desirable, to modify the built-in types so 
> that we could re-use them in the type annotations?
> 
>    def word_count(input: list[str]) -> dict[str, int]:
> 
> 
> Since types are otherwise unlikely to be indexable like that, I think 
> that might work.

-1 on that idea. Actually, -1 on List, Dict and friends as well.

Square brackets are for lookup (indexing, key-based, or slicing). Saying here that you?re ?looking up? a subtype of list that holds strings is a far stretch.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/9750c534/attachment.html>

From phd at phdru.name  Thu Aug 14 21:23:34 2014
From: phd at phdru.name (Oleg Broytman)
Date: Thu, 14 Aug 2014 21:23:34 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140814185245.GT4525@ando>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
 <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>
 <20140814185245.GT4525@ando>
Message-ID: <20140814192334.GA27244@phdru.name>

On Fri, Aug 15, 2014 at 04:52:45AM +1000, Steven D'Aprano <steve at pearwood.info> wrote:
> But with this proposal, Fred may write his function like this:
> 
> def foo(x:float)->float:
>     return (x+1)/2
> 
> and rely on mypy to check the types at compile time. Fred is happy: he 
> has static type checks, Python does it automatically for him (once he 
> has set up his build system to call mypy), and he is now convinced that 
> foo() is type-safe and an isinstance check at run-time would be a waste 
> of cycles.
> 
> I want to pass a Decimal to foo(). All I have to do is *not* install 
> mypy, or disable it, and lo and behold, like magic, the type checking 
> doesn't happen, and foo() operates by duck-typing just like in the glory 
> days of Python 1.5. Both Fred and I are now happy, and with the explicit 
> isinstance check removed, the only type checking that occurs when I run 
> Fred's library are the run-time duck-typing checks.

   Well, that's funny. Static type checking as a way to subvert type
checking! (-:

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From ethan at stoneleaf.us  Thu Aug 14 21:43:22 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 14 Aug 2014 12:43:22 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140814185555.GU4525@ando>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CA+VECoVO0COba4u3o5YD0shE0r-P8-q99sWMQtpRkisQ8wa8UQ@mail.gmail.com>
 <CAP7+vJLKNydsdLuc8jRjhfYr_R8P7u1Jg1aZ4Yb1LA1c-t3w1g@mail.gmail.com>
 <CA+VECoXkb+=UeSQbckY+prOgKx7AY4yUwFRV9ogPTTQ+mR=DOA@mail.gmail.com>
 <20140814185555.GU4525@ando>
Message-ID: <53ED115A.1090306@stoneleaf.us>

On 08/14/2014 11:55 AM, Steven D'Aprano wrote:
> On Thu, Aug 14, 2014 at 12:28:26PM +0200, Manuel Cer?n wrote:
>
>> One interesting feature of TypeScript is that it allows you to annotate
>> existing code without modifying it, by using external definition files. In
>> the JavaScript world, many people have contributed TypeScript annotation
>> files for popular JS libraries (http://definitelytyped.org/).
>>
>> I think this is possible in Python as well doing something like this:
>>
>> @annotate('math.ciel')
>> def ciel(x: float) -> int:
>>      pass
>
> I'm afraid I don't understand what the annotate decorator is doing here.
> Can you explain please?

My understanding is it's using the 'math.ciel' file in order to understand how it should treat the annotations of 
'float' and 'int'.

--
~Ethan~

From brakhane at googlemail.com  Thu Aug 14 21:43:39 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Thu, 14 Aug 2014 21:43:39 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CANc-5UwTy0b6Jo_ZYBMy1AUVMxTFj3dV_H05xK700XvgmyGG9A@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>	<53EBC3BA.10808@stoneleaf.us>	<CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>	<53ED0DCB.2000306@googlemail.com>
 <CANc-5UwTy0b6Jo_ZYBMy1AUVMxTFj3dV_H05xK700XvgmyGG9A@mail.gmail.com>
Message-ID: <53ED116B.2010300@googlemail.com>

Am 14.08.2014 21:34, schrieb Skip Montanaro:

> Couldn't you do that today in Python with a suitably sophisticated
> function decorator? The range/type checking would happen before the
> user's actual function is called.

I suppose so, but I'd have to repeat myself and it would look ugly,
because I
have to tell the decorator which parameter I'm talking about.

Something like

  @do_range_check('age', 18, 100)
  @do_email_check('email')
  def register(age: int, email: str):

looks not nearly as nice.

Furthermore, my proposal allows multiple uses of annotations, without
restricting them
to a single use.

If you only use mypy, you can keep using annotations as type
declarations, when you use
some other framework that uses annotations for a different thing, you
can still use them,
only once you want to use both *and* you have a method that needs both
types of annotations
are you forced to use the tuple notation.







From varma.sunjay at gmail.com  Thu Aug 14 21:40:26 2014
From: varma.sunjay at gmail.com (Sunjay Varma)
Date: Thu, 14 Aug 2014 15:40:26 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53ED0DCB.2000306@googlemail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBC3BA.10808@stoneleaf.us>
 <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
 <53ED0DCB.2000306@googlemail.com>
Message-ID: <CAJaQC31NtmOvGW9r6=6kB+A=vVqqyirZa042ceYsoQkqLUCbmw@mail.gmail.com>

See responses inline

On Aug 14, 2014 3:28 PM, "Dennis Brakhane" <brakhane at googlemail.com> wrote:

> 1. Annotations can be used to communicate additional restrictions on
> values that must be checked on run time
>
> Let's assume a simple Web service that is called via HTTP to register a
> user, and the underlying framework decodes
> the request and finally calls a simple controller function, it could
> look like this
>
> (Java code, @ signifies an annotation)
>
>   public Response register(@Range(18,100) int age, @ValidEmail String
> email) { ... }
>
> The framework would check the range of the age parameter and the
> validity of the email and if there are validation errors,
> refusing the request with a suitable error message without ever calling
> our function.
>

This is exactly the kind of syntax I want to avoid. Python should not
attempt to just blindly become like Java or any other language. Though
Dennis was just illustrating his point (this was not a suggestion of an
alternate syntax), I feel like Python is moving further and further towards
code like this. Python programmers should not be forcing as much as
possible into each line. Explicit is better than implicit.

> Even if we assume that mypy's type system will be incredibly complex and
> allowing to specify all kinds of restrictions on a type,
> it won't help because those checks have to be done at run time, and are
> not optional.

This is a great point. Regardless of the nature of the annotation, we can't
let this become a way out for lazy programmers. Having annotations is no
excuse for poor error checking.

> 2. They can give meta information to a framework
>
> An admittedly contrieved example, let's expand our register function:
>
> public Response register( int age, @Inject @Scope("session") UserService
> userService, @Authenticated User user) ....

This is too much to put on one line in Python. We should be aiming to make
code cleaner and promote proper error checking and documentation.

> The flexibility annotations give in Java makes programming much less
> frustrating. It also took quite some time before
> annotations were widly used (they were only available starting in Java
> 5) and people started finding more uses for them.
> I think this will also be true for Python, it will take time before
> people find useful ways for them. Redefining them now to
> be "not much more" than static type information feels wrong to me.

I agree that annotations can be useful, but I don't think they should be
used for type checking at this scale.

Sunjay _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/95f043eb/attachment.html>

From ben+python at benfinney.id.au  Thu Aug 14 21:48:50 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Fri, 15 Aug 2014 05:48:50 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
Message-ID: <851tsi7syl.fsf@benfinney.id.au>

Steven D'Aprano <steve at pearwood.info> writes:

> On Wed, Aug 13, 2014 at 10:29:48PM +0200, Christian Heimes wrote:
>
> > 1) I'm not keen with the naming of mypy's typing classes. The visual
> > distinction between e.g. dict() and Dict() is too small and IMHO
> > confusing for newcomers. [?]
>
> Would it be possible, and desirable, to modify the built-in types so
> that we could re-use them in the type annotations?

That would address my concern. With that change, when the programmer who
reads the code encounters mention of a type, it means precisely the same
type object as it appears to be and not some special beast.

-- 
 \      ?Science embraces facts and debates opinion; religion embraces |
  `\    opinion and debates the facts.? ?Tom Heehler, _The Well-Spoken |
_o__)                                                       Thesaurus_ |
Ben Finney


From tjreedy at udel.edu  Thu Aug 14 21:59:39 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 14 Aug 2014 15:59:39 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <E200320E-014F-46DC-A2C8-E7F221CD6976@yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <lsh6r6$rkn$1@ger.gmane.org>
 <CAO41-mPOE7TnJsmoNH38XKW7KjSCwpi0RkwuHwwwDR=zSwcRAQ@mail.gmail.com>
 <E200320E-014F-46DC-A2C8-E7F221CD6976@yahoo.com>
Message-ID: <lsj4g5$7b6$1@ger.gmane.org>

On 8/14/2014 11:21 AM, Andrew Barnert wrote:
> On Aug 14, 2014, at 7:37, Ryan Gonzalez
> <rymg19 at gmail.com
> <mailto:rymg19 at gmail.com>> wrote:
>
>>     On 8/13/2014 3:44 PM, Guido van Rossum wrote:
>
>>     Now consider an extended version (after Lucas).
>>
>>     def fib(n, a, b):
>>         i = 0
>>         while i <= n:
>>             print(i,a)
>>             i += 1
>>             a, b = b, a+b
>>
>>     The only requirement of a, b is that they be addable. Any numbers
>>     should be allowed, as in fib(10, 1, 1+1j), but so should fib(5,
>>     '0', '1'). Addable would be approximated from below by
>>     Union(Number, str).
>>
>>
>> Unless MyPy added some sort of type classes...
>
> By "type classes", do you mean this in the Haskell sense, or do you mean
> classes used just for typing--whether more granular ABCs (like an
> Addable which both Number and AnyStr and probably inherit) or typing.py
> type specifiers (like an Addable defined as Union(Number, AnyStr)?

What I like is the idea of protocol based types, as in
Andrey Vlasovskikh's slide 26
http://blog.pirx.ru/media/files/2013/python-optional-typing/#26

class Addable(Protocol):
     @abstractmethod
     def __add__(self, other):
         pass

Even this does not capture the actual requirement that a and b be 
addable together, in that order.  Addable_together is inherently a pair 
concept.  Syntactically, that could be handled by passing a pair.

def fib(n:Comparable_to_int, pair:Addable_together) -> Type resulting 
from pair:

However, the actual Python rules for Addable_together are rather 
complex. A static type checker for this would be extremely difficult to 
write. The best dynamic type checker is to try and let Python say no by 
raising.

> It's also worth noting that the idea that this function should take a
> Number or a str seems way off.

As written, it *does* take any pair that can be repeatedly added together.

> It's questionable whether it should accept str,

I recently read Douglas Hofstadler's Fluid Concepts and Creative 
Analogies: Computer Models of the Fundamental Mechanisms of Thought. In 
the first chapter he discussed puzzles like: A series begins 0, 1, 01, 
101, ..., what is the next item.  He started with number sequences and 
moved on to digit string sequences like the above, where the digit 
strings are *not* interpreted as number. Generic functions and 
duck-typing encourages this sort of creative thinking.

 > but if it does, shouldn't it also accept bytes,
 > bytearray, and other string-like types?

Of course. A descriptive type should not invalidate anything that is 
allowed.

 > What about sequences? And meanwhile,
> whether or not it accepts str, it should probably accept np.ndarray and
> other types of element-wise adding types.

The function above already does accept such.

> If you create an Addable type,
> it has to define, globally, which of those counts as addable, but
> different functions will have different definitions that make sense.

A single arg having .__(r)add__ is trivial.  The real difficultly is 
expressing *addable to each other* and relating the result type to the 
types of the members of the pair.




-- 
Terry Jan Reedy


From brakhane at googlemail.com  Thu Aug 14 22:05:59 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Thu, 14 Aug 2014 22:05:59 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJLFP4BaRtp0Mhg3d2A56YVLj=odUbY94XtyFKtpyUoOFA@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CAP7+vJLFP4BaRtp0Mhg3d2A56YVLj=odUbY94XtyFKtpyUoOFA@mail.gmail.com>
Message-ID: <53ED16A7.5080007@googlemail.com>

Am 13.08.2014 23:46, schrieb Guido van Rossum:
>
>
> Mypy has a cast() operator that you can use to shut it up when you
> (think you) know the conversion is safe.
>
Does Mypy provide a way to "fix/monkeypatch" incorrect type declarations
in function signatures? For example, by modifying __annotations__?

My pet peeve of static languages is that programmers are often too
fixated on their particular problem that they don't think about alternate
uses for their code and make the type declarations uncessarily complex.

For example, in Java nearly every method in Java that deals with
character strings uses "String" as parameter type, while they should
have used "CharSequence". Having to read an entire input stream and
storing it in a String just to be able to use a method is not fun
(String is final in Java)

I'm worried that in Python we will have utility functions that declare
they require a List[int], when in fact they actually only require a
Sequence[int] or
Sequence[Number].

While Mypy's cast is nice in that I won't have to wrap my Integer Tuple
in list like object, having to cast it every time I use a particular
broken utility method
feels very ugly to me; and defining a wrapper function with the correct
type information feels like unnecessary run time overhead for no gain.

Don't get me wrong, I'm not entirely against some kind of type checking,
but I fear that there must exist possible workarounds for badly written
code.



From varma.sunjay at gmail.com  Thu Aug 14 22:16:20 2014
From: varma.sunjay at gmail.com (Sunjay Varma)
Date: Thu, 14 Aug 2014 16:16:20 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53ED16A7.5080007@googlemail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CAP7+vJLFP4BaRtp0Mhg3d2A56YVLj=odUbY94XtyFKtpyUoOFA@mail.gmail.com>
 <53ED16A7.5080007@googlemail.com>
Message-ID: <CAJaQC32Rb-snR2kd+21AsvjxAhgSPZDQ=5ctoPpQCqip3jS_Xg@mail.gmail.com>

+1 This is definitely something to consider.

One of the many benefits of Python is that you can use objects with
equivalent interfaces in functions that may not have expected that type
while they were being written.

One thing to note: if we try to make this syntax too verbose, it may lose
all of its purpose all together.

For example: one (bad) solution to support what I described above is to
define some sort of convoluted syntax for specifying the exact interface a
function supports. At this point, any addition to the language would do
nothing but hinder it. Anything that verbose loses is too complex to be
valuable.

We have to be careful with this. If we do accept it (or any of the many
alternatives suggested so far), then we should choose a few use cases and
focus on solving them as best as possible.

On Aug 14, 2014 4:06 PM, "Dennis Brakhane" <brakhane at googlemail.com> wrote:
>
> Am 13.08.2014 23:46, schrieb Guido van Rossum:
> >
> >
> > Mypy has a cast() operator that you can use to shut it up when you
> > (think you) know the conversion is safe.
> >
> Does Mypy provide a way to "fix/monkeypatch" incorrect type declarations
> in function signatures? For example, by modifying __annotations__?
>
> My pet peeve of static languages is that programmers are often too
> fixated on their particular problem that they don't think about alternate
> uses for their code and make the type declarations uncessarily complex.
>
> For example, in Java nearly every method in Java that deals with
> character strings uses "String" as parameter type, while they should
> have used "CharSequence". Having to read an entire input stream and
> storing it in a String just to be able to use a method is not fun
> (String is final in Java)
>
> I'm worried that in Python we will have utility functions that declare
> they require a List[int], when in fact they actually only require a
> Sequence[int] or
> Sequence[Number].
>
> While Mypy's cast is nice in that I won't have to wrap my Integer Tuple
> in list like object, having to cast it every time I use a particular
> broken utility method
> feels very ugly to me; and defining a wrapper function with the correct
> type information feels like unnecessary run time overhead for no gain.
>
> Don't get me wrong, I'm not entirely against some kind of type checking,
> but I fear that there must exist possible workarounds for badly written
> code.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/191ebbf6/attachment.html>

From ceronman at gmail.com  Thu Aug 14 22:29:31 2014
From: ceronman at gmail.com (=?UTF-8?Q?Manuel_Cer=C3=B3n?=)
Date: Thu, 14 Aug 2014 22:29:31 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140814185555.GU4525@ando>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CA+VECoVO0COba4u3o5YD0shE0r-P8-q99sWMQtpRkisQ8wa8UQ@mail.gmail.com>
 <CAP7+vJLKNydsdLuc8jRjhfYr_R8P7u1Jg1aZ4Yb1LA1c-t3w1g@mail.gmail.com>
 <CA+VECoXkb+=UeSQbckY+prOgKx7AY4yUwFRV9ogPTTQ+mR=DOA@mail.gmail.com>
 <20140814185555.GU4525@ando>
Message-ID: <CA+VECoVPye38je-dN=HLLGGso_uxfphK550wXj3sAPwbTRfZeA@mail.gmail.com>

On Thu, Aug 14, 2014 at 8:55 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Thu, Aug 14, 2014 at 12:28:26PM +0200, Manuel Cer?n wrote:
>
> > One interesting feature of TypeScript is that it allows you to annotate
> > existing code without modifying it, by using external definition files.
> In
> > the JavaScript world, many people have contributed TypeScript annotation
> > files for popular JS libraries (http://definitelytyped.org/).
> >
> > I think this is possible in Python as well doing something like this:
> >
> > @annotate('math.ciel')
> > def ciel(x: float) -> int:
> >     pass
>
> I'm afraid I don't understand what the annotate decorator is doing here.
> Can you explain please?
>

The idea is to add type annotations to modules without modifying them. For
example, in this case, the stdlib math module is defined and implemented in
C, but you still want to have annotations for it so that if you write
math.ciel('foo'), the static type analyzer gives you a error or warning. By
defining a new module, for example math_annotations.py with empty functions
with annotated signatures, you can let the static analyzer know what are
the annotations for another module. In this example, the annotate decorator
is just a way of telling the static analyzer that these annotations apply
to the math.ceil function, not math_annotations.ceil.

This is what TypeScript does to annotate popular libraries written in plain
JavaScript with zero type information.

Manuel.



>
>
>
>
> --
> Steven
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/5a07ff32/attachment.html>

From cory at lukasa.co.uk  Thu Aug 14 22:33:00 2014
From: cory at lukasa.co.uk (Cory Benfield)
Date: Thu, 14 Aug 2014 21:33:00 +0100
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140814185245.GT4525@ando>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
 <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>
 <20140814185245.GT4525@ando>
Message-ID: <CAH_hAJF8YDtCgLLmhmoKF8BK31ugzyZT4KqPmVuWREGkJxA4Vg@mail.gmail.com>

On 14 August 2014 19:52, Steven D'Aprano <steve at pearwood.info> wrote:
> I want to pass a Decimal to foo(). All I have to do is *not* install
> mypy, or disable it, and lo and behold, like magic, the type checking
> doesn't happen, and foo() operates by duck-typing just like in the glory
> days of Python 1.5. Both Fred and I are now happy, and with the explicit
> isinstance check removed, the only type checking that occurs when I run
> Fred's library are the run-time duck-typing checks.

Thanks for explaining Steven, that's a lot clearer. I understand where
you're coming from now.

I still don't agree, however. I suspect what's more likely to happen
is that Fred writes his code, a user goes to run it with duck typing,
and it breaks. Assuming the static checker is in CPython and on by
default, there are a number of options here, most of which are bad:

1. The user doesn't know about the type checker and Googles the
problem. They find there's a flag they can pass to make the problem go
away, so they do. They have now learned a bad habit: to silence these
errors, pass this flag. They can no longer gain any benefits from the
type checker: it may as well have been not there (or off by default).

2. The user doesn't know about the type checker and blames Fred's
library, opening a bug report. In extreme cases, for popular
libraries, this will happen so often that Fred will either relent and
remove the annotations or get increasingly frustrated and take it out
on the users. (I'm speaking from experience in this regard.)

3. The user knows about the type checker and isn't using it. They turn
it off. Fine, this is ok.

4. The user knows about the type checker but is using it for their own
code in the same program. They're between a rock and a hard place:
either they turn off the checker and lose the benefit in their own
code, or they stop duck typing. This is actually the worst of these
cases.

Basically, my objection is to the following (admittedly extreme) case:
a static type checker that is a) present in the core distribution, b)
on by default, and c) with the only available scope being per-program.
I think that such an implementation is a recipe for having everyone
learn to turn the checker off, wasting all the effort associated with
it.

I am much happier if (b) goes away. Off by default is fine. Not in the
core distribution at all is also fine (because it's effectively
off-by-default). Allowing refined scopes is also a good idea, but
doesn't solve the core problem: people will continue to just turn it
off.

I am not averse to having static checking be an option for Python and
for annotations to be the mechanism by which such typing is done. I
just think we should be really cautious about ever including it in
CPython.

From apalala at gmail.com  Thu Aug 14 22:12:09 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Thu, 14 Aug 2014 15:42:09 -0430
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140814174512.GQ4525@ando>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAN1YFWtvpmRauVr6qH+Xv1HuDBuJdZppRY6FZ+JFBViNvH-7_A@mail.gmail.com>
 <20140814174512.GQ4525@ando>
Message-ID: <CAN1YFWumG67Xk3hQLh3kxzRKt5oZVU_nNMDqMAQLKa0Z9DNWPQ@mail.gmail.com>

On Thu, Aug 14, 2014 at 1:15 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

> Yes. You say that as if it were a bad thing. It is not. Python 3 is
> here to stay and we should be promoting Python 3 only code. There is
> absolutely no need to apologise for that fact. If people are happy with
> Python the way it is in 2.7, or 1.5 for that matter, that's great, they
> can stay on it for ever, but all new features are aimed at 3.x and not
> 2.x or 1.x.
>

That's reasonable..., in theory.

Reality is that most of the people most supportive of the migration towards
Python 3 are currently writing code that is compatible with both 3.x and
2.[67].

You don't leave your people behind (not without a lifeboat).

Since its decided there will not be a 2.8, the right thing to do is to
delay decisions about static typing (or restrictions on annotations) till
4.0. It would be "a bad thing" to break or deprecate existing 3.x code with
3.5.

Cheers,
-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/ff8bb3e5/attachment-0001.html>

From ethan at stoneleaf.us  Thu Aug 14 22:44:56 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 14 Aug 2014 13:44:56 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CA+VECoVPye38je-dN=HLLGGso_uxfphK550wXj3sAPwbTRfZeA@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CA+VECoVO0COba4u3o5YD0shE0r-P8-q99sWMQtpRkisQ8wa8UQ@mail.gmail.com>
 <CAP7+vJLKNydsdLuc8jRjhfYr_R8P7u1Jg1aZ4Yb1LA1c-t3w1g@mail.gmail.com>
 <CA+VECoXkb+=UeSQbckY+prOgKx7AY4yUwFRV9ogPTTQ+mR=DOA@mail.gmail.com>
 <20140814185555.GU4525@ando>
 <CA+VECoVPye38je-dN=HLLGGso_uxfphK550wXj3sAPwbTRfZeA@mail.gmail.com>
Message-ID: <53ED1FC8.1040102@stoneleaf.us>

On 08/14/2014 01:29 PM, Manuel Cer?n wrote:
> On Thu, Aug 14, 2014 at 8:55 PM, Steven D'Aprano wrote:
>> On Thu, Aug 14, 2014 at 12:28:26PM +0200, Manuel Cer?n wrote:
>>>
>>>
>>> @annotate('math.ciel')
>>> def ciel(x: float) -> int:
>>>     pass
>>
>> I'm afraid I don't understand what the annotate decorator is doing here.
>> Can you explain please?
>
> The idea is to add type annotations to modules without modifying them. For example, in this case, the stdlib math module
> is defined and implemented in C, but you still want to have annotations for it so that if you write math.ciel('foo'),
> the static type analyzer gives you a error or warning. By defining a new module, for example math_annotations.py with
> empty functions with annotated signatures, you can let the static analyzer know what are the annotations for another
> module. In this example, the annotate decorator is just a way of telling the static analyzer that these annotations
> apply to the math.ceil function, not math_annotations.ceil.

To make sure I understand:  The above snippet is located in a file named 'math_annotations.py', and the annotate 
decorator says "the following function annotation should be stored against the 'ceil' function in the 'math' module, not 
the 'ceil' function in this current module" ?

--
~Ethan~

From ethan at stoneleaf.us  Thu Aug 14 22:51:53 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 14 Aug 2014 13:51:53 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAH_hAJF8YDtCgLLmhmoKF8BK31ugzyZT4KqPmVuWREGkJxA4Vg@mail.gmail.com>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
 <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>
 <20140814185245.GT4525@ando>
 <CAH_hAJF8YDtCgLLmhmoKF8BK31ugzyZT4KqPmVuWREGkJxA4Vg@mail.gmail.com>
Message-ID: <53ED2169.6080800@stoneleaf.us>

On 08/14/2014 01:33 PM, Cory Benfield wrote:
> On 14 August 2014 19:52, Steven D'Aprano wrote:
>>
>> I want to pass a Decimal to foo(). All I have to do is *not* install
>> mypy, or disable it, and lo and behold, like magic, the type checking
>> doesn't happen, and foo() operates by duck-typing just like in the glory
>> days of Python 1.5. Both Fred and I are now happy, and with the explicit
>> isinstance check removed, the only type checking that occurs when I run
>> Fred's library are the run-time duck-typing checks.
>
> Thanks for explaining Steven, that's a lot clearer. I understand where
> you're coming from now.
>
> I still don't agree, however. I suspect what's more likely to happen
> is that Fred writes his code, a user goes to run it with duck typing,
> and it breaks. Assuming the static checker is in CPython and on by
> default, there are a number of options here, most of which are bad:

These are bad assumptions, since the PEP is about defining how annotations are to be used and specifically states there 
will be *no run-time checking*.  To be of use at all you have to get a third-party program (mypy at this point) and use it.

So the scenario you list simply isn't going to happen... at least, not like that.

What could happen is newbie team member tries to check something in, but mypy and annotations are in the pre-check, 
no-one has told newbie team member about mypy or newbie forgot and is too embarrassed to go ask someone, so same basic 
problem arises.

That, however, is mostly outside the concerns of developing Python.

--
~Ethan~

From apalala at gmail.com  Thu Aug 14 23:03:13 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Thu, 14 Aug 2014 16:33:13 -0430
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140814185245.GT4525@ando>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
 <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>
 <20140814185245.GT4525@ando>
Message-ID: <CAN1YFWviwO8JJK2ROe1TWKYo9HgYNz8H=8xAg8k3WBhtRFoJOw@mail.gmail.com>

On Thu, Aug 14, 2014 at 2:22 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

> def foo(x:float)->float:
>     return (x+1)/2
>
> and rely on mypy to check the types at compile time. Fred is happy: he
> has static type checks, Python does it automatically for him (once he
> has set up his build system to call mypy), and he is now convinced that
> foo() is type-safe and an isinstance check at run-time would be a waste
> of cycles.
>

The foo() kind of examples won't cut it.

The standard library and other important Python libraries will take an
argument of any of several "reasonable" but otherwise unrelated types...
and do the right thing. Such is duck-typing. Trying to find the "common
abstract type" to cast it in stone is a waste of time.

For example, see https://docs.python.org/3.3/library/json.html#json.dump,
or take a look at http://pyyaml.org/.

In fact, Why aren't we instead discussing the much more interesting topic
of *type inference*, as it's going on in projects like PyDev, PyCharm,
Rope, and others?

I would be all-in for an approach that helps make code-completion tools
more precise and more available, and that would enable linters to tell me
why a certain call will probably not work. It could be a milder approach,
with "type hinting" for when the tools can't infer the type, instead of one
of "type specification".

def set_color(self, color):
   ...

o.set_color("red")

It could well be that 'color' should actually be one of the constants in a
Color enum, and the call should fail statically... if it was Java. In the
Python way, the method will probably figure out the caller's intentions,
and do the right thing.

Cheers,

p.s. I just had an epiphany: This discussion is not about empowering Python
programmers, but again about finding "magic" to allow bad (read "cheap")
programmers to write good programs. I'm out!

-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/a9e834cd/attachment.html>

From bob at redivi.com  Thu Aug 14 22:54:20 2014
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 14 Aug 2014 13:54:20 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAH_hAJF8YDtCgLLmhmoKF8BK31ugzyZT4KqPmVuWREGkJxA4Vg@mail.gmail.com>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
 <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>
 <20140814185245.GT4525@ando>
 <CAH_hAJF8YDtCgLLmhmoKF8BK31ugzyZT4KqPmVuWREGkJxA4Vg@mail.gmail.com>
Message-ID: <CACwMPm8w34=+ndBcUsyE=hr4GTZK0+DDZ=0n4Du41HGRsgMyHg@mail.gmail.com>

On Thu, Aug 14, 2014 at 1:33 PM, Cory Benfield <cory at lukasa.co.uk> wrote:

> On 14 August 2014 19:52, Steven D'Aprano <steve at pearwood.info> wrote:
> > I want to pass a Decimal to foo(). All I have to do is *not* install
> > mypy, or disable it, and lo and behold, like magic, the type checking
> > doesn't happen, and foo() operates by duck-typing just like in the glory
> > days of Python 1.5. Both Fred and I are now happy, and with the explicit
> > isinstance check removed, the only type checking that occurs when I run
> > Fred's library are the run-time duck-typing checks.
>
> Thanks for explaining Steven, that's a lot clearer. I understand where
> you're coming from now.
>
> I still don't agree, however. I suspect what's more likely to happen
> is that Fred writes his code, a user goes to run it with duck typing,
> and it breaks. Assuming the static checker is in CPython and on by
> default, there are a number of options here, most of which are bad:
>

These assumptions are incorrect.

* Adding the checker to CPython is not part of this proposal.
* Turning it on by default is not part of this proposal.
* Having the type checker in any way associated with the runtime is not
part of this proposal.

What is part of this proposal?

* An effort to standardize on a particular syntax for optional type
annotations using Python 3 function annotations
* The syntax will be handwavingly based on what's in mypy, and thus implies
some semantics but not the implementation of the type checker/inference/etc.
* A suggestion to use tools such as mypy to provide a static type checking
pass in much the same way that people use linters today
* A suggestion that IDEs, documentation tools, etc. take advantage of the
information provided by the type annotations
* Deprecation of using function annotations for any other purpose. They
can't really be composed, and ideally type checkers can work primarily at
the syntax level and not have to evaluate the module with a full Python
interpreter in order to extract the annotations, so it's best to keep the
syntax uniform.


> I am not averse to having static checking be an option for Python and
> for annotations to be the mechanism by which such typing is done. I
> just think we should be really cautious about ever including it in
> CPython.


So, it sounds like you're not averse to what is has been proposed. :)

-bob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/40e77b5c/attachment-0001.html>

From brakhane at googlemail.com  Fri Aug 15 00:11:16 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Fri, 15 Aug 2014 00:11:16 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAJaQC32Rb-snR2kd+21AsvjxAhgSPZDQ=5ctoPpQCqip3jS_Xg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>	<loom.20140813T222929-838@post.gmane.org>	<CAP7+vJLFP4BaRtp0Mhg3d2A56YVLj=odUbY94XtyFKtpyUoOFA@mail.gmail.com>	<53ED16A7.5080007@googlemail.com>
 <CAJaQC32Rb-snR2kd+21AsvjxAhgSPZDQ=5ctoPpQCqip3jS_Xg@mail.gmail.com>
Message-ID: <53ED3404.7080309@googlemail.com>

Am 14.08.2014 22:16, schrieb Sunjay Varma:
> One of the many benefits of Python is that you can use objects with
> equivalent interfaces in functions that may not have expected that
> type while they were being written. 
Exactly. As others have already noted, Guido's example is actually
restricting code from calling it with a file object for no particular
reason.

Another thing is that it restricts the entries of the iterables to be
str, while it actually only requires them to provide a split method.

For example, and I'm aware that this is a contrieved case, let's assume
word_count would actually be somehow extremely optimized code that
I don't want to rewrite.

I now want to count all prime factors in a list of integers.

Without annotations I *could* do something like this ("Language for
consenting adults"):

  class IntWrapper(int):
      def split(self):
          return primefactors(self)

  word_count(IntWrapper(i) for i in my_integer_list)

I don't want to argue whether that's good code or not (it probably
isn't), it would be possible to write such code and it will work correctly.

For mypy to accept such code, the type declaration of word_count would
have to be something like

  def word_count(input: Iterable[has("split() -> T")]) -> Dict[T, int]

Which would require a very complex type system.

If I understand MyPy correctly, the above would be possible, but it
would have to look something like

  cast(Dict[int, int], word_count(cast(List[str], (IntWrapper(i) for i
in my_integer_list)))

One could argue that this ugly piece of code is the rightful punishment
for abusing duck typing, but I'm not convinced
you should be forced to make the code unreadable (and therefore refactor
your code and reimplement word_count yourself).


> We have to be careful with this. If we do accept it (or any of the
> many alternatives suggested so far), then we should choose a few use
> cases and focus on solving them as best as possible. 
Yes, my fear is that we will either end up with a broken type system
like Java (which provides more problems than it solves) or end up with a
hugely complex type system and type hierachy like Scala
(see http://www.scala-lang.org/api/2.11.2/#scala.collection.MapLike for
a quick example what you end up with)




From ahammel87 at gmail.com  Fri Aug 15 00:35:04 2014
From: ahammel87 at gmail.com (Alex Hammel)
Date: Thu, 14 Aug 2014 15:35:04 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53ED3404.7080309@googlemail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CAP7+vJLFP4BaRtp0Mhg3d2A56YVLj=odUbY94XtyFKtpyUoOFA@mail.gmail.com>
 <53ED16A7.5080007@googlemail.com>
 <CAJaQC32Rb-snR2kd+21AsvjxAhgSPZDQ=5ctoPpQCqip3jS_Xg@mail.gmail.com>
 <53ED3404.7080309@googlemail.com>
Message-ID: <CA+_xFeoeGhMQr+B1G4hmyBMzm+vZV8QR8p-JWB=uHcfpxQjvSQ@mail.gmail.com>

If we're looking for precedents in other languages, the Dialyzer project
for Erlang is worth a look. It does completely optional static type
checking without affecting the runtime at all.

It was pretty widely adopted and it works well in my experience. I think
it's a bit different from what's being proposed here (it does type
inference as well, for instance), but it is a more or less successful
example of adding optional static checks to a dynamically typed language.

Based on my experiences with Dialyzer and the way I've used function
annotations in Python to date, I'm really excited about this proposal.



On Thu, Aug 14, 2014 at 3:11 PM, Dennis Brakhane <brakhane at googlemail.com>
wrote:

> Am 14.08.2014 22:16, schrieb Sunjay Varma:
> > One of the many benefits of Python is that you can use objects with
> > equivalent interfaces in functions that may not have expected that
> > type while they were being written.
> Exactly. As others have already noted, Guido's example is actually
> restricting code from calling it with a file object for no particular
> reason.
>
> Another thing is that it restricts the entries of the iterables to be
> str, while it actually only requires them to provide a split method.
>
> For example, and I'm aware that this is a contrieved case, let's assume
> word_count would actually be somehow extremely optimized code that
> I don't want to rewrite.
>
> I now want to count all prime factors in a list of integers.
>
> Without annotations I *could* do something like this ("Language for
> consenting adults"):
>
>   class IntWrapper(int):
>       def split(self):
>           return primefactors(self)
>
>   word_count(IntWrapper(i) for i in my_integer_list)
>
> I don't want to argue whether that's good code or not (it probably
> isn't), it would be possible to write such code and it will work correctly.
>
> For mypy to accept such code, the type declaration of word_count would
> have to be something like
>
>   def word_count(input: Iterable[has("split() -> T")]) -> Dict[T, int]
>
> Which would require a very complex type system.
>
> If I understand MyPy correctly, the above would be possible, but it
> would have to look something like
>
>   cast(Dict[int, int], word_count(cast(List[str], (IntWrapper(i) for i
> in my_integer_list)))
>
> One could argue that this ugly piece of code is the rightful punishment
> for abusing duck typing, but I'm not convinced
> you should be forced to make the code unreadable (and therefore refactor
> your code and reimplement word_count yourself).
>
>
> > We have to be careful with this. If we do accept it (or any of the
> > many alternatives suggested so far), then we should choose a few use
> > cases and focus on solving them as best as possible.
> Yes, my fear is that we will either end up with a broken type system
> like Java (which provides more problems than it solves) or end up with a
> hugely complex type system and type hierachy like Scala
> (see http://www.scala-lang.org/api/2.11.2/#scala.collection.MapLike for
> a quick example what you end up with)
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/f22bb16f/attachment.html>

From rosuav at gmail.com  Fri Aug 15 00:44:54 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 15 Aug 2014 08:44:54 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53ED16A7.5080007@googlemail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CAP7+vJLFP4BaRtp0Mhg3d2A56YVLj=odUbY94XtyFKtpyUoOFA@mail.gmail.com>
 <53ED16A7.5080007@googlemail.com>
Message-ID: <CAPTjJmo6NBouWtyAaUD1xdtpZShuVc4QDtY8TpjCnQB0Tt+VDw@mail.gmail.com>

On Fri, Aug 15, 2014 at 6:05 AM, Dennis Brakhane
<brakhane at googlemail.com> wrote:
> Does Mypy provide a way to "fix/monkeypatch" incorrect type declarations
> in function signatures? For example, by modifying __annotations__?

AIUI there's a separation of execution and type checking here - you
*either* run under mypy to check types, *or* run under Python to
execute the code. If that's the case, it should be possible to create
a magic module that you import which overwrites a few functions with
stubs.

# spam.py (which you can't change)
def func(x: int) -> int:
    return x * 3 + 5

# type_fixes.py in the regular PYTHONPATH
"""This is a stub; the mypy equivalent fixes some type
annotations used in third-party libraries."""

# type_fixes.py in the mypy PYTHONPATH
import spam
def func(x: Number) -> Number: pass
spam.func = func

# in main_module.py
import type_fixes
import spam
x = func(42.5)


Monkey-patching entire functions shouldn't be a problem if they don't
need bodies.

ChrisA

From brakhane at googlemail.com  Fri Aug 15 00:59:36 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Fri, 15 Aug 2014 00:59:36 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CA+_xFeoeGhMQr+B1G4hmyBMzm+vZV8QR8p-JWB=uHcfpxQjvSQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CAP7+vJLFP4BaRtp0Mhg3d2A56YVLj=odUbY94XtyFKtpyUoOFA@mail.gmail.com>
 <53ED16A7.5080007@googlemail.com>
 <CAJaQC32Rb-snR2kd+21AsvjxAhgSPZDQ=5ctoPpQCqip3jS_Xg@mail.gmail.com>
 <53ED3404.7080309@googlemail.com>
 <CA+_xFeoeGhMQr+B1G4hmyBMzm+vZV8QR8p-JWB=uHcfpxQjvSQ@mail.gmail.com>
Message-ID: <53ED3F58.1040004@googlemail.com>

Am 15.08.2014 00:35, schrieb Alex Hammel:

> I think it's a bit different from what's being proposed here (it does
> type inference as well, for instance)

FWIW, David Halter is currently working on extending Jedi to include a
linter that does type inference. For example, the current development
version can do the following


$ cat foo.py

def foo(x):
    return x + "foo"

foo(1)


$ python -m jedi linter foo.py
foo.py:2:13: E11 TypeError: unsupported operand type(s) for +:
<CompiledObject: 1> and <CompiledObject: 'foo'>




From greg.ewing at canterbury.ac.nz  Fri Aug 15 00:59:33 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 15 Aug 2014 10:59:33 +1200
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
In-Reply-To: <CAKkef2yzJZ5m6B1wLOW2bxXsPCPCF2sZa9ctsrHYckOYHLf4yw@mail.gmail.com>
References: <CAKkef2yzJZ5m6B1wLOW2bxXsPCPCF2sZa9ctsrHYckOYHLf4yw@mail.gmail.com>
Message-ID: <53ED3F55.6030502@canterbury.ac.nz>

willvarfar at gmail.com wrote:
> But proper annotation support in the language rather than overloading
> comments would definitely be my preference.

I've been thinking the same thing.

-- 
Greg

From bob at redivi.com  Thu Aug 14 22:58:45 2014
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 14 Aug 2014 13:58:45 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53ED1FC8.1040102@stoneleaf.us>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CA+VECoVO0COba4u3o5YD0shE0r-P8-q99sWMQtpRkisQ8wa8UQ@mail.gmail.com>
 <CAP7+vJLKNydsdLuc8jRjhfYr_R8P7u1Jg1aZ4Yb1LA1c-t3w1g@mail.gmail.com>
 <CA+VECoXkb+=UeSQbckY+prOgKx7AY4yUwFRV9ogPTTQ+mR=DOA@mail.gmail.com>
 <20140814185555.GU4525@ando>
 <CA+VECoVPye38je-dN=HLLGGso_uxfphK550wXj3sAPwbTRfZeA@mail.gmail.com>
 <53ED1FC8.1040102@stoneleaf.us>
Message-ID: <CACwMPm_rbrakMTAQL3SYWLRKAHa1mr9MAPs9m4J80jURbP=-zw@mail.gmail.com>

On Thu, Aug 14, 2014 at 1:44 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 08/14/2014 01:29 PM, Manuel Cer?n wrote:
>
>> On Thu, Aug 14, 2014 at 8:55 PM, Steven D'Aprano wrote:
>>
>>> On Thu, Aug 14, 2014 at 12:28:26PM +0200, Manuel Cer?n wrote:
>>>
>>>>
>>>>
>>>> @annotate('math.ciel')
>>>> def ciel(x: float) -> int:
>>>>     pass
>>>>
>>>
>>> I'm afraid I don't understand what the annotate decorator is doing here.
>>> Can you explain please?
>>>
>>
>> The idea is to add type annotations to modules without modifying them.
>> For example, in this case, the stdlib math module
>> is defined and implemented in C, but you still want to have annotations
>> for it so that if you write math.ciel('foo'),
>> the static type analyzer gives you a error or warning. By defining a new
>> module, for example math_annotations.py with
>> empty functions with annotated signatures, you can let the static
>> analyzer know what are the annotations for another
>> module. In this example, the annotate decorator is just a way of telling
>> the static analyzer that these annotations
>> apply to the math.ceil function, not math_annotations.ceil.
>>
>
> To make sure I understand:  The above snippet is located in a file named
> 'math_annotations.py', and the annotate decorator says "the following
> function annotation should be stored against the 'ceil' function in the
> 'math' module, not the 'ceil' function in this current module" ?
>

mypy has existing infrastructure in the form of stub modules to support
this use case:
https://github.com/JukkaL/mypy/blob/master/stubs/3.2/math.py

I'm not sure if stubs can easily be pulled in from outside of mypy, but
that would be straightforward to support if it doesn't work already.

-bob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/b1a13d2a/attachment.html>

From joseph.martinot-lagarde at m4x.org  Fri Aug 15 01:43:33 2014
From: joseph.martinot-lagarde at m4x.org (Joseph Martinot-Lagarde)
Date: Fri, 15 Aug 2014 01:43:33 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAN1YFWumG67Xk3hQLh3kxzRKt5oZVU_nNMDqMAQLKa0Z9DNWPQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAN1YFWtvpmRauVr6qH+Xv1HuDBuJdZppRY6FZ+JFBViNvH-7_A@mail.gmail.com>
 <20140814174512.GQ4525@ando>
 <CAN1YFWumG67Xk3hQLh3kxzRKt5oZVU_nNMDqMAQLKa0Z9DNWPQ@mail.gmail.com>
Message-ID: <53ED49A5.1070404@m4x.org>

Le 14/08/2014 22:12, Juancarlo A?ez a ?crit :
>
> On Thu, Aug 14, 2014 at 1:15 PM, Steven D'Aprano
> <steve at pearwood.info
> <mailto:steve at pearwood.info>> wrote:
>
>     Yes. You say that as if it were a bad thing. It is not. Python 3 is
>     here to stay and we should be promoting Python 3 only code. There is
>     absolutely no need to apologise for that fact. If people are happy with
>     Python the way it is in 2.7, or 1.5 for that matter, that's great, they
>     can stay on it for ever, but all new features are aimed at 3.x and not
>     2.x or 1.x.
>
>
> That's reasonable..., in theory.
>
> Reality is that most of the people most supportive of the migration
> towards Python 3 are currently writing code that is compatible with both
> 3.x and 2.[67].
>
> You don't leave your people behind (not without a lifeboat).
>
> Since its decided there will not be a 2.8, the right thing to do is to
> delay decisions about static typing (or restrictions on annotations)
> till 4.0. It would be "a bad thing" to break or deprecate existing 3.x
> code with 3.5.

It breaks nothing since it's optional. What you say is true for every 
new feature in python: using a feature present in on version of python 
prevents the code to be compatible with previous versions.

I mostly write code compatible with 2.7 and 3.x, I won't use annotations 
the same way I won't use asyncio or yield from...

>
> Cheers,
> --
> Juancarlo *A?ez*
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



From guido at python.org  Fri Aug 15 01:56:39 2014
From: guido at python.org (Guido van Rossum)
Date: Thu, 14 Aug 2014 16:56:39 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
Message-ID: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>

I have read pretty much the entire thread up and down, and I don't think I
can keep up with responding to every individual piece of feedback. (Also, a
lot of responses cancel each other out. :-)

I think there are three broad categories of questions to think about next.

(A) Do we even need this?

(B) What syntax to use?

(C) Does/should it support <feature X>?

Taking these in turn:

(A) Do we even need a standard for optional static typing?

Many people have shown either support for the idea, or pointed to some
other system that addresses the same issue. On the other hand, several
people have claimed that they don't need it, or that they worry it will
make Python less useful for them. (However, many of the detractors seem to
have their own alternative proposal. :-)

In the end I don't think we can ever know for sure -- but my intuition
tells me that as long as we keep it optional, there is a real demand. In
any case, if we don't start building something we'll never know whether
it'll be useful, so I am going to take a leap of faith and continue to
promote this idea.

I am going to make one additional assumption: the main use cases will be
linting, IDEs, and doc generation. These all have one thing in common: it
should be possible to run a program even though it fails to type check.
Also, adding types to a program should not hinder its performance (nor will
it help :-).

(B) What syntax should a standard system for optional static typing use?

There are many interesting questions here, but at the highest level there
are a few choices that constrain the rest of the discussion, and I'd like
to start with these. I see three or four "families" of approaches, and I
think the first order is to pick a family.

(1) The mypy family. (http://mypy-lang.org/) This is characterized by its
use of PEP 3107 function annotations and the constraint that its syntax
must be valid (current) Python syntax that can be evaluated without errors
at function definition time. However, mypy also supports collecting
annotations in separate "stub" files; this is how it handles annotations
for the stdlib and C extensions. When mypy annotations occur inline (not in
a stub file) they are used to type check the body of the annotated function
as well as input for type checking its callers.

(2) The pytypedecl family. (https://github.com/google/pytypedecl) This is a
custom syntax that can only be used in separate stub files. Because it is
not constrained by Python's current syntax, its syntax is slightly more
elegant than mypy.

(3) The PyCharm family. (
http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.html)
This is a custom syntax that lives entirely in docstrings. There is also a
way to use stub files with this. (In fact, every viable approach has to
support some form of stub files, if only to describe signatures for C
extensions.)

(I suppose we could add a 4th family that puts everything in comments, but
I don't think anyone is seriously working on such a thing, and I don't see
any benefits.)

There's also a variant of (1) that ?ukasz Langa would like to see -- use
the syntactic position of function annotations but using a custom syntax
(e.g. one similar to the pytypedecl syntax) that isn't evaluated at
function-definition time. This would have to use "from __future__ import
<something>" for backward compatibility. I'm skeptical about this though;
it is only slightly more elegant than mypy, and it would open the
floodgates of unconstrained language design.

So how to choose? I've read passionate attacks and defenses of each
approach. I've got a feeling that the three projects aren't all that
different in maturity (all are well beyond the toy stage, none are quite
ready for prime time). In terms of specific type system features (e.g.
forward references, generic types, duck typing) I expect they are all
acceptable, and all probably need some work (and there's no reason to
assume that work can't be done). All support stubs so you can specify
signatures for code you can't edit (whether C extension, stdlib or just
opaque 3rd party code).

To me there is no doubt that (1) is the most Pythonic approach. When we
discussed PEP 3107 (function annotations) it was always my goal that these
would eventually be used for type annotations. There was no consensus at
the time on what the rules for type checking should be, but their syntactic
position was never in doubt. So we decided to introduce "annotations" in
Python 3 in the hope that 3rd party experiments would eventually produce
something satisfactory. Mypy is one such experiment. One of the important
lessons I draw from mypy is that type annotations are most useful to
linters, and should (normally) not be used to enforce types at run time.
They are also not useful for code generation. None of that was obvious when
we were discussing PEP 3107!

I don't buy the argument that PEP 3107 promises that annotations are
completely free of inherent semantics. It promises compatibility, and I
take that very seriously, but I think it is reasonable to eventually
deprecate other uses of annotations -- there aren't enough significant
other uses for them to warrant crippling type annotations forever. In the
meantime, we won't be breaking existing use of annotations -- but they may
confuse a type checker, whether a stand-alone linter like mypy or built
into an IDE like PyCharm, and that may serve as an encouragement to look
for a different solution.

Most of the thornier issues brought up against mypy wouldn't go away if we
adopted another approach: whether to use concrete or abstract types, the
use of type variables, how to define type equivalence, the relationship
between a list of ints and a list of objects, how to spell "something that
implements the buffer interface", what to do about JSON, binary vs. text
I/O and the signature of open(), how to check code that uses isinstance(),
how to shut up the type checker when you know better... The list goes on.
There will be methods whose type signature can't be spelled (yet). There
will be code distributed with too narrowly defined types. Some programmers
will uglify their code to please the type checker.

There are questions about what to do for older versions of Python. I find
mypy's story here actually pretty good -- the mypy codec may be a hack, but
so is any other approach. Only the __future__ approach really loses out
here, because you can't add a new __future__ import to an old version.

So there you have it. I am picking the mypy family and I hope we can start
focusing on specific improvements to mypy. I also hope that somebody will
write converters from pytypedecl and PyCharm stubs into mypy stubs, so that
we can reuse the work already put into stub definitions for those two
systems. And of course I hope that PyCharm and pytypedecl will adopt mypy's
syntax (initially in addition to their native syntax, eventually as their
sole syntax).

PS. I realize I didn't discuss question (C) much. That's intentional -- we
can now start discussing specific mypy features in separate threads (or in
this one :-).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/82c1429e/attachment-0001.html>

From steve at pearwood.info  Fri Aug 15 02:14:39 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Aug 2014 10:14:39 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CACwMPm8w34=+ndBcUsyE=hr4GTZK0+DDZ=0n4Du41HGRsgMyHg@mail.gmail.com>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
 <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>
 <20140814185245.GT4525@ando>
 <CAH_hAJF8YDtCgLLmhmoKF8BK31ugzyZT4KqPmVuWREGkJxA4Vg@mail.gmail.com>
 <CACwMPm8w34=+ndBcUsyE=hr4GTZK0+DDZ=0n4Du41HGRsgMyHg@mail.gmail.com>
Message-ID: <20140815001439.GX4525@ando>

On Thu, Aug 14, 2014 at 01:54:20PM -0700, Bob Ippolito wrote:

> What is part of this proposal?
> 
> * An effort to standardize on a particular syntax for optional type
> annotations using Python 3 function annotations

+1 on that.

> * The syntax will be handwavingly based on what's in mypy, and thus implies
> some semantics but not the implementation of the type checker/inference/etc.

I have some reservations as to whether the mypy syntax is mature enough 
to bake in as standard, but other than that reservation, I prefer the 
mypy approach over putting type declarations in docstrings.

+1

> * A suggestion to use tools such as mypy to provide a static type checking
> pass in much the same way that people use linters today

"Tools such as mypy" is a very important point. If there is a standard 
syntax for type annotations, there's no reason why other linters 
couldn't support it as well.

+1

> * A suggestion that IDEs, documentation tools, etc. take advantage of the
> information provided by the type annotations

+1

> * Deprecation of using function annotations for any other purpose. They
> can't really be composed, and ideally type checkers can work primarily at
> the syntax level and not have to evaluate the module with a full Python
> interpreter in order to extract the annotations, so it's best to keep the
> syntax uniform.

-1

I don't think this one is justified. At the very least, I think the 
decision to deprecate or not should be deferred until at least 3.7. It's 
enough to say that:

- you can use function annotations for type checking; or

- you can use function annotations for something else;

but not both at the same time. I don't think it is a big burden to have 
mypy and other linters support an "opt-out" decorator, say, so that 
projects can use annotations for something else without confusing the 
linter.

As I understand it, the current behaviour of mypy is that you have to 
import typing in the module before it will type check the module, so 
that already gives you a way to skip type annotations on a per-module 
basis: just don't import typing.


-- 
Steven

From rosuav at gmail.com  Fri Aug 15 02:29:21 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 15 Aug 2014 10:29:21 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140815001439.GX4525@ando>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
 <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>
 <20140814185245.GT4525@ando>
 <CAH_hAJF8YDtCgLLmhmoKF8BK31ugzyZT4KqPmVuWREGkJxA4Vg@mail.gmail.com>
 <CACwMPm8w34=+ndBcUsyE=hr4GTZK0+DDZ=0n4Du41HGRsgMyHg@mail.gmail.com>
 <20140815001439.GX4525@ando>
Message-ID: <CAPTjJmpdVk1fAq9R=euQq_6eCsROKmgYALTHg3J0F9AHOtvOEw@mail.gmail.com>

On Fri, Aug 15, 2014 at 10:14 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> I don't think this one is justified. At the very least, I think the
> decision to deprecate or not should be deferred until at least 3.7. It's
> enough to say that:
>
> - you can use function annotations for type checking; or
>
> - you can use function annotations for something else;

But who is "you"? Presumably the application author. What about
imported modules - what will they use annotations for? Will
conflicting uses break stuff? Or, conversely, will every use of
annotations have to be meta-annotated with its purpose, to try to
avoid breaking things?

Simpler to just say "this feature is standardly used for this
purpose", and let people break that convention at their own risk if
they like. Deprecation of other options fits this.

ChrisA

From bob at redivi.com  Fri Aug 15 02:46:51 2014
From: bob at redivi.com (Bob Ippolito)
Date: Thu, 14 Aug 2014 17:46:51 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140815001439.GX4525@ando>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
 <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>
 <20140814185245.GT4525@ando>
 <CAH_hAJF8YDtCgLLmhmoKF8BK31ugzyZT4KqPmVuWREGkJxA4Vg@mail.gmail.com>
 <CACwMPm8w34=+ndBcUsyE=hr4GTZK0+DDZ=0n4Du41HGRsgMyHg@mail.gmail.com>
 <20140815001439.GX4525@ando>
Message-ID: <CACwMPm8unXi8m31FJ9TOjRyEX--EdpH7H21KGNVRJSk2z93P8A@mail.gmail.com>

On Thu, Aug 14, 2014 at 5:14 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Thu, Aug 14, 2014 at 01:54:20PM -0700, Bob Ippolito wrote:
> > * Deprecation of using function annotations for any other purpose. They
> > can't really be composed, and ideally type checkers can work primarily at
> > the syntax level and not have to evaluate the module with a full Python
> > interpreter in order to extract the annotations, so it's best to keep the
> > syntax uniform.
>
> -1
>
> I don't think this one is justified. At the very least, I think the
> decision to deprecate or not should be deferred until at least 3.7. It's
> enough to say that:
>
> - you can use function annotations for type checking; or
>
> - you can use function annotations for something else;
>
> but not both at the same time. I don't think it is a big burden to have
> mypy and other linters support an "opt-out" decorator, say, so that
> projects can use annotations for something else without confusing the
> linter.
>

Certainly using annotations for purposes other than type checking would be
permissible, but discouraged. No enforcement of any kind is proposed, just
language to go in the documentation. It would be non-trivial for the Python
compiler or runtime to enforce it anyway!

For that sort of annotation, a decorator probably isn't best, some
module-level syntax would likely be more appropriate.


> As I understand it, the current behaviour of mypy is that you have to
> import typing in the module before it will type check the module, so
> that already gives you a way to skip type annotations on a per-module
> basis: just don't import typing.
>

The current implementation of mypy will typecheck any module that uses
annotations or imports from typing.

-bob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/7845ecc4/attachment.html>

From steve at pearwood.info  Fri Aug 15 03:13:14 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Aug 2014 11:13:14 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAPTjJmpdVk1fAq9R=euQq_6eCsROKmgYALTHg3J0F9AHOtvOEw@mail.gmail.com>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
 <CAH_hAJFi6V-pxhoN7CUnAF4_2iw2tk0GAYzQLGfZmwSSGMQJAw@mail.gmail.com>
 <20140814185245.GT4525@ando>
 <CAH_hAJF8YDtCgLLmhmoKF8BK31ugzyZT4KqPmVuWREGkJxA4Vg@mail.gmail.com>
 <CACwMPm8w34=+ndBcUsyE=hr4GTZK0+DDZ=0n4Du41HGRsgMyHg@mail.gmail.com>
 <20140815001439.GX4525@ando>
 <CAPTjJmpdVk1fAq9R=euQq_6eCsROKmgYALTHg3J0F9AHOtvOEw@mail.gmail.com>
Message-ID: <20140815011314.GY4525@ando>

On Fri, Aug 15, 2014 at 10:29:21AM +1000, Chris Angelico wrote:
> On Fri, Aug 15, 2014 at 10:14 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> > I don't think this one is justified. At the very least, I think the
> > decision to deprecate or not should be deferred until at least 3.7. It's
> > enough to say that:
> >
> > - you can use function annotations for type checking; or
> >
> > - you can use function annotations for something else;
> 
> But who is "you"? Presumably the application author.

The author of the code containing the annotations.


> What about imported modules - what will they use annotations for? 

Whatever they choose.

If they import typing and don't explicitly disable typechecking, they 
will get the default meaning of annotations, which is typechecking. If 
they perform whatever step is required to tell the linter "don't check 
here", the linter will treat that module as if it had no annotations.

(That doesn't necessarily mean ignoring the module, it only means ignore 
the annotations. A type checker with type inference might still be able 
to work with the module, so long as it needs no type hints.)


> Will
> conflicting uses break stuff? Or, conversely, will every use of
> annotations have to be meta-annotated with its purpose, to try to
> avoid breaking things?

No. I think Guido has the right instinct to make annotations' default 
purpose be for type-checking, but it's easy enough to make it opt-out. 
Since this is Python-Ideas, here are some ideas:


* If "typing" is not imported at all in the module, linters should
  ignore annotations inside that module.

* Have a decorator that tells linters and other tools "these are
  not type annotations":

  from typing import skip

  @skip
  def function(x: Spam, y: Eggs)->Cheese:
      pass

  Compliant linters and IDEs will now skip function, treating it as
  if it had no annotations at all, leaving the interpretation of the
  annotations up to the author of the module.

Non-compliant linters, of course, should be beaten with a large halibut, 
like any other buggy software :-)

The last one will probably require a standardized convention for how 
compliant tools recognise whether or not annotations are for them. 
Perhaps something like:

    if '+state' in func.__annotations__:
        # skip type-checking.

I picked '+state' because it is an invalid identifier and so cannot 
clash with any parameter name. The value of __annotations__['+state'] 
can remain unspecified, different tools could use it for whatever they 
like without Python's blessing, only the existence of that key is enough 
to mark the function as "don't treat these as type annotations".



-- 
Steven

From stephen at xemacs.org  Fri Aug 15 04:15:10 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 15 Aug 2014 11:15:10 +0900
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
In-Reply-To: <20140814190237.GV4525@ando>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
Message-ID: <87wqaah51t.fsf@uwakimon.sk.tsukuba.ac.jp>

Steven D'Aprano writes:

 > > 2) PEP 3107 only specifies arguments and return values but not
 > > exceptions that can be raised by a function. Java has the "throws"
 > > syntax to list possible exceptions:
 > > 
 > >  public void readFile() throws IOException {}
 > 
 > I understand that this is called a "checked exception" in Java. I also 
 > understand that they are hated and derided

Sure, but that's because it's hard for a human to guess what might
happen down in lower-level functions, let alone 3rd-party libraries
and the runtime -- and your program fails if you guess wrong.

Maybe the fact that type-checking is going to be optional mitigates
that.

I suspect it's not terrible useful, but another idea is to invert the
sense, i.e. say what exceptions the function believes it handles, and
therefore cannot be raised.

 > Perhaps with a few years of experience, we might be able to extend this 
 > to exceptions without making the same mistakes as Java's checked 
 > exceptions, but I wouldn't rush into it.

+1 to that!


From stephen at xemacs.org  Fri Aug 15 04:39:55 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 15 Aug 2014 11:39:55 +0900
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
In-Reply-To: <CAN1YFWumG67Xk3hQLh3kxzRKt5oZVU_nNMDqMAQLKa0Z9DNWPQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAN1YFWtvpmRauVr6qH+Xv1HuDBuJdZppRY6FZ+JFBViNvH-7_A@mail.gmail.com>
 <20140814174512.GQ4525@ando>
 <CAN1YFWumG67Xk3hQLh3kxzRKt5oZVU_nNMDqMAQLKa0Z9DNWPQ@mail.gmail.com>
Message-ID: <87vbpuh3wk.fsf@uwakimon.sk.tsukuba.ac.jp>

Juancarlo A?ez writes:

 > Reality is that most of the people most supportive of the migration
 > towards Python 3 are currently writing code that is compatible with
 > both 3.x and 2.[67].

And under Guido's proposal, they can continue to do so.  They just
have to live without the benefits of function annotations in that
code, same as they already do.  Or they can write a 3to2 tool to strip
the annotations.  It's just that the benefits to using Python 3 vs.
2/3-compatible syntax may take a quantum leap upward.  I don't see a
real problem here.

All that Guido is proposing to do is to rationalize development effort
on something people are already doing by standardizing a good-enough
approach, on an opt-in basis.  Sure, there are people who want
something more coercive, but Python developers aren't going to stand
for that, and Guido himself is the first line of defense.  He's
already said that's not going to happen, not now (and IIUC probably
never).


From apalala at gmail.com  Fri Aug 15 05:01:26 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Thu, 14 Aug 2014 22:31:26 -0430
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <87vbpuh3wk.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAN1YFWtvpmRauVr6qH+Xv1HuDBuJdZppRY6FZ+JFBViNvH-7_A@mail.gmail.com>
 <20140814174512.GQ4525@ando>
 <CAN1YFWumG67Xk3hQLh3kxzRKt5oZVU_nNMDqMAQLKa0Z9DNWPQ@mail.gmail.com>
 <87vbpuh3wk.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAN1YFWtGGug9MBDN0ZsQ9OniKypsB+e1BmvjhTi4J_jRuR83iw@mail.gmail.com>

On Thu, Aug 14, 2014 at 10:09 PM, Stephen J. Turnbull <stephen at xemacs.org>
wrote:

> All that Guido is proposing to do is to rationalize development effort
> on something people are already doing by standardizing a good-enough
> approach, on an opt-in basis.  Sure, there are people who want
> something more coercive, but Python developers aren't going to stand
> for that, and Guido himself is the first line of defense.  He's
> already said that's not going to happen, not now (and IIUC probably
> never).
>

The change has been decreed, so it will be.

It's all right, because (as Guido suggests in the decree) we can now take
the discussion to how to make it good for all involved. The Python
community will likely succeed at that.

It is a tautology that only experience will reveal the truth, at least so
in our endeavour.

Signing off this thread, yours truly,

-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/5c388df1/attachment.html>

From ryan at ryanhiebert.com  Fri Aug 15 05:33:24 2014
From: ryan at ryanhiebert.com (Ryan Hiebert)
Date: Thu, 14 Aug 2014 22:33:24 -0500
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140814190237.GV4525@ando>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
Message-ID: <7CFFAB82-8F95-41FA-A356-F5D1CAE2DA1C@ryanhiebert.com>


> On Aug 14, 2014, at 2:02 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> 
> Would it be possible, and desirable, to modify the built-in types so 
> that we could re-use them in the type annotations?
> 
>    def word_count(input: list[str]) -> dict[str, int]:
> 
> 
> Since types are otherwise unlikely to be indexable like that, I think 
> that might work.

Huge +1 from me. I know that extending types to do more might not be the first thing someone would want to do, but I think describing types well is exactly the right thing to extend type instances to do.

If that were to be considered plausible, it would also make sense to my mind to implement the union operator (|) on types, so that we could use:

int | None

Which I think would be quite beautiful. (or would it need to be NoneType? Perhaps we can give some special consideration to the None singleton)



It?s only tangentially related, but my (likely broken) work on a TypeSet metaclass might also be informative. It?s goal was to bring set operations to types, which has some overlap with the above discussion.

https://github.com/ryanhiebert/typeset

From abarnert at yahoo.com  Fri Aug 15 05:51:16 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 14 Aug 2014 20:51:16 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
In-Reply-To: <20140814190237.GV4525@ando>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
Message-ID: <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com>

On Thursday, August 14, 2014 12:03 PM, Steven D'Aprano <steve at pearwood.info> wrote:

> > On Wed, Aug 13, 2014 at 10:29:48PM +0200, Christian Heimes wrote:
> 
>>  1) I'm not keen with the naming of mypy's typing classes. The 
> visual
>>  distinction between e.g. dict() and Dict() is too small and IMHO
>>  confusing for newcomers. How about an additional 'T' prefix to make
>>  clear that the objects are referring to typing objects?
>> 
>> ?  from typing import TList, TDict
>> 
>> ?  def word_count(input: TList[str]) -> TDict[str, int]:
>> ? ? ?  ...
> 
> Would it be possible, and desirable, to modify the built-in types so 
> that we could re-use them in the type annotations?
> 
> ? ? def word_count(input: list[str]) -> dict[str, int]:
> 
> 
> Since types are otherwise unlikely to be indexable like that, I think 
> that might work.

I strongly agree with the basic sentiment here, but if you take it a little bit farther, it makes things a lot simpler.


Most of the code in typing.py is a duplication of the ABCs in the collections.abc module (and io and maybe others). I understand that MyPy couldn't monkeypatch that code into the stdlib, so it had to fork the contents, but if this is going into the stdlib, can't we just modify collections/abc.py instead? Why not have the type information on collections.abc.Sequence be introspectable at runtime? More importantly, why not have the interface defined by collections.abc.Sequence be _exactly_ the same as the one checked by the static type checker, instead of just very similar?

This would also make it easier for some libraries to document their types. For example, dbm.* could inherit from or register with MutableMapping[bytes, bytes] instead of just MutableMapping, and then it wouldn't have to explain in the docstring that the keys and values have to be bytes.

Once you move the ABC typing to the actual ABCs, the only problem left is the built-in (concrete) collections. But I still don't understand why people even _want_ those to be checked. Does anyone have a good example of a function that needs to restrict its arguments to list[str] instead of Sequence[str]? Guido gave us a _terrible_ example: a function whose most obvious use was on text files, but he added a static type check that prevents it being used that way. And I think that will be the case the vast majority of the time. When I see documentation in a PyPI library or a colleague's code that says a function takes a list, it's almost always a lie; the function in fact takes any iterable (or occasionally any sequence or mutable sequence). (In fact, every exception I can think of is written in C, so it couldn't be easily annotated anyway.) It seems like we're making things a lot harder for ourselves, just for a handful of types that people are almost
 always going to use wrong.

(As a side note, generic ABCs might even be the answer to the AnyStr problem: str implements String[str], bytes implements String[bytes], bytearray implements MutableString[bytes], and classes like PyObjC's NSString can now document that they implement String[str] or String[bytes] as appropriate.)

Anyway, these leaves typing.py as nothing but functions (Union, Optional, etc.) on types defined elsewhere, in the rest of the stdlib.

From stefan_ml at behnel.de  Fri Aug 15 06:12:18 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 15 Aug 2014 06:12:18 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJK6rOQTG37P9z6DfSAKBMx-wpCGTQdzG12G4bK_fatO+w@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
 <CAA_f+LxPbisNd949SmiKhkQNM_N12e0qvQ+aO9uhn1qQ0ji6+g@mail.gmail.com>
 <CAP7+vJK6rOQTG37P9z6DfSAKBMx-wpCGTQdzG12G4bK_fatO+w@mail.gmail.com>
Message-ID: <lsk1b2$tkn$1@ger.gmane.org>

Guido van Rossum schrieb am 14.08.2014 um 07:24:
> On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo wrote:
>> You could use AnyStr to make the example work with bytes as well:
>>
>>   def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]:
>>       result = {}  #type: Dict[AnyStr, int]
>>
>>       for line in input:
>>           for word in line.split():
>>               result[word] = result.get(word, 0) + 1
>>       return result
>>
>> Again, if this is just a simple utility function that you use once or
>> twice, I see no reason to spend a lot of effort in coming up with the most
>> general signature. Types are an abstraction and they can't express
>> everything precisely -- there will always be a lot of cases where you can't
>> express the most general type. However, I think that relatively simple
>> types work well enough most of the time, and give the most bang for the
>> buck.
> 
> I heartily agree. But just for the type theorists amongst us, if I really
> wanted to write the most general type, how would I express that the AnyStr
> in the return type matches the one in the argument? (I think pytypedecl
> would use something like T <= AnyStr.)

That's how Cython's "fused types" (generics) work, at least. They go by
name: same name of the type, same type. Otherwise, use alias names, which
make the types independent from each other.

http://docs.cython.org/src/userguide/fusedtypes.html

While it's a matter of definition what way to go here (same type or not),
practice has shown that it's clearly the right decision to make identical
types the default.

Stefan



From guido at python.org  Fri Aug 15 06:15:15 2014
From: guido at python.org (Guido van Rossum)
Date: Thu, 14 Aug 2014 21:15:15 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
 <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com>
Message-ID: <CAP7+vJLcsqVPYh2a6VJgci=t3rp9-pMFBV4xHdERbeemNBxZ8A@mail.gmail.com>

On Thu, Aug 14, 2014 at 8:51 PM, Andrew Barnert <
abarnert at yahoo.com.dmarc.invalid> wrote:

>
> Most of the code in typing.py is a duplication of the ABCs in the
> collections.abc module (and io and maybe others). I understand that MyPy
> couldn't monkeypatch that code into the stdlib, so it had to fork the
> contents, but if this is going into the stdlib, can't we just modify
> collections/abc.py instead? Why not have the type information on
> collections.abc.Sequence be introspectable at runtime? More importantly,
> why not have the interface defined by collections.abc.Sequence be _exactly_
> the same as the one checked by the static type checker, instead of just
> very similar?
>
> This would also make it easier for some libraries to document their types.
> For example, dbm.* could inherit from or register with
> MutableMapping[bytes, bytes] instead of just MutableMapping, and then it
> wouldn't have to explain in the docstring that the keys and values have to
> be bytes.
>
> Once you move the ABC typing to the actual ABCs, the only problem left is
> the built-in (concrete) collections. But I still don't understand why
> people even _want_ those to be checked. Does anyone have a good example of
> a function that needs to restrict its arguments to list[str] instead of
> Sequence[str]? Guido gave us a _terrible_ example: a function whose most
> obvious use was on text files, but he added a static type check that
> prevents it being used that way. And I think that will be the case the vast
> majority of the time. When I see documentation in a PyPI library or a
> colleague's code that says a function takes a list, it's almost always a
> lie; the function in fact takes any iterable (or occasionally any sequence
> or mutable sequence). (In fact, every exception I can think of is written
> in C, so it couldn't be easily annotated anyway.) It seems like we're
> making things a lot harder for ourselves, just for a handful of types that
> people are almost
>  always going to use wrong.
>
> (As a side note, generic ABCs might even be the answer to the AnyStr
> problem: str implements String[str], bytes implements String[bytes],
> bytearray implements MutableString[bytes], and classes like PyObjC's
> NSString can now document that they implement String[str] or String[bytes]
> as appropriate.)
>
> Anyway, these leaves typing.py as nothing but functions (Union, Optional,
> etc.) on types defined elsewhere, in the rest of the stdlib.
>

I think I already responded to a similar proposal. I think modifying the
existing ABCs is fine for Python 3.5 and beyond, but I think I'd like to
keep aliasing in the typing module to make it easier to use annotations in
earlier Python versions (using a typing.py installed from PyPI).

I feel similar about using a|b as a concise way to spell Union[a, n], and
int|None would be quite decent as a way to spell optional int. But again,
these could only be used in code meant exclusively for Python 3.5 and later.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/8e809879/attachment.html>

From greg at krypto.org  Fri Aug 15 06:18:50 2014
From: greg at krypto.org (Gregory P. Smith)
Date: Fri, 15 Aug 2014 04:18:50 +0000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CAP7+vJLFP4BaRtp0Mhg3d2A56YVLj=odUbY94XtyFKtpyUoOFA@mail.gmail.com>
 <53ED16A7.5080007@googlemail.com>
Message-ID: <CAGE7PN+LKN1mmb8UWKiXXc1QBtuAeot4pZ=Vn1LLA9CqgKvpCg@mail.gmail.com>

On Thu Aug 14 2014 at 1:06:20 PM Dennis Brakhane <brakhane at googlemail.com>
wrote:

> Am 13.08.2014 23:46, schrieb Guido van Rossum:
> >
> >
> > Mypy has a cast() operator that you can use to shut it up when you
> > (think you) know the conversion is safe.
> >
> Does Mypy provide a way to "fix/monkeypatch" incorrect type declarations
> in function signatures? For example, by modifying __annotations__?
>
> My pet peeve of static languages is that programmers are often too
> fixated on their particular problem that they don't think about alternate
> uses for their code and make the type declarations uncessarily complex.
>
> For example, in Java nearly every method in Java that deals with
> character strings uses "String" as parameter type, while they should
> have used "CharSequence". Having to read an entire input stream and
> storing it in a String just to be able to use a method is not fun
> (String is final in Java)
>
> I'm worried that in Python we will have utility functions that declare
> they require a List[int], when in fact they actually only require a
> Sequence[int] or
> Sequence[Number].
>
>
This always happens, even in docstrings where people state types today. For
example, I often correct people during code reviews to say sequence of
instead of list of in docstrings. Nothing can really prevent that beyond
documentation about annotations having good examples and promoting the use
of concept or interface annotations over specific type annotations.

But what's being proposed here isn't a strict static type check, it's an
annotation.  Sure, if someone misannotates their code and runs it through a
"compile time" checker (aka superlint) such as mypy or something even more
advanced, it will highlight problems where there may in fact be none. But
the problem being highlighted there is that the annotation is wrong.

Any system doing that level of code analysis will need a way to deal with
loading corrected annotations for code that for some reason cannot be
changed directly in the annotated files themselves (standard library, third
party library from pypi, etc). But this is really no different than loading
annotations for extension modules and the stdlib builtins. Those are
already likely to be separate files declaring interface annotations only
(there is a pile of these that we've generated via code analysis for
builtins and parts of the stdlib in the pytypedecl repo, expect more of
that to come from anyone working on this kind of thing).

While Mypy's cast is nice in that I won't have to wrap my Integer Tuple
> in list like object, having to cast it every time I use a particular
> broken utility method
> feels very ugly to me; and defining a wrapper function with the correct
> type information feels like unnecessary run time overhead for no gain.
>
> Don't get me wrong, I'm not entirely against some kind of type checking,
> but I fear that there must exist possible workarounds for badly written
> code.
>

always.  but this is more a task for all checker implementations.  not
something the syntax for annotations itself can prevent.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/fc34037f/attachment.html>

From guido at python.org  Fri Aug 15 06:34:31 2014
From: guido at python.org (Guido van Rossum)
Date: Thu, 14 Aug 2014 21:34:31 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lsk1b2$tkn$1@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
 <CAA_f+LxPbisNd949SmiKhkQNM_N12e0qvQ+aO9uhn1qQ0ji6+g@mail.gmail.com>
 <CAP7+vJK6rOQTG37P9z6DfSAKBMx-wpCGTQdzG12G4bK_fatO+w@mail.gmail.com>
 <lsk1b2$tkn$1@ger.gmane.org>
Message-ID: <CAP7+vJLGFC7RQSxr_+_WhjD21t1=Vwz+FOsKV3CwXhHp3FNi0g@mail.gmail.com>

On Thu, Aug 14, 2014 at 9:12 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Guido van Rossum schrieb am 14.08.2014 um 07:24:
> > On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo wrote:
> >> You could use AnyStr to make the example work with bytes as well:
> >>
> >>   def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]:
> >>       result = {}  #type: Dict[AnyStr, int]
> >>
> >>       for line in input:
> >>           for word in line.split():
> >>               result[word] = result.get(word, 0) + 1
> >>       return result
> >>
> >> Again, if this is just a simple utility function that you use once or
> >> twice, I see no reason to spend a lot of effort in coming up with the
> most
> >> general signature. Types are an abstraction and they can't express
> >> everything precisely -- there will always be a lot of cases where you
> can't
> >> express the most general type. However, I think that relatively simple
> >> types work well enough most of the time, and give the most bang for the
> >> buck.
> >
> > I heartily agree. But just for the type theorists amongst us, if I really
> > wanted to write the most general type, how would I express that the
> AnyStr
> > in the return type matches the one in the argument? (I think pytypedecl
> > would use something like T <= AnyStr.)
>
> That's how Cython's "fused types" (generics) work, at least. They go by
> name: same name of the type, same type. Otherwise, use alias names, which
> make the types independent from each other.
>
> http://docs.cython.org/src/userguide/fusedtypes.html
>
> While it's a matter of definition what way to go here (same type or not),
> practice has shown that it's clearly the right decision to make identical
> types the default.
>

I don't understand those docs at all, but I do think I understand the rule
"same name, same type" and I think I like it. Let me be clear -- in this
example:

def word_count(input: Iterable[AnyStr]) -> Mapping[AnyStr, int]:
    ...

the implication would be that if the input is Iterable[bytes] the output is
Mapping[bytes, int] while if the input is Iterable[str] the output is
Mapping[str, int]. Have I got that right? I hope so, because I think it is
a nice simplifying rule that covers a lot of cases in practice. (Note:
AnyStr is a predefined type in mypy that means "str or bytes".)

BTW there are a lot of messy things to consider around bytes, and IIUC mypy
currently doesn't really cover them. Often when you write code that accepts
a bytes instance, in practice it will accept anything that supports the
buffer protocol (e.g. bytearray and memoryview). Except when you are going
to use it as a dict key, then bytearray won't work. And if you say that you
are returning bytes, you probably shouldn't be returning a memoryview or
bytearray. I don't expect that any type system we can come up with will be
quite precise enough to cover all the cases, so we probably shouldn't lose
too much sleep over this.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/50588737/attachment-0001.html>

From ncoghlan at gmail.com  Fri Aug 15 06:40:37 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 15 Aug 2014 14:40:37 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
Message-ID: <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>

On 15 August 2014 09:56, Guido van Rossum <guido at python.org> wrote:
>
> I don't buy the argument that PEP 3107 promises that annotations are
> completely free of inherent semantics.

It's also worth noting the corresponding bullet point in PEP 3100
(under http://www.python.org/dev/peps/pep-3100/#core-language):

* Add optional declarations for static typing [45] [10] [done]

[10] Guido's blog ("Python Optional Typechecking Redux")
http://www.artima.com/weblogs/viewpost.jsp?thread=89161
[45] PEP 3107 (Function Annotations) http://www.python.org/dev/peps/pep-3107

> It promises compatibility, and I take
> that very seriously, but I think it is reasonable to eventually deprecate
> other uses of annotations -- there aren't enough significant other uses for
> them to warrant crippling type annotations forever. In the meantime, we
> won't be breaking existing use of annotations -- but they may confuse a type
> checker, whether a stand-alone linter like mypy or built  into an IDE like
> PyCharm, and that may serve as an encouragement to look for a different
> solution.

Linters/checkers may also want to provide a configurable way to say
"the presence of decorator <X> means the annotations on that function
aren't type markers". That ties in with the recommendation we added to
PEP 8 a while back: "It is recommended that third party experiments
with annotations use an associated decorator to indicate how the
annotation should be interpreted."

> So there you have it. I am picking the mypy family and I hope we can start
> focusing on specific improvements to mypy. I also hope that somebody will
> write converters from pytypedecl and PyCharm stubs into mypy stubs, so that
> we can reuse the work already put into stub definitions for those two
> systems. And of course I hope that PyCharm and pytypedecl will adopt mypy's
> syntax (initially in addition to their native syntax, eventually as their
> sole syntax).

Having Argument Clinic generate appropriate annotations automatically
could also be interesting.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Fri Aug 15 06:41:02 2014
From: guido at python.org (Guido van Rossum)
Date: Thu, 14 Aug 2014 21:41:02 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAGE7PN+LKN1mmb8UWKiXXc1QBtuAeot4pZ=Vn1LLA9CqgKvpCg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CAP7+vJLFP4BaRtp0Mhg3d2A56YVLj=odUbY94XtyFKtpyUoOFA@mail.gmail.com>
 <53ED16A7.5080007@googlemail.com>
 <CAGE7PN+LKN1mmb8UWKiXXc1QBtuAeot4pZ=Vn1LLA9CqgKvpCg@mail.gmail.com>
Message-ID: <CAP7+vJJ17B2LjETPMBBCQkgTUCx-YESw1RcBNd3-fZELktYz4Q@mail.gmail.com>

On Thu, Aug 14, 2014 at 9:18 PM, Gregory P. Smith <greg at krypto.org> wrote:

> (there is a pile of these that we've generated via code analysis for
> builtins and parts of the stdlib in the pytypedecl repo, expect more of
> that to come from anyone working on this kind of thing).
>

This is exciting news! It means that we should be able to convert
pytypedecl stubs to mypy stubs and mypy will get a lot of stdlib stubs for
free. I filed https://github.com/JukkaL/mypy/issues/382 for this. Hopefully
you are releasing Python 3 stubs along these lines too?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/68fa1aac/attachment.html>

From guido at python.org  Fri Aug 15 06:46:47 2014
From: guido at python.org (Guido van Rossum)
Date: Thu, 14 Aug 2014 21:46:47 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
Message-ID: <CAP7+vJLxNsgJ6fQfxCxdqeahyA7=9ihBAVrEoS-iDBXqBEHMGQ@mail.gmail.com>

On Thu, Aug 14, 2014 at 9:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 15 August 2014 09:56, Guido van Rossum <guido at python.org> wrote:
> >
> > I don't buy the argument that PEP 3107 promises that annotations are
> > completely free of inherent semantics.
>
> It's also worth noting the corresponding bullet point in PEP 3100
> (under http://www.python.org/dev/peps/pep-3100/#core-language):
>
> * Add optional declarations for static typing [45] [10] [done]
>
> [10] Guido's blog ("Python Optional Typechecking Redux")
> http://www.artima.com/weblogs/viewpost.jsp?thread=89161
> [45] PEP 3107 (Function Annotations)
> http://www.python.org/dev/peps/pep-3107
>

Such youthful optimism. :-)


> Having Argument Clinic generate appropriate annotations automatically
> could also be interesting.
>

How much of the 3.5 stdlib is currently covered by Argument Clinic? I
thought there's still a lot left to do. Might it be possible to convert
some of pytypedecl's stubs into AC stubs? Alternatively, the AC info could
be turned into mypy stubs. I'm just really hoping that between AC,
pytypedecl, PyCharm and mypy we have specs for most builtins and extension
modules in machine-readable form already, and we could use this combined
information to bootstrap mypy's collection of stubs.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140814/8ded6d12/attachment.html>

From abarnert at yahoo.com  Fri Aug 15 06:43:54 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 14 Aug 2014 21:43:54 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
In-Reply-To: <CAGE7PN+LKN1mmb8UWKiXXc1QBtuAeot4pZ=Vn1LLA9CqgKvpCg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <loom.20140813T222929-838@post.gmane.org>
 <CAP7+vJLFP4BaRtp0Mhg3d2A56YVLj=odUbY94XtyFKtpyUoOFA@mail.gmail.com>
 <53ED16A7.5080007@googlemail.com>
 <CAGE7PN+LKN1mmb8UWKiXXc1QBtuAeot4pZ=Vn1LLA9CqgKvpCg@mail.gmail.com> 
Message-ID: <1408077834.74431.YahooMailNeo@web181002.mail.ne1.yahoo.com>

On Thursday, August 14, 2014 9:20 PM, Gregory P. Smith <greg at krypto.org> wrote:


>On Thu Aug 14 2014 at 1:06:20 PM Dennis Brakhane <brakhane at googlemail.com> wrote:
>
>>My pet peeve of static languages is that programmers are often too
>>fixated on their particular problem that they don't think about alternate
>>uses for their code and make the type declarations uncessarily complex.
>>
>>For example, in Java nearly every method in Java that deals with
>>character strings uses "String" as parameter type, while they should
>>have used "CharSequence". Having to read an entire input stream and
>>storing it in a String just to be able to use a method is not fun
>>(String is final in Java)
>>
>>I'm worried that in Python we will have utility functions that declare
>>they require a List[int], when in fact they actually only require a
>>Sequence[int] or
>>Sequence[Number].
>
>This always happens, even in docstrings where people state types today. For example, I often correct people during code reviews to say sequence of instead of list of in docstrings. Nothing can really prevent that

We can't _prevent_ it, but we certainly can _drastically reduce_ it, just by not providing List[int].

Again, I think 90% or more of the uses of List[int] in type annotations will be incorrect. Why go out of our way to make things more complicated just so people can make more mistakes?

If anyone really does need a list of ints, implementing List should be pretty simple (in fact, it might even just be `class List(MutableSequence, list): pass`). Why bend over backwards to make it easier for people to make mistakes?


From stefan_ml at behnel.de  Fri Aug 15 07:35:28 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 15 Aug 2014 07:35:28 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJLGFC7RQSxr_+_WhjD21t1=Vwz+FOsKV3CwXhHp3FNi0g@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <1407980355.652.YahooMailNeo@web181004.mail.ne1.yahoo.com>
 <CAA_f+LxPbisNd949SmiKhkQNM_N12e0qvQ+aO9uhn1qQ0ji6+g@mail.gmail.com>
 <CAP7+vJK6rOQTG37P9z6DfSAKBMx-wpCGTQdzG12G4bK_fatO+w@mail.gmail.com>
 <lsk1b2$tkn$1@ger.gmane.org>
 <CAP7+vJLGFC7RQSxr_+_WhjD21t1=Vwz+FOsKV3CwXhHp3FNi0g@mail.gmail.com>
Message-ID: <lsk671$dpt$1@ger.gmane.org>

Guido van Rossum schrieb am 15.08.2014 um 06:34:
> On Thu, Aug 14, 2014 at 9:12 PM, Stefan Behnel wrote:
>> Guido van Rossum schrieb am 14.08.2014 um 07:24:
>>> On Wed, Aug 13, 2014 at 9:06 PM, Jukka Lehtosalo wrote:
>>>> You could use AnyStr to make the example work with bytes as well:
>>>>
>>>>   def word_count(input: Iterable[AnyStr]) -> Dict[AnyStr, int]:
>>>>       result = {}  #type: Dict[AnyStr, int]
>>>>
>>>>       for line in input:
>>>>           for word in line.split():
>>>>               result[word] = result.get(word, 0) + 1
>>>>       return result
>>>>
>>>> Again, if this is just a simple utility function that you use once or
>>>> twice, I see no reason to spend a lot of effort in coming up with the
>> most
>>>> general signature. Types are an abstraction and they can't express
>>>> everything precisely -- there will always be a lot of cases where you
>> can't
>>>> express the most general type. However, I think that relatively simple
>>>> types work well enough most of the time, and give the most bang for the
>>>> buck.
>>>
>>> I heartily agree. But just for the type theorists amongst us, if I really
>>> wanted to write the most general type, how would I express that the
>> AnyStr
>>> in the return type matches the one in the argument? (I think pytypedecl
>>> would use something like T <= AnyStr.)
>>
>> That's how Cython's "fused types" (generics) work, at least. They go by
>> name: same name of the type, same type. Otherwise, use alias names, which
>> make the types independent from each other.
>>
>> http://docs.cython.org/src/userguide/fusedtypes.html
>>
>> While it's a matter of definition what way to go here (same type or not),
>> practice has shown that it's clearly the right decision to make identical
>> types the default.
> 
> I don't understand those docs at all

I'm not surprised. ;)

The main idea is that you declare (typedef) a "fused" type that means "any
of the following list of types". Then you use it in a function signature
and the compiler explodes it into multiple specialised implementations that
get separately optimised for the specific type(s) they use. Compile time
generic functions, essentially. You get the cross product of all different
fused types that your function uses, but in practice, you almost always
want only one specialisation for each type, regardless of how often you
used it in the argument list.


> but I do think I understand the rule
> "same name, same type" and I think I like it. Let me be clear -- in this
> example:
> 
> def word_count(input: Iterable[AnyStr]) -> Mapping[AnyStr, int]:
>     ...
> 
> the implication would be that if the input is Iterable[bytes] the output is
> Mapping[bytes, int] while if the input is Iterable[str] the output is
> Mapping[str, int]. Have I got that right? I hope so, because I think it is
> a nice simplifying rule that covers a lot of cases in practice.

Yes, absolutely. One caveat for Python: static analysis tools (including
Cython) will usually have the AST available and thus see the type name
used. Once the annotation is in the __annotations__ dict, however, it's
lost and reduced to the base type object instance. So renaming types would
have to be an explicit operation that the type object knows about.
Otherwise, you'd loose semantics at runtime (not sure it matters much in
practice, but it would when used for documentation purposes). Not very DRY,
but as I said, the hugely more normal case is to want all types the same.


> BTW there are a lot of messy things to consider around bytes, and IIUC mypy
> currently doesn't really cover them. Often when you write code that accepts
> a bytes instance, in practice it will accept anything that supports the
> buffer protocol (e.g. bytearray and memoryview).

Yes, totally. I've been teaching people that for years now, but it's so
much easier for them to write "x: bytes" than to remember to be forgiving
about input and think about what that means for their specific code. Not
typing the input at all is actually the best solution in many cases, but
getting that into the head of users who are just discovering the beauty of
an optionally typed Python language is a true up-hill battle. Sometimes I
even encourage them to use memory views instead of expecting "real" byte
string input, even though that can make working with the data less
"stringish". But it's what the users of their code will want.

Cython essentially uses NumPy-ish syntax for (compile time) memory views,
i.e. you'd write "int[:] x" to unpack a 1-dimensional buffer of item type C
int, or "unsigned char[:] b" for a plain uchar buffer. Here's a bunch of
examples:

http://docs.cython.org/src/userguide/memoryviews.html

This is a very well established syntax by now, with lots of code out there
using it. Makes working with arbitrary buffer providers a charm. Note that
Cython has its own C level memory view implementation, so this is way more
efficient than Python's generic memoryview objects (but PEP 3118 based, so
compatible).


> Except when you are going
> to use it as a dict key, then bytearray won't work. And if you say that you
> are returning bytes, you probably shouldn't be returning a memoryview or
> bytearray.

Right. Hashability, strict output, all that.


> I don't expect that any type system we can come up with will be
> quite precise enough to cover all the cases, so we probably shouldn't lose
> too much sleep over this.

Well, Cython's type system has pretty much all you'd need. But it's linked
to the compiler in the sense that some features exist because Cython can do
them at compile time. Not everything can be done in pure Python at runtime
or import time.

Stefan



From stephen at xemacs.org  Fri Aug 15 07:46:10 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 15 Aug 2014 14:46:10 +0900
Subject: [Python-ideas] Proposal: Use mypy syntax for function
 annotations
In-Reply-To: <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
 <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com>
Message-ID: <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp>

It was written:

 > a static type check that prevents it being used that way.

Can we please refrain from using the word "prevent" in this thread?

I suggest instead

    a static type check that generates [scads of|tons of|way too many]
    [spurious|bogus|annoying] warnings when used that way.

which is IMO a genuine concern but also probably has technological
fixes like the one Andrew proposes (automatically promoting many
container types to the ABC actually needed).

In some sense I'm not really bothered by this (I'd be willing to omit
all the bracketed phrases :-).  If an upstream author writes

    def somefilter(l_of_s: List[str]) -> List[int]:
        pass

I think it's reasonable to assume that Ms. U. S. Author didn't think
about the implications of tuples or dictionary views or iterators or
whatever.  I personally think a warning here is a *good* thing: if
Ms. Author didn't intend those usages, why should she have to check
them?  If somebody else does the checking and believes the function is
properly prepared for those usages, they should prepare an RFE (*not*
a bug report! -- Ms. Author will have to recheck that the code works
as claimed in the more general context, and perhaps fix bugs to ensure
that it does work to *her* standards for code *she* distributes).


From nicholas.cole at gmail.com  Fri Aug 15 07:46:15 2014
From: nicholas.cole at gmail.com (Nicholas Cole)
Date: Fri, 15 Aug 2014 06:46:15 +0100
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53ECF1DD.8040808@stoneleaf.us>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <53ECF1DD.8040808@stoneleaf.us>
Message-ID: <CAAu18heOXeJzHJ8T7eHHzGXJmAODvRPJc3a6LOdhtbNW=8EU9g@mail.gmail.com>

On Thu, Aug 14, 2014 at 6:29 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 08/14/2014 09:01 AM, Sunjay Varma wrote:
>>
>>
>> Additionally, this approach can be used by documentation generators as
>> well and removes any duplication from the
>> function declaration and the docstring.
>>
>> Here's a taste of what that looks like:
>>      class SimpleEquation(object):
>>          def demo(self, a, b, c):
>>              """
>>              This function returns the product of a, b and c
>>              @type self: SimpleEquation
>>              :param a: int - The first number
>>              :param b: int
>>              :param c: int - The third number should not be zero and
>> should also
>>                  only be -1 if you enjoy carrots (this comment spans 2
>> lines)
>>              :return: int
>>              """
>>              return a * b * c
>
>
>
> +1  I like this much more.

+1 from me as well.

I like this much, much more as well.  It is simply far more readable
and easier on the eyes, especially for functions with complicated
definitions.  This feels like something I would actually use, without
it being a burden.

I think it could also work very nicely for keyword arguments.

N.

From abarnert at yahoo.com  Fri Aug 15 08:08:15 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 14 Aug 2014 23:08:15 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
 <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com>
 <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <1408082895.44451.YahooMailNeo@web181003.mail.ne1.yahoo.com>

On Thursday, August 14, 2014 10:46 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:

?
> I suggest instead
> 
> ? ? a static type check that generates [scads of|tons of|way too many]
> ? ? [spurious|bogus|annoying] warnings when used that way.
> 
> which is IMO a genuine concern but also probably has technological
> fixes like the one Andrew proposes (automatically promoting many
> container types to the ABC actually needed).
> 
> In some sense I'm not really bothered by this (I'd be willing to omit
> all the bracketed phrases :-).? If an upstream author writes
> 
> ? ? def somefilter(l_of_s: List[str]) -> List[int]:
> ? ? ? ? pass
> 
> I think it's reasonable to assume that Ms. U. S. Author didn't think
> about the implications of tuples or dictionary views or iterators or
> whatever.? I personally think a warning here is a *good* thing: if
> Ms. Author didn't intend those usages, why should she have to check
> them?

You're forgetting that linting is not the only purpose of static type annotations in this proposal.

If you write what should be perfectly valid code and it spits out scads of warnings when linted, yes, you can quickly figure out that it's a bug in the library and report it upstream.

But if, say, your IDE doesn't suggest a function that it could have, you may never notice the problem, and just find the library to be less fun to use than you expected.

From abarnert at yahoo.com  Fri Aug 15 08:28:11 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 14 Aug 2014 23:28:11 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJLcsqVPYh2a6VJgci=t3rp9-pMFBV4xHdERbeemNBxZ8A@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
 <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com>
 <CAP7+vJLcsqVPYh2a6VJgci=t3rp9-pMFBV4xHdERbeemNBxZ8A@mail.gmail.com>
Message-ID: <1408084091.95412.YahooMailNeo@web181006.mail.ne1.yahoo.com>

On Thursday, August 14, 2014 9:15 PM, Guido van Rossum <guido at python.org> wrote:

>On Thu, Aug 14, 2014 at 8:51 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote:
>
>>Most of the code in typing.py is a duplication of the ABCs in the collections.abc module (and io and maybe others). I understand that MyPy couldn't monkeypatch that code into the stdlib, so it had to fork the contents, but if this is going into the stdlib, can't we just modify collections/abc.py instead? Why not have the type information on collections.abc.Sequence be introspectable at runtime? More importantly, why not have the interface defined by collections.abc.Sequence be _exactly_ the same as the one checked by the static type checker, instead of just very similar?


[snip]

>I think I already responded to a similar proposal. I think modifying the existing ABCs is fine for Python 3.5 and beyond, but I think I'd like to keep aliasing in the typing module to make it easier to use annotations in earlier Python versions (using a typing.py installed from PyPI).

That sounds like a great idea. My worry wasn't the extra few KB of code sitting on my computer, it was the extra set of not-quite-the-same parallel concepts that are in typing.py today. If it instead has exactly the same classes as collections.abc, io, etc., everything's great.

>I feel similar about using a|b as a concise way to spell Union[a, n], and int|None would be quite decent as a way to spell optional int.


Swift makes you sound like a 12-year-old girl? who makes every phrase? sound like a question? by writing `int?` everywhere? Haskell maybe makes you sound wishy-washy and maybe a little uncertain by having you say `Maybe int`. ?Python lets you declare, `int, or None` like Patrick Henry. Sounds good.

From lukasz at langa.pl  Fri Aug 15 09:48:12 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 15 Aug 2014 00:48:12 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
Message-ID: <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>

On Aug 14, 2014, at 4:56 PM, Guido van Rossum <guido at python.org> wrote:

> There's also a variant of (1) that ?ukasz Langa would like to see -- use the syntactic position of function annotations but using a custom syntax (e.g. one similar to the pytypedecl syntax) that isn't evaluated at function-definition time. This would have to use "from __future__ import <something>" for backward compatibility. I'm skeptical about this though; it is only slightly more elegant than mypy, and it would open the floodgates of unconstrained language design.

I see the decision has been made. For the curious, the design would be as close as possible to PEP 3107. The biggest wins would be first-class annotations for variables (supported in Mypy as comments) and support for forward-references without the need to fall back to strings.

I?m also not a fan of the square brackets for generics (those brackets mean lookup!) but a BDFL once said that ?language evolution is the art of compromise? and one cannot disagree with that.


> So there you have it. I am picking the mypy family and I hope we can start focusing on specific improvements to mypy.

Alright! That sounds good. With the syntax mostly out of the way, the next issue to handle is the typing module. dict, MutableMapping, and now Dict? One step too far. We should be able to re-use ABCs for that, e.g. to add support for union types and generics. Lots of decisions ahead (covariance, casting, multiple dispatch, etc.) but we?ll get there.

The typing module as aliases for the updated ABCs sounds like a fair compromise, although I?m worried about the details of that approach (for starters: we need to bundle the new ABCMeta to support union types but having both implementations at runtime feels very wrong). Moreover, with dynamic type registration and duck typing isinstance(), there will be challenges to cover. I?m happy to go and slay that dragon.

As a side note, I?m happy you?re willing to agree on str | None. This reads really well and is concise enough to not require aliasing to be usable.

While we?re at slaying dragons, I?ll also silently make str non-iterable so that we can use Sequence[str] meaningfully from now on? How about that?


-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/fc869c83/attachment-0001.html>

From stephen at xemacs.org  Fri Aug 15 11:11:50 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 15 Aug 2014 18:11:50 +0900
Subject: [Python-ideas] Proposal: Use mypy syntax for function
 annotations
In-Reply-To: <1408082895.44451.YahooMailNeo@web181003.mail.ne1.yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
 <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com>
 <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp>
 <1408082895.44451.YahooMailNeo@web181003.mail.ne1.yahoo.com>
Message-ID: <87k36aglrd.fsf@uwakimon.sk.tsukuba.ac.jp>

Andrew Barnert writes:
 > On Thursday, August 14, 2014 10:46 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:

 > > I think it's reasonable to assume that Ms. U. S. Author didn't think
 > > about the implications of tuples or dictionary views or iterators or
 > > whatever.? I personally think a warning here is a *good* thing: if
 > > Ms. Author didn't intend those usages, why should she have to check
 > > them?
 > 
 > You're forgetting that linting is not the only purpose of static
 > type annotations in this proposal.

Hardly.

 > But if, say, your IDE doesn't suggest a function that it could
 > have, you may never notice the problem, and just find the library
 > to be less fun to use than you expected.

But who decides it *should* be perfectly valid code?  *Somebody* has
to check that, and if code is used naively in a case where it
shouldn't be (cf the "why doesn't sum() handle iter_of_str" thread!),
the user is screwed again.

Have you never written code that was perfectly sound for your purpose,
but could be used in other contexts according to ducktyping?  Are you
*sure* that it is valid in those other contexts?  (Think sum vs.
math.fsum -- accuracy matters.)

Of course it's a matter of balance, and *maybe* it tips more in the
direction of "default to the most general type where the code will run
without crashing or raising".  I'm just saying it's not obvious to me
that it's such a bad thing that one-off piece of junk libraries may
not be as re-abusable if you use a type checker as they are when you
don't.

From tjreedy at udel.edu  Fri Aug 15 11:38:03 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 15 Aug 2014 05:38:03 -0400
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
Message-ID: <lskkem$8m7$1@ger.gmane.org>

On 8/15/2014 12:40 AM, Nick Coghlan wrote:
> On 15 August 2014 09:56, Guido van Rossum <guido at python.org> wrote:
>>
>> I don't buy the argument that PEP 3107 promises that annotations are
>> completely free of inherent semantics.
>
> It's also worth noting the corresponding bullet point in PEP 3100
> (under http://www.python.org/dev/peps/pep-3100/#core-language):
>
> * Add optional declarations for static typing [45] [10] [done]
...
> Linters/checkers may also want to provide a configurable way to say
> "the presence of decorator <X> means the annotations on that function
> aren't type markers". That ties in with the recommendation we added to
> PEP 8 a while back: "It is recommended that third party experiments
> with annotations use an associated decorator to indicate how the
> annotation should be interpreted."

Depending on the checker, this suggests that non-type-check annotations 
need not be deprecated. If a decorator wraps a function with an 
unannotated wrapper, then the checker should see the result as 
unannotated, rather than looking for a wrapped attribute. Also, a 
decorator can remove non-type annotations and act on them, store them in 
a closure variable, or store them on the function in a different name. 
For example.

 >>> def doodad(f):
	f.doodad = f.__annotations__
	f.__annotations__ = {}
	return f

 >>> @doodad
def f(x:'arg doodad')->'return:doodad':
     pass

 >>> f.__annotations__
{}
 >>> f.doodad
{'x': 'arg doodad', 'return': 'return:doodad'}

Given these possibilities, all that is needs be said is "After a 
function is post-processed by decorators, any remaining annotations 
should be for type-checking or documentation."

For checkers that do look at the source, or the AST before compiling, 
the rule could be to ignore string annotations. Decorators can always 
eval, or perhaps safe_eval, strings.

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Fri Aug 15 11:48:10 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 15 Aug 2014 19:48:10 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <lskkem$8m7$1@ger.gmane.org>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
 <lskkem$8m7$1@ger.gmane.org>
Message-ID: <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>

On 15 August 2014 19:38, Terry Reedy <tjreedy at udel.edu> wrote:
> On 8/15/2014 12:40 AM, Nick Coghlan wrote:
>>
>> On 15 August 2014 09:56, Guido van Rossum <guido at python.org> wrote:
>>>
>>>
>>> I don't buy the argument that PEP 3107 promises that annotations are
>>> completely free of inherent semantics.
>>
>>
>> It's also worth noting the corresponding bullet point in PEP 3100
>> (under http://www.python.org/dev/peps/pep-3100/#core-language):
>>
>> * Add optional declarations for static typing [45] [10] [done]
>
> ...
>
>> Linters/checkers may also want to provide a configurable way to say
>> "the presence of decorator <X> means the annotations on that function
>> aren't type markers". That ties in with the recommendation we added to
>> PEP 8 a while back: "It is recommended that third party experiments
>> with annotations use an associated decorator to indicate how the
>> annotation should be interpreted."
>
>
> Depending on the checker, this suggests that non-type-check annotations need
> not be deprecated. If a decorator wraps a function with an unannotated
> wrapper, then the checker should see the result as unannotated, rather than
> looking for a wrapped attribute. Also, a decorator can remove non-type
> annotations and act on them, store them in a closure variable, or store them
> on the function in a different name.

No, many (most?) linters and IDEs will run off the AST without
actually executing the code, so they'll see the annotations, even if
they get stripped by the decorator at runtime.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From brakhane at googlemail.com  Fri Aug 15 12:02:21 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Fri, 15 Aug 2014 12:02:21 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <87k36aglrd.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
 <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com>
 <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp>
 <1408082895.44451.YahooMailNeo@web181003.mail.ne1.yahoo.com>
 <87k36aglrd.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <53EDDAAD.8040907@gmail.com>

Am 15.08.2014 11:11, schrieb Stephen J. Turnbull:
> Have you never written code that was perfectly sound for your purpose,
> but could be used in other contexts according to ducktyping?  Are you
> *sure* that it is valid in those other contexts?  (Think sum vs.
> math.fsum -- accuracy matters.)
>
Well, if someone uses my frobnicate method that I intended to be only
usable by lists, but
he uses it with generators, it's his responsibility to write a unit test
for his code (which he
should do anyway). Therefore he's responsible for guaranteeing that his
code continues to work,
and a linter should not tell him "you can't do that, the author hasn't
thought of that". At least
I should be able to silence the warning *once* and for all. Furthermore,
I want my IDE to suggest me the frobnicate
method even if I'm working with generators, so there should be a way for
me to tell the linter and IDE
"frobnicate takes an iterable, ignore the silly original annotation"

Unit testing your own code is the only way you can "guarantee" that your
code runs correctly.

After all, even if you use my frobnicate method only with lists, you
will still have to write a test, maybe I
introduce a subtle bug in the new version of my library.

From ndbecker2 at gmail.com  Fri Aug 15 12:56:46 2014
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 15 Aug 2014 06:56:46 -0400
Subject: [Python-ideas] generic code and dependent types
Message-ID: <lskp1f$vli$1@ger.gmane.org>

I'm interested in the proposals for adding type annotation.  Coming from some 
experience with generic code in c++, one of the difficult issues has been 
reasoning about dependent types in generic code.

In general, we need to be able to declare a type via an arbitrary metafunction

some_metafunction<T>::type F (...

It is also useful to be able to write something like:

typedef typeof (int() * float()) my_type;

I wonder if any of the proposals will be able to handle this sort of algebra on 
types?  I think it's needed to truly support generic code.


From bunslow at gmail.com  Fri Aug 15 13:01:23 2014
From: bunslow at gmail.com (Bill Winslow)
Date: Fri, 15 Aug 2014 06:01:23 -0500
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
 <lskkem$8m7$1@ger.gmane.org>
 <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>
Message-ID: <CAAWJOpYO+-+vpj29BBKvjbuJWFvK+trA53YYKQ02uChoo1fmXw@mail.gmail.com>

As a lurker who barely reads this list at all, let me just add here that
many of the typing-related questions that the BDFL presented as follow-up
are, in my opinion, questions worth asking even outside of the annotation
issue.

(Mini wall of text incoming:)

What I mean is, one of my grievances with Python is that the type hierarchy
is poorly defined and difficult to use, in that the "types" presented in
collections.abc in no way whatsoever interact with the builtins, and
considering how many library and user types inherit from those ABCs (as is
the purpose of those ABCs), it seems to me like a rather serious issue with
using the type system.

Consider the following:


>>> from collections import abc as types
>>> isinstance(dict, types.Mapping)
False
>>> isinstance(types.Mapping, dict)
False
>>> isinstance(list, types.MutableSequence)
False
>>> isinstance(types.MutableSequence, list)
False
>>> isinstance(list, types.Sized)
False


Furthermore:


>>> class DictThing(types.MutableMapping): pass #Easier to subclass
...
>>> isinstance(DictThing, dict)
False
>>> class DicterThing(dict): pass # Simpler
...
>>> isinstance(DicterThing, types.MutableMapping)
False


And finally:


>>> from collections import defaultdict
>>> isinstance(defaultdict, dict)
False
>>> isinstance(defaultdict, types.MutableMapping)
False


My conclusion is that to make lint-sort type-checkers worth their salt, the
Python type hierarchy needs to be fixed and properly integrated. There is
currently no obvious way to check if an object either is a list or behaves
like a list (the duck typing philosophy equates the two). (The current
shortest way is "isinstance(myvar, (list, types.MutableSequence)", but
that's not very obvious or Pythonic IMO.)

Before now I haven't bothered to put my thoughts to words, but this is
pretty decent motivation.

---------------------------------------------------------------------------

If people agree with my conclusion, then the ideal solution would be to
somehow merge the builtin types and the collections.abc types into one
single hierarchy, one single isinstance check, but that's probably
impossible.

The next thing would be to make one hierarchy the appropriate subclasses of
the other hierarchy, but either of those solutions has its own problems.
There're probably better solutions out there.



--Bill

On Fri, Aug 15, 2014 at 4:48 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 15 August 2014 19:38, Terry Reedy <tjreedy at udel.edu> wrote:
> > On 8/15/2014 12:40 AM, Nick Coghlan wrote:
> >>
> >> On 15 August 2014 09:56, Guido van Rossum <guido at python.org> wrote:
> >>>
> >>>
> >>> I don't buy the argument that PEP 3107 promises that annotations are
> >>> completely free of inherent semantics.
> >>
> >>
> >> It's also worth noting the corresponding bullet point in PEP 3100
> >> (under http://www.python.org/dev/peps/pep-3100/#core-language):
> >>
> >> * Add optional declarations for static typing [45] [10] [done]
> >
> > ...
> >
> >> Linters/checkers may also want to provide a configurable way to say
> >> "the presence of decorator <X> means the annotations on that function
> >> aren't type markers". That ties in with the recommendation we added to
> >> PEP 8 a while back: "It is recommended that third party experiments
> >> with annotations use an associated decorator to indicate how the
> >> annotation should be interpreted."
> >
> >
> > Depending on the checker, this suggests that non-type-check annotations
> need
> > not be deprecated. If a decorator wraps a function with an unannotated
> > wrapper, then the checker should see the result as unannotated, rather
> than
> > looking for a wrapped attribute. Also, a decorator can remove non-type
> > annotations and act on them, store them in a closure variable, or store
> them
> > on the function in a different name.
>
> No, many (most?) linters and IDEs will run off the AST without
> actually executing the code, so they'll see the annotations, even if
> they get stripped by the decorator at runtime.
>
> Cheers,
> Nick.
>
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/4ada1de6/attachment.html>

From ncoghlan at gmail.com  Fri Aug 15 13:50:31 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 15 Aug 2014 21:50:31 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAAWJOpYO+-+vpj29BBKvjbuJWFvK+trA53YYKQ02uChoo1fmXw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
 <lskkem$8m7$1@ger.gmane.org>
 <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>
 <CAAWJOpYO+-+vpj29BBKvjbuJWFvK+trA53YYKQ02uChoo1fmXw@mail.gmail.com>
Message-ID: <CADiSq7cm9iw_+AFoBR81uM6frnb4ceiVRsT1ecVFS9pSBo98PA@mail.gmail.com>

On 15 August 2014 21:01, Bill Winslow <bunslow at gmail.com> wrote:
> Consider the following:
>
>
>>>> from collections import abc as types
>>>> isinstance(dict, types.Mapping)
> False
>>>> isinstance(types.Mapping, dict)
> False
>>>> isinstance(list, types.MutableSequence)
> False
>>>> isinstance(types.MutableSequence, list)
> False
>>>> isinstance(list, types.Sized)
> False

You're doing instance checks on subclasses - that's never going to
work. Once you account for the type/instance distinction, you can see
everything is correctly registered:

>>> from collections import abc as cabc
>>> issubclass(dict, cabc.Mapping)
True
>>> issubclass(list, cabc.Sequence)
True
>>> issubclass(list, cabc.Sized)
True
>>> isinstance(dict(), cabc.Mapping)
True
>>> isinstance(list(), cabc.Sequence)
True
>>> isinstance(list(), cabc.Sized)
True

(in Python 2, a couple of builtins claim ABCs they don't actually
implement fully, but that's addressed in newer versions of Python 3)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From bunslow at gmail.com  Fri Aug 15 13:53:31 2014
From: bunslow at gmail.com (Bill Winslow)
Date: Fri, 15 Aug 2014 06:53:31 -0500
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CADiSq7cm9iw_+AFoBR81uM6frnb4ceiVRsT1ecVFS9pSBo98PA@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
 <lskkem$8m7$1@ger.gmane.org>
 <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>
 <CAAWJOpYO+-+vpj29BBKvjbuJWFvK+trA53YYKQ02uChoo1fmXw@mail.gmail.com>
 <CADiSq7cm9iw_+AFoBR81uM6frnb4ceiVRsT1ecVFS9pSBo98PA@mail.gmail.com>
Message-ID: <CAAWJOpYZTXkcvNOFngBiDp5PtcWAuZjKFStdE4vaD6bkRCg=pw@mail.gmail.com>

Oh dear lord, it's like a nightmare where you're late to class without
clothing... except it actually happened... I knew something like this would
happen when I decided to actually do something.




On Fri, Aug 15, 2014 at 6:50 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 15 August 2014 21:01, Bill Winslow <bunslow at gmail.com> wrote:
> > Consider the following:
> >
> >
> >>>> from collections import abc as types
> >>>> isinstance(dict, types.Mapping)
> > False
> >>>> isinstance(types.Mapping, dict)
> > False
> >>>> isinstance(list, types.MutableSequence)
> > False
> >>>> isinstance(types.MutableSequence, list)
> > False
> >>>> isinstance(list, types.Sized)
> > False
>
> You're doing instance checks on subclasses - that's never going to
> work. Once you account for the type/instance distinction, you can see
> everything is correctly registered:
>
> >>> from collections import abc as cabc
> >>> issubclass(dict, cabc.Mapping)
> True
> >>> issubclass(list, cabc.Sequence)
> True
> >>> issubclass(list, cabc.Sized)
> True
> >>> isinstance(dict(), cabc.Mapping)
> True
> >>> isinstance(list(), cabc.Sequence)
> True
> >>> isinstance(list(), cabc.Sized)
> True
>
> (in Python 2, a couple of builtins claim ABCs they don't actually
> implement fully, but that's addressed in newer versions of Python 3)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/386bb0d7/attachment.html>

From ncoghlan at gmail.com  Fri Aug 15 14:01:30 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 15 Aug 2014 22:01:30 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAAWJOpYZTXkcvNOFngBiDp5PtcWAuZjKFStdE4vaD6bkRCg=pw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
 <lskkem$8m7$1@ger.gmane.org>
 <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>
 <CAAWJOpYO+-+vpj29BBKvjbuJWFvK+trA53YYKQ02uChoo1fmXw@mail.gmail.com>
 <CADiSq7cm9iw_+AFoBR81uM6frnb4ceiVRsT1ecVFS9pSBo98PA@mail.gmail.com>
 <CAAWJOpYZTXkcvNOFngBiDp5PtcWAuZjKFStdE4vaD6bkRCg=pw@mail.gmail.com>
Message-ID: <CADiSq7e7Pyoj7W_t5MZ0NY_iaWEhAFdiVD6mqHG+KgvmdPnF-w@mail.gmail.com>

On 15 August 2014 21:53, Bill Winslow <bunslow at gmail.com> wrote:
> Oh dear lord, it's like a nightmare where you're late to class without
> clothing... except it actually happened... I knew something like this would
> happen when I decided to actually do something.

If it helps any, I had to type it into the interactive interpreter and
get very confused for a moment before I realised what had happened.
And I *knew* they were integrated, because I'd helped fix some of the
bugs with the ABC non-conformance :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From steve at pearwood.info  Fri Aug 15 14:26:09 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 15 Aug 2014 22:26:09 +1000
Subject: [Python-ideas] generic code and dependent types
In-Reply-To: <lskp1f$vli$1@ger.gmane.org>
References: <lskp1f$vli$1@ger.gmane.org>
Message-ID: <20140815122608.GZ4525@ando>

On Fri, Aug 15, 2014 at 06:56:46AM -0400, Neal Becker wrote:
> I'm interested in the proposals for adding type annotation.  Coming from some 
> experience with generic code in c++, one of the difficult issues has been 
> reasoning about dependent types in generic code.
> 
> In general, we need to be able to declare a type via an arbitrary metafunction
> 
> some_metafunction<T>::type F (...
> 
> It is also useful to be able to write something like:
> 
> typedef typeof (int() * float()) my_type;
> 
> I wonder if any of the proposals will be able to handle this sort of algebra on 
> types?  I think it's needed to truly support generic code.


Let's pretend that this is a Python mailing list, and that some readers 
aren't familiar with C++ generic syntax :-) What does 

typedef typeof (int() * float()) my_type;

do? From a Python perspecive, that looks like type(0*0.0) which will 
return float. Presumably that's not what you want, or you would have 
just said "float". So what am I missing?

(Apart from years of C++ experience.)


-- 
Steven

From tinchester at gmail.com  Fri Aug 15 14:32:04 2014
From: tinchester at gmail.com (=?UTF-8?Q?Tin_Tvrtkovi=C4=87?=)
Date: Fri, 15 Aug 2014 14:32:04 +0200
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 31
In-Reply-To: <mailman.72122.1407959084.18129.python-ideas@python.org>
References: <mailman.72122.1407959084.18129.python-ideas@python.org>
Message-ID: <CAEJ-mUrEcEzsiiU+v9mMMMnuGBa3QeAPOLjmODowMYfbJ9Hbsw@mail.gmail.com>

Hi,

I realize I'm very late to the party, but just in the interest of
completeness I thought I'd mention Typed Clojure (http://typedclojure.org/),
which is an effort to add optional typing to Clojure. It has even had a
successful crowdfunding campaign:
https://www.indiegogo.com/projects/typed-clojure. I haven't had an
opportunity of using it yet so this is all I know.

P.S. Very excited about this effort!


> Date: Wed, 13 Aug 2014 12:44:21 -0700
> From: Guido van Rossum <guido at python.org>
> To: Python-Ideas <python-ideas at python.org>
> Cc: Jukka Lehtosalo <jlehtosalo at gmail.com>, Bob Ippolito
>         <bob at redivi.com>
> Subject: [Python-ideas] Proposal: Use mypy syntax for function
>         annotations
> Message-ID:
>         <
> CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> [There is no TL;DR other than the subject line. Please read the whole thing
> before replying. I do have an appendix with some motivations for adding
> type annotations at the end.]
>
> Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man
> of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy:
> http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python
> can
> learn from Haskell (and other languages); yesterday he gave the same talk
> at Dropbox. The talk is online (
> https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad
> strokes comes down to three suggestions:
>
>   (a) Python should adopt mypy's syntax for function annotations
>   (b) Python's use of mutabe containers by default is wrong
>   (c) Python should adopt some kind of Abstract Data Types
>
> Proposals (b) and (c) don't feel particularly actionable (if you disagree
> please start a new thread, I'd be happy to discuss these further if there's
> interest) but proposal (a) feels right to me.
>
> So what is mypy?  It is a static type checker for Python written by Jukka
> for his Ph.D. thesis. The basic idea is that you add type annotations to
> your program using some custom syntax, and when running your program using
> the mypy interpreter, type errors will be found during compilation (i.e.,
> before the program starts running).
>
> The clever thing here is that the custom syntax is actually valid Python 3,
> using (mostly) function annotations: your annotated program will still run
> with the regular Python 3 interpreter. In the latter case there will be no
> type checking, and no runtime overhead, except to evaluate the function
> annotations (which are evaluated at function definition time but don't have
> any effect when the function is called).
>
> In fact, it is probably more useful to think of mypy as a heavy-duty linter
> than as a compiler or interpreter; leave the type checking to mypy, and the
> execution to Python. It is easy to integrate mypy into a continuous
> integration setup, for example.
>
> To read up on mypy's annotation syntax, please see the mypy-lang.org
> website. Here's just one complete example, to give a flavor:
>
>   from typing import List, Dict
>
>   def word_count(input: List[str]) -> Dict[str, int]:
>       result = {}  #type: Dict[str, int]
>       for line in input:
>           for word in line.split():
>               result[word] = result.get(word, 0) + 1
>       return result
>
> Note that the #type: comment is part of the mypy syntax; mypy uses comments
> to declare types in situations where no syntax is available -- although
> this particular line could also be written as follows:
>
>     result = Dict[str, int]()
>
> Either way the entire function is syntactically valid Python 3, and a
> suitable implementation of typing.py (containing class definitions for List
> and Dict, for example) can be written to make the program run correctly.
> One is provided as part of the mypy project.
>
> I should add that many of mypy's syntactic choices aren't actually new. The
> basis of many of its ideas go back at least a decade: I blogged about this
> topic in 2004 (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 --
> see also the two followup posts linked from the top there).
>
> I'll emphasize once more that mypy's type checking happens in a separate
> pass: no type checking happens at run time (other than what the interpreter
> already does, like raising TypeError on expressions like 1+"1").
>
> There's a lot to this proposal, but I think it's possible to get a PEP
> written, accepted and implemented in time for Python 3.5, if people are
> supportive. I'll go briefly over some of the action items.
>
> *(1) A change of direction for function annotations*
>
> PEP 3107 <http://legacy.python.org/dev/peps/pep-3107/>, which introduced
> function annotations, is intentional non-committal about how function
> annotations should be used. It lists a number of use cases, including but
> not limited to type checking. It also mentions some rejected proposals that
> would have standardized either a syntax for indicating types and/or a way
> for multiple frameworks to attach different annotations to the same
> function. AFAIK in practice there is little use of function annotations in
> mainstream code, and I propose a conscious change of course here by stating
> that annotations should be used to indicate types and to propose a standard
> notation for them.
>
> (We may have to have some backwards compatibility provision to avoid
> breaking code that currently uses annotations for some other purpose.
> Fortunately the only issue, at least initially, will be that when running
> mypy to type check such code it will produce complaints about the
> annotations; it will not affect how such code is executed by the Python
> interpreter. Nevertheless, it would be good to deprecate such alternative
> uses of annotations.)
>
> *(2) A specification for what to add to Python 3.5*
>
> There needs to be at least a rough consensus on the syntax for annotations,
> and the syntax must cover a large enough set of use cases to be useful.
> Mypy is still under development, and some of its features are still
> evolving (e.g. unions were only added a few weeks ago). It would be
> possible to argue endlessly about details of the notation, e.g. whether to
> use 'list' or 'List', what either of those means (is a duck-typed list-like
> type acceptable?) or how to declare and use type variables, and what to do
> with functions that have no annotations at all (mypy currently skips those
> completely).
>
> I am proposing that we adopt whatever mypy uses here, keeping discussion of
> the details (mostly) out of the PEP. The goal is to make it possible to add
> type checking annotations to 3rd party modules (and even to the stdlib)
> while allowing unaltered execution of the program by the (unmodified)
> Python 3.5 interpreter. The actual type checker will not be integrated with
> the Python interpreter, and it will not be checked into the CPython
> repository. The only thing that needs to be added to the stdlib is a copy
> of mypy's typing.py module. This module defines several dozen new classes
> (and a few decorators and other helpers) that can be used in expressing
> argument types. If you want to type-check your code you have to download
> and install mypy and run it separately.
>
> The curious thing here is that while standardizing a syntax for type
> annotations, we technically still won't be adopting standard rules for type
> checking. This is intentional. First of all, fully specifying all the type
> checking rules would make for a really long and boring PEP (a much better
> specification would probably be the mypy source code). Second, I think it's
> fine if the type checking algorithm evolves over time, or if variations
> emerge. The worst that can happen is that you consider your code correct
> but mypy disagrees; your code will still run.
>
> That said, I don't want to *completely* leave out any specification. I want
> the contents of the typing.py module to be specified in the PEP, so that it
> can be used with confidence. But whether mypy will complain about your
> particular form of duck typing doesn't have to be specified by the PEP.
> Perhaps as mypy evolves it will take options to tell it how to handle
> certain edge cases. Forks of mypy (or entirely different implementations of
> type checking based on the same annotation syntax) are also a possibility.
> Maybe in the distant future a version of Python will take a different
> stance, once we have more experience with how this works out in practice,
> but for Python 3.5 I want to restrict the scope of the upheaval.
>
>
> *Appendix -- Why Add Type Annotations?*
> The argument between proponents of static typing and dynamic typing has
> been going on for many decades. Neither side is all wrong or all right.
> Python has traditionally fallen in the camp of extremely dynamic typing,
> and this has worked well for most users, but there are definitely some
> areas where adding type annotations would help.
>
> - Editors (IDEs) can benefit from type annotations; they can call out
> obvious mistakes (like misspelled method names or inapplicable operations)
> and suggest possible method names. Anyone who has used IntelliJ or Xcode
> will recognize how powerful these features are, and type annotations will
> make such features more useful when editing Python source code.
>
> - Linters are an important tool for teams developing software. A linter
> doesn't replace a unittest, but can find certain types of errors better or
> quicker. The kind of type checking offered by mypy works much like a
> linter, and has similar benefits; but it can find problems that are beyond
> the capabilities of most linters.
>
> - Type annotations are useful for the human reader as well! Take the above
> word_count() example. How long would it have taken you to figure out the
> types of the argument and return value without annotations? Currently most
> people put the types in their docstrings; developing a standard notation
> for type annotations will reduce the amount of documentation that needs to
> be written, and running the type checker might find bugs in the
> documentation, too. Once a standard type annotation syntax is introduced,
> it should be simple to add support for this notation to documentation
> generators like Sphinx.
>
> - Refactoring. Bob's talk has a convincing example of how type annotations
> help in (manually) refactoring code. I also expect that certain automatic
> refactorings will benefit from type annotations -- imagine a tool like 2to3
> (but used for some other transformation) augmented by type annotations, so
> it will know whether e.g. x.keys() is referring to the keys of a dictionary
> or not.
>
> - Optimizers. I believe this is actually the least important application,
> certainly initially. Optimizers like PyPy or Pyston
> <https://github.com/dropbox/pyston> wouldn't be able to fully trust the
> type annotations, and they are better off using their current strategy of
> optimizing code based on the types actually observed at run time. But it's
> certainly feasible to imagine a future optimizer also taking type
> annotations into account.
>
> --
> --Guido "I need a new hobby" van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/29c08cbe/attachment-0001.html>

From brakhane at googlemail.com  Fri Aug 15 14:37:45 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Fri, 15 Aug 2014 14:37:45 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
Message-ID: <53EDFF19.7060305@gmail.com>

Am 15.08.2014 01:56, schrieb Guido van Rossum:

> PS. I realize I didn't discuss question (C) much. That's intentional
-- we can now start discussing specific mypy features in separate
threads (or in this one :-).

So is this the place to discuss "the thornier issues brought up against
mypy"? Because I think it's important that we get an idea of what we
want mypy to be able to do and what
not. After all, mypy will probably end up as a kind of reference
implementation for static type checkers. And I'm worried there might be
real damage to Python as a language
if they aren't thought through:

> Many people have shown either support for the idea, or pointed to some
other system that addresses the same issue. On the other hand, several
people have claimed
> that they don't need it, or that they worry it will make Python less
useful for them. (However, many of the detractors seem to have their own
alternative proposal. :-)
> In the end I don't think we can ever know for sure -- but my intuition
tells me that as long as we keep it optional, there is a real demand.

It won't be optional for programmers who work in a corporate environment
where mypy happens to be required. Those teams will probably also use
third party libraries,
and they might want them to be "type safe" as well and file RFEs for it;
in a few years, we might end up with a situation where "serious" code is
expected to provide static type info.

Even if mypy is "just a linter", a linter is supposed to find bugs and
promote best practices. And if the Python reference doc somehow named
mypy as an example for a static
type checker, it will be probably seen as enforcing Python best practices.

I really think there's a good chance/risk that mypy will change how
Python programs are written in the future. For example, people wouldn't
probably
call the word_count method with a file object and write a test case,
instead, they will read the file into a list and call it instead. After
all, the linter would complain otherwise.

IMO, the question is how much "staticness" we want to encourage. If we
want a really useful and flexible static type checking system, we would
need a very complex type system.
If we'd go that route, I fear that one of Pythons main features, its
dynamic nature will be seen by new programmers as kind of "deprecated
legacy", and turning Python into some
poor-man's-Scala.

If we take the current mypy approach, the type system would probably end
up a lot like Java's, useful for simple cases, and useless/a PITA otherwise.

And if we make the system minimal by design, there's a good chance that
it will be completely useless for static type checking for all but the
most trivial cases, helping no one.

I'm not sure which one I prefer, although I'm leaning into the minimal
to Java level direction; that way, the type system might be "just bad
enough" for people to see when
dynamic typing is an advantage and using that instead.


One thing where I do have a clear opinion is that it should/must be
possible to override (bad) type annotations.
Providing a seperate file seems ok to me. But the open problem with this
approach is

1) how to tell IDEs and linters which overrides are in effect when and
where (there might be different override for different modules)

2)how we should handle different versions of libraries

As an example for 2), let's say I'm the author of the frobnicate
library, which depends on spammatron 0.3 from pip.

Let's also say that - because it's a good idea - mypy will check
override modules for consistency: An override cannot declare a
completely different signature than the original.

Spammatro's author declared "def foo(x: float) -> float", but actually,
it should have been "Number" instead of float, as I'm using Decimal. So
I define a override module to fix it.

Now spammatron 0.3.1 is released, and foo has gained an additional
optional parameter: "def foo(x: float, y: float = 0) -> float". So I
have to update my override module, but then my
library isn't compatbile with 0.3 anymore; or at least will give linter
errors.

Providing both overrides would be very difficult; the static analyser
would have to know which versions of the library it is using.

I suppose it could look at the __version__ attribute, but what if it's
missing?


From brakhane at googlemail.com  Fri Aug 15 14:44:04 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Fri, 15 Aug 2014 14:44:04 +0200
Subject: [Python-ideas] generic code and dependent types
In-Reply-To: <20140815122608.GZ4525@ando>
References: <lskp1f$vli$1@ger.gmane.org> <20140815122608.GZ4525@ando>
Message-ID: <53EE0094.7080303@gmail.com>

Am 15.08.2014 14:26, schrieb Steven D'Aprano:
>
> Let's pretend that this is a Python mailing list, and that some readers 
> aren't familiar with C++ generic syntax :-) What does 
>
> typedef typeof (int() * float()) my_type;
>
> do? 

I think its "whatever type operator*(int a, float b)" returns.

Or more generally: "whatever type a suitable * operator returns after,
where suitable means after int and float might or might not have been
casted into another type so that a matching operator was found"

So, for example, if there were no operator*(int a, float b), it would
look if there's at operator*(int a, int b), and if not, operator*(float,
float) (or vice versa).

The "advantage" is that you do not need to change your code if the
meaning of multiplying an int with a float changes.

From brakhane at googlemail.com  Fri Aug 15 15:06:31 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Fri, 15 Aug 2014 15:06:31 +0200
Subject: [Python-ideas] generic code and dependent types
In-Reply-To: <lskp1f$vli$1@ger.gmane.org>
References: <lskp1f$vli$1@ger.gmane.org>
Message-ID: <53EE05D7.6010202@gmail.com>

Am 15.08.2014 12:56, schrieb Neal Becker:
> I wonder if any of the proposals will be able to handle this sort of algebra on 
> types?  I think it's needed to truly support generic code.
>
I hope it won't ;-)

I agree that it is needed for truly generic code, but if Python gains a
truly complete type system, then what's the point of having dynamic
typing at all? At this point,
you might as well program in Scala instead.

What's the point in trying to turn Python into a statically typed
language that can optionally be dynamically typed? There are enough good
static languages out there.

IMHO, the type information in Python should be considered part of the
documentation, if the type system gets too complex and can only be
parsed by the linter, then what's the point?

Is the information: "this function takes an Iterable of T and a
Map[U,Map[U,T]|T|int] and returns the type of the result of foo applied
to parameters of type T and U" really useful
for a programmer anymore?

Does it help me if the linter says that "Map[int,float]" is not
compatible with "T|Map[U,V subtype str]|Map[U subtype str, V]"? I think
if my unit test fails with a "TypeError: float has no method strip"
that's a much more useful information to me.

So I think limiting the power of the type system will actually be a good
thing.



From ndbecker2 at gmail.com  Fri Aug 15 15:10:51 2014
From: ndbecker2 at gmail.com (Neal Becker)
Date: Fri, 15 Aug 2014 09:10:51 -0400
Subject: [Python-ideas] generic code and dependent types
References: <lskp1f$vli$1@ger.gmane.org> <20140815122608.GZ4525@ando>
 <53EE0094.7080303@gmail.com>
Message-ID: <lsl0ss$3oi$1@ger.gmane.org>

Dennis Brakhane wrote:

> Am 15.08.2014 14:26, schrieb Steven D'Aprano:
>>
>> Let's pretend that this is a Python mailing list, and that some readers
>> aren't familiar with C++ generic syntax :-) What does
>>
>> typedef typeof (int() * float()) my_type;
>>
>> do?
> 
> I think its "whatever type operator*(int a, float b)" returns.
> 
> Or more generally: "whatever type a suitable * operator returns after,
> where suitable means after int and float might or might not have been
> casted into another type so that a matching operator was found"
> 
> So, for example, if there were no operator*(int a, float b), it would
> look if there's at operator*(int a, int b), and if not, operator*(float,
> float) (or vice versa).
> 
> The "advantage" is that you do not need to change your code if the
> meaning of multiplying an int with a float changes.

I really meant more like

typeof (A() * B())

where A and B are 2 types, and A() is a call to the default constructor for A.

For example

ret_t multiply (A a, B b): return a * b

what is ret_t?


From apalala at gmail.com  Fri Aug 15 15:12:43 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Fri, 15 Aug 2014 08:42:43 -0430
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAAWJOpYZTXkcvNOFngBiDp5PtcWAuZjKFStdE4vaD6bkRCg=pw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
 <lskkem$8m7$1@ger.gmane.org>
 <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>
 <CAAWJOpYO+-+vpj29BBKvjbuJWFvK+trA53YYKQ02uChoo1fmXw@mail.gmail.com>
 <CADiSq7cm9iw_+AFoBR81uM6frnb4ceiVRsT1ecVFS9pSBo98PA@mail.gmail.com>
 <CAAWJOpYZTXkcvNOFngBiDp5PtcWAuZjKFStdE4vaD6bkRCg=pw@mail.gmail.com>
Message-ID: <CAN1YFWv9gXdqH1BzDhtqRA3mTOx-DygKDK3VMMXcfM8PjjcB4A@mail.gmail.com>

On Fri, Aug 15, 2014 at 7:23 AM, Bill Winslow <bunslow at gmail.com> wrote:

> Oh dear lord, it's like a nightmare where you're late to class without
> clothing... except it actually happened... I knew something like this would
> happen when I decided to actually do something.


Don't worry. It's a too common happening, in one of the few places in which
Python has no Zen.

-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/7ad155d6/attachment-0001.html>

From steve at pearwood.info  Fri Aug 15 16:04:52 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 16 Aug 2014 00:04:52 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53EDDAAD.8040907@gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBCABC.7010801@python.org> <20140814190237.GV4525@ando>
 <1408074676.18275.YahooMailNeo@web181003.mail.ne1.yahoo.com>
 <87ppg2gva5.fsf@uwakimon.sk.tsukuba.ac.jp>
 <1408082895.44451.YahooMailNeo@web181003.mail.ne1.yahoo.com>
 <87k36aglrd.fsf@uwakimon.sk.tsukuba.ac.jp> <53EDDAAD.8040907@gmail.com>
Message-ID: <20140815140452.GB4525@ando>

On Fri, Aug 15, 2014 at 12:02:21PM +0200, Dennis Brakhane wrote:

> Unit testing your own code is the only way you can "guarantee" that your
> code runs correctly.

That is actually backwards. Unit testing cannot guarantee the code is 
correct. If you have 1000 tests, all that you have proved is that the 
code works for those 1000 cases. It tells you nothing about all the 
uncounted billions of other cases you haven't written tests for.

Given some set of tests which pass, you can be confident that your code 
gets at least those things correct (assuming the tests are themselves 
correct -- you write tests for your tests, don't you? *wink*). But 
that's all. As Dijkstra said, ?Program testing can be used to show the 
presence of bugs, but never to show their absence.?

To put it another way, there could be a million bugs hiding in all the 
corners of your code that you don't test, and you can never test 
*everything*.

Static typing is complementary to unit testing. Static typing can 
eliminate a lot of the unit tests that you otherwise would have to 
write. For example, you might have unit tests that try to demonstrate 
that given arguments of type float, the function will return a float; 
given Fraction arguments, the function returns a Fraction; and so on. A 
static type checker can *prove* that[1] instead of merely testing it 
with a few dozen examples and hoping the result generalises to all the 
billions of other floats and Fractions not tested. Static typing is 
actually a form of automated correctness testing.




[1] Assuming you trust that the checker itself is bug free.

-- 
Steven

From antoine at python.org  Fri Aug 15 16:30:24 2014
From: antoine at python.org (Antoine Pitrou)
Date: Fri, 15 Aug 2014 10:30:24 -0400
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
Message-ID: <lsl5i0$tu1$1@ger.gmane.org>

Le 15/08/2014 00:40, Nick Coghlan a ?crit :
>
> Having Argument Clinic generate appropriate annotations automatically
> could also be interesting.

That would be great actually. That's one of the things AC should 
eventually be able to bring to the table (thank you Larry :-)).
Being able to access the concrete implementation (without the boxing / 
unboxing wrapper) could also be quite useful for people who try to shave 
off some interpretation overhead :-)

Regards

Antoine.



From steve at pearwood.info  Fri Aug 15 16:38:22 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 16 Aug 2014 00:38:22 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53EDFF19.7060305@gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EDFF19.7060305@gmail.com>
Message-ID: <20140815143821.GC4525@ando>

On Fri, Aug 15, 2014 at 02:37:45PM +0200, Dennis Brakhane wrote:

> I really think there's a good chance/risk that mypy will change how
> Python programs are written in the future.

I certainly hope so. We're wasting our time if it doesn't. Why go 
through all the time and effort if nobody uses it?


> For example, people wouldn't probably call the word_count method with 
> a file object and write a test case, instead, they will read the file 
> into a list and call it instead. After all, the linter would complain 
> otherwise.

How do you know the linter will complain?

Many of the arguments against this proposal are based on the assumption 
that, given a type checker for Python, developers will suddenly abandon 
all the proven advantages of duck-typing and dynamic typing and rush to 
turn Python into a third-rate Java. I don't think this is a realistic 
fear. The mere fact we are having this argument proves that many Python 
developers will fight tooth and nail to keep using duck-typing and 
dynamic typing. If they do static type checks, they aren't going to give 
up those advantages. They'll check for Iterable, not list.

And those who don't? Do the same thing you would do *right now* when 
those authors write code like this:

    if type(argument) is not list:
        raise TypeError("list expected")


Report it as a bug, or request a feature enhancement. Patch the library. 
Use a different library. Or, *just don't use the linter*.


> IMO, the question is how much "staticness" we want to encourage. If we 
> want a really useful and flexible static type checking system, we 
> would need a very complex type system. If we'd go that route, I fear 
> that one of Pythons main features, its dynamic nature will be seen by 
> new programmers as kind of "deprecated legacy", and turning Python 
> into some poor-man's-Scala.

Static typing and dynamic typing are *not* opposites. The names are 
unfortunate, because they imply an opposition that doesn't necessarily 
exist. Both static and dynamic typing are attempts to solve certain 
problems in programming, and it is possible to do both at the same time.


[...]
> One thing where I do have a clear opinion is that it should/must be
> possible to override (bad) type annotations.

Why? Do you consider it a "must" to override functions that raise 
TypeError at run time?

py> len(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'NoneType' has no len()

So why do you consider it a "must" to be able to override functions that 
report type errors at compile time?

But for what it's worth, as this proposal has repeatedly said, the 
linter is optional. If you don't believe the linter, *just run the code* 
in Python like you have always done before.


-- 
Steven

From apalala at gmail.com  Fri Aug 15 16:39:01 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Fri, 15 Aug 2014 10:09:01 -0430
Subject: [Python-ideas] generic code and dependent types
In-Reply-To: <53EE05D7.6010202@gmail.com>
References: <lskp1f$vli$1@ger.gmane.org> <53EE05D7.6010202@gmail.com>
Message-ID: <CAN1YFWvNO=4-+u+xjdfc9TkYXrjOmzLpubHBe9dSNMiazW6gxg@mail.gmail.com>

On Fri, Aug 15, 2014 at 8:36 AM, Dennis Brakhane <brakhane at googlemail.com>
wrote:

> What's the point in trying to turn Python into a statically typed
> language that can optionally be dynamically typed? There are enough good
> static languages out there.
>

That's important.

Even within the standard lib, Python is a well behaved systems citizen, so
people are free to write parts of their software in any
language/environment, and Python will happily communicate with whatever.

Cheers,


-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/40fa1fd4/attachment.html>

From greg at krypto.org  Fri Aug 15 17:10:58 2014
From: greg at krypto.org (Gregory P. Smith)
Date: Fri, 15 Aug 2014 15:10:58 +0000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
Message-ID: <CAGE7PNJP2ASOuBTHaCpmrNesu45a_LFAUeaBq9Ta8LY_NHEKMw@mail.gmail.com>

On Thu Aug 14 2014 at 9:02:54 AM Sunjay Varma <varma.sunjay at gmail.com>
wrote:

> I am strongly opposed to this entire proposal. As Juancarlo points out,
> Python programs are small, but very understandable. I think this syntax
> detracts from that. I'll suggest an alternative further down in my reply.
>

Small? I've got tens of millions of lines of Python code to wrangle that
says otherwise. We're trying to create an analyzer and type inferencer so
that we can actually make sense of it all to make it easier to both (a)
maintain and (b) migrate to Python 3. :)


> One benefit of Python that makes it so attractive for new programmers and
> even old programmers alike is that you can usually pick out any piece of
> Python code and begin to understand it immediately. Even if you come from a
> different programming language, Python is written in
> english explicitly using words like "and" and "or". Those constructs, as
> opposed to "&&" or "||" make the language less scary for new developers and
> in general easier to read as well. It's also easier to type regular english
> words (no need to use the shift key). Using the annotation syntax this
> heavily will detract very much from the readability of Python and from the
> overall usability as well. Programs are read more times than they are
> written.
>
> Several years ago, before I had any programming experience in any language
> at all, I needed to edit some Python code to make something I was doing
> work. Without any experience at all, I was able to look through the (small)
> program I was editing and figure out exactly what I needed to adjust.
> Without Python being such a clean, almost English language, that would have
> been impossible.
>
> Though the annotation syntax is already present in Python 3, I would argue
> that using this for type annotations will get very messy very quickly. If
> I'm understanding the syntax correctly, writing any function using a large
> library with many nested subpackages could result in code like this:
>
>     import twisted.protocols.mice.mouseman
>
>     def
> process_mouseman(inputMouseMan: twisted.protocols.mice.mouseman.MouseMan)
> -> twisted.protocols.mice.mouseman.MouseMan:
>         pass
>
> That function definition is 122 characters long. Far more than what PEP8
> recommends. Though this example was crafted to illustrate my point (I don't
> think most people would really write code like this), it is easy to see
> that this kind of code is possible and may sometimes be written by some
> less experienced programmers. It demonstrates how messy things can get even
> with just one parameter.
>
> It is also easy to see that it is very difficult to parse out what is
> going on in that function. Adding type annotations inline makes it very
> difficult to quickly get an idea of what arguments a function takes and in
> what order. It detracts from the overall readability of a program and can
> also lead to very poorly formatted programs that break the guidelines in
> PEP8. Though I have only demonstrated this for function declarations, the
> example could also be extended to inline statement comments as well. Things
> get too messy too quickly.
>
> My Alternative Proposal:
> As an alternative, I would like to propose a syntax that Pycharm already
> supports:
> http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.html
>
> Since this type information isn't going to be used at runtime in the
> regular Python interpreter anyway, why not have it in the function
> docstring instead? This provides both readability and type checking.
> Standardizing that syntax or at least adding it as an optional way to check
> your program would in my opinion be a much better addition to the language.
> This approach needs no new syntax, keeps readability and allows the
> programmer to add additional documentation without going over the 80
> character limit.
>

Without commenting on the specific format of the docstring, there is an
added benefit to using docstrings for parameter and return value type
information: It encourages people to write documentation.

(the exact format of types in a docstring could turn into its own bikeshed
even grander than this thread already is)

JavaScript successfully uses this approach for type annotations. (sure, its
in comments, but that's because they don't _have_ docstrings).

In a way, using docstrings is similar to what argument clinic does for
extension modules. We could provide a standard way to do it and have the
language runtime parse them and turn them into actual annotation objects
when the __annotations__ attribute is first accessed on anything. If
someone really wanted to they could have it hide the information from the
docstring at the same time (I don't recommend that. Argument clinic is
"special" and had a real a need for this).

That laziness *mostly* avoids the forward referencing or forward
declaration mess that you'd otherwise have.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/5203b8f5/attachment-0001.html>

From guido at python.org  Fri Aug 15 17:17:33 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Aug 2014 08:17:33 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
Message-ID: <CAP7+vJLS_4Kr+7_SbdBXcEnjWVnLCvjm40FXMDLDC2d_DOaFAw@mail.gmail.com>

On Fri, Aug 15, 2014 at 12:48 AM, ?ukasz Langa <lukasz at langa.pl> wrote:

> On Aug 14, 2014, at 4:56 PM, Guido van Rossum <guido at python.org> wrote:
>
> There's also a variant of (1) that ?ukasz Langa would like to see -- use
> the syntactic position of function annotations but using a custom syntax
> (e.g. one similar to the pytypedecl syntax) that isn't evaluated at
> function-definition time. This would have to use "from __future__ import
> <something>" for backward compatibility. I'm skeptical about this though;
> it is only slightly more elegant than mypy, and it would open the
> floodgates of unconstrained language design.
>
>
> I see the decision has been made. For the curious, the design would be as
> close as possible to PEP 3107. The biggest wins would be first-class
> annotations for variables (supported in Mypy as comments) and support for
> forward-references without the need to fall back to strings.
>

You can probably come up with a notation for first-class variable
annotations, e.g.

    x: Sequence[int] = []

The value might be optional. The question is though, would the type
(Sequence[int]) be stored anyway? Also, in a class body, does it define a
class var or an instance var (or doesn't it matter?).

Does this need a 'var' keyword to be ambiguous?

I propose to disallow declaring multiple variables in this style, since
it's hard to decide whether the comma should bind tighter than the '=' sign
(as in assignments) or less tight (as in function headings).


>
> I?m also not a fan of the square brackets for generics (those brackets
> mean lookup!) but a BDFL once said that ?language evolution is the art of
> compromise? and one cannot disagree with that.
>
>
> So there you have it. I am picking the mypy family and I hope we can start
> focusing on specific improvements to mypy.
>
>
> Alright! That sounds good. With the syntax mostly out of the way, the next
> issue to handle is the typing module. dict, MutableMapping, and now Dict?
> One step too far. We should be able to re-use ABCs for that, e.g. to add
> support for union types and generics. Lots of decisions ahead (covariance,
> casting, multiple dispatch, etc.) but we?ll get there.
>
> The typing module as aliases for the updated ABCs sounds like a fair
> compromise, although I?m worried about the details of that approach (for
> starters: we need to bundle the new ABCMeta to support union types but
> having both implementations at runtime feels very wrong). Moreover, with
> dynamic type registration and duck typing isinstance(), there will be
> challenges to cover. I?m happy to go and slay that dragon.
>
> As a side note, I?m happy you?re willing to agree on str | None. This
> reads really well and is concise enough to not require aliasing to be
> usable.
>
> While we?re at slaying dragons, I?ll also silently make str non-iterable
> so that we can use Sequence[str] meaningfully from now on? How about that?
>

I hope you meant that as a joke. We missed our chance for that one with
Python 3.0. We must live with it.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/fa3f6258/attachment.html>

From mrocklin at gmail.com  Fri Aug 15 17:33:41 2014
From: mrocklin at gmail.com (Matthew Rocklin)
Date: Fri, 15 Aug 2014 08:33:41 -0700
Subject: [Python-ideas] RFC: Multiple Dispatch
Message-ID: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>

Hi all,

I curate the multipledispatch library on PyPI.  I believe that it is a
natural continuation of singledispatch outlined in PEP 443 and included in
functools 3.4.

For those unaware, dispatching correctly and unambiguously on multiple
inputs is a somewhat more complex problem than on a single input.  I
believe the approach taken in multipledispatch is fairly robust.

I wrote a blogpost about MD a while ago:
http://matthewrocklin.com/blog/work/2014/02/25/Multiple-Dispatch/

The docs pages live here:
http://multiple-dispatch.readthedocs.org/en/latest/

And the github page lives here:
https://github.com/mrocklin/multipledispatch

Recommendations for improvement welcome.  Thoughts on whether or not this
is appropriate to include in the standard library also welcome.

Best,
-Matthew Rocklin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/8f4cbcec/attachment.html>

From greg at krypto.org  Fri Aug 15 17:41:47 2014
From: greg at krypto.org (Gregory P. Smith)
Date: Fri, 15 Aug 2014 15:41:47 +0000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAN1YFWtvpmRauVr6qH+Xv1HuDBuJdZppRY6FZ+JFBViNvH-7_A@mail.gmail.com>
 <20140814174512.GQ4525@ando>
 <CAN1YFWumG67Xk3hQLh3kxzRKt5oZVU_nNMDqMAQLKa0Z9DNWPQ@mail.gmail.com>
 <87vbpuh3wk.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAN1YFWtGGug9MBDN0ZsQ9OniKypsB+e1BmvjhTi4J_jRuR83iw@mail.gmail.com>
Message-ID: <CAGE7PNKMNxxiwFeDOMhQBRBpdKU=8s4woDBER6U=zPPe2cuYxg@mail.gmail.com>

On Thu Aug 14 2014 at 8:02:14 PM Juancarlo A?ez <apalala at gmail.com> wrote:

>
> On Thu, Aug 14, 2014 at 10:09 PM, Stephen J. Turnbull <stephen at xemacs.org>
> wrote:
>
>> All that Guido is proposing to do is to rationalize development effort
>> on something people are already doing by standardizing a good-enough
>> approach, on an opt-in basis.  Sure, there are people who want
>> something more coercive, but Python developers aren't going to stand
>> for that, and Guido himself is the first line of defense.  He's
>> already said that's not going to happen, not now (and IIUC probably
>> never).
>>
>
> The change has been decreed, so it will be.
>

Where?  Please link to that in the mailing list archives if so.


>
> It's all right, because (as Guido suggests in the decree) we can now take
> the discussion to how to make it good for all involved. The Python
> community will likely succeed at that.
>
> It is a tautology that only experience will reveal the truth, at least so
> in our endeavour.
>
> Signing off this thread, yours truly,
>
> --
> Juancarlo *A?ez*
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/b986a132/attachment.html>

From encukou at gmail.com  Fri Aug 15 17:42:05 2014
From: encukou at gmail.com (Petr Viktorin)
Date: Fri, 15 Aug 2014 17:42:05 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
Message-ID: <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>

On Fri, Aug 15, 2014 at 9:48 AM, ?ukasz Langa <lukasz at langa.pl> wrote:
...
> As a side note, I?m happy you?re willing to agree on str | None. This reads
> really well and is concise enough to not require aliasing to be usable.

The common use is not all that concise:
    def foo(bar: int | None=None): pass

Or alternatively it could be:
    def foo(bar: int=None): pass
if the default was automatically allowed.


Also... Does None magically mean NoneType in type definitions?

From guido at python.org  Fri Aug 15 17:55:36 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Aug 2014 08:55:36 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
Message-ID: <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>

On Aug 15, 2014 8:43 AM, "Petr Viktorin" <encukou at gmail.com> wrote:
>
> On Fri, Aug 15, 2014 at 9:48 AM, ?ukasz Langa <lukasz at langa.pl> wrote:
> ...
> > As a side note, I?m happy you?re willing to agree on str | None. This
reads
> > really well and is concise enough to not require aliasing to be usable.
>
> The common use is not all that concise:
>     def foo(bar: int | None=None): pass
>
> Or alternatively it could be:
>     def foo(bar: int=None): pass
> if the default was automatically allowed.

Good idea.

> Also... Does None magically mean NoneType in type definitions?

Yes.

--Guido
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/c7b8d5b5/attachment.html>

From ethan at stoneleaf.us  Fri Aug 15 18:15:17 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 15 Aug 2014 09:15:17 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAAWJOpYZTXkcvNOFngBiDp5PtcWAuZjKFStdE4vaD6bkRCg=pw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
 <lskkem$8m7$1@ger.gmane.org>
 <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>
 <CAAWJOpYO+-+vpj29BBKvjbuJWFvK+trA53YYKQ02uChoo1fmXw@mail.gmail.com>
 <CADiSq7cm9iw_+AFoBR81uM6frnb4ceiVRsT1ecVFS9pSBo98PA@mail.gmail.com>
 <CAAWJOpYZTXkcvNOFngBiDp5PtcWAuZjKFStdE4vaD6bkRCg=pw@mail.gmail.com>
Message-ID: <53EE3215.6030907@stoneleaf.us>

On 08/15/2014 04:53 AM, Bill Winslow wrote:
>
> I knew something like this would happen when I decided to actually do something.

Take heart, we've all done it, and you've learned something important that I bet you don't soon forget.  ;)

--
~Ethan~

From greg at krypto.org  Fri Aug 15 18:24:09 2014
From: greg at krypto.org (Gregory P. Smith)
Date: Fri, 15 Aug 2014 16:24:09 +0000
Subject: [Python-ideas] Optional static typing -- the crossroads
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
 <CAP7+vJLxNsgJ6fQfxCxdqeahyA7=9ihBAVrEoS-iDBXqBEHMGQ@mail.gmail.com>
Message-ID: <CAGE7PN+6p+1NbFB-SZpw9eAZeUHUZBSJKvfqsfqaoz79QfUZ+g@mail.gmail.com>

On Thu Aug 14 2014 at 9:47:45 PM Guido van Rossum <guido at python.org> wrote:

> On Thu, Aug 14, 2014 at 9:40 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> On 15 August 2014 09:56, Guido van Rossum <guido at python.org> wrote:
>> >
>> > I don't buy the argument that PEP 3107 promises that annotations are
>> > completely free of inherent semantics.
>>
>> It's also worth noting the corresponding bullet point in PEP 3100
>> (under http://www.python.org/dev/peps/pep-3100/#core-language):
>>
>> * Add optional declarations for static typing [45] [10] [done]
>>
>> [10] Guido's blog ("Python Optional Typechecking Redux")
>> http://www.artima.com/weblogs/viewpost.jsp?thread=89161
>> [45] PEP 3107 (Function Annotations)
>> http://www.python.org/dev/peps/pep-3107
>>
>
> Such youthful optimism. :-)
>
>
>> Having Argument Clinic generate appropriate annotations automatically
>> could also be interesting.
>>
>
> How much of the 3.5 stdlib is currently covered by Argument Clinic? I
> thought there's still a lot left to do. Might it be possible to convert
> some of pytypedecl's stubs into AC stubs? Alternatively, the AC info could
> be turned into mypy stubs. I'm just really hoping that between AC,
> pytypedecl, PyCharm and mypy we have specs for most builtins and extension
> modules in machine-readable form already, and we could use this combined
> information to bootstrap mypy's collection of stubs.
>
>
I believe we have partially intersecting subsets of builtins and stdlib
coverage, union them all together and sanity check them and it's a good
start but will likely still have giant holes to be filled in.

We've been concentrating on 2.7 with the code analysis to generate
pytypedecl pytd's for but have always assumed that argument clinic would be
useful in providing annotation details for 3.4 onwards. Should it generate
annotation files itself? possibly, but I'm not sure it is expressive enough
to generate an ideal annotation.

To start with I'd leave generating annotations Python builtins, extensions
and internals itself out of CPython itself in 3.5.

Such things can be pulled in with tools to generate them in later release
once we're happy it is easy to maintain via the tools without a much human
tweaking being required.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/6b66e29b/attachment.html>

From brakhane at googlemail.com  Fri Aug 15 18:25:33 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Fri, 15 Aug 2014 18:25:33 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <20140815143821.GC4525@ando>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EDFF19.7060305@gmail.com> <20140815143821.GC4525@ando>
Message-ID: <53EE347D.2040801@googlemail.com>

On 15.08.2014 16:38, Steven D'Aprano wrote:
> How do you know the linter will complain?
Because a file object is not a list of strings, which is what word_count
incorrectly declares it needs. (it actually requires a iterable of
things that can be split()ed and doesn't return a Dict[str, int] but a
Dict[return_type_of_split(), int])

> So why do you consider it a "must" to be able to override functions
> that report type errors at compile time?

The difference  in that latter case is that the code runs perfectly fine
and correctly. It's just that the linter implies you've done something bad.

Using isinstance checks is kinda frowned upon, using type annotations
will be probably considered to be totally acceptable practise (if not,
what's the point of this proposal). It's much harder to argue with the
original author that it's a bug in the latter case. Furthermore, the
author might not want to loose the type requirement, because he doesn't
want to guarantee those semantics, for example. If I'm willing to take
that risk (and have a test case in my code) why shouldn't I be allowed
to silence those errors in a simple way that doesn't require casting at
every method call?

To me, one of the things that sets Python apart from other languages is
the fact that code will and can be used in a way the original author
might not have thought of.

Giving the original author of methods the means to dictate what types
are acceptable and not with no clean and simple way of overriding it
just feels Java-esque to me. (Again, isinstance checks are considered a
bad practise, but I doubt declaring too restrictive types will be)

I do not want to be forced to litter my code with casts, making it ugly
and feel bad about what seems to me a reasonable method call.


> But for what it's worth, as this proposal has repeatedly said, the
> linter is optional. If you don't believe the linter, *just run the
> code* in Python like you have always done before. 
As already said, I might be forced to run it because of company policy.
I might contribute to a library that uses mypy.

I don't have a big problem with complying with strange code style
requirements enforced by a linter ("in this project, all variables must
begin with foobar and end with a number"), that's just naming.

I do have a problem when a linter will de facto enforce rules like "all
code must only call  methods in the way the original author thought of",
as this might lead to more ugly code because of workarounds/casts.

Cheers,
  Dennis



From greg at krypto.org  Fri Aug 15 18:35:54 2014
From: greg at krypto.org (Gregory P. Smith)
Date: Fri, 15 Aug 2014 16:35:54 +0000
Subject: [Python-ideas] Optional static typing -- the crossroads
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EDFF19.7060305@gmail.com>
Message-ID: <CAGE7PNJ6MRPDhWvVrqhJDk2JFeZxR4aASPqW6ND3P8PdgeAS2A@mail.gmail.com>

On Fri Aug 15 2014 at 5:46:12 AM Dennis Brakhane <brakhane at googlemail.com>
wrote:

> Am 15.08.2014 01:56, schrieb Guido van Rossum:
>
> > PS. I realize I didn't discuss question (C) much. That's intentional
> -- we can now start discussing specific mypy features in separate
> threads (or in this one :-).
>
> So is this the place to discuss "the thornier issues brought up against
> mypy"? Because I think it's important that we get an idea of what we
> want mypy to be able to do and what
> not. After all, mypy will probably end up as a kind of reference
> implementation for static type checkers. And I'm worried there might be
> real damage to Python as a language
> if they aren't thought through:
>

I'm not concerned about that myself. If this syntax doesn't work out, the
the existing status quo prevails and the syntax is ignored by other tools
that need more than it can provide.

That still leads to the same feedback cycle we already have today such that
the language syntax for type annotations can evolve and improve again in
the future.

[Caution: I'm commenting about mypy below after having spent less than 10
minutes looking at its website to pretend I know what it can and can't do
already. Assume I'm wrong.]

ie: there are things I don't know about mypy.  Does it have the ability to
specify that the return type of "def foo(A, B)" is the same type as
whatever the caller passed in for parameter B?  That is a pretty common
thing in Python.  Even if it doesn't have it today, I suspect it can be
added in the future.

There are other things mypy didn't appear to deal with at first glance
either, specific sets of possible inputs -> outputs rather than always
listing inputs as a union of all possible types for that parameter and
outputs as a union of all possible types to be output.

I may well be wrong about the above.  But even if I'm not, I'm not worried.
Deeper analysis and annotation tools will simply do what they are already
doing: plowing on ahead with their own extended annotation format.

I understand your concerns (in the rest of our message that i've elided)
but I think a "try it and see" approach will actually work here.

Libraries are already released where people have gone overboard with
incorrect overly strict isinstance or issubclass checks. Those are _worse_
than something that merely lists overly strict types as you literally
cannot use them without modifying the code or complying.

Something any code analyzer implementation needs is an ability to be told
"ignore this module, it's full of crap." as part of its analysis process. :)

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/c42b05c7/attachment-0001.html>

From brakhane at googlemail.com  Fri Aug 15 18:43:02 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Fri, 15 Aug 2014 18:43:02 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
Message-ID: <53EE3896.2050804@googlemail.com>

On 15.08.2014 17:42, Petr Viktorin wrote:
>
> The common use is not all that concise:
>     def foo(bar: int | None=None): pass
>
> Or alternatively it could be:
>     def foo(bar: int=None): pass
> if the default was automatically allowed.
(Assuming you mean "the type of the default")

While I like the second form a bit more, it kinda goes against "explicit
is better than implicit".

Also, if I change the default value from None to 42, I've either changed
the allowable types,
or need to remember to turn "bar: int=None" into "bar: int|None = 42".

Furthermore, what should happen in the following case:


# no annotations here
def foo(): ...

def bar(evil: int = foo()): ...


Should this be disallowed, as the type checker will not be able to know
what type foo is? Should it just assume int?
And in the latter case, what should happen if foo now gains a "-> float"
annotation?


From encukou at gmail.com  Fri Aug 15 18:48:54 2014
From: encukou at gmail.com (Petr Viktorin)
Date: Fri, 15 Aug 2014 18:48:54 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
Message-ID: <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>

On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum <guido at python.org> wrote:
...
>> Also... Does None magically mean NoneType in type definitions?
>
> Yes.

This would mean either that `(None | None) is None`, or that (x |
None) is not always "optional x".
And if type objects grow any other common functionality, None will
have to support that as well.

From guido at python.org  Fri Aug 15 18:56:35 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Aug 2014 09:56:35 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53EE347D.2040801@googlemail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EDFF19.7060305@gmail.com> <20140815143821.GC4525@ando>
 <53EE347D.2040801@googlemail.com>
Message-ID: <CAP7+vJJ9rx+pTuZU2OpR50HNby8cEimDxy4sXG2Xxv1DmCzKCg@mail.gmail.com>

On Fri, Aug 15, 2014 at 9:25 AM, Dennis Brakhane <brakhane at googlemail.com>
wrote:

> On 15.08.2014 16:38, Steven D'Aprano wrote:
> > How do you know the linter will complain?
> Because a file object is not a list of strings, which is what word_count
> incorrectly declares it needs. (it actually requires a iterable of
> things that can be split()ed and doesn't return a Dict[str, int] but a
> Dict[return_type_of_split(), int])
>
> > So why do you consider it a "must" to be able to override functions
> > that report type errors at compile time?
>
> The difference  in that latter case is that the code runs perfectly fine
> and correctly. It's just that the linter implies you've done something bad.
>
> Using isinstance checks is kinda frowned upon, using type annotations
> will be probably considered to be totally acceptable practise (if not,
> what's the point of this proposal). It's much harder to argue with the
> original author that it's a bug in the latter case. Furthermore, the
> author might not want to loose the type requirement, because he doesn't
> want to guarantee those semantics, for example. If I'm willing to take
> that risk (and have a test case in my code) why shouldn't I be allowed
> to silence those errors in a simple way that doesn't require casting at
> every method call?
>
> To me, one of the things that sets Python apart from other languages is
> the fact that code will and can be used in a way the original author
> might not have thought of.
>
> Giving the original author of methods the means to dictate what types
> are acceptable and not with no clean and simple way of overriding it
> just feels Java-esque to me. (Again, isinstance checks are considered a
> bad practise, but I doubt declaring too restrictive types will be)
>
> I do not want to be forced to litter my code with casts, making it ugly
> and feel bad about what seems to me a reasonable method call.
>
>
> > But for what it's worth, as this proposal has repeatedly said, the
> > linter is optional. If you don't believe the linter, *just run the
> > code* in Python like you have always done before.
> As already said, I might be forced to run it because of company policy.
> I might contribute to a library that uses mypy.
>
> I don't have a big problem with complying with strange code style
> requirements enforced by a linter ("in this project, all variables must
> begin with foobar and end with a number"), that's just naming.
>
> I do have a problem when a linter will de facto enforce rules like "all
> code must only call  methods in the way the original author thought of",
> as this might lead to more ugly code because of workarounds/casts.
>

This attitude will not help you when interviewing at such a company. Did it
occur to you that there might actually be a good reason for the lint rule,
and that you, as a new hire, might not yet be aware of that reason?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/6b1833ac/attachment.html>

From guido at python.org  Fri Aug 15 19:00:13 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Aug 2014 10:00:13 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
Message-ID: <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>

On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin <encukou at gmail.com> wrote:

> On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum <guido at python.org>
> wrote:
> ...
> >> Also... Does None magically mean NoneType in type definitions?
> >
> > Yes.
>
> This would mean either that `(None | None) is None`, or that (x |
> None) is not always "optional x".
> And if type objects grow any other common functionality, None will
> have to support that as well.
>

Perhaps None itself should not implement any of this, and the __ror__
method on ABCs should implement it. That way, None|Mapping and Mapping|None
would both work, yet None|None would still be the TypeError it is today.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/d20f70ec/attachment.html>

From encukou at gmail.com  Fri Aug 15 19:11:38 2014
From: encukou at gmail.com (Petr Viktorin)
Date: Fri, 15 Aug 2014 19:11:38 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53EE3896.2050804@googlemail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <53EE3896.2050804@googlemail.com>
Message-ID: <CA+=+wqD+aZ2SYTidwDX3d2NJ-TqP=C3TnCxcLwMcZhyE96vnew@mail.gmail.com>

On Fri, Aug 15, 2014 at 6:43 PM, Dennis Brakhane
<brakhane at googlemail.com> wrote:
> On 15.08.2014 17:42, Petr Viktorin wrote:
>>
>> The common use is not all that concise:
>>     def foo(bar: int | None=None): pass
>>
>> Or alternatively it could be:
>>     def foo(bar: int=None): pass
>> if the default was automatically allowed.
> (Assuming you mean "the type of the default")

I really meant *only* the default. This really only works for None,
but that's a good thing, since something like:
    def foo(bar:int=''): pass
looks very suspicious. I'd be fine with the linter complaining about
foo('hello').

Of course you can always do:
    def foo(bar: (int | str)=''): pass

> While I like the second form a bit more, it kinda goes against "explicit
> is better than implicit".
>
> Also, if I change the default value from None to 42, I've either changed
> the allowable types,
> or need to remember to turn "bar: int=None" into "bar: int|None = 42".
>
> Furthermore, what should happen in the following case:
>
>
> # no annotations here
> def foo(): ...
>
> def bar(evil: int = foo()): ...
>
>
> Should this be disallowed, as the type checker will not be able to know
> what type foo is? Should it just assume int?
> And in the latter case, what should happen if foo now gains a "-> float"
> annotation?

If your linter can't figure it out, just specify the default's type
explicitly. Always a good thing to do when something's not immediately
obvious.

From brakhane at googlemail.com  Fri Aug 15 19:12:04 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Fri, 15 Aug 2014 19:12:04 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAGE7PNJ6MRPDhWvVrqhJDk2JFeZxR4aASPqW6ND3P8PdgeAS2A@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EDFF19.7060305@gmail.com>
 <CAGE7PNJ6MRPDhWvVrqhJDk2JFeZxR4aASPqW6ND3P8PdgeAS2A@mail.gmail.com>
Message-ID: <53EE3F64.5090600@googlemail.com>

On 15.08.2014 18:35, Gregory P. Smith wrote:
>
> Libraries are already released where people have gone overboard with
> incorrect overly strict isinstance or issubclass checks. Those are
> _worse_ than something that merely lists overly strict types as you
> literally cannot use them without modifying the code or complying.
>
I agree.

> That still leads to the same feedback cycle we already have today such
that the language syntax for
> type annotations can evolve and improve again in the future.
>
> Does it have the ability to specify that the return type of "def
> foo(A, B)" is the same type as whatever the caller passed in for
> parameter B?  That is a pretty common thing in Python.  Even if it
> doesn't have it today, I suspect it can be added in the future.
That's a good example of what I'm worried about: to be really useful,
the type declarations have to be really flexible. My (maybe irrational)
fear is that we end up with a turing complete type system.

If a programmers needs to spend 10 minutes thinking about how exactly he
has to declare his method parameters or "class interfaces" (does my
container type behave covariantly or contravariantly?), and all that
just so his code passes a linter, something went wrong ;)

We shouldn't try to make the signatures carry all information needed for
type checkers, if some things can only be found out by code analysis,
that's fine by me.

For example, Jedi can handle the following silly example:

def foo():
  return [os, sys]

x = foo()


Jedi knows that x[1] is sys and will only propose members of sys, and
only os members for x[0]. And the linter of the current development
version will actually barf on x[1].walk():

/tmp/a.py:12:5: E1 AttributeError: <CompiledObject: <module 'sys'
(built-in)>> has no attribute walk.

Trying to get this kind information encoded into some kind of type
signature would be insane, IMO.


From encukou at gmail.com  Fri Aug 15 19:19:38 2014
From: encukou at gmail.com (Petr Viktorin)
Date: Fri, 15 Aug 2014 19:19:38 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
Message-ID: <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>

On Fri, Aug 15, 2014 at 7:00 PM, Guido van Rossum <guido at python.org> wrote:
> On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin <encukou at gmail.com> wrote:
>>
>> On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum <guido at python.org>
>> wrote:
>> ...
>> >> Also... Does None magically mean NoneType in type definitions?
>> >
>> > Yes.
>>
>> This would mean either that `(None | None) is None`, or that (x |
>> None) is not always "optional x".
>> And if type objects grow any other common functionality, None will
>> have to support that as well.
>
>
> Perhaps None itself should not implement any of this, and the __ror__ method
> on ABCs should implement it. That way, None|Mapping and Mapping|None would
> both work, yet None|None would still be the TypeError it is today.

... and that (x|None) does not always mean "optional x".
Is this case special enough?

From guido at python.org  Fri Aug 15 19:28:20 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Aug 2014 10:28:20 -0700
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
Message-ID: <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>

Have you written anything significant that you're distributing with it? Has
anyone else?

Are there good uses of singledispatch in the wild even?


On Fri, Aug 15, 2014 at 8:33 AM, Matthew Rocklin <mrocklin at gmail.com> wrote:

> Hi all,
>
> I curate the multipledispatch library on PyPI.  I believe that it is a
> natural continuation of singledispatch outlined in PEP 443 and included in
> functools 3.4.
>
> For those unaware, dispatching correctly and unambiguously on multiple
> inputs is a somewhat more complex problem than on a single input.  I
> believe the approach taken in multipledispatch is fairly robust.
>
> I wrote a blogpost about MD a while ago:
> http://matthewrocklin.com/blog/work/2014/02/25/Multiple-Dispatch/
>
> The docs pages live here:
> http://multiple-dispatch.readthedocs.org/en/latest/
>
> And the github page lives here:
> https://github.com/mrocklin/multipledispatch
>
> Recommendations for improvement welcome.  Thoughts on whether or not this
> is appropriate to include in the standard library also welcome.
>
> Best,
> -Matthew Rocklin
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/9b80ec7e/attachment.html>

From ethan at stoneleaf.us  Fri Aug 15 19:34:12 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 15 Aug 2014 10:34:12 -0700
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
Message-ID: <53EE4494.8050609@stoneleaf.us>

On 08/15/2014 10:28 AM, Guido van Rossum wrote:
>
> Are there good uses of singledispatch in the wild even?

Well, I don't know if it's a "good use", but I have used singledispatch to override 'float' in certain modules so that I 
can say, for example, float(DateTime.Time) or float("time string").

It's really quite handy.

--
~Ethan~

From guido at python.org  Fri Aug 15 19:36:50 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Aug 2014 10:36:50 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
Message-ID: <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>

On Fri, Aug 15, 2014 at 10:19 AM, Petr Viktorin <encukou at gmail.com> wrote:

> On Fri, Aug 15, 2014 at 7:00 PM, Guido van Rossum <guido at python.org>
> wrote:
> > On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin <encukou at gmail.com>
> wrote:
> >>
> >> On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum <guido at python.org>
> >> wrote:
> >> ...
> >> >> Also... Does None magically mean NoneType in type definitions?
> >> >
> >> > Yes.
> >>
> >> This would mean either that `(None | None) is None`, or that (x |
> >> None) is not always "optional x".
> >> And if type objects grow any other common functionality, None will
> >> have to support that as well.
> >
> >
> > Perhaps None itself should not implement any of this, and the __ror__
> method
> > on ABCs should implement it. That way, None|Mapping and Mapping|None
> would
> > both work, yet None|None would still be the TypeError it is today.
>
> ... and that (x|None) does not always mean "optional x".
> Is this case special enough?
>

I'm not following. The proposal seems to be to add __or__ and __ror__
methods to type itself requiring the other argument to be also a type, or
the special case None (which is a value, not a type).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/96355deb/attachment-0001.html>

From alexander.belopolsky at gmail.com  Fri Aug 15 19:47:22 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 15 Aug 2014 13:47:22 -0400
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
Message-ID: <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>

On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido at python.org> wrote:

> Are there good uses of singledispatch in the wild even?


In one of my projects, I have a todo item to convert a homegrown
{type:func} dictionary based dispatch to singledispatch.  It has been open
for 5 months already.  I wouldn't be surprised if others had something
similar: a good-enough (and probably buggy) solution that is not bad enough
to justify adding extra dependency in 2.x or replacing with a 3.x only
solution.

Multiple dispatch is much harder to get right or even "good enough," so I
don't think singledispatch popularity or lack thereof is a good predictor
for multipledispatch.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/845c6a3d/attachment.html>

From encukou at gmail.com  Fri Aug 15 19:51:12 2014
From: encukou at gmail.com (Petr Viktorin)
Date: Fri, 15 Aug 2014 19:51:12 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
Message-ID: <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>

On Fri, Aug 15, 2014 at 7:36 PM, Guido van Rossum <guido at python.org> wrote:
> On Fri, Aug 15, 2014 at 10:19 AM, Petr Viktorin <encukou at gmail.com> wrote:
>>
>> On Fri, Aug 15, 2014 at 7:00 PM, Guido van Rossum <guido at python.org>
>> wrote:
>> > On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin <encukou at gmail.com>
>> > wrote:
>> >>
>> >> On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum <guido at python.org>
>> >> wrote:
>> >> ...
>> >> >> Also... Does None magically mean NoneType in type definitions?
>> >> >
>> >> > Yes.
>> >>
>> >> This would mean either that `(None | None) is None`, or that (x |
>> >> None) is not always "optional x".
>> >> And if type objects grow any other common functionality, None will
>> >> have to support that as well.
>> >
>> >
>> > Perhaps None itself should not implement any of this, and the __ror__
>> > method
>> > on ABCs should implement it. That way, None|Mapping and Mapping|None
>> > would
>> > both work, yet None|None would still be the TypeError it is today.
>>
>> ... and that (x|None) does not always mean "optional x".
>> Is this case special enough?
>
>
> I'm not following. The proposal seems to be to add __or__ and __ror__
> methods to type itself requiring the other argument to be also a type, or
> the special case None (which is a value, not a type).

My concern is that if someone does programmatic type declaration
manipulation/generation, there's now a special case to keep in mind.
Instead of

    def optional(t):
        return t | None

it's now:

    def optional(t):
        if t is None:
            return t
        else:
            return t | None

because unlike other type declarations, None doesn't have __or__, or
any other operation that types will gain in the future as this
proposal matures.

But maybe this is will never be a valid use case?

From guido at python.org  Fri Aug 15 19:55:06 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Aug 2014 10:55:06 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
Message-ID: <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>

Doesn't seem a big deal. The only place where you'd see an implementation
of optional() would be in typing.py, and optional(None) is redundant anyway.


On Fri, Aug 15, 2014 at 10:51 AM, Petr Viktorin <encukou at gmail.com> wrote:

> On Fri, Aug 15, 2014 at 7:36 PM, Guido van Rossum <guido at python.org>
> wrote:
> > On Fri, Aug 15, 2014 at 10:19 AM, Petr Viktorin <encukou at gmail.com>
> wrote:
> >>
> >> On Fri, Aug 15, 2014 at 7:00 PM, Guido van Rossum <guido at python.org>
> >> wrote:
> >> > On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin <encukou at gmail.com>
> >> > wrote:
> >> >>
> >> >> On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum <guido at python.org>
> >> >> wrote:
> >> >> ...
> >> >> >> Also... Does None magically mean NoneType in type definitions?
> >> >> >
> >> >> > Yes.
> >> >>
> >> >> This would mean either that `(None | None) is None`, or that (x |
> >> >> None) is not always "optional x".
> >> >> And if type objects grow any other common functionality, None will
> >> >> have to support that as well.
> >> >
> >> >
> >> > Perhaps None itself should not implement any of this, and the __ror__
> >> > method
> >> > on ABCs should implement it. That way, None|Mapping and Mapping|None
> >> > would
> >> > both work, yet None|None would still be the TypeError it is today.
> >>
> >> ... and that (x|None) does not always mean "optional x".
> >> Is this case special enough?
> >
> >
> > I'm not following. The proposal seems to be to add __or__ and __ror__
> > methods to type itself requiring the other argument to be also a type, or
> > the special case None (which is a value, not a type).
>
> My concern is that if someone does programmatic type declaration
> manipulation/generation, there's now a special case to keep in mind.
> Instead of
>
>     def optional(t):
>         return t | None
>
> it's now:
>
>     def optional(t):
>         if t is None:
>             return t
>         else:
>             return t | None
>
> because unlike other type declarations, None doesn't have __or__, or
> any other operation that types will gain in the future as this
> proposal matures.
>
> But maybe this is will never be a valid use case?
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/776456a6/attachment.html>

From mrocklin at gmail.com  Fri Aug 15 19:55:57 2014
From: mrocklin at gmail.com (Matthew Rocklin)
Date: Fri, 15 Aug 2014 10:55:57 -0700
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
 <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>
Message-ID: <CAJ8oX-EEHamhjUY-Ub8APYsOf1koXC=AN8mgobin0OA-nSRP-A@mail.gmail.com>

I've definitely seen homegrown solutions like Alexander talks about in a
variety of projects.  It has popped up a few times in SymPy.

My implementation was influenced somewhat by Julia's solution which seems
fairly sober.  Multiple dispatch has demonstrated value in that community.

I used the multipledispatch library in a few of my projects.  I can provide
examples of where it's been helpful if desired.


On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky <
alexander.belopolsky at gmail.com> wrote:

>
> On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido at python.org>
> wrote:
>
>> Are there good uses of singledispatch in the wild even?
>
>
> In one of my projects, I have a todo item to convert a homegrown
> {type:func} dictionary based dispatch to singledispatch.  It has been open
> for 5 months already.  I wouldn't be surprised if others had something
> similar: a good-enough (and probably buggy) solution that is not bad enough
> to justify adding extra dependency in 2.x or replacing with a 3.x only
> solution.
>
> Multiple dispatch is much harder to get right or even "good enough," so I
> don't think singledispatch popularity or lack thereof is a good predictor
> for multipledispatch.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/5ac26da7/attachment-0001.html>

From guido at python.org  Fri Aug 15 20:01:07 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Aug 2014 11:01:07 -0700
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <CAJ8oX-EEHamhjUY-Ub8APYsOf1koXC=AN8mgobin0OA-nSRP-A@mail.gmail.com>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
 <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>
 <CAJ8oX-EEHamhjUY-Ub8APYsOf1koXC=AN8mgobin0OA-nSRP-A@mail.gmail.com>
Message-ID: <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>

Please do write about non-toy examples!


On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin <mrocklin at gmail.com>
wrote:

> I've definitely seen homegrown solutions like Alexander talks about in a
> variety of projects.  It has popped up a few times in SymPy.
>
> My implementation was influenced somewhat by Julia's solution which seems
> fairly sober.  Multiple dispatch has demonstrated value in that community.
>
> I used the multipledispatch library in a few of my projects.  I can
> provide examples of where it's been helpful if desired.
>
>
> On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky <
> alexander.belopolsky at gmail.com> wrote:
>
>>
>> On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> Are there good uses of singledispatch in the wild even?
>>
>>
>> In one of my projects, I have a todo item to convert a homegrown
>> {type:func} dictionary based dispatch to singledispatch.  It has been open
>> for 5 months already.  I wouldn't be surprised if others had something
>> similar: a good-enough (and probably buggy) solution that is not bad enough
>> to justify adding extra dependency in 2.x or replacing with a 3.x only
>> solution.
>>
>> Multiple dispatch is much harder to get right or even "good enough," so I
>> don't think singledispatch popularity or lack thereof is a good predictor
>> for multipledispatch.
>>
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/f3dca8b1/attachment.html>

From lukasz at langa.pl  Fri Aug 15 20:25:35 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 15 Aug 2014 11:25:35 -0700
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
 <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>
 <CAJ8oX-EEHamhjUY-Ub8APYsOf1koXC=AN8mgobin0OA-nSRP-A@mail.gmail.com>
 <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>
Message-ID: <E05E59D2-11C2-4F31-BF17-7B62478839A5@langa.pl>

It has over 25k downloads from PyPI per month. It?s used in OpenStack?s Nova. It?s used at Facebook as part of a deadline based, resource aware, dependency based, fault tolerant scheduler.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev



On Aug 15, 2014, at 11:01 AM, Guido van Rossum <guido at python.org> wrote:

> Please do write about non-toy examples!
> 
> 
> On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin <mrocklin at gmail.com> wrote:
> I've definitely seen homegrown solutions like Alexander talks about in a variety of projects.  It has popped up a few times in SymPy.  
> 
> My implementation was influenced somewhat by Julia's solution which seems fairly sober.  Multiple dispatch has demonstrated value in that community.
> 
> I used the multipledispatch library in a few of my projects.  I can provide examples of where it's been helpful if desired.
> 
> 
> On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote:
> 
> On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido at python.org> wrote:
> Are there good uses of singledispatch in the wild even?
> 
> In one of my projects, I have a todo item to convert a homegrown {type:func} dictionary based dispatch to singledispatch.  It has been open for 5 months already.  I wouldn't be surprised if others had something similar: a good-enough (and probably buggy) solution that is not bad enough to justify adding extra dependency in 2.x or replacing with a 3.x only solution.
> 
> Multiple dispatch is much harder to get right or even "good enough," so I don't think singledispatch popularity or lack thereof is a good predictor for multipledispatch.
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/ae2cd31f/attachment.html>

From mrocklin at gmail.com  Fri Aug 15 20:33:06 2014
From: mrocklin at gmail.com (Matthew Rocklin)
Date: Fri, 15 Aug 2014 11:33:06 -0700
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <E05E59D2-11C2-4F31-BF17-7B62478839A5@langa.pl>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
 <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>
 <CAJ8oX-EEHamhjUY-Ub8APYsOf1koXC=AN8mgobin0OA-nSRP-A@mail.gmail.com>
 <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>
 <E05E59D2-11C2-4F31-BF17-7B62478839A5@langa.pl>
Message-ID: <CAJ8oX-EzLtn4JubP7aVFGH=JaGNcwz4CFGnzfQF009oYo0k0eg@mail.gmail.com>

I assume that Lukasz is referrring to singledispatch?  multipledispatch
doesn't yet have that level of adoption as far as I'm aware.


On Fri, Aug 15, 2014 at 11:25 AM, ?ukasz Langa <lukasz at langa.pl> wrote:

> It has over 25k downloads from PyPI per month. It?s used in OpenStack?s
> Nova. It?s used at Facebook as part of a deadline based, resource aware,
> dependency based, fault tolerant scheduler.
>
> --
> Best regards,
> ?ukasz Langa
>
> WWW: http://lukasz.langa.pl/
> Twitter: @llanga
> IRC: ambv on #python-dev
>
>
>
> On Aug 15, 2014, at 11:01 AM, Guido van Rossum <guido at python.org> wrote:
>
> Please do write about non-toy examples!
>
>
> On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin <mrocklin at gmail.com>
> wrote:
>
>> I've definitely seen homegrown solutions like Alexander talks about in a
>> variety of projects.  It has popped up a few times in SymPy.
>>
>> My implementation was influenced somewhat by Julia's solution which seems
>> fairly sober.  Multiple dispatch has demonstrated value in that community.
>>
>> I used the multipledispatch library in a few of my projects.  I can
>> provide examples of where it's been helpful if desired.
>>
>>
>> On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky <
>> alexander.belopolsky at gmail.com> wrote:
>>
>>>
>>> On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido at python.org>
>>> wrote:
>>>
>>>> Are there good uses of singledispatch in the wild even?
>>>
>>>
>>> In one of my projects, I have a todo item to convert a homegrown
>>> {type:func} dictionary based dispatch to singledispatch.  It has been open
>>> for 5 months already.  I wouldn't be surprised if others had something
>>> similar: a good-enough (and probably buggy) solution that is not bad enough
>>> to justify adding extra dependency in 2.x or replacing with a 3.x only
>>> solution.
>>>
>>> Multiple dispatch is much harder to get right or even "good enough," so
>>> I don't think singledispatch popularity or lack thereof is a good predictor
>>> for multipledispatch.
>>>
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/4a2dcf08/attachment-0001.html>

From lukasz at langa.pl  Fri Aug 15 20:38:02 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 15 Aug 2014 11:38:02 -0700
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <CAJ8oX-EzLtn4JubP7aVFGH=JaGNcwz4CFGnzfQF009oYo0k0eg@mail.gmail.com>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
 <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>
 <CAJ8oX-EEHamhjUY-Ub8APYsOf1koXC=AN8mgobin0OA-nSRP-A@mail.gmail.com>
 <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>
 <E05E59D2-11C2-4F31-BF17-7B62478839A5@langa.pl>
 <CAJ8oX-EzLtn4JubP7aVFGH=JaGNcwz4CFGnzfQF009oYo0k0eg@mail.gmail.com>
Message-ID: <967197E8-94BD-45C3-BCD8-BFD0690EFF12@langa.pl>

Ah, that?s right, I meant singledispatch. Sorry for the confusion! :)

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

On Aug 15, 2014, at 11:33 AM, Matthew Rocklin <mrocklin at gmail.com> wrote:

> I assume that Lukasz is referrring to singledispatch?  multipledispatch doesn't yet have that level of adoption as far as I'm aware.
> 
> 
> On Fri, Aug 15, 2014 at 11:25 AM, ?ukasz Langa <lukasz at langa.pl> wrote:
> It has over 25k downloads from PyPI per month. It?s used in OpenStack?s Nova. It?s used at Facebook as part of a deadline based, resource aware, dependency based, fault tolerant scheduler.
> 
> -- 
> Best regards,
> ?ukasz Langa
> 
> WWW: http://lukasz.langa.pl/
> Twitter: @llanga
> IRC: ambv on #python-dev
> 
> 
> 
> On Aug 15, 2014, at 11:01 AM, Guido van Rossum <guido at python.org> wrote:
> 
>> Please do write about non-toy examples!
>> 
>> 
>> On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin <mrocklin at gmail.com> wrote:
>> I've definitely seen homegrown solutions like Alexander talks about in a variety of projects.  It has popped up a few times in SymPy.  
>> 
>> My implementation was influenced somewhat by Julia's solution which seems fairly sober.  Multiple dispatch has demonstrated value in that community.
>> 
>> I used the multipledispatch library in a few of my projects.  I can provide examples of where it's been helpful if desired.
>> 
>> 
>> On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky <alexander.belopolsky at gmail.com> wrote:
>> 
>> On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido at python.org> wrote:
>> Are there good uses of singledispatch in the wild even?
>> 
>> In one of my projects, I have a todo item to convert a homegrown {type:func} dictionary based dispatch to singledispatch.  It has been open for 5 months already.  I wouldn't be surprised if others had something similar: a good-enough (and probably buggy) solution that is not bad enough to justify adding extra dependency in 2.x or replacing with a 3.x only solution.
>> 
>> Multiple dispatch is much harder to get right or even "good enough," so I don't think singledispatch popularity or lack thereof is a good predictor for multipledispatch.
>> 
>> 
>> 
>> 
>> -- 
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/d1d2bfec/attachment.html>

From varma.sunjay at gmail.com  Fri Aug 15 20:43:23 2014
From: varma.sunjay at gmail.com (Sunjay Varma)
Date: Fri, 15 Aug 2014 14:43:23 -0400
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
Message-ID: <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>

Hi all,
Has the syntax for specifying type been fully decided on already?

Using brackets may confuse new Python programmers. Since specifying type in
Python is fairly new anyway, what do you all think of introducing angle
brackets into Python instead? Other languages use angle brackets to specify
types. It provides a good separation between type specification and list
indexing.

I'm also worried that using square brackets will cause confusion as that
notation is generally associated with array declarations in other
languages. Even in Python, MyClass[int] may be confused with getting a key
called int from some MyClass.

dict<str, int> seems to tell me more explicitly that I'm dealing with a
declaration of an expected type. dict[str, int] looks very much like I'm
getting an item (str, int) from some class.

The angle bracket (or any other suggestions you have in mind) provides a
more concrete separation between when we are performing item indexing and
when we're specifying a type to validate.

Sunjay
On Aug 15, 2014 1:55 PM, "Guido van Rossum" <guido at python.org> wrote:

> Doesn't seem a big deal. The only place where you'd see an implementation
> of optional() would be in typing.py, and optional(None) is redundant anyway.
>
>
> On Fri, Aug 15, 2014 at 10:51 AM, Petr Viktorin <encukou at gmail.com> wrote:
>
>> On Fri, Aug 15, 2014 at 7:36 PM, Guido van Rossum <guido at python.org>
>> wrote:
>> > On Fri, Aug 15, 2014 at 10:19 AM, Petr Viktorin <encukou at gmail.com>
>> wrote:
>> >>
>> >> On Fri, Aug 15, 2014 at 7:00 PM, Guido van Rossum <guido at python.org>
>> >> wrote:
>> >> > On Fri, Aug 15, 2014 at 9:48 AM, Petr Viktorin <encukou at gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> On Fri, Aug 15, 2014 at 5:55 PM, Guido van Rossum <guido at python.org
>> >
>> >> >> wrote:
>> >> >> ...
>> >> >> >> Also... Does None magically mean NoneType in type definitions?
>> >> >> >
>> >> >> > Yes.
>> >> >>
>> >> >> This would mean either that `(None | None) is None`, or that (x |
>> >> >> None) is not always "optional x".
>> >> >> And if type objects grow any other common functionality, None will
>> >> >> have to support that as well.
>> >> >
>> >> >
>> >> > Perhaps None itself should not implement any of this, and the __ror__
>> >> > method
>> >> > on ABCs should implement it. That way, None|Mapping and Mapping|None
>> >> > would
>> >> > both work, yet None|None would still be the TypeError it is today.
>> >>
>> >> ... and that (x|None) does not always mean "optional x".
>> >> Is this case special enough?
>> >
>> >
>> > I'm not following. The proposal seems to be to add __or__ and __ror__
>> > methods to type itself requiring the other argument to be also a type,
>> or
>> > the special case None (which is a value, not a type).
>>
>> My concern is that if someone does programmatic type declaration
>> manipulation/generation, there's now a special case to keep in mind.
>> Instead of
>>
>>     def optional(t):
>>         return t | None
>>
>> it's now:
>>
>>     def optional(t):
>>         if t is None:
>>             return t
>>         else:
>>             return t | None
>>
>> because unlike other type declarations, None doesn't have __or__, or
>> any other operation that types will gain in the future as this
>> proposal matures.
>>
>> But maybe this is will never be a valid use case?
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/a0d065c0/attachment.html>

From lukasz at langa.pl  Fri Aug 15 20:55:44 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 15 Aug 2014 11:55:44 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
Message-ID: <2971B072-78B1-4E51-9B0F-50BFFDD66535@langa.pl>

On Aug 15, 2014, at 11:43 AM, Sunjay Varma <varma.sunjay at gmail.com> wrote:
> Hi all,
> Has the syntax for specifying type been fully decided on already?
> 
> Using brackets may confuse new Python programmers. (?) Other languages use angle brackets to specify types.
> 


I also agree that angle brackets would be nicer. Guido decided against it for pragmatic reasons:
1. angle brackets would create Python source code incompatible with any version lower than 3.5
2. angle brackets would complicate the lexer (normally you expect < and > to be spaced, in this case it wouldn?t)
3. angle brackets would require a new mechanism in Python to store this kind of expression within the type; this is still true for generics expressed with square brackets but at least you can use the existing nuts and bolts of Python classes

All in all, this is more trouble than it?s worth.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/e32a2e06/attachment-0001.html>

From encukou at gmail.com  Fri Aug 15 20:56:22 2014
From: encukou at gmail.com (Petr Viktorin)
Date: Fri, 15 Aug 2014 20:56:22 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
Message-ID: <CA+=+wqBdqan1LyfoVNxWh73tEaRtJJJ-33DXMbexTNRWzMFbFQ@mail.gmail.com>

On Fri, Aug 15, 2014 at 8:43 PM, Sunjay Varma <varma.sunjay at gmail.com> wrote:
> Hi all,
> Has the syntax for specifying type been fully decided on already?
>
> Using brackets may confuse new Python programmers. Since specifying type in
> Python is fairly new anyway, what do you all think of introducing angle
> brackets into Python instead? Other languages use angle brackets to specify
> types. It provides a good separation between type specification and list
> indexing.
>
> I'm also worried that using square brackets will cause confusion as that
> notation is generally associated with array declarations in other languages.
> Even in Python, MyClass[int] may be confused with getting a key called int
> from some MyClass.
>
> dict<str, int> seems to tell me more explicitly that I'm dealing with a
> declaration of an expected type. dict[str, int] looks very much like I'm
> getting an item (str, int) from some class.
>
> The angle bracket (or any other suggestions you have in mind) provides a
> more concrete separation between when we are performing item indexing and
> when we're specifying a type to validate.

Square brackets have the advantage of being valid Python now, so typed
code would be backwards compatible.

If the syntax was to change, what about a new operator?
    def sum(seq: Iterable of Number, start: Number):

    def print_grades(p: Mapping of (Student, Grade)):

Just an idea.

From kaiser.yann at gmail.com  Fri Aug 15 20:57:04 2014
From: kaiser.yann at gmail.com (Yann Kaiser)
Date: Fri, 15 Aug 2014 20:57:04 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CA+=+wqD+aZ2SYTidwDX3d2NJ-TqP=C3TnCxcLwMcZhyE96vnew@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <53EE3896.2050804@googlemail.com>
 <CA+=+wqD+aZ2SYTidwDX3d2NJ-TqP=C3TnCxcLwMcZhyE96vnew@mail.gmail.com>
Message-ID: <CANUJvPVka3Tm6XzJfJaJ3YCznTWkB8imKyz0Vw=WyCq07O+Xxg@mail.gmail.com>

So, if you need to change how many things you specify typing information
for depending on which parser you will use, how does putting those typing
information stub objects into the standard library advance anyone?


On 15 August 2014 19:11, Petr Viktorin <encukou at gmail.com> wrote:

> On Fri, Aug 15, 2014 at 6:43 PM, Dennis Brakhane
> <brakhane at googlemail.com> wrote:
> > On 15.08.2014 17:42, Petr Viktorin wrote:
> >>
> >> The common use is not all that concise:
> >>     def foo(bar: int | None=None): pass
> >>
> >> Or alternatively it could be:
> >>     def foo(bar: int=None): pass
> >> if the default was automatically allowed.
> > (Assuming you mean "the type of the default")
>
> I really meant *only* the default. This really only works for None,
> but that's a good thing, since something like:
>     def foo(bar:int=''): pass
> looks very suspicious. I'd be fine with the linter complaining about
> foo('hello').
>
> Of course you can always do:
>     def foo(bar: (int | str)=''): pass
>
> > While I like the second form a bit more, it kinda goes against "explicit
> > is better than implicit".
> >
> > Also, if I change the default value from None to 42, I've either changed
> > the allowable types,
> > or need to remember to turn "bar: int=None" into "bar: int|None = 42".
> >
> > Furthermore, what should happen in the following case:
> >
> >
> > # no annotations here
> > def foo(): ...
> >
> > def bar(evil: int = foo()): ...
> >
> >
> > Should this be disallowed, as the type checker will not be able to know
> > what type foo is? Should it just assume int?
> > And in the latter case, what should happen if foo now gains a "-> float"
> > annotation?
>
> If your linter can't figure it out, just specify the default's type
> explicitly. Always a good thing to do when something's not immediately
> obvious.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/3e4ea7e5/attachment.html>

From ryan at ryanhiebert.com  Fri Aug 15 20:56:17 2014
From: ryan at ryanhiebert.com (Ryan Hiebert)
Date: Fri, 15 Aug 2014 13:56:17 -0500
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
Message-ID: <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com>


> On Aug 15, 2014, at 1:43 PM, Sunjay Varma <varma.sunjay at gmail.com> wrote:
> 
> Using brackets may confuse new Python programmers. Since specifying type in Python is fairly new anyway, what do you all think of introducing angle brackets into Python instead? Other languages use angle brackets to specify types. It provides a good separation between type specification and list indexing.

Angle brackets already have meaning in Python, as comparison operators. The current surrounding operators ([], (), {}) require a matched pair in all cases. Breaking that rule would be confusing, though I know there are languages that do that.
> 
> I'm also worried that using square brackets will cause confusion as that notation is generally associated with array declarations in other languages. Even in Python, MyClass[int] may be confused with getting a key called int from some MyClass.
> 
> dict<str, int> seems to tell me more explicitly that I'm dealing with a declaration of an expected type. dict[str, int] looks very much like I'm getting an item (str, int) from some class.
> 
Getting an item from a class has no meaning for any classes that I?ve ever used, and I haven?t come up with any hypothetical one that would want to do that.

I think that the parallel between item access and item declaration is a great argument in favor of using the dict[str, int] (or perhaps dict[str: int]) syntax as a type declaration.

Ryan

From ethan at stoneleaf.us  Fri Aug 15 21:06:55 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 15 Aug 2014 12:06:55 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
 <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com>
Message-ID: <53EE5A4F.5060501@stoneleaf.us>

On 08/15/2014 11:56 AM, Ryan Hiebert wrote:
>
> Getting an item from a class has no meaning for any classes that I?ve ever used, and I haven?t come up with any hypothetical one that would want to do that.

--> class Foo(Enum):
...    spam = 'meat flavored'
...    eggs = 'chicken by-product'
...
--> Foo
<enum 'Foo'>

--> Foo['spam']
<Foo.spam: 'meat flavored'>

--
~Ethan~

From varma.sunjay at gmail.com  Fri Aug 15 21:11:49 2014
From: varma.sunjay at gmail.com (Sunjay Varma)
Date: Fri, 15 Aug 2014 15:11:49 -0400
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <2971B072-78B1-4E51-9B0F-50BFFDD66535@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
 <2971B072-78B1-4E51-9B0F-50BFFDD66535@langa.pl>
Message-ID: <CAJaQC31C0ywvr-J=fv1xtkw9+oTThjRO3AkCsq0n55mo6KwmnA@mail.gmail.com>

On Aug 15, 2014 2:55 PM, "?ukasz Langa" <lukasz at langa.pl> wrote:
>
> On Aug 15, 2014, at 11:43 AM, Sunjay Varma <varma.sunjay at gmail.com> wrote:
>>
>> Hi all,
>> Has the syntax for specifying type been fully decided on already?
>>
>> Using brackets may confuse new Python programmers. (?) Other languages
use angle brackets to specify types.
>
>
> I also agree that angle brackets would be nicer. Guido decided against it
for pragmatic reasons:
> 1. angle brackets would create Python source code incompatible with any
version lower than 3.5

I'm all for compatibility, but Python 3 already breaks compatibility with
Python 2. Why not add this feature in Python 4 (or whatever the next
breaking release is) and do it "right" the first time. I don't think it
makes sense to start muddling up the different semantic meanings of
Python's operations just because we think it will break in an older
version.

This is such a big and important change. It deserves its own syntax (and if
necessary a new version number as well).

> 2. angle brackets would complicate the lexer (normally you expect < and >
to be spaced, in this case it wouldn?t)
> 3. angle brackets would require a new mechanism in Python to store this
kind of expression within the type; this is still true for generics
expressed with square brackets but at least you can use the existing nuts
and bolts of Python classes

Angle brackets were just a suggestion as they are used frequently by other
languages. Even braces would be more appropriate as they're already built
into the lexer and dict{int, str} clearly means something different than
dict[int, str].

>
> All in all, this is more trouble than it?s worth.

I can understand that it's easier to use what's already there, but I don't
agree with doing something just because it's easier. Especially when the
side effects are not at all appealing.

Sunjay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/d4e1386b/attachment-0001.html>

From skip at pobox.com  Fri Aug 15 21:13:39 2014
From: skip at pobox.com (Skip Montanaro)
Date: Fri, 15 Aug 2014 14:13:39 -0500
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
Message-ID: <CANc-5UxHv-Z=8mjTrN-ShK85qjp7w+ncv=45kC=93O1OEjWQ1g@mail.gmail.com>

On Fri, Aug 15, 2014 at 1:43 PM, Sunjay Varma <varma.sunjay at gmail.com>
wrote:

> Using brackets may confuse new Python programmers. Since specifying type
> in Python is fairly new anyway, what do you all think of introducing angle
> brackets into Python instead?


Is this a facility which new programmers are likely to encounter right off
the bat or is it going to mostly be buried from casual view?

The use of paired angle brackets has been suggested over the years for
other purposes in Python. I no longer recall the arguments against them,
but ISTR issues with grammar complexity and syntax highlighting in editors.
Still, since C++ somehow managed to use them that way, perhaps all the
various tools which might be exposed to them have been fixed by now.

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/0ee6640e/attachment.html>

From lukasz at langa.pl  Fri Aug 15 21:14:33 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 15 Aug 2014 12:14:33 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53EE5A4F.5060501@stoneleaf.us>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
 <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com>
 <53EE5A4F.5060501@stoneleaf.us>
Message-ID: <A80AFA8D-CD3B-4487-A7F1-8A8EA28A7880@langa.pl>

On Aug 15, 2014, at 12:06 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 08/15/2014 11:56 AM, Ryan Hiebert wrote:
>> 
>> Getting an item from a class has no meaning for any classes that I?ve ever used, and I haven?t come up with any hypothetical one that would want to do that.
> 
> --> class Foo(Enum):
> ...    spam = 'meat flavored'
> ...    eggs = 'chicken by-product'
> ...
> --> Foo
> <enum 'Foo'>
> 
> --> Foo['spam']
> <Foo.spam: 'meat flavored?>

I also thought of enums. Looks fairly innocent to me, though. Do you see any cases where the two would conflict?

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/348e68bc/attachment.html>

From lukasz at langa.pl  Fri Aug 15 21:24:10 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Fri, 15 Aug 2014 12:24:10 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJLS_4Kr+7_SbdBXcEnjWVnLCvjm40FXMDLDC2d_DOaFAw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CAP7+vJLS_4Kr+7_SbdBXcEnjWVnLCvjm40FXMDLDC2d_DOaFAw@mail.gmail.com>
Message-ID: <A0A1D905-E138-4D54-B47C-A437C0EE3BAB@langa.pl>

On Aug 15, 2014, at 8:17 AM, Guido van Rossum <guido at python.org> wrote:

> You can probably come up with a notation for first-class variable annotations, e.g.
> 
>     x: Sequence[int] = []

Yes, that syntax is out of scope for now, though, right? If I understand your reasoning behind choosing Mypy?s function annotation syntax, we don?t want to create programs that require Python 3.5+ just to be parsed.

If we were to introduce first-class variable typing, yes, the syntax you propose is what I also had in mind.


> The value might be optional. The question is though, would the type (Sequence[int]) be stored anyway? Also, in a class body, does it define a class var or an instance var (or doesn't it matter?).

I wouldn?t change the current behaviour:

class C:
  cls_member: str = ?on the class?

  def __init__(self):
    self.obj_member: str = ?on the instance'
    self.cls_member = 2   # that?s the real question: type error or an instance member?

For that last case, even though it?s currently valid Python, my intuition tells me for Mypy to treat it as an error. 


> Does this need a 'var' keyword to be ambiguous?

I fail to see any additional value provided by such keyword. What would stop people from doing

var i = 1

I don?t think we want to end up with that.


> I propose to disallow declaring multiple variables in this style, since it's hard to decide whether the comma should bind tighter than the '=' sign (as in assignments) or less tight (as in function headings).

Right. I wonder if we even need this. For lines that use multiple assignment just for brevity, they can switch to multiple lines for typing. For common cases like:

host, port = origin.rsplit(?:?, 1)
successful, errors = query_the_world(hostnames)

I think the types can be easily inferred (assuming rsplit and query_the_world are annotated).


> While we?re at slaying dragons, I?ll also silently make str non-iterable so that we can use Sequence[str] meaningfully from now on? How about that?
> 
> I hope you meant that as a joke. We missed our chance for that one with Python 3.0. We must live with it.

Yes, that was obviously just a joke. By the way, is the PEP number 4000 free? Asking for a friend.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/62fdb37d/attachment.html>

From varma.sunjay at gmail.com  Fri Aug 15 21:25:53 2014
From: varma.sunjay at gmail.com (Sunjay Varma)
Date: Fri, 15 Aug 2014 15:25:53 -0400
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <A80AFA8D-CD3B-4487-A7F1-8A8EA28A7880@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
 <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com>
 <53EE5A4F.5060501@stoneleaf.us>
 <A80AFA8D-CD3B-4487-A7F1-8A8EA28A7880@langa.pl>
Message-ID: <CAJaQC326UXu+q9FGjdU6EQ6=gHeWHYTRMqPHamDpLZP7_7VCHw@mail.gmail.com>

Another such example, since names are just names:

    dict = {"a": 2}
    print(dict["a"])  # 2

Overwriting a built in type name is bad, but entirely possible.

dict["a"] here is also confusing with dict[str]. This kind of use could
also potentially throw off a type linter too.

These probably aren't the best examples out there, but I can definitely see
this operator's meaning becoming very confused as more people start to
apply it in different ways.

We should not be just using something because it's there. Especially if it
causes other problems.
list[str] may be valid syntax in old Python 3 versions, but it's still not
going to be correct if used in those versions. You're going to get some
breakage no matter what.

This feature is very new to Python as a whole, why not give it a syntax
that provides a proper separation from what already was?

Sunjay
On Aug 15, 2014 3:14 PM, "?ukasz Langa" <lukasz at langa.pl> wrote:

> On Aug 15, 2014, at 12:06 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
>
> On 08/15/2014 11:56 AM, Ryan Hiebert wrote:
>
>
> Getting an item from a class has no meaning for any classes that I?ve ever
> used, and I haven?t come up with any hypothetical one that would want to do
> that.
>
>
> --> class Foo(Enum):
> ...    spam = 'meat flavored'
> ...    eggs = 'chicken by-product'
> ...
> --> Foo
> <enum 'Foo'>
>
> --> Foo['spam']
> <Foo.spam: 'meat flavored?>
>
>
> I also thought of enums. Looks fairly innocent to me, though. Do you see
> any cases where the two would conflict?
>
> --
> Best regards,
> ?ukasz Langa
>
> WWW: http://lukasz.langa.pl/
> Twitter: @llanga
> IRC: ambv on #python-dev
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/9c86f82f/attachment-0001.html>

From ryan at ryanhiebert.com  Fri Aug 15 21:24:09 2014
From: ryan at ryanhiebert.com (Ryan Hiebert)
Date: Fri, 15 Aug 2014 14:24:09 -0500
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53EE5A4F.5060501@stoneleaf.us>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
 <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com>
 <53EE5A4F.5060501@stoneleaf.us>
Message-ID: <F654DBF5-9220-47DE-A82B-A4FC38B201E5@ryanhiebert.com>


> On Aug 15, 2014, at 2:06 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
> 
> On 08/15/2014 11:56 AM, Ryan Hiebert wrote:
>> 
>> Getting an item from a class has no meaning for any classes that I?ve ever used, and I haven?t come up with any hypothetical one that would want to do that.
> 
> --> class Foo(Enum):
> ...    spam = 'meat flavored'
> ...    eggs = 'chicken by-product'
> ...
> --> Foo
> <enum 'Foo'>
> 
> --> Foo['spam']
> <Foo.spam: 'meat flavored?>

Well thanks for that ;-)

I don?t think it would conflict in this case, since I don?t think there?d be a reason to mix Enum item access with Type item access. Especially because the Enum itself might be used in the type signature (though not item access on the Enum, I?d think).

I think it?s enough of an edge-case that I still like the parallel it provides, but I appreciate the reality check.


From guido at python.org  Fri Aug 15 21:31:26 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Aug 2014 12:31:26 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53EE5A4F.5060501@stoneleaf.us>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
 <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com>
 <53EE5A4F.5060501@stoneleaf.us>
Message-ID: <CAP7+vJ+=R_bY0rjvNUb+kHPuHuPXsX=djBqQtf4r_rxYbDEYBw@mail.gmail.com>

On Fri, Aug 15, 2014 at 12:06 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 08/15/2014 11:56 AM, Ryan Hiebert wrote:
>
>>
>> Getting an item from a class has no meaning for any classes that I?ve
>> ever used, and I haven?t come up with any hypothetical one that would want
>> to do that.
>>
>
> --> class Foo(Enum):
> ...    spam = 'meat flavored'
> ...    eggs = 'chicken by-product'
> ...
> --> Foo
> <enum 'Foo'>
>
> --> Foo['spam']
> <Foo.spam: 'meat flavored'>
>

That's a little unfortunate, but I don't think it's harmful, as I don't
expect a use case for parametrized enums. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/3bc2c9f2/attachment.html>

From guido at python.org  Fri Aug 15 21:33:07 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Aug 2014 12:33:07 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAJaQC31C0ywvr-J=fv1xtkw9+oTThjRO3AkCsq0n55mo6KwmnA@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
 <2971B072-78B1-4E51-9B0F-50BFFDD66535@langa.pl>
 <CAJaQC31C0ywvr-J=fv1xtkw9+oTThjRO3AkCsq0n55mo6KwmnA@mail.gmail.com>
Message-ID: <CAP7+vJJ2Bvxips=wDDQJAZ0g-eQf3vo4FV6om9hxM1X_sgcUqg@mail.gmail.com>

On Fri, Aug 15, 2014 at 12:11 PM, Sunjay Varma <varma.sunjay at gmail.com>
wrote:

>
> On Aug 15, 2014 2:55 PM, "?ukasz Langa" <lukasz at langa.pl> wrote:
> >
> > On Aug 15, 2014, at 11:43 AM, Sunjay Varma <varma.sunjay at gmail.com>
> wrote:
> >>
> >> Hi all,
> >> Has the syntax for specifying type been fully decided on already?
> >>
> >> Using brackets may confuse new Python programmers. (?) Other languages
> use angle brackets to specify types.
> >
> >
> > I also agree that angle brackets would be nicer. Guido decided against
> it for pragmatic reasons:
> > 1. angle brackets would create Python source code incompatible with any
> version lower than 3.5
>
> I'm all for compatibility, but Python 3 already breaks compatibility with
> Python 2. Why not add this feature in Python 4 (or whatever the next
> breaking release is) and do it "right" the first time. I don't think it
> makes sense to start muddling up the different semantic meanings of
> Python's operations just because we think it will break in an older
> version.
>
There won't *be* a "next breaking release".

> This is such a big and important change. It deserves its own syntax (and
> if necessary a new version number as well).
>
> > 2. angle brackets would complicate the lexer (normally you expect < and
> > to be spaced, in this case it wouldn?t)
> > 3. angle brackets would require a new mechanism in Python to store this
> kind of expression within the type; this is still true for generics
> expressed with square brackets but at least you can use the existing nuts
> and bolts of Python classes
>
> Angle brackets were just a suggestion as they are used frequently by other
> languages. Even braces would be more appropriate as they're already built
> into the lexer and dict{int, str} clearly means something different than
> dict[int, str].
>
> >
> > All in all, this is more trouble than it?s worth.
>
> I can understand that it's easier to use what's already there, but I don't
> agree with doing something just because it's easier. Especially when the
> side effects are not at all appealing.
>
I don't think you quite appreciate the art of language evolution.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/159bdae4/attachment.html>

From haoyi.sg at gmail.com  Fri Aug 15 21:33:20 2014
From: haoyi.sg at gmail.com (Haoyi Li)
Date: Fri, 15 Aug 2014 12:33:20 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAJaQC326UXu+q9FGjdU6EQ6=gHeWHYTRMqPHamDpLZP7_7VCHw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
 <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com>
 <53EE5A4F.5060501@stoneleaf.us>
 <A80AFA8D-CD3B-4487-A7F1-8A8EA28A7880@langa.pl>
 <CAJaQC326UXu+q9FGjdU6EQ6=gHeWHYTRMqPHamDpLZP7_7VCHw@mail.gmail.com>
Message-ID: <CALruUQJtiR27nNPDbnO6eMLXPnckgtiJVQoaMyNbHisc3j31aA@mail.gmail.com>

> This feature is very new to Python as a whole, why not give it a syntax
that provides a proper separation from what already was?

Mainly because angle brackets are one of C++'s
<http://stackoverflow.com/questions/7304699/c-templates-angle-brackets-problems>
 worst <http://blog.aaronballman.com/2011/05/semantic-whitespace/> mistakes
<http://stackoverflow.com/questions/15785496/c-templates-angle-brackets-pitfall-what-is-the-c11-fix>.
Well, worst among many other equally-worst things, but it's pretty bad. Say
goodbye to your simple LL1 parser!


On Fri, Aug 15, 2014 at 12:25 PM, Sunjay Varma <varma.sunjay at gmail.com>
wrote:

> Another such example, since names are just names:
>
>     dict = {"a": 2}
>     print(dict["a"])  # 2
>
> Overwriting a built in type name is bad, but entirely possible.
>
> dict["a"] here is also confusing with dict[str]. This kind of use could
> also potentially throw off a type linter too.
>
> These probably aren't the best examples out there, but I can definitely
> see this operator's meaning becoming very confused as more people start to
> apply it in different ways.
>
> We should not be just using something because it's there. Especially if it
> causes other problems.
> list[str] may be valid syntax in old Python 3 versions, but it's still not
> going to be correct if used in those versions. You're going to get some
> breakage no matter what.
>
> This feature is very new to Python as a whole, why not give it a syntax
> that provides a proper separation from what already was?
>
> Sunjay
> On Aug 15, 2014 3:14 PM, "?ukasz Langa" <lukasz at langa.pl> wrote:
>
>> On Aug 15, 2014, at 12:06 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>> On 08/15/2014 11:56 AM, Ryan Hiebert wrote:
>>
>>
>> Getting an item from a class has no meaning for any classes that I?ve
>> ever used, and I haven?t come up with any hypothetical one that would want
>> to do that.
>>
>>
>> --> class Foo(Enum):
>> ...    spam = 'meat flavored'
>> ...    eggs = 'chicken by-product'
>> ...
>> --> Foo
>> <enum 'Foo'>
>>
>> --> Foo['spam']
>> <Foo.spam: 'meat flavored?>
>>
>>
>> I also thought of enums. Looks fairly innocent to me, though. Do you see
>> any cases where the two would conflict?
>>
>> --
>> Best regards,
>> ?ukasz Langa
>>
>> WWW: http://lukasz.langa.pl/
>> Twitter: @llanga
>> IRC: ambv on #python-dev
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/cbbbb0e1/attachment-0001.html>

From guido at python.org  Fri Aug 15 21:46:27 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Aug 2014 12:46:27 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <A0A1D905-E138-4D54-B47C-A437C0EE3BAB@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CAP7+vJLS_4Kr+7_SbdBXcEnjWVnLCvjm40FXMDLDC2d_DOaFAw@mail.gmail.com>
 <A0A1D905-E138-4D54-B47C-A437C0EE3BAB@langa.pl>
Message-ID: <CAP7+vJK3Ok_PJYXjnhQcYDScwqS24LUwtLa5M+8abKBj3gPu-g@mail.gmail.com>

On Fri, Aug 15, 2014 at 12:24 PM, ?ukasz Langa <lukasz at langa.pl> wrote:

> On Aug 15, 2014, at 8:17 AM, Guido van Rossum <guido at python.org> wrote:
>
> You can probably come up with a notation for first-class variable
> annotations, e.g.
>
>     x: Sequence[int] = []
>
> Yes, that syntax is out of scope for now, though, right? If I understand
> your reasoning behind choosing Mypy?s function annotation syntax, we don?t
> want to create programs that require Python 3.5+ just to be parsed.
>

That's stronger than I meant it. The List<T> proposal would completely
prevent the new typing syntax from being backported. Having to refrain from
adding types for variables (or being required to use the inferior "magic
comment" syntax) is a much smaller burden.


> If we were to introduce first-class variable typing, yes, the syntax you
> propose is what I also had in mind.
>

It might be a separate PEP.

>  The value might be optional. The question is though, would the type
> (Sequence[int]) be stored anyway? Also, in a class body, does it define a
> class var or an instance var (or doesn't it matter?).
>
>
> I wouldn?t change the current behaviour:
>
> class C:
>   cls_member: str = ?on the class?
>
>   def __init__(self):
>     self.obj_member: str = ?on the instance'
>     self.cls_member = 2   # that?s the real question: type error or an
> instance member?
>
> For that last case, even though it?s currently valid Python, my intuition
> tells me for Mypy to treat it as an error.
>

I disagree -- it's a very common idiom to set (immutable) default values on
the class for what is meant to be an instance variable. This is why I
called it out as a question we need to answer.

> Does this need a 'var' keyword to be ambiguous?
>
> I fail to see any additional value provided by such keyword. What would
> stop people from doing
>
> var i = 1
>
> I don?t think we want to end up with that.
>

In a different world it could be used to address the issue of typos going
unnoticed, but I think it would be too big a departure from current PYthon
practice.


> I propose to disallow declaring multiple variables in this style, since
> it's hard to decide whether the comma should bind tighter than the '=' sign
> (as in assignments) or less tight (as in function headings).
>
>
> Right. I wonder if we even need this. For lines that use multiple
> assignment just for brevity, they can switch to multiple lines for typing.
> For common cases like:
>
> host, port = origin.rsplit(?:?, 1)
> successful, errors = query_the_world(hostnames)
>
> I think the types can be easily inferred (assuming rsplit and
> query_the_world are annotated).
>

Sure. It's just that people would be expecting it to work based on
generalizations from other forms -- it's just that if you generalize from
argument lists you end up with something different than when you generalize
from assignment. I think it's reasonable to disallow

  a, b: Tuple[int, float] = 42, 3.14

but to allow

  (a, b): Tuple[int, float] = (42, 3.14)


>  While we?re at slaying dragons, I?ll also silently make str non-iterable
>> so that we can use Sequence[str] meaningfully from now on? How about that?
>>
>
> I hope you meant that as a joke. We missed our chance for that one with
> Python 3.0. We must live with it.
>
>
> Yes, that was obviously just a joke. By the way, is the PEP number 4000
> free? Asking for a friend.
>

I totally missed the joke. :-(

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/8bc0cfdb/attachment.html>

From tjreedy at udel.edu  Fri Aug 15 23:18:11 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 15 Aug 2014 17:18:11 -0400
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
 <lskkem$8m7$1@ger.gmane.org>
 <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>
Message-ID: <lsltff$aba$1@ger.gmane.org>

On 8/15/2014 5:48 AM, Nick Coghlan wrote:
> On 15 August 2014 19:38, Terry Reedy <tjreedy at udel.edu> wrote:
>> On 8/15/2014 12:40 AM, Nick Coghlan wrote:
>>>
>>> On 15 August 2014 09:56, Guido van Rossum <guido at python.org> wrote:
>>>>
>>>>
>>>> I don't buy the argument that PEP 3107 promises that annotations are
>>>> completely free of inherent semantics.
>>>
>>>
>>> It's also worth noting the corresponding bullet point in PEP 3100
>>> (under http://www.python.org/dev/peps/pep-3100/#core-language):
>>>
>>> * Add optional declarations for static typing [45] [10] [done]
>>
>> ...
>>
>>> Linters/checkers may also want to provide a configurable way to say
>>> "the presence of decorator <X> means the annotations on that function
>>> aren't type markers". That ties in with the recommendation we added to
>>> PEP 8 a while back: "It is recommended that third party experiments
>>> with annotations use an associated decorator to indicate how the
>>> annotation should be interpreted."

I claim that the mere presence of a decorator in the *source* is not 
enough. The decorator for non-type markers should do something with 
.__annotations__ -- in particular, remove the non-type markers.  The 
presence of a decorator in the source does not necessarily leave a trace 
on the returned function object for runtime detection.

>> Depending on the checker, this suggests that non-type-check annotations need
>> not be deprecated. If a decorator wraps a function with an unannotated
>> wrapper, then the checker should see the result as unannotated, rather than
>> looking for a wrapped attribute. Also, a decorator can remove non-type
>> annotations and act on them, store them in a closure variable, or store them
>> on the function in a different name.

"Depending on the checker" alludes to the fact that there are two types 
of annotation consumers: those that read the source or parsed source 
(ast) and the annotations therein and those that look at the function 
.__annotations__ attribute after compilation.  Inspect.signature is an 
example of the latter. (If there were none, there would be no purpose to 
.__annotations__!)

Suppose an integer square root function were annotated with with type 
and non-type information.

 >>> def nsqrt(n:(int, 'random non-type info'))->(int, 'more non-type 
info'): pass

To me, inspect.signature already assumes that annotations are about 
type, which means that this horse has already left barn. In 3.4.1:

 >>> from inspect import signature as sig
 >>> str(sig(nsqrt))
"(n:(<class 'int'>, 'random non-type info')) -> (<class 'int'>, 'more 
non-type info')"

Typing 'nsqrt(' in Idle and pausing a fraction of a second brings up a 
calltip with the same string (without the outer quotes). To me, having 
random non-type info in the signature and calltip is noise and therefore 
wrong. So I agree that the non-standard annotation should be signaled by 
a decorator *and* suggest that the decorator should remove the 
non-standard annotation, which is easily done, so that the signature 
string for the above would be

"(n:<class 'int') -> <class 'int'>"

I summarized the above, perhaps not the best I could have, with

"Given these possibilities, all that is needs be said is "After a 
function is post-processed by decorators, any remaining annotations 
should be for type-checking or documentation."

> No, many (most?)

The future relative proportion of pre- and post-compile annotation 
consumers is not relevant to my argument. The stdlib already has a 
important post-compile consumer that is already somewhat broken by 
non-type info remaining in .__annotations__.

 > linters and IDEs will run off the AST without
> actually executing the code, so they'll see the annotations, even if
> they get stripped by the decorator at runtime.

Being aware of this, I concluded the post with the following that 
already said this.

"For checkers that do look at the source, or the AST before compiling,"

*and* I went on to suggest a solution.

"the rule could be to ignore string annotations. Decorators can always 
eval, or perhaps safe_eval, strings."

In other words, if type annotations were to be classes, as proposed, 
then non-type annotations should not be classes, so that pre-compile 
annotation consumers could easily ignore them. In particular, I 
suggested literal strings, which are easily recognized in source, as 
well as in asts.


To put this all another way --

The new-in-3.0 annotation feature has two components: a python function 
source syntax, and a function compile behavior of adding a dict 
attribute as .__annotations__.  If I understand correctly, Argument 
Clinic piggybacks on this by adding a mechanism to produce 
.__annotations__ from C source -- mainly for use by .signature, but also 
by another other .__annotations__ users. The two components -- source 
annotations and .__annotations__ dict -- each have their consumers.

Currently, annotation values are untyped (or AnyTyped).  Guido has 
proposed favoring type annotations. I support that and suggest that such 
favoritism is already needed for the annotation dict. So I would 
strengthen the PEP 8 recommendation.

Guido has also suggested 'deprecating' non-type annotations.  That would 
literally mean raising an error either when source is compiled or when 
def statements are executed.  I think deprecation in this sense is both 
unwise, since non-type annotation were explicitly invited, and 
unnecessary for the purpose of favoring type annotations. The point of 
my previous post was to explore what restrictions *are* necessary.  In 
summary, I suggest 1. use distinct syntax (this depends on what is 
adopted for type annotations); 2. decorate (as already suggested in PEP 
8); 3. clean .__annotations__ (which should also be suggested in PEP 8).

-- 
Terry Jan Reedy


From barry at python.org  Sat Aug 16 00:17:37 2014
From: barry at python.org (Barry Warsaw)
Date: Fri, 15 Aug 2014 18:17:37 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAGE7PN+TyBsUfE_yjZR5VLsJ4oNkJ6zgr_gR_KkGZGvSGYSkTQ@mail.gmail.com>
 <CAP7+vJJwsR0cWPhwuPcX6gk1m-CNaOySY2BaLLb_pUZkAGqpLg@mail.gmail.com>
Message-ID: <20140815181737.1e1279b4@anarchist.localdomain>

On Aug 13, 2014, at 10:11 PM, Guido van Rossum wrote:

>We could refrain from using type annotations in the stdlib (similar to how
>we refrain from using Unicode identifiers). Mypy's stubs mechanism makes it
>possible to ship the type declarations for stdlib modules with mypy instead
>of baking them into the stdlib.

That would address my visceral reaction to the proposal.  I haven't had time
to read the entire thread, or supporting material, but my main concern is
about readability.  I don't think it's clear whether the impact of adding type
annotations is a pure win or loss.

OT1H maybe it helps because you would have to read less of the body of the
code to guess what the implicit types are for the arguments and return values
of a function.  I myself often have to "annotate" (i.e. add comments) around
code such as:

    # Map UUIDs to user object.
    users_by_uid = {}

Admittedly, it's often quite difficult when reading foreign code to understand
what are the types of the various constructs involved.  It can involve reading
more deeply into the call structure, or as I usually find, writing a bit of
exploratory code and then just stepping through with the debugger.  So type
annotations could be a win here.

OTOH there *is* a cost that could negatively impact readability.  It's more
text visually assaulting you. :) It can reduce the speed at which you can scan
code.  I'd say most of the time, you can pretty much deduce what's happening
fairly easily (with molasses-inducing outliers like above) so type annotations
may make things more confusing, especially if you have to mentally map between
annotation types and run time types that are similar but not quite the same.
If even more complex types are used, then you have to go digging into other
material (docs or code) to understand what the author meant by some of the
annotations.  Type annotations likely increase line length or vertical
real-estate, both of which can negatively impact readability.  And if
annotations are buggy, then it imposes even more cognitive dissonance.

In practice, who knows whether wholesale adoption of type annotation in the
stdlib *code* would be a net win or loss?  I don't think it's at all clear.

The other concern I'd have is whether annotating new stdlib code would become
mandatory, and whether there'd be a wholesale campaign to annotate existing
code.  Using stubs and not baking them into the stdlib alleviates those
concerns too.

My gut reaction is more negative than positive but I remain open minded until
I've had time to read more.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/8e2a877e/attachment-0001.sig>

From barry at python.org  Sat Aug 16 00:33:00 2014
From: barry at python.org (Barry Warsaw)
Date: Fri, 15 Aug 2014 18:33:00 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
Message-ID: <20140815183300.04a3ad33@anarchist.localdomain>

On Aug 14, 2014, at 12:01 PM, Sunjay Varma wrote:

>Here's a taste of what that looks like:
>    class SimpleEquation(object):
>
>        def demo(self, a, b, c):
>            """
>            This function returns the product of a, b and c
>            @type self: SimpleEquation
>            :param a: int - The first number
>            :param b: int
>            :param c: int - The third number should not be zero and should
>also
>                only be -1 if you enjoy carrots (this comment spans 2 lines)
>            :return: int
>            """
>            return a * b * c

I use this all the time.  I think I originally adopted this from epydoc:

http://epydoc.sourceforge.net/manual-fields.html

(i.e. the reStructuredText flavor of epydoc fields.)

E.g.

def inject_message(mlist, msg, recipients=None, switchboard=None, **kws):
    """Inject a message into a queue.

    If the message does not have a Message-ID header, one is added.  An
    X-Message-Id-Hash header is also always added.

    :param mlist: The mailing list this message is destined for.
    :type mlist: IMailingList
    :param msg: The Message object to inject.
    :type msg: a Message object
    :param recipients: Optional set of recipients to put into the message's
        metadata.
    :type recipients: sequence of strings
    :param switchboard: Optional name of switchboard to inject this message
        into.  If not given, the 'in' switchboard is used.
    :type switchboard: string
    :param kws: Additional values for the message metadata.
    :type kws: dictionary
    """

With perhaps a little more formalism (or care on the part of the author
<wink>), e.g. a better spelling of "sequence of strings", this format seems
much more readable to me.  And being tucked away in a docstring, it can really
be safely ignored.  It also seems like a processor like mypy could use this
information just as easily as type annotations.

For whatever reason, this style seems much more comfortable to me than trying
to encode everything in the function signature.  it also has the added benefit
of actually describing the purpose and detail of the arguments to a human
reader rather than leaving it up to a mental mapping to translate annotated
types to semantics or content detail.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/1d4356ec/attachment.sig>

From barry at python.org  Sat Aug 16 00:40:44 2014
From: barry at python.org (Barry Warsaw)
Date: Fri, 15 Aug 2014 18:40:44 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
Message-ID: <20140815184044.2aaeca8e@anarchist.localdomain>

On Aug 15, 2014, at 04:15 AM, Steven D'Aprano wrote:

>> Here's a taste of what that looks like:
>>     class SimpleEquation(object):
>> 
>>         def demo(self, a, b, c):
>>             """
>>             This function returns the product of a, b and c
>>             @type self: SimpleEquation
>>             :param a: int - The first number
>>             :param b: int
>>             :param c: int - The third number should not be zero and should
>> also
>>                 only be -1 if you enjoy carrots (this comment spans 2 lines)
>>             :return: int
>>             """
>>             return a * b * c
>
>I really dislike that syntax. I dislike adding cruft like "@type" and
>":param" into docstrings, which should be written for human readers, not
>linters.

For me, the whole point of using syntax like this is for the human reader,
especially because I don't have a linter that parses this... yet. <wink>  Used
judiciously, this syntax could benefit both the human reader *and* automated
tools.

>I dislike that you have documented that self is a SimpleEquation. (What else
>could it be?)

FWIW, I never document 'self'.

>I dislike that the syntax clashes with ReST syntax.

It needn't.  http://epydoc.sourceforge.net/manual-fields.html

>I dislike that it isn't obvious to me why the first parameter uses @type
>while the second parameter uses :param.

It needn't.

>I like the annotation syntax. I'm not completely convinced that the mypy
>syntax is mature enough to bless, but the basic idea of type annotations is
>pretty common in dozens of languages. I think you are in a tiny minority if
>you think that putting the type declaration right next to the parameter make
>it *less* clear that putting the type declaration in a completely different
>part of the code.

Of course, it's not a completely different part of the code.  docstrings
naturally live right after the function signature (indeed, or it wouldn't get
stuffed into __doc__), so it's always close to the source.  That makes it
quite easy for the third party human reader, but also for the author to keep
up-to-date.

Plus, you can *always* fit these reST-ish epydoc field descriptions in PEP 8
line lengths.  It always bugs me to see multiline function signatures.
Currently those are pretty rare (and IMHO are a code smell), but with type
annotations, I suspect they'll be the norm.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/6665bd3b/attachment.sig>

From barry at python.org  Sat Aug 16 00:49:09 2014
From: barry at python.org (Barry Warsaw)
Date: Fri, 15 Aug 2014 18:49:09 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando> <53ED0109.6070207@stoneleaf.us>
Message-ID: <20140815184909.61c4ee53@anarchist.localdomain>

On Aug 14, 2014, at 11:33 AM, Ethan Furman wrote:

>> def frobinate(x, y):
>>      """Return the frobinated x and y.
>>
>>      Some more text goes here. Perhaps lots of text.
>>
>>      :param x: Spam
>>      :param y: Eggs
>>      :return: Breakfast
>>      """
>
>Sure, keeping that info in the annotations makes more sense, but I'd rather
>see it in the doc string instead of ruling out all other possible uses of
>annotations -- particularly for something that's supposed to be /optional/.

Docstring annotations almost by definition can contain more information useful
to the human reader than type annotations can, especially if you carefully use
the reST-ish epydoc convention of both :param: and :type:.  The latter
contains the type annotation (which an automated system could utilize) while
the former contains the exposition (for the benefit of the human reader).
It's the explanations that are missing from any type annotations.

I suppose you could intersperse comments with your type annotations, resulting
in a long multiline function signature.  I doubt that would be a readability
improvement.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/353bc1fd/attachment.sig>

From greg.ewing at canterbury.ac.nz  Sat Aug 16 00:51:38 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 16 Aug 2014 10:51:38 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
Message-ID: <53EE8EFA.7040005@canterbury.ac.nz>

?ukasz Langa wrote:
> I?m also not a fan of the square brackets for generics (those brackets 
> mean lookup!) but a BDFL once said that ?language evolution is the art 
> of compromise? and one cannot disagree with that.

I think there's sufficient precedent in other languages
to justify using square brackets that way. I much prefer
them over the angle brackets of C++ and Java, which are
ugly to my eyes and make the code hard to read. They
also make parsing tricky if you have "<<" and ">>"
operators.

-- 
Greg

From kaiser.yann at gmail.com  Sat Aug 16 00:44:27 2014
From: kaiser.yann at gmail.com (Yann Kaiser)
Date: Sat, 16 Aug 2014 00:44:27 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <lsltff$aba$1@ger.gmane.org>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
 <lskkem$8m7$1@ger.gmane.org>
 <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>
 <lsltff$aba$1@ger.gmane.org>
Message-ID: <CANUJvPUoECZnYNEUYtR4A9p3dN1yRRrinqHfPSx2yWoh9jzgmw@mail.gmail.com>

On 15 August 2014 23:18, Terry Reedy <tjreedy at udel.edu> wrote:
>
>
> I claim that the mere presence of a decorator in the *source* is not
> enough. The decorator for non-type markers should do something with
> .__annotations__ -- in particular, remove the non-type markers.  The
> presence of a decorator in the source does not necessarily leave a trace on
> the returned function object for runtime detection.
>
>
The way I understand it, mypy, which is what Guido's proposal sees its main
potential user, operates at a stage similar to compilation in CPython. At
which point anything after the colon in a parameter would be in its
equivalent of __annotation__, regardless of what decorators add or remove.
How would your other-annotation-removing decorator help mypy at all? By
having it examine the decorator source and infer what kind of operation the
decorator does? That doesn't sound very compile-stagey at all.



>
> To me, inspect.signature already assumes that annotations are about type,
> which means that this horse has already left barn. In 3.4.1:
>
> >>> from inspect import signature as sig
> >>> str(sig(nsqrt))
> "(n:(<class 'int'>, 'random non-type info')) -> (<class 'int'>, 'more
> non-type info')"
>
>
`inspect.signature` makes no such assumption, it only relays what it found
on the function's __annotation__ attribute. I don't know where this nsqrt
function comes from, but it is responsible for having set up those
annotations, which seem to be mere documentation if the "random non-type
info" is a string.


> "the rule could be to ignore string annotations. Decorators can always
> eval, or perhaps safe_eval, strings."
>
> In other words, if type annotations were to be classes, as proposed, then
> non-type annotations should not be classes, so that pre-compile annotation
> consumers could easily ignore them. In particular, I suggested literal
> strings, which are easily recognized in source, as well as in asts.
>
>
"Oh, everything that's not type-checking can be expressed in a string,
maybe to be eval'ed, say, with sys,_getframe(-1) locals."

No, and that's incredibly ugly and may not work on alternate
implementations of Python.

If a PEP is made to standardize typing attributes for annotations, perhaps
in the form of a typing type hierarchy or a type decorator(ie.
typing.Like(AClass)), or annotation namespacing of some sort(decorate the
function with "hey! please typecheck me!"?), then couldn't tools that rely
on those attributes pick out what's relevant to them on their own, thanks
to the standardization?

I find the whole idea of having so far equal uses of function annotations
be bullied aside for the good of a concept yet foreign to Python very
arrogant and unnecessary.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/9e17bac2/attachment-0001.html>

From ethan at stoneleaf.us  Sat Aug 16 01:46:55 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 15 Aug 2014 16:46:55 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140815184909.61c4ee53@anarchist.localdomain>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando> <53ED0109.6070207@stoneleaf.us>
 <20140815184909.61c4ee53@anarchist.localdomain>
Message-ID: <53EE9BEF.8040001@stoneleaf.us>

On 08/15/2014 03:49 PM, Barry Warsaw wrote:
> On Aug 14, 2014, at 11:33 AM, Ethan Furman wrote:
>
>>> def frobinate(x, y):
>>>       """Return the frobinated x and y.
>>>
>>>       Some more text goes here. Perhaps lots of text.
>>>
>>>       :param x: Spam
>>>       :param y: Eggs
>>>       :return: Breakfast
>>>       """
>>
>> Sure, keeping that info in the annotations makes more sense, but I'd rather
>> see it in the doc string instead of ruling out all other possible uses of
>> annotations -- particularly for something that's supposed to be /optional/.
>
> Docstring annotations almost by definition can contain more information useful
> to the human reader than type annotations can, especially if you carefully use
> the reST-ish epydoc convention of both :param: and :type:.  The latter
> contains the type annotation (which an automated system could utilize) while
> the former contains the exposition (for the benefit of the human reader).
> It's the explanations that are missing from any type annotations.
>
> I suppose you could intersperse comments with your type annotations, resulting
> in a long multiline function signature.  I doubt that would be a readability
> improvement.

Sounds like what we *really* need is a decorator that will parse the docstring and fill in the annotations 
automatically.  :)

--
~Ethan~

From greg.ewing at canterbury.ac.nz  Sat Aug 16 01:57:48 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 16 Aug 2014 11:57:48 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
Message-ID: <53EE9E7C.6000004@canterbury.ac.nz>

Petr Viktorin wrote:
> Instead of
> 
>     def optional(t):
>         return t | None
> 
> it's now:
> 
>     def optional(t):
>         if t is None:
>             return t
>         else:
>             return t | None

I don't think so. Code that's manipulating types shouldn't be
using None as a stand-in for NoneType in the first place.

Think of it this way: None is *not* a type, just a special-case
value of x in "<type> | x". So optional(None) is an error, just
as much as optional(42) would be.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sat Aug 16 02:10:54 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 16 Aug 2014 12:10:54 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
Message-ID: <53EEA18E.3050600@canterbury.ac.nz>

Sunjay Varma wrote:
> dict<str, int> seems to tell me more explicitly that I'm dealing with a 
> declaration of an expected type.

In addition to my earlier objections to angle brackets,
there would be a big problem with parsing this notation
in Python.

In languages that use syntax like this, there is a clear
division between type descriptions and expressions --
they belong to completely separate areas of the grammar.

However, we need to be able to parse our type descriptions
as expressions, because they *are* expressions, just like
any other.

Now consider:

    dict<str, int>
    ^^^^^^^^^^^^^

Everything up to the final '>' is a valid Python expression,
equivalent to

    ((dict < str), int)

so the parser would have to be capable of completely changing
its mind about how to interpret all of that when it saw the
'>'. I'm fairly sure that Python's mostly-LL parser can't do
that.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sat Aug 16 02:14:36 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 16 Aug 2014 12:14:36 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
Message-ID: <53EEA26C.3040907@canterbury.ac.nz>

Sunjay Varma wrote:
> dict[str, int] looks very much like I'm 
> getting an item (str, int) from some class.

If you consider that 'dict' on its own represents a set of
possible types, then it's not unreasonable that 'dict[str, int]'
selects one of the types from that set.

In other words, dict is a lazy collection of types.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sat Aug 16 02:20:11 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 16 Aug 2014 12:20:11 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <F654DBF5-9220-47DE-A82B-A4FC38B201E5@ryanhiebert.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CA+=+wqAr3O45Aj6sOrqO6z-2VUmvkeMNn6XzRUE3jDj5r=LOmw@mail.gmail.com>
 <CAP7+vJKQUefWT7CpdhSaoY1sGE0Mk4kAf+1FoMR-HcuDEB9MVQ@mail.gmail.com>
 <CA+=+wqBAs=ON7gr9sPnYa7CaSGJ9Q0K74f2hGeJnbpe4OCOH=g@mail.gmail.com>
 <CAP7+vJJZ7PcvfkZugXuQ7O7YOZTOiGNjE20UYHU9TQWGspUSUQ@mail.gmail.com>
 <CA+=+wqC0j5+00Q-4hzjdDzHD_-VTfqnJ3kGc8RcmqPGmZJMnPw@mail.gmail.com>
 <CAP7+vJJaroLgu62TxY2j1da08NLgo1wBasMBV-RE7PWmSR6qBA@mail.gmail.com>
 <CA+=+wqBVCLZwzOsVLq6Ai3W6edE1McowUX4tTXzDgjy_KHJf1Q@mail.gmail.com>
 <CAP7+vJKS6=aF6RpsYb6C6GM8KfR9eSJPG+DoR40HSi+CnispBQ@mail.gmail.com>
 <CAJaQC33xro0q50pZTQ4S8NQfo6dpnNGPbqza1f_DHPhkPvJSBQ@mail.gmail.com>
 <6B90B508-7961-4623-B2B3-FCF1BA9BBB17@ryanhiebert.com>
 <53EE5A4F.5060501@stoneleaf.us>
 <F654DBF5-9220-47DE-A82B-A4FC38B201E5@ryanhiebert.com>
Message-ID: <53EEA3BB.5030304@canterbury.ac.nz>

Ryan Hiebert wrote:
> I don?t think it would conflict in this case, since I don?t think there?d be
> a reason to mix Enum item access with Type item access.

It would conflict if you somehow needed to define an
Enum subclass with type parameters. I'm having trouble
thinking of a use case for such a thing, though.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sat Aug 16 02:26:09 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 16 Aug 2014 12:26:09 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <A0A1D905-E138-4D54-B47C-A437C0EE3BAB@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CAP7+vJLS_4Kr+7_SbdBXcEnjWVnLCvjm40FXMDLDC2d_DOaFAw@mail.gmail.com>
 <A0A1D905-E138-4D54-B47C-A437C0EE3BAB@langa.pl>
Message-ID: <53EEA521.7020406@canterbury.ac.nz>

?ukasz Langa wrote:

> class C:
>   cls_member: str = ?on the class?
> 
>   def __init__(self):
>     self.obj_member: str = ?on the instance'
>     self.cls_member = 2   # that?s the real question: type error or an 
> instance member?

I think not treating it as an error would make it hard
to reason about the type of x.cls_member for an instance
x of C. Its type would depend on whether del x.cls_member
had been performed on x.

Code which relied on them being different types would
be rather confusing to a human reader too, so it's
probably fine to discourage that.

-- 
Greg

From alexander.belopolsky at gmail.com  Sat Aug 16 04:07:45 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Fri, 15 Aug 2014 22:07:45 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53EE9BEF.8040001@stoneleaf.us>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando> <53ED0109.6070207@stoneleaf.us>
 <20140815184909.61c4ee53@anarchist.localdomain>
 <53EE9BEF.8040001@stoneleaf.us>
Message-ID: <CAP7h-xZovDSrTGyS9KqBC1CmAC9LSiArXEO5fSHLN0JtVgjmKQ@mail.gmail.com>

On Fri, Aug 15, 2014 at 7:46 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> Sounds like what we *really* need is a decorator that will parse the
> docstring and fill in the annotations automatically.  :)


While it may be contrary to the TOOWTDI principle, but I think decorator-
and annotations-based type specifications can co-exist.  I have been using
the following decorator for about a year

def returns(type):
    """emulate -> attribute setting in python 2"""

    def decorator(func):
        func.__dict__.setdefault('__attributes__', {})['return'] = type
        return func

    return decorator

and I often find code that uses for example @returns(int) more readable
than the code that uses .. -> int:

The advantage of argument annotations over any decorator-based solution is
avoidance of repetition, but I often miss K&R-style declarations in C.
 Probably because

int
copy(from, to)
char *from;
char *to;
{..}

looks more "pythonic" than

int
copy(char *from, char *to)
{}

In python, what is more readable:

def copy(from:Sequence, to:MutableSequence):
       ..

or

@arg_types(from=Sequence, to=MutableSequence)
def copy(from, to):
    ..

or

def copy(from, to):
    ..

set_arg_types(copy, from=Sequence, to=MutableSequence)

?


I believe that having type specifications out of the way in the last
variant more than outweighs the need to repeat the names of arguments.

Even the repetition  can be avoided if we do something like

@arg_types(Sequence, MutableSequence)
def copy(from, to):
    ..

but this is still more intrusive than the after-body variant where function
name repetition is unavoidable and argument names repetition improves
readability.

I can imagine cases where having type specifications follow each argument
is helpful.  For example

def configure(d:Drawing, c:Configuration):
   d.background = c.get_parameter('background')
   d.height = c.get_parameter('height')
   ..

It really depends on the problem at hand and on the coding style whether
you may want type specification before or within the function declaration
or out of the way in docstring, after the function body or in a separate
"stubs" file altogether.

If Python gets an official standard for specifying types in function
attributes, it should not be hard to standardize docstring and decorator
alternatives as well.  As a bonus, these alternatives will be immediately
available to 2.x users.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140815/ea715471/attachment-0001.html>

From stephen at xemacs.org  Sat Aug 16 05:07:29 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 16 Aug 2014 12:07:29 +0900
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
In-Reply-To: <20140815181737.1e1279b4@anarchist.localdomain>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAGE7PN+TyBsUfE_yjZR5VLsJ4oNkJ6zgr_gR_KkGZGvSGYSkTQ@mail.gmail.com>
 <CAP7+vJJwsR0cWPhwuPcX6gk1m-CNaOySY2BaLLb_pUZkAGqpLg@mail.gmail.com>
 <20140815181737.1e1279b4@anarchist.localdomain>
Message-ID: <87egwhgmj2.fsf@uwakimon.sk.tsukuba.ac.jp>

Barry Warsaw writes:

 > OTOH there *is* a cost that could negatively impact readability.
 > It's more text visually assaulting you. :)

Not if you have a syntax-highlighting editor that has a no-see-um
face or text property.


From wes.turner at gmail.com  Sat Aug 16 05:16:30 2014
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 15 Aug 2014 22:16:30 -0500
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CANc-5UwTy0b6Jo_ZYBMy1AUVMxTFj3dV_H05xK700XvgmyGG9A@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <53EBC3BA.10808@stoneleaf.us>
 <CAP7+vJJP_4pq6woFrWBDOxd6yiHoZXgSJ8seZwSAspzF-z+YSg@mail.gmail.com>
 <53ED0DCB.2000306@googlemail.com>
 <CANc-5UwTy0b6Jo_ZYBMy1AUVMxTFj3dV_H05xK700XvgmyGG9A@mail.gmail.com>
Message-ID: <CACfEFw-gknH=-HUR1yaACZwQ5s7YAGo1KQWW+KNBvaMcDchYrA@mail.gmail.com>

> Couldn't you do that today in Python with a suitably sophisticated
> function decorator? The range/type checking would happen before the
> user's actual function is called.

PyContracts does this; with (1) an @contract decorator,
(2) Python 3 annotations,
(3) with :type: and :rtype: docstrings

... http://www.reddit.com/r/Python/comments/2dh036/pythonideas_proposal_use_mypy_syntax_for_function/#cjq09bu
:

> +1 for static typing for certain problems.
> [...]
> Static type checking at compile time (linting) looks really neat.
> [...]
> Do we need a separate approach for actual type assertions
> at runtime? Will that ever be in scope for mypy?

From ncoghlan at gmail.com  Sat Aug 16 06:07:12 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 16 Aug 2014 14:07:12 +1000
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <967197E8-94BD-45C3-BCD8-BFD0690EFF12@langa.pl>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
 <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>
 <CAJ8oX-EEHamhjUY-Ub8APYsOf1koXC=AN8mgobin0OA-nSRP-A@mail.gmail.com>
 <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>
 <E05E59D2-11C2-4F31-BF17-7B62478839A5@langa.pl>
 <CAJ8oX-EzLtn4JubP7aVFGH=JaGNcwz4CFGnzfQF009oYo0k0eg@mail.gmail.com>
 <967197E8-94BD-45C3-BCD8-BFD0690EFF12@langa.pl>
Message-ID: <CADiSq7cRJmA8+agik4PJps_i8LkTx0z+mHfF3Mh=o-7O_7zuUA@mail.gmail.com>

On 16 August 2014 04:38, ?ukasz Langa <lukasz at langa.pl> wrote:
> Ah, that?s right, I meant singledispatch. Sorry for the confusion! :)

I think "singledispatch" is still potentially relevant to Matthew's
idea of a library specifically devoted to dispatch and signature
matching. My rationale is that "functools.singledispatch" actually has
some pretty neat funcationality behind it. I'm talking things like
functools._c3_merge, functools._compose_mro, functools._find_impl -
all the machinery that is needed to take the complexity of ABC
registration and turn it into something that can be checked quickly at
runtime. Even the trick with using abc.get_cache_token() to invalidate
the dispatch caches whenever the object graph changes via explicit ABC
registration is worth illuminating further.

Decoupling the underlying dispatch machinery from the specific
functools use case also opens up additional possibilities for type
based dispatch.

The way Julia uses multiple dispatch for binary operators is
interesting (http://julia.readthedocs.org/en/latest/manual/methods/),
although not directly applicable to Python's binary operators, since
we use the "return NotImplemented" dance to decide which
implementation to use.

That's where I can see a possible fit for something like multiple
dispatch support in the standard library: making it easier to write
binary operator overloads correctly. A *lot* of folks make the mistake
of raising TypeError or NotImplementedError directly in their operator
overload implementations, rather than returning the NotImplemented
singleton that tells the interpreter to try the other type. Even some
of the CPython builtins get that wrong, since the sq_concat and
sq_repeat slots in C don't properly support the type coercion dance,
so you *have* to raise the exception yourself if you're only
implementing those without implementing nb_add and nb_mul (types
defined in Python automatically populate both sets of C level slots if
you define __add__ or __mul__).

Dealing with the "NotImplemented dance" properly is also why
functools.total_ordering got substantially slower in Python 3.4 - it
isn't at risk of blowing up with RecursionError in some cases any
more, but it paid a hefty price in speed to get there.

Notation wise, I strongly encourage going with the format defined in
PEP 443: a "default implementation" that defines the namespace all the
other implementations will hook into, along with explicit registration
of additional overloads. If there's no sensible default
implementation, it can be written to raise an appropriate exception (a
dispatch library could even help with raising an appropriately
formatted type error).

For example, here's how a PEP 443 inspired notation would handle the
task of defining a stricter version of sequence repetition than the
default "*" binary operator:


    @functools.multidispatch(2) # Multiple dispatch on the first 2
positional arguments
    def repeat(lhs, rhs):
        raise BinaryDispatchError('repeat', lhs, rhs) # See below for
possible definition

   @example.register(numbers.Integral, collections.abc.Sequence):
   @example.register(collections.abc.Sequence, numbers.Integral):
    def repeat(lhs, rhs):
        return lhs * rhs


    # Assume this useful helper class is defined somewhere...
    class BinaryDispatchError(TypeError):
        def __init__(self, fname, lhs, rhs):
            self.fname = fname
            self.lhs_kind = lhs_kind = lhs.__class__.__name__
            self.rhs_kind = rhs_kind = rhs.__class__.__name__
            msg = "Unsupported operand type(s) for {!r}: {!r} and
{!r}".format(fname, lhs_kind, rhs_kind)
            super().__init__(msg)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From antoine at python.org  Sat Aug 16 06:24:19 2014
From: antoine at python.org (Antoine Pitrou)
Date: Sat, 16 Aug 2014 00:24:19 -0400
Subject: [Python-ideas] generic code and dependent types
In-Reply-To: <lskp1f$vli$1@ger.gmane.org>
References: <lskp1f$vli$1@ger.gmane.org>
Message-ID: <lsmmdj$a8d$1@ger.gmane.org>

Le 15/08/2014 06:56, Neal Becker a ?crit :
> I'm interested in the proposals for adding type annotation.  Coming from some
> experience with generic code in c++, one of the difficult issues has been
> reasoning about dependent types in generic code.
>
> In general, we need to be able to declare a type via an arbitrary metafunction
>
> some_metafunction<T>::type F (...

What is a metafunction?




From steve at pearwood.info  Sat Aug 16 07:04:33 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 16 Aug 2014 15:04:33 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140815184044.2aaeca8e@anarchist.localdomain>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando> <20140815184044.2aaeca8e@anarchist.localdomain>
Message-ID: <20140816050431.GE4525@ando>

Some people like the epydoc-style convention of putting type annotations 
in docstrings:

[...]
> >>         def demo(self, a, b, c):
> >>             """
> >>             This function returns the product of a, b and c
> >>             @type self: SimpleEquation
> >>             :param a: int - The first number
> >>             :param b: int
> >>             :param c: int - The third number should not be zero and should
> >> also
> >>                 only be -1 if you enjoy carrots (this comment spans 2 lines)
> >>             :return: int
> >>             """

One issue I haven't see raised is that annotations are available at 
runtime, whereas docstrings may not be. (The -OO switch removes 
docstrings.) A linter may be able to parse the docstrings at compile 
time before the docstrings are discarded, or it may not, but using 
docstrings means the information is not always available for 
introspection at runtime. I think that's a major disadvantage.

Although I admit I don't always remember to test my code using -O and 
-OO, I do try very hard to do this and I have found bugs in my code from 
doing so. I think anything which makes testing -O and -OO modes harder 
is a bad thing.

[Quoting Barry Warsaw]
> docstrings
> naturally live right after the function signature (indeed, or it wouldn't get
> stuffed into __doc__), so it's always close to the source.  That makes it
> quite easy for the third party human reader, but also for the author to keep
> up-to-date.

*Close to the source* is not the same as *part of the source*. In the 
example above, the difference is as high as eight lines, compared to 
zero:

    # Function annotations
    def demo(self, a:int, b:int, c:int)->int:


Using docstring annotations splits the information about parameters into 
two places. Those two places might be close, but there are still two 
sources of ultimate truth instead of one. You have the name of the 
parameter in the parameter list, and the type of the parameter inside 
the docstring separated by some arbitrary number of lines of code.


-- 
Steven

From steve at pearwood.info  Sat Aug 16 07:16:40 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 16 Aug 2014 15:16:40 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140815184909.61c4ee53@anarchist.localdomain>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando> <53ED0109.6070207@stoneleaf.us>
 <20140815184909.61c4ee53@anarchist.localdomain>
Message-ID: <20140816051640.GF4525@ando>

On Fri, Aug 15, 2014 at 06:49:09PM -0400, Barry Warsaw wrote:

> Docstring annotations almost by definition can contain more information useful
> to the human reader than type annotations can, especially if you carefully use
> the reST-ish epydoc convention of both :param: and :type:.  The latter
> contains the type annotation (which an automated system could utilize) while
> the former contains the exposition (for the benefit of the human reader).
> It's the explanations that are missing from any type annotations.
> 
> I suppose you could intersperse comments with your type annotations, resulting
> in a long multiline function signature.  I doubt that would be a readability
> improvement.

I'm sorry, I missed the part of Guido's proposal where he said that 
docstrings would cease to be used for documentation :-)

I don't think that it is a serious risk that docstrings will disappear, 
or that people will try to shove usage comments inside the parameter 
list:

def frobnicate(
    # this is the thing to be frobnicated
    obj:int,
    # pass a truthy object to use the blue frob instead of red frob
    blue:object=False,
    # an extra helping of spam
    yes_please_more_spam:list[Spam]
    ):

any more than they already do. (I think I've written a comment inside a 
parameter list maybe twice in the last decade.) I don't think there's 
much risk of that changing.

A more exciting outcome would be for documentation tools to start using 
the type annotations directly, without needing the writer to include the 
type annotation in two places (the parameter list and the docstring). 
But before the doc tools can do that, there needs to be a standard for 
annotations.


-- 
Steven

From donald at stufft.io  Sat Aug 16 07:22:05 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 16 Aug 2014 01:22:05 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140816051640.GF4525@ando>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando> <53ED0109.6070207@stoneleaf.us>
 <20140815184909.61c4ee53@anarchist.localdomain> <20140816051640.GF4525@ando>
Message-ID: <F8014846-8893-4DC9-A05B-4BC24876F1CF@stufft.io>


> On Aug 16, 2014, at 1:16 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> 
> On Fri, Aug 15, 2014 at 06:49:09PM -0400, Barry Warsaw wrote:
> 
>> Docstring annotations almost by definition can contain more information useful
>> to the human reader than type annotations can, especially if you carefully use
>> the reST-ish epydoc convention of both :param: and :type:.  The latter
>> contains the type annotation (which an automated system could utilize) while
>> the former contains the exposition (for the benefit of the human reader).
>> It's the explanations that are missing from any type annotations.
>> 
>> I suppose you could intersperse comments with your type annotations, resulting
>> in a long multiline function signature.  I doubt that would be a readability
>> improvement.
> 
> I'm sorry, I missed the part of Guido's proposal where he said that 
> docstrings would cease to be used for documentation :-)
> 
> I don't think that it is a serious risk that docstrings will disappear, 
> or that people will try to shove usage comments inside the parameter 
> list:
> 
> def frobnicate(
>    # this is the thing to be frobnicated
>    obj:int,
>    # pass a truthy object to use the blue frob instead of red frob
>    blue:object=False,
>    # an extra helping of spam
>    yes_please_more_spam:list[Spam]
>    ):
> 
> any more than they already do. (I think I've written a comment inside a 
> parameter list maybe twice in the last decade.) I don't think there's 
> much risk of that changing.
> 
> A more exciting outcome would be for documentation tools to start using 
> the type annotations directly, without needing the writer to include the 
> type annotation in two places (the parameter list and the docstring). 
> But before the doc tools can do that, there needs to be a standard for 
> annotations.
> 


Couldn?t the documentation tools also just pull that information from the
annotations?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/419a23e9/attachment.html>

From ethan at stoneleaf.us  Sat Aug 16 07:59:05 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 15 Aug 2014 22:59:05 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140816050431.GE4525@ando>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando> <20140815184044.2aaeca8e@anarchist.localdomain>
 <20140816050431.GE4525@ando>
Message-ID: <53EEF329.6080402@stoneleaf.us>

On 08/15/2014 10:04 PM, Steven D'Aprano wrote:
>
> Using docstring annotations splits the information about parameters into
> two places. Those two places might be close, but there are still two
> sources of ultimate truth instead of one. You have the name of the
> parameter in the parameter list, and the type of the parameter inside
> the docstring separated by some arbitrary number of lines of code.

You say that like it's a bad thing.  Not having everything crammed into one spot can be good.  Too dense is just as bad 
as too sparse.

--
~Ethan~

From wichert at wiggy.net  Sat Aug 16 08:39:56 2014
From: wichert at wiggy.net (Wichert Akkerman)
Date: Sat, 16 Aug 2014 08:39:56 +0200
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
 <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>
 <CAJ8oX-EEHamhjUY-Ub8APYsOf1koXC=AN8mgobin0OA-nSRP-A@mail.gmail.com>
 <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>
Message-ID: <BF2D830D-DB62-4109-A8AA-B1BFC04399D0@wiggy.net>


> On 15 Aug 2014, at 20:01, Guido van Rossum <guido at python.org> wrote:
> 
> Please do write about non-toy examples!

The entire Zope stack has been build around multiple-dispatch for the last 8 years or so. Of course that uses zope.interface/zope.component, which predate singledispatch by about 10 years as far as I can tell.

Wichert.

From brakhane at googlemail.com  Sat Aug 16 12:09:10 2014
From: brakhane at googlemail.com (Dennis Brakhane)
Date: Sat, 16 Aug 2014 12:09:10 +0200
Subject: [Python-ideas] generic code and dependent types
In-Reply-To: <lsmmdj$a8d$1@ger.gmane.org>
References: <lskp1f$vli$1@ger.gmane.org> <lsmmdj$a8d$1@ger.gmane.org>
Message-ID: <53EF2DC6.7040703@gmail.com>

Am 16.08.2014 06:24, schrieb Antoine Pitrou:
>
> What is a metafunction?
In C++, roughly speaking a function that is evaluated at compile time
not runtime, used for template metaprogramming. They
accept compile time constants and types as parameters and return
constants or types.

So as a silly example, if we were to have metafunctions in Python's type
system, it would allow me to define a metafunction
NestedList, and if I wrote

  def (x: NestedList[3, int])

it could be evaluated to

  def(x: List[List[List[int]]])

or IntOrString which would evaluate to Int if the parameter was true, or
String otherwise.



From steve at pearwood.info  Sat Aug 16 12:47:00 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 16 Aug 2014 20:47:00 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <lsltff$aba$1@ger.gmane.org>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CADiSq7eOYzQXOrb=BY1ETnzGYjAaxYfDtGn8-t-0pBeMOJoZvA@mail.gmail.com>
 <lskkem$8m7$1@ger.gmane.org>
 <CADiSq7cW5U_a+YTrO9MjxhYsPh6GZKmAysjRaSUa8aDX+Qiq5w@mail.gmail.com>
 <lsltff$aba$1@ger.gmane.org>
Message-ID: <20140816104700.GH4525@ando>

On Fri, Aug 15, 2014 at 05:18:11PM -0400, Terry Reedy wrote:
[...]
> "Depending on the checker" alludes to the fact that there are two types 
> of annotation consumers: those that read the source or parsed source 
> (ast) and the annotations therein and those that look at the function 
> .__annotations__ attribute after compilation.  Inspect.signature is an 
> example of the latter. (If there were none, there would be no purpose to 
> .__annotations__!)

Agreed.


> Suppose an integer square root function were annotated with with type 
> and non-type information.
> 
> >>> def nsqrt(n:(int, 'random non-type info'))->(int, 'more non-type 
> info'): pass

I don't think it is reasonable to expect arbitrary annotation tools to 
interoperate, unless they are specifically designed to interoperate. If 
tool A expects annotations to be a certain thing, and tool B expects 
them to be a different thing, they are going to confuse each other. We 
need a convention so that tools which expect to work with annotations 
can identify which annotations are aimed at them.

E.g. tool A might decorate the function with a marker that says "A", so 
that tool B knows to skip those functions. And vice versa. In the 
absence of any such marker, annotations can be assumed to be standard 
mypy-style type annotations.

(The nature of this marker probably should be standardized. I suggest a 
key/item in __annotations__, where the key cannot clash with parameter 
names.)

The easiest way to apply that marker is with a decorator: perhaps the 
typing module could provide a standard decorator that all annotation 
tools can recognise at compile-time:

@register_annotations(A.marker)  # for example
def function(x:"something understandable by A"):
    ...

# inside module A
marker = object()
def introspect(func):
    if magic(func) is marker:
        # okay to operate on func


> To me, inspect.signature already assumes that annotations are about 
> type, which means that this horse has already left barn. In 3.4.1:

I don't see how that follows. Your example demonstrates that inspect 
treats annotations as arbitrary Python expressions (which is what they 
are). You annotate the n parameter with the expression (int, "random 
non-type info"), which is a tuple of two objects. And signature 
dutifully reports that:

> >>> from inspect import signature as sig
> >>> str(sig(nsqrt))
> "(n:(<class 'int'>, 'random non-type info')) -> (<class 'int'>, 'more 
> non-type info')"
> 
> Typing 'nsqrt(' in Idle and pausing a fraction of a second brings up a 
> calltip with the same string (without the outer quotes). To me, having 
> random non-type info in the signature and calltip is noise and therefore 
> wrong.

Then don't put random info in the annotations :-)

If Idle has documented that the calltip is *always* type information, 
then Idle is wrong. Currently, there is no standard interpretation of 
function annotations, and if Idle documentation says otherwise, it is 
the documentation which is wrong.

In the future, I can see that Idle might want to only display calltips 
that it knows contain type information, or perhaps show them slightly 
differently if they are not type annotations. Or perhaps not... see 
below.


> So I agree that the non-standard annotation should be signaled by 
> a decorator 

I agree whole-heartedly to this.


> *and* suggest that the decorator should remove the 
> non-standard annotation, which is easily done, so that the signature 
> string for the above would be

But I disagree equally as strongly to this. Other uses of annotations 
are just as valid as static typing, and may equally want to be available 
for runtime introspection.

All we need is some sort of standardised runtime marker whereby tools 
can decide whether or not they should use the annotations.


> The future relative proportion of pre- and post-compile annotation 
> consumers is not relevant to my argument. The stdlib already has a 
> important post-compile consumer that is already somewhat broken by 
> non-type info remaining in .__annotations__.

I think it is only broken if you treat Idle calltips as displaying 
*types*. If you treat Idle calltips as displaying *annotations* no 
matter what the nature of those annotations, then it is not broken in 
the least. You yourself call them *call* tips, not "type tips", so there 
could be useful information provided other than the types of arguments. 

Consider two (imaginary) annotations in a graphics library:

def move(x: int, y:int): ...

def move(x: "distance along the left-right axis", 
         y: "distance along the up-down axis"): ...

I think that the second would be far more useful in a library 
aimed at beginners.


[...]
> Guido has also suggested 'deprecating' non-type annotations.  That would 
> literally mean raising an error either when source is compiled or when 
> def statements are executed.

Not necessarily. It could mean just documenting that we shouldn't use 
function annotations for anything other than specifying types. The usual 
procedure for deprecations is:

* for at least one release, tell people not to do this, but don't 
  raise a warning or an exception;
* for at least one release, raise a warning but not an exception;
* for at least one release, raise an exception

"For at least one release" might mean "until Python 5000".


> I think deprecation in this sense is both 
> unwise, since non-type annotation were explicitly invited, and 
> unnecessary for the purpose of favoring type annotations.

I whole-heartedly agree with this part!



-- 
Steven

From stefan at bytereef.org  Sat Aug 16 13:01:56 2014
From: stefan at bytereef.org (Stefan Krah)
Date: Sat, 16 Aug 2014 13:01:56 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53EEF329.6080402@stoneleaf.us>
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando>
 <20140815184044.2aaeca8e@anarchist.localdomain>
 <20140816050431.GE4525@ando> <53EEF329.6080402@stoneleaf.us>
Message-ID: <20140816110156.GA15938@sleipnir.bytereef.org>

Ethan Furman <ethan at stoneleaf.us> wrote:
> On 08/15/2014 10:04 PM, Steven D'Aprano wrote:
> >Using docstring annotations splits the information about parameters into
> >two places. Those two places might be close, but there are still two
> >sources of ultimate truth instead of one. You have the name of the
> >parameter in the parameter list, and the type of the parameter inside
> >the docstring separated by some arbitrary number of lines of code.
> 
> You say that like it's a bad thing.  Not having everything crammed
> into one spot can be good.  Too dense is just as bad as too sparse.

It *is* a bad thing. ;)  Humans can recognize patterns easily, and Steven's
example can be understood at a single glance.

The information is declarative and the density isn't that high.  Density
may become a problem if the information is functional and dense, like in
APL.  Even that can be overcome by training.


Stefan Krah



From davidhalter88 at gmail.com  Sat Aug 16 15:00:15 2014
From: davidhalter88 at gmail.com (Dave Halter)
Date: Sat, 16 Aug 2014 15:00:15 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
Message-ID: <CAA=HWYgZdwgDbg0RM3AHp_E+wVArxdjeWQf8oksBvcxudPQcdQ@mail.gmail.com>

I'm late, but it's also quite hard to read 200 mails in two days
(BlaBlaOverflow) :) I've read most of it. I'm also top-posting. Sorry :)
This mail is going to be way shorter this way.

*EuroPython*
I'm a core-dev of Jedi <https://github.com/davidhalter/jedi> (the
autocompletion library). We were having discussions
<https://github.com/davidhalter/jedi/issues/170> about using annotations
for a longer time, but we couldn't decide on a good approach. This has
changed quite drastically at EuroPython were I realized that Jedi is not
alone in wanting annotations to become a way of specifying types. Most
prominently the some of the pylint guys and Andrey of PyCharm seemed
interested to standardize type annotations. This would have led me to
eventually write a mail here on Python ideas (probably together with those
mentioned), describing our needs. Furthermore there were other interested
parties like Cython and Numba devs, because they might eventually use
annotations to improve their type knowledge.

Conclusion of the conference is that Python's static analysis community
wants type annotations. We want them standardized to be actually able to
use them. A third party library would be useful only if it had the same
opportunities like tulip: Being a third party library with the clear goal
of inclusion in the standard library.

*The current proposal*
While I don't fully support mypy's type system, I think it's a step into
the right direction, but I don't think we need it now (BTW: Mypy's
``Protocol`` class is genius and should be adapted into the stdlib anyway).
In the Appendix below, I've described what I came up with. I think my
solution inferior in capabilities, but way easier to understand and without
clutter. Something like Mypy's `typing` should probably be adopted and
standardized in a separate PEP, possibly for 3.6, once we have a few first
experiences. (Maybe if we start early it could still make 3.5).

I think a resulting PEP of this discussion should contain a deprecation
note for all usages other than type checking in Python 3.5. The current
ambiguous nature of annotations is the fact why no static analysis tool
ever checked them.

One thing you have to remember at this point, *that the two biggest issues
for static analysis are builtins and functions that are never called.
Function annotations would solve both partially.*

*Argument Clinic*
One notable side effect of Guido's proposal would be that Argument Clinic
could use annotations in its signatures. This would be a big benefit for
static analysis. because it would finally reveal the types of builtin
functions (input and output!). I tried to convince Larry Hastings to
implement that, but he refused, citing PEP 3107 which states that "This
work will be left to third-party libraries.".

Argument Clinic in combination with type annotations would be a huge win
for the static analysis community.

*Issues*

By far my biggest concern is the fact that type annotations in CPython
don't have any effect on run-time or compile-time. This is really an issue,
because people from other languages will actually think that this is a type
checker. This could be fixed partially by checking annotations at run-time
(and raising warnings [or exceptions with a command line switch]). This way
annotations wouldn't be without semantic meaning, they would actually be
some kind of pre/post conditions. However, this should also not be the goal
of a current PEP. I just wanted to mention it, so that we can keep it in
mind.

*Appendix (My Proposal)*

My proposal (as discussed and evolved with a few good people at EuroPython)
for containers would look something like this:

    def foo(index: int, x: [float], y: {int: str}) -> (float, str):
        return x[index], y[index]

The "default" containers (set, list, dict and tuple) would just serve as a
way of specifying containers. This makes a lot of things less complicated:

- It's easier to understand (everybody knows builtin types). It also feels
natural to me. The example above covers also almost all variations. A tuple
could be expanded by using the ellipsis: `(int, object, ..., float)`.
- No imports needed. People are more likely to use it, if they don't have
to import typing all the time. This is important for static analysis,
people are only likely to use it if it's easier than writing docstrings
with type information.
- Argument clinic could use the same. The standard library quite often
doesn't accept abstract data types, btw.
- It's what people sometimes use in docstrings: ``@rtype: (float, str)`` or
``:rtype: (float, str)``.

I know this doesn't solve the duck typing issue, but if you look at
real-life Python code bases, there are very few instances of actually
implementing a ``Mapping``, etc. For all of that we should write a separate
proposal (with abstract type classes). My proposal is also missing a Union
type and a few other things, but I'd rather start type annotations really
simple and soon than to wait for good libraries to emerge.
I think the typing module (without my additions) would be
counterproductive, because it's just too complicated too understand. Most
people that are using Python don't know ABCs and would have a hard time
dealing with such optional typing. In the end most people would just ignore
it entirely.

Godspeed!
~ Dave


2014-08-15 1:56 GMT+02:00 Guido van Rossum <guido at python.org>:

> I have read pretty much the entire thread up and down, and I don't think I
> can keep up with responding to every individual piece of feedback. (Also, a
> lot of responses cancel each other out. :-)
>
> I think there are three broad categories of questions to think about next.
>
> (A) Do we even need this?
>
> (B) What syntax to use?
>
> (C) Does/should it support <feature X>?
>
> Taking these in turn:
>
> (A) Do we even need a standard for optional static typing?
>
> Many people have shown either support for the idea, or pointed to some
> other system that addresses the same issue. On the other hand, several
> people have claimed that they don't need it, or that they worry it will
> make Python less useful for them. (However, many of the detractors seem to
> have their own alternative proposal. :-)
>
> In the end I don't think we can ever know for sure -- but my intuition
> tells me that as long as we keep it optional, there is a real demand. In
> any case, if we don't start building something we'll never know whether
> it'll be useful, so I am going to take a leap of faith and continue to
> promote this idea.
>
> I am going to make one additional assumption: the main use cases will be
> linting, IDEs, and doc generation. These all have one thing in common: it
> should be possible to run a program even though it fails to type check.
> Also, adding types to a program should not hinder its performance (nor will
> it help :-).
>
> (B) What syntax should a standard system for optional static typing use?
>
> There are many interesting questions here, but at the highest level there
> are a few choices that constrain the rest of the discussion, and I'd like
> to start with these. I see three or four "families" of approaches, and I
> think the first order is to pick a family.
>
> (1) The mypy family. (http://mypy-lang.org/) This is characterized by its
> use of PEP 3107 function annotations and the constraint that its syntax
> must be valid (current) Python syntax that can be evaluated without errors
> at function definition time. However, mypy also supports collecting
> annotations in separate "stub" files; this is how it handles annotations
> for the stdlib and C extensions. When mypy annotations occur inline (not in
> a stub file) they are used to type check the body of the annotated function
> as well as input for type checking its callers.
>
> (2) The pytypedecl family. (https://github.com/google/pytypedecl) This is
> a custom syntax that can only be used in separate stub files. Because it is
> not constrained by Python's current syntax, its syntax is slightly more
> elegant than mypy.
>
> (3) The PyCharm family. (
> http://www.jetbrains.com/pycharm/webhelp/using-docstrings-to-specify-types.html)
> This is a custom syntax that lives entirely in docstrings. There is also a
> way to use stub files with this. (In fact, every viable approach has to
> support some form of stub files, if only to describe signatures for C
> extensions.)
>
> (I suppose we could add a 4th family that puts everything in comments, but
> I don't think anyone is seriously working on such a thing, and I don't see
> any benefits.)
>
> There's also a variant of (1) that ?ukasz Langa would like to see -- use
> the syntactic position of function annotations but using a custom syntax
> (e.g. one similar to the pytypedecl syntax) that isn't evaluated at
> function-definition time. This would have to use "from __future__ import
> <something>" for backward compatibility. I'm skeptical about this though;
> it is only slightly more elegant than mypy, and it would open the
> floodgates of unconstrained language design.
>
> So how to choose? I've read passionate attacks and defenses of each
> approach. I've got a feeling that the three projects aren't all that
> different in maturity (all are well beyond the toy stage, none are quite
> ready for prime time). In terms of specific type system features (e.g.
> forward references, generic types, duck typing) I expect they are all
> acceptable, and all probably need some work (and there's no reason to
> assume that work can't be done). All support stubs so you can specify
> signatures for code you can't edit (whether C extension, stdlib or just
> opaque 3rd party code).
>
> To me there is no doubt that (1) is the most Pythonic approach. When we
> discussed PEP 3107 (function annotations) it was always my goal that these
> would eventually be used for type annotations. There was no consensus at
> the time on what the rules for type checking should be, but their syntactic
> position was never in doubt. So we decided to introduce "annotations" in
> Python 3 in the hope that 3rd party experiments would eventually produce
> something satisfactory. Mypy is one such experiment. One of the important
> lessons I draw from mypy is that type annotations are most useful to
> linters, and should (normally) not be used to enforce types at run time.
> They are also not useful for code generation. None of that was obvious when
> we were discussing PEP 3107!
>
> I don't buy the argument that PEP 3107 promises that annotations are
> completely free of inherent semantics. It promises compatibility, and I
> take that very seriously, but I think it is reasonable to eventually
> deprecate other uses of annotations -- there aren't enough significant
> other uses for them to warrant crippling type annotations forever. In the
> meantime, we won't be breaking existing use of annotations -- but they may
> confuse a type checker, whether a stand-alone linter like mypy or built
> into an IDE like PyCharm, and that may serve as an encouragement to look
> for a different solution.
>
> Most of the thornier issues brought up against mypy wouldn't go away if we
> adopted another approach: whether to use concrete or abstract types, the
> use of type variables, how to define type equivalence, the relationship
> between a list of ints and a list of objects, how to spell "something that
> implements the buffer interface", what to do about JSON, binary vs. text
> I/O and the signature of open(), how to check code that uses isinstance(),
> how to shut up the type checker when you know better... The list goes on.
> There will be methods whose type signature can't be spelled (yet). There
> will be code distributed with too narrowly defined types. Some programmers
> will uglify their code to please the type checker.
>
> There are questions about what to do for older versions of Python. I find
> mypy's story here actually pretty good -- the mypy codec may be a hack, but
> so is any other approach. Only the __future__ approach really loses out
> here, because you can't add a new __future__ import to an old version.
>
> So there you have it. I am picking the mypy family and I hope we can start
> focusing on specific improvements to mypy. I also hope that somebody will
> write converters from pytypedecl and PyCharm stubs into mypy stubs, so that
> we can reuse the work already put into stub definitions for those two
> systems. And of course I hope that PyCharm and pytypedecl will adopt mypy's
> syntax (initially in addition to their native syntax, eventually as their
> sole syntax).
>
> PS. I realize I didn't discuss question (C) much. That's intentional -- we
> can now start discussing specific mypy features in separate threads (or in
> this one :-).
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/56fe97fd/attachment-0001.html>

From antoine at python.org  Sat Aug 16 15:34:51 2014
From: antoine at python.org (Antoine Pitrou)
Date: Sat, 16 Aug 2014 09:34:51 -0400
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
 <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>
 <CAJ8oX-EEHamhjUY-Ub8APYsOf1koXC=AN8mgobin0OA-nSRP-A@mail.gmail.com>
 <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>
Message-ID: <lsnmls$l8f$1@ger.gmane.org>

Le 15/08/2014 14:01, Guido van Rossum a ?crit :
> Please do write about non-toy examples!

Are you looking for examples using the multipledispatch library, or 
multiple dispatch in general?

As for multiple dispatch in general, Numba uses something which is 
morally one in order to select the right specialization of, say, an 
operator (for example to choose amongst '+ between int and int', '+ 
between numpy.datetime64 and numpy.timedelta64', '+ between 
numpy.timedelta64 and numpy.timedelta64', etc.).

Regards

Antoine.



From mrocklin at gmail.com  Sat Aug 16 15:58:43 2014
From: mrocklin at gmail.com (Matthew Rocklin)
Date: Sat, 16 Aug 2014 06:58:43 -0700
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <lsnmls$l8f$1@ger.gmane.org>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
 <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>
 <CAJ8oX-EEHamhjUY-Ub8APYsOf1koXC=AN8mgobin0OA-nSRP-A@mail.gmail.com>
 <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>
 <lsnmls$l8f$1@ger.gmane.org>
Message-ID: <CAJ8oX-H1JEyArKk0cn9THdje5gp8hHOyTqXKR=TuDDcSSqoFWw@mail.gmail.com>

Here is a non-trivial example of multiple dispatch.  I want to convert data
between container types, i.e.  given

into(a, b)

I want to return something with the information content of b in a container
like a, e.g.

In [24]: into([], (1, 2, 3))
Out[24]: [1, 2, 3]

We use this abstraction pretty heavily in Blaze, a project that tries to
map relational algebra onto a variety of projects that might possibly be
used to do relational-algebra-like tasks.  Projects in this scope include
sqlalchemy, pandas, numpy, pyspark, pytables, etc..

In [26]: from blaze import into

A dataframe with some test data

In [25]: df = DataFrame([[1, 'Alice',   100],
                         [2, 'Bob',    -200],
                         [3, 'Charlie', 300],
                         [4, 'Dennis',   400],
                         [5, 'Edith',  -500]],
                         columns=['id', 'name', 'amount'])

migrate list <- DataFrame

In [27]: into([], df)
Out[27]:
[[1, 'Alice', 100],
 [2, 'Bob', -200],
 [3, 'Charlie', 300],
 [4, 'Dennis', 400],
 [5, 'Edith', -500]]

migrate numpy array <- DataFrame

In [28]: into(np.ndarray(0), df)
Out[28]:
rec.array([(1, 'Alice', 100), (2, 'Bob', -200), (3, 'Charlie', 300),
       (4, 'Dennis', 400), (5, 'Edith', -500)],
      dtype=[('id', '<i8'), ('name', 'O'), ('amount', '<i8')])

In [29]: x = into(np.ndarray(0), df)  # store for later


connect to local pymongo database

In [30]: import pymongo

In [31]: db = pymongo.MongoClient().db

In [34]: into(db.my_collection, df)  # migrate mongo <- pandas
Out[34]: Collection(Database(MongoClient('localhost', 27017), u'db'),
u'my_collection')

In [35]: into(db.my_collection2, x)  # migrate mongo <- numpy
Out[35]: Collection(Database(MongoClient('localhost', 27017), u'db'),
u'my_collection2')

In [36]: list(db.my_collection2.find())  # verify that things transferred
well
Out[36]:
[{u'_id': ObjectId('53ef6167fb5d1b34b9fd00e2'),
  u'amount': 100,
  u'id': 1,
  u'name': u'Alice'},
 {u'_id': ObjectId('53ef6167fb5d1b34b9fd00e3'),
  u'amount': -200,
  u'id': 2,
  u'name': u'Bob'},
 {u'_id': ObjectId('53ef6167fb5d1b34b9fd00e4'),
  u'amount': 300,
  u'id': 3,
  u'name': u'Charlie'},
 {u'_id': ObjectId('53ef6167fb5d1b34b9fd00e5'),
  u'amount': 400,
  u'id': 4,
  u'name': u'Dennis'},
 {u'_id': ObjectId('53ef6167fb5d1b34b9fd00e6'),
  u'amount': -500,
  u'id': 5,
  u'name': u'Edith'}]

migrate bcolz <- mongo

In [37]: into(bcolz.ctable(), db.my_collection)
Out[37]:
ctable((5,), [('amount', '<i8'), ('id', '<i8'), ('name', '<U7')])
  nbytes: 220; cbytes: 63.99 KB; ratio: 0.00
  cparams := cparams(clevel=5, shuffle=True, cname='blosclz')
[(100, 1, u'Alice') (-200, 2, u'Bob') (300, 3, u'Charlie')
 (400, 4, u'Dennis') (-500, 5, u'Edith')]

Note in this last case that the two libraries, bcolz (a compressed on-disk
storage library) and pymongo know absolutely nothing about each other.

Many of these into definitions are very simple

@dispatch(np.ndarray, DataFrame)
def into(a, df):
    return df.to_records(index=False)

While some of them rely on others, or on inheritance

@dispatch(Collection, np.ndarray)
def into(coll, x, **kwargs):
    return into(coll, into(DataFrame(), x), **kwargs)


But remembering all of the appropriate .to_foo and .from_bar methods can be
a real pain.  Collecting them all into a single abstraction cuts down
significantly on the administrative burden of data migrations.



On Sat, Aug 16, 2014 at 6:34 AM, Antoine Pitrou <antoine at python.org> wrote:

> Le 15/08/2014 14:01, Guido van Rossum a ?crit :
>
>  Please do write about non-toy examples!
>>
>
> Are you looking for examples using the multipledispatch library, or
> multiple dispatch in general?
>
> As for multiple dispatch in general, Numba uses something which is morally
> one in order to select the right specialization of, say, an operator (for
> example to choose amongst '+ between int and int', '+ between
> numpy.datetime64 and numpy.timedelta64', '+ between numpy.timedelta64 and
> numpy.timedelta64', etc.).
>
> Regards
>
> Antoine.
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/9683a5fd/attachment.html>

From abarnert at yahoo.com  Sat Aug 16 16:30:47 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 16 Aug 2014 07:30:47 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAA=HWYgZdwgDbg0RM3AHp_E+wVArxdjeWQf8oksBvcxudPQcdQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CAA=HWYgZdwgDbg0RM3AHp_E+wVArxdjeWQf8oksBvcxudPQcdQ@mail.gmail.com>
Message-ID: <F392A71E-CD09-489A-8D6F-AE51AAB7B32F@yahoo.com>

On Aug 16, 2014, at 6:00, Dave Halter <davidhalter88 at gmail.com> wrote:

> Appendix (My Proposal)
> 
> My proposal (as discussed and evolved with a few good people at EuroPython) for containers would look something like this:
> 
>     def foo(index: int, x: [float], y: {int: str}) -> (float, str):
>         return x[index], y[index]
> 
> The "default" containers (set, list, dict and tuple) would just serve as a way of specifying containers. This makes a lot of things less complicated:
> 
> - It's easier to understand (everybody knows builtin types). It also feels natural to me. The example above covers also almost all variations. A tuple could be expanded by using the ellipsis: `(int, object, ..., float)`.
> - No imports needed. People are more likely to use it, if they don't have to import typing all the time. This is important for static analysis, people are only likely to use it if it's easier than writing docstrings with type information.
> - Argument clinic could use the same. The standard library quite often doesn't accept abstract data types, btw. 
> - It's what people sometimes use in docstrings: ``@rtype: (float, str)`` or ``:rtype: (float, str)``.
> 
> I know this doesn't solve the duck typing issue, but if you look at real-life Python code bases, there are very few instances of actually implementing a ``Mapping``, etc.

For "Mapping", maybe (although anyone who uses some form of tree-based mapping, either to avoid hash-collision attacks or because he needs sorting, may disagree).

But for "etc.", people implement them all the time. Especially "Iterable" and "Callable". And, even some of the ones that people don't implement often, they use often, implicitly or otherwise, like TextIOBase.

Anything that encourages people to restrict their code to only working on lists instead of iterables would be a huge step backward to Python 2.2. And your argument for it ("everybody knows builtin types") implies that's exactly what you're expecting with this proposal.

However, there's an easy way around this: just let [spam] in your syntax mean  what Iterable[spam] means in MyPy's. If someone really needs to declare that they will only accept a list or whatever, that's the uncommon case, and can be written more verbosely. And I don't see any reason why this can't be added on top of genericizing the ABCs a la MyPy.

Meanwhile, your tuple doesn't fit the same pattern as the others, because it's explicitly fixed-size and heterogeneous. And I think this is a good thing. And I think it's pretty close to what MyPy does with an expression list of types already; if not, it seems like what MyPy _should_ do. If I loop over a zip of a [str] and an [int], the loop variable is a (str, int), not a Tuple[str or int].

So, making it easier to specify generic builtins is a bad idea, but using builtins to make it easier to specify common types is a great idea.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/b07f98f3/attachment-0001.html>

From brett at python.org  Sat Aug 16 16:46:47 2014
From: brett at python.org (Brett Cannon)
Date: Sat, 16 Aug 2014 14:46:47 +0000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAJaQC30kW5PGiTvGOzB3p5Mf3v_tjjt4bzvT6AkR-duLVwaLSg@mail.gmail.com>
 <20140814181554.GR4525@ando> <20140815184044.2aaeca8e@anarchist.localdomain>
 <20140816050431.GE4525@ando>
Message-ID: <CAP1=2W5fGvj747Ecz_iJyu2hVvUBCmVXUcSFLr2K3Uu+mhCXYg@mail.gmail.com>

On Sat Aug 16 2014 at 1:05:16 AM Steven D'Aprano <steve at pearwood.info>
wrote:

> Some people like the epydoc-style convention of putting type annotations
> in docstrings:
>
> [...]
> > >>         def demo(self, a, b, c):
> > >>             """
> > >>             This function returns the product of a, b and c
> > >>             @type self: SimpleEquation
> > >>             :param a: int - The first number
> > >>             :param b: int
> > >>             :param c: int - The third number should not be zero and
> should
> > >> also
> > >>                 only be -1 if you enjoy carrots (this comment spans 2
> lines)
> > >>             :return: int
> > >>             """
>
> One issue I haven't see raised is that annotations are available at
> runtime, whereas docstrings may not be. (The -OO switch removes
> docstrings.) A linter may be able to parse the docstrings at compile
> time before the docstrings are discarded, or it may not, but using
> docstrings means the information is not always available for
> introspection at runtime. I think that's a major disadvantage.
>
> Although I admit I don't always remember to test my code using -O and
> -OO, I do try very hard to do this and I have found bugs in my code from
> doing so. I think anything which makes testing -O and -OO modes harder
> is a bad thing.
>
> [Quoting Barry Warsaw]
> > docstrings
> > naturally live right after the function signature (indeed, or it
> wouldn't get
> > stuffed into __doc__), so it's always close to the source.  That makes it
> > quite easy for the third party human reader, but also for the author to
> keep
> > up-to-date.
>
> *Close to the source* is not the same as *part of the source*. In the
> example above, the difference is as high as eight lines, compared to
> zero:
>
>     # Function annotations
>     def demo(self, a:int, b:int, c:int)->int:
>
>
> Using docstring annotations splits the information about parameters into
> two places. Those two places might be close, but there are still two
> sources of ultimate truth instead of one. You have the name of the
> parameter in the parameter list, and the type of the parameter inside
> the docstring separated by some arbitrary number of lines of code.
>

I'm with Steven on this. I actively hate docstrings that list every
parameter, their expected interface, etc. The parameter list exists for a
reason and a majority of the time I don't need an explanation of what a
parameter does. In those rare instances where I need clarification I can
write a quick sentence in the docstring explaining the special case.

"""Returns the product of a, b, and c.

The 'c' parameter should not be zero. If you like carrots, set it to -1
(this comment spans two lines).
"""

That's 4 lines compared to 7 (which was missing a blank line to begin with
so it really should be 8). We're all adults and properly worded parameter
names tell you a lot. What we are trying to do here is help programmatic
tools know things that we know to be true.

Now that is not to say whatever comes out of typing.py shouldn't be legible
and not noisy. I'm sure the reason it uses CapWords for e.g. Dict is so you
can do `from typing import *` which makes `def demo(a: Int, b: Int, c: Int)
-> Int` read just as cleanly as if you used 'int' itself (this might become
the one time I promote using import * so enjoy it while you can =).

And as others have pointed out, if you really like the docstring approach
you can always set up a decorator to do the translation for you, but you
can't go the other way from annotation to docstring when examining source.
So while you can promote and use your docstring approach and even argue for
the inclusion of such a decorator in typing.py, you can't promote
docstrings exclusively without completely cutting off the function
annotation approach. And I think enough of us like the function annotation
approach that cutting it off entirely isn't acceptable.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/82ce28f6/attachment.html>

From brett at python.org  Sat Aug 16 17:02:44 2014
From: brett at python.org (Brett Cannon)
Date: Sat, 16 Aug 2014 15:02:44 +0000
Subject: [Python-ideas] Optional static typing -- the crossroads
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CAA=HWYgZdwgDbg0RM3AHp_E+wVArxdjeWQf8oksBvcxudPQcdQ@mail.gmail.com>
 <F392A71E-CD09-489A-8D6F-AE51AAB7B32F@yahoo.com>
Message-ID: <CAP1=2W7V3i35f_CiRgLqQ88=EQA0jmd=0vaHxSs9c--8dvWMRg@mail.gmail.com>

On Sat Aug 16 2014 at 10:38:34 AM Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:

> On Aug 16, 2014, at 6:00, Dave Halter <davidhalter88 at gmail.com> wrote:
>
> *Appendix (My Proposal)*
>
> My proposal (as discussed and evolved with a few good people at
> EuroPython) for containers would look something like this:
>
>     def foo(index: int, x: [float], y: {int: str}) -> (float, str):
>         return x[index], y[index]
>
> The "default" containers (set, list, dict and tuple) would just serve as a
> way of specifying containers. This makes a lot of things less complicated:
>
> - It's easier to understand (everybody knows builtin types). It also feels
> natural to me. The example above covers also almost all variations. A tuple
> could be expanded by using the ellipsis: `(int, object, ..., float)`.
> - No imports needed. People are more likely to use it, if they don't have
> to import typing all the time. This is important for static analysis,
> people are only likely to use it if it's easier than writing docstrings
> with type information.
> - Argument clinic could use the same. The standard library quite often
> doesn't accept abstract data types, btw.
> - It's what people sometimes use in docstrings: ``@rtype: (float, str)``
> or ``:rtype: (float, str)``.
>
> I know this doesn't solve the duck typing issue, but if you look at
> real-life Python code bases, there are very few instances of actually
> implementing a ``Mapping``, etc.
>
>
> For "Mapping", maybe (although anyone who uses some form of tree-based
> mapping, either to avoid hash-collision attacks or because he needs
> sorting, may disagree).
>
> But for "etc.", people implement them all the time. Especially "Iterable"
> and "Callable". And, even some of the ones that people don't implement
> often, they use often, implicitly or otherwise, like TextIOBase.
>
> Anything that encourages people to restrict their code to only working on
> lists instead of iterables would be a huge step backward to Python 2.2. And
> your argument for it ("everybody knows builtin types") implies that's
> exactly what you're expecting with this proposal.
>
> However, there's an easy way around this: just let [spam] in your syntax
> mean  what Iterable[spam] means in MyPy's. If someone really needs to
> declare that they will only accept a list or whatever, that's the uncommon
> case, and can be written more verbosely. And I don't see any reason why
> this can't be added on top of genericizing the ABCs a la MyPy.
>
> Meanwhile, your tuple doesn't fit the same pattern as the others, because
> it's explicitly fixed-size and heterogeneous. And I think this is a good
> thing. And I think it's pretty close to what MyPy does with an expression
> list of types already; if not, it seems like what MyPy _should_ do. If I
> loop over a zip of a [str] and an [int], the loop variable is a (str, int),
> not a Tuple[str or int].
>
> So, making it easier to specify generic builtins is a bad idea, but using
> builtins to make it easier to specify common types is a great idea.
>

The trick in all of this is making sure people instinctively know what the
builtin types represent in terms of an interface w/o necessarily
over-specifying. For instance, a list could be viewed as MutableSequence
when all that is really necessary is Sequence or Iterable. Just think of
those situations where a list or tuple both work as an argument; how do you
specify that without assuming mutability? It's tricky to figure out what a
proper assumption of what the built-ins represent should be.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/0be2105b/attachment.html>

From steve at pearwood.info  Sat Aug 16 19:15:09 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 17 Aug 2014 03:15:09 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAA=HWYgZdwgDbg0RM3AHp_E+wVArxdjeWQf8oksBvcxudPQcdQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CAA=HWYgZdwgDbg0RM3AHp_E+wVArxdjeWQf8oksBvcxudPQcdQ@mail.gmail.com>
Message-ID: <20140816171502.GK4525@ando>

On Sat, Aug 16, 2014 at 03:00:15PM +0200, Dave Halter wrote:

> I think a resulting PEP of this discussion should contain a deprecation
> note for all usages other than type checking in Python 3.5.

Can you explain your reasoning for this?

I understand that unless they are deliberately built to cooperate, two 
users of annotations are likely to interfere with each other. The jedi 
tool wants annotations to be type information; the sith tool wants 
annotations to be X-Face pictures of kittens. They can't both get what 
they want.

You are asking for alternative uses of annotations, such as pictures of 
kittens, to be deprecated. But it seems to me that all you really need 
is some standard way for jedi to look at the function and cheaply see 
that it shouldn't try interpreting the annotations as types. (And, 
mutatis mutandis, the same applies to sith.) That allows jedi and sith 
to co-exist, although we can't use both on the same function at the same 
time.

I think it is fair for Python to standardise on type checking as the 
default semantics of annotations, but I would like to see a way to 
opt-out and still use annotations for other purposes.


> The current
> ambiguous nature of annotations is the fact why no static analysis tool
> ever checked them.

mypy does. Hence this proposal.

So does PyCharm:

http://www.jetbrains.com/pycharm/webhelp/type-hinting-in-pycharm.html

On the other hand, in this thread we've heard from two others who use 
function annotations for purposes other than types. So it seems to me 
that usage of annotations is split right down the middle between typing 
and non-typing.


-- 
Steven

From ethan at stoneleaf.us  Sat Aug 16 22:22:48 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 16 Aug 2014 13:22:48 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
Message-ID: <53EFBD98.2090802@stoneleaf.us>

As a test case for what code may soon look like, here's a bit from one of my code bases:


--------------------------------------------------------------------
class ACHPayment(object):
     """A single payment from company to a vendor."""

     def __init__(self,
             description, sec_code,
             vendor_name, vendor_inv_num, vendor_rtng, vendor_acct,
             transaction_code, vendor_acct_type, amount, payment_date):
         """
         description:  10 chars
         sec_code: 'CCD' or 'CTX'
         vendor_name: 22 chars
         vendor_inv_num: 15 chars
         vendor_rtng: 9 chars
         vendor_acct: 17 chars
         transaction_code: ACH_ETC code (enum)
         vendor_acct_type: 'domestic' or 'foreign'
         amount: 10 digits (pennies)
         payment_date: date payment should occur on (datetime.date type class)
         """
--------------------------------------------------------------------


The question:  what would this look like with type annotations?  As a point of interest, the last parameter, 
payment_date, can be /anything/ that quacks like a datetime.date -- I tend to use my own dbf.Date class, which 
subclasses object, not datetime.date itself.

--
~Ethan~

From rymg19 at gmail.com  Sat Aug 16 22:48:10 2014
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Sat, 16 Aug 2014 15:48:10 -0500
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53EFBD98.2090802@stoneleaf.us>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us>
Message-ID: <CAO41-mOYY8zMGgS7RzJ0b5hOyt+g6sy6hfTfNhFkKHc+aNMdeg@mail.gmail.com>

Maybe that point of interest could be solved by using some kind of type
class/interface(in the Obj C/Java sense). That way, external types that the
user has no control of can be added to the interface.


On Sat, Aug 16, 2014 at 3:22 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> As a test case for what code may soon look like, here's a bit from one of
> my code bases:
>
>
> --------------------------------------------------------------------
> class ACHPayment(object):
>     """A single payment from company to a vendor."""
>
>     def __init__(self,
>             description, sec_code,
>             vendor_name, vendor_inv_num, vendor_rtng, vendor_acct,
>             transaction_code, vendor_acct_type, amount, payment_date):
>         """
>         description:  10 chars
>         sec_code: 'CCD' or 'CTX'
>         vendor_name: 22 chars
>         vendor_inv_num: 15 chars
>         vendor_rtng: 9 chars
>         vendor_acct: 17 chars
>         transaction_code: ACH_ETC code (enum)
>         vendor_acct_type: 'domestic' or 'foreign'
>         amount: 10 digits (pennies)
>         payment_date: date payment should occur on (datetime.date type
> class)
>         """
> --------------------------------------------------------------------
>
>
> The question:  what would this look like with type annotations?  As a
> point of interest, the last parameter, payment_date, can be /anything/ that
> quacks like a datetime.date -- I tend to use my own dbf.Date class, which
> subclasses object, not datetime.date itself.
>
> --
> ~Ethan~
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Ryan
If anybody ever asks me why I prefer C++ to C, my answer will be simple:
"It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was
nul-terminated."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/3e7eb889/attachment.html>

From mistersheik at gmail.com  Sat Aug 16 23:46:42 2014
From: mistersheik at gmail.com (Neil Girdhar)
Date: Sat, 16 Aug 2014 14:46:42 -0700 (PDT)
Subject: [Python-ideas] keyword for introducing generators
Message-ID: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com>

I'm sure this has been suggested before, but I just spent two days trying 
to figure out why a method wasn't being called only to find that I'd 
accidentally pasted a yield into the function.  What is the argument 
against a different keyword for introducing generator functions/methods?

If it's backward compatibility, then my suggestion to have a from 
__future__ and then make it real in Python 4.

Neil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/671279ad/attachment.html>

From rosuav at gmail.com  Sat Aug 16 23:55:36 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 17 Aug 2014 07:55:36 +1000
Subject: [Python-ideas] keyword for introducing generators
In-Reply-To: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com>
References: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com>
Message-ID: <CAPTjJmprOX0Yxv6xomZThuPe+uq_Mkj32wMr4GmHDX22vpiEmw@mail.gmail.com>

On Sun, Aug 17, 2014 at 7:46 AM, Neil Girdhar <mistersheik at gmail.com> wrote:
> I'm sure this has been suggested before, but I just spent two days trying to
> figure out why a method wasn't being called only to find that I'd
> accidentally pasted a yield into the function.  What is the argument against
> a different keyword for introducing generator functions/methods?

There are quite a few changes to a function based on its body, like
how the presence of assignment causes a name to be local unless
explicitly declared otherwise. It's not necessary to predeclare
everything.

But if you're having trouble with a function like that, maybe a little
decorator would help:

def announce(f):
    def inner(*a,**kw):
        print("Calling:",f.__name__)
        ret=f(*a,**kw)
        print("Return value is a",type(ret))
        return ret
    return inner

Decorate a function @announce, and it'll tell you (a) that it's being
called, and (b) what type its return value is. If that's a generator,
well, there's your answer.

ChrisA

From mistersheik at gmail.com  Sun Aug 17 00:04:45 2014
From: mistersheik at gmail.com (Neil Girdhar)
Date: Sat, 16 Aug 2014 18:04:45 -0400
Subject: [Python-ideas] keyword for introducing generators
In-Reply-To: <CAPTjJmprOX0Yxv6xomZThuPe+uq_Mkj32wMr4GmHDX22vpiEmw@mail.gmail.com>
References: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com>
 <CAPTjJmprOX0Yxv6xomZThuPe+uq_Mkj32wMr4GmHDX22vpiEmw@mail.gmail.com>
Message-ID: <CAA68w_k9phVyw+3jZ8zxSgTsYDgt4gWi8TgdQVhhkoSqO_om6w@mail.gmail.com>

This is only a good solution once I realize that the method isn't being
called after which, I now know what to look for.  The whole point is to
defensively design a language so that I don't get into this problem in the
first place.  Since I always know when I'm writing a function whether I
want it to be automatically generator-returning using the yield keyword or
not, then why not let me specify that?  That way if the function gets long
and I forget yield or accidentally include it, I get a reasonable error.

You're right that it's not necessary to predeclare everything, but a
different keyword is hardly more work.  (And with respect to local,
nonlocal and global variables, we often do predeclare those.)

Best,
Neil


On Sat, Aug 16, 2014 at 5:55 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Sun, Aug 17, 2014 at 7:46 AM, Neil Girdhar <mistersheik at gmail.com>
> wrote:
> > I'm sure this has been suggested before, but I just spent two days
> trying to
> > figure out why a method wasn't being called only to find that I'd
> > accidentally pasted a yield into the function.  What is the argument
> against
> > a different keyword for introducing generator functions/methods?
>
> There are quite a few changes to a function based on its body, like
> how the presence of assignment causes a name to be local unless
> explicitly declared otherwise. It's not necessary to predeclare
> everything.
>
> But if you're having trouble with a function like that, maybe a little
> decorator would help:
>
> def announce(f):
>     def inner(*a,**kw):
>         print("Calling:",f.__name__)
>         ret=f(*a,**kw)
>         print("Return value is a",type(ret))
>         return ret
>     return inner
>
> Decorate a function @announce, and it'll tell you (a) that it's being
> called, and (b) what type its return value is. If that's a generator,
> well, there's your answer.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/5-Qm2od4xQ8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/6163c44d/attachment-0001.html>

From abarnert at yahoo.com  Sun Aug 17 00:16:10 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 16 Aug 2014 15:16:10 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP1=2W7V3i35f_CiRgLqQ88=EQA0jmd=0vaHxSs9c--8dvWMRg@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CAA=HWYgZdwgDbg0RM3AHp_E+wVArxdjeWQf8oksBvcxudPQcdQ@mail.gmail.com>
 <F392A71E-CD09-489A-8D6F-AE51AAB7B32F@yahoo.com>
 <CAP1=2W7V3i35f_CiRgLqQ88=EQA0jmd=0vaHxSs9c--8dvWMRg@mail.gmail.com>
Message-ID: <4C70181D-C011-4B20-9CB9-C1A6BE40DFD5@yahoo.com>

On Aug 16, 2014, at 8:02, Brett Cannon <brett at python.org> wrote:

> On Sat Aug 16 2014 at 10:38:34 AM Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote:
>> On Aug 16, 2014, at 6:00, Dave Halter <davidhalter88 at gmail.com> wrote:
>> 
>>> Appendix (My Proposal)
>>> 
>>> My proposal (as discussed and evolved with a few good people at EuroPython) for containers would look something like this:
>>> 
>>>     def foo(index: int, x: [float], y: {int: str}) -> (float, str):
>>>         return x[index], y[index]
>>> 
>>> The "default" containers (set, list, dict and tuple) would just serve as a way of specifying containers. This makes a lot of things less complicated:
>>> 
>>> - It's easier to understand (everybody knows builtin types). It also feels natural to me. The example above covers also almost all variations. A tuple could be expanded by using the ellipsis: `(int, object, ..., float)`.
>>> - No imports needed. People are more likely to use it, if they don't have to import typing all the time. This is important for static analysis, people are only likely to use it if it's easier than writing docstrings with type information.
>>> - Argument clinic could use the same. The standard library quite often doesn't accept abstract data types, btw. 
>>> - It's what people sometimes use in docstrings: ``@rtype: (float, str)`` or ``:rtype: (float, str)``.
>>> 
>>> I know this doesn't solve the duck typing issue, but if you look at real-life Python code bases, there are very few instances of actually implementing a ``Mapping``, etc.
>> 
>> For "Mapping", maybe (although anyone who uses some form of tree-based mapping, either to avoid hash-collision attacks or because he needs sorting, may disagree).
>> 
>> But for "etc.", people implement them all the time. Especially "Iterable" and "Callable". And, even some of the ones that people don't implement often, they use often, implicitly or otherwise, like TextIOBase.
>> 
>> Anything that encourages people to restrict their code to only working on lists instead of iterables would be a huge step backward to Python 2.2. And your argument for it ("everybody knows builtin types") implies that's exactly what you're expecting with this proposal.
>> 
>> However, there's an easy way around this: just let [spam] in your syntax mean  what Iterable[spam] means in MyPy's. If someone really needs to declare that they will only accept a list or whatever, that's the uncommon case, and can be written more verbosely. And I don't see any reason why this can't be added on top of genericizing the ABCs a la MyPy.
>> 
>> Meanwhile, your tuple doesn't fit the same pattern as the others, because it's explicitly fixed-size and heterogeneous. And I think this is a good thing. And I think it's pretty close to what MyPy does with an expression list of types already; if not, it seems like what MyPy _should_ do. If I loop over a zip of a [str] and an [int], the loop variable is a (str, int), not a Tuple[str or int].
>> 
>> So, making it easier to specify generic builtins is a bad idea, but using builtins to make it easier to specify common types is a great idea.
> 
> The trick in all of this is making sure people instinctively know what the builtin types represent in terms of an interface w/o necessarily over-specifying. For instance, a list could be viewed as MutableSequence when all that is really necessary is Sequence or Iterable. Just think of those situations where a list or tuple both work as an argument; how do you specify that without assuming mutability? It's tricky to figure out what a proper assumption of what the built-ins represent should be. 

Honestly, I think [str] is the only case where this is an important question, so let's think about that rather than trying to think about a more general and abstract problem.

I'm not sure whether, when people say they need a list of strings, they more often mean Iterable[str] rather than Sequence[str] or MutableSequence[str]. I suspect it's the former, but I can't prove it, and I also suspect it's more of a 70/10/20 case than a 98/1/1 case. So maybe that argues that [str] just shouldn't be allowed.

But if it meant Iterable[str], someone who used it when they needed a Sequence or MutableSequence would get an error when they tried to MyPy their library, app, whatever. And it wouldn't be that hard to make that error explain the problem to them--the same way clang tries to explain template errors in C++ and suggest fixes, except that it would be orders of magnitude easier. "spam is an iterable, so you can't assign to its indexes. Did you mean to declare it as MutableSequence[str]?" On the other hand, if [str] meant MutableSequence[str] (or list[str]), the author who used it on a function that did nothing but loop over spam would not get an error; he'd have to wait until he published his code and someone filed a bug report saying "You declared spam as a MutableSequence, even though all you do is loop over it, and now my code that worked correctly with your spam-1.7, and that still works correctly with spam-1.8 if I don't use static checking, fails the linter. Did you mean to declare it as Iterable[str]?"

This obviously isn't a slam-sunk argument. If MutableSequence were used far more often than Iterable, or were more pythonic in some way, then it would make sense for [str] to mean MutableSequence despite the fact that it puts the errors in the less convenient place. But if they're both reasonably common, I think this argues for making it mean Iterable.

For the last part:

> Just think of those situations where a list or tuple both work as an argument; how do you specify that without assuming mutability?

That one's easy: you write Sequence. A lot of similar questions were already answered when abc, Number, collections.abc, and io were designed, and they did a great job answering some tricky questions--which is exactly why I think static typing should use those already-worked-out cases rather than trying to answer all those questions again with a parallel type hierarchy. (And if use of static typing leads people to realize that one of those ABCs got something wrong, better to fix that bug in the ABC than to leave it incorrect and make the typing type different.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/ada7a1f0/attachment.html>

From j.wielicki at sotecware.net  Sun Aug 17 00:17:53 2014
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Sun, 17 Aug 2014 00:17:53 +0200
Subject: [Python-ideas] keyword for introducing generators
In-Reply-To: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com>
References: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com>
Message-ID: <53EFD891.5030602@sotecware.net>

On 16.08.2014 23:46, Neil Girdhar wrote:
> I'm sure this has been suggested before, but I just spent two days trying 
> to figure out why a method wasn't being called only to find that I'd 
> accidentally pasted a yield into the function.  What is the argument 
> against a different keyword for introducing generator functions/methods?
> 
> If it's backward compatibility, then my suggestion to have a from 
> __future__ and then make it real in Python 4.

For what it?s worth, I know this problem very well, and it can take
hours to figure out whats wrong.

regards,
jwi

> 
> Neil
> 
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 


From abarnert at yahoo.com  Sun Aug 17 00:30:03 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 16 Aug 2014 15:30:03 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAO41-mOYY8zMGgS7RzJ0b5hOyt+g6sy6hfTfNhFkKHc+aNMdeg@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us>
 <CAO41-mOYY8zMGgS7RzJ0b5hOyt+g6sy6hfTfNhFkKHc+aNMdeg@mail.gmail.com>
Message-ID: <0B37B957-01A6-41BC-A416-94EF81D4FBA2@yahoo.com>

On Aug 16, 2014, at 13:48, Ryan Gonzalez <rymg19 at gmail.com> wrote:

> Maybe that point of interest could be solved by using some kind of type class/interface(in the Obj C/Java sense). That way, external types that the user has no control of can be added to the interface.

We already have that. ABCs are enough like Java interfaces, ObjC mandatory protocols, C++ (non-auto) concepts, etc. to do everything we need here (except genericity, which it seems like everyone agrees with adding) if you want to do it nominatively. You just need to write a Date ABC. Or argue that there should be a datetime.abc library in the stdlib that does it for you. And that's simple.

And if you want to do it structurally, like Go protocols, C++ auto concepts, ObjC optional protocols, etc., ABCs can also do that. It's not _quite_ as simple today, but it's not hard, and there are a half dozen libraries that make it easy (I wrote one in a couple hours, and didn't bother publishing it because a quick search turned up so many pre-existing alternatives, and at least three classes in the stdlib that just did it manually without help...). Of course the existing implementations don't give you a way to statically declare the types of method arguments, attribute/properties, etc., but MyPy.Protocol does, and that can easily be adopted into the stdlib as part of this proposal. (In fact, I think it's already on the list.)

So, for Ethan's case, the last argument is just "payment_date: datetime.abc.Date", except that nobody has added that to the stdlib yet, so instead he has to write it himself and use it.

From abarnert at yahoo.com  Sun Aug 17 00:36:26 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 16 Aug 2014 15:36:26 -0700
Subject: [Python-ideas] keyword for introducing generators
In-Reply-To: <CAPTjJmprOX0Yxv6xomZThuPe+uq_Mkj32wMr4GmHDX22vpiEmw@mail.gmail.com>
References: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com>
 <CAPTjJmprOX0Yxv6xomZThuPe+uq_Mkj32wMr4GmHDX22vpiEmw@mail.gmail.com>
Message-ID: <17013F47-88B8-4A1C-8EC9-8841DBD7BB1D@yahoo.com>

On Aug 16, 2014, at 14:55, Chris Angelico <rosuav at gmail.com> wrote:

> On Sun, Aug 17, 2014 at 7:46 AM, Neil Girdhar <mistersheik at gmail.com> wrote:
>> I'm sure this has been suggested before, but I just spent two days trying to
>> figure out why a method wasn't being called only to find that I'd
>> accidentally pasted a yield into the function.  What is the argument against
>> a different keyword for introducing generator functions/methods?
> 
> There are quite a few changes to a function based on its body, like
> how the presence of assignment causes a name to be local unless
> explicitly declared otherwise. It's not necessary to predeclare
> everything.
> 
> But if you're having trouble with a function like that, maybe a little
> decorator would help:
> 
> def announce(f):
>    def inner(*a,**kw):
>        print("Calling:",f.__name__)
>        ret=f(*a,**kw)
>        print("Return value is a",type(ret))
>        return ret
>    return inner

I think it would be both simpler and more useful for him to write:

def generator(f):
    assert inspect.isgeneratorfunction(f)
    return f

def function(f):
    asset not inspect.isgeneratorfunction(f)
    return f

Then he can just declare his functions as @generator or @function as appropriate and get an error at definition time if he accidentally got something wrong.

From barry at python.org  Sun Aug 17 01:26:46 2014
From: barry at python.org (Barry Warsaw)
Date: Sat, 16 Aug 2014 19:26:46 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAGE7PN+TyBsUfE_yjZR5VLsJ4oNkJ6zgr_gR_KkGZGvSGYSkTQ@mail.gmail.com>
 <CAP7+vJJwsR0cWPhwuPcX6gk1m-CNaOySY2BaLLb_pUZkAGqpLg@mail.gmail.com>
 <20140815181737.1e1279b4@anarchist.localdomain>
 <87egwhgmj2.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <20140816192646.78d2c76e@anarchist.wooz.org>

On Aug 16, 2014, at 12:07 PM, Stephen J. Turnbull wrote:

>Barry Warsaw writes:
>
> > OTOH there *is* a cost that could negatively impact readability.
> > It's more text visually assaulting you. :)
>
>Not if you have a syntax-highlighting editor that has a no-see-um
>face or text property.

Feel free to share your font-lock definitions <wink>

-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/93822c7d/attachment.sig>

From lukasz at langa.pl  Sun Aug 17 01:28:24 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sat, 16 Aug 2014 16:28:24 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53EEA521.7020406@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <BC3A5CB6-68E7-4DDF-B9B3-65F2FB24BF97@langa.pl>
 <CAP7+vJLS_4Kr+7_SbdBXcEnjWVnLCvjm40FXMDLDC2d_DOaFAw@mail.gmail.com>
 <A0A1D905-E138-4D54-B47C-A437C0EE3BAB@langa.pl>
 <53EEA521.7020406@canterbury.ac.nz>
Message-ID: <F4058614-29DD-4C7B-8A88-FF560731CF68@langa.pl>

On Aug 15, 2014, at 5:26 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> ?ukasz Langa wrote:
> 
>> class C:
>>  cls_member: str = ?on the class?
>>  def __init__(self):
>>    self.obj_member: str = ?on the instance'
>>    self.cls_member = 2   # that?s the real question: type error or an instance member?
> 
> I think not treating it as an error would make it hard
> to reason about the type of x.cls_member for an instance
> x of C. Its type would depend on whether del x.cls_member
> had been performed on x.
> 
> Code which relied on them being different types would
> be rather confusing to a human reader too, so it's
> probably fine to discourage that.

That was my reasoning exactly. +1

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/f5e3e861/attachment.html>

From guido at python.org  Sun Aug 17 01:42:36 2014
From: guido at python.org (Guido van Rossum)
Date: Sat, 16 Aug 2014 16:42:36 -0700
Subject: [Python-ideas] keyword for introducing generators
In-Reply-To: <53EFD891.5030602@sotecware.net>
References: <084223ab-aa29-43e3-b3bd-1af57b07d859@googlegroups.com>
 <53EFD891.5030602@sotecware.net>
Message-ID: <CAP7+vJKALLVRJofmuontt3Tt23p4EyHrsrrOdMj8cpFZMCrrag@mail.gmail.com>

On Sat, Aug 16, 2014 at 3:17 PM, Jonas Wielicki <j.wielicki at sotecware.net>
wrote:

> On 16.08.2014 23:46, Neil Girdhar wrote:
> > I'm sure this has been suggested before, but I just spent two days trying
> > to figure out why a method wasn't being called only to find that I'd
> > accidentally pasted a yield into the function.  What is the argument
> > against a different keyword for introducing generator functions/methods?
> >
> > If it's backward compatibility, then my suggestion to have a from
> > __future__ and then make it real in Python 4.
>
> For what it?s worth, I know this problem very well, and it can take
> hours to figure out whats wrong.
>

A linter should be able to figure this out. For example, mypy will insist
that a generator has a return type of Iterable[...]. So maybe you won't
have to wait for Python 4; if the mypy proposal goes forward you will be
able to use type annotations to distinguish generators.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/c8b8a747/attachment.html>

From steve at pearwood.info  Sun Aug 17 04:08:51 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 17 Aug 2014 12:08:51 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53EFBD98.2090802@stoneleaf.us>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us>
Message-ID: <20140817020851.GL4525@ando>

On Sat, Aug 16, 2014 at 01:22:48PM -0700, Ethan Furman wrote:

> As a test case for what code may soon look like, here's a bit from one of 
> my code bases:
> 
> 
> --------------------------------------------------------------------
> class ACHPayment(object):
>     """A single payment from company to a vendor."""
> 
>     def __init__(self,
>             description, sec_code,
>             vendor_name, vendor_inv_num, vendor_rtng, vendor_acct,
>             transaction_code, vendor_acct_type, amount, payment_date):
>         """
>         description:  10 chars
>         sec_code: 'CCD' or 'CTX'
>         vendor_name: 22 chars
>         vendor_inv_num: 15 chars
>         vendor_rtng: 9 chars
>         vendor_acct: 17 chars
>         transaction_code: ACH_ETC code (enum)
>         vendor_acct_type: 'domestic' or 'foreign'
>         amount: 10 digits (pennies)
>         payment_date: date payment should occur on (datetime.date type 
>         class)
>         """
> --------------------------------------------------------------------
> 
> 
> The question:  what would this look like with type annotations?  As a point 
> of interest, the last parameter, payment_date, can be /anything/ that 
> quacks like a datetime.date -- I tend to use my own dbf.Date class, which 
> subclasses object, not datetime.date itself.


I don't think this is a shining example of the value of static typing, 
at least not by default. As I see it, you would get something like this:

    def __init__(self,
            description:str, sec_code:str,
            vendor_name:str, vendor_inv_num:str, 
            vendor_rtng:str, vendor_acct:str, 
            transaction_code:str, vendor_acct_type:str, 
            amount:int, payment_date:Any)->None:

which may not give you much additional value. In this case, I think that 
the static checks will add nothing except (perhaps) allow you to forgo 
writing a few isinstance checks. You still have to check that the 
strings are the right length, and so on.

But if you're willing to invest some time creating individual str 
subclasses, you can push the length checks into the subclass 
constructor, and write something like this:

    def __init__(self,
            description:Str10, sec_code:SecurityCode,
            vendor_name:Str22, vendor_inv_num:Str15, 
            vendor_rtng:Str9, vendor_acct:Str17, 
            transaction_code:ACH_ETC, vendor_acct_type:VendorAcctType, 
            amount:Pennies, payment_date:DateABC)->None:

Without knowing your application in detail, it is difficult to know how 
much work you should hand over to the type system, and how much you 
should continue to do in Python. If all you're doing is pushing strings 
from one place to another, you might not care exactly how long the 
string is, say because they're truncated when you print them.



-- 
Steven

From guido at python.org  Sun Aug 17 07:03:48 2014
From: guido at python.org (Guido van Rossum)
Date: Sat, 16 Aug 2014 22:03:48 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <20140817020851.GL4525@ando>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
Message-ID: <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>

I'd like to summarize the main issues that have come up. As an experiment,
I'm not changing the subject, but I am still not quoting anything in
particular. Only two issues (or issue clusters) really seem contentious:

(1) Should the function annotation syntax (eventually) be reserved for type
annotations in a standard syntax? Or can multiple different uses of
annotations coexist? And if they can, how should a specific use be
indicated? (Also, some questions about compile-time vs. run-time use.)

(2) For type annotations, should we adopt (roughly) the mypy syntax or the
alternative proposed by Dave Halter? This uses built-in container notations
as a shorthand, e.g. {str: int} instead of Dict[str, int]. This also
touches on the issue of abstract vs. concrete types (e.g. iterable vs.
list).

Regarding (1), I continue to believe that we should eventually reserve
annotations for types, to avoid confusing both humans and tools, but I
think there's nothing we have to do in 3.5 -- 3.5 must preserve backward
compatibility, and we're not proposing to give annotations any new
semantics anyway -- the actual changes to CPython are limited to a new
stdlib module (typing) and some documentation.

Perhaps a thornier issue is how mypy should handle decorators that
manipulate the signature or annotations of the function they wrap. But I
think the only reasonable answer here can be that mypy must understand what
decorators do if it wants to have any chance at type-checking decorated
functions. I don't actually know how sophisticated mypy's understanding of
decorators is, currently, but I don't think there's anything fundamentally
more difficult than all the other things it must understand.

Moving on to (2), the proposal is elegant enough by itself, and indeed has
the advantage of being clear and concise: [T] instead of List[T], {T: U}
instead of Dict[T, U], and so on. However, there are a few concerns.

My first concern is that these expressions are only unambiguous in the
context of function annotations. I want to promote the use of type aliases,
and I think in general a type alias should behave similarly to an ABC. In
particular, I think that any object used to represent a type in an
annotation should itself be a type object (though you may not be able to
instantiate it), and e.g. [int] doesn't satisfy that requirement. Without
this, it would be difficult to implement isinstance() and issubclass() for
type aliases -- and while we could special-case lists, sets and dicts,
using a tuple *already* has a meaning!

The second concern is that the proposal seems to steer users in the
direction of using concrete types. A lot of Python's power stems from
concepts like iterable and mapping and their variants (e.g. iterable,
container, sequence, mutable sequence). There are justified concerns that
users will unnecessarily constrain the argument types more than necessary
(e.g. specifying a sequence where any iterable would do), and this proposal
lacks the subtlety to express the difference.

A third (minor) concern reflects issue (1): until we have agreement that
annotations should only be used as type annotations, a type checker cannot
assume that the presence of annotations means that types should be checked.
Using e.g. Iterable[int] is pretty unambiguous (especially when Iterable is
imported from typing.py), whereas just [int] is somewhat ambiguous. I call
this only a minor issue because it still occurs for simple types like int
or str, so if we can live with it for those we could presumably live with
[int] and {str: float}.

All in all I prefer the mypy syntax, despite being somewhat more verbose
and requiring an import, with one caveat: I agree that it would be nicer if
the mypy abstract collection types were the same objects as the ABCs
exported by collections.abc. I'm not quite sure whether we should also
change the concrete collection types from List, Dict, Set, Tuple to list,
dict, set, tuple; the concrete types are so ubiquitous that I worry that
there may be working code out there that somehow relies on the type objects
themselves not being subscriptable.

A mostly unrelated issue: there are two different uses of tuples, and we
need a notation for both. One is a tuple of fixed length with
heterogeneous, specific types for the elements; for example Tuple[int,
float]. But I think we also need a way to indicate that a function expects
(or returns) a variable-length tuple with a homogeneous element type.
Perhaps we should call this type frozenlist, analogous to frozenset (and it
seems there's a proposal for frozendict making the rounds as well).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140816/9d2eb0bf/attachment.html>

From abarnert at yahoo.com  Sun Aug 17 08:02:01 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 16 Aug 2014 23:02:01 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
Message-ID: <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com>

On Aug 16, 2014, at 22:03, Guido van Rossum <guido at python.org> wrote:

> Moving on to (2), the proposal is elegant enough by itself, and indeed has the advantage of being clear and concise: [T] instead of List[T], {T: U} instead of Dict[T, U], and so on. However, there are a few concerns.
> 
> My first concern is that these expressions are only unambiguous in the context of function annotations.

Good point. Together with your third point (that [str] could be meaningful as a different type of annotation, while Iterable[str] is incredibly unlikely to mean anything other than a static type check--except maybe a runtime type check, but I think it's reasonable to assume they can share annotations), I think this kills the idea. Pity, because I like the way it looked.

> All in all I prefer the mypy syntax, despite being somewhat more verbose and requiring an import, with one caveat: I agree that it would be nicer if the mypy abstract collection types were the same objects as the ABCs exported by collections.abc. I'm not quite sure whether we should also change the concrete collection types from List, Dict, Set, Tuple to list, dict, set, tuple; the concrete types are so ubiquitous that I worry that there may be working code out there that somehow relies on the type objects themselves not being subscriptable.

I won't belabor the point, but again: I don't think we need a generic list type object, and without it, this entire problem--your only remaining problem that isn't a mere stylistic choice--vanishes.

> A mostly unrelated issue: there are two different uses of tuples, and we need a notation for both. One is a tuple of fixed length with heterogeneous, specific types for the elements; for example Tuple[int, float]. But I think we also need a way to indicate that a function expects (or returns) a variable-length tuple with a homogeneous element type. Perhaps we should call this type frozenlist, analogous to frozenset (and it seems there's a proposal for frozendict making the rounds as well).

Even if you drop the idea for [str] and {int: str}, which I agree seems unavoidable, I think it may still make sense for (int, str) to mean a heterogeneous iterable.

Python already has target lists, argument lists, parameter lists, and expression lists that all have the same syntax as tuples or a superset thereof, but don't define tuples. In (a, b) = zip(c, d), neither (a, b) nor (c, d) is a tuple, and I don't think anyone is confused by that. So, why can't def foo(spam: (int, str)) mean that spam is an iterable of an int and a str, in exactly the same way that the assignment statement means that a and b are assigned the result of unpacking the iterable returned by zip when called with c and d?

And this leaves Tuple[str] or tuple[str] free to mean a homogenous tuple (although, again, I don't think we even want or need that...).


From steve at pearwood.info  Sun Aug 17 09:26:02 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 17 Aug 2014 17:26:02 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com>
Message-ID: <20140817072602.GN4525@ando>

On Sat, Aug 16, 2014 at 11:02:01PM -0700, Andrew Barnert wrote:
> On Aug 16, 2014, at 22:03, Guido van Rossum <guido at python.org> wrote:
> 
> > Moving on to (2), the proposal is elegant enough by itself, and indeed has the advantage of being clear and concise: [T] instead of List[T], {T: U} instead of Dict[T, U], and so on. However, there are a few concerns.
> > 
> > My first concern is that these expressions are only unambiguous in the context of function annotations.
> 
> Good point. Together with your third point (that [str] could be 
> meaningful as a different type of annotation, while Iterable[str] is 
> incredibly unlikely to mean anything other than a static type 
> check--except maybe a runtime type check, but I think it's reasonable 
> to assume they can share annotations), I think this kills the idea. 
> Pity, because I like the way it looked.

[str] looks nice, but it looks like a list of str, or possibly an 
optional str, e.g. from the docstring of int:

    int(x[, base]) -> integer

What the [str] syntax doesn't look like is an Iterable of str. Or should 
that be Sequence of str? MutableSequence perhaps? If [str] means 
something other than list of str, it is going to be some arbitrary 
special case to be memorized.

Have pity on people teaching Python. I don't want to have to try to 
explain to beginners why [str] sometimes means a list and sometimes an 
arbitrary Iterable (or whatever). This is just downright confusing:

def func(arg:[str]):
    x = [str]
    assert isinstance(x, list)  # Always passes.
    assert isinstance(arg, list)  # Sometimes fails.

func(iter("abc"))  # Fails.


[Guido]
> > All in all I prefer the mypy syntax, despite being somewhat more 
> > verbose and requiring an import, with one caveat: I agree that it 
> > would be nicer if the mypy abstract collection types were the same 
> > objects as the ABCs exported by collections.abc. I'm not quite sure 
> > whether we should also change the concrete collection types from 
> > List, Dict, Set, Tuple to list, dict, set, tuple;

We can start with typing.List, Dict, etc., and later on consider using 
builtins.

I worry that if we use builtins, people will declare x:list[int] not 
because they *need* a list of int, but because it saves typing over 

from typing import Sequence, Integer
def func(x:Sequence[Integer]):

So even though I suggested earlier that the builtins grow appropriate 
__getitem__ methods, on second thoughts I would be very cautious about 
introducing that.


> > the concrete types 
> > are so ubiquitous that I worry that there may be working code out 
> > there that somehow relies on the type objects themselves not being 
> > subscriptable.

[Andrew] 
> I won't belabor the point, but again: I don't think we need a generic 
> list type object, and without it, this entire problem--your only 
> remaining problem that isn't a mere stylistic choice--vanishes.

I don't understand. If there's no list typing object, how do you declare 
a variable must be a list and nothing but a list? Or that it returns a 
list?


[Guido]
> > A mostly unrelated issue: there are two different uses of tuples, 
> > and we need a notation for both. One is a tuple of fixed length with 
> > heterogeneous, specific types for the elements; for example 
> > Tuple[int, float]. But I think we also need a way to indicate that a 
> > function expects (or returns) a variable-length tuple with a 
> > homogeneous element type.

Throwing this idea out to be shot down: use some sort of slice notation.

Tuple[int, float, str]  # Like (23, 1.5, "spam")

Tuple[::int, float, str]  # Like (1, 2, 3, 4) or (1.5,) or ("x", "y")

That is, if the argument to __getitem__ is a slice (None, None, T), T is 
either a type or a tuple of types. Any other kind of slice is reserved 
for the future, or an error.

> > Perhaps we should call this type 
> > frozenlist, analogous to frozenset (and it seems there's a proposal 
> > for frozendict making the rounds as well).

[Andrew]
> Even if you drop the idea for [str] and {int: str}, which I agree 
> seems unavoidable, I think it may still make sense for (int, str) to 
> mean a heterogeneous iterable.

That makes no sense to me. It looks like a tuple, not a generic iterable 
object. Your interpretation has the same problems I discussed above for 
[str] notation: it is an arbitrary choice whether (...) means Iterable, 
Sequence or ImmutableSequence, and it doesn't fit nicely with other 
common uses of parens. See below.


> Python already has target lists, argument lists, parameter lists, and 
> expression lists that all have the same syntax as tuples or a superset 
> thereof, but don't define tuples. In (a, b) = zip(c, d), neither (a, 
> b) nor (c, d) is a tuple, and I don't think anyone is confused by 
> that. 

Ha , you've obviously stopped reading the "Multi-line with statement" 
thread on Python-Dev :-)


> So, why can't def foo(spam: (int, str)) mean that spam is an 
> iterable of an int and a str, in exactly the same way that the 
> assignment statement means that a and b are assigned the result of 
> unpacking the iterable returned by zip when called with c and d?

But a, b = zip(c, d) requires that there be exactly two elements, not 
some unspecified number.

To me, spam:(int, str) has a natural interpretation that spam can be 
either an int or a str, not an Iterable or Sequence or even a tuple.

> And this leaves Tuple[str] or tuple[str] free to mean a homogenous 
> tuple (although, again, I don't think we even want or need that...).

We do. Consider the isinstance() function. Here's the signature 
according to its docstring:

isinstance(object, class-or-type-or-tuple) -> bool

The second argument can be a single type, or a tuple of an arbitrary 
number of types. I'd write it with annotations like:

def isinstance(object:Any, 
               class_or_type_or_tuple:(Type, Tuple[::Type])
               )->Bool:

assuming (a,b) means "either a or b" and Tuple[::a] means a homogenous 
tuple of a. (With shorter argument names, it even fits on one line.)

And, here's issubclass:

def issubclass(C:Type, D:(Type, Tuple[::Type]))->Bool:



-- 
Steven

From stefan_ml at behnel.de  Sun Aug 17 09:28:56 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 17 Aug 2014 09:28:56 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
Message-ID: <lspljq$bqt$1@ger.gmane.org>

Guido van Rossum schrieb am 17.08.2014 um 07:03:
> I'd like to summarize the main issues that have come up. As an experiment,
> I'm not changing the subject, but I am still not quoting anything in
> particular. Only two issues (or issue clusters) really seem contentious:
> 
> (1) Should the function annotation syntax (eventually) be reserved for type
> annotations in a standard syntax? Or can multiple different uses of
> annotations coexist? And if they can, how should a specific use be
> indicated? (Also, some questions about compile-time vs. run-time use.)
>
> Regarding (1), I continue to believe that we should eventually reserve 
> annotations for types, to avoid confusing both humans and tools, but I 
> think there's nothing we have to do in 3.5 -- 3.5 must preserve
> backward compatibility, and we're not proposing to give annotations any
> new semantics anyway -- the actual changes to CPython are limited to a
> new stdlib module (typing) and some documentation.

As I mentioned before, there is more than one kind of type, even if we
stick to reserving annotations for type declarations. That's why Cython
currently supports these four ways of type annotations (in addition to its
own non-Python way with "cdef"):

    x: dict
    x: {"type": dict}
    x: {"type": "dict"}
    x: {"ctype": "long double"}

The latter three can also be combined, so you could declare a C type for
Cython compilation and a Python type for your IDE and other static Python
analysis tools, e.g.

    x: {"type": int, "ctype": "size_t"}

Note that this also helps at a documentation level. The expected input is a
Python int, but in fact it's restricted to a C size_t by the native
implementation.

I'd still vote for allowing the simpler "x: dict" as well for cases where
it's the only annotation. It's easy enough to switch to the explicit
notation when you want to add a second (potentially non-type) annotation.

So, rather than "reserving" annotations for type declarations, I vote for
making type annotations the default, but allowing other annotations by
putting them into a dict that gives each annotation a string name. That
name could be a module name in the stdlib or on PyPI, for example.


> (2) For type annotations, should we adopt (roughly) the mypy syntax or the
> alternative proposed by Dave Halter? This uses built-in container notations
> as a shorthand, e.g. {str: int} instead of Dict[str, int]. This also
> touches on the issue of abstract vs. concrete types (e.g. iterable vs.
> list).

I talked to him at EP14 and we agreed that the simpler syntax looks
tempting. However, it does not support protocols, so it still needs
something that allows us to say Iterable(int) in some way. I always thought
that the ABCs were made for that, but so far everyone seemed to think that
we need something different again. I'm happy to see that your preference
also goes in that direction now. Having yet another typing module seems
like an unnecessary duplication of the type system.

Stefan



From nicholas.cole at gmail.com  Sun Aug 17 09:41:33 2014
From: nicholas.cole at gmail.com (Nicholas Cole)
Date: Sun, 17 Aug 2014 08:41:33 +0100
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <20140817020851.GL4525@ando>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
Message-ID: <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>

On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano <steve at pearwood.info> wrote:

> I don't think this is a shining example of the value of static typing,
> at least not by default. As I see it, you would get something like this:
>
>     def __init__(self,
>             description:str, sec_code:str,
>             vendor_name:str, vendor_inv_num:str,
>             vendor_rtng:str, vendor_acct:str,
>             transaction_code:str, vendor_acct_type:str,
>             amount:int, payment_date:Any)->None:
>
> which may not give you much additional value. In this case, I think that
> the static checks will add nothing except (perhaps) allow you to forgo
> writing a few isinstance checks. You still have to check that the
> strings are the right length, and so on.
>
> But if you're willing to invest some time creating individual str
> subclasses, you can push the length checks into the subclass
> constructor, and write something like this:
>
>     def __init__(self,
>             description:Str10, sec_code:SecurityCode,
>             vendor_name:Str22, vendor_inv_num:Str15,
>             vendor_rtng:Str9, vendor_acct:Str17,
>             transaction_code:ACH_ETC, vendor_acct_type:VendorAcctType,
>             amount:Pennies, payment_date:DateABC)->None:

I know that the BDFL has spoken on this issue and said that he finds
all of this readable and "pythonic", but these examples perfectly
capture what I am going to dislike about this syntax as it becomes
popular.

I suppose it will be better when I am reading it in an editor that has
syntax highlighting but as it stands I had to stare at that block of
code for a long time to see how many and what type of arguments it
called.  At first I thought you had one per line, then I thought you
had variable numbers per line.  On about the fourth or fifth reading,
I saw you had two per line.  The problem with the syntax (I think) is
that it relies on readers spotting characters like ":" and "[",
characters which change how the eye should parse the line (assuming
this is going to be optional).  I find that those characters get lost
very easily in long function definitions, leaving the reading having
to read and re-read the block to answer questions like, 1. how many
arguments are there? 2. Are any of them keyword arguments? Are they
all the same type? 3. What are their names? In the example above, my
eye keeps wanting to tell me that one of them is called SecurityCode,
for example, even though I know that is the name of a class. This all
seems unpythonic to me.  Most of python's syntax is expressed in words
rather than compact symbols.

My fear with all of this is that it turns python into a language that
is harder for humans to read.  I much prefer the PyCharm docstring
approach, because the eye can scan the function signature quickly and
then the brain can say, "Ah - 10 arguments, oh, and I see that they
have to be particular types and the return code is specified."  To put
it another way, current python function signatures are immediately
intuitive even to someone who is unfamiliar with the language.  There
is nothing intuitive about this.  It is more like looking at
ObjectiveC or similar.

At root, I don't totally understand what is "Pythonic" about function
signatures. On the other hand, more expert people than me seem to like
the above, and so I am sure that I am missing something.  Perhaps it
is simply the DRY principle.  On the other hand, I am sure that
readability issues are not simply a matter of personal taste.  I've
just re-read Tog on Inerterface, and perhaps that is colouring my
thought!

Anyone who is at all dyslexic is, I think, going to struggle!

PEP8 should probably specify one argument per line if this kind of
syntax is going to be at all re-readable.

However, I do accept that the BDFL has spoken, and I'll "get with the
program"!  I'm sure I'll get used to it.

N.

From sf at fermigier.com  Sun Aug 17 10:07:22 2014
From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=)
Date: Sun, 17 Aug 2014 10:07:22 +0200
Subject: [Python-ideas]  RFC: Multiple Dispatch
Message-ID: <CABuJJj4kv+g_3c9Q_FbVSL7LXeU-7_Ow1hYcoBeFjhTD7xfHRg@mail.gmail.com>

GvR wrote:
> Are there good uses of singledispatch in the wild even?

Martijn Faassen wrote a whole Web framework (Morepath =
http://morepath.readthedocs.org/en/latest/) on top of it:

http://blog.startifact.com/posts/reg-now-with-more-generic.html

S.

-- 
Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier -
http://linkedin.com/in/sfermigier
Founder & CEO, Abilian - Enterprise Social Software -
http://www.abilian.com/
Founder, Nuxeo - Enterprise Content Management Platform -
http://www.nuxeo.com/
Co-Founder and Chairman, Free&OSS Group / Systematic Cluster -
http://www.gt-logiciel-libre.org/
Co-Founder & Vice-President, National Council for Free & Open Source
Software (CNLL) - http://cnll.fr/
Co-Founder & Co-organiser, Open World Forum -
http://www.openworldforum.paris/fr/
---
"No problem is too small or too trivial if we can really do something about
it." - Richard P. Feynman
"Well done is better than well said." - Benjamin Franklin
"There's no such thing as can't. You always have a choice." - Ken Gor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/cb73519d/attachment.html>

From ben+python at benfinney.id.au  Sun Aug 17 10:23:52 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 17 Aug 2014 18:23:52 +1000
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing -- the
	crossroads)
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
Message-ID: <854mxb5xt3.fsf_-_@benfinney.id.au>

Guido van Rossum <guido at python.org> writes:

> A mostly unrelated issue: there are two different uses of tuples, and
> we need a notation for both. One is a tuple of fixed length with
> heterogeneous, specific types for the elements; for example Tuple[int,
> float].

That's the meaning of a tuple data structure, to me.

> But I think we also need a way to indicate that a function expects (or
> returns) a variable-length tuple with a homogeneous element type.

Why? What real-world uses are there, where a list won't do the job
adequately?

I have encountered many uses of ?homogeneous, variable-length sequence?
and every time a Python tuple is used for that, I perceive a Python list
would be better precisely *because* it better indicates that semantic
meaning.

I'd like to know how you think that's not true, and what real-world code
makes you think so.

-- 
 \      ?Contentment is a pearl of great price, and whosoever procures |
  `\        it at the expense of ten thousand desires makes a wise and |
_o__)                                      happy purchase.? ?J. Balguy |
Ben Finney


From steve at pearwood.info  Sun Aug 17 10:34:53 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 17 Aug 2014 18:34:53 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>
Message-ID: <20140817083453.GO4525@ando>

On Sun, Aug 17, 2014 at 08:41:33AM +0100, Nicholas Cole wrote:
> On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano <steve at pearwood.info> wrote:
[...]
> >     def __init__(self,
> >             description:Str10, sec_code:SecurityCode,
> >             vendor_name:Str22, vendor_inv_num:Str15,
> >             vendor_rtng:Str9, vendor_acct:Str17,
> >             transaction_code:ACH_ETC, vendor_acct_type:VendorAcctType,
> >             amount:Pennies, payment_date:DateABC)->None:
> 
> I know that the BDFL has spoken on this issue and said that he finds
> all of this readable and "pythonic", but these examples perfectly
> capture what I am going to dislike about this syntax as it becomes
> popular.

Even though I am in favour of the proposal, I do sympathise, and I see 
what you mean. But, I think it is important to realise that a method 
with ten arguments (plus self) is not going to be exactly readable at 
the best of times.


> I suppose it will be better when I am reading it in an editor that has
> syntax highlighting but as it stands I had to stare at that block of
> code for a long time to see how many and what type of arguments it
> called.  At first I thought you had one per line, then I thought you
> had variable numbers per line.  On about the fourth or fifth reading,
> I saw you had two per line.

Look for the commas :-)

But yes, as given that makes a big wall of text. I suppose it will take 
some time for people to decide what formatting works best for them. It 
might help to align the arguments in columns (even though that goes 
against PEP-8):

     def __init__(self,
            description:Str10,        sec_code:SecurityCode,
            vendor_name:Str22,        vendor_inv_num:Str15,
            vendor_rtng:Str9,         vendor_acct:Str17,
            transaction_code:ACH_ETC, vendor_acct_type:VendorAcctType,
            amount:Pennies,           payment_date:DateABC,
            ) -> None:

That works for me.


> My fear with all of this is that it turns python into a language that
> is harder for humans to read.

Declaring types in the function parameter list is very common, in many 
languages. If it were *that* much harder to read, languages wouldn't 
keep using it. (Not many languages follow Forth or APL syntax.) Perhaps 
because I learned to program in Pascal, I find the annotation syntax 
very easy to read, but, yes, anything which increases the density of 
information per line risks hurting readability a little.

It's a tradeoff, and of course all of this is optional. Syntax 
highlighting will help, and I expect that in a few years time emacs and 
vim will have some way to hide annotations when editing code :-)


-- 
Steven

From ncoghlan at gmail.com  Sun Aug 17 10:50:08 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Aug 2014 18:50:08 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <20140817083453.GO4525@ando>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>
 <20140817083453.GO4525@ando>
Message-ID: <CADiSq7dHi1UdyZd3yNz8ifwvKMC-q8K8JkCUX4_objyYLdbk0g@mail.gmail.com>

On 17 August 2014 18:34, Steven D'Aprano <steve at pearwood.info> wrote:
>
> Declaring types in the function parameter list is very common, in many
> languages. If it were *that* much harder to read, languages wouldn't
> keep using it. (Not many languages follow Forth or APL syntax.) Perhaps
> because I learned to program in Pascal, I find the annotation syntax
> very easy to read, but, yes, anything which increases the density of
> information per line risks hurting readability a little.

I once had the "pleasure" of inheriting some code written in K&R style
C, where the parameter type declarations were separate from the
signature line:

    void foo(a, b, c)
        double a;
        char b;
    {
          ...
    }

ANSI C, with inline typing,  is far more readable :)

When it comes to the readability of function headers with lots and
lots of parameters... I'm in the "those are inherently unreadable,
even if sometimes an unfortunate necessity" camp :)

Reorganising-the-subprocess-module-docs-was-interesting'ly,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tjreedy at udel.edu  Sun Aug 17 10:51:46 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 17 Aug 2014 04:51:46 -0400
Subject: [Python-ideas] Variable-length, homogeneous tuple: why?
In-Reply-To: <854mxb5xt3.fsf_-_@benfinney.id.au>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
Message-ID: <lspqfv$vop$1@ger.gmane.org>

On 8/17/2014 4:23 AM, Ben Finney wrote:
> Guido van Rossum <guido at python.org> writes:
>
>> A mostly unrelated issue: there are two different uses of tuples, and
>> we need a notation for both. One is a tuple of fixed length with
>> heterogeneous, specific types for the elements; for example Tuple[int,
>> float].
>
> That's the meaning of a tuple data structure, to me.
>
>> But I think we also need a way to indicate that a function expects (or
>> returns) a variable-length tuple with a homogeneous element type.

There are also fixed-length homogeneous structures, like points.

> Why? What real-world uses are there, where a list won't do the job
> adequately?

Variable-length homogenous tuples are part of python syntax in multiple 
places.

Tuples can be hashed and put in sets an used as dict keys, lists cannot. 
Tuple contants are calculated just once when the code is compiled (and 
typically saved as .pyc).



-- 
Terry Jan Reedy


From abarnert at yahoo.com  Sun Aug 17 10:52:21 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 17 Aug 2014 01:52:21 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <20140817072602.GN4525@ando>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com> <20140817072602.GN4525@ando>
Message-ID: <828F3377-EEF0-4AC6-B721-3DB65FBFA062@yahoo.com>

On Aug 17, 2014, at 0:26, Steven D'Aprano <steve at pearwood.info> wrote:

> On Sat, Aug 16, 2014 at 11:02:01PM -0700, Andrew Barnert wrote:
>> 
> 
>> I won't belabor the point, but again: I don't think we need a generic 
>> list type object, and without it, this entire problem--your only 
>> remaining problem that isn't a mere stylistic choice--vanishes.
> 
> I don't understand. If there's no list typing object, how do you declare 
> a variable must be a list and nothing but a list? Or that it returns a 
> list?

You think about it and make sure you really do need a list and nothing but a list. Most of the time (as in all three of the examples given in this thread) this is a mistake. If it's not, then you use List. (Or, if the stdlib doesn't provide that, you have to write one line of code: List = TypeAlias(list), and then you can use it.)

If having list[T] is going to be more of an attractive nuisance than a useful feature, and it will be especially attractive and nuisanceful for exactly the same novices who are unlikely to know how to TypeAlias it themselves, why is it a problem to leave it out?

>> Even if you drop the idea for [str] and {int: str}, which I agree 
>> seems unavoidable, I think it may still make sense for (int, str) to 
>> mean a heterogeneous iterable.
> 
> That makes no sense to me. It looks like a tuple, not a generic iterable 
> object. Your interpretation has the same problems I discussed above for 
> [str] notation: it is an arbitrary choice whether (...) means Iterable, 
> Sequence or ImmutableSequence, and it doesn't fit nicely with other 
> common uses of parens. See below.
> 
>> Python already has target lists, argument lists, parameter lists, and 
>> expression lists that all have the same syntax as tuples or a superset 
>> thereof, but don't define tuples. In (a, b) = zip(c, d), neither (a, 
>> b) nor (c, d) is a tuple, and I don't think anyone is confused by 
>> that.
> 
> Ha , you've obviously stopped reading the "Multi-line with statement" 
> thread on Python-Dev :-)

OK, granted, only 4 of the 5 attempts to reuse the comma-separated lists in Python have been 100% successful. Still not a bad batting average.

>> So, why can't def foo(spam: (int, str)) mean that spam is an 
>> iterable of an int and a str, in exactly the same way that the 
>> assignment statement means that a and b are assigned the result of 
>> unpacking the iterable returned by zip when called with c and d?
> 
> But a, b = zip(c, d) requires that there be exactly two elements, not 
> some unspecified number.

And spam:(int, str) requires that there be exactly two elements (and that the first be an int and the second a str), not some unspecified number. How is that any different?

> To me, spam:(int, str) has a natural interpretation that spam can be 
> either an int or a str, not an Iterable or Sequence or even a tuple.

OK, I see the parallel there with exception statements now that you mention it.

But almost anywhere else in Python, a comma-separated list is a sequence of values, targets, parameters, etc., not a disjunction. The obvious way to spell what you want here is "int | str" (and the fact that it was independently suggested three times on this thread and no other alternatives have been suggested until now makes me feel pretty confident that it really is the obvious way).

Of course there is a _different_ alternative that could be borrowed from some of the typed functional languages: int * str. But I don't think that's at all obvious to a Python reader.


From greg.ewing at canterbury.ac.nz  Sun Aug 17 11:22:45 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 17 Aug 2014 21:22:45 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
Message-ID: <53F07465.1070103@canterbury.ac.nz>

Guido van Rossum wrote:

> Perhaps a thornier issue is how mypy should handle decorators that 
> manipulate the signature or annotations of the function they wrap. But I 
> think the only reasonable answer here can be that mypy must understand 
> what decorators do if it wants to have any chance at type-checking 
> decorated functions.

Seems to me the only way to do that in general is to
execute the decorators. That means importing everything
the decorators depend on and probably running at least
the top-level module code. Is executing arbitrary
code at type-checking time really desirable?

-- 
Greg

From greg.ewing at canterbury.ac.nz  Sun Aug 17 11:33:13 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 17 Aug 2014 21:33:13 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <lspljq$bqt$1@ger.gmane.org>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <lspljq$bqt$1@ger.gmane.org>
Message-ID: <53F076D9.4020807@canterbury.ac.nz>

Stefan Behnel wrote:
> However, it does not support protocols, so it still needs
> something that allows us to say Iterable(int) in some way.

Just had a thought -- does mypy provide a way to express
a type that supports more than one protocol? E.g. can you
say that something must be both Iterable and Hashable?

-- 
Greg

From lukasz at langa.pl  Sun Aug 17 11:41:17 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sun, 17 Aug 2014 02:41:17 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
Message-ID: <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl>

On Aug 16, 2014, at 10:03 PM, Guido van Rossum <guido at python.org> wrote:

> I'd like to summarize the main issues that have come up. As an experiment, I'm not changing the subject, but I am still not quoting anything in particular. Only two issues (or issue clusters) really seem contentious:
> 
> (1) Should the function annotation syntax (eventually) be reserved for type annotations in a standard syntax? Or can multiple different uses of annotations coexist? And if they can, how should a specific use be indicated? (Also, some questions about compile-time vs. run-time use.)

Consider what Stefan Behnel is proposing: using function annotations for types by default, but in the presence of a dictionary, search for type in the 'type' key. This is very nice, provides a way to be concise if possible, and generic, if needed.

My suggestion: we should support that.


> All in all I prefer the mypy syntax, despite being somewhat more verbose and requiring an import, with one caveat: I agree that it would be nicer if the mypy abstract collection types were the same objects as the ABCs exported by collections.abc.

Good :) If the functionality will be implemented in the ABCs, what is the purpose of the typing module?

My suggestion: if the functionality will be implemented in the ABCs, there's no need to introduce the "typing" module. We can back-port the new ABCs, for sure, but for Python 3.5 `collections` is enough (already has aliases to collections.abc0.


> I'm not quite sure whether we should also change the concrete collection types from List, Dict, Set, Tuple to list, dict, set, tuple; the concrete types are so ubiquitous that I worry that there may be working code out there that somehow relies on the type objects themselves not being subscriptable.

While unlikely, such code can exist in the wild. That being said, I think builtins should support the one-obvious-way-to-do-it syntax for generics, if only for uniformity. Please note that there also can be code in the future that relies on type objects not to support binary-or. We will still need to add this, though, a type union of (int | str) or (str | None) will be a common thing.

My suggestion: add __getitem__ and __or__/__ror__ to both builtins and ABCs.

Steven D'Aprano touches on an interesting point that list[int] will be tempting for users because Iterable[int] is both longer and requires an import. We could extend PEP 8 to talk about typing and how people should think about introducing hints, but I think Steven is generally right: list[int] will sadly win.

The real reason that we'll see list[T] everywhere is that Iterable[str] == str and Sequence[str] == str. Whoever will try and fail to use those abstract types to specify a collection of strings, but *not* a single string, will migrate to using concrete data types. And that's such a common use case!

AFAIK, there is no good workaround at the moment. The solution for that, which we sadly can't implement, would be to make strings non-iterable. 

My suggestion: two new ABCs, let me temporarily call them StrictIterable and StrictSequence. Those would return False for issubclass(str, ...). 
My other suggestion: deprecate iterating over strings (and possibly bytes, too?). I'm not saying "remove", but just officially say "this was a bad idea, don't use it".


> A mostly unrelated issue: there are two different uses of tuples, and we need a notation for both. One is a tuple of fixed length with heterogeneous, specific types for the elements; for example Tuple[int, float]. But I think we also need a way to indicate that a function expects (or returns) a variable-length tuple with a homogeneous element type. Perhaps we should call this type frozenlist, analogous to frozenset (and it seems there's a proposal for frozendict making the rounds as well).

On one hand, not being able to represent variable-length homogeneous tuples in the type hints will be a strong signal that this usage of tuples is disputable. On the other hand, we might *need* to support this to be compatible with existing framework code in the wild. frozenlists would be a nice, explicit solution for that but sadly they'd be a new collection, so no today's code returning/accepting var-length tuples would use that.

My suggestion: tuple[int, ...]
For uniformity, we would also accept this form for other iterables, I suppose.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/f73f9e1f/attachment-0001.html>

From steve at pearwood.info  Sun Aug 17 11:41:52 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 17 Aug 2014 19:41:52 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <828F3377-EEF0-4AC6-B721-3DB65FBFA062@yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com> <20140817072602.GN4525@ando>
 <828F3377-EEF0-4AC6-B721-3DB65FBFA062@yahoo.com>
Message-ID: <20140817094152.GP4525@ando>

On Sun, Aug 17, 2014 at 01:52:21AM -0700, Andrew Barnert wrote:
> On Aug 17, 2014, at 0:26, Steven D'Aprano <steve at pearwood.info> wrote:
> 
> > On Sat, Aug 16, 2014 at 11:02:01PM -0700, Andrew Barnert wrote:
> >> 
> > 
> >> I won't belabor the point, but again: I don't think we need a generic 
> >> list type object, and without it, this entire problem--your only 
> >> remaining problem that isn't a mere stylistic choice--vanishes.
> > 
> > I don't understand. If there's no list typing object, how do you declare 
> > a variable must be a list and nothing but a list? Or that it returns a 
> > list?
> 
> You think about it and make sure you really do need a list and nothing 
> but a list. Most of the time (as in all three of the examples given in 
> this thread) this is a mistake. If it's not, then you use List. (Or, 
> if the stdlib doesn't provide that, you have to write one line of 
> code: List = TypeAlias(list), and then you can use it.)

Ah, that is the point I missed. You think that the stdlib shouldn't 
provide a standard typing object for lists, but that people should just 
create their own if they need it.

Okay, but I don't understand why you're singling out lists. If you want 
to propose providing only abstract classes (Sequence, Mapping, etc.) 
and not concrete classes (list, dict, etc.) by default, that makes 
some sense to me. But I don't understand including typing.Dict as 
an alias for dict (say) but not List.


> If having list[T] is going to be more of an attractive nuisance than a 
> useful feature, and it will be especially attractive and nuisanceful 
> for exactly the same novices who are unlikely to know how to TypeAlias 
> it themselves, why is it a problem to leave it out?

There are at least three scenarios:

(1) Built-ins can be used directly in static type annotations:

    x:list[dict]

    This has the advantage of not needing special names, but the 
    disadvantage of encouraging lazy programmers to specify concrete
    types when they should be using abstract Sequence[Mapping].

(2) Built-ins *cannot* be used, you have to import them from typing:

    from typing import List, Dict
    x:List[Dict]

    The advantage is that since you have to do an import anyway,
    it is not much more effort to Do The Right Thing:

    from typing import Sequence, Mapping
    x:Sequence[Mapping]

(3) And the final scenario, the one which confuses me, but seems 
    to be what you are suggesting: you can use the built-ins, 
    *but not list*, and there is no List to import either:

    from typing import Sequence
    x:Sequence[dict]

    I don't understand the advantage of this.


[...]
> >> So, why can't def foo(spam: (int, str)) mean that spam is an 
> >> iterable of an int and a str, in exactly the same way that the 
> >> assignment statement means that a and b are assigned the result of 
> >> unpacking the iterable returned by zip when called with c and d?
> > 
> > But a, b = zip(c, d) requires that there be exactly two elements, not 
> > some unspecified number.
> 
> And spam:(int, str) requires that there be exactly two elements (and 
> that the first be an int and the second a str), not some unspecified 
> number. How is that any different?

Ah, that's what I didn't understand. I thought you meant an iterable of 
either ints or strs, without requiring a fixed number of them.

I must admit, I just assumed that (based on the example of isinstance, 
and general Python practice), unions of types would be represented as a 
tuple, but I see that mypy uses Union[int, str].

In other words, I was thinking:

Iterable[Union[int, str]] == Iterable[(int, str)]

and thought you wanted to drop the Iterable[ ] and just be left with the 
(int, str).


> > To me, spam:(int, str) has a natural interpretation that spam can be 
> > either an int or a str, not an Iterable or Sequence or even a tuple.
> 
> OK, I see the parallel there with exception statements now that you 
> mention it.
> 
> But almost anywhere else in Python, a comma-separated list is a 
> sequence of values, targets, parameters, etc., not a disjunction. The 
> obvious way to spell what you want here is "int | str"

Which mypy spells as Union[ ].

http://mypy-lang.org/tutorial.html



-- 
Steven


From lukasz at langa.pl  Sun Aug 17 11:44:32 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sun, 17 Aug 2014 02:44:32 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53F076D9.4020807@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <lspljq$bqt$1@ger.gmane.org> <53F076D9.4020807@canterbury.ac.nz>
Message-ID: <05931994-74E5-4F9E-9D84-3B272A2C4368@langa.pl>

On Aug 17, 2014, at 2:33 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> Stefan Behnel wrote:
>> However, it does not support protocols, so it still needs
>> something that allows us to say Iterable(int) in some way.
> 
> Just had a thought -- does mypy provide a way to express
> a type that supports more than one protocol? E.g. can you
> say that something must be both Iterable and Hashable?

That would be union types. Current syntax:
Union[Iterable[int], Hashable]

Proposed more concise syntax:
Iterable[int] | Hashable

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/105e9476/attachment.html>

From steve at pearwood.info  Sun Aug 17 11:45:36 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 17 Aug 2014 19:45:36 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53F076D9.4020807@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <lspljq$bqt$1@ger.gmane.org> <53F076D9.4020807@canterbury.ac.nz>
Message-ID: <20140817094536.GQ4525@ando>

On Sun, Aug 17, 2014 at 09:33:13PM +1200, Greg Ewing wrote:
> Stefan Behnel wrote:
> >However, it does not support protocols, so it still needs
> >something that allows us to say Iterable(int) in some way.
> 
> Just had a thought -- does mypy provide a way to express
> a type that supports more than one protocol? E.g. can you
> say that something must be both Iterable and Hashable?

mypy has Union[Iterable, Hashable], but that would mean anything 
iterable, or anything hashable, but not necessarily both at the same 
time. I don't see anything that says it must support both, but I've 
only gone through the tutorial:

http://mypy-lang.org/tutorial.html




-- 
Steven

From lukasz at langa.pl  Sun Aug 17 11:49:59 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sun, 17 Aug 2014 02:49:59 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <05931994-74E5-4F9E-9D84-3B272A2C4368@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <lspljq$bqt$1@ger.gmane.org> <53F076D9.4020807@canterbury.ac.nz>
 <05931994-74E5-4F9E-9D84-3B272A2C4368@langa.pl>
Message-ID: <E55F0C16-3DFB-4F80-93D7-6E4CFD78A0C4@langa.pl>

On Aug 17, 2014, at 2:44 AM, ?ukasz Langa <lukasz at langa.pl> wrote:

> On Aug 17, 2014, at 2:33 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> 
>> Stefan Behnel wrote:
>>> However, it does not support protocols, so it still needs
>>> something that allows us to say Iterable(int) in some way.
>> 
>> Just had a thought -- does mypy provide a way to express
>> a type that supports more than one protocol? E.g. can you
>> say that something must be both Iterable and Hashable?
> 
> That would be union types. Current syntax:
> Union[Iterable[int], Hashable]
> 
> Proposed more concise syntax:
> Iterable[int] | Hashable

Ah, scratch that. What Greg asked about would be

Iterable[int] & Hashable

;-)

Don't know if this is supported but makes sense.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/9fa7e673/attachment.html>

From steve at pearwood.info  Sun Aug 17 11:51:52 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 17 Aug 2014 19:51:52 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <05931994-74E5-4F9E-9D84-3B272A2C4368@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <lspljq$bqt$1@ger.gmane.org> <53F076D9.4020807@canterbury.ac.nz>
 <05931994-74E5-4F9E-9D84-3B272A2C4368@langa.pl>
Message-ID: <20140817095152.GR4525@ando>

On Sun, Aug 17, 2014 at 02:44:32AM -0700, ?ukasz Langa wrote:
> On Aug 17, 2014, at 2:33 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> 
> > Stefan Behnel wrote:
> >> However, it does not support protocols, so it still needs
> >> something that allows us to say Iterable(int) in some way.
> > 
> > Just had a thought -- does mypy provide a way to express
> > a type that supports more than one protocol? E.g. can you
> > say that something must be both Iterable and Hashable?
> 
> That would be union types. Current syntax:
> Union[Iterable[int], Hashable]

I don't think so. The mypy tutorial says:

    Use the Union[...] type constructor to construct a union type. For 
    example, the type Union[int, str] is compatible with both integers 
    and strings. You can use an isinstance check to narrow down the type 
    to a specific type ...

which implies that x:Union[a, b] means:

    isinstance(x, (a, b))

rather than:

    isinstance(x, a) and isinstance(x, b)

which I think is what Greg is asking for.




-- 
Steven

From nicholas.cole at gmail.com  Sun Aug 17 11:52:52 2014
From: nicholas.cole at gmail.com (Nicholas Cole)
Date: Sun, 17 Aug 2014 10:52:52 +0100
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CADiSq7dHi1UdyZd3yNz8ifwvKMC-q8K8JkCUX4_objyYLdbk0g@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>
 <20140817083453.GO4525@ando>
 <CADiSq7dHi1UdyZd3yNz8ifwvKMC-q8K8JkCUX4_objyYLdbk0g@mail.gmail.com>
Message-ID: <CAAu18hfyj5ZqvnMi1wB7dN6WwMYcyrLZ+bmsO=U9GwU2yMTdEA@mail.gmail.com>

On Sun, Aug 17, 2014 at 9:50 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 17 August 2014 18:34, Steven D'Aprano <steve at pearwood.info> wrote:
>>
>> Declaring types in the function parameter list is very common, in many
>> languages. If it were *that* much harder to read, languages wouldn't
>> keep using it. (Not many languages follow Forth or APL syntax.) Perhaps
>> because I learned to program in Pascal, I find the annotation syntax
>> very easy to read, but, yes, anything which increases the density of
>> information per line risks hurting readability a little.
>
> I once had the "pleasure" of inheriting some code written in K&R style
> C, where the parameter type declarations were separate from the
> signature line:
>
>     void foo(a, b, c)
>         double a;
>         char b;
>     {
>           ...
>     }
>
> ANSI C, with inline typing,  is far more readable :)

I think you've put your finger on it.  It comes down to a disagreement
over density of information.  You'd like everything in "one pass" as
it were. The way my brain is wired, I read your example here as:

"This function takes three (non-named) parameters, the a is a double,
the b is a char."  I find that faster to process than the inline
alternative, especially when the alternative is optional.

With meaningful parameter names it would be even easier.

If I re-write your example:

>     void foo(double a, void b, c)

My brain takes an extra fraction of a second to count the number of
arguments.  Syntax highlighting would help, of course.

Some of this can be very subtle.  For example, it's important for
readability that in Python positional parameters are all specified,
then keyword ones, so the brain doesn't have to keep switching
backwards and forwards.

In C, of course, everything has to be typed, and so I can see it makes
sense to put it all inline.  But what if you are mixing the two, and
some are typed and some not?  I think it is all going to get very
dense and hard to read.

The same tension occurs in natural languages.  I tend to write quite
dense English prose, myself. My editors always want me to write less
densely, and over time I've come to see that they are right!  Dense
prose is fine for the specialist, but it doesn't help the student or
the casual reader.

N.

From willvarfar at gmail.com  Sun Aug 17 12:13:24 2014
From: willvarfar at gmail.com (willvarfar at gmail.com)
Date: Sun, 17 Aug 2014 12:13:24 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAKkef2yzJZ5m6B1wLOW2bxXsPCPCF2sZa9ctsrHYckOYHLf4yw@mail.gmail.com>
References: <CAKkef2yzJZ5m6B1wLOW2bxXsPCPCF2sZa9ctsrHYckOYHLf4yw@mail.gmail.com>
Message-ID: <CAKkef2wECOEO3f=zdaZE-R9Y1aZ5N9+xussDWh8yCOjbJLZyYw@mail.gmail.com>

When discussed on reddit programming ("proggit" as its called), there were
plenty of people saying they really didn't like the mypy syntax:

http://www.reddit.com/r/programming/comments/2disob/proposal_for_python_type_annotations_from_guido/

(Declaration: I am the author of obiwan https://pypi.python.org/pypi/obiwan
and I got a lot of upvotes on that proggit thread when I promoted the
obiwan style instead)
On 14 Aug 2014 13:16, "willvarfar at gmail.com" <willvarfar at gmail.com> wrote:

> I fully support formalizing Python 3's annotations for type checking.
>
> I wrote - and use daily - my own type checker called obiwan
> https://pypi.python.org/pypi/obiwan
>
> Its a runtime type checker, and if enabled will check and enforce
> types on every call.
>
> I support the wider adoption and standardization of static type
> checkers, but runtime checkers are still wanted for very dynamic code
> and for checking external data e.g. I use obiwan for validating JSON.
>
> One small detail is that I feel the obiwan annotations are more
> pythonic than the mypy examples given.
>
> E.g. instead of:
>
>   from typing import List, Dict
>
>   def word_count(input: List[str]) -> Dict[str, int]:
>       ...
>
> It would be:
>
>   def word_count(input: [str]) -> {str, int}:
>       ...
>
> Obiwan does not check types within functions; I was unwilling to try
> and overload comments!  You can invoke obiwan to check things
> explicitly, but these are more as assertions.
>
> Anyway, when I look at the mypy in-function annotations (where
> comments are overloaded) I am cautious.  It would be far nicer if we
> had annotations as part of the language instead, e.g. instead of:
>
>   result = {}  #type: Dict[str, int]
>
> It would be:
>
>   result = {} -> {str, int}
>
> where we use the -> arrow again.  I can see pros and cons for any
> implementation (as we'd want the annotation to be both to declare a
> type and to check a type, and want the annotation to be attached to
> the variable forever etc) so this would need a full PEP treatment and
> possibly be configurable as asserts are.
>
> But proper annotation support in the language rather than overloading
> comments would definitely be my preference.
>
> /Will
>
> /Will
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/163463e9/attachment.html>

From steve at pearwood.info  Sun Aug 17 12:20:59 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 17 Aug 2014 20:20:59 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
Message-ID: <20140817102059.GS4525@ando>

On Sat, Aug 16, 2014 at 10:03:48PM -0700, Guido van Rossum wrote:
> I'd like to summarize the main issues that have come up. As an experiment,
> I'm not changing the subject, but I am still not quoting anything in
> particular. Only two issues (or issue clusters) really seem contentious:
> 
> (1) Should the function annotation syntax (eventually) be reserved for type
> annotations in a standard syntax? 

Reserved, as in alternatives are prohibited? No.

But assumed to be types by default? I think so.


> Or can multiple different uses of
> annotations coexist? And if they can, how should a specific use be
> indicated? (Also, some questions about compile-time vs. run-time use.)

Determining whether annotations should be interpreted as type 
annotations or not needs to be possible both at compile-time and 
run-time. I suggest a special decorator, imported from the typing 
module:


from typing import skip_types  # or some better name?

@skip_types(marker)  # see below for the purpose of marker
@another_decorator  # the order of decorators doesn't matter
def function(x:"this is not a type")->[int]:
    ...


At compile-time, static typing tools should treat any function decorated 
by skip_types as if it were dynamic.

The skip_types decorator also writes marker to the __annotations__ dict, 
using a key which cannot be used as an identifier. Say, "+mark+". The 
purpose of the marker is so that other tools can determine whether the 
annotations are aimed at them.

def handle_annotations(func):
    if func.__annotations__.get("+mark+", None) is MyMarker:
        do_stuff(func.__annotations__)
    else:
        pass

Any time a function is not decorated by skip_types (or whatever name it 
has), or doesn't have that "+mark+" key in the __annotations__ dict, 
static type checking tools are entitled to assume the annotations are 
used for typing.


> (2) For type annotations, should we adopt (roughly) the mypy syntax or the
> alternative proposed by Dave Halter? This uses built-in container notations
> as a shorthand, e.g. {str: int} instead of Dict[str, int]. This also
> touches on the issue of abstract vs. concrete types (e.g. iterable vs.
> list).

I prefer the mypy syntax.

 
> Regarding (1), I continue to believe that we should eventually reserve
> annotations for types, to avoid confusing both humans and tools, but I
> think there's nothing we have to do in 3.5 -- 3.5 must preserve backward
> compatibility, and we're not proposing to give annotations any new
> semantics anyway -- the actual changes to CPython are limited to a new
> stdlib module (typing) and some documentation.
> 
> Perhaps a thornier issue is how mypy should handle decorators that
> manipulate the signature or annotations of the function they wrap. But I
> think the only reasonable answer here can be that mypy must understand what
> decorators do if it wants to have any chance at type-checking decorated
> functions. 

Hmmm. Anything the decorators do to the annotations will be at runtime, 
so is it reasonable to say that the static typing tool will only operate 
on the annotations available at compile time?

That is, given:

@mangle_annotations
def spam(x:int)->List[str]: ...

the type checker is expected to use the annotations seen at 
compile-time, no matter what the mangle_annotations decorator happens to 
do at run-time.

Otherwise, the type-checker needs to be a full Python interpreter, in 
order to see what mangle_annotations does. And that could be an
intractible problem:

def mangle_annotations(func):
    if random.random() < 0.5:
        func.__annotations__['x'] = List[str]
    else:
        func.__annotations__['x'] = float


I don't see how any type-checker is supposed to take that into account.

(1) Compile-time type checkers should ignore the decorator and just use 
the annotations available at compile-time;

(2) Run-time analysis tools should use the annotations available at 
run-time;

(3) If they happen to be different, oh well, consenting adults.



-- 
Steven

From davidhalter88 at gmail.com  Sun Aug 17 12:26:35 2014
From: davidhalter88 at gmail.com (Dave Halter)
Date: Sun, 17 Aug 2014 12:26:35 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <F392A71E-CD09-489A-8D6F-AE51AAB7B32F@yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <CAA=HWYgZdwgDbg0RM3AHp_E+wVArxdjeWQf8oksBvcxudPQcdQ@mail.gmail.com>
 <F392A71E-CD09-489A-8D6F-AE51AAB7B32F@yahoo.com>
Message-ID: <CAA=HWYi9fEa0Dh9mf4DzH=2CQeJSUvVUXO56q-XARLRB934-Ww@mail.gmail.com>

2014-08-16 16:30 GMT+02:00 Andrew Barnert <abarnert at yahoo.com>:

> On Aug 16, 2014, at 6:00, Dave Halter <davidhalter88 at gmail.com> wrote:
>
> *Appendix (My Proposal)*
>
> My proposal (as discussed and evolved with a few good people at
> EuroPython) for containers would look something like this:
>
>     def foo(index: int, x: [float], y: {int: str}) -> (float, str):
>         return x[index], y[index]
>
> The "default" containers (set, list, dict and tuple) would just serve as a
> way of specifying containers. This makes a lot of things less complicated:
>
> - It's easier to understand (everybody knows builtin types). It also feels
> natural to me. The example above covers also almost all variations. A tuple
> could be expanded by using the ellipsis: `(int, object, ..., float)`.
> - No imports needed. People are more likely to use it, if they don't have
> to import typing all the time. This is important for static analysis,
> people are only likely to use it if it's easier than writing docstrings
> with type information.
> - Argument clinic could use the same. The standard library quite often
> doesn't accept abstract data types, btw.
> - It's what people sometimes use in docstrings: ``@rtype: (float, str)``
> or ``:rtype: (float, str)``.
>
> I know this doesn't solve the duck typing issue, but if you look at
> real-life Python code bases, there are very few instances of actually
> implementing a ``Mapping``, etc.
>
>
> For "Mapping", maybe (although anyone who uses some form of tree-based
> mapping, either to avoid hash-collision attacks or because he needs
> sorting, may disagree).
>
> But for "etc.", people implement them all the time. Especially "Iterable"
> and "Callable". And, even some of the ones that people don't implement
> often, they use often, implicitly or otherwise, like TextIOBase.
>

> Anything that encourages people to restrict their code to only working on
> lists instead of iterables would be a huge step backward to Python 2.2. And
> your argument for it ("everybody knows builtin types") implies that's
> exactly what you're expecting with this proposal.
>
> However, there's an easy way around this: just let [spam] in your syntax
> mean  what Iterable[spam] means in MyPy's. If someone really needs to
> declare that they will only accept a list or whatever, that's the uncommon
> case, and can be written more verbosely. And I don't see any reason why
> this can't be added on top of genericizing the ABCs a la MyPy.
>
> Meanwhile, your tuple doesn't fit the same pattern as the others, because
> it's explicitly fixed-size and heterogeneous. And I think this is a good
> thing. And I think it's pretty close to what MyPy does with an expression
> list of types already; if not, it seems like what MyPy _should_ do. If I
> loop over a zip of a [str] and an [int], the loop variable is a (str, int),
> not a Tuple[str or int].
>
> So, making it easier to specify generic builtins is a bad idea, but using
> builtins to make it easier to specify common types is a great idea.
>

Very interesting ideas. I'm not sure if this is the right way to go, but
it's definitely an option. It could confuse people, if they cannot use
`isinstance(x, list)` anymore. But then again it would just be a really
neat way of expressing abstract types.

One other confusion might come from the fact that `-> list` would not be an
abstract data type, but `[str]` will. It's ok, but also not very obvious.
I've also thought about adding something like an `->
abstract([abstract(str)])` which would return the abstract data types for
builtins.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/47996462/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Sun Aug 17 12:31:28 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 17 Aug 2014 22:31:28 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>
Message-ID: <53F08480.6060305@canterbury.ac.nz>

Nicholas Cole wrote:
> On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> 
>>    def __init__(self,
>>            description:str, sec_code:str,
>>            vendor_name:str, vendor_inv_num:str,
>>            vendor_rtng:str, vendor_acct:str,
>>            transaction_code:str, vendor_acct_type:str,
>>            amount:int, payment_date:Any)->None:
>
>I had to stare at that block of
> code for a long time to see how many and what type of arguments it
> called.

Pascal's function signature syntax had a nice feature
that everyone else seems to have forgotten about. If you
had multiple parameters of the same type, you only had
to write the type once:

     procedure Init(description, sec_code, vendor_name,
         vendor_inv_num, vendor_rtng, vendor_acct,
         transaction_code, vendor_acct_type, amount: str;
         payment_date: Any)

Disappointingly, Python's annotations make the same
blunder as C, and most other languages since, in
requiring each parameter to have its own individual
annotation.

-- 
Greg

From andrey.vlasovskikh at gmail.com  Sun Aug 17 12:35:57 2014
From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh)
Date: Sun, 17 Aug 2014 14:35:57 +0400
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl>
Message-ID: <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com>

2014-08-17, 13:41, ?ukasz Langa <lukasz at langa.pl> wrote:

> On Aug 16, 2014, at 10:03 PM, Guido van Rossum <guido at python.org> wrote:
> 
>> All in all I prefer the mypy syntax, despite being somewhat more verbose and requiring an import, with one caveat: I agree that it would be nicer if the mypy abstract collection types were the same objects as the ABCs exported by collections.abc.
> 
> Good :) If the functionality will be implemented in the ABCs, what is the purpose of the typing module?
> 
> My suggestion: if the functionality will be implemented in the ABCs, there's no need to introduce the "typing" module. We can back-port the new ABCs, for sure, but for Python 3.5 `collections` is enough (already has aliases to collections.abc0.

-1 for collections.abc classes, +1 for mypy's typing classes.

There is a problem in static analysis of current types that are instances of abc.ABCMeta or types that just define their own __instancecheck__ / __subclasscheck__. Static analyzers cannot infer in general case what attributes of an instance / subclass do these methods check, because their body can be arbitrarily complex.

Mypy's typing.Protocol subclasses are much easier to analyze statically, since they are required to explicitly define abstract methods as function defintions inside the class body.

Current collectoins.abc classes do define their methods explicitly too, so it seems that at least these classes are fine. But their inheritors don't have to do it, they may just override __subclasshook__. And promoting abc.ABCMeta-based ABCs would mean that not all ABCs can be used as static type annotations.

-- 
Andrey Vlasovskikh

Web: http://pirx.ru/


From andrey.vlasovskikh at gmail.com  Sun Aug 17 12:36:07 2014
From: andrey.vlasovskikh at gmail.com (Andrey Vlasovskikh)
Date: Sun, 17 Aug 2014 14:36:07 +0400
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
Message-ID: <84F83885-BD67-45E0-9521-7742546E8279@gmail.com>

2014-08-17, 9:03, Guido van Rossum <guido at python.org> wrote:

> My first concern is that these expressions are only unambiguous in the context of function annotations. I want to promote the use of type aliases, and I think in general a type alias should behave similarly to an ABC. In particular, I think that any object used to represent a type in an annotation should itself be a type object (though you may not be able to instantiate it), and e.g. [int] doesn't satisfy that requirement. Without this, it would be difficult to implement isinstance() and issubclass() for type aliases -- and while we could special-case lists, sets and dicts, using a tuple *already* has a meaning!

Having type annotations as type objects sounds good. The fact that we can use isinstance() and issubclass() for all type annotations would provide some level of compatibility between static type checking and potential dynamic type checking: if "x: <type-expr>" then "isinstance(x, <type-expr>)".

Note, that not all type annotations of mypy are currently type objects. Probably this should be fixed. 

-- 
Andrey Vlasovskikh

Web: http://pirx.ru/


From abarnert at yahoo.com  Sun Aug 17 12:41:17 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 17 Aug 2014 03:41:17 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <20140817094152.GP4525@ando>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <2CF4047F-8A85-4CAB-8AEC-1993274F6302@yahoo.com> <20140817072602.GN4525@ando>
 <828F3377-EEF0-4AC6-B721-3DB65FBFA062@yahoo.com> <20140817094152.GP4525@ando>
Message-ID: <7BC3BC4C-F209-4509-90E3-559BE6D15FE1@yahoo.com>

On Aug 17, 2014, at 2:41, Steven D'Aprano <steve at pearwood.info> wrote:

> On Sun, Aug 17, 2014 at 01:52:21AM -0700, Andrew Barnert wrote:
>> On Aug 17, 2014, at 0:26, Steven D'Aprano <steve at pearwood.info> wrote:
>> 
>>> On Sat, Aug 16, 2014 at 11:02:01PM -0700, Andrew Barnert wrote:
>>> 
>>>> I won't belabor the point, but again: I don't think we need a generic 
>>>> list type object, and without it, this entire problem--your only 
>>>> remaining problem that isn't a mere stylistic choice--vanishes.
>>> 
>>> I don't understand. If there's no list typing object, how do you declare 
>>> a variable must be a list and nothing but a list? Or that it returns a 
>>> list?
>> 
>> You think about it and make sure you really do need a list and nothing 
>> but a list. Most of the time (as in all three of the examples given in 
>> this thread) this is a mistake. If it's not, then you use List. (Or, 
>> if the stdlib doesn't provide that, you have to write one line of 
>> code: List = TypeAlias(list), and then you can use it.)
> 
> Ah, that is the point I missed. You think that the stdlib shouldn't 
> provide a standard typing object for lists, but that people should just 
> create their own if they need it.
> 
> Okay, but I don't understand why you're singling out lists. If you want 
> to propose providing only abstract classes (Sequence, Mapping, etc.) 
> and not concrete classes (list, dict, etc.) by default, that makes 
> some sense to me. But I don't understand including typing.Dict as 
> an alias for dict (say) but not List.

I'm not singling out lists. I referred to tuple in exactly the same way in the same message. I didn't mention dict, frozenset, etc. because I'd already given the full argument a few hundred messages ago and I didn't think anyone wanted to read it again. But the "if" sentence in your paragraph is a good summary of the whole thing.

>> If having list[T] is going to be more of an attractive nuisance than a 
>> useful feature, and it will be especially attractive and nuisanceful 
>> for exactly the same novices who are unlikely to know how to TypeAlias 
>> it themselves, why is it a problem to leave it out?
> 
> There are at least three scenarios:
> 
> (1) Built-ins can be used directly in static type annotations:
> 
>    x:list[dict]
> 
>    This has the advantage of not needing special names, but the 
>    disadvantage of encouraging lazy programmers to specify concrete
>    types when they should be using abstract Sequence[Mapping].
> 
> (2) Built-ins *cannot* be used, you have to import them from typing:
> 
>    from typing import List, Dict
>    x:List[Dict]
> 
>    The advantage is that since you have to do an import anyway,
>    it is not much more effort to Do The Right Thing:
> 
>    from typing import Sequence, Mapping
>    x:Sequence[Mapping]
> 
> (3) And the final scenario, the one which confuses me, but seems 
>    to be what you are suggesting: you can use the built-ins, 
>    *but not list*, and there is no List to import either:
> 
>    from typing import Sequence
>    x:Sequence[dict]
> 
>    I don't understand the advantage of this.

As explained above, I'm suggesting (2), not (3).

> [...]
>>>> So, why can't def foo(spam: (int, str)) mean that spam is an 
>>>> iterable of an int and a str, in exactly the same way that the 
>>>> assignment statement means that a and b are assigned the result of 
>>>> unpacking the iterable returned by zip when called with c and d?
>>> 
>>> But a, b = zip(c, d) requires that there be exactly two elements, not 
>>> some unspecified number.
>> 
>> And spam:(int, str) requires that there be exactly two elements (and 
>> that the first be an int and the second a str), not some unspecified 
>> number. How is that any different?
> 
> Ah, that's what I didn't understand. I thought you meant an iterable of 
> either ints or strs, without requiring a fixed number of them.

Right. The "paradigm case" for tuples is as fixed-length, heterogeneous collections, where each index has a specific semantic meaning (and therefore type).

The problem is that, at least in Python, tuples are _also_ used as general-purpose immutable sequences--and, in a few cases, that's specifically enshrined in syntax (e.g., the exception types in an except statement) or builtins (e.g., the arguments to str.__mod__), so we can't just ignore that.

My suggestion is that (int, str) means the first (what you'd call int * str if you wanted your language to look more like type theory than something a normal human would write), so Tuple[int] works exactly like every other generic type: it's a homogenous collection of ints.

>>> To me, spam:(int, str) has a natural interpretation that spam can be 
>>> either an int or a str, not an Iterable or Sequence or even a tuple.
>> 
>> OK, I see the parallel there with exception statements now that you 
>> mention it.
>> 
>> But almost anywhere else in Python, a comma-separated list is a 
>> sequence of values, targets, parameters, etc., not a disjunction. The 
>> obvious way to spell what you want here is "int | str"
> 
> Which mypy spells as Union[ ].
> 
> http://mypy-lang.org/tutorial.html

Yes, but multiple people in this thread have suggested spelling it as int | str, nobody's objected, and both Jukka and Guido have given at least tentative assent.

(I realize there's a whole lot of messages to read through and remember. And I could easily have missed an argument against this syntax just as you missed the suggestions for it.)

From ben+python at benfinney.id.au  Sun Aug 17 12:53:18 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sun, 17 Aug 2014 20:53:18 +1000
Subject: [Python-ideas] Variable-length, homogeneous tuple: why?
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au> <lspqfv$vop$1@ger.gmane.org>
Message-ID: <85zjf34cbl.fsf@benfinney.id.au>

Terry Reedy <tjreedy at udel.edu> writes:

> On 8/17/2014 4:23 AM, Ben Finney wrote:
> > Guido van Rossum <guido at python.org> writes:
> >
> >> A mostly unrelated issue: there are two different uses of tuples, and
> >> we need a notation for both. One is a tuple of fixed length with
> >> heterogeneous, specific types for the elements; for example Tuple[int,
> >> float].
> >
> > That's the meaning of a tuple data structure, to me.
> >
> >> But I think we also need a way to indicate that a function expects
> >> (or returns) a variable-length tuple with a homogeneous element
> >> type.
>
> There are also fixed-length homogeneous structures, like points.

I assume you mean where the sequence is something like ?(x, y, z)? where
each position has a meaning specific to that position.

That's not homogeneous as I understood this usage, because the psitions
are not homogeneous in meaning. The number 7.03 has very different
meaning in the first, second, or third positions.

A homogeneous sequence would imply there are deliberately *no* specific
meanings to each position. The value 7.03 would have the same semantic
value at any position in the sequence.

So a fixed-length sequence where each position implies a special meaning
is a heterogeneous sequence, and I agree that's an excellent use for a
tuple.

> > Why? What real-world uses are there, where a list won't do the job
> > adequately?
>
> Variable-length homogenous tuples are part of python syntax in
> multiple places.
>
> Tuples can be hashed and put in sets an used as dict keys, lists
> cannot. Tuple contants are calculated just once when the code is
> compiled (and typically saved as .pyc).

Okay, so these are not for the semantic purpose a tuple is for. I think
a putative ?frozenlist? type is best for that, to keep the heterogeneous
implication of a tuple separate from the homogeneous implication of a
list.

-- 
 \           ?I prayed for twenty years but received no answer until I |
  `\          prayed with my legs.? ?Frederick Douglass, escaped slave |
_o__)                                                                  |
Ben Finney


From abarnert at yahoo.com  Sun Aug 17 12:53:47 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 17 Aug 2014 03:53:47 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53F076D9.4020807@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <lspljq$bqt$1@ger.gmane.org> <53F076D9.4020807@canterbury.ac.nz>
Message-ID: <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com>

On Aug 17, 2014, at 2:33, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> Stefan Behnel wrote:
>> However, it does not support protocols, so it still needs
>> something that allows us to say Iterable(int) in some way.
> 
> Just had a thought -- does mypy provide a way to express
> a type that supports more than one protocol? E.g. can you
> say that something must be both Iterable and Hashable?

The obvious way that already works (with MyPy, and also with ABCs for isinstance checking):

    class HashableIterable(Iterable, Hashable): pass

    def spam(a: HashableIterable):
        pass

But, there's no reason typing.py couldn't add a wrapper that does this automatically:

    class Multiple:
        @staticmethod
        def __getitem__(*types);
            return type(types[0])(
                '_'.join(t.__name__ for t in types), types, {})

    def spam(a: Multiple[Iterable, Hashable]):
        pass

And, on analogy with the proposal for | as a shortcut for Union, the base class could add:

    def __and__(self, other):
        return Multiple[self, other]

    def spam(a: Iterable & Hashable):
        pass


From skip at pobox.com  Sun Aug 17 13:31:55 2014
From: skip at pobox.com (Skip Montanaro)
Date: Sun, 17 Aug 2014 06:31:55 -0500
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <854mxb5xt3.fsf_-_@benfinney.id.au>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
Message-ID: <CANc-5UzG5e_m1Ghn3TdWLU7XSqELGb1RreS01h5We1QtAYg1Kw@mail.gmail.com>

On Sun, Aug 17, 2014 at 3:23 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
> I have encountered many uses of ?homogeneous, variable-length sequence?
> and every time a Python tuple is used for that, I perceive a Python list
> would be better precisely *because* it better indicates that semantic
> meaning.

Agreed. While "variable-length" doesn't imply mutability, they often
seem to go hand-in-hand.

Skip

From jeanpierreda at gmail.com  Sun Aug 17 13:44:31 2014
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sun, 17 Aug 2014 04:44:31 -0700
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <854mxb5xt3.fsf_-_@benfinney.id.au>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
Message-ID: <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>

On Sun, Aug 17, 2014 at 1:23 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
> I have encountered many uses of ?homogeneous, variable-length sequence?
> and every time a Python tuple is used for that, I perceive a Python list
> would be better precisely *because* it better indicates that semantic
> meaning.
>
> I'd like to know how you think that's not true, and what real-world code
> makes you think so.

isinstance is real world code that for the second parameter accepts
types and (recursively) tuples of any length of things it accepts.

It doesn't accept lists, because then it would need to check for cycles.

-- Devin

From jeanpierreda at gmail.com  Sun Aug 17 14:05:24 2014
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sun, 17 Aug 2014 05:05:24 -0700
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
Message-ID: <CABicbJJmbo34nOs5zZ5n5UhRT3bP5og8ywCXsHR9inkMgWNosg@mail.gmail.com>

On Sun, Aug 17, 2014 at 4:44 AM, Devin Jeanpierre
<jeanpierreda at gmail.com> wrote:
> On Sun, Aug 17, 2014 at 1:23 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
>> I have encountered many uses of ?homogeneous, variable-length sequence?
>> and every time a Python tuple is used for that, I perceive a Python list
>> would be better precisely *because* it better indicates that semantic
>> meaning.
>>
>> I'd like to know how you think that's not true, and what real-world code
>> makes you think so.
>
> isinstance is real world code that for the second parameter accepts
> types and (recursively) tuples of any length of things it accepts.
>
> It doesn't accept lists, because then it would need to check for cycles.

I was going to leave it at that -- the fact that a builtin works this
way is an argument in favor of any standardized typing syntax
supporting it -- but maybe it deserves justification, too.

The key question is, given two values A and B that are valid second
parameters to isinstance, how do you combine them into one valid
second parameter, which forms the union of A and B? if A and B were
always tuples, you could do A + B (isinstance predates sets). But we
want to be able to do isinstance(x, SomeType), so A and B can't always
be tuples. So Python adopts the convention that you can do (A, B) and,
no matter what they are, isinstance(x, (A, B)) holds if and only if
(isinstance(x, A) or isinstance(x, B)).

As it happens we could use frozensets instead, but, oops. Given that
frozensets are out of the question (in this case, because of dating,
in other cases, because of unhashability or whatever) using sequences
seems reasonable, and specifically using tuples so that we can avoid
cycle checks (a messy algorithm) also seems entirely reasonable.

I've also seen people use tuples to emphasize that the sequence is not
mutated, but there's no reason to require that from a type signature
in that case.

-- Devin

From stefan_ml at behnel.de  Sun Aug 17 14:10:27 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 17 Aug 2014 14:10:27 +0200
Subject: [Python-ideas] RFC: Multiple Dispatch
In-Reply-To: <lsnmls$l8f$1@ger.gmane.org>
References: <CAJ8oX-FSs7YhPUHb9QEtBwFHaD_6jBU_p+5fwJxpx_=sGBUrgQ@mail.gmail.com>
 <CAP7+vJKWvuCxmJM_0p4Y+2wM+oLW+Ao0CBF-wUw-ewqe7h49Cw@mail.gmail.com>
 <CAP7h-xbWjCe6iJQkHtNmMBAj9hcwPJ7ZTyXFtLuaxiYsRxNmAA@mail.gmail.com>
 <CAJ8oX-EEHamhjUY-Ub8APYsOf1koXC=AN8mgobin0OA-nSRP-A@mail.gmail.com>
 <CAP7+vJLGbCWpoxZffMEVH=HQdBzqD2Cr0YOp1BGUqVX3B77rbw@mail.gmail.com>
 <lsnmls$l8f$1@ger.gmane.org>
Message-ID: <53F09BB3.5090508@behnel.de>

Antoine Pitrou schrieb am 16.08.2014 um 15:34:
> Le 15/08/2014 14:01, Guido van Rossum a ?crit :
>> Please do write about non-toy examples!
> 
> Are you looking for examples using the multipledispatch library, or
> multiple dispatch in general?
> 
> As for multiple dispatch in general, Numba uses something which is morally
> one in order to select the right specialization of, say, an operator (for
> example to choose amongst '+ between int and int', '+ between
> numpy.datetime64 and numpy.timedelta64', '+ between numpy.timedelta64 and
> numpy.timedelta64', etc.).

Similar for Cython's "fused types", i.e. compile time generics. It
generates specialised function implementations for a cross product of the
type sets used in the signature and then calls the right specialisation
based on the input types at runtime, when called from Python code.
Understands input buffer types and C types in signatures, though, so it's a
bit more complex than the "average Python generic function dispatch" (same
applies to Numba, I guess).

Stefan



From steve at pearwood.info  Sun Aug 17 14:31:27 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 17 Aug 2014 22:31:27 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAKkef2wECOEO3f=zdaZE-R9Y1aZ5N9+xussDWh8yCOjbJLZyYw@mail.gmail.com>
References: <CAKkef2yzJZ5m6B1wLOW2bxXsPCPCF2sZa9ctsrHYckOYHLf4yw@mail.gmail.com>
 <CAKkef2wECOEO3f=zdaZE-R9Y1aZ5N9+xussDWh8yCOjbJLZyYw@mail.gmail.com>
Message-ID: <20140817123127.GT4525@ando>

On Sun, Aug 17, 2014 at 12:13:24PM +0200, willvarfar at gmail.com wrote:
> When discussed on reddit programming ("proggit" as its called), there were
> plenty of people saying they really didn't like the mypy syntax:
> 
> http://www.reddit.com/r/programming/comments/2disob/proposal_for_python_type_annotations_from_guido/
> 
> (Declaration: I am the author of obiwan https://pypi.python.org/pypi/obiwan
> and I got a lot of upvotes on that proggit thread when I promoted the
> obiwan style instead)

My thoughts on the obiwan style?

Quicker, easier, more seductive. Easily they flow. But once you start 
down the dark path, forever will it dominate your destiny.

*wink*

Your example:

[mypy]
def word_count(input: List[str]) -> Dict[str, int]:

[obiwan]
def word_count(input: [str]) -> {str, int}:

Now consider that annotations are just expressions. Inside or outside of 
a function parameter definition, Dict[str, int] is exactly the same 
thing: it's an abstract type. I should be able to use it at runtime:

isinstance(x, word_count.__annotations__['return'])

But obiwan's style (like the old Jedi himself) twists the truth around. 
The return annotation is only a type from a certain perspective, but in 
reality it is an instance of a concrete class, not an abstract class.

It's not even a dict. {str, int} is a set, but obiwan uses it to 
represent a dict. (Although maybe that's a typo, and you meant to write 
{str: int}.)



-- 
Steven

From ncoghlan at gmail.com  Sun Aug 17 14:43:45 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Aug 2014 22:43:45 +1000
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
Message-ID: <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>

On 17 August 2014 21:44, Devin Jeanpierre <jeanpierreda at gmail.com> wrote:
> On Sun, Aug 17, 2014 at 1:23 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
>> I have encountered many uses of ?homogeneous, variable-length sequence?
>> and every time a Python tuple is used for that, I perceive a Python list
>> would be better precisely *because* it better indicates that semantic
>> meaning.
>>
>> I'd like to know how you think that's not true, and what real-world code
>> makes you think so.
>
> isinstance is real world code that for the second parameter accepts
> types and (recursively) tuples of any length of things it accepts.

There are a few other cases where tuples are special cased as arguments:

- str.__mod__
- str.startswith (ditto for binary sequences)
- str.endswith (ditto for binary sequences)

>>> "aa".endswith(['a', 'b', 'c'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: endswith first arg must be str or a tuple of str, not list

Searching the C files for "tuple of" turned up a couple more:

* N-dimensional indexing also relies specifically on
tuples-of-integers, rather than arbitrary iterators.

* dynamic type creation expects to receive the bases as a tuple

* the decimal module uses tuples of digits for internal data representations

And that inspired recollection of several other cases where mutability
would be wrong, because the tuple represents cached information rather
than dynamic state:

* various "*args" related APIs use "tuple of object" or "tuple of
thing" (e.g. attributes of partial objects, internal storage in
contextlib.ExitStack when used with arbitrary callbacks)

* other introspection related APIs use tuples to report information
about inspected objects

* namedtuple _fields attributes are a tuple of strings

* BaseException.args publishes the full args tuple passed to the constructor

str.startswith, str.endswith, isinstance and issubclass use the
"implied or" interpretation, everything else does not. In most cases,
the immutability conveys relevant semantic information (usually
indicating that it's a read-only API).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rymg19 at gmail.com  Sun Aug 17 16:09:07 2014
From: rymg19 at gmail.com (Ryan)
Date: Sun, 17 Aug 2014 09:09:07 -0500
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53F08480.6060305@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>
 <53F08480.6060305@canterbury.ac.nz>
Message-ID: <a9312da9-6974-4a0e-a851-2f9b267cc7d5@email.android.com>

Nimrod has that feature, too, which makes type lists easier on the eyes.

Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>Nicholas Cole wrote:
>> On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano
><steve at pearwood.info> wrote:
>> 
>>>    def __init__(self,
>>>            description:str, sec_code:str,
>>>            vendor_name:str, vendor_inv_num:str,
>>>            vendor_rtng:str, vendor_acct:str,
>>>            transaction_code:str, vendor_acct_type:str,
>>>            amount:int, payment_date:Any)->None:
>>
>>I had to stare at that block of
>> code for a long time to see how many and what type of arguments it
>> called.
>
>Pascal's function signature syntax had a nice feature
>that everyone else seems to have forgotten about. If you
>had multiple parameters of the same type, you only had
>to write the type once:
>
>     procedure Init(description, sec_code, vendor_name,
>         vendor_inv_num, vendor_rtng, vendor_acct,
>         transaction_code, vendor_acct_type, amount: str;
>         payment_date: Any)
>
>Disappointingly, Python's annotations make the same
>blunder as C, and most other languages since, in
>requiring each parameter to have its own individual
>annotation.
>
>-- 
>Greg
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>https://mail.python.org/mailman/listinfo/python-ideas
>Code of Conduct: http://python.org/psf/codeofconduct/

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/ffb439f7/attachment.html>

From rosuav at gmail.com  Sun Aug 17 16:30:49 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 18 Aug 2014 00:30:49 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <20140817102059.GS4525@ando>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <20140817102059.GS4525@ando>
Message-ID: <CAPTjJmogZmhtVTcQEG00-zhhJ-F66qGC+7waYD0jdcBCYJjV6g@mail.gmail.com>

On Sun, Aug 17, 2014 at 8:20 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Hmmm. Anything the decorators do to the annotations will be at runtime,
> so is it reasonable to say that the static typing tool will only operate
> on the annotations available at compile time?
>
> That is, given:
>
> @mangle_annotations
> def spam(x:int)->List[str]: ...
>
> the type checker is expected to use the annotations seen at
> compile-time, no matter what the mangle_annotations decorator happens to
> do at run-time.
>
> Otherwise, the type-checker needs to be a full Python interpreter, in
> order to see what mangle_annotations does. And that could be an
> intractible problem:
>
> def mangle_annotations(func):
>     if random.random() < 0.5:
>         func.__annotations__['x'] = List[str]
>     else:
>         func.__annotations__['x'] = float
>
>
> I don't see how any type-checker is supposed to take that into account.

You give an example of a malicious mangling, but more significant is
the naive mangling - wrapping the decorated function in a
non-annotated outer function, without using functools.wraps() or
equivalent (I'm sure it'd be possible to propagate the annotations
through wraps(), so that would take care of a lot of cases).

IMO the right handling here is to completely ignore all unrecognized
decorators, on the assumption that most decorators should be returning
an "equivalently usable" function. I don't, for instance, see
real-world examples of decorators that add extra parameters to a
function, even though it would be plausible (maybe you have a whole
bunch of functions that all take an optional mode parameter, which
causes other arguments to be translated automatically by the
decorator?). If you're annotating the function, the type checker can
assume that that's intended to be correct.

ChrisA

From stefan_ml at behnel.de  Sun Aug 17 16:37:13 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 17 Aug 2014 16:37:13 +0200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53F08480.6060305@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>
 <53F08480.6060305@canterbury.ac.nz>
Message-ID: <lsqemt$obe$1@ger.gmane.org>

Greg Ewing schrieb am 17.08.2014 um 12:31:
> Nicholas Cole wrote:
>> On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano wrote:
>>
>>>    def __init__(self,
>>>            description:str, sec_code:str,
>>>            vendor_name:str, vendor_inv_num:str,
>>>            vendor_rtng:str, vendor_acct:str,
>>>            transaction_code:str, vendor_acct_type:str,
>>>            amount:int, payment_date:Any)->None:
>>
>> I had to stare at that block of
>> code for a long time to see how many and what type of arguments it
>> called.
> 
> Pascal's function signature syntax had a nice feature
> that everyone else seems to have forgotten about. If you
> had multiple parameters of the same type, you only had
> to write the type once:
> 
>     procedure Init(description, sec_code, vendor_name,
>         vendor_inv_num, vendor_rtng, vendor_acct,
>         transaction_code, vendor_acct_type, amount: str;
>         payment_date: Any)
> 
> Disappointingly, Python's annotations make the same
> blunder as C, and most other languages since, in
> requiring each parameter to have its own individual
> annotation.

The difference is that Pascal requires type declarations whereas they are
purely optional in Python. That makes the case that they are "missing" the
right thing to optimise for, i.e. they should be explicit where they are
and not take away space where they are not. Allowing argument sequences
under a single type annotation would require some kind of marker for either
that list or for the other arguments that are not typed. If you have a long
argument list of, say, three positional arguments and ten optional keyword
arguments, and you only want to annotate the first two positional arguments
with types and leave the rest free, that's a lot nicer to express with two
explicit type annotations than with grouped annotations and (potentially)
explicit non-annotations.

Stefan



From rosuav at gmail.com  Sun Aug 17 16:37:46 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 18 Aug 2014 00:37:46 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53F08480.6060305@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>
 <53F08480.6060305@canterbury.ac.nz>
Message-ID: <CAPTjJmr61=GA=H3pnPB6kHASQuVrfJm8NcMpWg+Ya1SJuUoSXA@mail.gmail.com>

On Sun, Aug 17, 2014 at 8:31 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Disappointingly, Python's annotations make the same
> blunder as C, and most other languages since, in
> requiring each parameter to have its own individual
> annotation.

Python doesn't really have any option here, because a non-annotated
parameter already has meaning. But even in C-family languages, I'm not
sure that it's all that advantageous; it makes editing less clear, so
it's really only beneficial when you have sets of related arguments
(eg "int r,g,b" to specify a color). With variable declarations,
there's a difference between "int r,g,b; double x,y,z;", where the
block of integers is terminated by a semicolon (and, conventionally, a
line ending); in argument lists, you don't get that, so it's not as
clear where one starts and one stops. (Imagine you misspell a type
name. It's no longer a keyword. How will your mistake be reported?)

ChrisA

From guido at python.org  Sun Aug 17 17:03:29 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 17 Aug 2014 08:03:29 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53F07465.1070103@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <53F07465.1070103@canterbury.ac.nz>
Message-ID: <CAP7+vJJxQ2kTQ6KJ6Wb687tQoeA0Yoo3znBZ=SXzrW+-8DEZTA@mail.gmail.com>

On Sun, Aug 17, 2014 at 2:22 AM, Greg Ewing <greg.ewing at canterbury.ac.nz>
wrote:

> Guido van Rossum wrote:
>
>  Perhaps a thornier issue is how mypy should handle decorators that
>> manipulate the signature or annotations of the function they wrap. But I
>> think the only reasonable answer here can be that mypy must understand what
>> decorators do if it wants to have any chance at type-checking decorated
>> functions.
>>
>
> Seems to me the only way to do that in general is to
> execute the decorators. That means importing everything
> the decorators depend on and probably running at least
> the top-level module code. Is executing arbitrary
> code at type-checking time really desirable?


Nah, you just have to type-check the decorators. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/e813718e/attachment.html>

From python at mrabarnett.plus.com  Sun Aug 17 17:27:36 2014
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 17 Aug 2014 16:27:36 +0100
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <lsqemt$obe$1@ger.gmane.org>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>
 <53F08480.6060305@canterbury.ac.nz> <lsqemt$obe$1@ger.gmane.org>
Message-ID: <53F0C9E8.1090803@mrabarnett.plus.com>

On 2014-08-17 15:37, Stefan Behnel wrote:
> Greg Ewing schrieb am 17.08.2014 um 12:31:
>> Nicholas Cole wrote:
>>> On Sun, Aug 17, 2014 at 3:08 AM, Steven D'Aprano wrote:
>>>
>>>>    def __init__(self,
>>>>            description:str, sec_code:str,
>>>>            vendor_name:str, vendor_inv_num:str,
>>>>            vendor_rtng:str, vendor_acct:str,
>>>>            transaction_code:str, vendor_acct_type:str,
>>>>            amount:int, payment_date:Any)->None:
>>>
>>> I had to stare at that block of
>>> code for a long time to see how many and what type of arguments it
>>> called.
>>
>> Pascal's function signature syntax had a nice feature
>> that everyone else seems to have forgotten about. If you
>> had multiple parameters of the same type, you only had
>> to write the type once:
>>
>>     procedure Init(description, sec_code, vendor_name,
>>         vendor_inv_num, vendor_rtng, vendor_acct,
>>         transaction_code, vendor_acct_type, amount: str;
>>         payment_date: Any)
>>
>> Disappointingly, Python's annotations make the same
>> blunder as C, and most other languages since, in
>> requiring each parameter to have its own individual
>> annotation.
>
> The difference is that Pascal requires type declarations whereas they are
> purely optional in Python. That makes the case that they are "missing" the
> right thing to optimise for, i.e. they should be explicit where they are
> and not take away space where they are not. Allowing argument sequences
> under a single type annotation would require some kind of marker for either
> that list or for the other arguments that are not typed. If you have a long
> argument list of, say, three positional arguments and ten optional keyword
> arguments, and you only want to annotate the first two positional arguments
> with types and leave the rest free, that's a lot nicer to express with two
> explicit type annotations than with grouped annotations and (potentially)
> explicit non-annotations.
>
I wonder whether you could include the colon but omit the type if it's 
the same
as that of the following parameter:

     def __init__(self,
             description:, sec_code:,
             vendor_name:, vendor_inv_num:,
             vendor_rtng:, vendor_acct:,
             transaction_code:, vendor_acct_type:str,
             amount:int, payment_date:Any)->None:


From brett at python.org  Sun Aug 17 17:37:31 2014
From: brett at python.org (Brett Cannon)
Date: Sun, 17 Aug 2014 15:37:31 +0000
Subject: [Python-ideas] Optional static typing -- the crossroads
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl>
 <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com>
Message-ID: <CAP1=2W7ciu_5zudeYGp0k26mjUtFtu8uekGiOMer5bL_vKuwvQ@mail.gmail.com>

On Sun Aug 17 2014 at 6:36:43 AM Andrey Vlasovskikh <
andrey.vlasovskikh at gmail.com> wrote:

> 2014-08-17, 13:41, ?ukasz Langa <lukasz at langa.pl> wrote:
>
> > On Aug 16, 2014, at 10:03 PM, Guido van Rossum <guido at python.org> wrote:
> >
> >> All in all I prefer the mypy syntax, despite being somewhat more
> verbose and requiring an import, with one caveat: I agree that it would be
> nicer if the mypy abstract collection types were the same objects as the
> ABCs exported by collections.abc.
> >
> > Good :) If the functionality will be implemented in the ABCs, what is
> the purpose of the typing module?
> >
> > My suggestion: if the functionality will be implemented in the ABCs,
> there's no need to introduce the "typing" module. We can back-port the new
> ABCs, for sure, but for Python 3.5 `collections` is enough (already has
> aliases to collections.abc0.
>
> -1 for collections.abc classes, +1 for mypy's typing classes.
>
> There is a problem in static analysis of current types that are instances
> of abc.ABCMeta or types that just define their own __instancecheck__ /
> __subclasscheck__. Static analyzers cannot infer in general case what
> attributes of an instance / subclass do these methods check, because their
> body can be arbitrarily complex.
>

That's only an issue if the type-checking code chooses to care about
__instancecheck__/__subclasscheck__. The tool could choose to simply ignore
those methods and treat them as a run-time only benefit for
isinstance/issubclass checks but not for type checking. This is especially
true if the check is being done on the AST instead of imported code. People
can simply be told that their linter tool will not pick up magical
__instancecheck__/__subclasscheck__ implementations.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/e7527541/attachment.html>

From lukasz at langa.pl  Sun Aug 17 17:55:11 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sun, 17 Aug 2014 08:55:11 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl>
 <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com>
Message-ID: <94148077-5CFD-4D85-B867-6563B78A4450@langa.pl>

On Aug 17, 2014, at 3:35 AM, Andrey Vlasovskikh <andrey.vlasovskikh at gmail.com> wrote:

> There is a problem in static analysis of current types that are instances of abc.ABCMeta or types that just define their own __instancecheck__ / __subclasscheck__. Static analyzers cannot infer in general case what attributes of an instance / subclass do these methods check, because their body can be arbitrarily complex.

That's right. Moreover, arbitrary classes can be register()'ed on an ABC to respond to subclass and instance checks during runtime. Meta-classes and __new__ can do surprising things with returned class objects, too. In all cases Mypy would generate a false-positive type error. That's fine, we can improve on that in multiple ways. [1]

People will put classes with ABCMeta in function annotations whether we design for it or not. Subclassing a MutableMapping is the easiest way to implement the whole protocol. As you say, __instancecheck__ and friends aren't limited to ABCs. People will put classes implementing those in function annotations, too. People will have classes with elaborate metaclasses as well.

Most importantly: often those classes will come from libraries and frameworks that those people didn't write themselves. We can't expect them to open every black box to see if it works with type hinting. Lastly, classes may and will evolve, sometimes to grow more static and sometimes to become more dynamic.

What I'm saying is that the typing module does not shield you from any of it. We need to define type hinting as a best-effort solution to set reasonable expectations. You need to understand the additional cognitive burden a new module like 'typing' this would impose on us and our users.


> Mypy's typing.Protocol subclasses are much easier to analyze statically, since they are required to explicitly define abstract methods as function defintions inside the class body.



[1] My suggestions:
- ABCMeta is special and we can reasonably improve on the runtime .register() invocations by scanning the source for those
- a redefined __instancecheck__ would generate a warning
- we can define a way for a Python program to say: this module is safe to be imported independently for executing __subclasscheck__ and __subclasshook__ checks on its classes
- failing all of the above, this is a perfectly fine case for optional runtime type checking; I don't think anybody has the expectation that we will statically analyse their highly dynamic codebases

All the above suggestions might be incremental. It's totally fine if we just begin with the last one.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/3ffe0b38/attachment-0001.html>

From lukasz at langa.pl  Sun Aug 17 17:56:45 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sun, 17 Aug 2014 08:56:45 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP1=2W7ciu_5zudeYGp0k26mjUtFtu8uekGiOMer5bL_vKuwvQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl>
 <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com>
 <CAP1=2W7ciu_5zudeYGp0k26mjUtFtu8uekGiOMer5bL_vKuwvQ@mail.gmail.com>
Message-ID: <AB3BD77A-A4A8-49EF-A37B-CF87B1CF94D6@langa.pl>

On Aug 17, 2014, at 8:37 AM, Brett Cannon <brett at python.org> wrote:

> On Sun Aug 17 2014 at 6:36:43 AM Andrey Vlasovskikh <andrey.vlasovskikh at gmail.com> wrote:
> There is a problem in static analysis of current types that are instances of abc.ABCMeta or types that just define their own __instancecheck__ / __subclasscheck__. Static analyzers cannot infer in general case what attributes of an instance / subclass do these methods check, because their body can be arbitrarily complex.
> 
> That's only an issue if the type-checking code chooses to care about __instancecheck__/__subclasscheck__. The tool could choose to simply ignore those methods and treat them as a run-time only benefit for isinstance/issubclass checks but not for type checking. This is especially true if the check is being done on the AST instead of imported code. People can simply be told that their linter tool will not pick up magical __instancecheck__/__subclasscheck__ implementations.

+1

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/e24d65f5/attachment.html>

From bruce at leban.us  Sun Aug 17 18:54:23 2014
From: bruce at leban.us (Bruce Leban)
Date: Sun, 17 Aug 2014 09:54:23 -0700
Subject: [Python-ideas] Variable-length, homogeneous tuple: why?
In-Reply-To: <85zjf34cbl.fsf@benfinney.id.au>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au> <lspqfv$vop$1@ger.gmane.org>
 <85zjf34cbl.fsf@benfinney.id.au>
Message-ID: <CAGu0Anvm8RuO8jEMaJ_AU_vYurJDNP5VRGM0-qEJ0vmSoNDkFg@mail.gmail.com>

On Sun, Aug 17, 2014 at 3:53 AM, Ben Finney <ben+python at benfinney.id.au>
wrote:

> A homogeneous sequence would imply there are deliberately *no* specific
> meanings to each position.
>

That's a set. A sequence implies order matters in some way.


> The value 7.03 would have the same semantic
> value at any position in the sequence.
>

Not at all. The 7 implies a units value, the 0 is a tenths value and the 3
is the hundredths. Those are different semantics. In Roman numerals before
they invented subtraction, XVI = VIX so order did not matter. Roman
numerals were basically sets of numbers to be added.

Here's another: (KentuckyDerby, PreaknessStakes, BelmontStakes) -- these
three objects are all instances of the class StakesRace and the order is
significant.

--- Bruce
Learn how hackers think: http://j.mp/gruyere-security
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/df9204c7/attachment.html>

From abarnert at yahoo.com  Sun Aug 17 21:31:19 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 17 Aug 2014 12:31:19 -0700
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing -- the
	crossroads)
In-Reply-To: <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
Message-ID: <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>

I think we're conflating multiple problems here.

Sometimes we use tuple to mean a homogeneous, arbitrary-length, immutable sequence, and other times we use it to mean a heterogeneous, fixed-length sequence.?Nick's list demonstrates that the former is (a) common enough to worry about, and (b) not always a mistake.

But even if such uses were always a mistake, Python is obviously not going to add a frozenlist, with new display syntax, and change over all existing misuses of tuple (which would clearly require a full deprecation cycle) before adding static typing. So, we'd still need a way to statically type both uses.

And even if the homogeneous uses had never existed in the first place, a heterogeneous tuple would still not be a parametric type in the same sense as all of the other collections, the io classes, etc. So, we'd still need to distinguish it syntactically from all of the other generics.

So, arguing about whether we need to handle heterogeneous tuples specially is pointless; the only question is how we do it.

I can think of four possibilities:

1. Use a tuple of types to mean a tuple of types: (int, str).

This is exactly what we already do in isinstance, the except statement, etc. And it's how Swift, C++, D, and other languages specify a tuple of types (although Swift can also use a product).

This definitely would be potentially confusing if we went with the obiwan-inspired syntax of [str] for lists (or iterables or mutable sequences) of str, but since I think Guido has pretty conclusively argued against that syntax on other grounds, this isn't a problem.


This probably implies that function types should be written as Function[(str, int), int] instead of Function[[str, int], int], but I think MyPy already handles that, and I think it makes more sense anyway.

2. Use a product of types to mean a tuple: int*str.


Tuples really are just product types. This is how you specify them in type theory (and relational theory, and elsewhere).?This is exactly how ML, Haskell, and other languages that designed their type systems carefully instead of haphazardly represent tuples of types.?It also nicely parallels the suggestions for str|int for union types and str&int for multi-inherited subtypes.

The first big problem here is that there's no way to specify a tuple of one type. That's not a problem for theory-inspired languages, because in those languages, a tuple of one value is the same thing as that value, but that's obviously not true for Python. The fact that Python doesn't have an appropriate unit type (None is a value that people frequently want to use in tuples) is also at least a theoretical problem.

Also, unless we were going to change isinstance, except, etc. to accept (or require) this syntax instead of a tuple of types, I think it would be confusing to remember that you use a tuple of types in some places, a product of types in others. Unlike subscripting, this looks too similar, and too syntactic, to avoid confusion.


3. Use subscripting, but a different form of subscripting, as Steven suggested: Tuple[::int, str].


This isn't actually right; it doesn't mean Tuple[::(int, str)], but Tuple[(::int), str]. But let's assume it's possible to come up with a readable syntax that doesn't require parentheses and ignore that problem.

This implies a more general use of slicing in type subscription: the start is the type parameter, and the step is some other stuff to be used in a special way that's specific to that type. If we have other such uses, that would be a door worth leaving open, but I suspect we don't. (If we had dynamic/implicit named tuples, as people have suggested a few times, would that be relevant here?)

Also notice that this is still passing a tuple of types, it's just wrapping it up in a slice and then passing that to Tuple just so it can mark the tuple of types as actually meaning a tuple of types. Is that adding enough additional information to be worth all that additional verbosity and complexity?

4. Just use Tuple[int, str] and note that this is a special case that doesn't mean the same thing as other generic types.

It seems to me potentially very confusing to have Tuple[int] mean a homogeneous arbitrary-length immutable sequence of ints, while Tuple[int, str] means exactly one int and one str (and Tuple[int,] means exactly one int). You could argue that this confusion is inherent in Python's use of tuples for those two different cases in the first place, but we're still spreading that confusion further.

If there were no better alternatives, I think this might be better than Steven's suggestion (practicality beats purity, and his suggestion really doesn't remove that much confusion), but I think there are better alternatives?namely, the first one.



On Sunday, August 17, 2014 5:44 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
 

>
>
>On 17 August 2014 21:44, Devin Jeanpierre <jeanpierreda at gmail.com> wrote:
>> On Sun, Aug 17, 2014 at 1:23 AM, Ben Finney <ben+python at benfinney.id.au> wrote:
>>> I have encountered many uses of ?homogeneous, variable-length sequence?
>>> and every time a Python tuple is used for that, I perceive a Python list
>>> would be better precisely *because* it better indicates that semantic
>>> meaning.
>>>
>>> I'd like to know how you think that's not true, and what real-world code
>>> makes you think so.
>>
>> isinstance is real world code that for the second parameter accepts
>> types and (recursively) tuples of any length of things it accepts.
>
>There are a few other cases where tuples are special cased as arguments:
>
>- str.__mod__
>- str.startswith (ditto for binary sequences)
>- str.endswith (ditto for binary sequences)
>
>>>> "aa".endswith(['a', 'b', 'c'])
>Traceback (most recent call last):
>? File "<stdin>", line 1, in <module>
>TypeError: endswith first arg must be str or a tuple of str, not list
>
>Searching the C files for "tuple of" turned up a couple more:
>
>* N-dimensional indexing also relies specifically on
>tuples-of-integers, rather than arbitrary iterators.
>
>* dynamic type creation expects to receive the bases as a tuple
>
>* the decimal module uses tuples of digits for internal data representations
>
>And that inspired recollection of several other cases where mutability
>would be wrong, because the tuple represents cached information rather
>than dynamic state:
>
>* various "*args" related APIs use "tuple of object" or "tuple of
>thing" (e.g. attributes of partial objects, internal storage in
>contextlib.ExitStack when used with arbitrary callbacks)
>
>* other introspection related APIs use tuples to report information
>about inspected objects
>
>* namedtuple _fields attributes are a tuple of strings
>
>* BaseException.args publishes the full args tuple passed to the constructor
>
>str.startswith, str.endswith, isinstance and issubclass use the
>"implied or" interpretation, everything else does not. In most cases,
>the immutability conveys relevant semantic information (usually
>indicating that it's a read-only API).
>
>Cheers,
>Nick.
>
>-- 
>Nick Coghlan?  |? ncoghlan at gmail.com?  |?  Brisbane, Australia
>
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>https://mail.python.org/mailman/listinfo/python-ideas
>Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/c9848854/attachment-0001.html>

From guido at python.org  Sun Aug 17 22:34:36 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 17 Aug 2014 13:34:36 -0700
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
Message-ID: <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>

Where is it said that Tuple[int] is a homogeneous variable size list?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/d4057a32/attachment.html>

From abarnert at yahoo.com  Mon Aug 18 00:46:49 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 17 Aug 2014 15:46:49 -0700
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing -- the
	crossroads)
In-Reply-To: <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>	<53EFBD98.2090802@stoneleaf.us>	<20140817020851.GL4525@ando>	<CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>	<854mxb5xt3.fsf_-_@benfinney.id.au>	<CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>	<CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>	<1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
Message-ID: <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>

On Sunday, August 17, 2014 1:34 PM, Guido van Rossum <guido at python.org> wrote:

>Where is it said that Tuple[int] is a homogeneous variable size list?


(I'm assuming you're referring to the homogeneity and arbitrary length here, not the fact that someone presumably said "list" when they meant "tuple", because otherwise the answer is trivial?)

First, that's how the current typing.py interprets it: Tuple[str] is a homogeneous, arbitrary-length (although of course unchanging, because it's immutable) tuple of strings.


Second, what else _would_ it mean? If List[str] and Set[str] mean homogeneous arbitrary-length lists and sets of strs, and the same goes for Iterable[str] and MutableSequence[str] and IO[str] and AnyStr[str] and every other example in typing.py, it would be pretty surprising if it weren't the same for Tuple[str].

Third, if it didn't mean that, how would you define the argument types to any of Nick's examples? For example:

? ? def isinstance(obj: object, types: type | Tuple[type]) -> bool:

That had better mean a homogeneous arbitrary-length tuple of types; if not, there doesn't seem to be any other way to declare its type.

From guido at python.org  Mon Aug 18 01:15:27 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 17 Aug 2014 16:15:27 -0700
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
Message-ID: <CAP7+vJLTQZRVYbOKLx9nEg=K4sn8=jF0rV6yMtVNuhku9TBhug@mail.gmail.com>

I still think you are mistaken. I don't think mypy has a way to spell a
homogeneous arbitrary-length tuple. All uses of Tuple[...] refer to
"anonymous struct" tuples.

I tried this:

from typing import Tuple

def f(a: Tuple[int]) -> None:
    pass

def main() -> None:
    f((1,))
    f((1, 2))

and I get an error for the second call, f((1, 2)):

a.py: In function "main":
a.py, line 8: Argument 1 to "f" has incompatible type "Tuple[int, int]";
expected "Tuple[int]"



On Sun, Aug 17, 2014 at 3:46 PM, Andrew Barnert <
abarnert at yahoo.com.dmarc.invalid> wrote:

> On Sunday, August 17, 2014 1:34 PM, Guido van Rossum <guido at python.org>
> wrote:
>
> >Where is it said that Tuple[int] is a homogeneous variable size list?
>
>
> (I'm assuming you're referring to the homogeneity and arbitrary length
> here, not the fact that someone presumably said "list" when they meant
> "tuple", because otherwise the answer is trivial?)
>
> First, that's how the current typing.py interprets it: Tuple[str] is a
> homogeneous, arbitrary-length (although of course unchanging, because it's
> immutable) tuple of strings.
>
>
> Second, what else _would_ it mean? If List[str] and Set[str] mean
> homogeneous arbitrary-length lists and sets of strs, and the same goes for
> Iterable[str] and MutableSequence[str] and IO[str] and AnyStr[str] and
> every other example in typing.py, it would be pretty surprising if it
> weren't the same for Tuple[str].
>
> Third, if it didn't mean that, how would you define the argument types to
> any of Nick's examples? For example:
>
>     def isinstance(obj: object, types: type | Tuple[type]) -> bool:
>
> That had better mean a homogeneous arbitrary-length tuple of types; if
> not, there doesn't seem to be any other way to declare its type.
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/3c6338b7/attachment.html>

From steve at pearwood.info  Mon Aug 18 01:16:39 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 18 Aug 2014 09:16:39 +1000
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing -- the
	crossroads)
In-Reply-To: <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
Message-ID: <20140817231638.GC25957@ando>

On Sun, Aug 17, 2014 at 01:34:36PM -0700, Guido van Rossum wrote:

> Where is it said that Tuple[int] is a homogeneous variable size list?

Did you mean variable sized tuple rather than list? The mypy tutorial 
says the opposite:

http://www.mypy-lang.org/tutorial.html#tuples

which implies that Tuple[int] would accept (23,) but not (23, 42). 


There's two, or three, cases to consider:

Variable sized tuple of some homogenous type
- e.g. (1,), (1, 2)
- Tuple[int] for consistency with other types
- mypy doesn't appear to support this

Fixed size tuple of given hetrogeneous types
- e.g. (23, "abc"), (42, "xyz")
- mypy uses Tuple[int, str] which is a special case

Some way to specify two or more types, e.g. 
- (str, int) for consistency with isinstance, issubclass
- Union[str, int] as used by mypy
- str|int obvious short-cut for Union
- str*int will make type theorists happy


Making Tuple[] a special case troubles me, I strongly prefer Tuple[int] 
to mean a homogenous tuple of ints.

mypy already has Union[str, int] for union types (giving str|int as the 
obvious short-cut), which leaves (str, int) for the hetrogenous 2-tuple 
case.

Perhaps in 3.6 we can consider allowing isinstance and issubclass to 
accept Union types as well as tuples of types.


-- 
Steven

From guido at python.org  Mon Aug 18 01:17:39 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 17 Aug 2014 16:17:39 -0700
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <CAP7+vJLTQZRVYbOKLx9nEg=K4sn8=jF0rV6yMtVNuhku9TBhug@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJLTQZRVYbOKLx9nEg=K4sn8=jF0rV6yMtVNuhku9TBhug@mail.gmail.com>
Message-ID: <CAP7+vJ+=zF+Lh3OX5_zpSDbntnZAeV=fvj8m29LU5kkhh4vKzw@mail.gmail.com>

I sent that a little too soon; I should add that I think this is the right
way; and that's why I keep suggesting a different way to spell a
homogeneous variable-length tuple. 1-tuples should not be special.


On Sun, Aug 17, 2014 at 4:15 PM, Guido van Rossum <guido at python.org> wrote:

> I still think you are mistaken. I don't think mypy has a way to spell a
> homogeneous arbitrary-length tuple. All uses of Tuple[...] refer to
> "anonymous struct" tuples.
>
> I tried this:
>
> from typing import Tuple
>
> def f(a: Tuple[int]) -> None:
>     pass
>
> def main() -> None:
>     f((1,))
>     f((1, 2))
>
> and I get an error for the second call, f((1, 2)):
>
> a.py: In function "main":
> a.py, line 8: Argument 1 to "f" has incompatible type "Tuple[int, int]";
> expected "Tuple[int]"
>
>
>
> On Sun, Aug 17, 2014 at 3:46 PM, Andrew Barnert <
> abarnert at yahoo.com.dmarc.invalid> wrote:
>
>> On Sunday, August 17, 2014 1:34 PM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>> >Where is it said that Tuple[int] is a homogeneous variable size list?
>>
>>
>> (I'm assuming you're referring to the homogeneity and arbitrary length
>> here, not the fact that someone presumably said "list" when they meant
>> "tuple", because otherwise the answer is trivial?)
>>
>> First, that's how the current typing.py interprets it: Tuple[str] is a
>> homogeneous, arbitrary-length (although of course unchanging, because it's
>> immutable) tuple of strings.
>>
>>
>> Second, what else _would_ it mean? If List[str] and Set[str] mean
>> homogeneous arbitrary-length lists and sets of strs, and the same goes for
>> Iterable[str] and MutableSequence[str] and IO[str] and AnyStr[str] and
>> every other example in typing.py, it would be pretty surprising if it
>> weren't the same for Tuple[str].
>>
>> Third, if it didn't mean that, how would you define the argument types to
>> any of Nick's examples? For example:
>>
>>     def isinstance(obj: object, types: type | Tuple[type]) -> bool:
>>
>> That had better mean a homogeneous arbitrary-length tuple of types; if
>> not, there doesn't seem to be any other way to declare its type.
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/ecbb78fc/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Mon Aug 18 01:19:01 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 18 Aug 2014 11:19:01 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <7DA218AC-2E88-4643-8EFC-C9E4EFB31136@langa.pl>
 <2E3F6DE2-2247-4349-A22F-7E22A8BDBFCD@gmail.com>
Message-ID: <53F13865.4040302@canterbury.ac.nz>

Andrey Vlasovskikh wrote:
> There is a problem in static analysis of current types that are instances of
> abc.ABCMeta or types that just define their own __instancecheck__ /
> __subclasscheck__. Static analyzers cannot infer in general case what
> attributes of an instance / subclass do these methods check, because their
> body can be arbitrarily complex.

However, I'd be worried about having two very similar
but subtly different sets of type objects floating around.
That just seems like a recipe for massive confusion.

-- 
Greg

From lukasz at langa.pl  Mon Aug 18 01:19:16 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sun, 17 Aug 2014 16:19:16 -0700
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing -- the
	crossroads)
In-Reply-To: <20140817231638.GC25957@ando>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <20140817231638.GC25957@ando>
Message-ID: <8AAD22FE-7366-4BC6-9ABE-8126C9BE7924@langa.pl>

On Aug 17, 2014, at 4:16 PM, Steven D'Aprano <steve at pearwood.info> wrote:

> Perhaps in 3.6 we can consider allowing isinstance and issubclass to 
> accept Union types as well as tuples of types.


Why not 3.5?

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/cd30432f/attachment.html>

From steve at pearwood.info  Mon Aug 18 01:24:29 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 18 Aug 2014 09:24:29 +1000
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing -- the
	crossroads)
In-Reply-To: <8AAD22FE-7366-4BC6-9ABE-8126C9BE7924@langa.pl>
References: <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <20140817231638.GC25957@ando> <8AAD22FE-7366-4BC6-9ABE-8126C9BE7924@langa.pl>
Message-ID: <20140817232429.GD25957@ando>

On Sun, Aug 17, 2014 at 04:19:16PM -0700, ?ukasz Langa wrote:
> On Aug 17, 2014, at 4:16 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> 
> > Perhaps in 3.6 we can consider allowing isinstance and issubclass to 
> > accept Union types as well as tuples of types.
> 
> 
> Why not 3.5?

In case Guido changes his mind :-)



-- 
Steven

From lukasz at langa.pl  Mon Aug 18 01:27:56 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sun, 17 Aug 2014 16:27:56 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <lspljq$bqt$1@ger.gmane.org> <53F076D9.4020807@canterbury.ac.nz>
 <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com>
Message-ID: <D71E65AB-6766-4270-8E9D-5FFEED942556@langa.pl>

On Aug 17, 2014, at 3:53 AM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote:

> The obvious way that already works (with MyPy, and also with ABCs for isinstance checking):
> 
>    class HashableIterable(Iterable, Hashable): pass
> 
>    def spam(a: HashableIterable):
>        pass

No. Specifying a subclass of Iterable and Hashable cannot possibly mean that *any* type that is both Iterable and Hashable is behaving like your subclass. 

>>> from collections import *
>>> class IterableHashable(Iterable, Hashable): pass
...
>>> isinstance("str", Iterable)
True
>>> isinstance("str", Hashable)
True
>>> isinstance("str", IterableHashable)
False

We would need a new construct like Union, maybe named AnySubclass, that checks True when all its base classes are in a given type's MRO. Some ABCs don't explicitly appear in a type's MRO but I fixed this problem with singledispatch.

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/721c777c/attachment.html>

From lukasz at langa.pl  Mon Aug 18 01:35:11 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sun, 17 Aug 2014 16:35:11 -0700
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing -- the
	crossroads)
In-Reply-To: <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>	<53EFBD98.2090802@stoneleaf.us>	<20140817020851.GL4525@ando>	<CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>	<854mxb5xt3.fsf_-_@benfinney.id.au>	<CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>	<CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>	<1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
Message-ID: <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>

On Aug 17, 2014, at 3:46 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote:

> Second, what else _would_ it mean? If List[str] and Set[str] mean homogeneous arbitrary-length lists and sets of strs, and the same goes for Iterable[str] and MutableSequence[str] and IO[str] and AnyStr[str] and every other example in typing.py, it would be pretty surprising if it weren't the same for Tuple[str].

You're playing the uniformity card and usually I would agree with you. In this case, there is no uniformity between different *data structures*. Nobody expects to be able to say: int[int], dict[str], set[str: int]. Tuples were meant to be heterogenic, Raymond draws that distinction in many classes and talks he gives. On the other hand, some types are hard to be typed heterogenically, e.g. you can't reasonably specify an Iterable that yield strings except for the third yield, which is an int.

All in all, I think it's not at all confusing to say that tuple[int, int] means a point, for instance (1, 1). That being said, we will need support for homogenous tuples, too, simply because they are already in the wild. I proposed tuple[int, ...], which is explicit and obvious (if you're Polish, that is).

-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/393a2ce8/attachment-0001.html>

From rosuav at gmail.com  Mon Aug 18 01:40:53 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 18 Aug 2014 09:40:53 +1000
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
Message-ID: <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>

On Mon, Aug 18, 2014 at 9:35 AM, ?ukasz Langa <lukasz at langa.pl> wrote:
> That being said, we will need support for homogenous tuples, too, simply
> because they are already in the wild. I proposed tuple[int, ...], which is
> explicit and obvious (if you're Polish, that is).

Conceptually, these kinds of constructs are sometimes called
frozenlists. Why not actually create that type?

frozenlist = tuple;

Et voila. Now just define that frozenlist[int] is like list[int]
rather than like tuple[int], and there you are, out of your difficulty
at once!

ChrisA

From greg.ewing at canterbury.ac.nz  Mon Aug 18 01:43:51 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 18 Aug 2014 11:43:51 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAPTjJmogZmhtVTcQEG00-zhhJ-F66qGC+7waYD0jdcBCYJjV6g@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <20140817102059.GS4525@ando>
 <CAPTjJmogZmhtVTcQEG00-zhhJ-F66qGC+7waYD0jdcBCYJjV6g@mail.gmail.com>
Message-ID: <53F13E37.5090807@canterbury.ac.nz>

Chris Angelico wrote:
> I don't, for instance, see
> real-world examples of decorators that add extra parameters to a
> function, even though it would be plausible

Some decorators, such as property(), don't return a
function at all. Ignoring the decorator in that case
would give completely the wrong idea.

I'm now thinking the right thing to do with decorators
is to analyse them statically if possible, otherwise
treat the result as untyped.

If a decorator is well-behaved, type inference should
be able to propagate the types of the decorated
function through to the result. If not, all bets
are off.

-- 
Greg

From lukasz at langa.pl  Mon Aug 18 01:47:49 2014
From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=)
Date: Sun, 17 Aug 2014 16:47:49 -0700
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing -- the
	crossroads)
In-Reply-To: <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
Message-ID: <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>

On Aug 17, 2014, at 4:40 PM, Chris Angelico <rosuav at gmail.com> wrote:

> Et voila. Now just define that frozenlist[int] is like list[int]
> rather than like tuple[int], and there you are, out of your difficulty
> at once!

That's a neat trick and I think Guido suggested this naming before. It feels a little icky though, consider:

- issubclass(tuple[int, int, int], frozenlist[int]) ?  I think True.
- issubclass(frozenlist[int], tuple[int, int, int]) ?  I would think False. But because that's technically the same tuple underneath, *sometimes* instances of frozenlist[int] will respond True to isinstance(tuple[int, int, int]).

Saying explicitly tuple[int, ...] takes that riddle away.


-- 
Best regards,
?ukasz Langa

WWW: http://lukasz.langa.pl/
Twitter: @llanga
IRC: ambv on #python-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/0ba9cd52/attachment.html>

From rosuav at gmail.com  Mon Aug 18 01:59:24 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 18 Aug 2014 09:59:24 +1000
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
 <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
Message-ID: <CAPTjJmq-EjBXsGi=v4C3zESo2G8mq+8VCzNCioyykz0p7AZ23A@mail.gmail.com>

On Mon, Aug 18, 2014 at 9:47 AM, ?ukasz Langa <lukasz at langa.pl> wrote:
> - issubclass(tuple[int, int, int], frozenlist[int]) ?  I think True.
> - issubclass(frozenlist[int], tuple[int, int, int]) ?  I would think False.
> But because that's technically the same tuple underneath, *sometimes*
> instances of frozenlist[int] will respond True to isinstance(tuple[int, int,
> int]).

I would say False to both of those, because they have different
implications of intent. But it is arguable, so I'd be happy with the
first one being True if it's easier to code that way.

ChrisA

From greg.ewing at canterbury.ac.nz  Mon Aug 18 02:04:20 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 18 Aug 2014 12:04:20 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <lspljq$bqt$1@ger.gmane.org> <53F076D9.4020807@canterbury.ac.nz>
 <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com>
Message-ID: <53F14304.6040705@canterbury.ac.nz>

Andrew Barnert wrote:
> The obvious way that already works (with MyPy, and also with ABCs for isinstance checking):
> 
>     class HashableIterable(Iterable, Hashable): pass
> 
>     def spam(a: HashableIterable):
>         pass

That way might be obvious, but it's wrong. It says that spam()
only takes an instance of the specific class HashableIterable
or a subclass thereof. It won't accept anything else, even if
it implements Iterable and Hashable perfectly well.

> 
> But, there's no reason typing.py couldn't add a wrapper that does this automatically:
> 
>     class Multiple:
>         @staticmethod
>         def __getitem__(*types);
>             return type(types[0])(
>                 '_'.join(t.__name__ for t in types), types, {})

Automating it won't make it any more right. There needs to
be a distinct concept such as Intersection() as a counterpart
to Union().

-- 
Greg

From guido at python.org  Mon Aug 18 02:30:37 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 17 Aug 2014 17:30:37 -0700
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
 <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
Message-ID: <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>

On Sun, Aug 17, 2014 at 4:47 PM, ?ukasz Langa <lukasz at langa.pl> wrote:

> On Aug 17, 2014, at 4:40 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
> Et voila. Now just define that frozenlist[int] is like list[int]
> rather than like tuple[int], and there you are, out of your difficulty
> at once!
>
> That's a neat trick and I think Guido suggested this naming before. It
> feels a little icky though, consider:
>
> - issubclass(tuple[int, int, int], frozenlist[int]) ?  I think True.
> - issubclass(frozenlist[int], tuple[int, int, int]) ?  I would think
> False. But because that's technically the same tuple underneath,
> *sometimes* instances of frozenlist[int] will respond True to
> isinstance(tuple[int, int, int]).
>
> Saying explicitly tuple[int, ...] takes that riddle away.
>

Hm. That looks just as [tr]icky. Plus, I expect both pedants and
ignoramuses may wonder about the empty tuple. :-) I think it deserves a
proper name, not magic syntax.

I do agree that frozentuple sounds odd, but perhaps we just need to get
used to it.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/c0d181c8/attachment-0001.html>

From jlehtosalo at gmail.com  Mon Aug 18 03:01:47 2014
From: jlehtosalo at gmail.com (Jukka Lehtosalo)
Date: Sun, 17 Aug 2014 18:01:47 -0700
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
 <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
 <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
Message-ID: <CAA_f+LxGqincxq-AXOACFgCV74KWtx6P0cm9=K-sRfK3uoJJFA@mail.gmail.com>

On Sun, Aug 17, 2014 at 5:30 PM, Guido van Rossum <guido at python.org> wrote:

> On Sun, Aug 17, 2014 at 4:47 PM, ?ukasz Langa <lukasz at langa.pl> wrote:
>
>> That's a neat trick and I think Guido suggested this naming before. It
>> feels a little icky though, consider:
>>
>> - issubclass(tuple[int, int, int], frozenlist[int]) ?  I think True.
>> - issubclass(frozenlist[int], tuple[int, int, int]) ?  I would think
>> False. But because that's technically the same tuple underneath,
>> *sometimes* instances of frozenlist[int] will respond True to
>> isinstance(tuple[int, int, int]).
>>
>> Saying explicitly tuple[int, ...] takes that riddle away.
>>
>
> Hm. That looks just as [tr]icky. Plus, I expect both pedants and
> ignoramuses may wonder about the empty tuple. :-) I think it deserves a
> proper name, not magic syntax.
>
> I do agree that frozentuple sounds odd, but perhaps we just need to get
> used to it.
>

I've considered adding a mypy type for arbitrary-length tuples [1]. My
original idea was to call it TupleSequence[T], since it's basically a
concrete tuple that is used like a sequence.

Fixed-length tuples can already be used as abstract iterables and sequences:

    def f(x: Iterable[int]) -> None: pass
    f((1, 2, 3))   # Ok

[1] https://github.com/JukkaL/mypy/issues/184

Jukka


>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/9dcd4f6e/attachment.html>

From abarnert at yahoo.com  Mon Aug 18 03:05:02 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 17 Aug 2014 18:05:02 -0700
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing --
	the	crossroads)
In-Reply-To: <20140817231638.GC25957@ando>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <20140817231638.GC25957@ando>
Message-ID: <1408323902.81771.YahooMailNeo@web181001.mail.ne1.yahoo.com>

On Sunday, August 17, 2014 4:17 PM, Steven D'Aprano <steve at pearwood.info> wrote:



> There's two, or three, cases to consider:
> 
> Variable sized tuple of some homogenous type
> - e.g. (1,), (1, 2)
> - Tuple[int] for consistency with other types
> - mypy doesn't appear to support this

Are you sure?

In [typing.py](https://github.com/JukkaL/mypy/blob/master/lib-typing/3.2/typing.py#L17), Tuple is defined as TypeAlias(tuple), exactly the same way List is defined as TypeAlias(list). And I don't see any code anywhere else in that module that adds the special-casing to it.

So, isn't this what MyPy is already doing? Or is there some hidden functionality outside of typing.py that changes it?

Anyway, I agree with you that, whatever it _currently_ means in MyPy, it _should_ mean a tuple of an arbitrary number of int values.

> Fixed size tuple of given hetrogeneous types
> - e.g. (23, "abc"), (42, "xyz")
> - mypy uses Tuple[int, str] which is a special case

This is the one I'd like to write as (int, str).

> Some way to specify two or more types, e.g. 
> - (str, int) for consistency with isinstance, issubclass
> - Union[str, int] as used by mypy
> - str|int obvious short-cut for Union
> - str*int will make type theorists happy

That last one will not make type theorists happy. A product of two types is a tuple (your case #2, not #3). What you want is a sum or union of two types, which you'd write str|int, str U int, or maybe str+int.

> Making Tuple[] a special case troubles me, I strongly prefer Tuple[int]?

> to mean a homogenous tuple of ints.
> 
> mypy already has Union[str, int] for union types (giving str|int as the 
> obvious short-cut), which leaves (str, int) for the hetrogenous 2-tuple 
> case.
> 
> Perhaps in 3.6 we can consider allowing isinstance and issubclass to 
> accept Union types as well as tuples of types.
> 
> 
> -- 
> Steven
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

From rosuav at gmail.com  Mon Aug 18 03:11:23 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 18 Aug 2014 11:11:23 +1000
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53F13E37.5090807@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <20140817102059.GS4525@ando>
 <CAPTjJmogZmhtVTcQEG00-zhhJ-F66qGC+7waYD0jdcBCYJjV6g@mail.gmail.com>
 <53F13E37.5090807@canterbury.ac.nz>
Message-ID: <CAPTjJmqXiBoE7vaQk+=TLh1mO=7KYsWSn+S9=4wfZwZ72259+A@mail.gmail.com>

On Mon, Aug 18, 2014 at 9:43 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Chris Angelico wrote:
>>
>> I don't, for instance, see
>> real-world examples of decorators that add extra parameters to a
>> function, even though it would be plausible
>
>
> Some decorators, such as property(), don't return a
> function at all. Ignoring the decorator in that case
> would give completely the wrong idea.

So you don't annotate the function that handles the property. Simple!

There may need to be some other handling of it (telling mypy that this
property will always be a str, for instance), but the function's
arguments aren't significant to the type system, so they don't need
annotations. Same is true of anything else that decorates the function
away altogether.

ChrisA

From abarnert at yahoo.com  Mon Aug 18 03:12:58 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 17 Aug 2014 18:12:58 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <53F14304.6040705@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <lspljq$bqt$1@ger.gmane.org> <53F076D9.4020807@canterbury.ac.nz>
 <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com>
 <53F14304.6040705@canterbury.ac.nz>
Message-ID: <1408324378.30223.YahooMailNeo@web181003.mail.ne1.yahoo.com>

> On Sunday, August 17, 2014 5:04 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> > Andrew Barnert wrote:
>>  The obvious way that already works (with MyPy, and also with ABCs for 
> isinstance checking):
>> 
>> ? ?  class HashableIterable(Iterable, Hashable): pass
>> 
>> ? ?  def spam(a: HashableIterable):
>> ? ? ? ?  pass
> 
> That way might be obvious, but it's wrong. It says that spam()
> only takes an instance of the specific class HashableIterable
> or a subclass thereof. It won't accept anything else, even if
> it implements Iterable and Hashable perfectly well.

typing.Iterable and typing.Hashable are both typing.Protocols, so HashableIterable is also a typing.Protocol, so, if I understand Protocol correctly, isinstance will return True iff every method and attribute in HashableIterable (that is, every method and attribute in Iterable, plus every method and attribute in Hashable) is implemented by the type. Which is what we want here, right?

That obviously doesn't work for nominal rather than structural ABCs, or for ad-hoc structural ABCs like the ones in collections.abc that don't automatically compose, but isn't that the whole point of Protocol, to provide structural subtyping that follows the rules of structural rather than nominative subtyping?

> 
>> 
>>  But, there's no reason typing.py couldn't add a wrapper that does 
> this automatically:
>> 
>> ? ?  class Multiple:
>> ? ? ? ?  @staticmethod
>> ? ? ? ?  def __getitem__(*types);
>> ? ? ? ? ? ?  return type(types[0])(
>> ? ? ? ? ? ? ? ?  '_'.join(t.__name__ for t in types), types, 
> {})
> 
> Automating it won't make it any more right. There needs to
> be a distinct concept such as Intersection() as a counterpart
> to Union().
> 
> 
> -- 
> Greg
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> 

From schesis at gmail.com  Mon Aug 18 03:21:21 2014
From: schesis at gmail.com (Zero Piraeus)
Date: Sun, 17 Aug 2014 21:21:21 -0400
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
Message-ID: <20140818012121.GA10658@piedra>

:

On Sun, Aug 17, 2014 at 12:31:19PM -0700, Andrew Barnert wrote:
> I think we're conflating multiple problems here.
> 
> Sometimes we use tuple to mean a homogeneous, arbitrary-length,
> immutable sequence, and other times we use it to mean a heterogeneous,
> fixed-length sequence.?Nick's list demonstrates that the former is (a)
> common enough to worry about, and (b) not always a mistake.
> 
> [...]
> 
> I can think of four possibilities:

Since we're already repurposing a number of operators - TypeA[TypeB],
TypeA|TypeB and possibly TypeA&TypeB, here's a fifth possibility:

Leave Tuple[int] with the same semantic meaning as List[int], Set[int]
etc., and use e.g. Tuple%(int, str, float) to declare the signature of a
fixed-length, heterogenous sequence (borrowing from the use of % in
string interpolation).

This has the advantages that it 

a) doesn't make tuple a special case, and so allows other container
types to use it (including user-defined types which aren't tuples, but
are intended to behave similarly to them), and

b) is both reasonably compact, and easily distinguished from Tuple[int].

The specific operator used doesn't really matter - when this idea came
to me I thought of @ (since it's shiny and new in 3.5), but % seems a
better conceptual fit.

 -[]z.

-- 
Zero Piraeus: unus multorum
http://etiol.net/pubkey.asc

From ryan at ryanhiebert.com  Mon Aug 18 03:25:35 2014
From: ryan at ryanhiebert.com (Ryan Hiebert)
Date: Sun, 17 Aug 2014 20:25:35 -0500
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing -- the
	crossroads)
In-Reply-To: <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
 <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
 <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
Message-ID: <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com>


> On Aug 17, 2014, at 7:30 PM, Guido van Rossum <guido at python.org> wrote:
> 
> I do agree that frozentuple sounds odd, but perhaps we just need to get used to it.

I?d expect that most uses of Tuple[int] to mean an arbitrary length tuple of integers would be better served with Sequence[int] anyway, so I?d definitely find it odd for Tuple[int] to mean anything other than a tuple with exactly one integer in it.

I could imagine somebody wanting to say that it could be a Sequence but NOT a MutableSequence. Perhaps that could be spelled:

Sequence[int] - MutableSequence[int]

and/or

(Sequence - MutableSequence)[int]

From abarnert at yahoo.com  Mon Aug 18 03:41:08 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 17 Aug 2014 18:41:08 -0700
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing -- the
	crossroads)
In-Reply-To: <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>	<53EFBD98.2090802@stoneleaf.us>	<20140817020851.GL4525@ando>	<CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>	<854mxb5xt3.fsf_-_@benfinney.id.au>	<CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>	<CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>	<1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
Message-ID: <1408326068.7864.YahooMailNeo@web181006.mail.ne1.yahoo.com>

On Sunday, August 17, 2014 4:36 PM, ?ukasz Langa <lukasz at langa.pl> wrote:

>On Aug 17, 2014, at 3:46 PM, Andrew Barnert <abarnert at yahoo.com.dmarc.invalid> wrote:
>
>>Second, what else _would_ it mean? If List[str] and Set[str] mean homogeneous arbitrary-length lists and sets of strs, and the same goes for Iterable[str] and MutableSequence[str] and IO[str] and AnyStr[str] and every other example in typing.py, it would be pretty surprising if it weren't the same for Tuple[str].
>>
>
>
>You're playing the uniformity card and usually I would agree with you. In this case, there is no uniformity between different *data structures*.

There is uniformity between every single generic data structure defined by MyPy except Tuple. That makes Tuple an exceedingly special special case.

And its specialness is almost invisible. I went through typing.py in more detail, and I'm pretty sure there is nothing there to indicate that the Tuple = TypeAlias(tuple) line is going to do anything different than all of the other TypeAlias calls. Whatever difference there is must be in the private, implementation-specific code.


> Nobody expects to be able to say: int[int], dict[str], set[str: int].

But int and dict are not sequences; tuple is.

> Tuples were meant to be heterogenic, Raymond draws that distinction in many classes and talks he gives.

Sure. But?as Nick pointed out, there are places all over even the core language and the stdlib where they're used homogeneously. You can argue that was a mistake, but you can't just wish it away.

>All in all, I think it's not at all confusing to say that tuple[int, int] means a point, for instance (1, 1).?

I think it's far less confusing to say that (int, int) is the type of (1, 1).

And other languages agree:

? ? $ ghci
? ? Prelude> set +t
? ? Prelude> (1, 1)
? ? (1,1)
? ? it :: (Integer, Integer)
? ? Prelude> ^D
? ? Leaving GHCi.
? ? $ xcrun swift
? ? ? 1> (1, 1)

? ? $R0: (Int, Int) = {
? ? ? 0 = 1
? ? ? 1 = 1
? ? }
? ? ? 2> ^D

Those languages don't try to make heterogeneous tuples look like parametric collection types. Why should Python?

From guido at python.org  Mon Aug 18 03:49:59 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 17 Aug 2014 18:49:59 -0700
Subject: [Python-ideas] Variable-length,
 homogeneous tuple: why? (was: Optional static typing -- the
 crossroads)
In-Reply-To: <1408323902.81771.YahooMailNeo@web181001.mail.ne1.yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <20140817231638.GC25957@ando>
 <1408323902.81771.YahooMailNeo@web181001.mail.ne1.yahoo.com>
Message-ID: <CAP7+vJKL+o_oO=VWwkPvRShpoQ9a0ioPkXtpuzZ7dB7OAO9=1w@mail.gmail.com>

On Sun, Aug 17, 2014 at 6:05 PM, Andrew Barnert <
abarnert at yahoo.com.dmarc.invalid> wrote:

> On Sunday, August 17, 2014 4:17 PM, Steven D'Aprano <steve at pearwood.info>
> wrote:
>
> > There's two, or three, cases to consider:
> >
> > Variable sized tuple of some homogenous type
> > - e.g. (1,), (1, 2)
> > - Tuple[int] for consistency with other types
> > - mypy doesn't appear to support this
>
> Are you sure?
>
> In [typing.py](
> https://github.com/JukkaL/mypy/blob/master/lib-typing/3.2/typing.py#L17),
> Tuple is defined as TypeAlias(tuple), exactly the same way List is defined
> as TypeAlias(list). And I don't see any code anywhere else in that module
> that adds the special-casing to it.
>
> So, isn't this what MyPy is already doing? Or is there some hidden
> functionality outside of typing.py that changes it?
>

There is -- typing.py is not the typechecker, it's just a bunch of dummies
crafted so that your code can also be executed by vanilla Python 3.2. (If
you look at the definition of TypeAlias a few lines earlier, it has almost
no semantics, and you can see that it is also used for wildly other types,
e.g. Function and Union.


> Anyway, I agree with you that, whatever it _currently_ means in MyPy, it
> _should_ mean a tuple of an arbitrary number of int values.
>

I disagree. (I've said that before, but there seems to be a large delay
between our exchanges.)

The Foo[bar, ...] notation has no inherent semantics -- compare for example
Dict[str, int], Tuple[str, int], and Union[str, int]. The most useful way
to define Tuple[T1, T2, ..., Tn] is to use it for an n-tuple whose element
types are T1 etc.


> > Fixed size tuple of given hetrogeneous types
> > - e.g. (23, "abc"), (42, "xyz")
> > - mypy uses Tuple[int, str] which is a special case
>
> This is the one I'd like to write as (int, str).
>

I know that's your proposal, but it seems to blind you for the other
position.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/1b92438c/attachment-0001.html>

From abarnert at yahoo.com  Mon Aug 18 03:58:00 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 17 Aug 2014 18:58:00 -0700
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing --
	the	crossroads)
In-Reply-To: <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
 <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
 <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
 <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com>
Message-ID: <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com>

On Sunday, August 17, 2014 6:26 PM, Ryan Hiebert <ryan at ryanhiebert.com> wrote:
?
> I?d expect that most uses of Tuple[int] to mean an arbitrary length tuple of 
> integers would be better served with Sequence[int] anyway

It would be nice if that were true, but unfortunately, Python has a long history of using tuple, and only tuple, both in the core language and in the stdlib, specifically to mean an arbitrary-length homogeneous tuple in APIs or syntax that take a single value or tuple of values.

I don't want to repeat the whole list that Nick Coghlan provided, but I would like to add that any third-party code that interacts with those parts of the syntax or stdlib has to do the same thing (e.g., an exception-logging decorator has to taken an exception type or tuple of exception types to use in an except statement), and there's probably some third-party code that's done the same thing completely independently, just because it's endorsed by the stdlib.

From ryan at ryanhiebert.com  Mon Aug 18 04:18:57 2014
From: ryan at ryanhiebert.com (Ryan Hiebert)
Date: Sun, 17 Aug 2014 21:18:57 -0500
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing --
	the	crossroads)
In-Reply-To: <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
 <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
 <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
 <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com>
 <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com>
Message-ID: <F3C427BA-6ACB-4BAB-A118-4B806DED2C02@ryanhiebert.com>


> On Aug 17, 2014, at 8:58 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
> 
> On Sunday, August 17, 2014 6:26 PM, Ryan Hiebert <ryan at ryanhiebert.com> wrote:
>  
>> I?d expect that most uses of Tuple[int] to mean an arbitrary length tuple of 
>> integers would be better served with Sequence[int] anyway
> 
> It would be nice if that were true, but unfortunately, Python has a long history of using tuple, and only tuple, both in the core language and in the stdlib, specifically to mean an arbitrary-length homogeneous tuple in APIs or syntax that take a single value or tuple of values.
> 
> I don't want to repeat the whole list that Nick Coghlan provided, but I would like to add that any third-party code that interacts with those parts of the syntax or stdlib has to do the same thing (e.g., an exception-logging decorator has to taken an exception type or tuple of exception types to use in an except statement), and there's probably some third-party code that's done the same thing completely independently, just because it's endorsed by the stdlib.

Thanks. As I read more, I?m recalling some apis that use tuples as special markers for iterables rather than, say, strings, so that there can be polymorphism based on the arguments, allowing both a string and a tuple. Iterable strings bite again ;-)

From alexander.belopolsky at gmail.com  Mon Aug 18 07:02:36 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 18 Aug 2014 01:02:36 -0400
Subject: [Python-ideas] Variable-length,
	homogeneous tuple: why? (was: Optional static typing -- the
	crossroads)
In-Reply-To: <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
 <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
 <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
Message-ID: <B6FDCDD7-4AF5-40C1-9853-C9DB343DE9F9@gmail.com>



> On Aug 17, 2014, at 8:30 PM, Guido van Rossum <guido at python.org> wrote:
> 
> Plus, I expect both pedants and ignoramuses may wonder about the empty tuple. :-) I think it deserves a proper name, not magic syntax.

I agree, even though covering all cases in a consistent syntax does not seem hard:

Tuple[int] - variable length homogeneous tuple.
Tuple[int,float] - length 2
Tuple[int,] - length 1
Tuple[()] - length 0

Since length 1 and more so length 0 cases don't seem to get much use, a somewhat non-obvious syntax should be fine. 

From greg.ewing at canterbury.ac.nz  Mon Aug 18 08:06:47 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 18 Aug 2014 18:06:47 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAPTjJmr61=GA=H3pnPB6kHASQuVrfJm8NcMpWg+Ya1SJuUoSXA@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAAu18hew2L4zmoAcSpD9827YAXzJG2OdaAmQ94Ou4ROvE2Xc3w@mail.gmail.com>
 <53F08480.6060305@canterbury.ac.nz>
 <CAPTjJmr61=GA=H3pnPB6kHASQuVrfJm8NcMpWg+Ya1SJuUoSXA@mail.gmail.com>
Message-ID: <53F197F7.4090105@canterbury.ac.nz>

Chris Angelico wrote:
> With variable declarations,
> there's a difference between "int r,g,b; double x,y,z;", where the
> block of integers is terminated by a semicolon ...
 > in argument lists, you don't get that,

Python's argument annotation syntax *could* have been
defined to work the Pascal way, with semicolons
separating the groups. But it wasn't, and it's too
late to change now.

 > (Imagine you misspell a type
> name. It's no longer a keyword. How will your mistake be reported?)

I don't understand that. Type names are always clearly
separated from keywords in either style (they come after
a colon), so there's no danger of confusing them.

-- 
Greg



From greg.ewing at canterbury.ac.nz  Mon Aug 18 08:19:36 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 18 Aug 2014 18:19:36 +1200
Subject: [Python-ideas] Variable-length, homogeneous tuple: why?
In-Reply-To: <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
Message-ID: <53F19AF8.1050204@canterbury.ac.nz>

Andrew Barnert wrote:
> First, that's how the current typing.py interprets it: Tuple[str] is a
> homogeneous, arbitrary-length (although of course unchanging, because it's
> immutable) tuple of strings.

So how do you spell a heterogeneous tuple of length 1
containing a string?

-- 
Greg

From jlehtosalo at gmail.com  Mon Aug 18 08:29:01 2014
From: jlehtosalo at gmail.com (Jukka Lehtosalo)
Date: Sun, 17 Aug 2014 23:29:01 -0700
Subject: [Python-ideas] Variable-length, homogeneous tuple: why?
In-Reply-To: <53F19AF8.1050204@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <53F19AF8.1050204@canterbury.ac.nz>
Message-ID: <CAA_f+LyFmH-35Z8H-SnnA+KCbzEFkfh-+WagLpcaataVv+8dLQ@mail.gmail.com>

On Sun, Aug 17, 2014 at 11:19 PM, Greg Ewing <greg.ewing at canterbury.ac.nz>
wrote:

> Andrew Barnert wrote:
>
>> First, that's how the current typing.py interprets it: Tuple[str] is a
>> homogeneous, arbitrary-length (although of course unchanging, because it's
>> immutable) tuple of strings.
>>
>
> So how do you spell a heterogeneous tuple of length 1
> containing a string?
>

Tuple[str] :-)

The only arbitrary-length tuple that mypy knows about is just 'tuple' and
it isn't generic or necessarily homogeneous (it's dynamically typed,
basically a catch-all for a completely arbitrary tuple). However,
arbitrary-length, homogeneous tuples would be a nice addition, once we
figure out how the type should be named.

Jukka


>
> --
> Greg
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140817/64ac7f0d/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Mon Aug 18 08:33:11 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 18 Aug 2014 18:33:11 +1200
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <1408324378.30223.YahooMailNeo@web181003.mail.ne1.yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <lspljq$bqt$1@ger.gmane.org> <53F076D9.4020807@canterbury.ac.nz>
 <01CFE39F-8418-4316-91AB-E64C1F72ED0E@yahoo.com>
 <53F14304.6040705@canterbury.ac.nz>
 <1408324378.30223.YahooMailNeo@web181003.mail.ne1.yahoo.com>
Message-ID: <53F19E27.6010807@canterbury.ac.nz>

Andrew Barnert wrote:

> typing.Iterable and typing.Hashable are both typing.Protocols, so
> HashableIterable is also a typing.Protocol, so, if I understand Protocol
> correctly, isinstance will return True iff every method and attribute in
> HashableIterable (that is, every method and attribute in Iterable, plus every
> method and attribute in Hashable) is implemented by the type.

Urg. In that case, how do you spell a type that *is*
a concrete class that implements Hashable and Iterable,
rather than a new protocol?

There's too much magic going on here for my liking.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Mon Aug 18 08:41:07 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 18 Aug 2014 18:41:07 +1200
Subject: [Python-ideas] Variable-length, homogeneous tuple: why?
In-Reply-To: <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
 <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
 <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
 <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com>
 <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com>
Message-ID: <53F1A003.1010001@canterbury.ac.nz>

So we need a name for a type that's a tuple used
as a sequence.

How about Tuppence?

-- 
Greg


From ben+python at benfinney.id.au  Mon Aug 18 08:53:04 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 18 Aug 2014 16:53:04 +1000
Subject: [Python-ideas] Variable-length, homogeneous tuple: why?
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
 <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
 <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
 <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com>
 <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com>
 <53F1A003.1010001@canterbury.ac.nz>
Message-ID: <85a97247cf.fsf@benfinney.id.au>

Greg Ewing <greg.ewing at canterbury.ac.nz> writes:

> So we need a name for a type that's a tuple used
> as a sequence.

Um. Isn't every tuple used as a sequence? So, the name ?tuple? fits.

Maybe you mean ?used as a *homogeneous* sequence?. Or maybe you mean
?used as a *variable-length* sequence?. Or something else?

-- 
 \                   ?Too many Indians spoil the golden egg.? ?Sir Joh |
  `\                                                   Bjelke-Petersen |
_o__)                                                                  |
Ben Finney


From rosuav at gmail.com  Mon Aug 18 08:53:53 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 18 Aug 2014 16:53:53 +1000
Subject: [Python-ideas] Variable-length, homogeneous tuple: why?
In-Reply-To: <53F1A003.1010001@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
 <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
 <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
 <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com>
 <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com>
 <53F1A003.1010001@canterbury.ac.nz>
Message-ID: <CAPTjJmr3qG9hVpMEiEh-Hd0HTyJ_P4DNSjJN0B9TagkJOy3ABQ@mail.gmail.com>

On Mon, Aug 18, 2014 at 4:41 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> So we need a name for a type that's a tuple used
> as a sequence.
>
> How about Tuppence?

King Tuppence - or - A Good Deal Less Than Half a Sovereign?

ChrisA

From daviesk24 at yahoo.com  Mon Aug 18 17:24:58 2014
From: daviesk24 at yahoo.com (Kevin Davies)
Date: Mon, 18 Aug 2014 05:24:58 -1000
Subject: [Python-ideas] float comparison in doctest
In-Reply-To: <53E9B914.1000702@yahoo.com>
References: <53E8EE7B.50408@yahoo.com> <lsb183$f7f$1@ger.gmane.org>
 <lscbe0$ba4$1@ger.gmane.org> <53E9B914.1000702@yahoo.com>
Message-ID: <53F21ACA.9080206@yahoo.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140818/c46eae6e/attachment.html>

From daviesk24 at yahoo.com  Mon Aug 18 17:24:45 2014
From: daviesk24 at yahoo.com (Kevin Davies)
Date: Mon, 18 Aug 2014 05:24:45 -1000
Subject: [Python-ideas] float comparison in doctest
In-Reply-To: <53E9B914.1000702@yahoo.com>
References: <53E8EE7B.50408@yahoo.com> <lsb183$f7f$1@ger.gmane.org>
 <lscbe0$ba4$1@ger.gmane.org> <53E9B914.1000702@yahoo.com>
Message-ID: <53F21ABD.5000107@yahoo.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140818/439a384c/attachment.html>

From ethan at stoneleaf.us  Mon Aug 18 19:00:38 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 18 Aug 2014 10:00:38 -0700
Subject: [Python-ideas] Variable-length, homogeneous tuple: why?
In-Reply-To: <53F1A003.1010001@canterbury.ac.nz>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <854mxb5xt3.fsf_-_@benfinney.id.au>
 <CABicbJJ+qjG3Pd3vTv8M=DmdUuTE7ioS4Nc_PK8EQTVy_QoiiQ@mail.gmail.com>
 <CADiSq7c1kH96=N8h6GEZdn5xYoOZtLi3VcdMVYcjQdQu-gx5ww@mail.gmail.com>
 <1408303879.87633.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJK+srui=mawRV8WTABoCvp8931SAQ7-6+PLLnk3wu_Anw@mail.gmail.com>
 <1408315609.97650.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <BD320DDD-25E9-4229-834A-7710930AC821@langa.pl>
 <CAPTjJmpnb-m3YorCGJhg6PkuxwFJe35sGsT5EaekK_BsU7wB_Q@mail.gmail.com>
 <E7680560-9FEB-4F04-9B3E-7D446B3341A4@langa.pl>
 <CAP7+vJKiGcYstF171cAhNnxwDYt8uRZn5-tmSfXqx8GLj5FbVg@mail.gmail.com>
 <0F822F76-7A6E-4BB8-ACF8-A3D916778116@ryanhiebert.com>
 <1408327080.88601.YahooMailNeo@web181006.mail.ne1.yahoo.com>
 <53F1A003.1010001@canterbury.ac.nz>
Message-ID: <53F23136.2060200@stoneleaf.us>

On 08/17/2014 11:41 PM, Greg Ewing wrote:
> So we need a name for a type that's a tuple used
> as a sequence.
>
> How about Tuppence?

Only if we also get Pidgeons!  ;)

--
~Ethan~

From ethan at stoneleaf.us  Mon Aug 18 22:15:36 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 18 Aug 2014 13:15:36 -0700
Subject: [Python-ideas] Optional Static Typing -- the Python Way
Message-ID: <53F25EE8.8000900@stoneleaf.us>

After reading through the OST threads, and learning a bit more about what static typing means, and the difference 
between nominal and structural subclassing, I think I have a better appreciation for the knee-jerk "no way!" reaction, 
and also how to make what seems like a foreign-to-Python concept pythonistic.

The most-feared problem seems to be over-specification (it is certainly mine). I think the answer has been alluded to a 
couple times already, but just to hopefully bring it front and center:

Instead of either 'nominal' or 'structural', use our own 'dynamical' subclassing scheme.

In other words:

   def spamify(current: datetime.date, moved:int) -> bool:
      # ONE_DAY is a one day time delta, previously declared
      days = ONE_DAY * moved
      proposed_date = current + days
      return it_works(proposed_date)

The type checker will not look to see if 'current' is a datetime.date, but rather will check that what datetime.date's 
__add__ works with (datetime.delta) and then check that what is passed in for current has an __add__ that also works 
with datetime.delta.

I think this would be far more valuable than just nominal as it follows duck-typing, and hopefully less work than 
structural as it's only checking the methods and attributes actually used.

On the minus side, there are times when we need *exactly* a datetime.date, so we would need a way to specify exact vs 
dynamic, but dynamic should be the default.

--
~Ethan~

From ncoghlan at gmail.com  Tue Aug 19 15:27:08 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 19 Aug 2014 23:27:08 +1000
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <53F25EE8.8000900@stoneleaf.us>
References: <53F25EE8.8000900@stoneleaf.us>
Message-ID: <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>

On 19 August 2014 06:15, Ethan Furman <ethan at stoneleaf.us> wrote:
> After reading through the OST threads, and learning a bit more about what
> static typing means, and the difference between nominal and structural
> subclassing, I think I have a better appreciation for the knee-jerk "no
> way!" reaction, and also how to make what seems like a foreign-to-Python
> concept pythonistic.
>
> The most-feared problem seems to be over-specification (it is certainly
> mine). I think the answer has been alluded to a couple times already, but
> just to hopefully bring it front and center:

The exact same fear was raised when ABCs were added in PEP 3119 - that
by formalising ABCs, we'd see a plethora of additional isinstance
checks as people tried to make things "fail fast". That fear turned
out to be spurious then, and I think it's spurious now (unless we
allow builtins to be used for type annotations - we shouldn't do that,
as I think it *would* cause problems when developers migrate to Python
from languages like C, Java and C#).

Most Python code should continue to be written without type
annotations - the use of annotations (and type checkers) should be
limited to projects (and organisations) that are large enough for the
additional complexity to be worth the hassle.

Anecdotally, my experience is that the split between proponents of
static and dynamic typing tends to divide along "team size" lines. If
someone thinks a team of 20 working on one project is "a lot of
people", then they're likely to favour dynamic typing for the
additional flexibility it offers. If they think such a team is small,
then they're more likely to want compiler enforced assistance in
ensuring that everyone is playing by the rules of the API.

Myself, I think the kind of structural typechecking offered by "pylint
-E" and the "optional type declarations" proposal starts looking far
more attractive the first time you're stuck at work after hours
debugging a production failure that happened due to a typo in an error
handling path that wasn't covered by your test suite. However, if your
code isn't in the category of "if this doesn't work, my
company/project/organisation stops until it is fixed", then type
assertions are unlikely to be worth the investment needed to write
them in the first place :)

Ultimately, my perspective is that Guido's proposal boils down to
having a nice syntax where:

    def myfunc(a : KindX, b: KindY, c: KindZ):
        ...

is the moral equivalent of:

    def myfunc(a, b, c):
        assert isinstance(a, KindX)
        assert isinstance(b, KindY)
        assert isinstance(c, KindZ)

except done in a way that is more amenable to static analysis by
looking at the compiled AST, rather than actually executing the code.

Calling it "optional static typing" is probably a bad idea, since the
proposal isn't to change the type system itself - that will be just as
dynamic as it always has been. "optional type assertions" is probably
the most accurate, but then people may think it is *actually*
translated to runtime checks as I show above, which it won't be.

"optional type hinting" is probably the most reassuring term that
could be used, while still remaining accurate.

> Instead of either 'nominal' or 'structural', use our own 'dynamical'
> subclassing scheme.
>
> In other words:
>
>   def spamify(current: datetime.date, moved:int) -> bool:
>      # ONE_DAY is a one day time delta, previously declared
>      days = ONE_DAY * moved
>      proposed_date = current + days
>      return it_works(proposed_date)
>
> The type checker will not look to see if 'current' is a datetime.date, but
> rather will check that what datetime.date's __add__ works with
> (datetime.delta) and then check that what is passed in for current has an
> __add__ that also works with datetime.delta.
>
> I think this would be far more valuable than just nominal as it follows
> duck-typing, and hopefully less work than structural as it's only checking
> the methods and attributes actually used.

ABCs already support ducktyping, and explicit registration, etc, etc.
We don't need a completely new type system, we can just file the rough
edges off the categories we have already defined, and fill in some of
the missing pieces (like allowing union types, which I seem to recall
being discussed back in the PEP 3119 time frame, but postponed until
more concrete use cases presented themselves).

We may even need to finally add String and BytesLike ABCs, but we've
been muttering about doing that for years.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Tue Aug 19 19:01:12 2014
From: guido at python.org (Guido van Rossum)
Date: Tue, 19 Aug 2014 10:01:12 -0700
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
Message-ID: <CAP7+vJKJP6Nh42X70VmmFKwrf3XpJX3-JC5R9guNgmW8mk7_oA@mail.gmail.com>

On Tue, Aug 19, 2014 at 6:27 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
[Agreeable musings about team size...]

Ultimately, my perspective is that Guido's proposal boils down to
> having a nice syntax where:
>
>     def myfunc(a : KindX, b: KindY, c: KindZ):
>         ...
>
> is the moral equivalent of:
>
>     def myfunc(a, b, c):
>         assert isinstance(a, KindX)
>         assert isinstance(b, KindY)
>         assert isinstance(c, KindZ)
>

Please no. The asserts affect runtime. The type declarations are for
linting, IDEs and docs. I don't want the difference to be swept under the
rug. I want it to be the key feature of the proposal.


> except done in a way that is more amenable to static analysis by
> looking at the compiled AST, rather than actually executing the code.
>

Ironically, mypy also understands isinstance() checks, using them as a kind
of implied "typecase" (though I don't know if it understands assert -- it
definitely understands "if isinstance(x, int): x = x**2".


> Calling it "optional static typing" is probably a bad idea, since the
> proposal isn't to change the type system itself - that will be just as
> dynamic as it always has been. "optional type assertions" is probably
> the most accurate, but then people may think it is *actually*
> translated to runtime checks as I show above, which it won't be.
>

I don't know what you mean by "accurate".


> "optional type hinting" is probably the most reassuring term that
> could be used, while still remaining accurate.
>

I like that.


>  > Instead of either 'nominal' or 'structural', use our own 'dynamical'
> > subclassing scheme.
> >
> > In other words:
> >
> >   def spamify(current: datetime.date, moved:int) -> bool:
> >      # ONE_DAY is a one day time delta, previously declared
> >      days = ONE_DAY * moved
> >      proposed_date = current + days
> >      return it_works(proposed_date)
> >
> > The type checker will not look to see if 'current' is a datetime.date,
> but
> > rather will check that what datetime.date's __add__ works with
> > (datetime.delta) and then check that what is passed in for current has an
> > __add__ that also works with datetime.delta.
> >
> > I think this would be far more valuable than just nominal as it follows
> > duck-typing, and hopefully less work than structural as it's only
> checking
> > the methods and attributes actually used.
>
> ABCs already support ducktyping, and explicit registration, etc, etc.
> We don't need a completely new type system, we can just file the rough
> edges off the categories we have already defined, and fill in some of
> the missing pieces (like allowing union types, which I seem to recall
> being discussed back in the PEP 3119 time frame, but postponed until
> more concrete use cases presented themselves).
>

Yeah, my response to Ethan's use case is that either he shouldn't bother
with type annotations, or he should probably care enough to also declare
his duck type for datetime.date as such, and an ABC regsitration sounds
perfect for that.


> We may even need to finally add String and BytesLike ABCs, but we've
> been muttering about doing that for years.
>

Mypy has some ideas here.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140819/30dc54e8/attachment.html>

From censi at mit.edu  Tue Aug 19 19:49:37 2014
From: censi at mit.edu (Andrea Censi)
Date: Tue, 19 Aug 2014 13:49:37 -0400
Subject: [Python-ideas] type checking using PyContracts;
 was: Proposal: Use mypy syntax for function annotations
Message-ID: <CAJfDr9400XJMAo-kyxbSau9k6nkjPLKy6A+eSWC+P9TQRW+zyQ@mail.gmail.com>

Dear all,

(Apologies for not replying to the original thread,  I only signed up
to the mailing list today)

I wrote PyContracts[1], which implements type-checking functionality
using decorators for Python 2.7+ and annotations in Python 3.

 [1] http://andreacensi.github.com/contracts/

> def word_count(input: List[str]) -> Dict[str, int]:

Here's that example done using PyContracts and Python 3,
and note the extra constraint '>=1' that PyContract can express:

    from contracts import contract

    @contract
    def word_count(input:'list(str)') -> 'dict(str:int,>=1):
        ...

Here's that example done using PyContracts and Python 2.7, using
a decorator instead of annotations:

    from contracts import contract

    @contract(input='list(str)', returns='dict(str:int,>=1)')
    def word_count(input):
        ...

I would like to make a few points from my experience of writing
PyContracts regarding the general issues that one encounters in adding
type checking to Python.

1) Once one starts to think about it, "types" are very broad and if you
   design a system from scratch, make sure it is extensible. For example,
   you might want to have "positive integer" in addition to "integer"
   ('int', and 'int,>=0' in PyContracts). Also, let the users add
their own types.

2) Dependent types are a must and they must have a simple syntax.
   For example, this is a PyContracts annotation that says the function takes
   a list of strings and returns a list of integers of the same size N:

       @contract(x='list[N](str)', returns='list[N](int)')
       def f(x):
           return list(map(len, x))

   PyContract's convention is that and a,b,c,...,y,z bind to any variables
   and A,B,C,...,Z bind to integer variables. This makes numpy contracts
   very nice looking:

       @contract
       def gray2rgb(gray:'array[HxW](uint8)') -> 'array[HxWx3](uint8)':
         ...

3) There is an argument for using regular strings to describe annotations
   rather than Python constructs such as MyPy. One advantage
   is that the system is backwards compatible. It also makes the code
   much cleaner because you don't need to import the various symbols.

   Here's a simple example in MyPy:

        from typing typevar, Sequence

        T = typevar('T')      # Declare type variable

        def first(seq: Sequence[T]) -> T:
            return seq[0]

   Here's the same example in PyContracts (using Python 3's annotations):

        from contracts import contract

        @contract
        def first(s: 'seq(type(t))') -> 'type(t)':
            return s[0]

   Same example in Python 2.7+ (using decorators):

     from contracts import contract

        @contract(s='seq(type(t))', returns='type(t)')
        def first(s):
            return s[0]

(By the way, after a while, I actually like the decorator version
 better than Python 3's annotations, because you can see very
easily the function's name and parameters, and only if you want
you can then read the function annotations above.)

cheers,
A.

-- 
Andrea Censi | http://censi.mit.edu | "Not all those who wander are lost."
research scientist @ LIDS / Massachusetts Institute of Technology

From apalala at gmail.com  Tue Aug 19 23:18:07 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Tue, 19 Aug 2014 16:48:07 -0430
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
Message-ID: <CAN1YFWuXn1sAp8iVF8_reS86ux_96rscUUxWzfv7ofdErUy6fQ@mail.gmail.com>

On Sun, Aug 17, 2014 at 12:33 AM, Guido van Rossum <guido at python.org> wrote:

>
> (2) For type annotations, should we adopt (roughly) the mypy syntax or the
> alternative proposed by Dave Halter? This uses built-in container notations
> as a shorthand, e.g. {str: int} instead of Dict[str, int]. This also
> touches on the issue of abstract vs. concrete types (e.g. iterable vs.
> list).


Maybe we've been missing that in most programming languages the
sub-language for describing types is separate from the sub-language for
describing algorithms. It is so in Haskell, and in C, and in LISP, and many
others.

It should be OK if similar constructs mean different things in each
sub-language, as the necessary symbol reuse (if one omits APL) come mostly
from the IBM keyboard and ASCII.

Personally, I'd like to type less, as is most often the case with current
Python. Beyond that, I agree it is important that the adopted syntax does
not mislead.

Cheers,


-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140819/2cbfd767/attachment.html>

From tjreedy at udel.edu  Wed Aug 20 00:50:13 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 19 Aug 2014 18:50:13 -0400
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
Message-ID: <lt0kc6$s6n$1@ger.gmane.org>

On 8/19/2014 9:27 AM, Nick Coghlan wrote:

What I like: 'optional type hints' based on a fleshed-out ABC systems 
that collects all ABCs together on one module via import from current 
files. It seems that we have been slowly groping towards this for years.

What I also like: shadow skeleton files (for the stdlib) written by 
people other than core developers, who obviously have the energy to do 
so, having already started with various syntaxes, and who just need a 
standard to combine their efforts.  There should be one set that, as 
much as possible, fully allows duck-typing, which is to say, does not 
reject valid arguments.

What I want to add: the advantage of separate skeleton files, aside from 
parallel development by other people, is that there can be more than one 
skeleton file for a given stdlib module.  Linters typically allow users 
to set local standards by selecting options.  A group could also set 
local standards by restrictive type hints in their local skeletons. A 
realistic example would be sum(it: Iterable[Number]).

-- 
Terry Jan Reedy


From antoine at python.org  Wed Aug 20 01:02:50 2014
From: antoine at python.org (Antoine Pitrou)
Date: Tue, 19 Aug 2014 19:02:50 -0400
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <lt0kc6$s6n$1@ger.gmane.org>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org>
Message-ID: <lt0l2r$5ed$1@ger.gmane.org>

Le 19/08/2014 18:50, Terry Reedy a ?crit :
> On 8/19/2014 9:27 AM, Nick Coghlan wrote:
>
> What I like: 'optional type hints' based on a fleshed-out ABC systems
> that collects all ABCs together on one module via import from current
> files. It seems that we have been slowly groping towards this for years.

Hmm, I've been saying this already, but my intuition is that it's a bad 
idea to conflate *type descriptions* (what this proposal is about) and 
actual *runtime types* (what ABCs are).

Besides, the fact that type descriptions must be parametrable, and 
therefore end up being used as instances, sounds like it kills the idea 
of making them regular (runtime) types as well. Unless we make them 
metaclasses?

Regards

Antoine.



From guido at python.org  Wed Aug 20 01:45:27 2014
From: guido at python.org (Guido van Rossum)
Date: Tue, 19 Aug 2014 16:45:27 -0700
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAN1YFWuXn1sAp8iVF8_reS86ux_96rscUUxWzfv7ofdErUy6fQ@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <CAN1YFWuXn1sAp8iVF8_reS86ux_96rscUUxWzfv7ofdErUy6fQ@mail.gmail.com>
Message-ID: <CAP7+vJJGbkRtRWwLb04mTd+=MhjoZfeKTfp=XVLRBc5chnrW+Q@mail.gmail.com>

On Tue, Aug 19, 2014 at 2:18 PM, Juancarlo A?ez <apalala at gmail.com> wrote:

>
>
> On Sun, Aug 17, 2014 at 12:33 AM, Guido van Rossum <guido at python.org>
> wrote:
>
>>
>> (2) For type annotations, should we adopt (roughly) the mypy syntax or
>> the alternative proposed by Dave Halter? This uses built-in container
>> notations as a shorthand, e.g. {str: int} instead of Dict[str, int]. This
>> also touches on the issue of abstract vs. concrete types (e.g. iterable vs.
>> list).
>
>
> Maybe we've been missing that in most programming languages the
> sub-language for describing types is separate from the sub-language for
> describing algorithms. It is so in Haskell, and in C, and in LISP, and many
> others.
>
> It should be OK if similar constructs mean different things in each
> sub-language, as the necessary symbol reuse (if one omits APL) come mostly
> from the IBM keyboard and ASCII.
>
> Personally, I'd like to type less, as is most often the case with current
> Python. Beyond that, I agree it is important that the adopted syntax does
> not mislead.
>

Actually, Python has a long tradition of reusing the same sub-language for
things that are different sub-languages in other languages. Starting with
'int' being callable, in fact. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140819/b458764f/attachment.html>

From guido at python.org  Wed Aug 20 02:12:54 2014
From: guido at python.org (Guido van Rossum)
Date: Tue, 19 Aug 2014 17:12:54 -0700
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <lt0l2r$5ed$1@ger.gmane.org>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
Message-ID: <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>

On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou <antoine at python.org> wrote:

> Le 19/08/2014 18:50, Terry Reedy a ?crit :
>
>  On 8/19/2014 9:27 AM, Nick Coghlan wrote:
>>
>> What I like: 'optional type hints' based on a fleshed-out ABC systems
>> that collects all ABCs together on one module via import from current
>> files. It seems that we have been slowly groping towards this for years.
>>
>
> Hmm, I've been saying this already, but my intuition is that it's a bad
> idea to conflate *type descriptions* (what this proposal is about) and
> actual *runtime types* (what ABCs are).
>

But are they? I think the registration mechanism makes it clear that they
aren't (necessarily) runtime types -- by linking a concrete type with an
ABC through registration you are pretty clearly stating that the ABC
*describes* (an aspect of) the concrete class without automatically adding
any behavior from the ABC to it.


> Besides, the fact that type descriptions must be parametrable, and
> therefore end up being used as instances, sounds like it kills the idea of
> making them regular (runtime) types as well. Unless we make them
> metaclasses?
>

No, that doesn't follow at all. The concept of metaclasses means that
classes *are* instances (of the metaclass) and it lets us define operations
(in particular __getitem__ :-) on the classes.

(I was going to show a working example, but I ran out of time, and I'm
taking a vacation the rest of this week.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140819/fcfe482f/attachment-0001.html>

From apalala at gmail.com  Wed Aug 20 03:50:20 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Tue, 19 Aug 2014 21:20:20 -0430
Subject: [Python-ideas] Optional static typing -- the crossroads
In-Reply-To: <CAP7+vJJGbkRtRWwLb04mTd+=MhjoZfeKTfp=XVLRBc5chnrW+Q@mail.gmail.com>
References: <CAP7+vJLMet-r8OpsL4O6+LgOkL2m-v_RC_Cnz2qXZP_YnWX8xg@mail.gmail.com>
 <53EFBD98.2090802@stoneleaf.us> <20140817020851.GL4525@ando>
 <CAP7+vJKJjxEEQc2WctohTKf-b3tMN_bBB+fA=tJaLO0dod=Hbw@mail.gmail.com>
 <CAN1YFWuXn1sAp8iVF8_reS86ux_96rscUUxWzfv7ofdErUy6fQ@mail.gmail.com>
 <CAP7+vJJGbkRtRWwLb04mTd+=MhjoZfeKTfp=XVLRBc5chnrW+Q@mail.gmail.com>
Message-ID: <CAN1YFWtmqqPGRake8imXoSz=9sAgSLK-BSBC7TZrAhJaDkiWeQ@mail.gmail.com>

On Tue, Aug 19, 2014 at 7:15 PM, Guido van Rossum <guido at python.org> wrote:

> Actually, Python has a long tradition of reusing the same sub-language for
> things that are different sub-languages in other languages. Starting with
> 'int' being callable, in fact. :-)


I love the consistency of Python's type system. It's intuitive (difficult
to get wrong). Maybe that's why I'm one of the fearful ones regarding this
move towards formalizing type annotations.

BTW, today I took the time to read Mr. C.D. Smith's (old) article about
type systems, and I found it relevant, illustrative, smart, and
entertaining. May I remind the list about it?

I will:

http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/

-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140819/a5682fd2/attachment.html>

From ncoghlan at gmail.com  Wed Aug 20 13:27:34 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 20 Aug 2014 21:27:34 +1000
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
Message-ID: <CADiSq7d0zdZ7x5fgU5tM86HCS8ZCjc4UYP_CBF4L+yirHCnYQg@mail.gmail.com>

On 20 Aug 2014 10:14, "Guido van Rossum" <guido at python.org> wrote:
>
> On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou <antoine at python.org>
wrote:
>>
>> Le 19/08/2014 18:50, Terry Reedy a ?crit :
>>
>>> On 8/19/2014 9:27 AM, Nick Coghlan wrote:
>>>
>>> What I like: 'optional type hints' based on a fleshed-out ABC systems
>>> that collects all ABCs together on one module via import from current
>>> files. It seems that we have been slowly groping towards this for years.
>>
>>
>> Hmm, I've been saying this already, but my intuition is that it's a bad
idea to conflate *type descriptions* (what this proposal is about) and
actual *runtime types* (what ABCs are).
>
>
> But are they? I think the registration mechanism makes it clear that they
aren't (necessarily) runtime types -- by linking a concrete type with an
ABC through registration you are pretty clearly stating that the ABC
*describes* (an aspect of) the concrete class without automatically adding
any behavior from the ABC to it.

One of the ways I describe the explicit registration mechanism to people is
as giving you the option of lying to the interpreter. If you use explicit
registration, your registered type basically walks around with a sign
saying "I'm a duck!" and the interpreter goes "OK, you're a duck!" and
assumes it will be able to quack without actually checking in advance. It's
really just a constrained version of normal duck typing, where the
interpreter just assumes that *everything* may exhibit duck-like tendencies.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/c53cbd20/attachment.html>

From tleeuwenburg at gmail.com  Wed Aug 20 14:08:48 2014
From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg)
Date: Wed, 20 Aug 2014 22:08:48 +1000
Subject: [Python-ideas] Optional static typing -- late to the party
Message-ID: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>

Hi all,

I apologise in advance for being across only that proportion of the
previous correspondence that I could read in about half an hour. Too much
has been said to fully integrate it all.

I have some responses to what was written, and one of those I feel has been
largely missed. Please accept this as "just some stuff I think" and not any
kind of strident criticism or positioning.

The primary goal should be a syntax which enhances human readability. While
I agree that having good static code analysis tools is very useful for the
write --> test --> fix cycle, actually the more bugs you can catch during
the human read/understand/write cycle, the better. The earlier you can
communicate useful information about variable types, the better, and the
earliest stage of that is code comprehension at the human level.

I mostly agree with those who are preferring docstring embedding over
signature decoration, particularly for complex cases, due to the desire to
separate the complexity into structured sections. The signature is like the
heading, and the annotation is like additional detail.

I was led to a conclusion: what is wrong with allowing both? Indeed,
clearly both are actually already supported by various tools. So perhaps
there is actually a higher-level common concept -- what are the actual
assertion which are going to be supported? Can I declare a variable only to
be a list, or can it be a list-of-ints?

Further, is it possible to standardise between some of the
syntax/terminology, and some of the assertion types, such that they are
consistent between function annotation syntaxes and docstring embedding
syntaxes? Could I use function annotation for simple cases, but seamlessly
move to a docstring-embedded approach for a more complex example?

e.g.

def frobnicate(x: int, y: int) -> float
    '''
    Applies some kind of standard function
    '''
    return x**2 / y

is great.

But

def handle_complex_case(name, context, objective, datacube):
   '''
   @param name: str, contains a "Firstname Lastname" string
   @param context: dict, contains a data record
   @param objective: int, contains a coded directive
   @param datacube: numpy.array, contains a field of continuous spatial data

   @return: instructionset: list(callable), a list of callable instructions
   '''

allows me to describe what's going on in that function in a way which
supports both typing and static analysis. Apologies for inventing the
docstring syntax rather than correctly using an existing syntax. However,
my point is that in the first example, an annotation approach is *more*
readable. In the second example, you really want to spend more time
describing and annotating the function.

If there could be some agreement about the details and consistency, I see
no reason that a dual style could not be preferable, and no more complex to
understand and use.

Regards,
-Tennessee Leeuwenburg

-- 
--------------------------------------------------
Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/6c9e8f32/attachment.html>

From tleeuwenburg at gmail.com  Wed Aug 20 14:14:34 2014
From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg)
Date: Wed, 20 Aug 2014 22:14:34 +1000
Subject: [Python-ideas] type checking using PyContracts;
 was: Proposal: Use mypy syntax for function annotations
In-Reply-To: <CAJfDr9400XJMAo-kyxbSau9k6nkjPLKy6A+eSWC+P9TQRW+zyQ@mail.gmail.com>
References: <CAJfDr9400XJMAo-kyxbSau9k6nkjPLKy6A+eSWC+P9TQRW+zyQ@mail.gmail.com>
Message-ID: <CADhgxgcSCSvp4uqJmB2Un89w8qptSyrYdB_kPL1VnkY6V9kfyw@mail.gmail.com>

I like your comments about making the type system easily extensible. I can
well imagine this being used, particularly for data types with an inherent
structure such as tuples, lists, dicts and namedtuples.


On Wed, Aug 20, 2014 at 3:49 AM, Andrea Censi <censi at mit.edu> wrote:

> Dear all,
>
> (Apologies for not replying to the original thread,  I only signed up
> to the mailing list today)
>
> I wrote PyContracts[1], which implements type-checking functionality
> using decorators for Python 2.7+ and annotations in Python 3.
>
>  [1] http://andreacensi.github.com/contracts/
>
> > def word_count(input: List[str]) -> Dict[str, int]:
>
> Here's that example done using PyContracts and Python 3,
> and note the extra constraint '>=1' that PyContract can express:
>
>     from contracts import contract
>
>     @contract
>     def word_count(input:'list(str)') -> 'dict(str:int,>=1):
>         ...
>
> Here's that example done using PyContracts and Python 2.7, using
> a decorator instead of annotations:
>
>     from contracts import contract
>
>     @contract(input='list(str)', returns='dict(str:int,>=1)')
>     def word_count(input):
>         ...
>
> I would like to make a few points from my experience of writing
> PyContracts regarding the general issues that one encounters in adding
> type checking to Python.
>
> 1) Once one starts to think about it, "types" are very broad and if you
>    design a system from scratch, make sure it is extensible. For example,
>    you might want to have "positive integer" in addition to "integer"
>    ('int', and 'int,>=0' in PyContracts). Also, let the users add
> their own types.
>
> 2) Dependent types are a must and they must have a simple syntax.
>    For example, this is a PyContracts annotation that says the function
> takes
>    a list of strings and returns a list of integers of the same size N:
>
>        @contract(x='list[N](str)', returns='list[N](int)')
>        def f(x):
>            return list(map(len, x))
>
>    PyContract's convention is that and a,b,c,...,y,z bind to any variables
>    and A,B,C,...,Z bind to integer variables. This makes numpy contracts
>    very nice looking:
>
>        @contract
>        def gray2rgb(gray:'array[HxW](uint8)') -> 'array[HxWx3](uint8)':
>          ...
>
> 3) There is an argument for using regular strings to describe annotations
>    rather than Python constructs such as MyPy. One advantage
>    is that the system is backwards compatible. It also makes the code
>    much cleaner because you don't need to import the various symbols.
>
>    Here's a simple example in MyPy:
>
>         from typing typevar, Sequence
>
>         T = typevar('T')      # Declare type variable
>
>         def first(seq: Sequence[T]) -> T:
>             return seq[0]
>
>    Here's the same example in PyContracts (using Python 3's annotations):
>
>         from contracts import contract
>
>         @contract
>         def first(s: 'seq(type(t))') -> 'type(t)':
>             return s[0]
>
>    Same example in Python 2.7+ (using decorators):
>
>      from contracts import contract
>
>         @contract(s='seq(type(t))', returns='type(t)')
>         def first(s):
>             return s[0]
>
> (By the way, after a while, I actually like the decorator version
>  better than Python 3's annotations, because you can see very
> easily the function's name and parameters, and only if you want
> you can then read the function annotations above.)
>
> cheers,
> A.
>
> --
> Andrea Censi | http://censi.mit.edu | "Not all those who wander are lost."
> research scientist @ LIDS / Massachusetts Institute of Technology
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--------------------------------------------------
Tennessee Leeuwenburg
http://myownhat.blogspot.com/
"Don't believe everything you think"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/a7342a36/attachment-0001.html>

From steve at pearwood.info  Wed Aug 20 14:35:09 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 20 Aug 2014 22:35:09 +1000
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
Message-ID: <20140820123508.GK25957@ando>

On Wed, Aug 20, 2014 at 10:08:48PM +1000, Tennessee Leeuwenburg wrote:

[...]
> I was led to a conclusion: what is wrong with allowing both? 

"Both" being function annotations and docstring annotations.

Docstring annotations will not suddenly go away if (when) 
Guido's proposal is accepted.

But remember that any annotations in docstrings may not be available at 
runtime, whereas function annotations will be unless you deliberately 
delete them. Also, docstring annotations have the disadvantage of being 
text only, instead of expressions which evaluate to arbitrarily powerful 
or useful objects.

> Indeed,
> clearly both are actually already supported by various tools. So perhaps
> there is actually a higher-level common concept -- what are the actual
> assertion which are going to be supported? Can I declare a variable only to
> be a list, or can it be a list-of-ints?

Yes. Using Guido's suggestion of mypy's syntax:

from typing import List
def spam(x:List)->List[int]:
    ...

declares that spam accepts as argument a list of anything, and 
returns a list of ints. But of course you're more likely to want to 
accept anything which quacks like a list:

from typing import Sequence
def spam(x:Sequence)->List[int]:
    ...


The precise details of the syntax aren't yet finalised, but this is 
suggestive of what it will likely be (I hope).


> Further, is it possible to standardise between some of the
> syntax/terminology, and some of the assertion types, such that they are
> consistent between function annotation syntaxes and docstring embedding
> syntaxes? Could I use function annotation for simple cases, but seamlessly
> move to a docstring-embedded approach for a more complex example?

That entirely depends on the tool you are using for static type analysis 
and/or linting.


-- 
Steven


From ben+python at benfinney.id.au  Wed Aug 20 14:53:24 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 20 Aug 2014 22:53:24 +1000
Subject: [Python-ideas] Optional static typing -- late to the party
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <20140820123508.GK25957@ando>
Message-ID: <85sikr1fwb.fsf@benfinney.id.au>

Steven D'Aprano <steve at pearwood.info> writes:

> But remember that any annotations in docstrings may not be available
> at runtime

How so? What conformant Python implementation is discarding docstrings
from code objects?

> Also, docstring annotations have the disadvantage of being text only,
> instead of expressions which evaluate to arbitrarily powerful or
> useful objects.

True. This is why I'm a fan of reStructuredText for docstrings, and the
conventions that have arisen for using reST field lists for code
annotations <URL:http://sphinx-doc.org/domains.html#info-field-lists>.

The structure in a docstring isn't available without parsing the reST,
but that seems no worse than requiring new syntax and new tools to
process function annotations.

I'm not arguing against function annotations, merely addressing these
claims about docstrings.

-- 
 \     ?The cost of a thing is the amount of what I call life which is |
  `\       required to be exchanged for it, immediately or in the long |
_o__)                                       run.? ?Henry David Thoreau |
Ben Finney


From rosuav at gmail.com  Wed Aug 20 14:56:32 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 20 Aug 2014 22:56:32 +1000
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <85sikr1fwb.fsf@benfinney.id.au>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <20140820123508.GK25957@ando> <85sikr1fwb.fsf@benfinney.id.au>
Message-ID: <CAPTjJmpXZ+LH8SP-nZWN5cmU=BcpdA22Ckw6vmuYsUcmv1W5Gg@mail.gmail.com>

On Wed, Aug 20, 2014 at 10:53 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> Steven D'Aprano <steve at pearwood.info> writes:
>
>> But remember that any annotations in docstrings may not be available
>> at runtime
>
> How so? What conformant Python implementation is discarding docstrings
> from code objects?

CPython with the -OO flag. Or is that non-conformant?

ChrisA

From censi at mit.edu  Wed Aug 20 14:56:38 2014
From: censi at mit.edu (Andrea Censi)
Date: Wed, 20 Aug 2014 08:56:38 -0400
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
Message-ID: <CAJfDr96uTGBR7Nq0SqpYGajySSNUjk0RY-ftEh1wyiAHxRrUiQ@mail.gmail.com>

On Wed, Aug 20, 2014 at 5:08 AM, Tennessee Leeuwenburg
<tleeuwenburg at gmail.com> wrote:
>
> I was led to a conclusion: what is wrong with allowing both? Indeed, clearly
> both are actually already supported by various tools. So perhaps there is
> actually a higher-level common concept -- what are the actual assertion
> which are going to be supported? Can I declare a variable only to be a list,
> or can it be a list-of-ints?


Yes - why not both? or all three ways?

Here's the "nonempty list of positive int" example done
using PyContracts [1] in 3 ways:
1) Using decorators;
2) Using annotations (Python 3 only);
3) Using docstrings.

The type "list[length-type](entry-type)" means a list
where the length satisfies length-type and the entries
satisfy entry-type. So "list[>=1](int,>0)" means
a list of length at least 1 whose entries are positive integers.

1) Using decorators:

    @contract(l='list[>=1](int,>0)', returns='int,>0')
    def mysum(l):
        ...

2) Using annotations:

    @contract
    def mysum(l: 'list[>=1](int,>0)') -> 'int,>0':
        ...

3) Using docstrings, with the :type: and :rtype: tags:

    @contract
    def mysum(l):
        """
            :type l: list[>=1](int,>0)
            :rtype: int,>0
        """
        ...

If using (1) and (2), a docstring like in (3) is automatically generated.

[1]: https://pypi.python.org/pypi/PyContracts


Steven D'Aprano says:

> docstring annotations have the disadvantage of being
text only, instead of expressions which evaluate to arbitrarily powerful
or useful objects.

I disagree, it's actually the other way around: Python's general
purpose syntax is actually cumbersome with respect to a custom
language for types.
Once you have the freedom of using a custom language (e.g. using
PyParsing) then you can create very compact and at the same time
powerful contracts.
For example, expressing a type such as "list[>=1](int,>0)" using a
valid Python expression will take much more space.

And you have the ability of creating special syntax where it makes
sense, for example in Numpy --- look at this definition of matrix
multiplication with compatible dimensions:

   @contract
   def matrix_multiply(a,  b):
        ''' Multiplies two matrices together.

            :param a: The first matrix. Must be a 2D array.
            :type a: array[MxN],M>0,N>0

           :param b: The second matrix. Must be of compatible dimensions.
           :type b: array[NxP],P>0

            :rtype: array[MxP]
       '''
        return numpy.dot(a, b)

If the contract is violated, there will be a nice message saying
exactly what's wrong


Example:

    a = numpy.zeros((2,2))
    b = numpy.zeros((3,2))
    matrix_multiply(a,b)

Exception:

    Breach for argument 'b' to matrix_multiply().
    Expected value for 'N' was: Instance of int: 2
            instead I received: Instance of int: 3
    checking: N                for value: Instance of int: 3
    checking: NxP              for value: Instance of tuple: (3, 2)
    checking: array[NxP]       for value: array['3x2'](float64)
array([[ 0.,  0.],        [ 0.,  0.], ... [clip]
    checking: array[NxP],P>0   for value: array['3x2'](float64)
array([[ 0.,  0.],        [ 0.,  0.], ... [clip]


best,
A.

-- 
Andrea Censi | http://censi.mit.edu | "Not all those who wander are lost."
research scientist @ LIDS / Massachusetts Institute of Technology

From p.f.moore at gmail.com  Wed Aug 20 15:28:55 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 20 Aug 2014 14:28:55 +0100
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
Message-ID: <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>

On 14 August 2014 00:30, Guido van Rossum <guido at python.org> wrote:
> We certainly *could* do that. However, I haven't seen sufficient other uses
> of annotations. If there is only one use for annotations (going forward),
> annotations would be unambiguous. If we allow different types of
> annotations, there would have to be a way to tell whether a particular
> annotation is intended as a type annotation or not. Currently mypy ignores
> all modules that don't import typing.py (using any form of import
> statement), and we could continue this convention. But it would mean that
> something like this would still require the typing import in order to be
> checked by mypy:
>
> import typing
>
> def gcd(int a, int b) -> int:
>     <tralala>

Sorry, I'm slowly going through this thread, so my apologies if this
has been covered later, but it seems to me that projects (and in
particular libraries) that want to target 2.7 as well as 3.x will be
forced to avoid this feature. And avoid any dependencies that use it.

Specifically, the annotation syntax is purely Python 3.x, so without
some form of translation or import hook, 2.7 won't accept it. And
adding a dependency (even if it's only a very lightweight typing.py
plus a hook installed somewhere/somehow) for something that only has
value as part of a developer-level type check doesn't seem like a good
solution.

So, I'd like someone to explain (maybe by pointing me to relevant mypy
docs, I haven't investigated them yet) how I should modify my existing
code that supports 2.7 and 3.x so that it uses the new functionality
*without* extra runtime dependencies that only deliver build/test-time
benefits (that's the problem I always had with
setuptools/pkg_resources, and I'd not like to see it repeated here).

Paul

From p.f.moore at gmail.com  Wed Aug 20 16:06:09 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 20 Aug 2014 15:06:09 +0100
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140814173103.GO4525@ando>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <20140814173103.GO4525@ando>
Message-ID: <CACac1F8XA=8mHVFTgezRoz0x5+J6kC+zATTp2qFmkvdu_PPUpg@mail.gmail.com>

On 14 August 2014 18:31, Steven D'Aprano <steve at pearwood.info> wrote:
> opt-out and use something else:
>
> @use_spam_annotations
> def frobnicate(x: spam, y: eggs)->breakfast:
    ...
Which reminds me. Why is mypy exempt from the general recommendation
to only use annotations on functions that are tagged with a decorator?

Paul.

From brett at python.org  Wed Aug 20 16:11:21 2014
From: brett at python.org (Brett Cannon)
Date: Wed, 20 Aug 2014 14:11:21 +0000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
Message-ID: <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>

On Wed Aug 20 2014 at 9:29:34 AM Paul Moore <p.f.moore at gmail.com> wrote:

> On 14 August 2014 00:30, Guido van Rossum <guido at python.org> wrote:
> > We certainly *could* do that. However, I haven't seen sufficient other
> uses
> > of annotations. If there is only one use for annotations (going forward),
> > annotations would be unambiguous. If we allow different types of
> > annotations, there would have to be a way to tell whether a particular
> > annotation is intended as a type annotation or not. Currently mypy
> ignores
> > all modules that don't import typing.py (using any form of import
> > statement), and we could continue this convention. But it would mean that
> > something like this would still require the typing import in order to be
> > checked by mypy:
> >
> > import typing
> >
> > def gcd(int a, int b) -> int:
> >     <tralala>
>
> Sorry, I'm slowly going through this thread, so my apologies if this
> has been covered later, but it seems to me that projects (and in
> particular libraries) that want to target 2.7 as well as 3.x will be
> forced to avoid this feature. And avoid any dependencies that use it.
>
> Specifically, the annotation syntax is purely Python 3.x, so without
> some form of translation or import hook, 2.7 won't accept it. And
> adding a dependency (even if it's only a very lightweight typing.py
> plus a hook installed somewhere/somehow) for something that only has
> value as part of a developer-level type check doesn't seem like a good
> solution.
>
> So, I'd like someone to explain (maybe by pointing me to relevant mypy
> docs, I haven't investigated them yet) how I should modify my existing
> code that supports 2.7 and 3.x so that it uses the new functionality
> *without* extra runtime dependencies that only deliver build/test-time
> benefits (that's the problem I always had with
> setuptools/pkg_resources, and I'd not like to see it repeated here).
>

I suspect the answer is "you don't". Just like everything else that is
syntactically exclusive to Python 3, it's a perk you get with Python 3-only
code that you simply can't get if you want to maintain
backwards-compatibility.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/4d256229/attachment.html>

From ryan at ryanhiebert.com  Wed Aug 20 16:27:47 2014
From: ryan at ryanhiebert.com (Ryan Hiebert)
Date: Wed, 20 Aug 2014 09:27:47 -0500
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
Message-ID: <51F3C4F3-A6A9-4F97-A287-B3CC7608E130@ryanhiebert.com>


> On Aug 20, 2014, at 8:28 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
> Specifically, the annotation syntax is purely Python 3.x, so without
> some form of translation or import hook, 2.7 won't accept it. And
> adding a dependency (even if it's only a very lightweight typing.py
> plus a hook installed somewhere/somehow) for something that only has
> value as part of a developer-level type check doesn't seem like a good
> solution.
> 
> So, I'd like someone to explain (maybe by pointing me to relevant mypy
> docs, I haven't investigated them yet) how I should modify my existing
> code that supports 2.7 and 3.x so that it uses the new functionality
> *without* extra runtime dependencies that only deliver build/test-time
> benefits (that's the problem I always had with
> setuptools/pkg_resources, and I'd not like to see it repeated here).

As you?ve suggested, using annotations won?t work on Python 2 where there are no annotations. This feature would not available for applications that wish to support Python 2.

However, that stipulation isn?t unique to this feature. If you want be able to run on older versions of the language, you must forgo the features that are available in newer versions of the language.

You can?t use with_statement unless you?re willing to drop Python 2.4 support, you can?t use print_function unless you?re willing to drop 2.5 support, and you can?t use annotations unless you?re willing to drop all Python 2 support. This is the nature of language improvements.

From jean-charles.douet at laposte.net  Wed Aug 20 16:13:38 2014
From: jean-charles.douet at laposte.net (Jean-Charles Douet)
Date: Wed, 20 Aug 2014 16:13:38 +0200
Subject: [Python-ideas] Python-ideas Digest, Vol 93, Issue 118
In-Reply-To: <mailman.73481.1408536876.18129.python-ideas@python.org>
References: <mailman.73481.1408536876.18129.python-ideas@python.org>
Message-ID: <53F4AD12.8070209@laposte.net>

Hi All,

Just a few remarks :
- Here is what is said about literate programming : 
http://www.literateprogramming.com/quotes_sd.html. Isn't that Pythonic ?
- PyContracts, annotations, etc? look like ? literate programming ?. Doc 
is optional, type hint too, why not put the former with the latter ?
- Now about PyContracts: I cannot prevent myself from thinking of it as 
a ? CHECK CONSTRAINT ? in PostgreSQL and co.
- PyContracts aims at describing constraints, including them in 
docstrings. Thus documentation has to be more structured, more formatted 
and more informative about what's going on in the software being 
interpreted by Python automata. Perhaps structured doc becomes a 
solution to the Rice's theorem ?
- Moreover, the ? doctest ? module could be very useful too. Whenever 
you write a docstring, you can have a test too; this module could test 
annotations and contracts too. Why not include it in standard library ?
- Two systems of structured documentation :
     - CWEB : http://www-cs-faculty.stanford.edu/~uno/cweb.html
     - FunnelWeb
- Are there collisions with these formats : ReSt, GRASS GIS' ? 
boilerplating ? in comments, #FIXME and #TODO for the IDEs, etc?

Best regards,
   JCD.

From donald at stufft.io  Wed Aug 20 16:25:31 2014
From: donald at stufft.io (Donald Stufft)
Date: Wed, 20 Aug 2014 10:25:31 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
Message-ID: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>


> On Aug 20, 2014, at 10:11 AM, Brett Cannon <brett at python.org> wrote:
> 
> 
> 
> On Wed Aug 20 2014 at 9:29:34 AM Paul Moore <p.f.moore at gmail.com <mailto:p.f.moore at gmail.com>> wrote:
> On 14 August 2014 00:30, Guido van Rossum <guido at python.org <mailto:guido at python.org>> wrote:
> > We certainly *could* do that. However, I haven't seen sufficient other uses
> > of annotations. If there is only one use for annotations (going forward),
> > annotations would be unambiguous. If we allow different types of
> > annotations, there would have to be a way to tell whether a particular
> > annotation is intended as a type annotation or not. Currently mypy ignores
> > all modules that don't import typing.py (using any form of import
> > statement), and we could continue this convention. But it would mean that
> > something like this would still require the typing import in order to be
> > checked by mypy:
> >
> > import typing
> >
> > def gcd(int a, int b) -> int:
> >     <tralala>
> 
> Sorry, I'm slowly going through this thread, so my apologies if this
> has been covered later, but it seems to me that projects (and in
> particular libraries) that want to target 2.7 as well as 3.x will be
> forced to avoid this feature. And avoid any dependencies that use it.
> 
> Specifically, the annotation syntax is purely Python 3.x, so without
> some form of translation or import hook, 2.7 won't accept it. And
> adding a dependency (even if it's only a very lightweight typing.py
> plus a hook installed somewhere/somehow) for something that only has
> value as part of a developer-level type check doesn't seem like a good
> solution.
> 
> So, I'd like someone to explain (maybe by pointing me to relevant mypy
> docs, I haven't investigated them yet) how I should modify my existing
> code that supports 2.7 and 3.x so that it uses the new functionality
> *without* extra runtime dependencies that only deliver build/test-time
> benefits (that's the problem I always had with
> setuptools/pkg_resources, and I'd not like to see it repeated here).
> 
> I suspect the answer is "you don't". Just like everything else that is syntactically exclusive to Python 3, it's a perk you get with Python 3-only code that you simply can't get if you want to maintain backwards-compatibility. 
> 

mypy does have a codec that will ignore annotations on 2.x. But other than that the answer is you don?t.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/8c0fb229/attachment-0001.html>

From apieum at gmail.com  Wed Aug 20 16:34:06 2014
From: apieum at gmail.com (Gregory Salvan)
Date: Wed, 20 Aug 2014 16:34:06 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
Message-ID: <CAAZsQLDw_ZxRNXXrkNHa=ho5_ydR2XOjDbYZWHR948-mig6oaw@mail.gmail.com>

In mypy FAQ: http://www.mypy-lang.org/faq.html

"All of my code is still in Python 2. What are my options?
 Mypy currently supports Python 3 syntax. Python 2 support is still in
early stages of development. However, Python 2 support will be improving."

I don't know how.
Does someone know if a decorator that add "__annotations__" attribute to
the function can do the job ?

with something like that:


  @mypy(int, int, returns=int)
  def gcd(a, b):
     <tralala>



2014-08-20 16:11 GMT+02:00 Brett Cannon <brett at python.org>:

>
>
> On Wed Aug 20 2014 at 9:29:34 AM Paul Moore <p.f.moore at gmail.com> wrote:
>
>> On 14 August 2014 00:30, Guido van Rossum <guido at python.org> wrote:
>> > We certainly *could* do that. However, I haven't seen sufficient other
>> uses
>> > of annotations. If there is only one use for annotations (going
>> forward),
>> > annotations would be unambiguous. If we allow different types of
>> > annotations, there would have to be a way to tell whether a particular
>> > annotation is intended as a type annotation or not. Currently mypy
>> ignores
>> > all modules that don't import typing.py (using any form of import
>> > statement), and we could continue this convention. But it would mean
>> that
>> > something like this would still require the typing import in order to be
>> > checked by mypy:
>> >
>> > import typing
>> >
>> > def gcd(int a, int b) -> int:
>> >     <tralala>
>>
>> Sorry, I'm slowly going through this thread, so my apologies if this
>> has been covered later, but it seems to me that projects (and in
>> particular libraries) that want to target 2.7 as well as 3.x will be
>> forced to avoid this feature. And avoid any dependencies that use it.
>>
>> Specifically, the annotation syntax is purely Python 3.x, so without
>> some form of translation or import hook, 2.7 won't accept it. And
>> adding a dependency (even if it's only a very lightweight typing.py
>> plus a hook installed somewhere/somehow) for something that only has
>> value as part of a developer-level type check doesn't seem like a good
>> solution.
>>
>> So, I'd like someone to explain (maybe by pointing me to relevant mypy
>> docs, I haven't investigated them yet) how I should modify my existing
>> code that supports 2.7 and 3.x so that it uses the new functionality
>> *without* extra runtime dependencies that only deliver build/test-time
>> benefits (that's the problem I always had with
>> setuptools/pkg_resources, and I'd not like to see it repeated here).
>>
>
> I suspect the answer is "you don't". Just like everything else that is
> syntactically exclusive to Python 3, it's a perk you get with Python 3-only
> code that you simply can't get if you want to maintain
> backwards-compatibility.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/182a9788/attachment.html>

From steve at pearwood.info  Wed Aug 20 16:47:34 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 21 Aug 2014 00:47:34 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CACac1F8XA=8mHVFTgezRoz0x5+J6kC+zATTp2qFmkvdu_PPUpg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <20140814173103.GO4525@ando>
 <CACac1F8XA=8mHVFTgezRoz0x5+J6kC+zATTp2qFmkvdu_PPUpg@mail.gmail.com>
Message-ID: <20140820144734.GL25957@ando>

On Wed, Aug 20, 2014 at 03:06:09PM +0100, Paul Moore wrote:
> On 14 August 2014 18:31, Steven D'Aprano <steve at pearwood.info> wrote:
> > opt-out and use something else:
> >
> > @use_spam_annotations
> > def frobnicate(x: spam, y: eggs)->breakfast:
>     ...
> Which reminds me. Why is mypy exempt from the general recommendation
> to only use annotations on functions that are tagged with a decorator?

That's the whole point of Guido's proposal: he wants to bless, not mypy 
precisely since that's an alternative implementation of Python, but 
mypy's syntax as the standard use of annotations. Since it will be the 
official standard, there's no need to flag it with a decorator, even if 
CPython will not itself do anything with those annotations.

If Guido's suggestion is accepted, any linter, editor, IDE, Python 
compiler, or other static tool will be entitled to assume that 
annotations are type hints, using the mypy-derived syntax, at 
least unless explicitly told not to.



-- 
Steven

From abarnert at yahoo.com  Wed Aug 20 16:55:00 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 20 Aug 2014 07:55:00 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
Message-ID: <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>

On Aug 20, 2014, at 7:25, Donald Stufft <donald at stufft.io> wrote:

> 
>> On Aug 20, 2014, at 10:11 AM, Brett Cannon <brett at python.org> wrote:
>> 
>> 
>> 
>> On Wed Aug 20 2014 at 9:29:34 AM Paul Moore <p.f.moore at gmail.com> wrote:
>>> On 14 August 2014 00:30, Guido van Rossum <guido at python.org> wrote:
>>> > We certainly *could* do that. However, I haven't seen sufficient other uses
>>> > of annotations. If there is only one use for annotations (going forward),
>>> > annotations would be unambiguous. If we allow different types of
>>> > annotations, there would have to be a way to tell whether a particular
>>> > annotation is intended as a type annotation or not. Currently mypy ignores
>>> > all modules that don't import typing.py (using any form of import
>>> > statement), and we could continue this convention. But it would mean that
>>> > something like this would still require the typing import in order to be
>>> > checked by mypy:
>>> >
>>> > import typing
>>> >
>>> > def gcd(int a, int b) -> int:
>>> >     <tralala>
>>> 
>>> Sorry, I'm slowly going through this thread, so my apologies if this
>>> has been covered later, but it seems to me that projects (and in
>>> particular libraries) that want to target 2.7 as well as 3.x will be
>>> forced to avoid this feature. And avoid any dependencies that use it.
>>> 
>>> Specifically, the annotation syntax is purely Python 3.x, so without
>>> some form of translation or import hook, 2.7 won't accept it. And
>>> adding a dependency (even if it's only a very lightweight typing.py
>>> plus a hook installed somewhere/somehow) for something that only has
>>> value as part of a developer-level type check doesn't seem like a good
>>> solution.
>>> 
>>> So, I'd like someone to explain (maybe by pointing me to relevant mypy
>>> docs, I haven't investigated them yet) how I should modify my existing
>>> code that supports 2.7 and 3.x so that it uses the new functionality
>>> *without* extra runtime dependencies that only deliver build/test-time
>>> benefits (that's the problem I always had with
>>> setuptools/pkg_resources, and I'd not like to see it repeated here).
>> 
>> I suspect the answer is "you don't". Just like everything else that is syntactically exclusive to Python 3, it's a perk you get with Python 3-only code that you simply can't get if you want to maintain backwards-compatibility. 

This seems like much more of an issue for libraries than for applications. It's not a big deal for an app to require Python 3.5, but most libraries out there are supporting, e.g., 2.7/3.3+, and I think that will continue for quite some time. You want your library to be useful to people whose apps are in 2.x and who don't yet have a compelling reason to port them, unless you're confident that your library in itself is compelling enough to be worth the effort and risk.

> 
> mypy does have a codec that will ignore annotations on 2.x. But other than that the answer is you don?t.

If the issue here is not wanting to create a new runtime dependency, that should be pretty easy to solve by just applying the codec at build time for 2.x installs. If the codec isn't easy enough to use from setup.py, that shouldn't be too hard to fix.

And maybe the codec (or equivalent functionality) could be added to setuptools, so it doesn't add an extra build-time dependency to most projects.

I realize that at first glance this sounds similar to saying "just run 2to3 or 3to2 at install time", which turned out to be unrealistic for many libraries. But the difference is that this is a very simple, specific transformation that should have no semantic impact on your runtime code, so I don't think it's naive to expect that it would be simple and reliable as a fully automated transformation.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/8664fc42/attachment-0001.html>

From p.f.moore at gmail.com  Wed Aug 20 17:08:45 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 20 Aug 2014 16:08:45 +0100
Subject: [Python-ideas] Annotations (and static typing)
Message-ID: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>

Sigh. I go away for a week and come back to a mega-thread I can never
hope to catch up on :-)

TL; DR; - Although mypy looks interesting, I think it's too soon to
close the door on all other uses of annotations. Let's find a solution
that allows exploration of alternative uses for a while longer.

OK, can I just make some points regarding the static typing thread.
First of all, I have no issue with the idea of static typing, and in
fact I look forward to seeing what benefits it might have (if nothing
else, the pointer to mypy, which I'd never heard of before, is
appreciated). It won't be something I use soon (see below) but that's
fine.

But Guido seems to be saying (on a number of occasions) that nobody is
really using annotations, so he wants to focus on the static typing
use case alone. I think this is a mistake. First of all, I see no
reason why functions using typing annotations could not be introduced
with a decorator[1]. So why insist that this is the *only* use of
annotations, when it's pretty easy to allow others to co-exist?

Also, the "nobody is using annotations" argument? Personally, I know
of a few other uses:

1. Argument parsers - at least 3 have been mentioned in the thread.
2. Structure unpacking - I think there is a library that uses
annotations for this, although I may be wrong.
3. FFI bindings. I know I've seen this discussed, although I can't
find a reference just now.

There are probably other ideas around as well (GUI bindings,
documentation generation, ...) None are particularly mature, and most
are just at the "ideas" stage, but typically the ideas I have seen are
the sort of thing you'd write a library for, and Python 3 only
libraries *really* aren't common yet.

The problem for people wanting to experiment with annotations, is that
they need to be writing Python 3 only code. While Python 3 adoption is
growing rapidly, I suspect that large applications are typically still
focused on either going through, or tidying up after, a 2-3 migration.
And new projects, while they may be developed from the ground up using
Python 3, will typically be using programmers skilled in Python 2, to
whom Python 3 features are not yet an "instinctive" part of the
toolset. Apart from large standalone applications, there are smaller
scripts (which are typically going to be too small to need
programming-in-the-large features like annotations) and libraries
(which really aren't yet in a position to drop Python 2.x totally,
unless they have a fairly small user base).

So I don't see it as compelling that usage of annotations in the wild
is not yet extensive.

Rather than close the door on alternative uses of annotations, can I suggest:

1. By all means bless mypy syntax as the standard static typing
notation - this seems like a good thing.
2. Clarify that static typing annotations should be introduced with a
decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy
declaration in the stdlib, and have a registration protocols for tools
to hook their own implementation into it.[2]
3. Leave the door open for other uses of decorators, at least until
some of the more major libraries drop Python 2.x support completely
(and hence can afford to have a dependency on a Python 3 only module
that uses annotations). See at that stage if annotations take off.
4. If we get to a point where even libraries that *could* use
annotations don't, then revisit the idea of restricting usage to just
type information.

Paul

[1] Also, a decorator could allow a Python 2 compatible form by using
decorator arguments as an alternative to annotations.
[2] I've yet to see a clear explanation of how "a tool using type
annotations" like an linter, editor, IDE or Python compiler would use
them in such a way that precludes decoration as a means of signalling
the annotation semantics.

From guido at python.org  Wed Aug 20 17:53:02 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 20 Aug 2014 08:53:02 -0700
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CADiSq7d0zdZ7x5fgU5tM86HCS8ZCjc4UYP_CBF4L+yirHCnYQg@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <CADiSq7d0zdZ7x5fgU5tM86HCS8ZCjc4UYP_CBF4L+yirHCnYQg@mail.gmail.com>
Message-ID: <CAP7+vJK59UWOPo1v6d57pjFXCebth2Y4+ukQ1OGrfBaZ0Bzs3A@mail.gmail.com>

Was that meant to support Antoine's position (ABCs are not type
descriptions) or mine (they are)?


On Wed, Aug 20, 2014 at 4:27 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
> On 20 Aug 2014 10:14, "Guido van Rossum" <guido at python.org> wrote:
> >
> > On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou <antoine at python.org>
> wrote:
> >>
> >> Le 19/08/2014 18:50, Terry Reedy a ?crit :
> >>
> >>> On 8/19/2014 9:27 AM, Nick Coghlan wrote:
> >>>
> >>> What I like: 'optional type hints' based on a fleshed-out ABC systems
> >>> that collects all ABCs together on one module via import from current
> >>> files. It seems that we have been slowly groping towards this for
> years.
> >>
> >>
> >> Hmm, I've been saying this already, but my intuition is that it's a bad
> idea to conflate *type descriptions* (what this proposal is about) and
> actual *runtime types* (what ABCs are).
> >
> >
> > But are they? I think the registration mechanism makes it clear that
> they aren't (necessarily) runtime types -- by linking a concrete type with
> an ABC through registration you are pretty clearly stating that the ABC
> *describes* (an aspect of) the concrete class without automatically adding
> any behavior from the ABC to it.
>
> One of the ways I describe the explicit registration mechanism to people
> is as giving you the option of lying to the interpreter. If you use
> explicit registration, your registered type basically walks around with a
> sign saying "I'm a duck!" and the interpreter goes "OK, you're a duck!" and
> assumes it will be able to quack without actually checking in advance. It's
> really just a constrained version of normal duck typing, where the
> interpreter just assumes that *everything* may exhibit duck-like tendencies.
>
> Cheers,
> Nick.
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/ec7d294c/attachment.html>

From guido at python.org  Wed Aug 20 18:04:31 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 20 Aug 2014 09:04:31 -0700
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
Message-ID: <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>

I expect that we'll find a way for type and other annotations to coexist,
but I want to correct what I am quoted as having said. I didn't just say
there was little use; I also said that my original intent was to have type
annotations (see the 2004-2005 Artima blog posts referenced in the threads)
and accepted PEP 3107 in the expectation that it would enable 3rd party
experiments to determine the right syntax for type annotations. IMO mypy
fits right into that category. (Admittedly I had a long discussion with the
author at PyCon 2013 where I convinced him to change his syntax. :-)


On Wed, Aug 20, 2014 at 8:08 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> Sigh. I go away for a week and come back to a mega-thread I can never
> hope to catch up on :-)
>
> TL; DR; - Although mypy looks interesting, I think it's too soon to
> close the door on all other uses of annotations. Let's find a solution
> that allows exploration of alternative uses for a while longer.
>
> OK, can I just make some points regarding the static typing thread.
> First of all, I have no issue with the idea of static typing, and in
> fact I look forward to seeing what benefits it might have (if nothing
> else, the pointer to mypy, which I'd never heard of before, is
> appreciated). It won't be something I use soon (see below) but that's
> fine.
>
> But Guido seems to be saying (on a number of occasions) that nobody is
> really using annotations, so he wants to focus on the static typing
> use case alone. I think this is a mistake. First of all, I see no
> reason why functions using typing annotations could not be introduced
> with a decorator[1]. So why insist that this is the *only* use of
> annotations, when it's pretty easy to allow others to co-exist?
>
> Also, the "nobody is using annotations" argument? Personally, I know
> of a few other uses:
>
> 1. Argument parsers - at least 3 have been mentioned in the thread.
> 2. Structure unpacking - I think there is a library that uses
> annotations for this, although I may be wrong.
> 3. FFI bindings. I know I've seen this discussed, although I can't
> find a reference just now.
>
> There are probably other ideas around as well (GUI bindings,
> documentation generation, ...) None are particularly mature, and most
> are just at the "ideas" stage, but typically the ideas I have seen are
> the sort of thing you'd write a library for, and Python 3 only
> libraries *really* aren't common yet.
>
> The problem for people wanting to experiment with annotations, is that
> they need to be writing Python 3 only code. While Python 3 adoption is
> growing rapidly, I suspect that large applications are typically still
> focused on either going through, or tidying up after, a 2-3 migration.
> And new projects, while they may be developed from the ground up using
> Python 3, will typically be using programmers skilled in Python 2, to
> whom Python 3 features are not yet an "instinctive" part of the
> toolset. Apart from large standalone applications, there are smaller
> scripts (which are typically going to be too small to need
> programming-in-the-large features like annotations) and libraries
> (which really aren't yet in a position to drop Python 2.x totally,
> unless they have a fairly small user base).
>
> So I don't see it as compelling that usage of annotations in the wild
> is not yet extensive.
>
> Rather than close the door on alternative uses of annotations, can I
> suggest:
>
> 1. By all means bless mypy syntax as the standard static typing
> notation - this seems like a good thing.
> 2. Clarify that static typing annotations should be introduced with a
> decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy
> declaration in the stdlib, and have a registration protocols for tools
> to hook their own implementation into it.[2]
> 3. Leave the door open for other uses of decorators, at least until
> some of the more major libraries drop Python 2.x support completely
> (and hence can afford to have a dependency on a Python 3 only module
> that uses annotations). See at that stage if annotations take off.
> 4. If we get to a point where even libraries that *could* use
> annotations don't, then revisit the idea of restricting usage to just
> type information.
>
> Paul
>
> [1] Also, a decorator could allow a Python 2 compatible form by using
> decorator arguments as an alternative to annotations.
> [2] I've yet to see a clear explanation of how "a tool using type
> annotations" like an linter, editor, IDE or Python compiler would use
> them in such a way that precludes decoration as a means of signalling
> the annotation semantics.
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/65800ec6/attachment-0001.html>

From burak.arslan at arskom.com.tr  Wed Aug 20 17:58:52 2014
From: burak.arslan at arskom.com.tr (Burak Arslan)
Date: Wed, 20 Aug 2014 18:58:52 +0300
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <53F4C5BC.8080008@arskom.com.tr>


Chiming in as well (first post to python-ideas, fingers crossed!),
because Spyne (I'm the author) will have to interoperate with this
sooner or later :)

To put my opinions in context:
http://spyne.io/docs/2.10/manual/03_types.html


On 08/13/14 22:44, Guido van Rossum wrote:
> [There is no TL;DR other than the subject line. Please read the whole
> thing before replying. I do have an appendix with some motivations for
> adding type annotations at the end.]
>

Not only that, but I went through most of the thread as well :)

>
> To read up on mypy's annotation syntax, please see the mypy-lang.org
> <http://mypy-lang.org> website. Here's just one complete example, to
> give a flavor:
>
>   from typing import List, Dict
>
>   def word_count(input: List[str]) -> Dict[str, int]:
>       result = {}  #type: Dict[str, int]
>       for line in input:
>           for word in line.split():
>               result[word] = result.get(word, 0) + 1
>       return result
>


I couldn't agree more with Dennis Brakhane saying:

> "I would be very sad to see annotations being limited to convey type
> information."

It'd be very tedious for us corporate web-service guys having to specify
things twice to get the advantages of Python's static typing -- whatever
they might end up being.

But it's not just that -- str[120] is a kind of string (of length 120)
the same way list[str] is a kind of list (that only contains strings).
But this gets out of hand fast: Why not say str[6,32,'[A-Fa-f0-9]+'] is
a hex string between lengths 6 and 32?

So my proposition is to have __getitem__ accept **kwargs. That way the
above becomes:

HexString = str[min_length=6, max_length=32, pattern='[A-Fa-f0-9]+']

This would elegantly blend in with duck typing as well.
object[write=callable] is a file-like object. Or maybe I should say
object[write=callable[bytes]] ?

Alternatively, having what I call "specializers" in typing.py could work
just as well:

HexString = str[min_length(6), max_length(32), pattern('[A-Fa-f0-9]+')]
IntToStrMapping = dict[key(int), value(str)]

But it looks hideous and as a bonus, invalid specializations like 
dict[pattern('[a-z0-9]+')]  will have to be dealt with.

Or, it's also possible to do what we're doing with Spyne and leave
builtins like str, list etc. alone and introduce a separate set of
markers to be used exclusively for type annotation purposes:

HexString = Unicode(min_length=6, max_length=32, pattern='[A-Fa-f0-9]+')
IntToStrMapping = Dict(key=int, value=str)

This has the disadvantage of doing generics with parens, which turned
out to be pretty confusing for some new users of Spyne.

While we're into the run-time checked types territory, why make do with
declarative restrictions and not pass arbitrary callables to validate
incoming objects? e.g.

PngData = bytes[validator=ensure_png]

where ensure_png is a callable[bytes] that ensures that the incoming
bytes object is indeed a valid png data.

Now you might think this proposal an abomination in the context of a
community of "consenting adults". But when you let your functions be
called by untrusted sources (which is, I'll argue, pretty much anyone
whose code listens to a socket one way or the other) precisely
expressing and applying such constraints is very very important.

IMO, how this kind of information is passed around needs to be
standardized for the same reasons Python got a standard event loop. We
already have a high amount of largely incompatible systems of varying
levels of sophistication and purposes for enforcing constraints to
incoming data.

I hope all this makes sense.

Best regards,
Burak

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/db3c1f19/attachment.html>

From ethan at stoneleaf.us  Wed Aug 20 18:26:23 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 20 Aug 2014 09:26:23 -0700
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
Message-ID: <53F4CC2F.70308@stoneleaf.us>

On 08/20/2014 09:04 AM, Guido van Rossum wrote:
>
> I expect that we'll find a way for type and other annotations to coexist [...]

Another thing to keep in mind, and which my scription utility will probably switch to, is that we are not stuck only 
with __annotations__ -- we can also add other attributes, such as __scription__ or __plac__ or __my_custom_stuff__ or 
whatever.

It could easily be that coexisting means use a decorator instead of annotations, and store the info in a different 
attribute.  As a bonus this method is even 2.x compatible.

--
~Ethan~

From p.f.moore at gmail.com  Wed Aug 20 18:27:58 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 20 Aug 2014 17:27:58 +0100
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
Message-ID: <CACac1F8pWCA8MBvWgDpeJKJj26AJ3xDUgUQwr15pLB=Qwb=_0A@mail.gmail.com>

On 20 August 2014 17:04, Guido van Rossum <guido at python.org> wrote:
> I expect that we'll find a way for type and other annotations to coexist,
> but I want to correct what I am quoted as having said. I didn't just say
> there was little use; I also said that my original intent was to have type
> annotations (see the 2004-2005 Artima blog posts referenced in the threads)
> and accepted PEP 3107 in the expectation that it would enable 3rd party
> experiments to determine the right syntax for type annotations. IMO mypy
> fits right into that category. (Admittedly I had a long discussion with the
> author at PyCon 2013 where I convinced him to change his syntax. :-)

Sorry, yeah. I glossed over the point about having type annotations
and converging on the right syntax - as I said, I'm interested in
that, but not so much in the short term.

Paul

From edk141 at gmail.com  Wed Aug 20 18:22:13 2014
From: edk141 at gmail.com (Ed Kellett)
Date: Wed, 20 Aug 2014 17:22:13 +0100
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <CABmzr0i+-ecu-oDhApy+OhDA1NnxizGeavSGYW8th=sE1SABnw@mail.gmail.com>

This proposal effectively negates any benefit to adding annotations to Python.

Annotations' only function was to provide unspecified information about
parameters and return values to other Python at runtime. That goes with the
acceptance of this proposal: It's still their sole function, but now we're
*not allowed to use them* to do that any more. Annotations alone are syntax
that corresponds to exactly no semantic meaning, and far from encouraging
developers to assign meaning to them, you're banning them from doing so.

In its current form, Python code can't even make use of the annotations that
are allowed: mypy's type system is not Python's, and without a specification
of how the static type checker's type system actually works, anything looking
at the annotations from the Python side would have to come up with its own way
to interpret them.

You could have picked *any* other way to communicate static typing information,
and it wouldn't neuter a feature so completely:

- use a @typechecked decorator to signify that a decorated function's
  annotations are for use as type information
- use a certain set of annotations to denote type information, without
  deprecating the use of annotations for other purposes
- parse a standardized docstring format for type information
- add new syntax for expressing type information without conflicting with
  annotations: def foo(int bar), for example

Static typing is what it is and I won't try to convince you that it's not
worth adding to the language, but it's not worth rendering a feature useless
when you could *not* render that feature useless and retain all the same
benefits.

Ed Kellett

On 13 August 2014 20:44, Guido van Rossum <guido at python.org> wrote:
> [There is no TL;DR other than the subject line. Please read the whole thing
> before replying. I do have an appendix with some motivations for adding type
> annotations at the end.]
>
> Yesterday afternoon I had an inspiring conversation with Bob Ippolito (man
> of many trades, author of simplejson) and Jukka Lehtosalo (author of mypy:
> http://mypy-lang.org/). Bob gave a talk at EuroPython about what Python can
> learn from Haskell (and other languages); yesterday he gave the same talk at
> Dropbox. The talk is online
> (https://ep2014.europython.eu/en/schedule/sessions/121/) and in broad
> strokes comes down to three suggestions:
>
>   (a) Python should adopt mypy's syntax for function annotations
>   (b) Python's use of mutabe containers by default is wrong
>   (c) Python should adopt some kind of Abstract Data Types
>
> Proposals (b) and (c) don't feel particularly actionable (if you disagree
> please start a new thread, I'd be happy to discuss these further if there's
> interest) but proposal (a) feels right to me.
>
> So what is mypy?  It is a static type checker for Python written by Jukka
> for his Ph.D. thesis. The basic idea is that you add type annotations to
> your program using some custom syntax, and when running your program using
> the mypy interpreter, type errors will be found during compilation (i.e.,
> before the program starts running).
>
> The clever thing here is that the custom syntax is actually valid Python 3,
> using (mostly) function annotations: your annotated program will still run
> with the regular Python 3 interpreter. In the latter case there will be no
> type checking, and no runtime overhead, except to evaluate the function
> annotations (which are evaluated at function definition time but don't have
> any effect when the function is called).
>
> In fact, it is probably more useful to think of mypy as a heavy-duty linter
> than as a compiler or interpreter; leave the type checking to mypy, and the
> execution to Python. It is easy to integrate mypy into a continuous
> integration setup, for example.
>
> To read up on mypy's annotation syntax, please see the mypy-lang.org
> website. Here's just one complete example, to give a flavor:
>
>   from typing import List, Dict
>
>   def word_count(input: List[str]) -> Dict[str, int]:
>       result = {}  #type: Dict[str, int]
>       for line in input:
>           for word in line.split():
>               result[word] = result.get(word, 0) + 1
>       return result
>
> Note that the #type: comment is part of the mypy syntax; mypy uses comments
> to declare types in situations where no syntax is available -- although this
> particular line could also be written as follows:
>
>     result = Dict[str, int]()
>
> Either way the entire function is syntactically valid Python 3, and a
> suitable implementation of typing.py (containing class definitions for List
> and Dict, for example) can be written to make the program run correctly. One
> is provided as part of the mypy project.
>
> I should add that many of mypy's syntactic choices aren't actually new. The
> basis of many of its ideas go back at least a decade: I blogged about this
> topic in 2004 (http://www.artima.com/weblogs/viewpost.jsp?thread=85551 --
> see also the two followup posts linked from the top there).
>
> I'll emphasize once more that mypy's type checking happens in a separate
> pass: no type checking happens at run time (other than what the interpreter
> already does, like raising TypeError on expressions like 1+"1").
>
> There's a lot to this proposal, but I think it's possible to get a PEP
> written, accepted and implemented in time for Python 3.5, if people are
> supportive. I'll go briefly over some of the action items.
>
> (1) A change of direction for function annotations
>
> PEP 3107, which introduced function annotations, is intentional
> non-committal about how function annotations should be used. It lists a
> number of use cases, including but not limited to type checking. It also
> mentions some rejected proposals that would have standardized either a
> syntax for indicating types and/or a way for multiple frameworks to attach
> different annotations to the same function. AFAIK in practice there is
> little use of function annotations in mainstream code, and I propose a
> conscious change of course here by stating that annotations should be used
> to indicate types and to propose a standard notation for them.
>
> (We may have to have some backwards compatibility provision to avoid
> breaking code that currently uses annotations for some other purpose.
> Fortunately the only issue, at least initially, will be that when running
> mypy to type check such code it will produce complaints about the
> annotations; it will not affect how such code is executed by the Python
> interpreter. Nevertheless, it would be good to deprecate such alternative
> uses of annotations.)
>
> (2) A specification for what to add to Python 3.5
>
> There needs to be at least a rough consensus on the syntax for annotations,
> and the syntax must cover a large enough set of use cases to be useful. Mypy
> is still under development, and some of its features are still evolving
> (e.g. unions were only added a few weeks ago). It would be possible to argue
> endlessly about details of the notation, e.g. whether to use 'list' or
> 'List', what either of those means (is a duck-typed list-like type
> acceptable?) or how to declare and use type variables, and what to do with
> functions that have no annotations at all (mypy currently skips those
> completely).
>
> I am proposing that we adopt whatever mypy uses here, keeping discussion of
> the details (mostly) out of the PEP. The goal is to make it possible to add
> type checking annotations to 3rd party modules (and even to the stdlib)
> while allowing unaltered execution of the program by the (unmodified) Python
> 3.5 interpreter. The actual type checker will not be integrated with the
> Python interpreter, and it will not be checked into the CPython repository.
> The only thing that needs to be added to the stdlib is a copy of mypy's
> typing.py module. This module defines several dozen new classes (and a few
> decorators and other helpers) that can be used in expressing argument types.
> If you want to type-check your code you have to download and install mypy
> and run it separately.
>
> The curious thing here is that while standardizing a syntax for type
> annotations, we technically still won't be adopting standard rules for type
> checking. This is intentional. First of all, fully specifying all the type
> checking rules would make for a really long and boring PEP (a much better
> specification would probably be the mypy source code). Second, I think it's
> fine if the type checking algorithm evolves over time, or if variations
> emerge. The worst that can happen is that you consider your code correct but
> mypy disagrees; your code will still run.
>
> That said, I don't want to completely leave out any specification. I want
> the contents of the typing.py module to be specified in the PEP, so that it
> can be used with confidence. But whether mypy will complain about your
> particular form of duck typing doesn't have to be specified by the PEP.
> Perhaps as mypy evolves it will take options to tell it how to handle
> certain edge cases. Forks of mypy (or entirely different implementations of
> type checking based on the same annotation syntax) are also a possibility.
> Maybe in the distant future a version of Python will take a different
> stance, once we have more experience with how this works out in practice,
> but for Python 3.5 I want to restrict the scope of the upheaval.
>
> Appendix -- Why Add Type Annotations?
>
> The argument between proponents of static typing and dynamic typing has been
> going on for many decades. Neither side is all wrong or all right. Python
> has traditionally fallen in the camp of extremely dynamic typing, and this
> has worked well for most users, but there are definitely some areas where
> adding type annotations would help.
>
> - Editors (IDEs) can benefit from type annotations; they can call out
> obvious mistakes (like misspelled method names or inapplicable operations)
> and suggest possible method names. Anyone who has used IntelliJ or Xcode
> will recognize how powerful these features are, and type annotations will
> make such features more useful when editing Python source code.
>
> - Linters are an important tool for teams developing software. A linter
> doesn't replace a unittest, but can find certain types of errors better or
> quicker. The kind of type checking offered by mypy works much like a linter,
> and has similar benefits; but it can find problems that are beyond the
> capabilities of most linters.
>
> - Type annotations are useful for the human reader as well! Take the above
> word_count() example. How long would it have taken you to figure out the
> types of the argument and return value without annotations? Currently most
> people put the types in their docstrings; developing a standard notation for
> type annotations will reduce the amount of documentation that needs to be
> written, and running the type checker might find bugs in the documentation,
> too. Once a standard type annotation syntax is introduced, it should be
> simple to add support for this notation to documentation generators like
> Sphinx.
>
> - Refactoring. Bob's talk has a convincing example of how type annotations
> help in (manually) refactoring code. I also expect that certain automatic
> refactorings will benefit from type annotations -- imagine a tool like 2to3
> (but used for some other transformation) augmented by type annotations, so
> it will know whether e.g. x.keys() is referring to the keys of a dictionary
> or not.
>
> - Optimizers. I believe this is actually the least important application,
> certainly initially. Optimizers like PyPy or Pyston wouldn't be able to
> fully trust the type annotations, and they are better off using their
> current strategy of optimizing code based on the types actually observed at
> run time. But it's certainly feasible to imagine a future optimizer also
> taking type annotations into account.
>
> --
> --Guido "I need a new hobby" van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From p.f.moore at gmail.com  Wed Aug 20 18:31:50 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 20 Aug 2014 17:31:50 +0100
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <53F4CC2F.70308@stoneleaf.us>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
 <53F4CC2F.70308@stoneleaf.us>
Message-ID: <CACac1F8ALXmMEa61TV-WHfLfpuDE5Y9kmGOkMXS13b1nfX0nvg@mail.gmail.com>

On 20 August 2014 17:26, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 08/20/2014 09:04 AM, Guido van Rossum wrote:
>>
>> I expect that we'll find a way for type and other annotations to coexist
>> [...]
>
> Another thing to keep in mind, and which my scription utility will probably
> switch to, is that we are not stuck only with __annotations__ -- we can also
> add other attributes, such as __scription__ or __plac__ or
> __my_custom_stuff__ or whatever.
>
> It could easily be that coexisting means use a decorator instead of
> annotations, and store the info in a different attribute.  As a bonus this
> method is even 2.x compatible.

The obvious thought is, if a decorator is a sufficiently good
notation, is there any need for annotations at all (even for static
typing)?

But that argument does seem to imply that using annotations for
anything *other* than static typing is pointless. (As in, has more
disadvantages than advantages). I wonder if that's the real issue
here?

Paul

From ethan at stoneleaf.us  Wed Aug 20 18:43:03 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 20 Aug 2014 09:43:03 -0700
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <CACac1F8ALXmMEa61TV-WHfLfpuDE5Y9kmGOkMXS13b1nfX0nvg@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
 <53F4CC2F.70308@stoneleaf.us>
 <CACac1F8ALXmMEa61TV-WHfLfpuDE5Y9kmGOkMXS13b1nfX0nvg@mail.gmail.com>
Message-ID: <53F4D017.4040300@stoneleaf.us>

On 08/20/2014 09:31 AM, Paul Moore wrote:
> On 20 August 2014 17:26, Ethan Furman <ethan at stoneleaf.us> wrote:
>> On 08/20/2014 09:04 AM, Guido van Rossum wrote:
>>>
>>> I expect that we'll find a way for type and other annotations to coexist
>>> [...]
>>
>> Another thing to keep in mind, and which my scription utility will probably
>> switch to, is that we are not stuck only with __annotations__ -- we can also
>> add other attributes, such as __scription__ or __plac__ or
>> __my_custom_stuff__ or whatever.
>>
>> It could easily be that coexisting means use a decorator instead of
>> annotations, and store the info in a different attribute.  As a bonus this
>> method is even 2.x compatible.
>
> The obvious thought is, if a decorator is a sufficiently good
> notation, is there any need for annotations at all (even for static
> typing)?

What can one do with an annotation that one cannot do with a decorator?

--
~Ethan~

From alexander.belopolsky at gmail.com  Wed Aug 20 18:51:39 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 20 Aug 2014 12:51:39 -0400
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <53F4D017.4040300@stoneleaf.us>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
 <53F4CC2F.70308@stoneleaf.us>
 <CACac1F8ALXmMEa61TV-WHfLfpuDE5Y9kmGOkMXS13b1nfX0nvg@mail.gmail.com>
 <53F4D017.4040300@stoneleaf.us>
Message-ID: <CAP7h-xba9x8MB77mcWU_gGP7aRrAJiQ+UGtWcz25US=15abTiw@mail.gmail.com>

On Wed, Aug 20, 2014 at 12:43 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> What can one do with an annotation that one cannot do with a decorator?


Spell "returns" in two characters.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/8e8e9c3c/attachment.html>

From mertz at gnosis.cx  Wed Aug 20 19:23:02 2014
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 20 Aug 2014 10:23:02 -0700
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <CAJfDr96uTGBR7Nq0SqpYGajySSNUjk0RY-ftEh1wyiAHxRrUiQ@mail.gmail.com>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <CAJfDr96uTGBR7Nq0SqpYGajySSNUjk0RY-ftEh1wyiAHxRrUiQ@mail.gmail.com>
Message-ID: <CAEbHw4b4zRVqM-EABkKnj8oenZM3znYiMvvpQfYG2u6oCND1xA@mail.gmail.com>

I have to say that I find the capabilities in PyContracts--which are absent
currently in mypy--to be very compelling.  If we are going to add
encouragement to use type annotations, adding richer types seems extremely
worthwhile.

I do not see in any of the discussion a way that mypy/typing.py could
express a concept like "A list/sequence of at least N elements, each of
which has some property (as well as a basic data type)."  In fact, I have
trouble imagining how a native syntax could do this, rather than using a
custom DSL like PyContracts uses.  Well, we could do it by sticking lambdas
in for properties, but that would get very ugly quickly.


2) Using annotations:
>
>     @contract
>     def mysum(l: 'list[>=1](int,>0)') -> 'int,>0':
>         ...
>
> 3) Using docstrings, with the :type: and :rtype: tags:
>
>     @contract
>     def mysum(l):
>         """
>             :type l: list[>=1](int,>0)
>             :rtype: int,>0
>         """
>

I don't really care about the annotations vs. docstring issue much.  But
incorporating pre/post-conditions as part of typing makes a whole lot of
sense to me.


>    @contract
>    def matrix_multiply(a,  b):
>         ''' Multiplies two matrices together.
>
>             :param a: The first matrix. Must be a 2D array.
>             :type a: array[MxN],M>0,N>0
>
>            :param b: The second matrix. Must be of compatible dimensions.
>            :type b: array[NxP],P>0
>
>             :rtype: array[MxP]
>        '''
>         return numpy.dot(a, b)
>

This is also quite lovely and powerful (and not doable in mypy), as is the
error report produced.


> Example:
>
>     a = numpy.zeros((2,2))
>     b = numpy.zeros((3,2))
>     matrix_multiply(a,b)
>
> Exception:
>
>     Breach for argument 'b' to matrix_multiply().
>     Expected value for 'N' was: Instance of int: 2
>             instead I received: Instance of int: 3
>     checking: N                for value: Instance of int: 3
>     checking: NxP              for value: Instance of tuple: (3, 2)
>     checking: array[NxP]       for value: array['3x2'](float64)
> array([[ 0.,  0.],        [ 0.,  0.], ... [clip]
>     checking: array[NxP],P>0   for value: array['3x2'](float64)
> array([[ 0.,  0.],        [ 0.,  0.], ... [clip]
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/30600ce1/attachment.html>

From censi at mit.edu  Wed Aug 20 19:37:21 2014
From: censi at mit.edu (Andrea Censi)
Date: Wed, 20 Aug 2014 13:37:21 -0400
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <CAP7h-xba9x8MB77mcWU_gGP7aRrAJiQ+UGtWcz25US=15abTiw@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
 <53F4CC2F.70308@stoneleaf.us>
 <CACac1F8ALXmMEa61TV-WHfLfpuDE5Y9kmGOkMXS13b1nfX0nvg@mail.gmail.com>
 <53F4D017.4040300@stoneleaf.us>
 <CAP7h-xba9x8MB77mcWU_gGP7aRrAJiQ+UGtWcz25US=15abTiw@mail.gmail.com>
Message-ID: <CAJfDr96yuwyhvFuixg-VJHk=QaJtyDhHQr4KaX1X_B5tC9QkTg@mail.gmail.com>

On Wed, Aug 20, 2014 at 12:51 PM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
>> What can one do with an annotation that one cannot do with a decorator?
>
>
> Spell "returns" in two characters.

True. (Is there any alternative to using a "returns" argument in the decorator?)

We can also compile another list:

What can one do with a decorator that cannot be done with annotations?

1)  Comment out type checking with one character.

(Admittedly, this is thinking short term --- it's a limitation of the
editors rather than the language itself. We can imagine that
language-aware editors of the future will be more flexible. For
example, the editor would have a shortcut to hide or disable type
annotations.)

2) Easily parse (with your eyes) the function signature without having
to think about types.

...

A.

From mertz at gnosis.cx  Wed Aug 20 19:50:18 2014
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 20 Aug 2014 10:50:18 -0700
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <CACwMPm9Qmipvhc7MUGdqxNaPnrWwVMdKYscYi2udyKN0g1QR1w@mail.gmail.com>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <CAJfDr96uTGBR7Nq0SqpYGajySSNUjk0RY-ftEh1wyiAHxRrUiQ@mail.gmail.com>
 <CAEbHw4b4zRVqM-EABkKnj8oenZM3znYiMvvpQfYG2u6oCND1xA@mail.gmail.com>
 <CACwMPm9Qmipvhc7MUGdqxNaPnrWwVMdKYscYi2udyKN0g1QR1w@mail.gmail.com>
Message-ID: <CAEbHw4ZGWP-xo-M83xY=N9M=LJkryaSCwWUCX3uTnzZEjW5Kcg@mail.gmail.com>

Hi Bob,

I enjoyed watching your talk (on video, not live), and I certainly see the
appeal of a Haskell-like type system.  However, what we seem to be
discussing here with mypy annotations looks a lot closer to a C type system
than to a Haskell type system.  All those things that PyContracts get us
are easy enough in Haskell--why aim so low if we are thinking of a change
in Python?


On Wed, Aug 20, 2014 at 10:42 AM, Bob Ippolito <bob at redivi.com> wrote:

> On Wed, Aug 20, 2014 at 10:23 AM, David Mertz <mertz at gnosis.cx> wrote:
>
>> I have to say that I find the capabilities in PyContracts--which are
>> absent currently in mypy--to be very compelling.  If we are going to add
>> encouragement to use type annotations, adding richer types seems extremely
>> worthwhile.
>>
>
> It's easy to check any expressible condition at runtime, but this is not
> the case ahead of time as with a static analysis tool or linter. It'd be
> great to have this sort of capability in mypy, but what you're asking for
> is at the fringes of current research and will not be practical to add to
> Python any time soon.
>
> -bob
>
>


-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/af0274d4/attachment-0001.html>

From bob at redivi.com  Wed Aug 20 19:42:55 2014
From: bob at redivi.com (Bob Ippolito)
Date: Wed, 20 Aug 2014 10:42:55 -0700
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <CAEbHw4b4zRVqM-EABkKnj8oenZM3znYiMvvpQfYG2u6oCND1xA@mail.gmail.com>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <CAJfDr96uTGBR7Nq0SqpYGajySSNUjk0RY-ftEh1wyiAHxRrUiQ@mail.gmail.com>
 <CAEbHw4b4zRVqM-EABkKnj8oenZM3znYiMvvpQfYG2u6oCND1xA@mail.gmail.com>
Message-ID: <CACwMPm9Qmipvhc7MUGdqxNaPnrWwVMdKYscYi2udyKN0g1QR1w@mail.gmail.com>

On Wed, Aug 20, 2014 at 10:23 AM, David Mertz <mertz at gnosis.cx> wrote:

> I have to say that I find the capabilities in PyContracts--which are
> absent currently in mypy--to be very compelling.  If we are going to add
> encouragement to use type annotations, adding richer types seems extremely
> worthwhile.
>

It's easy to check any expressible condition at runtime, but this is not
the case ahead of time as with a static analysis tool or linter. It'd be
great to have this sort of capability in mypy, but what you're asking for
is at the fringes of current research and will not be practical to add to
Python any time soon.

-bob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/37d5ef85/attachment.html>

From p.f.moore at gmail.com  Wed Aug 20 20:13:06 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 20 Aug 2014 19:13:06 +0100
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <CAJfDr96yuwyhvFuixg-VJHk=QaJtyDhHQr4KaX1X_B5tC9QkTg@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
 <53F4CC2F.70308@stoneleaf.us>
 <CACac1F8ALXmMEa61TV-WHfLfpuDE5Y9kmGOkMXS13b1nfX0nvg@mail.gmail.com>
 <53F4D017.4040300@stoneleaf.us>
 <CAP7h-xba9x8MB77mcWU_gGP7aRrAJiQ+UGtWcz25US=15abTiw@mail.gmail.com>
 <CAJfDr96yuwyhvFuixg-VJHk=QaJtyDhHQr4KaX1X_B5tC9QkTg@mail.gmail.com>
Message-ID: <CACac1F_PuRh=SkheJU_V2z72DF+Du_GJTcwAsKt2BmRjaXnOTQ@mail.gmail.com>

On 20 August 2014 18:37, Andrea Censi <censi at mit.edu> wrote:
> What can one do with a decorator that cannot be done with annotations?

Python 2 compatibility.

Paul

PS This discussion is leading me to feel that annotations aren't
really that useful compared to a well-written decorator, so why *not*
leave them specifically for static type information, as Guido
suggested...

From bob at redivi.com  Wed Aug 20 20:14:31 2014
From: bob at redivi.com (Bob Ippolito)
Date: Wed, 20 Aug 2014 11:14:31 -0700
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <CAEbHw4ZGWP-xo-M83xY=N9M=LJkryaSCwWUCX3uTnzZEjW5Kcg@mail.gmail.com>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <CAJfDr96uTGBR7Nq0SqpYGajySSNUjk0RY-ftEh1wyiAHxRrUiQ@mail.gmail.com>
 <CAEbHw4b4zRVqM-EABkKnj8oenZM3znYiMvvpQfYG2u6oCND1xA@mail.gmail.com>
 <CACwMPm9Qmipvhc7MUGdqxNaPnrWwVMdKYscYi2udyKN0g1QR1w@mail.gmail.com>
 <CAEbHw4ZGWP-xo-M83xY=N9M=LJkryaSCwWUCX3uTnzZEjW5Kcg@mail.gmail.com>
Message-ID: <CACwMPm8BCHd89BA5=Z6Hxwx6WAbr79Vyeo+KABvTwEJFpiTgqA@mail.gmail.com>

There's the misunderstanding: PyContracts style contracts are not "easy
enough" in Haskell.

mypy's types are not like a C type system. There are a few missing features
that are being worked on, but it is much better than some here perceive it
to be.

On Wednesday, August 20, 2014, David Mertz <mertz at gnosis.cx> wrote:

> Hi Bob,
>
> I enjoyed watching your talk (on video, not live), and I certainly see the
> appeal of a Haskell-like type system.  However, what we seem to be
> discussing here with mypy annotations looks a lot closer to a C type system
> than to a Haskell type system.  All those things that PyContracts get us
> are easy enough in Haskell--why aim so low if we are thinking of a change
> in Python?
>
>
> On Wed, Aug 20, 2014 at 10:42 AM, Bob Ippolito <bob at redivi.com
> <javascript:_e(%7B%7D,'cvml','bob at redivi.com');>> wrote:
>
>> On Wed, Aug 20, 2014 at 10:23 AM, David Mertz <mertz at gnosis.cx
>> <javascript:_e(%7B%7D,'cvml','mertz at gnosis.cx');>> wrote:
>>
>>> I have to say that I find the capabilities in PyContracts--which are
>>> absent currently in mypy--to be very compelling.  If we are going to add
>>> encouragement to use type annotations, adding richer types seems extremely
>>> worthwhile.
>>>
>>
>> It's easy to check any expressible condition at runtime, but this is not
>> the case ahead of time as with a static analysis tool or linter. It'd be
>> great to have this sort of capability in mypy, but what you're asking for
>> is at the fringes of current research and will not be practical to add to
>> Python any time soon.
>>
>> -bob
>>
>>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/a13ce0ec/attachment.html>

From mertz at gnosis.cx  Wed Aug 20 20:38:46 2014
From: mertz at gnosis.cx (David Mertz)
Date: Wed, 20 Aug 2014 11:38:46 -0700
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <CACwMPm8BCHd89BA5=Z6Hxwx6WAbr79Vyeo+KABvTwEJFpiTgqA@mail.gmail.com>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <CAJfDr96uTGBR7Nq0SqpYGajySSNUjk0RY-ftEh1wyiAHxRrUiQ@mail.gmail.com>
 <CAEbHw4b4zRVqM-EABkKnj8oenZM3znYiMvvpQfYG2u6oCND1xA@mail.gmail.com>
 <CACwMPm9Qmipvhc7MUGdqxNaPnrWwVMdKYscYi2udyKN0g1QR1w@mail.gmail.com>
 <CAEbHw4ZGWP-xo-M83xY=N9M=LJkryaSCwWUCX3uTnzZEjW5Kcg@mail.gmail.com>
 <CACwMPm8BCHd89BA5=Z6Hxwx6WAbr79Vyeo+KABvTwEJFpiTgqA@mail.gmail.com>
Message-ID: <CAEbHw4a0gDaRGX0oh6hCO-fk_iO1azjGBEo8XuBcSxRuo55=XA@mail.gmail.com>

On Wed, Aug 20, 2014 at 11:14 AM, Bob Ippolito <bob at redivi.com> wrote:
> There's the misunderstanding: PyContracts style contracts are not "easy
enough" in Haskell.

Well, for example, I found something like this for Haskell:

newtype Digit = Digit { digitVal :: Int }
  deriving (Eq, Ord, Show)
mkDigit :: Int -> Maybe Digit
mkDigit n
  | n >= 0 && n < 10 = Just (Digit n)
  | otherwise = Nothing


OK, sure it's not just one line.  But now we have a predicate-restricted
data type right in the type system.  If we introduce a static type system
in Python, I'd like it to be able to do that.  PyContracts--or something
close to it--lets us do that in a DSL.  But I would be equally happy to use
some Python code that could be tucked away but reused.  E.g., completely
hypothetical syntax:

class Digit(mypy.Int):

    def __predicate__(self):

        return 0 <= self < 10

def times_modulo(incr: Digit, num: Int) -> Digit:

    result = 0

    for i in range(num):

        result += incr

        result %= 10

    return result


I could just stick Digit and all my other custom types in 'mytypes.py', and
the actual annotations would have a richer type system, plus remain
readable (if I chose decent names for the types).

> mypy's types are not like a C type system. There are a few missing
features that are being worked on, but it is much better than some here
perceive it to be.
>
>
> On Wednesday, August 20, 2014, David Mertz <mertz at gnosis.cx> wrote:
>>
>> Hi Bob,
>>
>> I enjoyed watching your talk (on video, not live), and I certainly see
the appeal of a Haskell-like type system.  However, what we seem to be
discussing here with mypy annotations looks a lot closer to a C type system
than to a Haskell type system.  All those things that PyContracts get us
are easy enough in Haskell--why aim so low if we are thinking of a change
in Python?
>>
>>
>> On Wed, Aug 20, 2014 at 10:42 AM, Bob Ippolito <bob at redivi.com> wrote:
>>>
>>> On Wed, Aug 20, 2014 at 10:23 AM, David Mertz <mertz at gnosis.cx> wrote:
>>>>
>>>> I have to say that I find the capabilities in PyContracts--which are
absent currently in mypy--to be very compelling.  If we are going to add
encouragement to use type annotations, adding richer types seems extremely
worthwhile.
>>>
>>>
>>> It's easy to check any expressible condition at runtime, but this is
not the case ahead of time as with a static analysis tool or linter. It'd
be great to have this sort of capability in mypy, but what you're asking
for is at the fringes of current research and will not be practical to add
to Python any time soon.
>>>
>>> -bob
>>>
>>
>>
>>
>> --
>> Keeping medicines from the bloodstreams of the sick; food
>> from the bellies of the hungry; books from the hands of the
>> uneducated; technology from the underdeveloped; and putting
>> advocates of freedom in prisons.  Intellectual property is
>> to the 21st century what the slave trade was to the 16th.




--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/d3976c35/attachment-0001.html>

From alexander.belopolsky at gmail.com  Wed Aug 20 20:44:59 2014
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Wed, 20 Aug 2014 14:44:59 -0400
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <CACac1F_PuRh=SkheJU_V2z72DF+Du_GJTcwAsKt2BmRjaXnOTQ@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
 <53F4CC2F.70308@stoneleaf.us>
 <CACac1F8ALXmMEa61TV-WHfLfpuDE5Y9kmGOkMXS13b1nfX0nvg@mail.gmail.com>
 <53F4D017.4040300@stoneleaf.us>
 <CAP7h-xba9x8MB77mcWU_gGP7aRrAJiQ+UGtWcz25US=15abTiw@mail.gmail.com>
 <CAJfDr96yuwyhvFuixg-VJHk=QaJtyDhHQr4KaX1X_B5tC9QkTg@mail.gmail.com>
 <CACac1F_PuRh=SkheJU_V2z72DF+Du_GJTcwAsKt2BmRjaXnOTQ@mail.gmail.com>
Message-ID: <CAP7h-xZfWXL8-OCps3rxObpQuxrAXrY09uDAFoZLN0d3u8aiJg@mail.gmail.com>

On Wed, Aug 20, 2014 at 2:13 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 20 August 2014 18:37, Andrea Censi <censi at mit.edu> wrote:
> > What can one do with a decorator that cannot be done with annotations?
>
> Python 2 compatibility.


Out of the way mark-up.  Compare


@types(int, float, returns=float)
def foo(x, y):
  ..

to

def foo(x:int, y:float) -> float:
  ..

The decorator variant is visually closer to plain unannotated Python.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/7003dce3/attachment.html>

From kaiser.yann at gmail.com  Wed Aug 20 21:02:32 2014
From: kaiser.yann at gmail.com (Yann Kaiser)
Date: Wed, 20 Aug 2014 21:02:32 +0200
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <CAP7h-xZfWXL8-OCps3rxObpQuxrAXrY09uDAFoZLN0d3u8aiJg@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
 <53F4CC2F.70308@stoneleaf.us>
 <CACac1F8ALXmMEa61TV-WHfLfpuDE5Y9kmGOkMXS13b1nfX0nvg@mail.gmail.com>
 <53F4D017.4040300@stoneleaf.us>
 <CAP7h-xba9x8MB77mcWU_gGP7aRrAJiQ+UGtWcz25US=15abTiw@mail.gmail.com>
 <CAJfDr96yuwyhvFuixg-VJHk=QaJtyDhHQr4KaX1X_B5tC9QkTg@mail.gmail.com>
 <CACac1F_PuRh=SkheJU_V2z72DF+Du_GJTcwAsKt2BmRjaXnOTQ@mail.gmail.com>
 <CAP7h-xZfWXL8-OCps3rxObpQuxrAXrY09uDAFoZLN0d3u8aiJg@mail.gmail.com>
Message-ID: <CANUJvPXX0E4HBi1C8F4dnc9fgmR_qX+bQrFb-MUQYEFLQv6yZQ@mail.gmail.com>

Let's add convenient namespacing to that list:

@annotate('typing', one=str, two=int, three=int)
@annotate('cli', two=int, three='t')
def func(one, two, *, three=5):
    ...

I might add something like this to sigtools.modifiers.annotate and
.specifiers.signature to extract it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/a92f44de/attachment.html>

From tjreedy at udel.edu  Wed Aug 20 21:21:38 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Aug 2014 15:21:38 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
Message-ID: <lt2sh5$9vi$1@ger.gmane.org>

On 8/20/2014 10:55 AM, Andrew Barnert wrote:

> This seems like much more of an issue for libraries than for
> applications. It's not a big deal for an app to require Python 3.5, but
> most libraries out there are supporting, e.g., 2.7/3.3+, and I think
> that will continue for quite some time. You want your library to be
> useful to people whose apps are in 2.x and who don't yet have a
> compelling reason to port them, unless you're confident that your
> library in itself is compelling enough to be worth the effort and risk.

If the annotation for a library are in a separate skeleton file (which 
would only run on 3.5+), then users of the library (including the 
library itself), would not see the annotations unless they run on 3.5+ 
and look for the annotation file.

-- 
Terry Jan Reedy


From ethan at stoneleaf.us  Wed Aug 20 21:37:55 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 20 Aug 2014 12:37:55 -0700
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <CANUJvPXX0E4HBi1C8F4dnc9fgmR_qX+bQrFb-MUQYEFLQv6yZQ@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
 <53F4CC2F.70308@stoneleaf.us>
 <CACac1F8ALXmMEa61TV-WHfLfpuDE5Y9kmGOkMXS13b1nfX0nvg@mail.gmail.com>
 <53F4D017.4040300@stoneleaf.us>
 <CAP7h-xba9x8MB77mcWU_gGP7aRrAJiQ+UGtWcz25US=15abTiw@mail.gmail.com>
 <CAJfDr96yuwyhvFuixg-VJHk=QaJtyDhHQr4KaX1X_B5tC9QkTg@mail.gmail.com>
 <CACac1F_PuRh=SkheJU_V2z72DF+Du_GJTcwAsKt2BmRjaXnOTQ@mail.gmail.com>
 <CAP7h-xZfWXL8-OCps3rxObpQuxrAXrY09uDAFoZLN0d3u8aiJg@mail.gmail.com>
 <CANUJvPXX0E4HBi1C8F4dnc9fgmR_qX+bQrFb-MUQYEFLQv6yZQ@mail.gmail.com>
Message-ID: <53F4F913.9030807@stoneleaf.us>

On 08/20/2014 12:02 PM, Yann Kaiser wrote:
> Let's add convenient namespacing to that list:
>
> @annotate('typing', one=str, two=int, three=int)
> @annotate('cli', two=int, three='t')
> def func(one, two, *, three=5):
>      ...

For what I was thinking, the above decorator would store the different annotations in, for example, func._typing and 
func._cli.

--
~Ethan~

From tjreedy at udel.edu  Wed Aug 20 21:40:29 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Aug 2014 15:40:29 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <51F3C4F3-A6A9-4F97-A287-B3CC7608E130@ryanhiebert.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <51F3C4F3-A6A9-4F97-A287-B3CC7608E130@ryanhiebert.com>
Message-ID: <lt2tkf$opv$1@ger.gmane.org>

On 8/20/2014 10:27 AM, Ryan Hiebert wrote:
>
>> On Aug 20, 2014, at 8:28 AM, Paul Moore <p.f.moore at gmail.com>
>> wrote:
>>
>> Specifically, the annotation syntax is purely Python 3.x, so
>> without some form of translation or import hook, 2.7 won't accept
>> it. And adding a dependency (even if it's only a very lightweight
>> typing.py plus a hook installed somewhere/somehow) for something
>> that only has value as part of a developer-level type check doesn't
>> seem like a good solution.
>>
>> So, I'd like someone to explain (maybe by pointing me to relevant
>> mypy docs, I haven't investigated them yet) how I should modify my
>> existing code that supports 2.7 and 3.x so that it uses the new
>> functionality *without* extra runtime dependencies that only
>> deliver build/test-time benefits (that's the problem I always had
>> with setuptools/pkg_resources, and I'd not like to see it repeated
>> here).
>
> As you?ve suggested, using annotations won?t work on Python 2 where
> there are no annotations.

Correct.

> This feature would not available for applications that wish to support Python 2.

Not exactly true, which is to say, true for run-time use, not true for 
pre-compile call checking.  Guido mostly wants type hints for the 
latter. Call checking mostly just needs to be done once (on 3.5+).

To expand on what I said elsewhere: suppose a 2&3 app A uses 2&3 library 
L, which has a separate annotation file.  You run the call checks on 
3.5.  Assuming that the 2&3 code of each use bytes and unicode and not 
str, calls should be good on all versions.  (They will be unless the 
same code produces objects of different types on different versions.)

-- 
Terry Jan Reedy



From ahammel87 at gmail.com  Wed Aug 20 21:42:42 2014
From: ahammel87 at gmail.com (Alex Hammel)
Date: Wed, 20 Aug 2014 12:42:42 -0700
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <CAEbHw4a0gDaRGX0oh6hCO-fk_iO1azjGBEo8XuBcSxRuo55=XA@mail.gmail.com>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <CAJfDr96uTGBR7Nq0SqpYGajySSNUjk0RY-ftEh1wyiAHxRrUiQ@mail.gmail.com>
 <CAEbHw4b4zRVqM-EABkKnj8oenZM3znYiMvvpQfYG2u6oCND1xA@mail.gmail.com>
 <CACwMPm9Qmipvhc7MUGdqxNaPnrWwVMdKYscYi2udyKN0g1QR1w@mail.gmail.com>
 <CAEbHw4ZGWP-xo-M83xY=N9M=LJkryaSCwWUCX3uTnzZEjW5Kcg@mail.gmail.com>
 <CACwMPm8BCHd89BA5=Z6Hxwx6WAbr79Vyeo+KABvTwEJFpiTgqA@mail.gmail.com>
 <CAEbHw4a0gDaRGX0oh6hCO-fk_iO1azjGBEo8XuBcSxRuo55=XA@mail.gmail.com>
Message-ID: <CA+_xFerJnXrF_J-ypfKPrhC7UojbN2+GB8kgaD0dhmLhgh9QTA@mail.gmail.com>

The problem is that the proposal is to use mypy syntax for *static* type
checking. It's really hard to prove that values of Digit never go below
zero without doing checks at runtime.

Your Haskell example doesn't do that either, actually. Sure, the value of
Digit can never be a negative number, but neither is it always a positive
number (it might be Nothing).

Say I've got a function that requires a non-negative number to work
properly, so I specify that it takes one of your Digits. If I mess up and
pass it a negative number, the Digit class converts it to Nothing,
resulting in a runtime error (or some exceptional runtime behaviour,
depending on what I do with that Nothing). What I really want if for the
compiler to say "sorry, I'm not compiling this because I can't prove that
the input to this function is non-negative". You could probably do this
with a sufficiently clever Digit type in Haskell, but I don't see how youc
could do it with __predicate__.


On Wed, Aug 20, 2014 at 11:38 AM, David Mertz <mertz at gnosis.cx> wrote:

> On Wed, Aug 20, 2014 at 11:14 AM, Bob Ippolito <bob at redivi.com> wrote:
> > There's the misunderstanding: PyContracts style contracts are not "easy
> enough" in Haskell.
>
> Well, for example, I found something like this for Haskell:
>
> newtype Digit = Digit { digitVal :: Int }
>   deriving (Eq, Ord, Show)
> mkDigit :: Int -> Maybe Digit
> mkDigit n
>   | n >= 0 && n < 10 = Just (Digit n)
>   | otherwise = Nothing
>
>
> OK, sure it's not just one line.  But now we have a predicate-restricted
> data type right in the type system.  If we introduce a static type system
> in Python, I'd like it to be able to do that.  PyContracts--or something
> close to it--lets us do that in a DSL.  But I would be equally happy to use
> some Python code that could be tucked away but reused.  E.g., completely
> hypothetical syntax:
>
> class Digit(mypy.Int):
>
>     def __predicate__(self):
>
>         return 0 <= self < 10
>
> def times_modulo(incr: Digit, num: Int) -> Digit:
>
>     result = 0
>
>     for i in range(num):
>
>         result += incr
>
>         result %= 10
>
>     return result
>
>
> I could just stick Digit and all my other custom types in 'mytypes.py',
> and the actual annotations would have a richer type system, plus remain
> readable (if I chose decent names for the types).
>
>
> > mypy's types are not like a C type system. There are a few missing
> features that are being worked on, but it is much better than some here
> perceive it to be.
> >
> >
> > On Wednesday, August 20, 2014, David Mertz <mertz at gnosis.cx> wrote:
> >>
> >> Hi Bob,
> >>
> >> I enjoyed watching your talk (on video, not live), and I certainly see
> the appeal of a Haskell-like type system.  However, what we seem to be
> discussing here with mypy annotations looks a lot closer to a C type system
> than to a Haskell type system.  All those things that PyContracts get us
> are easy enough in Haskell--why aim so low if we are thinking of a change
> in Python?
> >>
> >>
> >> On Wed, Aug 20, 2014 at 10:42 AM, Bob Ippolito <bob at redivi.com> wrote:
> >>>
> >>> On Wed, Aug 20, 2014 at 10:23 AM, David Mertz <mertz at gnosis.cx> wrote:
> >>>>
> >>>> I have to say that I find the capabilities in PyContracts--which are
> absent currently in mypy--to be very compelling.  If we are going to add
> encouragement to use type annotations, adding richer types seems extremely
> worthwhile.
> >>>
> >>>
> >>> It's easy to check any expressible condition at runtime, but this is
> not the case ahead of time as with a static analysis tool or linter. It'd
> be great to have this sort of capability in mypy, but what you're asking
> for is at the fringes of current research and will not be practical to add
> to Python any time soon.
> >>>
> >>> -bob
> >>>
> >>
> >>
> >>
> >> --
> >> Keeping medicines from the bloodstreams of the sick; food
> >> from the bellies of the hungry; books from the hands of the
> >> uneducated; technology from the underdeveloped; and putting
> >> advocates of freedom in prisons.  Intellectual property is
> >> to the 21st century what the slave trade was to the 16th.
>
>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/ba1c6ab6/attachment-0001.html>

From bob at redivi.com  Wed Aug 20 22:03:25 2014
From: bob at redivi.com (Bob Ippolito)
Date: Wed, 20 Aug 2014 13:03:25 -0700
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <CAEbHw4a0gDaRGX0oh6hCO-fk_iO1azjGBEo8XuBcSxRuo55=XA@mail.gmail.com>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <CAJfDr96uTGBR7Nq0SqpYGajySSNUjk0RY-ftEh1wyiAHxRrUiQ@mail.gmail.com>
 <CAEbHw4b4zRVqM-EABkKnj8oenZM3znYiMvvpQfYG2u6oCND1xA@mail.gmail.com>
 <CACwMPm9Qmipvhc7MUGdqxNaPnrWwVMdKYscYi2udyKN0g1QR1w@mail.gmail.com>
 <CAEbHw4ZGWP-xo-M83xY=N9M=LJkryaSCwWUCX3uTnzZEjW5Kcg@mail.gmail.com>
 <CACwMPm8BCHd89BA5=Z6Hxwx6WAbr79Vyeo+KABvTwEJFpiTgqA@mail.gmail.com>
 <CAEbHw4a0gDaRGX0oh6hCO-fk_iO1azjGBEo8XuBcSxRuo55=XA@mail.gmail.com>
Message-ID: <CACwMPm-dNYWjcii2y+sAvPG_Z1haa_CT8zSnryrqbNMzJjKznA@mail.gmail.com>

On Wed, Aug 20, 2014 at 11:38 AM, David Mertz <mertz at gnosis.cx> wrote:

> On Wed, Aug 20, 2014 at 11:14 AM, Bob Ippolito <bob at redivi.com> wrote:
> > There's the misunderstanding: PyContracts style contracts are not "easy
> enough" in Haskell.
>
> Well, for example, I found something like this for Haskell:
>
> newtype Digit = Digit { digitVal :: Int }
>   deriving (Eq, Ord, Show)
> mkDigit :: Int -> Maybe Digit
> mkDigit n
>   | n >= 0 && n < 10 = Just (Digit n)
>   | otherwise = Nothing
>
>
> OK, sure it's not just one line.  But now we have a predicate-restricted
> data type right in the type system.  If we introduce a static type system
> in Python, I'd like it to be able to do that. [?]
>

This isn't in the type system. This is what is called a smart constructor
[1]. These predicates are strictly a runtime construct, not compile time.
Guard syntax (a sequence of pipes each followed by a predicate and then an
equal sign and some term) is just a convenient way to write if[/else
if?]/else.

There are ways to do this specific sort of dependent typing [2] in Haskell
using an unholy combination of GHC-specific extensions and the absolute
latest compiler. This is exciting research, but not so practical today.

[1] http://www.haskell.org/haskellwiki/Smart_constructors
[2] http://en.wikipedia.org/wiki/Dependent_type

-bob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/3928677e/attachment.html>

From castironpi at gmail.com  Wed Aug 20 22:12:01 2014
From: castironpi at gmail.com (Aaron Brady)
Date: Wed, 20 Aug 2014 15:12:01 -0500
Subject: [Python-ideas] Mutating while iterating
In-Reply-To: <CAEX1CPNJBXnQn8UUe_ZObC2bNGb3_rFDpeuy-xSaOZtAA2W9fw@mail.gmail.com>
References: <cd149d5b-5b57-4fbe-a639-a7f13fae64cf@googlegroups.com>
 <CADiSq7cG0Cfp0AVn2xeVx-rRyqzeGUPV3XWmQQcXSeMa=02LPA@mail.gmail.com>
 <CAEX1CPOxUiqcC860zxb9kyNTM6Dc6jVmqe0FE0G151GUF==vhQ@mail.gmail.com>
 <CAEX1CPNJBXnQn8UUe_ZObC2bNGb3_rFDpeuy-xSaOZtAA2W9fw@mail.gmail.com>
Message-ID: <CAEX1CPON=Aq0BAP_nFggOdYn1rbu3DU7DnK1_sheM2P4UGndXw@mail.gmail.com>

On Wed, Aug 13, 2014 at 7:21 AM, Aaron Brady <castironpi at gmail.com> wrote:
> On Sun, Aug 3, 2014 at 12:46 PM, Aaron Brady <castironpi at gmail.com> wrote:
>> On Sat, Jul 26, 2014 at 10:39 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
[snip]
>>
>> Python is replete with examples of prohibiting structures which are
>> likely bugs but aren't segfaults.  There are also reciprocal-- though
>> not necessarily inverse-- limitations of the unordered collections
>> themselves ("set" and "dict").  The new behavior is transparent for
>> the programmer; no possible programs can /rely/ on the existing
>> behavior.  The new behavior introduces no new objects, possibly except
>> "IterationError", no new syntax, and no new costs.
>>
>> I propose we leave this discussion thread open for the time being.  I
>> also take the issue of /re/-assigning to keys during iteration to be
>> settled as permitted.
>
> Nick, It works by comparing the state of the container, to its state
> when the iterator was opened.  We're ensuring it will always have a
> unique state, up to comparison.  A state can be reused once no
> iterators refer to it, hence needing the reference count.  A full
> "object" is not needed for a memo, only the reference count, but the
> "object" is easier and only twice the size, as "PyBaseObject Type" is
> allocated anyway.
>
> I'll point out that among the additional costs that there aren't,
> garbage collection isn't any slower, as both "tp traverse" and "tp
> clear" are empty in the "PyBaseObject Type" definition, on line
> 3511-3512 in "typeobject.c" at time of writing [1].
>
> [1] http://svn.python.org/view/python/trunk/Objects/typeobject.c?revision=81744&view=markup#l3511

For ordered containers, there are several consistent behaviors if
there's an iterator on the element being removed.

1) Always advance
2) Always retreat
3) Always close
4) Specified in iterator / reverse iter's
5) Specified by "remove" caller

Iterators could define a 3rd method for it, or 4th counting "prev".
Iterators might be invalidated permanently or until the next call to
"Next" in some languages, after the mutating call.  It raises the
issue of when there's an iterator on the first or last item and it's
removed (and others are inserted there).

The same sentinel value can't be used before it's started and after
it's finished.  Either the container or the iterator needs extra
storage to distinguish.  Though the sizes can be kept the same by
changing the semantics slightly.

From abarnert at yahoo.com  Wed Aug 20 22:14:33 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 20 Aug 2014 13:14:33 -0700
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <CA+_xFerJnXrF_J-ypfKPrhC7UojbN2+GB8kgaD0dhmLhgh9QTA@mail.gmail.com>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <CAJfDr96uTGBR7Nq0SqpYGajySSNUjk0RY-ftEh1wyiAHxRrUiQ@mail.gmail.com>
 <CAEbHw4b4zRVqM-EABkKnj8oenZM3znYiMvvpQfYG2u6oCND1xA@mail.gmail.com>
 <CACwMPm9Qmipvhc7MUGdqxNaPnrWwVMdKYscYi2udyKN0g1QR1w@mail.gmail.com>
 <CAEbHw4ZGWP-xo-M83xY=N9M=LJkryaSCwWUCX3uTnzZEjW5Kcg@mail.gmail.com>
 <CACwMPm8BCHd89BA5=Z6Hxwx6WAbr79Vyeo+KABvTwEJFpiTgqA@mail.gmail.com>
 <CAEbHw4a0gDaRGX0oh6hCO-fk_iO1azjGBEo8XuBcSxRuo55=XA@mail.gmail.com>
 <CA+_xFerJnXrF_J-ypfKPrhC7UojbN2+GB8kgaD0dhmLhgh9QTA@mail.gmail.com>
Message-ID: <0BE0982A-54A7-4500-B08D-291956DA1EC6@yahoo.com>

On Aug 20, 2014, at 12:42, Alex Hammel <ahammel87 at gmail.com> wrote:

> The problem is that the proposal is to use mypy syntax for static type checking. It's really hard to prove that values of Digit never go below zero without doing checks at runtime.
> 
> Your Haskell example doesn't do that either, actually. Sure, the value of Digit can never be a negative number, but neither is it always a positive number (it might be Nothing).
> 
> Say I've got a function that requires a non-negative number to work properly, so I specify that it takes one of your Digits. If I mess up and pass it a negative number, the Digit class converts it to Nothing, resulting in a runtime error (or some exceptional runtime behaviour, depending on what I do with that Nothing). What I really want if for the compiler to say "sorry, I'm not compiling this because I can't prove that the input to this function is non-negative".

If the compiler could say, "I am compiling it because I can prove it's either non-negative or from an un-checkable source, but I'll keep track of which so I can propagate that information", that could be useful too.

Of course this is only useful if uncheckable sources are rare and contained in your app, the checker makes it easy to read out what did and didn't get infected with permissiveness, and the type system makes it easy to write things so that you don't have to make each type handle permissiveness manually the way you would in Haskell. There are a couple of ML variants aimed at making this easier, and there's no reason PyConstrajnts couldn't do something similar.

> You could probably do this with a sufficiently clever Digit type in Haskell, but I don't see how youc could do it with __predicate__.
> 
> 
> On Wed, Aug 20, 2014 at 11:38 AM, David Mertz <mertz at gnosis.cx> wrote:
>> On Wed, Aug 20, 2014 at 11:14 AM, Bob Ippolito <bob at redivi.com> wrote:
>> > There's the misunderstanding: PyContracts style contracts are not "easy enough" in Haskell.
>> 
>> Well, for example, I found something like this for Haskell:
>> 
>> newtype Digit = Digit { digitVal :: Int }
>>   deriving (Eq, Ord, Show)
>> mkDigit :: Int -> Maybe Digit
>> mkDigit n
>>   | n >= 0 && n < 10 = Just (Digit n)
>>   | otherwise = Nothing
>> 
>> OK, sure it's not just one line.  But now we have a predicate-restricted data type right in the type system.  If we introduce a static type system in Python, I'd like it to be able to do that.  PyContracts--or something close to it--lets us do that in a DSL.  But I would be equally happy to use some Python code that could be tucked away but reused.  E.g., completely hypothetical syntax:
>> 
>> class Digit(mypy.Int):
>>     def __predicate__(self):
>>         return 0 <= self < 10
>> 
>> def times_modulo(incr: Digit, num: Int) -> Digit:
>>     result = 0
>>     for i in range(num):
>>         result += incr
>>         result %= 10
>>     return result
>> 
>> I could just stick Digit and all my other custom types in 'mytypes.py', and the actual annotations would have a richer type system, plus remain readable (if I chose decent names for the types).
>> 
>> 
>> > mypy's types are not like a C type system. There are a few missing features that are being worked on, but it is much better than some here perceive it to be.
>> >
>> >
>> > On Wednesday, August 20, 2014, David Mertz <mertz at gnosis.cx> wrote:
>> >>
>> >> Hi Bob,
>> >>
>> >> I enjoyed watching your talk (on video, not live), and I certainly see the appeal of a Haskell-like type system.  However, what we seem to be discussing here with mypy annotations looks a lot closer to a C type system than to a Haskell type system.  All those things that PyContracts get us are easy enough in Haskell--why aim so low if we are thinking of a change in Python?
>> >>
>> >>
>> >> On Wed, Aug 20, 2014 at 10:42 AM, Bob Ippolito <bob at redivi.com> wrote:
>> >>>
>> >>> On Wed, Aug 20, 2014 at 10:23 AM, David Mertz <mertz at gnosis.cx> wrote:
>> >>>>
>> >>>> I have to say that I find the capabilities in PyContracts--which are absent currently in mypy--to be very compelling.  If we are going to add encouragement to use type annotations, adding richer types seems extremely worthwhile.  
>> >>>
>> >>>
>> >>> It's easy to check any expressible condition at runtime, but this is not the case ahead of time as with a static analysis tool or linter. It'd be great to have this sort of capability in mypy, but what you're asking for is at the fringes of current research and will not be practical to add to Python any time soon.
>> >>>
>> >>> -bob
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Keeping medicines from the bloodstreams of the sick; food
>> >> from the bellies of the hungry; books from the hands of the
>> >> uneducated; technology from the underdeveloped; and putting
>> >> advocates of freedom in prisons.  Intellectual property is
>> >> to the 21st century what the slave trade was to the 16th.
>> 
>> 
>> 
>> 
>> --
>> Keeping medicines from the bloodstreams of the sick; food
>> from the bellies of the hungry; books from the hands of the
>> uneducated; technology from the underdeveloped; and putting
>> advocates of freedom in prisons.  Intellectual property is
>> to the 21st century what the slave trade was to the 16th.
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/b60beb00/attachment-0001.html>

From ben+python at benfinney.id.au  Wed Aug 20 22:29:16 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 21 Aug 2014 06:29:16 +1000
Subject: [Python-ideas] Optional static typing -- late to the party
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <20140820123508.GK25957@ando> <85sikr1fwb.fsf@benfinney.id.au>
 <CAPTjJmpXZ+LH8SP-nZWN5cmU=BcpdA22Ckw6vmuYsUcmv1W5Gg@mail.gmail.com>
Message-ID: <85lhqi29cz.fsf@benfinney.id.au>

Chris Angelico <rosuav at gmail.com> writes:

> On Wed, Aug 20, 2014 at 10:53 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> > Steven D'Aprano <steve at pearwood.info> writes:
> >
> >> But remember that any annotations in docstrings may not be
> >> available at runtime
> >
> > How so? What conformant Python implementation is discarding
> > docstrings from code objects?
>
> CPython with the -OO flag. Or is that non-conformant?

Okay, one can deliberately remove docstrings with an option to the
interpreter (the ?-OO? option explicitly has that purpose). So a user
wanting to check annotations in docstrings wouldn't use that option. So
this case doesn't seem relevant to the discussion.

Remember that we're talking about type annotations that are for
*static code checkers* to inspect. Docstrings will certainly be
available there.

-- 
 \        ?? no testimony can be admitted which is contrary to reason; |
  `\   reason is founded on the evidence of our senses.? ?Percy Bysshe |
_o__)                        Shelley, _The Necessity of Atheism_, 1811 |
Ben Finney


From ncoghlan at gmail.com  Wed Aug 20 23:25:24 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Aug 2014 07:25:24 +1000
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CAP7+vJK59UWOPo1v6d57pjFXCebth2Y4+ukQ1OGrfBaZ0Bzs3A@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <CADiSq7d0zdZ7x5fgU5tM86HCS8ZCjc4UYP_CBF4L+yirHCnYQg@mail.gmail.com>
 <CAP7+vJK59UWOPo1v6d57pjFXCebth2Y4+ukQ1OGrfBaZ0Bzs3A@mail.gmail.com>
Message-ID: <CADiSq7f7cdf8HvYPi4jDx+SrFO8TfzqHQGaWn_4M=a8u9xh08Q@mail.gmail.com>

On 21 Aug 2014 01:53, "Guido van Rossum" <guido at python.org> wrote:
>
> Was that meant to support Antoine's position (ABCs are not type
descriptions) or mine (they are)?

Yours. The ability to register any type with any ABC means that even an
explicit isinstance() ABC check at runtime is only advisory - developers
are free to force a "true" answer if they choose to do so. You can even
disable a check entirely by registering "object" with the relevant ABC
(actually doing that would likely be a *bad idea*, but the interpreter
allows it). That puts them squarely in the "descriptions of intent"
category for me.

A bit more rambling, since I've only been skimming (at best) the type
hinting threads:

The thing I appreciate getting out of linters is picking up the "this is
almost certainly wrong" cases (like trying to call "indx" on something I
have said I expect to be a sequence). This is especially useful when code
is modified over time, but retains the original annotations documenting the
interface assumptions - if you expand the function to cover new types,
you'll need to change the annotation, which is also a good reminder that
the docs will need updating.

I'm not particularly worried if there are still cases where linters go "I
have no idea what that object is, so I'll assume anything goes". After all,
that's the baseline we start from when we don't run a linter at all.

Cheers,
Nick.

>
>
> On Wed, Aug 20, 2014 at 4:27 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>>
>> On 20 Aug 2014 10:14, "Guido van Rossum" <guido at python.org> wrote:
>> >
>> > On Tue, Aug 19, 2014 at 4:02 PM, Antoine Pitrou <antoine at python.org>
wrote:
>> >>
>> >> Le 19/08/2014 18:50, Terry Reedy a ?crit :
>> >>
>> >>> On 8/19/2014 9:27 AM, Nick Coghlan wrote:
>> >>>
>> >>> What I like: 'optional type hints' based on a fleshed-out ABC systems
>> >>> that collects all ABCs together on one module via import from current
>> >>> files. It seems that we have been slowly groping towards this for
years.
>> >>
>> >>
>> >> Hmm, I've been saying this already, but my intuition is that it's a
bad idea to conflate *type descriptions* (what this proposal is about) and
actual *runtime types* (what ABCs are).
>> >
>> >
>> > But are they? I think the registration mechanism makes it clear that
they aren't (necessarily) runtime types -- by linking a concrete type with
an ABC through registration you are pretty clearly stating that the ABC
*describes* (an aspect of) the concrete class without automatically adding
any behavior from the ABC to it.
>>
>> One of the ways I describe the explicit registration mechanism to people
is as giving you the option of lying to the interpreter. If you use
explicit registration, your registered type basically walks around with a
sign saying "I'm a duck!" and the interpreter goes "OK, you're a duck!" and
assumes it will be able to quack without actually checking in advance. It's
really just a constrained version of normal duck typing, where the
interpreter just assumes that *everything* may exhibit duck-like tendencies.
>>
>> Cheers,
>> Nick.
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140821/ca478814/attachment.html>

From antoine at python.org  Wed Aug 20 23:57:02 2014
From: antoine at python.org (Antoine Pitrou)
Date: Wed, 20 Aug 2014 17:57:02 -0400
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
Message-ID: <lt35je$tp3$1@ger.gmane.org>


Le 19/08/2014 20:12, Guido van Rossum a ?crit :
>     Hmm, I've been saying this already, but my intuition is that it's a
>     bad idea to conflate *type descriptions* (what this proposal is
>     about) and actual *runtime types* (what ABCs are).
>
>
> But are they? I think the registration mechanism makes it clear that
> they aren't (necessarily) runtime types -- by linking a concrete type
> with an ABC through registration you are pretty clearly stating that the
> ABC *describes* (an aspect of) the concrete class without automatically
> adding any behavior from the ABC to it.

Hmm... well, they are usable at runtime (if only for isinstance calls 
:-)). I admit my wording was a bit vague here. But our type descriptions 
should be able to express more information than ABCs currently do. For 
example, I don't how you'd express the idea of a "mapping from str to 
int" using the current Mapping ABC, while retaining the runtime 
properties of the Mapping class.

Regards

Antoine.



From steve at pearwood.info  Thu Aug 21 02:02:39 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 21 Aug 2014 10:02:39 +1000
Subject: [Python-ideas] Annotations (and static typing)
In-Reply-To: <CACac1F8ALXmMEa61TV-WHfLfpuDE5Y9kmGOkMXS13b1nfX0nvg@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <CAP7+vJJC9Gq5c+9DqWrzm8w=cznjyr3e7TTnGaewiJsshoSA9Q@mail.gmail.com>
 <53F4CC2F.70308@stoneleaf.us>
 <CACac1F8ALXmMEa61TV-WHfLfpuDE5Y9kmGOkMXS13b1nfX0nvg@mail.gmail.com>
Message-ID: <20140821000234.GN25957@ando>

On Wed, Aug 20, 2014 at 05:31:50PM +0100, Paul Moore wrote:

> The obvious thought is, if a decorator is a sufficiently good
> notation, is there any need for annotations at all (even for static
> typing)?

Decorators are *not* sufficiently good notation.

Decorators have many nice uses, but they are second class for this 
purpose since they require you to write the parameters twice: once in 
the decorator, and once in the actual function declaration. They also 
visually separate the parameter from its value.

If you're thinking about editing code with one or two such functions, 
with simple parameter declarations, it's no big deal. They're not *very* 
far away, a line or two maybe, it's easy to cope. But as the number of 
parameters increases, or the complexity of the type declaration 
increases, you get code like this:

@annotate(fe=Str[blah blah blah blah])
@annotate(fi=List[blah blah blah blah])
@annotate(fo=Dict[blah blah blah blah])
@annotate(fum=Tuple[blah blah blah blah])
@returns(int)
def spam(fe=aaa, fi=bbb, fo=ccc, fum=ddd):
    ...


and now you're looking at a solid wall of annotations, and the distance 
between the first decorator and the parameter list isn't "a line or two" 
any more. Or you get something like this, which may be a bit better:

@annotate(fe=Str[blah blah blah blah],
          fi=List[blah blah blah blah],
          fo=Dict[blah blah blah blah],
          fum=Tuple[blah blah blah blah])
@returns(int)
def spam(fe=aaa, fi=bbb, fo=ccc, fum=ddd):
    ...

but still separates the type of fe from the declaration of fe by four 
lines. With annotations, everything[1] stays together:

def spam(fe:Str[blah blah blah blah]=aaa,
         fi:List[blah blah blah blah]=bbb,
         fo:Dict[blah blah blah blah]=ccc,
         fum:Tuple[blah blah blah blah]=ddd,
         )->int:
    ...


I'm going to take the liberty of quote Nick from an earlier message:

[quote]
I once had the "pleasure" of inheriting some code written in K&R style
C, where the parameter type declarations were separate from the
signature line:

    void foo(a, b, c)
        double a;
        char b;
    {
          ...
    }

ANSI C, with inline typing,  is far more readable :)
[end quote]

C programmers, hands up if you prefer to use K&R style declarations? 
I expect you will be in a very small minority.

There is a reason why most languages with type declarations normally put 
them together with the parameter declarations.

If anyone wishes to argue for decorator style *in preference* to 
annotations, a good place to start is with a list of programming 
languages which use similar syntax for their type declarations.





[1] Not quite, you still have to write documentation separately, but 
since the documentation for a single parameter might be an entire 
paragraph of text, we don't want to put that in the parameter list. 
Some problems are just intractable.


-- 
Steven

From steve at pearwood.info  Thu Aug 21 02:09:54 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 21 Aug 2014 10:09:54 +1000
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <85lhqi29cz.fsf@benfinney.id.au>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <20140820123508.GK25957@ando> <85sikr1fwb.fsf@benfinney.id.au>
 <CAPTjJmpXZ+LH8SP-nZWN5cmU=BcpdA22Ckw6vmuYsUcmv1W5Gg@mail.gmail.com>
 <85lhqi29cz.fsf@benfinney.id.au>
Message-ID: <20140821000954.GO25957@ando>

On Thu, Aug 21, 2014 at 06:29:16AM +1000, Ben Finney wrote:
> Chris Angelico <rosuav at gmail.com> writes:
> 
> > On Wed, Aug 20, 2014 at 10:53 PM, Ben Finney <ben+python at benfinney.id.au> wrote:
> > > Steven D'Aprano <steve at pearwood.info> writes:
> > >
> > >> But remember that any annotations in docstrings may not be
> > >> available at runtime
> > >
> > > How so? What conformant Python implementation is discarding
> > > docstrings from code objects?
> >
> > CPython with the -OO flag. Or is that non-conformant?
> 
> Okay, one can deliberately remove docstrings with an option to the
> interpreter (the ?-OO? option explicitly has that purpose). So a user
> wanting to check annotations in docstrings wouldn't use that option. So
> this case doesn't seem relevant to the discussion.

But "the user" can be two different people: the writer of the code using 
the annotations, and the person running the code that wants to use 
annotations. It's the *second* user, not the first, who decides whether 
or not to use -OO.

If you're the original developer, you might insist that your code is not 
compatible with -OO, but (in my opinion) that's a mean thing to do if it 
can be avoided. Better to not rely on docstrings for anything other than 
documentation.


> Remember that we're talking about type annotations that are for
> *static code checkers* to inspect. Docstrings will certainly be
> available there.

True, but runtime checks may (or may not) be a part of the code checker. 
At least, we should not rule that out.


-- 
Steven

From ethan at stoneleaf.us  Thu Aug 21 02:22:45 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 20 Aug 2014 17:22:45 -0700
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <20140821000954.GO25957@ando>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <20140820123508.GK25957@ando> <85sikr1fwb.fsf@benfinney.id.au>
 <CAPTjJmpXZ+LH8SP-nZWN5cmU=BcpdA22Ckw6vmuYsUcmv1W5Gg@mail.gmail.com>
 <85lhqi29cz.fsf@benfinney.id.au> <20140821000954.GO25957@ando>
Message-ID: <53F53BD5.5090902@stoneleaf.us>

On 08/20/2014 05:09 PM, Steven D'Aprano wrote:
> On Thu, Aug 21, 2014 at 06:29:16AM +1000, Ben Finney wrote:
>>
>> Remember that we're talking about type annotations that are for
>> *static code checkers* to inspect. Docstrings will certainly be
>> available there.
>
> True, but runtime checks may (or may not) be a part of the code checker.
> At least, we should not rule that out.

This proposal is not about "a code checker" but about *static* typing.

Seems to be a lot of people forgetting that *static* (at least in this case), means not actually running the program -- 
hence, no loss of docstrings.

--
~Ethan~

From stephen at xemacs.org  Thu Aug 21 03:42:45 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 21 Aug 2014 10:42:45 +0900
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
In-Reply-To: <lt2sh5$9vi$1@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
 <lt2sh5$9vi$1@ger.gmane.org>
Message-ID: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>

Terry Reedy writes:

 > If the annotation for a library are in a separate skeleton file

You mean stub file, right?  (Stub is the term used so far in this
thread.)

To develop Terry's idea a bit more explicitly, ISTM that during early
development, you use Python 3 with annotations in file.  Then when you
need to test Python 2, you use a tool (as yet to be developed) which
generates a Python file that has had all annotations stripped, and a
stub file[1].  Throw away the stub file, test the Python file, and
return to development.

For distribution (to *both* Python 3 and Python 2 users) you ignore
the original annotated code and distribute the generated Python file
and the stubs file.  Probably there would be a tool for combining the
Python and stub files back into an annotated Python file.

The two tools (stripper and annotator) can be very accurate (although
the annotator may not produce precisely the style desired by users,
the tools probably can be designed so that the re-annotated file
accurately reflects the style used by the authors).

Is the implied workflow really so burdensome?  Especially to
downstream?

Note that Terry works on IDLE, so it seems likely that at least one
Python IDE would support this workflow very well. :-)

Footnotes: 
[1]  Yes, I know, the stub file is actually Python, too.  You know
what I mean.



From jim.baker at python.org  Thu Aug 21 05:28:34 2014
From: jim.baker at python.org (Jim Baker)
Date: Wed, 20 Aug 2014 21:28:34 -0600
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
 <lt2sh5$9vi$1@ger.gmane.org> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>

(My apologies in advance that I have not been able to read all the emails
on this set of threads.)

Mypy?s approach to parameterizing types using slice notation/__getitem__ is
a rather neat hack, given how it does not require any new notation, matches
rather similar syntax in languages like Scala and Haskell, and can be
readily added to existing types, whether builtins like list, collections
like deque, or ABC collection types like Iterator andSequence. Adding such
parameterization support to builtin/stdlib types is a compelling reason for
including in 3.5. An enthusiastic +1!

Such support is also a very good reason for Jython 3.x to target 3.5.

However, let's not restrict Python?s support for static typing, or related
schemes like gradual typing, to just what Mypy can do now. This should be
about a preferred syntax, especially since there?s other active work, some
of which I?m involved in like Reticulated (
http://wphomes.soic.indiana.edu/jsiek/), which has the possibility of
helping with such issues as better Java integration in Jython, such as with
the Clamp project (https://github.com/jimbaker/clamped).

Instead, I like what one message stated - that such type expressions in
function annotations can be arbitrarily complicated expressions. We can
have simple parameterization, but also disjunctive types, and more. In
certain simple cases, linters will be able to meaningfully work with such
type expressions without having to evaluate the defining module (if not
actually run it); in other cases, they won?t. But this is just what linters
cope with now. We see something similar in Clamp, since we need to evaluate
class definitions ahead-of-time to generate the bytecode for corresponding
Java proxy classes - otherwise, we would not have a jar that Java/JVM code
could use for linkage.

Using type variables is another aspect that can be addressed in this
proposed syntax, but it is readily doable without further modification.
Consider a type expression like Sequence[T] - the only thing that we need
to do is assign the type variable T in the scope of the annotation usage.
I?m pretty certain we can do some pretty nice things with such type
variable assignments to describe type bounds, variance, and what not, for
those so inclined and type checkers that can handle. Again, this keeps
things nicely open without undue restrictions.

One last thing: let?s not add List, TList, etc. Just use list and other
existing types to generate new types. I think this is the main reason why
3.5 support is needed. Being able to use types as objects in this fashion
is a wonderful pleasure of working with Python as of new style classes (so
I will assume now that we are talking about Python 3.5!). The same goes for
putting such support in docstrings (although static type checkers could
potentially attempt to cross validate) and in comments. Using stub files in
contrast is a great idea, and something that can also be standardized on.

Thanks again for a fantastic, if somewhat overwhelming conversation.

- Jim


On Wed, Aug 20, 2014 at 7:42 PM, Stephen J. Turnbull <stephen at xemacs.org>
wrote:

> Terry Reedy writes:
>
>  > If the annotation for a library are in a separate skeleton file
>
> You mean stub file, right?  (Stub is the term used so far in this
> thread.)
>
> To develop Terry's idea a bit more explicitly, ISTM that during early
> development, you use Python 3 with annotations in file.  Then when you
> need to test Python 2, you use a tool (as yet to be developed) which
> generates a Python file that has had all annotations stripped, and a
> stub file[1].  Throw away the stub file, test the Python file, and
> return to development.
>
> For distribution (to *both* Python 3 and Python 2 users) you ignore
> the original annotated code and distribute the generated Python file
> and the stubs file.  Probably there would be a tool for combining the
> Python and stub files back into an annotated Python file.
>
> The two tools (stripper and annotator) can be very accurate (although
> the annotator may not produce precisely the style desired by users,
> the tools probably can be designed so that the re-annotated file
> accurately reflects the style used by the authors).
>
> Is the implied workflow really so burdensome?  Especially to
> downstream?
>
> Note that Terry works on IDLE, so it seems likely that at least one
> Python IDE would support this workflow very well. :-)
>
> Footnotes:
> [1]  Yes, I know, the stub file is actually Python, too.  You know
> what I mean.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
- Jim

jim.baker@{colorado.edu|python.org|rackspace.com|zyasoft.com}
twitter.com/jimbaker
github.com/jimbaker
bitbucket.com/jimbaker
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140820/3913174e/attachment-0001.html>

From tjreedy at udel.edu  Thu Aug 21 08:03:46 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 21 Aug 2014 02:03:46 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <lt4254$f1a$1@ger.gmane.org>

On 8/20/2014 9:42 PM, Stephen J. Turnbull wrote:
> Terry Reedy writes:
>
>   > If the annotation for a library are in a separate skeleton file
>
> You mean stub file, right?  (Stub is the term used so far in this
> thread.)

Then yes, that I what i mean ;-).

> To develop Terry's idea a bit more explicitly, ISTM that during early
> development, you use Python 3 with annotations in file.  Then when you
> need to test Python 2, you use a tool (as yet to be developed) which
> generates a Python file that has had all annotations stripped, and a
> stub file[1].  Throw away the stub file, test the Python file, and
> return to development.

An alternative is to develop code and stub separately. There are people 
(not core developers) already working on stub files for the stdlib and 
other libraries. In fact there are multiple groups using different 
syntaxes.  Presumably some will join forces if there is a blessed, 
standard, syntax.  There are no plans to add annotations to the stdlib 
code itself.

Even if one does not add annotations to one own code, which many 
projects, especially smaller ones, will not, there will be the 
possibility of having calls into the stdlib and other annotated or 
stubbed libraries statically checked on 3.5 (or later).  One will be 
able to at least partially benefit without writing annotations oneself.

> For distribution (to *both* Python 3 and Python 2 users) you ignore
> the original annotated code and distribute the generated Python file
> and the stubs file.  Probably there would be a tool for combining the
> Python and stub files back into an annotated Python file.
>
> The two tools (stripper and annotator) can be very accurate (although
> the annotator may not produce precisely the style desired by users,
> the tools probably can be designed so that the re-annotated file
> accurately reflects the style used by the authors).

I imagine both tools will emerge.

> Is the implied workflow really so burdensome?  Especially to
> downstream?
>
> Note that Terry works on IDLE,

I have been watching this thread precisely from that viewpoint.

> so it seems likely that at least one
> Python IDE would support this workflow very well. :-)

Part this summer's Idle GSOC project was developing a framework to run 
3rd party code checkers on code in an Idle editor from the editor. I 
just need to review and possibly revise the code Saimadhav wrote and 
decide among the alternatives tried. Acesss to existing pyflakes, 
pylint, or anything else, once installed (easy now with pip), will soon 
be possible.

Anything in the stdlib, rather than at PyPI or wherever, would likely 
get even more direct support.  For instance, one can check syntax 
without running the edit buffer, and I believe that just imports and 
runs tokenize.tokenize.  I can imagine doing something similar with 
stripper, splitter, and joiner functions.  (This might be in an 
extension module so the feature can to disabled, as might be appropriate 
for a beginner class.)

-- 
Terry Jan Reedy


From antoine at python.org  Thu Aug 21 15:23:39 2014
From: antoine at python.org (Antoine Pitrou)
Date: Thu, 21 Aug 2014 09:23:39 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
Message-ID: <lt4rsr$a00$2@ger.gmane.org>

Le 20/08/2014 23:28, Jim Baker a ?crit :
>
> Mypy?s approach to parameterizing types using slice
> notation/|__getitem__| is a rather neat hack, given how it does not
> require any new notation, matches rather similar syntax in languages
> like Scala and Haskell, and can be readily added to existing types,
> whether builtins like |list|, collections like |deque|, or ABC
> collection types like |Iterator| and|Sequence|. Adding such
> parameterization support to builtin/stdlib types is a compelling reason
> for including in 3.5. An enthusiastic +1!

I'm -1 on using __getitem__ for parametering types. It is indeed a hack 
and its limitations will bite us harshly when wanting to refine the type 
descriptions (for example to describe a constraint on a collection's size).

> One last thing: let?s not add |List|, |TList|, etc. Just use |list| and
> other existing types to generate new types.

I'm -1 on that, for exactly the same reason as above.

Regards

Antoine.



From brett at python.org  Thu Aug 21 16:30:03 2014
From: brett at python.org (Brett Cannon)
Date: Thu, 21 Aug 2014 14:30:03 +0000
Subject: [Python-ideas] Optional Static Typing -- the Python Way
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org>
Message-ID: <CAP1=2W70P_Mdeup3bj6om+0N_HrR2cv-QMXHNKw5OiU79Je4Ww@mail.gmail.com>

On Wed Aug 20 2014 at 5:58:31 PM Antoine Pitrou <antoine at python.org> wrote:

>
> Le 19/08/2014 20:12, Guido van Rossum a ?crit :
> >     Hmm, I've been saying this already, but my intuition is that it's a
> >     bad idea to conflate *type descriptions* (what this proposal is
> >     about) and actual *runtime types* (what ABCs are).
> >
> >
> > But are they? I think the registration mechanism makes it clear that
> > they aren't (necessarily) runtime types -- by linking a concrete type
> > with an ABC through registration you are pretty clearly stating that the
> > ABC *describes* (an aspect of) the concrete class without automatically
> > adding any behavior from the ABC to it.
>
> Hmm... well, they are usable at runtime (if only for isinstance calls
> :-)). I admit my wording was a bit vague here. But our type descriptions
> should be able to express more information than ABCs currently do. For
> example, I don't how you'd express the idea of a "mapping from str to
> int" using the current Mapping ABC, while retaining the runtime
> properties of the Mapping class.
>

Isn't that the magic of __instancecheck__/__subclasscheck__? If typing.py
does the right thing here then it shouldn't matter. You could use __init__
or __getitem__ to construct an object who upon introspection provides the
details you want and do what you expect, and then override the appropriate
methods so that isinstance() checks check against the ABC instead of the
type class itself (heck we could make the base type class require that you
inherit from an ABC to promote working with ABCs instead of concrete
classes).

I have to also say that I like using ABCs because `from collections import
abc as a` leads to reading parameters like "x is a.Mapping" which is
linguistically quaint. =)


P.S.: Off-topic for this email but something I don't think has been
mentioned more than once, type hinting like we are proposing is exactly
what Dart does and it is nice. Built-in API documentation of interfaces
along with IDE support at the API makes all of this worth it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140821/c4d5628d/attachment.html>

From antoine at python.org  Thu Aug 21 16:54:17 2014
From: antoine at python.org (Antoine Pitrou)
Date: Thu, 21 Aug 2014 10:54:17 -0400
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CAP1=2W70P_Mdeup3bj6om+0N_HrR2cv-QMXHNKw5OiU79Je4Ww@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org>
 <CAP1=2W70P_Mdeup3bj6om+0N_HrR2cv-QMXHNKw5OiU79Je4Ww@mail.gmail.com>
Message-ID: <lt516p$j58$1@ger.gmane.org>


Le 21/08/2014 10:30, Brett Cannon a ?crit :
>
> Isn't that the magic of __instancecheck__/__subclasscheck__? If
> typing.py does the right thing here then it shouldn't matter.

What is "the right thing"?

> You could
> use __init__ or __getitem__ to construct an object who upon
> introspection provides the details you want and do what you expect, and
> then override the appropriate methods so that isinstance() checks check
> against the ABC instead of the type class itself (heck we could make the
> base type class require that you inherit from an ABC to promote working
> with ABCs instead of concrete classes).

So we agree that type descriptions should be separate things from ABCs, 
or not?

> I have to also say that I like using ABCs because `from collections
> import abc as a` leads to reading parameters like "x is a.Mapping" which
> is linguistically quaint. =)

Except that "x is a.Mapping" is False with a normal Python.

Regards

Antoine.



From abarnert at yahoo.com  Thu Aug 21 17:46:46 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 21 Aug 2014 08:46:46 -0700
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CAP1=2W70P_Mdeup3bj6om+0N_HrR2cv-QMXHNKw5OiU79Je4Ww@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org>
 <CAP1=2W70P_Mdeup3bj6om+0N_HrR2cv-QMXHNKw5OiU79Je4Ww@mail.gmail.com>
Message-ID: <1B33C3C0-A9E9-4380-BAD8-7B35A91FF367@yahoo.com>

On Aug 21, 2014, at 7:30, Brett Cannon <brett at python.org> wrote:

> I have to also say that I like using ABCs because `from collections import abc as a` leads to reading parameters like "x is a.Mapping" which is linguistically quaint. =)

Cute, but it seems like it actually impedes the reading of "x is an Iterable". (Although I guess as compensation you get "x is an Integer" if you import numbers as n, and you get to yodel like Adam Ant when reading code about file types.)

More seriously: I argued along the same lines as you as to the advantages of using the ABCs directly, but I understand the opposing viewpoint: although static type hints and runtime type constraints are _normally_ the same things, they're not _necessarily_ so, and forcing them to share a representation may preclude some of the fancy things people want to do with types beyond the simple generic model (e.g.; the PyConstraints sub-thread).

From ncoghlan at gmail.com  Thu Aug 21 18:15:53 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Aug 2014 02:15:53 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lt4rsr$a00$2@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
 <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
Message-ID: <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>

On 21 August 2014 23:23, Antoine Pitrou <antoine at python.org> wrote:
> Le 20/08/2014 23:28, Jim Baker a ?crit :
>>
>>
>> Mypy?s approach to parameterizing types using slice
>> notation/|__getitem__| is a rather neat hack, given how it does not
>>
>> require any new notation, matches rather similar syntax in languages
>> like Scala and Haskell, and can be readily added to existing types,
>> whether builtins like |list|, collections like |deque|, or ABC
>> collection types like |Iterator| and|Sequence|. Adding such
>>
>> parameterization support to builtin/stdlib types is a compelling reason
>> for including in 3.5. An enthusiastic +1!
>
>
> I'm -1 on using __getitem__ for parametering types. It is indeed a hack and
> its limitations will bite us harshly when wanting to refine the type
> descriptions (for example to describe a constraint on a collection's size).

Given:

    Sequence[Number]
    Sequence[Number|None]

There's still at least three options for extending the syntax even further:

    Sequence[Number](EXPR)
    Sequence[Number:EXPR]
    Sequence[Number,EXPR]

You'd probably want to use strings at that point, for the same reasons
PyContracts does. For example:

    Number,"N>0" # Positive number
    Sequence[Number,"N>0"] # Sequence of positive numbers
    Sequence[Number],"N>0" # Non-empty sequence

I doubt we'll ever get to that point, though. Type hinting and runtime
assertions aren't the same thing.

>> One last thing: let?s not add |List|, |TList|, etc. Just use |list| and
>>
>> other existing types to generate new types.
>
> I'm -1 on that, for exactly the same reason as above.

I'm also -1, on the grounds that concrete types and ABCs are
different, and I think type hinting should be heavily biased towards
ABCs.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From abarnert at yahoo.com  Thu Aug 21 18:16:18 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 21 Aug 2014 09:16:18 -0700
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <lt35je$tp3$1@ger.gmane.org>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org>
Message-ID: <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com>

On Aug 20, 2014, at 14:57, Antoine Pitrou <antoine at python.org> wrote:

> Le 19/08/2014 20:12, Guido van Rossum a ?crit :
>>    Hmm, I've been saying this already, but my intuition is that it's a
>>    bad idea to conflate *type descriptions* (what this proposal is
>>    about) and actual *runtime types* (what ABCs are).
>> 
>> 
>> But are they? I think the registration mechanism makes it clear that
>> they aren't (necessarily) runtime types -- by linking a concrete type
>> with an ABC through registration you are pretty clearly stating that the
>> ABC *describes* (an aspect of) the concrete class without automatically
>> adding any behavior from the ABC to it.
> 
> Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class.

Someone (I think Guido, but I can't find it) gave a concrete proposal here (which happens to match how MyPy already works): you make the collections ABCs implement subscripting behavior exactly the way the typing classes do--that is, they just return self. This means that at compile time Mapping[str, int] can be used as a static type hint that a function returns mappings from str to int, while a runtime isinstance check only verifies that the thing in question is a Mapping.

This avoids thorny questions like distinguishing covariant from contravariant contexts at runtime (which I don't think is solvable without a bunch of separate isinstance functions, and a more complicated __subclasshook__ protocol, but I could be wrong).

Also, there's no reason this couldn't be added in 3.5, and then an enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once someone works out those issues. (So isinstance still ignores the parameters, but iscovariant checks them covariantly, etc.) That wouldn't risk breaking 3.5 code in 3.6.

Also, keep in mind that, just because you _can_ write isinstance(d, Mapping[str, int]) and that might be confusing, that doesn't mean there's any reason you _would_ write that. It will only come up in metaprogramming contexts (e.g., where you're generating a function at runtime to match some type argument). Maybe you could argue that this means the compile-time machinery should be available at runtime, but I don't think you could argue that this means the compile-time types shouldn't be available at runtime.

I'll admit that I could be off on some of this. It's hard to discuss anything beyond the basics without either getting formal (which I don't think anyone wants) or providing some realistic examples of where you might want to runtime-check against an annotation (which I think would be very useful, if anyone has one).

From rosuav at gmail.com  Thu Aug 21 18:21:17 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 22 Aug 2014 02:21:17 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
 <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
 <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>
Message-ID: <CAPTjJmqNVNCSHdtc_apiyXsP1SbTrFGuM13XwHG5VHdpAvAY2Q@mail.gmail.com>

On Fri, Aug 22, 2014 at 2:15 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I'm also -1, on the grounds that concrete types and ABCs are
> different, and I think type hinting should be heavily biased towards
> ABCs.

We keep seeing this line of argument coming up, and while I've only
been skimming, I haven't seen anyone comment that this applies to
parameters but not so much to return values. A function might take a
Sequence and iterate over it, and then return a concrete integer. Or
it might take a file-like object and a string, and yield a series of
strings. Or it might take some set of parameters, and return None
unconditionally. There's nothing wrong with being concrete about
return values.

ChrisA

From ncoghlan at gmail.com  Thu Aug 21 18:26:24 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Aug 2014 02:26:24 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAPTjJmqNVNCSHdtc_apiyXsP1SbTrFGuM13XwHG5VHdpAvAY2Q@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
 <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
 <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>
 <CAPTjJmqNVNCSHdtc_apiyXsP1SbTrFGuM13XwHG5VHdpAvAY2Q@mail.gmail.com>
Message-ID: <CADiSq7dkmj4i=WJLdyMxTvPXu6m_J4FOU=8JTk2DYa_ysk1=bg@mail.gmail.com>

On 22 August 2014 02:21, Chris Angelico <rosuav at gmail.com> wrote:
> On Fri, Aug 22, 2014 at 2:15 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> I'm also -1, on the grounds that concrete types and ABCs are
>> different, and I think type hinting should be heavily biased towards
>> ABCs.
>
> We keep seeing this line of argument coming up, and while I've only
> been skimming, I haven't seen anyone comment that this applies to
> parameters but not so much to return values. A function might take a
> Sequence and iterate over it, and then return a concrete integer. Or
> it might take a file-like object and a string, and yield a series of
> strings. Or it might take some set of parameters, and return None
> unconditionally. There's nothing wrong with being concrete about
> return values.

For standalone functions, I agree. For methods and generics, I'd still
be somewhat wary of encouraging it.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From apieum at gmail.com  Thu Aug 21 18:43:00 2014
From: apieum at gmail.com (Gregory Salvan)
Date: Thu, 21 Aug 2014 18:43:00 +0200
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org>
 <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com>
Message-ID: <CAAZsQLC--MvZh6fsO-099pXDJtTN5Wg=3j5=bZVQvh9H=NSWRQ@mail.gmail.com>

> I'll admit that I could be off on some of this. It's hard to discuss
anything beyond the basics without either getting formal (which I don't
think anyone wants) or providing some
> realistic examples of where you might want to runtime-check against an
annotation (which I think would be very useful, if anyone has one).

The use case I see is functions composition.
I think, it's interesting to check at compile time if 2 functions can be
composed and at runtime if given arguments are valid.


2014-08-21 18:16 GMT+02:00 Andrew Barnert <abarnert at yahoo.com.dmarc.invalid>
:

> On Aug 20, 2014, at 14:57, Antoine Pitrou <antoine at python.org> wrote:
>
> > Le 19/08/2014 20:12, Guido van Rossum a ?crit :
> >>    Hmm, I've been saying this already, but my intuition is that it's a
> >>    bad idea to conflate *type descriptions* (what this proposal is
> >>    about) and actual *runtime types* (what ABCs are).
> >>
> >>
> >> But are they? I think the registration mechanism makes it clear that
> >> they aren't (necessarily) runtime types -- by linking a concrete type
> >> with an ABC through registration you are pretty clearly stating that the
> >> ABC *describes* (an aspect of) the concrete class without automatically
> >> adding any behavior from the ABC to it.
> >
> > Hmm... well, they are usable at runtime (if only for isinstance calls
> :-)). I admit my wording was a bit vague here. But our type descriptions
> should be able to express more information than ABCs currently do. For
> example, I don't how you'd express the idea of a "mapping from str to int"
> using the current Mapping ABC, while retaining the runtime properties of
> the Mapping class.
>
> Someone (I think Guido, but I can't find it) gave a concrete proposal here
> (which happens to match how MyPy already works): you make the collections
> ABCs implement subscripting behavior exactly the way the typing classes
> do--that is, they just return self. This means that at compile time
> Mapping[str, int] can be used as a static type hint that a function returns
> mappings from str to int, while a runtime isinstance check only verifies
> that the thing in question is a Mapping.
>
> This avoids thorny questions like distinguishing covariant from
> contravariant contexts at runtime (which I don't think is solvable without
> a bunch of separate isinstance functions, and a more complicated
> __subclasshook__ protocol, but I could be wrong).
>
> Also, there's no reason this couldn't be added in 3.5, and then an
> enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once
> someone works out those issues. (So isinstance still ignores the
> parameters, but iscovariant checks them covariantly, etc.) That wouldn't
> risk breaking 3.5 code in 3.6.
>
> Also, keep in mind that, just because you _can_ write isinstance(d,
> Mapping[str, int]) and that might be confusing, that doesn't mean there's
> any reason you _would_ write that. It will only come up in metaprogramming
> contexts (e.g., where you're generating a function at runtime to match some
> type argument). Maybe you could argue that this means the compile-time
> machinery should be available at runtime, but I don't think you could argue
> that this means the compile-time types shouldn't be available at runtime.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140821/14b51e37/attachment-0001.html>

From antoine at python.org  Thu Aug 21 19:24:15 2014
From: antoine at python.org (Antoine Pitrou)
Date: Thu, 21 Aug 2014 13:24:15 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
 <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>
Message-ID: <lt59vv$96v$1@ger.gmane.org>


Le 21/08/2014 12:15, Nick Coghlan a ?crit :
>
> Given:
>
>      Sequence[Number]
>      Sequence[Number|None]
>
> There's still at least three options for extending the syntax even further:
>
>      Sequence[Number](EXPR)
>      Sequence[Number:EXPR]
>      Sequence[Number,EXPR]
>
> You'd probably want to use strings at that point, for the same reasons
> PyContracts does. For example:
>
>      Number,"N>0" # Positive number
>      Sequence[Number,"N>0"] # Sequence of positive numbers
>      Sequence[Number],"N>0" # Non-empty sequence

All of those are horrible and un-Pythonic, though.
Python has one of the most powerful and user-friendly function call 
syntaxes around, why reinvent something clearly inferior and alien?

 > I doubt we'll ever get to that point, though. Type hinting and runtime
 > assertions aren't the same thing.

It's less about defining all those use cases up front and more about not 
painting ourselves in a corner, though.

Regards

Antoine.



From abarnert at yahoo.com  Thu Aug 21 20:36:44 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 21 Aug 2014 11:36:44 -0700
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CAAZsQLC--MvZh6fsO-099pXDJtTN5Wg=3j5=bZVQvh9H=NSWRQ@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org> <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com>
 <CAAZsQLC--MvZh6fsO-099pXDJtTN5Wg=3j5=bZVQvh9H=NSWRQ@mail.gmail.com>
Message-ID: <BE14B4CD-209C-48DF-A270-0C6DD49E9995@yahoo.com>

On Aug 21, 2014, at 9:43, Gregory Salvan <apieum at gmail.com> wrote:

> 
> > I'll admit that I could be off on some of this. It's hard to discuss anything beyond the basics without either getting formal (which I don't think anyone wants) or providing some 
> > realistic examples of where you might want to runtime-check against an annotation (which I think would be very useful, if anyone has one).
> 
> The use case I see is functions composition.
> I think, it's interesting to check at compile time if 2 functions can be composed and at runtime if given arguments are valid.

OK, how about a _concrete_ use case? Function composition is rare enough in Python (we don't even have a compose in functools, much less as an operator or builtin, and how often do you miss it?) that I'm finding it hard to imagine where you'd ever want to, e.g., accept a function that returns Sequence[int] and rcompose it with a function that takes Iterable[Number] and try to validate those compile-time type annotations at runtime.

For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable?

> 2014-08-21 18:16 GMT+02:00 Andrew Barnert <abarnert at yahoo.com.dmarc.invalid>:
>> On Aug 20, 2014, at 14:57, Antoine Pitrou <antoine at python.org> wrote:
>> 
>> > Le 19/08/2014 20:12, Guido van Rossum a ?crit :
>> >>    Hmm, I've been saying this already, but my intuition is that it's a
>> >>    bad idea to conflate *type descriptions* (what this proposal is
>> >>    about) and actual *runtime types* (what ABCs are).
>> >>
>> >>
>> >> But are they? I think the registration mechanism makes it clear that
>> >> they aren't (necessarily) runtime types -- by linking a concrete type
>> >> with an ABC through registration you are pretty clearly stating that the
>> >> ABC *describes* (an aspect of) the concrete class without automatically
>> >> adding any behavior from the ABC to it.
>> >
>> > Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class.
>> 
>> Someone (I think Guido, but I can't find it) gave a concrete proposal here (which happens to match how MyPy already works): you make the collections ABCs implement subscripting behavior exactly the way the typing classes do--that is, they just return self. This means that at compile time Mapping[str, int] can be used as a static type hint that a function returns mappings from str to int, while a runtime isinstance check only verifies that the thing in question is a Mapping.
>> 
>> This avoids thorny questions like distinguishing covariant from contravariant contexts at runtime (which I don't think is solvable without a bunch of separate isinstance functions, and a more complicated __subclasshook__ protocol, but I could be wrong).
>> 
>> Also, there's no reason this couldn't be added in 3.5, and then an enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once someone works out those issues. (So isinstance still ignores the parameters, but iscovariant checks them covariantly, etc.) That wouldn't risk breaking 3.5 code in 3.6.
>> 
>> Also, keep in mind that, just because you _can_ write isinstance(d, Mapping[str, int]) and that might be confusing, that doesn't mean there's any reason you _would_ write that. It will only come up in metaprogramming contexts (e.g., where you're generating a function at runtime to match some type argument). Maybe you could argue that this means the compile-time machinery should be available at runtime, but I don't think you could argue that this means the compile-time types shouldn't be available at runtime.
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140821/f9cd7c70/attachment.html>

From scottyblair at gmail.com  Thu Aug 21 21:23:16 2014
From: scottyblair at gmail.com (Scott Blair)
Date: Thu, 21 Aug 2014 14:23:16 -0500
Subject: [Python-ideas] Idea
Message-ID: <CAPeKT0Xg6r6wYcXeDQXcK1jwmK6nJFqumsCeUQBzU=BrZ7okfw@mail.gmail.com>

Add continue to a try, except statement to continue where the code threw an
error. Or a function of exceptions to get the line number at which it was
thrown along with a way to go back to that line.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140821/cf766d24/attachment.html>

From p.f.moore at gmail.com  Thu Aug 21 21:38:07 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 21 Aug 2014 20:38:07 +0100
Subject: [Python-ideas] Idea
In-Reply-To: <CAPeKT0Xg6r6wYcXeDQXcK1jwmK6nJFqumsCeUQBzU=BrZ7okfw@mail.gmail.com>
References: <CAPeKT0Xg6r6wYcXeDQXcK1jwmK6nJFqumsCeUQBzU=BrZ7okfw@mail.gmail.com>
Message-ID: <CACac1F_=xB99ScLyaoZhtQmHDbpBWLC0oYnAiKK+3WKe+MHtig@mail.gmail.com>

On 21 August 2014 20:23, Scott Blair <scottyblair at gmail.com> wrote:
> Add continue to a try, except statement to continue where the code threw an
> error. Or a function of exceptions to get the line number at which it was
> thrown along with a way to go back to that line.

I'm pretty sure this has been suggested before. The big problem is
that an exception can exit many levels of function invocation before
being caught. Restarting function call frames that have been exited is
a non-trivial exercise in general, and for all practical purposes not
possible in CPython. If you're interested in details, Stackless Python
does something similar, and for a more theoretical discussion you can
start at http://en.wikipedia.org/wiki/Continuation

Paul

From tjreedy at udel.edu  Thu Aug 21 22:19:33 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 21 Aug 2014 16:19:33 -0400
Subject: [Python-ideas] Idea
In-Reply-To: <CAPeKT0Xg6r6wYcXeDQXcK1jwmK6nJFqumsCeUQBzU=BrZ7okfw@mail.gmail.com>
References: <CAPeKT0Xg6r6wYcXeDQXcK1jwmK6nJFqumsCeUQBzU=BrZ7okfw@mail.gmail.com>
Message-ID: <lt5k9o$e6f$2@ger.gmane.org>

On 8/21/2014 3:23 PM, Scott Blair wrote:
> Add continue to a try, except statement to continue where the code threw
> an error.

As a practical matter, the way to do this now is

try:
   <statement>
except SomeError:
   pass

Anything more has technical difficulties discussed by Paul More.

-- 
Terry Jan Reedy


From apieum at gmail.com  Thu Aug 21 22:58:50 2014
From: apieum at gmail.com (Gregory Salvan)
Date: Thu, 21 Aug 2014 22:58:50 +0200
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <BE14B4CD-209C-48DF-A270-0C6DD49E9995@yahoo.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org>
 <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com>
 <CAAZsQLC--MvZh6fsO-099pXDJtTN5Wg=3j5=bZVQvh9H=NSWRQ@mail.gmail.com>
 <BE14B4CD-209C-48DF-A270-0C6DD49E9995@yahoo.com>
Message-ID: <CAAZsQLA3KQ3eU_1dMHFQBfrJs2-MrijY6X2fDvAo9q_kQ1-MOA@mail.gmail.com>

> OK, how about a _concrete_ use case? Function composition is rare enough
in Python (we don't even have a compose in functools, much less as an
operator or builtin,...

True, explicit composition is actually uncommon practice in python, whereas
there are lots of libs: toolz, funcy, fn.py, Pymonads....
Note that composition can be implicit: f(g(x))

Using mypy syntax was originally pushed with a presentation about what
python can learn from FP.
Personnally, I consider FP types have a different meaning than OO types, I
feel them more as a way to describe programs flows (by func de/composition)
and that's something I find really usefull in some cases (mainly for
parsing and parallel computing).
That's what I expect python would learn from FP and why I'm delighted about
this proposition. (my opinion doesn't matter but +1000 for the proposition
:)

> and, how often do you miss it ?
sufficiently to have written my own tool, which I started to extract from a
project few days before Guido announces his mypy proposal. (in development
but usable: https://github.com/apieum/lawvere )


> that I'm finding it hard to imagine where you'd ever want to, e.g.,
accept a function that returns Sequence[int] and rcompose it with a
function that
> takes Iterable[Number] and try to validate those compile-time type
annotations at runtime.

I've not such cases in mind, I thought more of these ones:

from functools import singledispatch
@singledispatch
def div1000(a: intexcept0) -> int:

  return 1000 / a

@div1000.register
def divby0(a: Just(0)):

  log('something')...


In this case it's interesting to know if for ex. div1000 can be composed
with sub500... when compiling
and if it's 0 or not at runtime.
(OK the example is simple and you can use a condition inside div1000 but
you can have more complex cases of this type or want to use divby0
elsewhere)


> For that matter, even if you wanted to, how _would_ you validate that
something matches Iterable[Number] at runtime, beyond validating that it's
iterable?

I'm not sure to understand what you are asking.
I see Iterable[Number] as a kind of type constructor (like Just(0)) even if
it's cached. I would probably check it at runtime with isinstance.



2014-08-21 20:36 GMT+02:00 Andrew Barnert <abarnert at yahoo.com>:

> On Aug 21, 2014, at 9:43, Gregory Salvan <apieum at gmail.com> wrote:
>
>
> > I'll admit that I could be off on some of this. It's hard to discuss
> anything beyond the basics without either getting formal (which I don't
> think anyone wants) or providing some
> > realistic examples of where you might want to runtime-check against an
> annotation (which I think would be very useful, if anyone has one).
>
> The use case I see is functions composition.
> I think, it's interesting to check at compile time if 2 functions can be
> composed and at runtime if given arguments are valid.
>
>
> OK, how about a _concrete_ use case? Function composition is rare enough
> in Python (we don't even have a compose in functools, much less as an
> operator or builtin, and how often do you miss it?) that I'm finding it
> hard to imagine where you'd ever want to, e.g., accept a function that
> returns Sequence[int] and rcompose it with a function that takes
> Iterable[Number] and try to validate those compile-time type annotations at
> runtime.
>
> For that matter, even if you wanted to, how _would_ you validate that
> something matches Iterable[Number] at runtime, beyond validating that it's
> iterable?
>
> 2014-08-21 18:16 GMT+02:00 Andrew Barnert <
> abarnert at yahoo.com.dmarc.invalid>:
>
>> On Aug 20, 2014, at 14:57, Antoine Pitrou <antoine at python.org> wrote:
>>
>> > Le 19/08/2014 20:12, Guido van Rossum a ?crit :
>> >>    Hmm, I've been saying this already, but my intuition is that it's a
>> >>    bad idea to conflate *type descriptions* (what this proposal is
>> >>    about) and actual *runtime types* (what ABCs are).
>> >>
>> >>
>> >> But are they? I think the registration mechanism makes it clear that
>> >> they aren't (necessarily) runtime types -- by linking a concrete type
>> >> with an ABC through registration you are pretty clearly stating that
>> the
>> >> ABC *describes* (an aspect of) the concrete class without automatically
>> >> adding any behavior from the ABC to it.
>> >
>> > Hmm... well, they are usable at runtime (if only for isinstance calls
>> :-)). I admit my wording was a bit vague here. But our type descriptions
>> should be able to express more information than ABCs currently do. For
>> example, I don't how you'd express the idea of a "mapping from str to int"
>> using the current Mapping ABC, while retaining the runtime properties of
>> the Mapping class.
>>
>> Someone (I think Guido, but I can't find it) gave a concrete proposal
>> here (which happens to match how MyPy already works): you make the
>> collections ABCs implement subscripting behavior exactly the way the typing
>> classes do--that is, they just return self. This means that at compile time
>> Mapping[str, int] can be used as a static type hint that a function returns
>> mappings from str to int, while a runtime isinstance check only verifies
>> that the thing in question is a Mapping.
>>
>> This avoids thorny questions like distinguishing covariant from
>> contravariant contexts at runtime (which I don't think is solvable without
>> a bunch of separate isinstance functions, and a more complicated
>> __subclasshook__ protocol, but I could be wrong).
>>
>> Also, there's no reason this couldn't be added in 3.5, and then an
>> enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once
>> someone works out those issues. (So isinstance still ignores the
>> parameters, but iscovariant checks them covariantly, etc.) That wouldn't
>> risk breaking 3.5 code in 3.6.
>>
>> Also, keep in mind that, just because you _can_ write isinstance(d,
>> Mapping[str, int]) and that might be confusing, that doesn't mean there's
>> any reason you _would_ write that. It will only come up in metaprogramming
>> contexts (e.g., where you're generating a function at runtime to match some
>> type argument). Maybe you could argue that this means the compile-time
>> machinery should be available at runtime, but I don't think you could argue
>> that this means the compile-time types shouldn't be available at runtime.
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140821/9eaa7395/attachment.html>

From rosuav at gmail.com  Fri Aug 22 02:37:31 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 22 Aug 2014 10:37:31 +1000
Subject: [Python-ideas] Idea
In-Reply-To: <CAPeKT0Xg6r6wYcXeDQXcK1jwmK6nJFqumsCeUQBzU=BrZ7okfw@mail.gmail.com>
References: <CAPeKT0Xg6r6wYcXeDQXcK1jwmK6nJFqumsCeUQBzU=BrZ7okfw@mail.gmail.com>
Message-ID: <CAPTjJmqcr9viKam+8PNsi+=S5Zk-W+5MQV82ZS+qw-7bsrExwA@mail.gmail.com>

On Fri, Aug 22, 2014 at 5:23 AM, Scott Blair <scottyblair at gmail.com> wrote:
> Add continue to a try, except statement to continue where the code threw an
> error. Or a function of exceptions to get the line number at which it was
> thrown along with a way to go back to that line.

You can get the line number from the traceback. However, I don't think
it's a good idea *in the general case* to jump back to the line that
started a problem, because you have no way of undoing anything that
got half done (Python is not transactional). But if you have a
specific situation in which you want to do this, chances are you can
wrap stuff up into a function. For instance, here's something that
would reasonably want a "RESUME" statement (like in BASIC):

try:
    install_file("base.py")
    install_file("subdir/foo.py")
    install_file("subdir/bar.py")
    install_file("otherdir/spam.py")
    install_file("otherdir/ham.py")
except PathNotFoundError:
    create_directory(whichever_one_is_missing)
    resume

But this is better served by a simple wrapper:

def install_and_create_dir(dir,fn):
    while "infinite retry"
        try:
            install_file(dir+"/"+fn)
            return
        except PathNotFoundError:
            create_directory(dir)
install_file("base.py")
install_and_create_dir("subdir","foo.py")
install_and_create_dir("subdir","bar.py")
install_and_create_dir("otherdir","spam.py")
install_and_create_dir("otherdir","ham.py")

Do you have a concrete use-case that begs for a RESUME statement and
can't easily be refactored into a (possibly nested) function?

ChrisA

From ncoghlan at gmail.com  Fri Aug 22 02:40:37 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Aug 2014 10:40:37 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lt59vv$96v$1@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
 <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
 <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>
 <lt59vv$96v$1@ger.gmane.org>
Message-ID: <CADiSq7c60OWVxkXhY1Pkub-cVdfbHx4BArUg8JZ4dvdFY5ZZoA@mail.gmail.com>

On 22 Aug 2014 03:27, "Antoine Pitrou" <antoine at python.org> wrote:
>
>
> Le 21/08/2014 12:15, Nick Coghlan a ?crit :
>
>>
>> Given:
>>
>>      Sequence[Number]
>>      Sequence[Number|None]
>>
>> There's still at least three options for extending the syntax even
further:
>>
>>      Sequence[Number](EXPR)
>>      Sequence[Number:EXPR]
>>      Sequence[Number,EXPR]
>>
>> You'd probably want to use strings at that point, for the same reasons
>> PyContracts does. For example:
>>
>>      Number,"N>0" # Positive number
>>      Sequence[Number,"N>0"] # Sequence of positive numbers
>>      Sequence[Number],"N>0" # Non-empty sequence
>
>
> All of those are horrible and un-Pythonic, though.
> Python has one of the most powerful and user-friendly function call
syntaxes around, why reinvent something clearly inferior and alien?

I don't think it's possible to have "Pythonic" design by contract
(PyContracts makes a laudable effort, but I think its decorator based
syntax is cleaner than its annotation based one)

Decorators aren't going anywhere, so I'm comfortable with the idea of
complexity constraints on the type hint notation - anything that doesn't
fit into the type hint syntax can still go in a decorator, and get a handy
decorator name for readers to look up in the docs or on Google.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140822/78c58ca4/attachment.html>

From antoine at python.org  Fri Aug 22 03:24:17 2014
From: antoine at python.org (Antoine Pitrou)
Date: Thu, 21 Aug 2014 21:24:17 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CADiSq7c60OWVxkXhY1Pkub-cVdfbHx4BArUg8JZ4dvdFY5ZZoA@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
 <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>
 <lt59vv$96v$1@ger.gmane.org>
 <CADiSq7c60OWVxkXhY1Pkub-cVdfbHx4BArUg8JZ4dvdFY5ZZoA@mail.gmail.com>
Message-ID: <53F69BC1.30205@python.org>

Le 21/08/2014 20:40, Nick Coghlan a ?crit :
> On 22 Aug 2014 03:27, "Antoine Pitrou"
> <antoine at python.org
> <mailto:antoine at python.org>> wrote:
>  >
>  >
>  > Le 21/08/2014 12:15, Nick Coghlan a ?crit :
>  >
>  >>
>  >> Given:
>  >>
>  >>      Sequence[Number]
>  >>      Sequence[Number|None]
>  >>
>  >> There's still at least three options for extending the syntax even
> further:
>  >>
>  >>      Sequence[Number](EXPR)
>  >>      Sequence[Number:EXPR]
>  >>      Sequence[Number,EXPR]
>  >>
>  >> You'd probably want to use strings at that point, for the same reasons
>  >> PyContracts does. For example:
>  >>
>  >>      Number,"N>0" # Positive number
>  >>      Sequence[Number,"N>0"] # Sequence of positive numbers
>  >>      Sequence[Number],"N>0" # Non-empty sequence
>  >
>  >
>  > All of those are horrible and un-Pythonic, though.
>  > Python has one of the most powerful and user-friendly function call
> syntaxes around, why reinvent something clearly inferior and alien?
>
> I don't think it's possible to have "Pythonic" design by contract

We're not talking about design by contract, we're talking about type 
annotations. "A list of X integers" isn't exactly an extremely 
complicated notion. We shouldn't need weird convoluted syntaxes to 
express it.

Regards

Antoine.



From ethan at stoneleaf.us  Fri Aug 22 03:31:57 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 21 Aug 2014 18:31:57 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53F69BC1.30205@python.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
 <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>
 <lt59vv$96v$1@ger.gmane.org>
 <CADiSq7c60OWVxkXhY1Pkub-cVdfbHx4BArUg8JZ4dvdFY5ZZoA@mail.gmail.com>
 <53F69BC1.30205@python.org>
Message-ID: <53F69D8D.5030803@stoneleaf.us>

On 08/21/2014 06:24 PM, Antoine Pitrou wrote:
> Le 21/08/2014 20:40, Nick Coghlan a ?crit :
>> On 22 Aug 2014 03:27, "Antoine Pitrou"
>> <antoine at python.org
>> <mailto:antoine at python.org>> wrote:
>>  >
>>  >
>>  > Le 21/08/2014 12:15, Nick Coghlan a ?crit :
>>  >
>>  >>
>>  >> Given:
>>  >>
>>  >>      Sequence[Number]
>>  >>      Sequence[Number|None]
>>  >>
>>  >> There's still at least three options for extending the syntax even
>> further:
>>  >>
>>  >>      Sequence[Number](EXPR)
>>  >>      Sequence[Number:EXPR]
>>  >>      Sequence[Number,EXPR]
>>  >>
>>  >> You'd probably want to use strings at that point, for the same reasons
>>  >> PyContracts does. For example:
>>  >>
>>  >>      Number,"N>0" # Positive number
>>  >>      Sequence[Number,"N>0"] # Sequence of positive numbers
>>  >>      Sequence[Number],"N>0" # Non-empty sequence
>>  >
>>  >
>>  > All of those are horrible and un-Pythonic, though.
>>  > Python has one of the most powerful and user-friendly function call
>> syntaxes around, why reinvent something clearly inferior and alien?
>>
>> I don't think it's possible to have "Pythonic" design by contract
>
> We're not talking about design by contract, we're talking about type annotations. "A list of X integers" isn't exactly
> an extremely complicated notion. We shouldn't need weird convoluted syntaxes to express it.

I don't know for whom I am arguing and against whom I am arguing, I just want to remind us that a type can be quite a 
bit more complicated than int or str:

   class Counting(int):
       def __new__(cls, value):
           if value < 1:
               raise ValueError('Counting can only be positive')
           ...

--
~Ethan~

From abarnert at yahoo.com  Fri Aug 22 05:11:15 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 21 Aug 2014 20:11:15 -0700
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CAAZsQLA3KQ3eU_1dMHFQBfrJs2-MrijY6X2fDvAo9q_kQ1-MOA@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org> <02961BAE-4ACF-48A8-BAAB-217710F681BC@yahoo.com>
 <CAAZsQLC--MvZh6fsO-099pXDJtTN5Wg=3j5=bZVQvh9H=NSWRQ@mail.gmail.com>
 <BE14B4CD-209C-48DF-A270-0C6DD49E9995@yahoo.com>
 <CAAZsQLA3KQ3eU_1dMHFQBfrJs2-MrijY6X2fDvAo9q_kQ1-MOA@mail.gmail.com>
Message-ID: <1408677075.31030.YahooMailNeo@web181002.mail.ne1.yahoo.com>

On Thursday, August 21, 2014 1:59 PM, Gregory Salvan <apieum at gmail.com> wrote:

>> OK, how about a _concrete_ use case??
>
>
>I've not such cases in mind, I thought more of these ones:
>
>
>from functools import singledispatch 
>@singledispatch
>def div1000(a: intexcept0) -> int:
>
>return 1000 / a
>
>
>@div1000.register
>def divby0(a: Just(0)):
>
>log('something')...
>In this case it's interesting to know if for ex. div1000 can be composed with sub500... when compiling
>and if it's 0 or not at runtime.?


Antoine was against using the same types for runtime ABCs and compile-time type-checking, because?"I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class."


You want something that's more powerful than existing ABCs at runtime, and more powerful than existing typing.py classes at compile-time, but I don't see any reason why you can't write them as the _same_ something. In fact, your example seems to be doing exactly that.


So, it seems like you're arguing for the exact same position as the one you disagreed with? What am I missing?

>> For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable?

>I'm not sure to understand what you are asking. 
>I see Iterable[Number] as a kind of type constructor (like Just(0)) even if it's cached. I would probably check it at runtime with isinstance.

But how would a runtime isinstance check on Iterable[Number] work??Either it has to ignore the generic part and just test Iterable (which is exactly what Guido's suggestion entails, which Antoine, and I thought you, didn't like), or it has to iterate all of the values and check them all, leaving you an empty iterable after checking (which may take infinite time, to boot).

>2014-08-21 20:36 GMT+02:00 Andrew Barnert <abarnert at yahoo.com>:

>
>On Aug 21, 2014, at 9:43, Gregory Salvan <apieum at gmail.com> wrote:
>>
>>
>>
>>>> I'll admit that I could be off on some of this. It's hard to discuss 
anything beyond the basics without either getting formal (which I don't 
think anyone wants) or providing some 
>>>> realistic examples of where you 
might want to runtime-check against an annotation (which I think would 
be very useful, if anyone has one).
>>>
>>>
>>>The use case I see is functions composition.
>>>I think, it's interesting to check at compile time if 2 functions can be composed and at runtime if given arguments are valid.
>>>
>>
>>OK, how about a _concrete_ use case? Function composition is rare enough in Python (we don't even have a compose in functools, much less as an operator or builtin, and how often do you miss it?)?that I'm finding it hard to imagine where you'd ever want to, e.g., accept a function that returns Sequence[int] and rcompose it with a function that takes Iterable[Number] and try to validate those compile-time type annotations at runtime.
>>
>>
>>For that matter, even if you wanted to, how _would_ you validate that something matches Iterable[Number] at runtime, beyond validating that it's iterable?
>>
>>
>>2014-08-21 18:16 GMT+02:00 Andrew Barnert <abarnert at yahoo.com.dmarc.invalid>:
>>>
>>>On Aug 20, 2014, at 14:57, Antoine Pitrou <antoine at python.org> wrote:
>>>>
>>>>> Le 19/08/2014 20:12, Guido van Rossum a ?crit :
>>>>>>? ? Hmm, I've been saying this already, but my intuition is that it's a
>>>>>>? ? bad idea to conflate *type descriptions* (what this proposal is
>>>>>>? ? about) and actual *runtime types* (what ABCs are).
>>>>>>
>>>>>>
>>>>>> But are they? I think the registration mechanism makes it clear that
>>>>>> they aren't (necessarily) runtime types -- by linking a concrete type
>>>>>> with an ABC through registration you are pretty clearly stating that the
>>>>>> ABC *describes* (an aspect of) the concrete class without automatically
>>>>>> adding any behavior from the ABC to it.
>>>>>
>>>>> Hmm... well, they are usable at runtime (if only for isinstance calls :-)). I admit my wording was a bit vague here. But our type descriptions should be able to express more information than ABCs currently do. For example, I don't how you'd express the idea of a "mapping from str to int" using the current Mapping ABC, while retaining the runtime properties of the Mapping class.
>>>>
>>>>Someone (I think Guido, but I can't find it) gave a concrete proposal here (which happens to match how MyPy already works): you make the collections ABCs implement subscripting behavior exactly the way the typing classes do--that is, they just return self. This means that at compile time Mapping[str, int] can be used as a static type hint that a function returns mappings from str to int, while a runtime isinstance check only verifies that the thing in question is a Mapping.
>>>>
>>>>This avoids thorny questions like distinguishing covariant from contravariant contexts at runtime (which I don't think is solvable without a bunch of separate isinstance functions, and a more complicated __subclasshook__ protocol, but I could be wrong).
>>>>
>>>>Also, there's no reason this couldn't be added in 3.5, and then an enhanced runtime meaning given to parameterized ABCs at runtime in 3.6 once someone works out those issues. (So isinstance still ignores the parameters, but iscovariant checks them covariantly, etc.) That wouldn't risk breaking 3.5 code in 3.6.
>>>>
>>>>Also, keep in mind that, just because you _can_ write isinstance(d, Mapping[str, int]) and that might be confusing, that doesn't mean there's any reason you _would_ write that. It will only come up in metaprogramming contexts (e.g., where you're generating a function at runtime to match some type argument). Maybe you could argue that this means the compile-time machinery should be available at runtime, but I don't think you could argue that this means the compile-time types shouldn't be available at runtime.
>>>>
>>>>
>>>>_______________________________________________
>>>>Python-ideas mailing list
>>>>Python-ideas at python.org
>>>>https://mail.python.org/mailman/listinfo/python-ideas
>>>>Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>_______________________________________________
>>>Python-ideas mailing list
>>>Python-ideas at python.org
>>>https://mail.python.org/mailman/listinfo/python-ideas
>>>Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>https://mail.python.org/mailman/listinfo/python-ideas
>Code of Conduct: http://python.org/psf/codeofconduct/
>
>

From greg at krypto.org  Fri Aug 22 08:42:59 2014
From: greg at krypto.org (Gregory P. Smith)
Date: Thu, 21 Aug 2014 23:42:59 -0700
Subject: [Python-ideas] Decorators that execute once rather than every
 import (was: codec and decorator hacks)
Message-ID: <CAGE7PN+_jnu_QGXcuJFLjGc1i_EfK8WkyR+mPtXeTqO5VEL3fw@mail.gmail.com>

On Wed, Aug 20, 2014 at 7:25 AM, Donald Stufft <donald at stufft.io> wrote:

>
> mypy does have a codec that will ignore annotations on 2.x.
>

Note that all codec hacks and decorator hacks have a down side other than
being hacks: They significantly slow down program import and start time. At
least the results of the codec hack are cached when a .pyc is generated.
Decorator executions are not.

hmm.. could a flavor of decorator that is cacheable in .pyc's be created so
the cost is only paid once rather than for each time the module is imported?

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140821/069f5c51/attachment.html>

From steve at pearwood.info  Fri Aug 22 09:41:04 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 22 Aug 2014 17:41:04 +1000
Subject: [Python-ideas] Decorators that execute once rather than every
	import (was: codec and decorator hacks)
In-Reply-To: <CAGE7PN+_jnu_QGXcuJFLjGc1i_EfK8WkyR+mPtXeTqO5VEL3fw@mail.gmail.com>
References: <CAGE7PN+_jnu_QGXcuJFLjGc1i_EfK8WkyR+mPtXeTqO5VEL3fw@mail.gmail.com>
Message-ID: <20140822074103.GR25957@ando>

On Thu, Aug 21, 2014 at 11:42:59PM -0700, Gregory P. Smith wrote:
> On Wed, Aug 20, 2014 at 7:25 AM, Donald Stufft <donald at stufft.io> wrote:
> 
> >
> > mypy does have a codec that will ignore annotations on 2.x.
> >
> 
> Note that all codec hacks and decorator hacks have a down side other than
> being hacks: They significantly slow down program import and start time. At
> least the results of the codec hack are cached when a .pyc is generated.
> Decorator executions are not.

Decorator executions are no different from any other function call. They 
occur when you call them. If you only call the decorator once, at the 
top level of the module, they don't get called again when you import the 
module again. That's no different from any other code at the top-level:


[steve at ando ~]$ cat test_deco.py
def decorate(func):
    print("calling an expensive decorator...")
    func.__name__ = "decorated_%s" % func.__name__
    return func

@decorate
def spam(n):
    return "spam"*n

print("%s: %s" % (spam.__name__, spam(3)))

[steve at ando ~]$ python3.3
Python 3.3.0rc3 (default, Sep 27 2012, 18:44:58)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux
Type "help", "copyright", "credits" or "license" for more information.
py> import test_deco as d
calling an expensive decorator...
decorated_spam: spamspamspam
py> import test_deco  # importing again doesn't re-call the decorator
py> test_deco.spam.__name__
'decorated_spam'


> hmm.. could a flavor of decorator that is cacheable in .pyc's be created so
> the cost is only paid once rather than for each time the module is imported?

I'm not sure that you're trying to solve a real problem. Can you give 
an example?



-- 
Steven

From miguelglafuente at gmail.com  Fri Aug 22 09:47:34 2014
From: miguelglafuente at gmail.com (Rock Neurotiko)
Date: Fri, 22 Aug 2014 09:47:34 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53F69D8D.5030803@stoneleaf.us>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
 <lt2sh5$9vi$1@ger.gmane.org> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
 <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>
 <lt59vv$96v$1@ger.gmane.org>
 <CADiSq7c60OWVxkXhY1Pkub-cVdfbHx4BArUg8JZ4dvdFY5ZZoA@mail.gmail.com>
 <53F69BC1.30205@python.org> <53F69D8D.5030803@stoneleaf.us>
Message-ID: <CAAhaj_gTBpwquXxw2PyxAvBKPCiuM48j3MBqTXVAxQta3rwwkg@mail.gmail.com>

First, sorry for my (probably)  bad english :)

I think that everyone who wants to learn about types, type system, what is
static type, what is not, why function annotations (or some other, but I
prefer it) refering to types is being proposed, should learn a little bit
(you won't regret!)

1) This article (What to know before debating type systems) has been sent
before in this thread (I think) but is a must read:
http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/

2) Bob's talk in Europython last month "What can python learn from haskell"
at least until minute 20, that talks about this things.
http://pyvideo.org/video/3046/what-can-python-learn-from-haskell

3) If you love maths, some book of Benjamin C. Pierce (about type systems,
of course!), this three are so good (but hard). Ordered from low to high
level.
- Type Systems for programming languages.
- Types for programming languages.
- Advanced topics in Types and programming languages.

4) If want to learn more, search in google, there are so much
books/articles about types, and if you can learn about haskell types (or
Idris if you are crazy), better ;)


As last word, I want to give my opinion about the proposal.

I like the proposal, and I would love to see it in Python 3.x, I don't
think that function notations can have a better purpose.

Of course, what is being proposed won't be needed to run a python program,
and you will still can write your function notations whatever you want, but
have a standard proposal will help a lot to build tools that read the
function notations (like IDEs, linters, static type checkers, ...)

About the syntax, I love the mypy's syntax, and remember me a lot to other
languages like scala.

def primes_to_n(n: Int) => List[Int]

def primes_to_n(n: int) -> List[int]


This two function definitions are a function that get an int and return a
list of integers. First definition is Scala, second one is Python with
mypy's syntax.

I like much more the Haskell syntax, but is another world and I think that
python shouldn't have a syntax like that:

primes_to_n :: Int -> [Int]

About other syntax that I had seen in this thread:
- decorators: I really don't like that one, is compatible with python 2,
ok, but looks so bad (to me)
- docs: That's not bad at all, and can still exist with the typing syntax
to declare the types, but we have function annotations where the types are
much close to the definition, and that's definitively good.
- PyContracts: That's not bad at all, but just for check types in runtime,
much things (the main diferences with mypy) can't be done in static, for
example, how you know if this will work *before* execute it:

@contract
def f(x: 'list[3>](str)') -> 'list(str)'

f(input().split(" "))


That's a function that take a list of three or more strings, and is called
with the user input.
You can't know if that list will be over three elements if you don't run it
and get the input, and even those, you can't say that will be true all the
times you execute it.


That's just my opinion :)

Cheers!

-- 
Miguel Garc?a Lafuente - Rock Neurotiko

Do it, the devil is in the details.
The quieter you are, the more you are able to hear.
Happy Coding. Code with Passion, Decode with Patience.
If we make consistent effort, based on proper education, we can change the
world.

El contenido de este e-mail es privado, no se permite la revelacion del
contenido de este e-mail a gente ajena a ?l.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140822/bf807fda/attachment.html>

From mal at egenix.com  Fri Aug 22 10:10:22 2014
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 22 Aug 2014 10:10:22 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53F69BC1.30205@python.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>	<CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>	<CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>	<CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>	<CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>	<8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>	<77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>	<lt2sh5$9vi$1@ger.gmane.org>	<87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>	<CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>	<lt4rsr$a00$2@ger.gmane.org>	<CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>	<lt59vv$96v$1@ger.gmane.org>	<CADiSq7c60OWVxkXhY1Pkub-cVdfbHx4BArUg8JZ4dvdFY5ZZoA@mail.gmail.com>
 <53F69BC1.30205@python.org>
Message-ID: <53F6FAEE.9010705@egenix.com>

On 22.08.2014 03:24, Antoine Pitrou wrote:
> Le 21/08/2014 20:40, Nick Coghlan a ?crit :
>> On 22 Aug 2014 03:27, "Antoine Pitrou"
>> <antoine at python.org
>> <mailto:antoine at python.org>> wrote:
>>  >
>>  >
>>  > Le 21/08/2014 12:15, Nick Coghlan a ?crit :
>>  >
>>  >>
>>  >> Given:
>>  >>
>>  >>      Sequence[Number]
>>  >>      Sequence[Number|None]
>>  >>
>>  >> There's still at least three options for extending the syntax even
>> further:
>>  >>
>>  >>      Sequence[Number](EXPR)
>>  >>      Sequence[Number:EXPR]
>>  >>      Sequence[Number,EXPR]
>>  >>
>>  >> You'd probably want to use strings at that point, for the same reasons
>>  >> PyContracts does. For example:
>>  >>
>>  >>      Number,"N>0" # Positive number
>>  >>      Sequence[Number,"N>0"] # Sequence of positive numbers
>>  >>      Sequence[Number],"N>0" # Non-empty sequence
>>  >
>>  >
>>  > All of those are horrible and un-Pythonic, though.
>>  > Python has one of the most powerful and user-friendly function call
>> syntaxes around, why reinvent something clearly inferior and alien?
>>
>> I don't think it's possible to have "Pythonic" design by contract
> 
> We're not talking about design by contract, we're talking about type annotations. "A list of X
> integers" isn't exactly an extremely complicated notion. We shouldn't need weird convoluted syntaxes
> to express it.

I agree with Antoine. Those EXPRs should not be needed for
type annotations. The above is mixing type information with
runtime information.

All a linter needs to know is that the result of operation1(type1)
is of type2, i.e. the information about certain features of those
types is implicit in the type name and the information about whether
types can be adapted to each other.

If the linter then finds that the result of operation1() is
being used as input for operation2(), which only supports
type1 arguments, and it doesn't have information about how
to adapt type2 to type1, it comes back with an error message.

It's all abstract. Trying to conflate arbitrary type properties
into the syntax is not needed - after all, type annotations are
not meant for runtime checks.

Regarding the whole idea of inline type annotations, I'm at most +0
on those. They add more complexity to the language and make it less
readable for little added value.

Making the type annotations external is much better in this respect
and I'm sure IDEs could even help with this by providing nice
interfaces for adding the type information. They could semi-
automate the process and make suggestions based on the functions
used inside the functions/method body.

Now, since these tools would have to read and write the annotations
programmatically, the syntax does not have to be nicely readable or
Pythonic. It could just as well be a YAML/JSON/XML file.

But hey, I don't want to spoil all the fun in defining a little
language for type annotations, so please do continue to have
fun defining Python .pyh header files ;-)

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 22 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2014-09-19: PyCon UK 2014, Coventry, UK ...                28 days to go
2014-09-30: Python Meeting Duesseldorf ...                 39 days to go

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ram.rachum at gmail.com  Fri Aug 22 13:40:49 2014
From: ram.rachum at gmail.com (Ram Rachum)
Date: Fri, 22 Aug 2014 04:40:49 -0700 (PDT)
Subject: [Python-ideas] Allow getting all the items out of a
	`threading.local`
Message-ID: <72de8dd9-4ce3-48a9-a22c-7c6dcf4b6d2a@googlegroups.com>

I propose adding a method to `threading.local` that allows to check the 
values stored for any and all threads.

I asked on Stack Overflow how to do 
that: http://stackoverflow.com/questions/25435908/python-getting-all-the-items-out-of-a-threading-local

But I got the answer that it's only possible on the pure Python 
implementation, not the C one. And it's quite ugly too.

I propose a method be added to both the Python implementation and the C 
implementation of `threading.local` that allows checking the values stored 
for all threads.


Thanks,
Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140822/cf1716b8/attachment.html>

From songofacandy at gmail.com  Fri Aug 22 13:50:52 2014
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 22 Aug 2014 20:50:52 +0900
Subject: [Python-ideas] Decorators that execute once rather than every
 import (was: codec and decorator hacks)
In-Reply-To: <20140822074103.GR25957@ando>
References: <CAGE7PN+_jnu_QGXcuJFLjGc1i_EfK8WkyR+mPtXeTqO5VEL3fw@mail.gmail.com>
 <20140822074103.GR25957@ando>
Message-ID: <CAEfz+TxnAVRPjido4Pt64H7kvPsM=6wNK1H94deL1AYbQmprJw@mail.gmail.com>

Longer import time makes CLI applications slower.
It's a real problem for people writing command line application.
For example, both of mercurial and Bazaar has lazy import system to
minimize import.

On Fri, Aug 22, 2014 at 4:41 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Thu, Aug 21, 2014 at 11:42:59PM -0700, Gregory P. Smith wrote:
>> On Wed, Aug 20, 2014 at 7:25 AM, Donald Stufft <donald at stufft.io> wrote:
>>
>> >
>> > mypy does have a codec that will ignore annotations on 2.x.
>> >
>>
>> Note that all codec hacks and decorator hacks have a down side other than
>> being hacks: They significantly slow down program import and start time. At
>> least the results of the codec hack are cached when a .pyc is generated.
>> Decorator executions are not.
>
> Decorator executions are no different from any other function call. They
> occur when you call them. If you only call the decorator once, at the
> top level of the module, they don't get called again when you import the
> module again. That's no different from any other code at the top-level:
>
>
> [steve at ando ~]$ cat test_deco.py
> def decorate(func):
>     print("calling an expensive decorator...")
>     func.__name__ = "decorated_%s" % func.__name__
>     return func
>
> @decorate
> def spam(n):
>     return "spam"*n
>
> print("%s: %s" % (spam.__name__, spam(3)))
>
> [steve at ando ~]$ python3.3
> Python 3.3.0rc3 (default, Sep 27 2012, 18:44:58)
> [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> py> import test_deco as d
> calling an expensive decorator...
> decorated_spam: spamspamspam
> py> import test_deco  # importing again doesn't re-call the decorator
> py> test_deco.spam.__name__
> 'decorated_spam'
>
>
>> hmm.. could a flavor of decorator that is cacheable in .pyc's be created so
>> the cost is only paid once rather than for each time the module is imported?
>
> I'm not sure that you're trying to solve a real problem. Can you give
> an example?
>
>
>
> --
> Steven
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
INADA Naoki  <songofacandy at gmail.com>

From brett at python.org  Fri Aug 22 14:53:25 2014
From: brett at python.org (Brett Cannon)
Date: Fri, 22 Aug 2014 12:53:25 +0000
Subject: [Python-ideas] Optional Static Typing -- the Python Way
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org>
 <CAP1=2W70P_Mdeup3bj6om+0N_HrR2cv-QMXHNKw5OiU79Je4Ww@mail.gmail.com>
 <lt516p$j58$1@ger.gmane.org>
Message-ID: <CAP1=2W7_qmZdGZ6WkDuaK1koP8G6eeZ_No94iArBiWmaXkTUgQ@mail.gmail.com>

On Thu Aug 21 2014 at 10:55:47 AM Antoine Pitrou <antoine at python.org> wrote:

>
> Le 21/08/2014 10:30, Brett Cannon a ?crit :
> >
> > Isn't that the magic of __instancecheck__/__subclasscheck__? If
> > typing.py does the right thing here then it shouldn't matter.
>
> What is "the right thing"?
>

I don't feel like scrolling all the way back through this thread to find
your original email, but "the right thing" is "whatever it takes". =) My
point is that I don't think there is any technical reason why we can't make
ABCs work.


>
> > You could
> > use __init__ or __getitem__ to construct an object who upon
> > introspection provides the details you want and do what you expect, and
> > then override the appropriate methods so that isinstance() checks check
> > against the ABC instead of the type class itself (heck we could make the
> > base type class require that you inherit from an ABC to promote working
> > with ABCs instead of concrete classes).
>
> So we agree that type descriptions should be separate things from ABCs,
> or not?
>

We don't agree. I'm fine with reusing ABCs. My only point is that if you
really don't want to tweak ABCs to work directly, typing.py can still use
ABCs as a subclass and then add whatever is necessary on top of them.


>
> > I have to also say that I like using ABCs because `from collections
> > import abc as a` leads to reading parameters like "x is a.Mapping" which
> > is linguistically quaint. =)
>
> Except that "x is a.Mapping" is False with a normal Python.
>

You're reading that too literally as code and not as a descriptive
sentence. What I meant to convey is that code `def foo(x: a.Mapping)` can
be read in your inner voice as "function foo has a parameter x which is
a.Mapping", where "a.Mapping" somewhat reads like "a Mapping".

-Brett


>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140822/9af5f5b7/attachment.html>

From skip at pobox.com  Fri Aug 22 15:42:00 2014
From: skip at pobox.com (Skip Montanaro)
Date: Fri, 22 Aug 2014 08:42:00 -0500
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
Message-ID: <CANc-5UyHg_V5dx6qSc0ZXX6y-KeNRXsRK6kYBr3Xa==X88K3JA@mail.gmail.com>

There's been a lot to read in this and related threads over the past nine
days. (According to Gmail. It qualitatively seems much longer to me.) I
think I've followed along reasonably well. Forgive me if I missed the
answers to these questions. The proposal on the table is to adopt MyPy's
type annotations for Python 3.mumble. I presume MyPy already supports them.

1. Can MyPy be used today as a standalone static type checker for Python
3.x code without actually compiling anything? That is, can I just sprinkle
type annotations into my code and run a front-end pass of MyPy much the
same way I'd run pylint, vulture, flake8, or other lint-ish program?

2. Assuming the answer to #1 is "yes," if you start sprinkling type
annotations into your code and running "mypy *.py", will it tell you when
it needs a missing type annotation to more fully check things, or will it
silently process code without annotations and not let you know that it's
not really checking much?

3. Will we get into a phase like the early days of "const" in ANSI C where
addition of "const" in one location in your existing code base forced you
into a never-ending iterations of adding const all over the place? I forget
what that was called ("const propagation"?), but I recall that it generally
wasn't a fun activity.

4. If the answer to #1 is "no," are there plans to produce a front-end-only
MyPy which can be used as a linter for Python code?

Thx,

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140822/1009e6c5/attachment.html>

From antoine at python.org  Fri Aug 22 16:36:26 2014
From: antoine at python.org (Antoine Pitrou)
Date: Fri, 22 Aug 2014 10:36:26 -0400
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CAP1=2W7_qmZdGZ6WkDuaK1koP8G6eeZ_No94iArBiWmaXkTUgQ@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org>
 <CAP1=2W70P_Mdeup3bj6om+0N_HrR2cv-QMXHNKw5OiU79Je4Ww@mail.gmail.com>
 <lt516p$j58$1@ger.gmane.org>
 <CAP1=2W7_qmZdGZ6WkDuaK1koP8G6eeZ_No94iArBiWmaXkTUgQ@mail.gmail.com>
Message-ID: <53F7556A.205@python.org>

Le 22/08/2014 08:53, Brett Cannon a ?crit :
>      > I have to also say that I like using ABCs because `from collections
>      > import abc as a` leads to reading parameters like "x is
>     a.Mapping" which
>      > is linguistically quaint. =)
>
>     Except that "x is a.Mapping" is False with a normal Python.
>
>
> You're reading that too literally as code and not as a descriptive
> sentence. What I meant to convey is that code `def foo(x: a.Mapping)`
> can be read in your inner voice as "function foo has a parameter x which
> is a.Mapping", where "a.Mapping" somewhat reads like "a Mapping".

So? That would be true with anything else that isn't an ABC, as long as 
you import the relevant module with the name "a" (which is silly, but 
hey :-)).

   def foo(x: a.List(a.Number)) -> a.Number

Regards

Antoine.



From antoine at python.org  Fri Aug 22 16:38:58 2014
From: antoine at python.org (Antoine Pitrou)
Date: Fri, 22 Aug 2014 10:38:58 -0400
Subject: [Python-ideas] Allow getting all the items out of a
	`threading.local`
In-Reply-To: <72de8dd9-4ce3-48a9-a22c-7c6dcf4b6d2a@googlegroups.com>
References: <72de8dd9-4ce3-48a9-a22c-7c6dcf4b6d2a@googlegroups.com>
Message-ID: <lt7km2$t69$2@ger.gmane.org>

Le 22/08/2014 07:40, Ram Rachum a ?crit :
> I propose adding a method to `threading.local` that allows to check the
> values stored for any and all threads.
>
> I asked on Stack Overflow how to do
> that: http://stackoverflow.com/questions/25435908/python-getting-all-the-items-out-of-a-threading-local
>
> But I got the answer that it's only possible on the pure Python
> implementation, not the C one. And it's quite ugly too.
>
> I propose a method be added to both the Python implementation and the C
> implementation of `threading.local` that allows checking the values
> stored for all threads.

Do other languages have such a functionality?
As for feasibility, the best way to check is to try to write a patch :-)

Regards

Antoine.



From bob at redivi.com  Fri Aug 22 16:30:36 2014
From: bob at redivi.com (Bob Ippolito)
Date: Fri, 22 Aug 2014 07:30:36 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CANc-5UyHg_V5dx6qSc0ZXX6y-KeNRXsRK6kYBr3Xa==X88K3JA@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CANc-5UyHg_V5dx6qSc0ZXX6y-KeNRXsRK6kYBr3Xa==X88K3JA@mail.gmail.com>
Message-ID: <CACwMPm9EkDT98gM-9_fwE_NqEwg5qA7QaR-nVN4WVYjowkc3Eg@mail.gmail.com>

On Friday, August 22, 2014, Skip Montanaro <skip at pobox.com> wrote:

> There's been a lot to read in this and related threads over the past nine
> days. (According to Gmail. It qualitatively seems much longer to me.) I
> think I've followed along reasonably well. Forgive me if I missed the
> answers to these questions. The proposal on the table is to adopt MyPy's
> type annotations for Python 3.mumble. I presume MyPy already supports them.
>
> 1. Can MyPy be used today as a standalone static type checker for Python
> 3.x code without actually compiling anything? That is, can I just sprinkle
> type annotations into my code and run a front-end pass of MyPy much the
> same way I'd run pylint, vulture, flake8, or other lint-ish program?
>

Yes. mypy -S performs checking without execution.

2. Assuming the answer to #1 is "yes," if you start sprinkling type
> annotations into your code and running "mypy *.py", will it tell you when
> it needs a missing type annotation to more fully check things, or will it
> silently process code without annotations and not let you know that it's
> not really checking much?
>

No. It happily and silently accepts dynamic code with no annotations. It
can't do much for you with that code, but it doesn't complain.

3. Will we get into a phase like the early days of "const" in ANSI C where
> addition of "const" in one location in your existing code base forced you
> into a never-ending iterations of adding const all over the place? I forget
> what that was called ("const propagation"?), but I recall that it generally
> wasn't a fun activity.
>

Nope.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140822/21f85fce/attachment.html>

From steve at pearwood.info  Fri Aug 22 18:15:01 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 23 Aug 2014 02:15:01 +1000
Subject: [Python-ideas] Optional static typing -- late to the party
In-Reply-To: <53F53BD5.5090902@stoneleaf.us>
References: <CADhgxgfGfAe-FdGebeqCt2+7-E=Dw1UrE5anK_qf2JS89sr7fQ@mail.gmail.com>
 <20140820123508.GK25957@ando> <85sikr1fwb.fsf@benfinney.id.au>
 <CAPTjJmpXZ+LH8SP-nZWN5cmU=BcpdA22Ckw6vmuYsUcmv1W5Gg@mail.gmail.com>
 <85lhqi29cz.fsf@benfinney.id.au> <20140821000954.GO25957@ando>
 <53F53BD5.5090902@stoneleaf.us>
Message-ID: <20140822161501.GU25957@ando>

On Wed, Aug 20, 2014 at 05:22:45PM -0700, Ethan Furman wrote:
> On 08/20/2014 05:09 PM, Steven D'Aprano wrote:
> >On Thu, Aug 21, 2014 at 06:29:16AM +1000, Ben Finney wrote:
> >>
> >>Remember that we're talking about type annotations that are for
> >>*static code checkers* to inspect. Docstrings will certainly be
> >>available there.
> >
> >True, but runtime checks may (or may not) be a part of the code checker.
> >At least, we should not rule that out.
> 
> This proposal is not about "a code checker" but about *static* typing.
> 
> Seems to be a lot of people forgetting that *static* (at least in this 
> case), means not actually running the program -- hence, no loss of 
> docstrings.

I'm aware of that, but I'm just saying that we shouldn't rule out the 
possibility of tools which operate at runtime using those same 
annotations. If the annotations are in the docstring, then they cannot 
be used at runtime with -OO. Function annotations can be used both 
statically and dynamically, while docstring annotations cannot be relied 
on to be present at runtime.

  
-- 
Steven

From tjreedy at udel.edu  Sat Aug 23 01:36:53 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 22 Aug 2014 19:36:53 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAAhaj_gTBpwquXxw2PyxAvBKPCiuM48j3MBqTXVAxQta3rwwkg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
 <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>
 <lt59vv$96v$1@ger.gmane.org>
 <CADiSq7c60OWVxkXhY1Pkub-cVdfbHx4BArUg8JZ4dvdFY5ZZoA@mail.gmail.com>
 <53F69BC1.30205@python.org> <53F69D8D.5030803@stoneleaf.us>
 <CAAhaj_gTBpwquXxw2PyxAvBKPCiuM48j3MBqTXVAxQta3rwwkg@mail.gmail.com>
Message-ID: <lt8k7p$oau$2@ger.gmane.org>

On 8/22/2014 3:47 AM, Rock Neurotiko wrote:

> About the syntax, I love the mypy's syntax, and remember me a lot to
> other languages like scala.
>
> def primes_to_n(n: Int) => List[Int]
>
> def primes_to_n(n: int) -> List[int]

The Python version need ':' at the end, after the return type.
I don't think you are the first to omit it.

-- 
Terry Jan Reedy


From miguelglafuente at gmail.com  Sat Aug 23 01:50:49 2014
From: miguelglafuente at gmail.com (Rock Neurotiko)
Date: Sat, 23 Aug 2014 01:50:49 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lt8k7p$oau$2@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
 <lt2sh5$9vi$1@ger.gmane.org> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
 <CADiSq7fiCPrCTRtj6RmsrVTGXdue+2ZDuK6VnazLgHdQjW=acQ@mail.gmail.com>
 <lt59vv$96v$1@ger.gmane.org>
 <CADiSq7c60OWVxkXhY1Pkub-cVdfbHx4BArUg8JZ4dvdFY5ZZoA@mail.gmail.com>
 <53F69BC1.30205@python.org> <53F69D8D.5030803@stoneleaf.us>
 <CAAhaj_gTBpwquXxw2PyxAvBKPCiuM48j3MBqTXVAxQta3rwwkg@mail.gmail.com>
 <lt8k7p$oau$2@ger.gmane.org>
Message-ID: <CAAhaj_hSwKzO3z_Tkc0E1vd=ZU4OWNhC2qvtD3BiDgf02sZ=uQ@mail.gmail.com>

Yes, and the Scala version need a " = {}", I was talking just about how the
syntax of the types and annotations are, I omitted the ':' because it was
irrelevant.



2014-08-23 1:36 GMT+02:00 Terry Reedy <tjreedy at udel.edu>:

> On 8/22/2014 3:47 AM, Rock Neurotiko wrote:
>
>  About the syntax, I love the mypy's syntax, and remember me a lot to
>> other languages like scala.
>>
>> def primes_to_n(n: Int) => List[Int]
>>
>> def primes_to_n(n: int) -> List[int]
>>
>
> The Python version need ':' at the end, after the return type.
> I don't think you are the first to omit it.
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Miguel Garc?a Lafuente - Rock Neurotiko

Do it, the devil is in the details.
The quieter you are, the more you are able to hear.
Happy Coding. Code with Passion, Decode with Patience.
If we make consistent effort, based on proper education, we can change the
world.

El contenido de este e-mail es privado, no se permite la revelacion del
contenido de este e-mail a gente ajena a ?l.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140823/81c78063/attachment-0001.html>

From steve at pearwood.info  Sat Aug 23 03:36:47 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 23 Aug 2014 11:36:47 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lt4rsr$a00$2@ger.gmane.org>
References: <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
Message-ID: <20140823013646.GV25957@ando>

On Thu, Aug 21, 2014 at 09:23:39AM -0400, Antoine Pitrou wrote:

> I'm -1 on using __getitem__ for parametering types. It is indeed a hack 
> and its limitations will bite us harshly when wanting to refine the type 
> descriptions (for example to describe a constraint on a collection's size).

> >One last thing: let?s not add |List|, |TList|, etc. Just use |list| and
> >other existing types to generate new types.
> 
> I'm -1 on that, for exactly the same reason as above.

It's not 100% clear to me whether you're -1 on re-using |list| or -1 or 
adding |List|. I'm going to assume that you mean -1 on re-using list 
(see below).

In a later message, Antoine wrote:

> Python has one of the most powerful and user-friendly function call
> syntaxes around, why reinvent something clearly inferior and alien?

I wouldn't say it is alien. abc[xyz] is clearly Python syntax. Nor would 
I necessarily say it is inferior, although it is more limited. 
Deliberately so, I think. How complex will the static type declarations 
need to be for us to want to accept arbitrary keyword arguments, for 
example? I think limiting the syntax possible is a good thing, it will 
discourage people from trying to declare arbitrarily complex information 
in the type notations.

But another reason for disliking call syntax for this is that it gets 
confusing:

def function(param:Sequence(Spam, Eggs))->Mapping(Cheese, Toast):

suffers compared to this:

def function(param:Sequence[Spam, Eggs])->Mapping[Cheese, Toast]:

In the second case, the type annotations stand out more easily because 
they use square brackets instead of round.

As far as the question of list versus List, I don't believe we can use 
built-ins if we follow Antoine's suggestion of call syntax.

def function(param:list(int))->list(str):

for example, intends to declare param is a list of ints. But list(x) 
already has a meaning, and if x is an int, it raises TypeError.

param:List(int) at least can work, but now consider int. Perhaps we 
might like to specify a range of ints, from 0 to 9 (say). The obvious 
syntax would be:

param:List(int(0, 10))  # inclusive lower bound, exclusive upper bound

but again that already has meaning. If we want the flexibility of 
providing arguments to scalar types like int, I believe we need to use 
subscripting syntax:

param:List[int[0, 10]]


-1 on re-using call syntax: it is more confusing to read inside a 
parameter list, too flexible, and clashes with existing use calling 
types.

+1 on subscripting syntax: it is more easily distinguishable from 
calling, just flexible enough, and doesn't clash with existing 
functionality.



-- 
Steven

From antoine at python.org  Sat Aug 23 04:31:18 2014
From: antoine at python.org (Antoine Pitrou)
Date: Fri, 22 Aug 2014 22:31:18 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140823013646.GV25957@ando>
References: <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
Message-ID: <lt8udm$jbd$1@ger.gmane.org>


Le 22/08/2014 21:36, Steven D'Aprano a ?crit :
>
> It's not 100% clear to me whether you're -1 on re-using |list| or -1 or
> adding |List|. I'm going to assume that you mean -1 on re-using list
> (see below).

Indeed.

> In a later message, Antoine wrote:
>
>> Python has one of the most powerful and user-friendly function call
>> syntaxes around, why reinvent something clearly inferior and alien?
>
> I wouldn't say it is alien. abc[xyz] is clearly Python syntax.

But it is completely uncommon for anything else than subscripting and 
indexing containers. Python isn't known for reusing operators for 
completely unrelated things, e.g. it doesn't overload ">>" for I/O 
(thank goodness).

> How complex will the static type declarations
> need to be for us to want to accept arbitrary keyword arguments, for
> example? I think limiting the syntax possible is a good thing, it will
> discourage people from trying to declare arbitrarily complex information
> in the type notations.

I think this is an incredibly weak argument, and the proof is that it 
hasn't been brought for anything else in the stdlib. Why the typing 
module should be any different and use a deliberately repressive 
parameter-passing syntax is beyond me - even though it's an advanced, 
optional, feature and will be put in the hands of consenting adults.

> But another reason for disliking call syntax for this is that it gets
> confusing:
>
> def function(param:Sequence(Spam, Eggs))->Mapping(Cheese, Toast):
>
> suffers compared to this:
>
> def function(param:Sequence[Spam, Eggs])->Mapping[Cheese, Toast]:

If you find it confusing (which may simply be a matter of habit, I don't 
know), you can simply alias the types:

   spam_eggs_sequence = Sequence(Spam, Eggs)
   cheese_toast_mapping = Mapping(Cheese, Toast)

   def function(param: spam_eggs_sequence) -> cheese_toast_mapping:
       ...

Regardless of readability, I would expect people to create such aliases 
anyway, the same way people tend to create "typedefs" in other 
languages, because it helps condense and clarify which actual types are 
in use in a module.

> As far as the question of list versus List, I don't believe we can use
> built-ins if we follow Antoine's suggestion of call syntax.
>
> def function(param:list(int))->list(str):
>
> for example, intends to declare param is a list of ints. But list(x)
> already has a meaning, and if x is an int, it raises TypeError.

And indeed this is exactly why I said I was against using built-ins for 
this (or the existing ABCs) :-))

Really, for it to be powerful enough, the type description system has to 
use its own set of classes. It can't try to hack away existing builtins 
and ABCs in the hope of expressing different things with them, or it 
*will* hit a wall some day (and, IMO, sooner rather than later).

Regards

Antoine.



From steve at pearwood.info  Sat Aug 23 07:13:57 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 23 Aug 2014 15:13:57 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lt8udm$jbd$1@ger.gmane.org>
References: <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org>
Message-ID: <20140823051357.GX25957@ando>

On Fri, Aug 22, 2014 at 10:31:18PM -0400, Antoine Pitrou wrote:

> >>Python has one of the most powerful and user-friendly function call
> >>syntaxes around, why reinvent something clearly inferior and alien?
> >
> >I wouldn't say it is alien. abc[xyz] is clearly Python syntax.
> 
> But it is completely uncommon for anything else than subscripting and 
> indexing containers. 

And function call syntax is completely uncommon for anything else than 
calling functions (including types and methods). One way or the other, 
we're adding a new use, type declarations.


> Python isn't known for reusing operators for 
> completely unrelated things, e.g. it doesn't overload ">>" for I/O 
> (thank goodness).

I see you've forgotten Python 2's 

    print >>sys.stderr, x, y, z

:-)


> >How complex will the static type declarations
> >need to be for us to want to accept arbitrary keyword arguments, for
> >example? I think limiting the syntax possible is a good thing, it will
> >discourage people from trying to declare arbitrarily complex information
> >in the type notations.
> 
> I think this is an incredibly weak argument, and the proof is that it 
> hasn't been brought for anything else in the stdlib. 

We frequently reject syntax because it will be (allegedly) confusing or 
hard to read. E.g. try...except expressions, anonymous code blocks, 
multiline lambda. Especially multiline lambda. Even something which I 
think is a no-brainer, allowing the with statement to use parentheses 
around the context managers:

    with (this_manager() as m1,
          that_manager() as m2):
        ...

was criticised by Nick on the basis that if you can't fit the context 
managers all on one line, you ought to refactor your code. I'm sure if 
you check the rejected PEPs and Python-Ideas archives, you'll find 
many examples of my argument applied to the standard library.

I don't think it's unPythonic to argue that type hinting syntax should 
by default use an annotation style which is not arbitrarily complex. The 
annotations themselves will remain arbitrary Python expressions, so if 
you really need a complex type declaration, you can create a helper and 
call it with whatever signature you like.[1]


> Why the typing 
> module should be any different and use a deliberately repressive 
> parameter-passing syntax is beyond me - even though it's an advanced, 
> optional, feature and will be put in the hands of consenting adults.

I can't speak for the author of mypy, Jukka Lehtosalo, but for me, I 
think the answer is that type declarations make a *mini-language*, not 
full-blown Python. It is unclear to me just how powerful the type 
language will be, but surely we don't expect it to evaluate arbitrarily 
complex Python expressions at compile time? Do we?

I find it unlikely that we expect a static linter to make sense of this:

def bizarre(param:int if random.random() > 0.5 else str)->float:

except to say it returns a float and takes who-knows-what as argument. 
An extreme case, I grant, but I expect that there are going to be limits 
to what can be checked at compile time based on static analysis. I admit 
that I don't fully understand all the implications of static versus 
runtime type checking, but it doesn't seem unreasonable to expect static 
declarations to be considerably simpler than what can be expressed at 
runtime. Hence, a simpler mini-language is appropriate.


> Really, for it to be powerful enough, the type description system has to 
> use its own set of classes. It can't try to hack away existing builtins 
> and ABCs in the hope of expressing different things with them, or it 
> *will* hit a wall some day (and, IMO, sooner rather than later).

How powerful is "powerful enough"? Powerful enough for what?

You've probably heard of the famous "the compiler found my infinite 
loop", where the ML type checker was able to detect that code would 
never terminate. I find that remarkable, and even more astonishing that, 
according to some, solving the halting problem is "nothing special" for 
type systems.[2] But many languages make do with less powerful type systems, 
and tools such as IDEs and editors surely don't need something that 
powerful. So:

- how powerful do we expect the type system to be?

- and how much of that power needs to be expressed using 
  function annotations?

The second question is critical, because there are alternatives to 
function annotations: decorators, docstring annotations, and external 
stub files.




[1] Although if you do so, it is not clear to me how much the static 
checker will be able to use it.


[2] http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/


-- 
Steven

From jeanpierreda at gmail.com  Sat Aug 23 10:09:56 2014
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sat, 23 Aug 2014 01:09:56 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140823051357.GX25957@ando>
References: <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
 <lt2sh5$9vi$1@ger.gmane.org> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
Message-ID: <CABicbJL=knPEc7qMWN6ERM+=ojaN=06f0t20d33ekEihoJv-cA@mail.gmail.com>

On Fri, Aug 22, 2014 at 10:13 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> You've probably heard of the famous "the compiler found my infinite
> loop", where the ML type checker was able to detect that code would
> never terminate. I find that remarkable, and even more astonishing that,
> according to some, solving the halting problem is "nothing special" for
> type systems.[2] But many languages make do with less powerful type systems,
> and tools such as IDEs and editors surely don't need something that
> powerful.
--snip--
> [2] http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/

You seem to be mischaracterizing ML's type system as something
magically powerful. The ML type checker can be implemented in a few
dozen lines of Python code, and resolved efficiently. ML does type
inference by looking at the constraints on arguments inside the body
of a function. Infinite recursion often accidentally results in you
omitting constraints that you didn't meant to. For example, "def
foo(): return foo()" will be straight up marked as not returning at
all, because no additional constraints are placed on the return value,
which can only mean that it recurses infinitely. Many infinite loops
will not be marked, though. It's a type system, not magic.

(The ML type system is actually substantially simpler than anything
one is likely to come up with to describe Python, because ML didn't
have an object system or structural subtyping.)

As for the comment that this is "nothing special", taken literally,
they said something wrong -- solving the halting problem for Turing
complete languages is impossible, static types or no.  The only way
static types can help is by allowing you to define a (sub-)language
that is not Turing-complete. I think they were alluding to the fact
that, counter to many peoples' intuition, creating a type system that
allows you to detect some of the code that halts isn't a matter of
creating a powerful type system. If you add even the simplest type
system to the lambda calculus, poof, now every well-typed expression
can only be "executed" finitely many steps!


As for whether IDEs need to make something complicated enough that it
can positively identify code that does or doesn't halt, I'd leave that
to the IDE folks. C and Java tools do this sort of thing all the time,
for things like detecting dead code. Why shouldn't Python tools? (This
isn't a feature of the type system so much as control flow analysis
under a given type system, so you're right that it doesn't belong in
the types themselves.)

-- Devin

From g.brandl at gmx.net  Sat Aug 23 11:21:58 2014
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 23 Aug 2014 11:21:58 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lt4rsr$a00$2@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org>
Message-ID: <lt9mfn$tpn$1@ger.gmane.org>

On 08/21/2014 03:23 PM, Antoine Pitrou wrote:

>> One last thing: let?s not add |List|, |TList|, etc. Just use |list| and
>> other existing types to generate new types.
> 
> I'm -1 on that, for exactly the same reason as above.

Another reason is that it makes it even easier for people to use the
usually too narrow and therefore annoying

   list[int]

instead of the correct

   Iterable[int] or Sequence[int]


Another thought is whether, assuming

   class MyList(list):
       ...

the type  "MyList[int]"  would do the expected thing.  (And if not,
how you would write the correct type generation code.)


Georg


From antoine at python.org  Sat Aug 23 15:44:02 2014
From: antoine at python.org (Antoine Pitrou)
Date: Sat, 23 Aug 2014 09:44:02 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140823051357.GX25957@ando>
References: <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
Message-ID: <lta5r3$nl$1@ger.gmane.org>

Le 23/08/2014 01:13, Steven D'Aprano a ?crit :
> On Fri, Aug 22, 2014 at 10:31:18PM -0400, Antoine Pitrou wrote:
>
>>>> Python has one of the most powerful and user-friendly function call
>>>> syntaxes around, why reinvent something clearly inferior and alien?
>>>
>>> I wouldn't say it is alien. abc[xyz] is clearly Python syntax.
>>
>> But it is completely uncommon for anything else than subscripting and
>> indexing containers.
>
> And function call syntax is completely uncommon for anything else than
> calling functions (including types and methods). One way or the other,
> we're adding a new use, type declarations.

It's not a new use. A type class is a class, and calling it is just 
instantiating that class. There's nothing new here. If you think that's 
a bit "meta", it's no different than e.g. higher-order functions.

>> Python isn't known for reusing operators for
>> completely unrelated things, e.g. it doesn't overload ">>" for I/O
>> (thank goodness).
>
> I see you've forgotten Python 2's
>
>      print >>sys.stderr, x, y, z
>
> :-)

I always forget about it, but the fact that we finally made print() a 
function in Python 3 actually supports my point: we had a special case, 
and we had to remove it - breaking compatibility in the process - 
because it was impeding on extensibility (adding keyword-only arguments) 
and flexibility (swapping out print() with another function).

"The brackets looks better" is a misguided argument, the same way the 
idea that "print" was cuter as a non-parenthetical statement (and that 
we wouldn't need the power of a regular function call; eventually we 
*did* need it) was a misguided argument.

> The
> annotations themselves will remain arbitrary Python expressions, so if
> you really need a complex type declaration, you can create a helper and
> call it with whatever signature you like.[1]

I don't think you understand what this discussion is about. It isn't 
only about annotations (they are only the channel used to convey the 
information), it is about standardizing a typing system in the stdlib - 
and, therefore, accross the Python community.

I want the *default* typing system to be extensible (not by me, but by 
the community, because use cases for that *will* arise). I would want 
the same thing even if the proposal was to use something else than 
annotation to convey type descriptions.

> I can't speak for the author of mypy, Jukka Lehtosalo, but for me, I
> think the answer is that type declarations make a *mini-language*, not
> full-blown Python.

No, the answer is that mypy wants to support runtime instantiations of 
its type descriptions. It wants to allow you to write e.g. "List[Int]()" 
to instantiate a regular Python list.

While that's one of mypy's design points, though (because mypy 
ultimately seems to aim at becoming a separate Python dialect), it's not 
desired for regular Python.

> It is unclear to me just how powerful the type
> language will be, but surely we don't expect it to evaluate arbitrarily
> complex Python expressions at compile time? Do we?

Why wouldn't it? Ideally, "compile time" is just the import of the module.
(yes, some libraries are ill-behaved on import; they break pydoc and the 
like; I'd say it's their problem)

> I find it unlikely that we expect a static linter to make sense of this:
>
> def bizarre(param:int if random.random() > 0.5 else str)->float:

Yeah... so what, exactly? Do you also criticize Python because it allows 
you to write absurd code? Does it make Python "too powerful"?

> So:
>
> - how powerful do we expect the type system to be?

This is a bad question to ask. That's like asking "how powerful does a 
function or decorator need to be?" The entire point of devising new 
language or library tools is to enable use cases that we *don't know 
about yet*.

I'm not the only one pointing this out. Other people have also come with 
examples where they'd like to make the type system more powerful. And 
that's right now. The future will see more interesting propositions, the 
same way decorators grew to a mechanism powering very diverse and 
sophisticated use cases.

Saying "I don't want to make this functionality powerful because I can't 
think of powerful use cases" is the mentality that leads to things such 
as PHP (closures? functions as objects? "why would I need that to write 
Web pages?").

> - and how much of that power needs to be expressed using
>    function annotations?
 >
> The second question is critical, because there are alternatives to
> function annotations: decorators, docstring annotations, and external
> stub files.

Why would those other channels use a type description syntax different 
from the one used in function annotations? That would be crazy.

You seem to think that a type system should simply be some kind of 
textual description. It's not. It's a set of objects (or classes) with 
some behaviour attached to them; that's why it uses Python syntax. 
Because it *is* Python code.

Regards

Antoine.



From warren.weckesser at gmail.com  Sat Aug 23 17:30:47 2014
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 23 Aug 2014 11:30:47 -0400
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
Message-ID: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>

I'd like to add an additional optional argument to functools.reduce.
The argument is the "nullifier" of the reducing operation.  It is a value
such that function(nullifier, anything) returns nullifier.  For example, if
function(x, y) computes x*y, the nullifier is 0.  If function(x, y) is
the intersection of the sets x and y, the nullifier is the empty set.

The argument would allow reduce to "short circuit" its calculation.   When
reduce encounters the nullifier, it can return immediately.  This can
provide
a significant improvement in performance in some cases.

The change is simple.  Here, for example, is  the "rough equivalent" for
functools.reduce from the docs:

    def reduce(function, iterable, initializer=None):
        it = iter(iterable)
        if initializer is None:
            try:
                initializer = next(it)
            except StopIteration:
                raise TypeError('reduce() of empty sequence with no initial
value')
        accum_value = initializer
        for x in it:
            accum_value = function(accum_value, x)
        return accum_value

Here's how it looks with the optional nullifier argument; the only
change is the new argument and an 'if' statement in the 'for' loop.

    def reduce(function, iterable, initializer=None, nullifier=None):
        it = iter(iterable)
        if initializer is None:
            try:
                initializer = next(it)
            except StopIteration:
                raise TypeError('reduce() of empty sequence with no initial
value')
        accum_value = initializer
        for x in it:
            if nullifier is not None and accum_value == nullifier:
                break
            accum_value = function(accum_value, x)
        return accum_value

(It might be better to use a distinct singleton for the default
value of nullifier, to allow None to be a valid nullifier.)

The actual implementation is in the extension module _functoolsmodule.c.
It looks like the changes to the C code should be straightforward.


Warren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140823/a3d61636/attachment.html>

From jeanpierreda at gmail.com  Sat Aug 23 17:47:22 2014
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sat, 23 Aug 2014 08:47:22 -0700
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
Message-ID: <CABicbJKMpabD9fD8YvWY83CxEGDS+AmC9EnekrwJRG9D16fg8A@mail.gmail.com>

On Sat, Aug 23, 2014 at 8:30 AM, Warren Weckesser
<warren.weckesser at gmail.com> wrote:
> The argument would allow reduce to "short circuit" its calculation.   When
> reduce encounters the nullifier, it can return immediately.  This can
> provide
> a significant improvement in performance in some cases.

If you want something like this you should probably use an explicit
loop. And I say that as someone who *likes* reduce. Loops are far more
flexible. That said, I guess I am +0.

Does this feature exist in any other programming languages?

-- Devin

From joshua at landau.ws  Sat Aug 23 18:01:20 2014
From: joshua at landau.ws (Joshua Landau)
Date: Sat, 23 Aug 2014 17:01:20 +0100
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
Message-ID: <CAN1F8qUBvptkn52+PWb5sDwbtu5OaALSqE9=UPnt-mS3WQfM4g@mail.gmail.com>

On 23 August 2014 16:30, Warren Weckesser <warren.weckesser at gmail.com> wrote:
> I'd like to add an additional optional argument to functools.reduce.
> The argument is the "nullifier" of the reducing operation.  It is a value
> such that function(nullifier, anything) returns nullifier.  For example, if
> function(x, y) computes x*y, the nullifier is 0.  If function(x, y) is
> the intersection of the sets x and y, the nullifier is the empty set.
>
> The argument would allow reduce to "short circuit" its calculation.   When
> reduce encounters the nullifier, it can return immediately.  This can
> provide
> a significant improvement in performance in some cases.

This hasn't been given a use-case and seems like needless complexity.
It also seems far too magical. -1 for those reasons.

If its only purpose is to speed up "reduce" it seems like a very bad
trade-off, as *everywhere else* has a new now-sanctioned behaviour to
worry about.

A better answer in my opinion would be something more like
"reduce(..., stop_on=sentinel)". This could even allow more
optimisations than your idea:

    reduce(operator.mul, iterable, stop_on=0)
    reduce(operator.add, iterable, stop_on=float("nan"))

etc.

I would be -0 on that, because there hasn't been a mentioned use-case.

From jeanpierreda at gmail.com  Sat Aug 23 18:15:02 2014
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Sat, 23 Aug 2014 09:15:02 -0700
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAN1F8qUBvptkn52+PWb5sDwbtu5OaALSqE9=UPnt-mS3WQfM4g@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAN1F8qUBvptkn52+PWb5sDwbtu5OaALSqE9=UPnt-mS3WQfM4g@mail.gmail.com>
Message-ID: <CABicbJJm3EOe7THXRCQbEid0sT8F8EyVUvUB1A-25mJd8Vo-bw@mail.gmail.com>

On Sat, Aug 23, 2014 at 9:01 AM, Joshua Landau <joshua at landau.ws> wrote:
> A better answer in my opinion would be something more like
> "reduce(..., stop_on=sentinel)". This could even allow more
> optimisations than your idea:
>
>     reduce(operator.mul, iterable, stop_on=0)
>     reduce(operator.add, iterable, stop_on=float("nan"))
>
> etc.

How is this any different?

-- Devin

From warren.weckesser at gmail.com  Sat Aug 23 18:27:12 2014
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sat, 23 Aug 2014 12:27:12 -0400
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAN1F8qUBvptkn52+PWb5sDwbtu5OaALSqE9=UPnt-mS3WQfM4g@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAN1F8qUBvptkn52+PWb5sDwbtu5OaALSqE9=UPnt-mS3WQfM4g@mail.gmail.com>
Message-ID: <CAGzF1ufC_KHkBGytoyR0nLynb_W+zqiX5+j7f4QL2F6vgNZVcg@mail.gmail.com>

On Sat, Aug 23, 2014 at 12:01 PM, Joshua Landau <joshua at landau.ws> wrote:

> On 23 August 2014 16:30, Warren Weckesser <warren.weckesser at gmail.com>
> wrote:
> > I'd like to add an additional optional argument to functools.reduce.
> > The argument is the "nullifier" of the reducing operation.  It is a value
> > such that function(nullifier, anything) returns nullifier.  For example,
> if
> > function(x, y) computes x*y, the nullifier is 0.  If function(x, y) is
> > the intersection of the sets x and y, the nullifier is the empty set.
> >
> > The argument would allow reduce to "short circuit" its calculation.
>  When
> > reduce encounters the nullifier, it can return immediately.  This can
> > provide
> > a significant improvement in performance in some cases.
>
> This hasn't been given a use-case and seems like needless complexity.
> It also seems far too magical. -1 for those reasons.
>


I guess "magical" is a matter of perspective ("Any sufficiently advanced
technology..." and all that).



>
> If its only purpose is to speed up "reduce" it seems like a very bad
> trade-off, as *everywhere else* has a new now-sanctioned behaviour to
> worry about.
>

I don't understand what you mean.  Could you elaborate?



>
> A better answer in my opinion would be something more like
> "reduce(..., stop_on=sentinel)". This could even allow more
> optimisations than your idea:
>
>     reduce(operator.mul, iterable, stop_on=0)
>     reduce(operator.add, iterable, stop_on=float("nan"))
>
>
Do you mean it would stop when the sentinel was encountered in iterable?
That wouldn't help in an example such as

    reduce(lambda x, y: x & y, [{1,2}, {3, 4}, {5, 6}], nullifier={})

The nullifier is the empty set, but the empty set does not occur in the
iterable.


 (Joshua, sorry for sending this to you again.  Forgot to "Reply all" the
first time.)


etc.
>
> I would be -0 on that, because there hasn't been a mentioned use-case.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140823/ed8c749d/attachment.html>

From mertz at gnosis.cx  Sat Aug 23 19:25:06 2014
From: mertz at gnosis.cx (David Mertz)
Date: Sat, 23 Aug 2014 10:25:06 -0700
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
Message-ID: <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>

This "nullifier" is mathematically called an "absorbing element", but
saying an "attractor" might be a little more general.  I.e. think of a
local optimization problem, where multiple local min/max points might
occur.  If you reach one, further iteration won't budge from that point,
even if it's not the "global absorbing element."

However, any such argument--including the much more useful
sentinel/'stop_on' idea--significantly changes the semantics of reduce.  In
particular, as is reduce() always consumes its iterator.  Under these
changes, it may or may not consume the iterator, depending on what elements
occur.

Given that one can easily write one's own three line wrapper
'reduce_with_attractor()' for this special semantics which hasn't been
given a use case, I can't see a point of including the argument in the
stdlib.

-1 on proposal.



On Sat, Aug 23, 2014 at 8:30 AM, Warren Weckesser <
warren.weckesser at gmail.com> wrote:

> I'd like to add an additional optional argument to functools.reduce.
> The argument is the "nullifier" of the reducing operation.  It is a value
> such that function(nullifier, anything) returns nullifier.  For example, if
> function(x, y) computes x*y, the nullifier is 0.  If function(x, y) is
> the intersection of the sets x and y, the nullifier is the empty set.
>
> The argument would allow reduce to "short circuit" its calculation.   When
> reduce encounters the nullifier, it can return immediately.  This can
> provide
> a significant improvement in performance in some cases.
>
> The change is simple.  Here, for example, is  the "rough equivalent" for
> functools.reduce from the docs:
>
>     def reduce(function, iterable, initializer=None):
>         it = iter(iterable)
>         if initializer is None:
>             try:
>                 initializer = next(it)
>             except StopIteration:
>                 raise TypeError('reduce() of empty sequence with no
> initial value')
>         accum_value = initializer
>         for x in it:
>             accum_value = function(accum_value, x)
>         return accum_value
>
> Here's how it looks with the optional nullifier argument; the only
> change is the new argument and an 'if' statement in the 'for' loop.
>
>     def reduce(function, iterable, initializer=None, nullifier=None):
>         it = iter(iterable)
>         if initializer is None:
>             try:
>                 initializer = next(it)
>             except StopIteration:
>                 raise TypeError('reduce() of empty sequence with no
> initial value')
>         accum_value = initializer
>         for x in it:
>             if nullifier is not None and accum_value == nullifier:
>                 break
>             accum_value = function(accum_value, x)
>         return accum_value
>
> (It might be better to use a distinct singleton for the default
> value of nullifier, to allow None to be a valid nullifier.)
>
> The actual implementation is in the extension module _functoolsmodule.c.
> It looks like the changes to the C code should be straightforward.
>
>
> Warren
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140823/123aeb67/attachment.html>

From steve at pearwood.info  Sat Aug 23 19:25:41 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 24 Aug 2014 03:25:41 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lta5r3$nl$1@ger.gmane.org>
References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org>
Message-ID: <20140823172536.GZ25957@ando>

On Sat, Aug 23, 2014 at 09:44:02AM -0400, Antoine Pitrou wrote:
> Le 23/08/2014 01:13, Steven D'Aprano a ?crit :
> >On Fri, Aug 22, 2014 at 10:31:18PM -0400, Antoine Pitrou wrote:
> >
> >>>>Python has one of the most powerful and user-friendly function call
> >>>>syntaxes around, why reinvent something clearly inferior and alien?
> >>>
> >>>I wouldn't say it is alien. abc[xyz] is clearly Python syntax.
> >>
> >>But it is completely uncommon for anything else than subscripting and
> >>indexing containers.
> >
> >And function call syntax is completely uncommon for anything else than
> >calling functions (including types and methods). One way or the other,
> >we're adding a new use, type declarations.
> 
> It's not a new use. A type class is a class, and calling it is just 
> instantiating that class. There's nothing new here. If you think that's 
> a bit "meta", it's no different than e.g. higher-order functions.

There's no instantiation during *static* analysis, because the code 
hasn't run yet.

I don't think it's productive to argue about what's new and what isn't. 
I think it is more important to discuss what appears to me to be a 
problem with your suggestion that annotations use call syntax rather 
than subscript syntax. Here's an annotation:

    param:Sequence[int]

or as you prefer:

    param:Sequence(int)

There's two problems with the later, as I see it. Of course, during 
static analysis, we can use any syntax we like, but Python annotations 
are also evaluated at runtime.

(1) Sequence is an ABC, and like all abstract classes, you're not 
supposed to be able to instantiate it:

py> Sequence()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class Sequence with abstract 
methods __getitem__, __len__

For your proposal to be accepted, that would have to change. I think 
that is a problem. At the very least, it's a backwards-incompatibility, 
even within Python 3.x.

(2) Even if we allow instantiating Sequence, and returning a type 
(rather than an instance of Sequence), I don't think that Sequence 
should accept arguments that concrete subclasses would accept:

py> list(int)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'type' object is not iterable

So we would have to allow list() to accept type arguments, and return a 
type, as well as the usual iterable arguments. That strikes me as messy, 
ugly and confusing. I would much prefer keeping call syntax on a type to 
instantiate the type, so that list(obj) either fails or returns a list. 
list[int] can return anything needed, it doesn't have to be an instance 
of list.


> "The brackets looks better" is a misguided argument, the same way the 
> idea that "print" was cuter as a non-parenthetical statement (and that 
> we wouldn't need the power of a regular function call; eventually we 
> *did* need it) was a misguided argument.

You're entitled to your opinion, but in my opinion, trying to avoid 
confusing, hard to read syntax is never misguided. Overloading call 
syntax to do two things (instantiate types, and type annotations), and 
placing such calls in the function parameter list which already has 
parens, risks creating hard to read, ugly code. I think that using [] 
helps the annotations stand out, and I think that allowing the full 
function call syntax complete with keyword-only arguments inside type 
annotations is a case of YAGNI as well as confusing and hard to read. 
That's my opinion; yours may differ.

In any case, I think this is a minor point, and would rather not get 
into arguments about aesthetics. Ultimately, Guido will decide which one 
he prefers the look of.


> >The
> >annotations themselves will remain arbitrary Python expressions, so if
> >you really need a complex type declaration, you can create a helper and
> >call it with whatever signature you like.[1]
> 
> I don't think you understand what this discussion is about. It isn't 
> only about annotations (they are only the channel used to convey the 
> information), it is about standardizing a typing system in the stdlib - 
> and, therefore, accross the Python community.

I think you've got that exactly backwards. Guido's original post seemed 
clear to me that this proposal was *not* about adding a typing system to 
CPython, at least not now, but *only* about standardizing on the syntax 
used. A couple of quotes from his original post:

    [GvR]
    The actual type checker will not be integrated with the Python
    interpreter, and it will not be checked into the CPython repository. 
    The only thing that needs to be added to the stdlib is a copy of
    mypy's typing.py module. [...]

    The curious thing here is that while standardizing a syntax for type
    annotations, we technically still won't be adopting standard rules for
    type checking. This is intentional.


> >It is unclear to me just how powerful the type
> >language will be, but surely we don't expect it to evaluate arbitrarily
> >complex Python expressions at compile time? Do we?
> 
> Why wouldn't it? Ideally, "compile time" is just the import of the module.
> (yes, some libraries are ill-behaved on import; they break pydoc and the 
> like; I'd say it's their problem)

When I say compile-time, I mean when the module is compiled to byte 
code, before it is executed or imported.


> >- how powerful do we expect the type system to be?
> 
> This is a bad question to ask. That's like asking "how powerful does a 
> function or decorator need to be?" The entire point of devising new 
> language or library tools is to enable use cases that we *don't know 
> about yet*.

Decorators are an excellent example. Have you noticed how limited the 
syntax for decorators are? Probably not, because you can do a lot with 
decorators even though the syntax is deliberately limited. But limited 
it is. For instance, you can't use decorator syntax with an index 
lookup:

py> @decorators[0]
  File "<stdin>", line 1
    @decorators[0]
               ^
SyntaxError: invalid syntax

or more than one call:

py> @decorate(x)(y)
  File "<stdin>", line 1
    @decorate(x)(y)
                ^
SyntaxError: invalid syntax

(Both examples in 3.3; 3.4 may differ.)

Does this limit the power of decorators? No, of course not. The sorts of 
things decorators can do is only lightly constrained by the restrictions 
on syntax, and I believe that the same will apply to type annotations.


[...]
> >- and how much of that power needs to be expressed using
> >   function annotations?
> >
> >The second question is critical, because there are alternatives to
> >function annotations: decorators, docstring annotations, and external
> >stub files.
> 
> Why would those other channels use a type description syntax different 
> from the one used in function annotations? That would be crazy.

Because function annotations are limited to a single expression, while 
the others are not.

You're talking about me wanting to limit the power of the static type 
system, but I'm not really. I'm just wanting to limit how much clutter 
ends up inside the function parameter list. Not every arbitrarily 
complex type description needs to be written as a single expression 
inside the parameter list. Static analysis tools can make use of:

* function annotations;
* class definitions outside of the function (e.g. ABCs);
* decorators;
* docstring annotations;
* stub files;
* type inference on variables;
* even inline comments (as mypy does)

Guido's proposal is only about the first, and possibly by implication 
the second. The rest aren't going away, nor are they being standardised 
yet. There's plenty of opportunity for future tools to develop.


> You seem to think that a type system should simply be some kind of 
> textual description. It's not. It's a set of objects (or classes) with 
> some behaviour attached to them; that's why it uses Python syntax. 
> Because it *is* Python code.

That may be true about *runtime* typing, but it is not true for *static* 
typing since that occurs before the Python code runs.

You're absolutely correct that there are other uses for type annotations 
apart from static type checking, and if you go back over my previous 
posts you'll see I've made a number of references to dynamic runtime 
behaviour. Ethan even (politely) told me off for forgetting the static 
part of the proposal, when he thought I was emphasising the runtime 
implications too much. For you now to think I've forgotten the runtime 
implications is ironic :-)


-- 
Steven

From steve at pearwood.info  Sat Aug 23 19:43:58 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 24 Aug 2014 03:43:58 +1000
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
Message-ID: <20140823174358.GA25957@ando>

On Sat, Aug 23, 2014 at 10:25:06AM -0700, David Mertz wrote:
> This "nullifier" is mathematically called an "absorbing element", but
> saying an "attractor" might be a little more general.  I.e. think of a
> local optimization problem, where multiple local min/max points might
> occur.  If you reach one, further iteration won't budge from that point,
> even if it's not the "global absorbing element."

Yes. Note that arbitrary systems may have more than one attractors, 
including cycles (a -> b -> c -> a) and even "strange attractors" of 
infinite complexity. It's probably too much to expect reduce to deal 
with cycles, but a really general solution should at least deal with 
multiple attractors.

> Given that one can easily write one's own three line wrapper
> 'reduce_with_attractor()' for this special semantics 

I don't think you can. Although it's 3:30am here and it's past my bed 
time, so perhaps I'm wrong. The problem is that the wrapper cannot see 
the reduced value until reduce() returns, and we want to short-circuit 
the call once the reduced value is the attractor.

Still, easy or not, I think the semantics are too specialised to 
justify in the standard library, especially given that Guido doesn't 
like reduce and it almost got removed. A better solution, I think, would 
be to stick this reduce_with_attractor() in some third-party functional 
tool library.


-- 
Steven

From bob at redivi.com  Sat Aug 23 19:44:28 2014
From: bob at redivi.com (Bob Ippolito)
Date: Sat, 23 Aug 2014 10:44:28 -0700
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
Message-ID: <CACwMPm9eW15P+eXBd9Y+jm2fF9Oh2=Na5rSSYKAEmDW9jw7JQg@mail.gmail.com>

Yes, such a 'reduce_with_attractor()' can be easily implemented with
itertools.accumulate. Maybe this would be good to add to the "Itertools
Recipes" section of the docs?



On Sat, Aug 23, 2014 at 10:25 AM, David Mertz <mertz at gnosis.cx> wrote:

> This "nullifier" is mathematically called an "absorbing element", but
> saying an "attractor" might be a little more general.  I.e. think of a
> local optimization problem, where multiple local min/max points might
> occur.  If you reach one, further iteration won't budge from that point,
> even if it's not the "global absorbing element."
>
> However, any such argument--including the much more useful
> sentinel/'stop_on' idea--significantly changes the semantics of reduce.  In
> particular, as is reduce() always consumes its iterator.  Under these
> changes, it may or may not consume the iterator, depending on what elements
> occur.
>
> Given that one can easily write one's own three line wrapper
> 'reduce_with_attractor()' for this special semantics which hasn't been
> given a use case, I can't see a point of including the argument in the
> stdlib.
>
> -1 on proposal.
>
>
>
> On Sat, Aug 23, 2014 at 8:30 AM, Warren Weckesser <
> warren.weckesser at gmail.com> wrote:
>
>> I'd like to add an additional optional argument to functools.reduce.
>> The argument is the "nullifier" of the reducing operation.  It is a value
>> such that function(nullifier, anything) returns nullifier.  For example,
>> if
>> function(x, y) computes x*y, the nullifier is 0.  If function(x, y) is
>> the intersection of the sets x and y, the nullifier is the empty set.
>>
>> The argument would allow reduce to "short circuit" its calculation.   When
>> reduce encounters the nullifier, it can return immediately.  This can
>> provide
>> a significant improvement in performance in some cases.
>>
>> The change is simple.  Here, for example, is  the "rough equivalent" for
>> functools.reduce from the docs:
>>
>>     def reduce(function, iterable, initializer=None):
>>         it = iter(iterable)
>>         if initializer is None:
>>             try:
>>                 initializer = next(it)
>>             except StopIteration:
>>                 raise TypeError('reduce() of empty sequence with no
>> initial value')
>>         accum_value = initializer
>>         for x in it:
>>             accum_value = function(accum_value, x)
>>         return accum_value
>>
>> Here's how it looks with the optional nullifier argument; the only
>> change is the new argument and an 'if' statement in the 'for' loop.
>>
>>     def reduce(function, iterable, initializer=None, nullifier=None):
>>         it = iter(iterable)
>>         if initializer is None:
>>             try:
>>                 initializer = next(it)
>>             except StopIteration:
>>                 raise TypeError('reduce() of empty sequence with no
>> initial value')
>>         accum_value = initializer
>>         for x in it:
>>             if nullifier is not None and accum_value == nullifier:
>>                 break
>>             accum_value = function(accum_value, x)
>>         return accum_value
>>
>> (It might be better to use a distinct singleton for the default
>> value of nullifier, to allow None to be a valid nullifier.)
>>
>> The actual implementation is in the extension module _functoolsmodule.c.
>> It looks like the changes to the C code should be straightforward.
>>
>>
>> Warren
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140823/02ad4474/attachment.html>

From mertz at gnosis.cx  Sat Aug 23 20:43:42 2014
From: mertz at gnosis.cx (David Mertz)
Date: Sat, 23 Aug 2014 11:43:42 -0700
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <20140823174358.GA25957@ando>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <20140823174358.GA25957@ando>
Message-ID: <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>

It's true, Steven, that we'd have to use itertools.accumulate() rather than
functools.reduce() to do the heavy lifting here (as Bob suggested).  But
using that, here are three lines:

  from itertools import *
  from operator import add

  def reduce_with_attractor(func, it, start=?, end_if=?):
      it = iter(it)
      start = start if start?? else it.__next__()
      return list(takewhile(? x: x?end_if,
                            accumulate(chain([start],it), func)))[-1]

Oh, that's cute... my copy-paste was using
https://github.com/ehamberg/vim-cute-python (somewhat customized further by
me).  In regular ASCII:

  def reduce_with_attractor(func, it, start=None, end_if=None):
      it = iter(it)
      start = start if start!=None else it.__next__()
      return list(takewhile(lambda x: x!=end_if,
                            accumulate(chain([start],it), func)))[-1]

This gives you the accumulation up-to-but-not-including the attractor.  I
guess the OP wanted to return the attractor itself (although that seems
slightly less useful to me).  I'd have to play around to get that version
in three lines... maybe it would take 4 or 5 lines to do it.


On Sat, Aug 23, 2014 at 10:43 AM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Sat, Aug 23, 2014 at 10:25:06AM -0700, David Mertz wrote:
> > This "nullifier" is mathematically called an "absorbing element", but
> > saying an "attractor" might be a little more general.  I.e. think of a
> > local optimization problem, where multiple local min/max points might
> > occur.  If you reach one, further iteration won't budge from that point,
> > even if it's not the "global absorbing element."
>
> Yes. Note that arbitrary systems may have more than one attractors,
> including cycles (a -> b -> c -> a) and even "strange attractors" of
> infinite complexity. It's probably too much to expect reduce to deal
> with cycles, but a really general solution should at least deal with
> multiple attractors.
>
> > Given that one can easily write one's own three line wrapper
> > 'reduce_with_attractor()' for this special semantics
>
> I don't think you can. Although it's 3:30am here and it's past my bed
> time, so perhaps I'm wrong. The problem is that the wrapper cannot see
> the reduced value until reduce() returns, and we want to short-circuit
> the call once the reduced value is the attractor.
>
> Still, easy or not, I think the semantics are too specialised to
> justify in the standard library, especially given that Guido doesn't
> like reduce and it almost got removed. A better solution, I think, would
> be to stick this reduce_with_attractor() in some third-party functional
> tool library.
>
>
> --
> Steven
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140823/6b3fdb47/attachment-0001.html>

From mertz at gnosis.cx  Sat Aug 23 20:54:37 2014
From: mertz at gnosis.cx (David Mertz)
Date: Sat, 23 Aug 2014 11:54:37 -0700
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <20140823174358.GA25957@ando>
 <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>
Message-ID: <CAEbHw4a8sXC2kgka6LwqPX4Bfg_qthS-qk8mQ0TMOQApsp5aEg@mail.gmail.com>

Completely off topic, but the symbol substitutions in
https://github.com/ehamberg/vim-cute-python/tree/moresymbols used '?' for
both 'set()' and 'None' which seemed confusing to me.  Using aleph ('?')
both somewhat resembles a Roman 'N(one)' and also makes a certain kind of
sense if you think of aleph0 as the first "inaccessible cardinal" in set
theory.

It's too bad I didn't have any loops or 'in' tests in my three liner, since
I love the look of 'i ? collection' on screen.


On Sat, Aug 23, 2014 at 11:43 AM, David Mertz <mertz at gnosis.cx> wrote:

> It's true, Steven, that we'd have to use itertools.accumulate() rather
> than functools.reduce() to do the heavy lifting here (as Bob suggested).
>  But using that, here are three lines:
>
>   from itertools import *
>   from operator import add
>
>   def reduce_with_attractor(func, it, start=?, end_if=?):
>       it = iter(it)
>       start = start if start?? else it.__next__()
>       return list(takewhile(? x: x?end_if,
>                             accumulate(chain([start],it), func)))[-1]
>
> Oh, that's cute... my copy-paste was using
> https://github.com/ehamberg/vim-cute-python (somewhat customized further
> by me).  In regular ASCII:
>
>   def reduce_with_attractor(func, it, start=None, end_if=None):
>       it = iter(it)
>       start = start if start!=None else it.__next__()
>       return list(takewhile(lambda x: x!=end_if,
>                             accumulate(chain([start],it), func)))[-1]
>
> This gives you the accumulation up-to-but-not-including the attractor.  I
> guess the OP wanted to return the attractor itself (although that seems
> slightly less useful to me).  I'd have to play around to get that version
> in three lines... maybe it would take 4 or 5 lines to do it.
>
>
> On Sat, Aug 23, 2014 at 10:43 AM, Steven D'Aprano <steve at pearwood.info>
> wrote:
>
>> On Sat, Aug 23, 2014 at 10:25:06AM -0700, David Mertz wrote:
>> > This "nullifier" is mathematically called an "absorbing element", but
>> > saying an "attractor" might be a little more general.  I.e. think of a
>> > local optimization problem, where multiple local min/max points might
>> > occur.  If you reach one, further iteration won't budge from that point,
>> > even if it's not the "global absorbing element."
>>
>> Yes. Note that arbitrary systems may have more than one attractors,
>> including cycles (a -> b -> c -> a) and even "strange attractors" of
>> infinite complexity. It's probably too much to expect reduce to deal
>> with cycles, but a really general solution should at least deal with
>> multiple attractors.
>>
>> > Given that one can easily write one's own three line wrapper
>> > 'reduce_with_attractor()' for this special semantics
>>
>> I don't think you can. Although it's 3:30am here and it's past my bed
>> time, so perhaps I'm wrong. The problem is that the wrapper cannot see
>> the reduced value until reduce() returns, and we want to short-circuit
>> the call once the reduced value is the attractor.
>>
>> Still, easy or not, I think the semantics are too specialised to
>> justify in the standard library, especially given that Guido doesn't
>> like reduce and it almost got removed. A better solution, I think, would
>> be to stick this reduce_with_attractor() in some third-party functional
>> tool library.
>>
>>
>> --
>> Steven
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>



-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140823/960a198d/attachment.html>

From joshua at landau.ws  Sat Aug 23 21:23:16 2014
From: joshua at landau.ws (Joshua Landau)
Date: Sat, 23 Aug 2014 20:23:16 +0100
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CABicbJJm3EOe7THXRCQbEid0sT8F8EyVUvUB1A-25mJd8Vo-bw@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAN1F8qUBvptkn52+PWb5sDwbtu5OaALSqE9=UPnt-mS3WQfM4g@mail.gmail.com>
 <CABicbJJm3EOe7THXRCQbEid0sT8F8EyVUvUB1A-25mJd8Vo-bw@mail.gmail.com>
Message-ID: <CAN1F8qX3f9ZaAvXHCEi0wPtPzq9PmGT4Pk6ddSaLs618i9sp-Q@mail.gmail.com>

On 23 August 2014 17:15, Devin Jeanpierre <jeanpierreda at gmail.com> wrote:
> On Sat, Aug 23, 2014 at 9:01 AM, Joshua Landau <joshua at landau.ws> wrote:
>> A better answer in my opinion would be something more like
>> "reduce(..., stop_on=sentinel)".
>
> How is this any different?

It's not. I'm a fool who doesn't read.

Apologies for the noise.

From tomirendo at gmail.com  Sat Aug 23 21:23:56 2014
From: tomirendo at gmail.com (Yotam Vaknin)
Date: Sat, 23 Aug 2014 22:23:56 +0300
Subject: [Python-ideas] Proposal : allowing grouping by relation
Message-ID: <A63F4ECC-8555-4C15-B5BD-F86168AA6F92@gmail.com>

Hi,

I am using groupby (from itertools) to group objects by a key. It would be very useful for me to be able to group objects by the relation of two consecutive  objects or by an object relation to the first object in the current group.

I think it should be done by adding a "relation" keyword to the function, that accept two argument functions that return true or false.

It would also be useful to create a function that enable easily creating relation functions. (Like attrgetter does for keys)
ls = "aaabcdddefgjklm"
groupby(ls, relation=difference(3,key = ord)) 
#[['a', 'a', 'a', 'b', 'c', 'd', 'd', 'd', 'e', 'f', 'g'], ['j', 'k', 'l', 'm']]

I think in this case the function won't return a key-group tuple, but just a group iterable.

This is already very useful for me, to group event objects in a list if they are close enough in time.

I wrote most of what I had in mind here:
https://github.com/tomirendo/Grouper


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140823/eb7382ae/attachment-0001.html>

From joshua at landau.ws  Sat Aug 23 21:29:32 2014
From: joshua at landau.ws (Joshua Landau)
Date: Sat, 23 Aug 2014 20:29:32 +0100
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAGzF1ufC_KHkBGytoyR0nLynb_W+zqiX5+j7f4QL2F6vgNZVcg@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAN1F8qUBvptkn52+PWb5sDwbtu5OaALSqE9=UPnt-mS3WQfM4g@mail.gmail.com>
 <CAGzF1ufC_KHkBGytoyR0nLynb_W+zqiX5+j7f4QL2F6vgNZVcg@mail.gmail.com>
Message-ID: <CAN1F8qWu6nsNPOBSN2GxhevTJPYaN8+TAtZh77sKs8MOEheRVw@mail.gmail.com>

On 23 August 2014 17:27, Warren Weckesser <warren.weckesser at gmail.com> wrote:
>
> On Sat, Aug 23, 2014 at 12:01 PM, Joshua Landau <joshua at landau.ws> wrote:
>>
>> On 23 August 2014 16:30, Warren Weckesser <warren.weckesser at gmail.com>
>> wrote:
>>>
>>> <sensible stuff>
>>
>> <not so sensible stuff>
>
> <sensible stuff>

I suggest you just ignore what I wrote, as I was obviously too out of
it to read properly. Let me try again:

"This sound good, but it lacks a use-case and has a somewhat poor name."

From stephen at xemacs.org  Sat Aug 23 22:42:53 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 24 Aug 2014 05:42:53 +0900
Subject: [Python-ideas]  Proposal : allowing grouping by relation
In-Reply-To: <A63F4ECC-8555-4C15-B5BD-F86168AA6F92@gmail.com>
References: <A63F4ECC-8555-4C15-B5BD-F86168AA6F92@gmail.com>
Message-ID: <87bnrbc4z6.fsf@uwakimon.sk.tsukuba.ac.jp>

Yotam Vaknin writes:
 > Hi,
 > 
 > I am using groupby (from itertools) to group objects by a key. It
 > would be very useful for me to be able to group objects by the
 > relation of two consecutive  objects or by an object relation to
 > the first object in the current group.

I don't think you need to extend groupby.  You can just cache the
object to compare to.  How about a decorator like

def relate_to_first(is_related):
    def wrapped(this, _first=[]):
        if _first:
            if is_related(this, _first[0]):
                pass
            else:
                _first[0] = this
                _first[1] += 1
        else:
            _first[0] = this
            _first[1] = 0
        return _first[1]
    return wrapped

@relate_to_first
def some_relation(this, that):
    pass

and similarly for a decorator relate_to_last?

There are probably more elegant ways to do this, such as a class whose
instances are callable.  Such a class could also provide a reset
method so you could reuse the relation

Warning: that code is untested.

From abarnert at yahoo.com  Sat Aug 23 22:48:26 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 23 Aug 2014 13:48:26 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140823051357.GX25957@ando>
References: <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
Message-ID: <CD5844DD-3224-41D0-9923-621B71650D89@yahoo.com>

On Aug 22, 2014, at 22:13, Steven D'Aprano <steve at pearwood.info> wrote:

> I can't speak for the author of mypy, Jukka Lehtosalo, but for me, I 
> think the answer is that type declarations make a *mini-language*, not 
> full-blown Python. It is unclear to me just how powerful the type 
> language will be, but surely we don't expect it to evaluate arbitrarily 
> complex Python expressions at compile time? Do we?
> 
> I find it unlikely that we expect a static linter to make sense of this:
> 
> def bizarre(param:int if random.random() > 0.5 else str)->float:
> 
> except to say it returns a float and takes who-knows-what as argument. 
> An extreme case, I grant, but I expect that there are going to be limits 
> to what can be checked at compile time based on static analysis. I admit 
> that I don't fully understand all the implications of static versus 
> runtime type checking, but it doesn't seem unreasonable to expect static 
> declarations to be considerably simpler than what can be expressed at 
> runtime. Hence, a simpler mini-language is appropriate.

I agree.

Trying to make the declarative type system more powerful is possible, but probably not a good idea. Look at C++98's template system. It's a complete compile-time language for defining types declaratively (in terms of other types, integer literals, and pointer literals), except that it's missing the ability to recursively decompose recursive types (like a compile-time cons). While that sounds nice in theory, in practice it's painful to use, and completely different from programming in C++. That's what happens if you take a declarative type system and try to expand it until it's fully general: you end up with a type system that looks like ML instead of your language.

C++14 lets you use almost the entire runtime language for computing at compile time (no goto or try, some limits on inheritance) with constexpr values and functions. You can always use the declarative language, but you can also write complex stuff imperatively, in a way that looks like C++. 

If we really need something as powerful as ML or Haskell for our compile-time type system, the latter would be the way to do it. And if this obviously takes too much effort to be worth doing, that means we probably don't actually need the former either.

So, trying to extend MyPy's declarative system to be fully general probably isn't worth doing.

>> Really, for it to be powerful enough, the type description system has to 
>> use its own set of classes. It can't try to hack away existing builtins 
>> and ABCs in the hope of expressing different things with them, or it 
>> *will* hit a wall some day (and, IMO, sooner rather than later).
> 
> How powerful is "powerful enough"? Powerful enough for what?
> 
> You've probably heard of the famous "the compiler found my infinite 
> loop", where the ML type checker was able to detect that code would 
> never terminate. I find that remarkable, and even more astonishing that, 
> according to some, solving the halting problem is "nothing special" for 
> type systems.[2] But many languages make do with less powerful type systems, 
> and tools such as IDEs and editors surely don't need something that 
> powerful. So:
> 
> - how powerful do we expect the type system to be?
> 
> - and how much of that power needs to be expressed using 
>  function annotations?
> 
> The second question is critical, because there are alternatives to 
> function annotations: decorators, docstring annotations, and external 
> stub files.
> 
> 
> 
> 
> [1] Although if you do so, it is not clear to me how much the static 
> checker will be able to use it.
> 
> 
> [2] http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/
> 
> 
> -- 
> Steven
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From abarnert at yahoo.com  Sat Aug 23 23:57:46 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 23 Aug 2014 14:57:46 -0700
Subject: [Python-ideas] Proposal : allowing grouping by relation
In-Reply-To: <A63F4ECC-8555-4C15-B5BD-F86168AA6F92@gmail.com>
References: <A63F4ECC-8555-4C15-B5BD-F86168AA6F92@gmail.com>
Message-ID: <ECF62218-A564-43FA-AC64-5F84859F54EC@yahoo.com>

On Aug 23, 2014, at 12:23, Yotam Vaknin <tomirendo at gmail.com> wrote:

> Hi,
> 
> I am using groupby (from itertools) to group objects by a key. It would be very useful for me to be able to group objects by the relation of two consecutive  objects or by an object relation to the first object in the current group.
> 
> I think it should be done by adding a "relation" keyword to the function, that accept two argument functions that return true or false.

This _should be_ easy to write as a wrapper around groupby with a key that checks your relation.

But there's one problem: groupby checks the _first_ key in a group against each new key, instead of the most recent one.

I wrote a blog post last year about this (http://stupidpythonideas.blogpost.com/2014/01/grouping-into-runs-of-adjacent-values.html). It turns our to be pretty easy if your relation is symmetric, but only one of the obvious ways to do it actually works.

Anyway, it might be worth changing groupby so it never compares x==y instead of y==x, and making the C implementation and the Python equivalent in the docs actually equivalent.

Beyond that, I think it might make sense to add a relation_to_key function and/or to change cmp_to_key so it's directly usable with groupby.

Then, it should be possible to make groupby_relation into a 3-line wrapper around groupby, in which case I think it might be better as a recipe (and submitted to more_itertools on PyPI) than to add it to itertools itself.

> It would also be useful to create a function that enable easily creating relation functions. (Like attrgetter does for keys)
> ls = "aaabcdddefgjklm"
> groupby(ls, relation=difference(3,key = ord)) 
> #[['a', 'a', 'a', 'b', 'c', 'd', 'd', 'd', 'e', 'f', 'g'], ['j', 'k', 'l', 'm']]
> 
> I think in this case the function won't return a key-group tuple, but just a group iterable.

The key actually can be useful here. You can use it as a label for the "column". Especially if you've written your key function so it keeps track of both the first and most recent values, instead of just the most recent, so you can label it "a-_", where that _ is the current value at any given point, and the last value once you've consumed the group iterator. Sure, you _could_ recover that information from the group itself if you need it, but isn't it even easier to discard it if you don't need it?

> 
> This is already very useful for me, to group event objects in a list if they are close enough in time.
> 
> I wrote most of what I had in mind here:
> https://github.com/tomirendo/Grouper
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140823/a6c18e07/attachment-0001.html>

From antoine at python.org  Sun Aug 24 01:00:54 2014
From: antoine at python.org (Antoine Pitrou)
Date: Sat, 23 Aug 2014 19:00:54 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140823172536.GZ25957@ando>
References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
Message-ID: <ltb6f6$65d$1@ger.gmane.org>


Le 23/08/2014 13:25, Steven D'Aprano a ?crit :
>>
>> It's not a new use. A type class is a class, and calling it is just
>> instantiating that class. There's nothing new here. If you think that's
>> a bit "meta", it's no different than e.g. higher-order functions.
>
> There's no instantiation during *static* analysis, because the code
> hasn't run yet.

In your idea of "static analysis", it hasn't. Because you think it 
should involve some kind of separate syntax analysis tool that has 
nothing to do with regular Python. But Python is powerful enough to let 
you do that using normal introspection of modules.

And it's *exactly* how we are exposing function annotations (and also 
docstrings, etc.): using runtime-accessible introspection information 
which is gathered by importing modules and therefore actually 
*executing* toplevel module code. Not merely compiling it.

The rest of your message I'm not motivated to respond to, sorry.

Regards

Antoine.



From cfkaran2 at gmail.com  Sun Aug 24 02:37:34 2014
From: cfkaran2 at gmail.com (Cem Karan)
Date: Sat, 23 Aug 2014 20:37:34 -0400
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CAP7+vJKJP6Nh42X70VmmFKwrf3XpJX3-JC5R9guNgmW8mk7_oA@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <CAP7+vJKJP6Nh42X70VmmFKwrf3XpJX3-JC5R9guNgmW8mk7_oA@mail.gmail.com>
Message-ID: <846C4603-A7EF-4394-8CE9-4B8C614E1E13@gmail.com>

As others have said on the list, 'go away for a few days, and find a mega-thread in your inbox'.  Yay.

On Aug 19, 2014, at 1:01 PM, Guido van Rossum <guido at python.org> wrote:

> On Tue, Aug 19, 2014 at 6:27 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> [Agreeable musings about team size...]
> 
> Ultimately, my perspective is that Guido's proposal boils down to
> having a nice syntax where:
> 
>     def myfunc(a : KindX, b: KindY, c: KindZ):
>         ...
> 
> is the moral equivalent of:
> 
>     def myfunc(a, b, c):
>         assert isinstance(a, KindX)
>         assert isinstance(b, KindY)
>         assert isinstance(c, KindZ)
> 
> Please no. The asserts affect runtime. The type declarations are for linting, IDEs and docs. I don't want the difference to be swept under the rug. I want it to be the key feature of the proposal.

<SNIP>

Going slightly sideways on this -- I think this is why we should use decorators instead of annotations; as has already been mentioned twice on the list, http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ does a good job pointing out that static and dynamic typing systems are separate but overlapping concepts.  It also points out that both have their place.  If we define decorators that can be turned on or off easily (command-line option?  Environment variable? other?), then the end user can choose if if her or she is going to do static, dynamic, both, or none.  This could be useful when trying to track down that annoying bug in long-running production code.

Also, Types and Programming Languages by Dr. Pierce has been mentioned at least twice as well; would it be useful to ask him to join in the discussion?

Thanks,
Cem Karan

From greg.ewing at canterbury.ac.nz  Sun Aug 24 03:11:34 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 24 Aug 2014 13:11:34 +1200
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
In-Reply-To: <lt9mfn$tpn$1@ger.gmane.org>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <lt9mfn$tpn$1@ger.gmane.org>
Message-ID: <53F93BC6.6030307@canterbury.ac.nz>

Georg Brandl wrote:
> Another thought is whether, assuming
> 
>    class MyList(list):
>        ...
> 
> the type  "MyList[int]"  would do the expected thing.  (And if not,
> how you would write the correct type generation code.)

I would expect that to be a error, unless MyList were
declared as accepting a type parameter, using whatever
means is decided on for that in the type language.

-- 
Greg


From steve at pearwood.info  Sun Aug 24 03:53:26 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 24 Aug 2014 11:53:26 +1000
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <20140823174358.GA25957@ando>
 <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>
Message-ID: <20140824015325.GB25957@ando>

On Sat, Aug 23, 2014 at 11:43:42AM -0700, David Mertz wrote:

>   def reduce_with_attractor(func, it, start=None, end_if=None):
>       it = iter(it)
>       start = start if start!=None else it.__next__()
>       return list(takewhile(lambda x: x!=end_if,
>                             accumulate(chain([start],it), func)))[-1]

A couple of points:

- Don't use it.__next__(), use the built-in next(it).

- To match the behaviour of reduce, you need to catch the StopIteration 
and raise TypeError.

- Your implementation eagerly generates a potentially enormous list of 
intermediate results, which strikes me as somewhat unfortunate for a 
functional tool like reduce. In other words, for some inputs, this is 
going to perform like a dog, generating a HUGE list up front, then 
throwing it all away except for the final value.

 
> This gives you the accumulation up-to-but-not-including the attractor.  I
> guess the OP wanted to return the attractor itself (although that seems
> slightly less useful to me).

Not really. His use-case seems to be to short-cut a lot of unnecessary 
calculations, e.g. suppose you write product() as reduce(operator.mul, 
iterable). In the event that the product reaches zero, you can[1] 
short-circuit the rest of the iterable and just return 0:

product([1, 2, 3, 0] + [4]*1000000)

ought to reduce 0, not 6, and the intent is for it to do so *quickly*, 
ignoring the 4s at the end of the list.




[1] Actually you can't. 0 is no longer an attractor in the presence of 
INF or NAN.



-- 
Steven

From mertz at gnosis.cx  Sun Aug 24 04:19:25 2014
From: mertz at gnosis.cx (David Mertz)
Date: Sat, 23 Aug 2014 19:19:25 -0700
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <20140824015325.GB25957@ando>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <20140823174358.GA25957@ando>
 <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>
 <20140824015325.GB25957@ando>
Message-ID: <CAEbHw4Yk4fx=T0+wX2xULYutfNE3AVzSgcD_C-R-zkzS7wQF8A@mail.gmail.com>

On Sat, Aug 23, 2014 at 6:53 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Sat, Aug 23, 2014 at 11:43:42AM -0700, David Mertz wrote:
>
> >   def reduce_with_attractor(func, it, start=None, end_if=None):
> >       it = iter(it)
> >       start = start if start!=None else it.__next__()
> >       return list(takewhile(lambda x: x!=end_if,
> >                             accumulate(chain([start],it), func)))[-1]
>
> A couple of points:
>
> - Don't use it.__next__(), use the built-in next(it).
>

Yeah, good point.


> - To match the behaviour of reduce, you need to catch the StopIteration
> and raise TypeError.
>

Oh, OK.  Hadn't thought of that.


> - Your implementation eagerly generates a potentially enormous list of
> intermediate results, which strikes me as somewhat unfortunate for a
> functional tool like reduce. In other words, for some inputs, this is
> going to perform like a dog, generating a HUGE list up front, then
> throwing it all away except for the final value.
>

I know.  I realized this flaw right away.  I was trying to be cute and fit
it in my promised 3 lines.  It would be better to put it in a loop to
realize the successive values, of course--but would take an extra line or
two.  Maybe there's a way to squeeze it in one line with itertools rather
than a regular loop though.


> > This gives you the accumulation up-to-but-not-including the attractor.  I
> > guess the OP wanted to return the attractor itself (although that seems
> > slightly less useful to me).
>
> Not really. His use-case seems to be to short-cut a lot of unnecessary
> calculations, e.g. suppose you write product() as reduce(operator.mul,
> iterable). In the event that the product reaches zero, you can[1]
> short-circuit the rest of the iterable and just return 0:
>
> product([1, 2, 3, 0] + [4]*1000000)
>
> ought to reduce 0, not 6, and the intent is for it to do so *quickly*,
> ignoring the 4s at the end of the list.
>

I guess that's true.  Although I can certainly imagine being interested not
only in the final attractor, but *that* it reached an attractor and ended
early.  Not sure how best to signal that.  Actually, now that I think of
it, it would be kinda nice to make the function 'reduce_with_attractorS()'
instead, and allow specification of multiple attractors.

I welcome your improved version of the code :-).  Feel free to take a whole
10 lines to do it right.


> [1] Actually you can't. 0 is no longer an attractor in the presence of
> INF or NAN.
>

I was sort of thinking of a "we're all adults here" attitude.  That is, the
"attractor" might not really be a genuine attractor, but we still trust the
caller to say it is.  I.e. my function would accept this call:

    reduce_with_attractor(operator.mul, range(1,1e6), end_if=6))

I'm making the claim that reaching '6' is a stopping point... which, well
it is.  No, it's not an actual attractor, but maybe a caller really does
want to stop iterating if it gets to that value anyway.  Hence 'end_if' is
actually an accurate name.

-- 
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140823/2eccd7c9/attachment-0001.html>

From cfkaran2 at gmail.com  Sun Aug 24 04:18:42 2014
From: cfkaran2 at gmail.com (Cem Karan)
Date: Sat, 23 Aug 2014 22:18:42 -0400
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
Message-ID: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>


On Aug 20, 2014, at 11:08 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> Sigh. I go away for a week and come back to a mega-thread I can never
> hope to catch up on :-)
> 
> TL; DR; - Although mypy looks interesting, I think it's too soon to
> close the door on all other uses of annotations. Let's find a solution
> that allows exploration of alternative uses for a while longer.
> 
> OK, can I just make some points regarding the static typing thread.
> First of all, I have no issue with the idea of static typing, and in
> fact I look forward to seeing what benefits it might have (if nothing
> else, the pointer to mypy, which I'd never heard of before, is
> appreciated). It won't be something I use soon (see below) but that's
> fine.
> 
> But Guido seems to be saying (on a number of occasions) that nobody is
> really using annotations, so he wants to focus on the static typing
> use case alone. I think this is a mistake. First of all, I see no
> reason why functions using typing annotations could not be introduced
> with a decorator[1]. So why insist that this is the *only* use of
> annotations, when it's pretty easy to allow others to co-exist?
> 
> Also, the "nobody is using annotations" argument? Personally, I know
> of a few other uses:
> 
> 1. Argument parsers - at least 3 have been mentioned in the thread.
> 2. Structure unpacking - I think there is a library that uses
> annotations for this, although I may be wrong.
> 3. FFI bindings. I know I've seen this discussed, although I can't
> find a reference just now.
> 
> There are probably other ideas around as well (GUI bindings,
> documentation generation, ...) None are particularly mature, and most
> are just at the "ideas" stage, but typically the ideas I have seen are
> the sort of thing you'd write a library for, and Python 3 only
> libraries *really* aren't common yet.
> 
> The problem for people wanting to experiment with annotations, is that
> they need to be writing Python 3 only code. While Python 3 adoption is
> growing rapidly, I suspect that large applications are typically still
> focused on either going through, or tidying up after, a 2-3 migration.
> And new projects, while they may be developed from the ground up using
> Python 3, will typically be using programmers skilled in Python 2, to
> whom Python 3 features are not yet an "instinctive" part of the
> toolset. Apart from large standalone applications, there are smaller
> scripts (which are typically going to be too small to need
> programming-in-the-large features like annotations) and libraries
> (which really aren't yet in a position to drop Python 2.x totally,
> unless they have a fairly small user base).
> 
> So I don't see it as compelling that usage of annotations in the wild
> is not yet extensive.
> 
> Rather than close the door on alternative uses of annotations, can I suggest:
> 
> 1. By all means bless mypy syntax as the standard static typing
> notation - this seems like a good thing.
> 2. Clarify that static typing annotations should be introduced with a
> decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy
> declaration in the stdlib, and have a registration protocols for tools
> to hook their own implementation into it.[2]
> 3. Leave the door open for other uses of decorators, at least until
> some of the more major libraries drop Python 2.x support completely
> (and hence can afford to have a dependency on a Python 3 only module
> that uses annotations). See at that stage if annotations take off.
> 4. If we get to a point where even libraries that *could* use
> annotations don't, then revisit the idea of restricting usage to just
> type information.
> 
> Paul
> 
> [1] Also, a decorator could allow a Python 2 compatible form by using
> decorator arguments as an alternative to annotations.
> [2] I've yet to see a clear explanation of how "a tool using type
> annotations" like an linter, editor, IDE or Python compiler would use
> them in such a way that precludes decoration as a means of signalling
> the annotation semantics.

A long while back I proposed a mechanism for sharing __annotations__ between multiple, non-cooperating projects.  The basic idea is that each annotation becomes a dictionary.  Each project (and 'project' is a very loosely defined concept here) chooses a UUID that it uses as key into the dictionary.  The value is up to the project.

The advantage to this is manifold:

- Annotations can still have multiple uses by different groups without stepping on each other's toes.
- If someone wants to make a standard, all they have to do is publish the UUID associated with their standard.  For example, we might choose UUID('2cca6238-9fca-4053-aa3d-db9050e6b26b') as the official type information UUID.  All projects that want to develop linters, documentation generators, etc., will use that UUID for all annotations, and the PEP will require it.
- de facto standards can become de jure standards by blessing a particular UUID.
- Guessing if this method is being used is relatively easy; if its a dictionary, and if every key is a UUID, it probably follows this standard.  We can tighten it further by requiring some key-value pair be in every dictionary (i.e., {UUID('1ad60d50-8237-4b98-b2b1-69fd08ed575c'):"PEPXXXX"} is always in the dictionary).  This makes it fairly simple to add without stomping on what people are already doing. 
- Finding the standard on the web should also be easy; while you might not find the PEP instantly, you'll probably zoom into it fairly fast.

Disadvantages:

- Typing out UUIDs is PAINFUL.  I highly recommend using decorators instead.
- Reading the __annotations__ dictionary will be difficult.  pprint() should make this easier.

I have working proof-of-concept code at https://github.com/oranguman/annotizer that defines a decorator class that handles the UUID for you.  It needs to be extended to parse out information, but it handles the 'other use cases' problem fairly well.

Thanks,
Cem Karan

From steve at pearwood.info  Sun Aug 24 05:13:30 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 24 Aug 2014 13:13:30 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for
	function	annotations
In-Reply-To: <53F93BC6.6030307@canterbury.ac.nz>
References: <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <lt9mfn$tpn$1@ger.gmane.org>
 <53F93BC6.6030307@canterbury.ac.nz>
Message-ID: <20140824031329.GD25957@ando>

On Sun, Aug 24, 2014 at 01:11:34PM +1200, Greg Ewing wrote:
> Georg Brandl wrote:
> >Another thought is whether, assuming
> >
> >   class MyList(list):
> >       ...
> >
> >the type  "MyList[int]"  would do the expected thing.  (And if not,
> >how you would write the correct type generation code.)
> 
> I would expect that to be a error, unless MyList were
> declared as accepting a type parameter, using whatever
> means is decided on for that in the type language.

I don't really understand what you're trying to say here, so I may be 
misinterpreting you. I *think* that you're trying to say that for every 
type in the standard library, and every class created by third parties 
(including subclasses), the author will have to "declare" (in some 
unknown sense) that it can be used for type annotations like MyList[T], 
for some type T.

I expect that will be painful and unnecessary. I expect that if we go 
down this path, somebody will have to add the appropriate code to the 
metaclass of all types (i.e. type itself) so that AnyArbitraryType[spam] 
will do the right thing without the author of AnyArbitraryType needing 
to do any additional work.

This is not entirely uncharted territory. Mypy has blazed the trail for 
us. The way I think Mypy works is that the classes in typing.py return 
themselves when subscripted. The arguments to __getitem__ are ignored 
except during static analysis. Here's the metaclass from Mypy:

class GenericMeta(type):
      """Metaclass for generic classes that support indexing by types."""
      def __getitem__(self, args):
          # Just ignore args; they are for compile-time checks only.
          return self

https://github.com/JukkaL/mypy/blob/master/lib-typing/3.2/typing.py


As I understand it, we are considering two possibilities:

(1) all types are directly usable, so we can say

    def func(param:list[int])->list:

    This will require type to grow a __getitem__ method similar to
    the metaclass above.

(2) Standard library types are not directly usable, you have to go 
    through the typing module, which has wrappers that include the
    __getitem__ metaclass magic:

    from typing import List
    def func(param:List[int])->List:

There are pros and cons to both approaches. 


-- 
Steven

From ncoghlan at gmail.com  Sun Aug 24 06:24:53 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Aug 2014 14:24:53 +1000
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
Message-ID: <CADiSq7ffiRmJK5YFWZHeBrj3Z3ZsGkGtXOFJ5hePmc4tv6BEwQ@mail.gmail.com>

On 24 August 2014 01:30, Warren Weckesser <warren.weckesser at gmail.com> wrote:
> I'd like to add an additional optional argument to functools.reduce.
> The argument is the "nullifier" of the reducing operation.  It is a value
> such that function(nullifier, anything) returns nullifier.  For example, if
> function(x, y) computes x*y, the nullifier is 0.  If function(x, y) is
> the intersection of the sets x and y, the nullifier is the empty set.

When it comes to judging the usefulness of functional programming
features these days, my first question is generally going to be "Does
PyToolz offer this?"

Grumblings about the name aside, it's still the solution I recommend
to folks that wish Python had more functional programming tools in the
standard library: http://toolz.readthedocs.org/en/latest/api.html

There's even a Cython accelerated version available (Cytoolz).

"pip install toolz" for the pure Python version, "pip install cytoolz"
for the accelerated one.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From greg.ewing at canterbury.ac.nz  Sun Aug 24 07:54:22 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 24 Aug 2014 17:54:22 +1200
Subject: [Python-ideas] Proposal: Use mypy syntax
	for	function	annotations
In-Reply-To: <20140824031329.GD25957@ando>
References: <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <lt9mfn$tpn$1@ger.gmane.org>
 <53F93BC6.6030307@canterbury.ac.nz> <20140824031329.GD25957@ando>
Message-ID: <53F97E0E.6030908@canterbury.ac.nz>

Steven D'Aprano wrote:
> I don't really understand what you're trying to say here, so I may be 
> misinterpreting you. I *think* that you're trying to say that for every 
> type in the standard library, and every class created by third parties 
> (including subclasses), the author will have to "declare" (in some 
> unknown sense) that it can be used for type annotations like MyList[T], 
> for some type T.

I suppose the run-time incarnations of the type descriptions
could be looser, but if you want to use them for static checking,
the static checker is going to have to know what MyList[T] means
in some detail (what effect the parameter has on the method
types, etc.) The programmer will have to specify all that
somehow.

The way this is done in other languages with static type
checking is to give the class declaration a parameter list.
I was envisaging that the mypy type description syntax would have
something equivalent. Not sure what form it would take, though.

-- 
Greg

From g.brandl at gmx.net  Sun Aug 24 10:00:32 2014
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 24 Aug 2014 10:00:32 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53F97E0E.6030908@canterbury.ac.nz>
References: <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <lt9mfn$tpn$1@ger.gmane.org>
 <53F93BC6.6030307@canterbury.ac.nz> <20140824031329.GD25957@ando>
 <53F97E0E.6030908@canterbury.ac.nz>
Message-ID: <ltc630$dlf$1@ger.gmane.org>

On 08/24/2014 07:54 AM, Greg Ewing wrote:
> Steven D'Aprano wrote:
>> I don't really understand what you're trying to say here, so I may be 
>> misinterpreting you. I *think* that you're trying to say that for every 
>> type in the standard library, and every class created by third parties 
>> (including subclasses), the author will have to "declare" (in some 
>> unknown sense) that it can be used for type annotations like MyList[T], 
>> for some type T.
> 
> I suppose the run-time incarnations of the type descriptions
> could be looser, but if you want to use them for static checking,
> the static checker is going to have to know what MyList[T] means
> in some detail (what effect the parameter has on the method
> types, etc.) The programmer will have to specify all that
> somehow.
> 
> The way this is done in other languages with static type
> checking is to give the class declaration a parameter list.
> I was envisaging that the mypy type description syntax would have
> something equivalent. Not sure what form it would take, though.

Exactly.  Does mypy handle that?  For example, for a custom mapping
type

class Mapping(object):
    def setitem(self, key, value):
        ...

how would one specify
a) that you can use Mapping[T1, T2] as a type annotation and
b) the type annotations for the "key" and "value" arguments?

Georg


From steve at pearwood.info  Sun Aug 24 10:50:03 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 24 Aug 2014 18:50:03 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <ltc630$dlf$1@ger.gmane.org>
References: <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
 <lt2sh5$9vi$1@ger.gmane.org> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <lt9mfn$tpn$1@ger.gmane.org>
 <53F93BC6.6030307@canterbury.ac.nz> <20140824031329.GD25957@ando>
 <53F97E0E.6030908@canterbury.ac.nz> <ltc630$dlf$1@ger.gmane.org>
Message-ID: <20140824085003.GE25957@ando>

On Sun, Aug 24, 2014 at 10:00:32AM +0200, Georg Brandl wrote:
> On 08/24/2014 07:54 AM, Greg Ewing wrote:
> > Steven D'Aprano wrote:
> >> I don't really understand what you're trying to say here, so I may be 
> >> misinterpreting you. I *think* that you're trying to say that for every 
> >> type in the standard library, and every class created by third parties 
> >> (including subclasses), the author will have to "declare" (in some 
> >> unknown sense) that it can be used for type annotations like MyList[T], 
> >> for some type T.
> > 
> > I suppose the run-time incarnations of the type descriptions
> > could be looser, but if you want to use them for static checking,
> > the static checker is going to have to know what MyList[T] means
> > in some detail (what effect the parameter has on the method
> > types, etc.) The programmer will have to specify all that
> > somehow.
> > 
> > The way this is done in other languages with static type
> > checking is to give the class declaration a parameter list.
> > I was envisaging that the mypy type description syntax would have
> > something equivalent. Not sure what form it would take, though.
> 
> Exactly.  Does mypy handle that?  For example, for a custom mapping
> type
> 
> class Mapping(object):
>     def setitem(self, key, value):
>         ...
> 
> how would one specify
> a) that you can use Mapping[T1, T2] as a type annotation and
> b) the type annotations for the "key" and "value" arguments?

I'm not an expert on Mypy, but I think the answer is, you can't. In 
order for Mypy to recognise your Mapping as an actual mapping, you have 
to either inherit from dict or some other mapping which Mypy knows 
about, or from typing.Generic.

http://mypy-lang.org/tutorial.html#genericclasses


from typing import typevar, Generic
T = typevar('T')
class Mapping(Generic[T, T]):
    ...


Now Mypy will know that your Mapping class should be considered a 
mapping from some type to another type, and you can use it:

def func(x:Mapping[int, str], n:int)->str:
    return x[n+1]


which will pass the static type checker.

(This is just what I understand from reading some of the docs, I welcome 
correction from anyone who knows better.)

But that's just Mypy. Another type checker might just trust you, 
regardless of what terrible lies you tell it, so long as they're 
consistent lies:

def func(x:set[int, str], n:int)->str:
    return x[n+1]


Since you've said x is a set which maps ints to strs, the code will pass 
the static check, but fail at run-time since sets don't actually support 
__getitem__. A third type checker might do structural typing instead of 
nominal typing, and be able to recognise that set doesn't have 
__getitem__ but Mapping does.

Remember that this proposal isn't about adding Mypy to the standard 
library or merging it with CPython. It remains a third-party 
implementation. People are encouraged to work on Mypy, or fork it, or 
build their own static tools, which may or may not do a better job of 
static analysis. Or runtime tools for that matter.


-- 
Steven

From abarnert at yahoo.com  Sun Aug 24 12:04:09 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 24 Aug 2014 03:04:09 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140824085003.GE25957@ando>
References: <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com>
 <lt2sh5$9vi$1@ger.gmane.org> <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <lt9mfn$tpn$1@ger.gmane.org>
 <53F93BC6.6030307@canterbury.ac.nz> <20140824031329.GD25957@ando>
 <53F97E0E.6030908@canterbury.ac.nz> <ltc630$dlf$1@ger.gmane.org>
 <20140824085003.GE25957@ando>
Message-ID: <389F0F28-31B1-4DBC-BA24-499CDE06FA38@yahoo.com>

On Aug 24, 2014, at 1:50, Steven D'Aprano <steve at pearwood.info> wrote:

> On Sun, Aug 24, 2014 at 10:00:32AM +0200, Georg Brandl wrote:
>> On 08/24/2014 07:54 AM, Greg Ewing wrote:
>>> Steven D'Aprano wrote:
>>>> I don't really understand what you're trying to say here, so I may be 
>>>> misinterpreting you. I *think* that you're trying to say that for every 
>>>> type in the standard library, and every class created by third parties 
>>>> (including subclasses), the author will have to "declare" (in some 
>>>> unknown sense) that it can be used for type annotations like MyList[T], 
>>>> for some type T.
>>> 
>>> I suppose the run-time incarnations of the type descriptions
>>> could be looser, but if you want to use them for static checking,
>>> the static checker is going to have to know what MyList[T] means
>>> in some detail (what effect the parameter has on the method
>>> types, etc.) The programmer will have to specify all that
>>> somehow.
>>> 
>>> The way this is done in other languages with static type
>>> checking is to give the class declaration a parameter list.
>>> I was envisaging that the mypy type description syntax would have
>>> something equivalent. Not sure what form it would take, though.
>> 
>> Exactly.  Does mypy handle that?  For example, for a custom mapping
>> type
>> 
>> class Mapping(object):
>>    def setitem(self, key, value):
>>        ...
>> 
>> how would one specify
>> a) that you can use Mapping[T1, T2] as a type annotation and
>> b) the type annotations for the "key" and "value" arguments?
> 
> I'm not an expert on Mypy, but I think the answer is, you can't. In 
> order for Mypy to recognise your Mapping as an actual mapping, you have 
> to either inherit from dict or some other mapping which Mypy knows 
> about, or from typing.Generic.
> 
> http://mypy-lang.org/tutorial.html#genericclasses
> 
> 
> from typing import typevar, Generic
> T = typevar('T')
> class Mapping(Generic[T, T]):
>    ...
> 
> 
> Now Mypy will know that your Mapping class should be considered a 
> mapping from some type to another type, and you can use it:

No it doesn't, it just knows that it's a generic type, meaning that you can index it with types, and those types will be used to fill in the corresponding typevars on its methods.

Consider Tuple[int, str]. That isn't a mapping from int to str, it's a 2-element tuple whose first element is int and whose second is str. And I'm sure you can come up with other uses for generic types of two type parameters that aren't mappings.

This is exactly the same as in C++, ML and its descendants, and Java and its descendants.

For example:

    class MyMapping(Generic[T, U]):
        def __getitem__(self, key: T) -> U:
            # ...

Now a function that takes a MyMapping[int, str] will bind T to int and U to str, so the type checker knows that the argument to __getitem__ is an int and the return value is a str.

And that still doesn't make it a Mapping. If collections.abc.Mapping (or typing.Mapping, if they're separate in the final version) is a Protocol, or otherwise implements structural matching, _that_ is what makes MyMapping a Mapping. And if if doesn't, then only inheriting or registering can make it a Mapping. Just like ABCs at runtime.


From edk141 at gmail.com  Sun Aug 24 13:39:30 2014
From: edk141 at gmail.com (Ed Kellett)
Date: Sun, 24 Aug 2014 12:39:30 +0100
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
Message-ID: <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>

I have a few questions:

- How often do multiple kinds of annotation end up on the same function?
- Why UUIDs (rather than for e.g. PyPI packages, or some other namespace)?
- What is the point? A decorator could process the annotations and put
some information in func._projectname__something instead of doing the
UUID dance
- What would pydoc print for the function signature?

On 24 August 2014 03:18, Cem Karan <cfkaran2 at gmail.com> wrote:
>
> On Aug 20, 2014, at 11:08 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> Sigh. I go away for a week and come back to a mega-thread I can never
>> hope to catch up on :-)
>>
>> TL; DR; - Although mypy looks interesting, I think it's too soon to
>> close the door on all other uses of annotations. Let's find a solution
>> that allows exploration of alternative uses for a while longer.
>>
>> OK, can I just make some points regarding the static typing thread.
>> First of all, I have no issue with the idea of static typing, and in
>> fact I look forward to seeing what benefits it might have (if nothing
>> else, the pointer to mypy, which I'd never heard of before, is
>> appreciated). It won't be something I use soon (see below) but that's
>> fine.
>>
>> But Guido seems to be saying (on a number of occasions) that nobody is
>> really using annotations, so he wants to focus on the static typing
>> use case alone. I think this is a mistake. First of all, I see no
>> reason why functions using typing annotations could not be introduced
>> with a decorator[1]. So why insist that this is the *only* use of
>> annotations, when it's pretty easy to allow others to co-exist?
>>
>> Also, the "nobody is using annotations" argument? Personally, I know
>> of a few other uses:
>>
>> 1. Argument parsers - at least 3 have been mentioned in the thread.
>> 2. Structure unpacking - I think there is a library that uses
>> annotations for this, although I may be wrong.
>> 3. FFI bindings. I know I've seen this discussed, although I can't
>> find a reference just now.
>>
>> There are probably other ideas around as well (GUI bindings,
>> documentation generation, ...) None are particularly mature, and most
>> are just at the "ideas" stage, but typically the ideas I have seen are
>> the sort of thing you'd write a library for, and Python 3 only
>> libraries *really* aren't common yet.
>>
>> The problem for people wanting to experiment with annotations, is that
>> they need to be writing Python 3 only code. While Python 3 adoption is
>> growing rapidly, I suspect that large applications are typically still
>> focused on either going through, or tidying up after, a 2-3 migration.
>> And new projects, while they may be developed from the ground up using
>> Python 3, will typically be using programmers skilled in Python 2, to
>> whom Python 3 features are not yet an "instinctive" part of the
>> toolset. Apart from large standalone applications, there are smaller
>> scripts (which are typically going to be too small to need
>> programming-in-the-large features like annotations) and libraries
>> (which really aren't yet in a position to drop Python 2.x totally,
>> unless they have a fairly small user base).
>>
>> So I don't see it as compelling that usage of annotations in the wild
>> is not yet extensive.
>>
>> Rather than close the door on alternative uses of annotations, can I suggest:
>>
>> 1. By all means bless mypy syntax as the standard static typing
>> notation - this seems like a good thing.
>> 2. Clarify that static typing annotations should be introduced with a
>> decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy
>> declaration in the stdlib, and have a registration protocols for tools
>> to hook their own implementation into it.[2]
>> 3. Leave the door open for other uses of decorators, at least until
>> some of the more major libraries drop Python 2.x support completely
>> (and hence can afford to have a dependency on a Python 3 only module
>> that uses annotations). See at that stage if annotations take off.
>> 4. If we get to a point where even libraries that *could* use
>> annotations don't, then revisit the idea of restricting usage to just
>> type information.
>>
>> Paul
>>
>> [1] Also, a decorator could allow a Python 2 compatible form by using
>> decorator arguments as an alternative to annotations.
>> [2] I've yet to see a clear explanation of how "a tool using type
>> annotations" like an linter, editor, IDE or Python compiler would use
>> them in such a way that precludes decoration as a means of signalling
>> the annotation semantics.
>
> A long while back I proposed a mechanism for sharing __annotations__ between multiple, non-cooperating projects.  The basic idea is that each annotation becomes a dictionary.  Each project (and 'project' is a very loosely defined concept here) chooses a UUID that it uses as key into the dictionary.  The value is up to the project.
>
> The advantage to this is manifold:
>
> - Annotations can still have multiple uses by different groups without stepping on each other's toes.
> - If someone wants to make a standard, all they have to do is publish the UUID associated with their standard.  For example, we might choose UUID('2cca6238-9fca-4053-aa3d-db9050e6b26b') as the official type information UUID.  All projects that want to develop linters, documentation generators, etc., will use that UUID for all annotations, and the PEP will require it.
> - de facto standards can become de jure standards by blessing a particular UUID.
> - Guessing if this method is being used is relatively easy; if its a dictionary, and if every key is a UUID, it probably follows this standard.  We can tighten it further by requiring some key-value pair be in every dictionary (i.e., {UUID('1ad60d50-8237-4b98-b2b1-69fd08ed575c'):"PEPXXXX"} is always in the dictionary).  This makes it fairly simple to add without stomping on what people are already doing.
> - Finding the standard on the web should also be easy; while you might not find the PEP instantly, you'll probably zoom into it fairly fast.
>
> Disadvantages:
>
> - Typing out UUIDs is PAINFUL.  I highly recommend using decorators instead.
> - Reading the __annotations__ dictionary will be difficult.  pprint() should make this easier.
>
> I have working proof-of-concept code at https://github.com/oranguman/annotizer that defines a decorator class that handles the UUID for you.  It needs to be extended to parse out information, but it handles the 'other use cases' problem fairly well.
>
> Thanks,
> Cem Karan
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

From steve at pearwood.info  Sun Aug 24 13:59:55 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 24 Aug 2014 21:59:55 +1000
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <CAP1=2W70P_Mdeup3bj6om+0N_HrR2cv-QMXHNKw5OiU79Je4Ww@mail.gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org>
 <CAP1=2W70P_Mdeup3bj6om+0N_HrR2cv-QMXHNKw5OiU79Je4Ww@mail.gmail.com>
Message-ID: <20140824115954.GF25957@ando>

On Thu, Aug 21, 2014 at 02:30:03PM +0000, Brett Cannon wrote:

> P.S.: Off-topic for this email but something I don't think has been
> mentioned more than once, type hinting like we are proposing is exactly
> what Dart does and it is nice. Built-in API documentation of interfaces
> along with IDE support at the API makes all of this worth it.

For anyone not familiar with Dart, I think it is worth reading the Dart 
justification for static type hinting in a dynamic language:

https://www.dartlang.org/articles/optional-types/

https://www.dartlang.org/articles/why-dart-types/

It's not quite the same as Guido's proposal, for example in Dart the 
type hints have no runtime effect at all, but I think it demonstrates 
that this is a proven approach and can work well with a dynamic language 
like Python without compromising the dynamic nature of the language.


-- 
Steven

From __peter__ at web.de  Sun Aug 24 14:35:57 2014
From: __peter__ at web.de (Peter Otten)
Date: Sun, 24 Aug 2014 14:35:57 +0200
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <20140823174358.GA25957@ando>
 <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>
Message-ID: <ltcm7e$fn$1@ger.gmane.org>

David Mertz wrote:

>   def reduce_with_attractor(func, it, start=None, end_if=None):
>       it = iter(it)
>       start = start if start!=None else it.__next__()
>       return list(takewhile(lambda x: x!=end_if,
>                             accumulate(chain([start],it), func)))[-1]

Wouldn't it be better to break this into a function that limits a sequence 
and to combine that with the original reduce()?

>>> from functools import reduce
>>> from itertools import takewhile
>>> def stop_on(value, items):
...     return takewhile(lambda item: item != value, items)
... 
>>> list(stop_on(0, [1, 2, 3, 0, 4]))
[1, 2, 3]
>>> from operator import mul
>>> reduce(mul, stop_on(0, [1, 2, 3, 0, 4]))
6

My suggestion is to add a value-limited version of itertools.takewhile() 
rather than making reduce() more more powerful/complex.


From steve at pearwood.info  Sun Aug 24 14:41:39 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 24 Aug 2014 22:41:39 +1000
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <846C4603-A7EF-4394-8CE9-4B8C614E1E13@gmail.com>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <CAP7+vJKJP6Nh42X70VmmFKwrf3XpJX3-JC5R9guNgmW8mk7_oA@mail.gmail.com>
 <846C4603-A7EF-4394-8CE9-4B8C614E1E13@gmail.com>
Message-ID: <20140824124139.GG25957@ando>

On Sat, Aug 23, 2014 at 08:37:34PM -0400, Cem Karan wrote:

> Going slightly sideways on this -- I think this is why we should use 
> decorators instead of annotations; as has already been mentioned twice 
> on the list, 
> http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ does a 
> good job pointing out that static and dynamic typing systems are 
> separate but overlapping concepts.  It also points out that both have 
> their place.  If we define decorators that can be turned on or off 
> easily (command-line option?  Environment variable? other?), then the 
> end user can choose if if her or she is going to do static, dynamic, 
> both, or none.  This could be useful when trying to track down that 
> annoying bug in long-running production code.

This applies equally to annotations, and in fact that's exactly what 
Mypy already does. You can run Mypy to do static type checking, or not 
run it, and the annotations will be ignored.

E.g. given a module program.py, you can:

# Type check and then run the program:
mypy program.py

# Just run it, with no extra type checks:
python3 program.py


Guido's proposal is *not* to add static types to the CPython 
interpreter, at least not yet. It is just to standardise on annotations 
for type hinting, agree on a syntax for those type hints, and then allow 
third-part tools (linters, editors, IDEs, etc.) and alternative 
interpreters (like mypy) to actually use the type hints.

Looking forward to the distant future, if CPython gains its own built-in 
type checker, it will probably come with a runtime switch to enable or 
disable such type checking. But that's possible regardless of whether we 
use decorators, annotations, or both.


> Also, Types and Programming Languages by Dr. Pierce has been mentioned 
> at least twice as well; would it be useful to ask him to join in the 
> discussion?

Does he know anything about Python? Will he care? There are hundreds of 
programming languages, unless he has a particular interest in Python I 
can't see why he would care about this discussion. But if you are a 
colleague or friend of his, by all means invite him to join up, this is 
a public forum.



-- 
Steven

From steve at pearwood.info  Sun Aug 24 15:09:08 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 24 Aug 2014 23:09:08 +1000
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <ltcm7e$fn$1@ger.gmane.org>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <20140823174358.GA25957@ando>
 <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>
 <ltcm7e$fn$1@ger.gmane.org>
Message-ID: <20140824130908.GH25957@ando>

On Sun, Aug 24, 2014 at 02:35:57PM +0200, Peter Otten wrote:
> David Mertz wrote:
> 
> >   def reduce_with_attractor(func, it, start=None, end_if=None):
> >       it = iter(it)
> >       start = start if start!=None else it.__next__()
> >       return list(takewhile(lambda x: x!=end_if,
> >                             accumulate(chain([start],it), func)))[-1]
> 
> Wouldn't it be better to break this into a function that limits a sequence 
> and to combine that with the original reduce()?

In general, if x is an attractor, then we want to stop and return x if 
either of these two scenarios occur:

- we come across x in the input;

- or the sequence of intermediate values reaches x.

You can do the first by filtering the input stream, but not the second.

Here's a concrete example, sticking to product() where 0 is an 
attractor:

# best viewed with a fixed-width font
product([1, 2, 3, 9, 0, 8, 7, 4])
.....................^ stop here


But multiplication can underflow to zero too, and once it does, 
we likewise want to stop:

product([1e-70, 2e-71, 6e-69, 4e-68, 1e-48, 2e-71, 1e-69, 3e-70])
.....................................^ stop here


Notice that 1e-48 is not only non-zero, but it's considerably bigger 
than the other numbers in the sequence. (About 100000000000000000000 
times bigger, give or take a factor of 10.) Yet it's enough to cause the 
product to underflow to zero, after which the product will never 
shift away from zero. (Ignoring NANs and INFs.)

I'm still not convinced this belongs in the standard library, but it's a 
nice functional, er, function to add to your private library or as a 
third-party module.



-- 
Steven

From cfkaran2 at gmail.com  Sun Aug 24 15:32:38 2014
From: cfkaran2 at gmail.com (Cem Karan)
Date: Sun, 24 Aug 2014 09:32:38 -0400
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
 <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
Message-ID: <DD3FE481-6C1B-409C-BB4E-1564376DE52D@gmail.com>


On Aug 24, 2014, at 7:39 AM, Ed Kellett <edk141 at gmail.com> wrote:

> I have a few questions:
> 
> - How often do multiple kinds of annotation end up on the same function?

In this thread, we've already talked about type checkers and documentation generators, both of which can use the __annotations__ dictionary legitimately.  Now imagine that you are an end user that has installed a documentation generator and a static type analyzer.  If both tools were to use the __annotations__ dictionary, then right now you could choose one or the other tool, but not both.  However, if the standard I'm proposing was adopted, then each tool would choose its own UUID as its key, which would mean they could share entries in the annotations dictionary.

> - Why UUIDs (rather than for e.g. PyPI packages, or some other namespace)?

Multiple reasons:

- Using PyPI means that every programmer that is trying to follow the standard (including anyone who is just learning python) will create some name while practicing.  That name will need to be pushed up to PyPI to ensure uniqueness.  Since most of these names are only for learning, PyPI will immediately get flooded with a bunch of project names that are probably going to be abandoned almost immediately (I'm thinking of beginning programming classes especially).  This would significantly degrade the utility of PyPI, which I want to avoid.  All similar centrally-managed systems will suffer from the same problem.  UUIDs don't have this problem; create and abandon them at will.

- All other namespace systems will either suffer from the possibility of collisions, require a centrally managed repository of names, or will eventually reinvent UUIDs.  We already have UUIDs, lets skip the first couple of headaches and just solve the problem.

- Centrally managed systems have a much higher barrier to entry than simple UUIDs.  Getting a new UUID to experiment with is trivial; "import uuid; uuid.uuid4()" is our complete program, requires no management on the part of PyPI (or any other third party), doesn't require internet access, etc.  

- UUIDs have no built-in human significance; it is VERY unlikely that multiple projects will accidentally choose the same UUID.  E.g., it is likely that programmers developing different type checkers would choose 'type checker' as a key, and each project will have incompatible meanings/values for the 'type checker' key.  This doesn't sound too bad until you start pulling in multiple frameworks from different sources, each of which uses a different, mutually incompatible type checker system.  At that point, running any type checker will cause a crash as the type checker tries to read information from the frameworks you've just pulled in.

- Google for a UUID.  Any UUID.  If you've just generated the UUID, you are unlikely to get one come up.  Now google for 'UUID('f9bbc165-d904-4452-b858-fc5c9f104c87')'.  I've just added it to the README for my annotizer project, and I expect googlebot to pick it up in the next few days.  At that point, the only two places you should find mention of that UUID is on github, and in this thread.  If you google for 'type checker', etc., how many hits do you get?  How many of them relate to mypy, or even this thread?  Once people are used to the standard, they'll know that to find out information about what project is associated with a given UUID they just need to google for it.  This is a big win.  Actually, do this as an experiment: don't look at the URL below, instead, wait a few days and google for the UUID above.  See what comes up.

- Making a standard for __annotations__ at this point isn't easy; we need a simple way of deciding if someone is complying the the standard.  This is pretty easy if we adopt some UUID as a required key as I mentioned earlier.  

> - What is the point? A decorator could process the annotations and put
> some information in func._projectname__something instead of doing the
> UUID dance

Again, this isn't for consumption within a project, it is for users across projects.  What if I want to use Sphinx (http://sphinx-doc.org/) and mypy (http://www.mypy-lang.org/) at the same time in my project?  What happens in the following code?

"""
@sphinx_decorator(a, "Some documentation about a")
@mypy_decorator(a, int))
def foo(a):
    pass
"""

Is it the same as:

"""
@mypy_decorator(a, int))
@sphinx_decorator(a, "Some documentation about a")
def foo(a):
    pass
"""

Right now, as I understand it, the last applied decorator would win, which means 'func._projectname__something' would be set to either sphinx or mypy.  That means that order matters for completely orthogonal concepts.  This is bad.  UUIDs solve this, and all the earlier problems.

> - What would pydoc print for the function signature?

As I mentioned earlier, certain UUIDs might become de facto or de jure standards.  In this case, projects that have common goals could settle on a common standard and publish a common UUID.  Pydoc would know about these UUIDs (they would be published), and would know what to do for them.  For UUIDs it doesn't understand, it could raise a warning, or simply ignore them.

Before you take my comments above as proof the we don't need UUIDs, consider the fact that we are currently discussing type systems, and our thoughts may change in the future.  I don't mean that there will be successive standards, I mean that there may be competing standards, at least until we really know what the best one will be.  This is a case where creating and abandoning UUIDs will be trivial, but where using 'type checker' is going to lead to confusion.

> On 24 August 2014 03:18, Cem Karan <cfkaran2 at gmail.com> wrote:
>> 
>> On Aug 20, 2014, at 11:08 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>> 
>>> Sigh. I go away for a week and come back to a mega-thread I can never
>>> hope to catch up on :-)
>>> 
>>> TL; DR; - Although mypy looks interesting, I think it's too soon to
>>> close the door on all other uses of annotations. Let's find a solution
>>> that allows exploration of alternative uses for a while longer.
>>> 
>>> OK, can I just make some points regarding the static typing thread.
>>> First of all, I have no issue with the idea of static typing, and in
>>> fact I look forward to seeing what benefits it might have (if nothing
>>> else, the pointer to mypy, which I'd never heard of before, is
>>> appreciated). It won't be something I use soon (see below) but that's
>>> fine.
>>> 
>>> But Guido seems to be saying (on a number of occasions) that nobody is
>>> really using annotations, so he wants to focus on the static typing
>>> use case alone. I think this is a mistake. First of all, I see no
>>> reason why functions using typing annotations could not be introduced
>>> with a decorator[1]. So why insist that this is the *only* use of
>>> annotations, when it's pretty easy to allow others to co-exist?
>>> 
>>> Also, the "nobody is using annotations" argument? Personally, I know
>>> of a few other uses:
>>> 
>>> 1. Argument parsers - at least 3 have been mentioned in the thread.
>>> 2. Structure unpacking - I think there is a library that uses
>>> annotations for this, although I may be wrong.
>>> 3. FFI bindings. I know I've seen this discussed, although I can't
>>> find a reference just now.
>>> 
>>> There are probably other ideas around as well (GUI bindings,
>>> documentation generation, ...) None are particularly mature, and most
>>> are just at the "ideas" stage, but typically the ideas I have seen are
>>> the sort of thing you'd write a library for, and Python 3 only
>>> libraries *really* aren't common yet.
>>> 
>>> The problem for people wanting to experiment with annotations, is that
>>> they need to be writing Python 3 only code. While Python 3 adoption is
>>> growing rapidly, I suspect that large applications are typically still
>>> focused on either going through, or tidying up after, a 2-3 migration.
>>> And new projects, while they may be developed from the ground up using
>>> Python 3, will typically be using programmers skilled in Python 2, to
>>> whom Python 3 features are not yet an "instinctive" part of the
>>> toolset. Apart from large standalone applications, there are smaller
>>> scripts (which are typically going to be too small to need
>>> programming-in-the-large features like annotations) and libraries
>>> (which really aren't yet in a position to drop Python 2.x totally,
>>> unless they have a fairly small user base).
>>> 
>>> So I don't see it as compelling that usage of annotations in the wild
>>> is not yet extensive.
>>> 
>>> Rather than close the door on alternative uses of annotations, can I suggest:
>>> 
>>> 1. By all means bless mypy syntax as the standard static typing
>>> notation - this seems like a good thing.
>>> 2. Clarify that static typing annotations should be introduced with a
>>> decorator. Maybe reserve a decorator name ("@typed"?) that has a dummy
>>> declaration in the stdlib, and have a registration protocols for tools
>>> to hook their own implementation into it.[2]
>>> 3. Leave the door open for other uses of decorators, at least until
>>> some of the more major libraries drop Python 2.x support completely
>>> (and hence can afford to have a dependency on a Python 3 only module
>>> that uses annotations). See at that stage if annotations take off.
>>> 4. If we get to a point where even libraries that *could* use
>>> annotations don't, then revisit the idea of restricting usage to just
>>> type information.
>>> 
>>> Paul
>>> 
>>> [1] Also, a decorator could allow a Python 2 compatible form by using
>>> decorator arguments as an alternative to annotations.
>>> [2] I've yet to see a clear explanation of how "a tool using type
>>> annotations" like an linter, editor, IDE or Python compiler would use
>>> them in such a way that precludes decoration as a means of signalling
>>> the annotation semantics.
>> 
>> A long while back I proposed a mechanism for sharing __annotations__ between multiple, non-cooperating projects.  The basic idea is that each annotation becomes a dictionary.  Each project (and 'project' is a very loosely defined concept here) chooses a UUID that it uses as key into the dictionary.  The value is up to the project.
>> 
>> The advantage to this is manifold:
>> 
>> - Annotations can still have multiple uses by different groups without stepping on each other's toes.
>> - If someone wants to make a standard, all they have to do is publish the UUID associated with their standard.  For example, we might choose UUID('2cca6238-9fca-4053-aa3d-db9050e6b26b') as the official type information UUID.  All projects that want to develop linters, documentation generators, etc., will use that UUID for all annotations, and the PEP will require it.
>> - de facto standards can become de jure standards by blessing a particular UUID.
>> - Guessing if this method is being used is relatively easy; if its a dictionary, and if every key is a UUID, it probably follows this standard.  We can tighten it further by requiring some key-value pair be in every dictionary (i.e., {UUID('1ad60d50-8237-4b98-b2b1-69fd08ed575c'):"PEPXXXX"} is always in the dictionary).  This makes it fairly simple to add without stomping on what people are already doing.
>> - Finding the standard on the web should also be easy; while you might not find the PEP instantly, you'll probably zoom into it fairly fast.
>> 
>> Disadvantages:
>> 
>> - Typing out UUIDs is PAINFUL.  I highly recommend using decorators instead.
>> - Reading the __annotations__ dictionary will be difficult.  pprint() should make this easier.
>> 
>> I have working proof-of-concept code at https://github.com/oranguman/annotizer that defines a decorator class that handles the UUID for you.  It needs to be extended to parse out information, but it handles the 'other use cases' problem fairly well.
>> 
>> Thanks,
>> Cem Karan

Thanks,
Cem Karan

From ncoghlan at gmail.com  Sun Aug 24 15:57:20 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Aug 2014 23:57:20 +1000
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <DD3FE481-6C1B-409C-BB4E-1564376DE52D@gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
 <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
 <DD3FE481-6C1B-409C-BB4E-1564376DE52D@gmail.com>
Message-ID: <CADiSq7cvoU=9+vgj4D9g96Dnkb+Xa=HzvZ35kNw4Ndb26=dE3A@mail.gmail.com>

On 24 August 2014 23:32, Cem Karan <cfkaran2 at gmail.com> wrote:
>
> - Using PyPI means that every programmer that is trying to follow the standard (including anyone who is just learning python) will create some name while practicing.  That name will need to be pushed up to PyPI to ensure uniqueness.  Since most of these names are only for learning, PyPI will immediately get flooded with a bunch of project names that are probably going to be abandoned almost immediately (I'm thinking of beginning programming classes especially).  This would significantly degrade the utility of PyPI, which I want to avoid.  All similar centrally-managed systems will suffer from the same problem.  UUIDs don't have this problem; create and abandon them at will.

If someone is just experimenting locally, it doesn't matter if they
pick a conflicting name. If they're distributing, they're going to
need to register a name on PyPI anyway.

The fact that potential conflicts only matter once people start to
consider distribution is crystal clear for distribution metadata,
which is why PyPI names are the proposed namespacing mechanism for
avoiding naming conflicts for extensions to metadata 2.0 (see the PEP
426 draft for details).

(Note I don't really like this idea in general - if it's not for
humans, then it doesn't need to be in the annotations dict, it can go
in a separate file. But the UUIDs for namespacing idea fails badly on
the "Readability counts" front)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From steve at pearwood.info  Sun Aug 24 16:03:14 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 25 Aug 2014 00:03:14 +1000
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <DD3FE481-6C1B-409C-BB4E-1564376DE52D@gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
 <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
 <DD3FE481-6C1B-409C-BB4E-1564376DE52D@gmail.com>
Message-ID: <20140824140313.GI25957@ando>

On Sun, Aug 24, 2014 at 09:32:38AM -0400, Cem Karan wrote:
> 
> On Aug 24, 2014, at 7:39 AM, Ed Kellett <edk141 at gmail.com> wrote:
> 
> > I have a few questions:
> > 
> > - How often do multiple kinds of annotation end up on the same function?
> 
> In this thread, we've already talked about type checkers and 
> documentation generators, both of which can use the __annotations__ 
> dictionary legitimately.  Now imagine that you are an end user that 
> has installed a documentation generator and a static type analyzer.  
> If both tools were to use the __annotations__ dictionary, then right 
> now you could choose one or the other tool, but not both.

Not necessarily. It depends on what the tools expect to find in the 
annotations. If type hints become the standard for annotations, which is 
Guido's proposal, then both tools will expect to find type hints, and 
both can use them.

As for the cases where the tool or library wants to use annotations for 
something else, I'm quite happy to say that, in those cases, you can't 
use both. It's like docstrings. Docstrings are for documentation, 
including the interactive help system. If a library wants to use __doc__ 
for something else, it can (within the limits that it can only be a 
string, or None). But a user of that library cannot expect to *both* use 
the function __doc__ as a docstring *and* for this other purpose at the 
same time. They have to pick one.


> However, if 
> the standard I'm proposing was adopted, then each tool would choose 
> its own UUID as its key, which would mean they could share entries in 
> the annotations dictionary.

I don't think Guido wants to encourage multiple, incompatible uses for 
function annotations. It is going to be tricky enough to persuade him to 
merely allow other uses.

My feeling is that this is a case of YAGNI. I don't think there are 
going to be so many competing uses for function annotations in common 
use that this will be necessary. 


-- 
Steven

From edk141 at gmail.com  Sun Aug 24 16:05:19 2014
From: edk141 at gmail.com (Ed Kellett)
Date: Sun, 24 Aug 2014 15:05:19 +0100
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <DD3FE481-6C1B-409C-BB4E-1564376DE52D@gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
 <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
 <DD3FE481-6C1B-409C-BB4E-1564376DE52D@gmail.com>
Message-ID: <CABmzr0ixTjL+a0GoHrT6TsWwt_zkQuDV42doDTA_LsApKUCuAg@mail.gmail.com>

> In this thread, we've already talked about type checkers and documentation generators, both of which can use the __annotations__ dictionary legitimately.

They shouldn't be putting things in it, though, they should be
extracting data from it when they're told to by means of a decorator.
(I know the original proposal didn't mention using a decorator for
standard type information annotations and advocated deprecating other
annotations instead, but I think that was wrong and judging by some of
the other posts in the thread it seems I'm not the only one).

You pick the thing you're going to use annotations for and decorate
your function with its "process this function by means of its
annotations" decorator; there's no potential for conflict between
tools since each function can have only one set of annotations anyway.

> Right now, as I understand it, the last applied decorator would win, which means 'func._projectname__something' would be set to either sphinx or mypy.  That means that order matters for completely orthogonal concepts.  This is bad.  UUIDs solve this, and all the earlier problems.

You've missed my point (that's not your fault; I could have expressed
it better): the key '_projectname__something' won't be used, the keys
_mypy__types and _sphinx__doc (or similar) would be.

>
>> - What would pydoc print for the function signature?
>
> As I mentioned earlier, certain UUIDs might become de facto or de jure standards.  In this case, projects that have common goals could settle on a common standard and publish a common UUID.  Pydoc would know about these UUIDs (they would be published), and would know what to do for them.  For UUIDs it doesn't understand, it could raise a warning, or simply ignore them.
>
> Before you take my comments above as proof the we don't need UUIDs, consider the fact that we are currently discussing type systems, and our thoughts may change in the future.  I don't mean that there will be successive standards, I mean that there may be competing standards, at least until we really know what the best one will be.  This is a case where creating and abandoning UUIDs will be trivial, but where using 'type checker' is going to lead to confusion.


I don't think your comments prove that we don't need UUIDs, I just
don't think they prove that we do. Decorators solve this problem
adequately; the only issue that will arise is if annotations without a
decorator are supposed to have some standard semantics.


As more or less a tangential point, I think most of the advantages
you've listed of UUIDs are counteracted by how unintuitive, unreadable
and forgettable they are.

From cfkaran2 at gmail.com  Sun Aug 24 16:11:30 2014
From: cfkaran2 at gmail.com (Cem Karan)
Date: Sun, 24 Aug 2014 10:11:30 -0400
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <20140824124139.GG25957@ando>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <CAP7+vJKJP6Nh42X70VmmFKwrf3XpJX3-JC5R9guNgmW8mk7_oA@mail.gmail.com>
 <846C4603-A7EF-4394-8CE9-4B8C614E1E13@gmail.com>
 <20140824124139.GG25957@ando>
Message-ID: <7C2692DB-36FE-43E8-B445-A87E7146E739@gmail.com>


On Aug 24, 2014, at 8:41 AM, Steven D'Aprano <steve at pearwood.info> wrote:

> On Sat, Aug 23, 2014 at 08:37:34PM -0400, Cem Karan wrote:
> 
>> Going slightly sideways on this -- I think this is why we should use 
>> decorators instead of annotations; as has already been mentioned twice 
>> on the list, 
>> http://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ does a 
>> good job pointing out that static and dynamic typing systems are 
>> separate but overlapping concepts.  It also points out that both have 
>> their place.  If we define decorators that can be turned on or off 
>> easily (command-line option?  Environment variable? other?), then the 
>> end user can choose if if her or she is going to do static, dynamic, 
>> both, or none.  This could be useful when trying to track down that 
>> annoying bug in long-running production code.
> 
> This applies equally to annotations, and in fact that's exactly what 
> Mypy already does. You can run Mypy to do static type checking, or not 
> run it, and the annotations will be ignored.
> 
> E.g. given a module program.py, you can:
> 
> # Type check and then run the program:
> mypy program.py
> 
> # Just run it, with no extra type checks:
> python3 program.py
> 
> 
> Guido's proposal is *not* to add static types to the CPython 
> interpreter, at least not yet. It is just to standardise on annotations 
> for type hinting, agree on a syntax for those type hints, and then allow 
> third-part tools (linters, editors, IDEs, etc.) and alternative 
> interpreters (like mypy) to actually use the type hints.

OK, I think I see what you're saying.  Since the annotations will always be available, the static/dynamic nature doesn't matter, since we just choose the flags/interpreter/whatever, and it will check the annotations, correct?  If so, then you're right, and we don't need decorators for this.

> Looking forward to the distant future, if CPython gains its own built-in 
> type checker, it will probably come with a runtime switch to enable or 
> disable such type checking. But that's possible regardless of whether we 
> use decorators, annotations, or both.
> 
>> Also, Types and Programming Languages by Dr. Pierce has been mentioned 
>> at least twice as well; would it be useful to ask him to join in the 
>> discussion?
> 
> Does he know anything about Python? Will he care? There are hundreds of 
> programming languages, unless he has a particular interest in Python I 
> can't see why he would care about this discussion. But if you are a 
> colleague or friend of his, by all means invite him to join up, this is 
> a public forum.

I just invited him.

Thanks,
Cem Karan

From antoine at python.org  Sun Aug 24 16:28:00 2014
From: antoine at python.org (Antoine Pitrou)
Date: Sun, 24 Aug 2014 10:28:00 -0400
Subject: [Python-ideas] [OT] Re: Add nullifier argument to functools.reduce?
In-Reply-To: <CADiSq7ffiRmJK5YFWZHeBrj3Z3ZsGkGtXOFJ5hePmc4tv6BEwQ@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CADiSq7ffiRmJK5YFWZHeBrj3Z3ZsGkGtXOFJ5hePmc4tv6BEwQ@mail.gmail.com>
Message-ID: <ltcspg$dn6$2@ger.gmane.org>

Le 24/08/2014 00:24, Nick Coghlan a ?crit :
> On 24 August 2014 01:30, Warren Weckesser <warren.weckesser at gmail.com> wrote:
>> I'd like to add an additional optional argument to functools.reduce.
>> The argument is the "nullifier" of the reducing operation.  It is a value
>> such that function(nullifier, anything) returns nullifier.  For example, if
>> function(x, y) computes x*y, the nullifier is 0.  If function(x, y) is
>> the intersection of the sets x and y, the nullifier is the empty set.
>
> When it comes to judging the usefulness of functional programming
> features these days, my first question is generally going to be "Does
> PyToolz offer this?"
>
> Grumblings about the name aside,

I suppose it's for functional whizkidz and cowboyz?

Regards

Antoine.



From warren.weckesser at gmail.com  Sun Aug 24 16:45:11 2014
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sun, 24 Aug 2014 10:45:11 -0400
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
Message-ID: <CAGzF1uf25d-mOC8iuoXnVXQCP4QjHi_21qpogjfGXcxZ_02xEQ@mail.gmail.com>

On Sat, Aug 23, 2014 at 1:25 PM, David Mertz <mertz at gnosis.cx> wrote:

> This "nullifier" is mathematically called an "absorbing element", but
> saying an "attractor" might be a little more general.  I.e. think of a
> local optimization problem, where multiple local min/max points might
> occur.  If you reach one, further iteration won't budge from that point,
> even if it's not the "global absorbing element."
>
>

I took the name "nullifier" from
    http://www.mayhematics.com/m/m2_operations.htm
but a less "mathy" name would probably be better.  For consistency, I'll
stick with it, but you can think of it as a placeholder for a better name
to be determined later.



> However, any such argument--including the much more useful
> sentinel/'stop_on' idea--significantly changes the semantics of reduce.  In
> particular, as is reduce() always consumes its iterator.  Under these
> changes, it may or may not consume the iterator, depending on what elements
> occur.
>
>

I don't agree that this change "significantly changes the semantics of
reduce".  The nullifier is optional.  The short-circuit can only occur when
the caller has specified a nullifier, so anyone using it will be aware that
the iterable might not be consumed.  Indeed, that's the *point* of using it.

Assuming the nullifier given is truly a nullifier of the function, the
result of the call to reduce should be the same (other than the amount by
which the iterable  has been consumed) whether or not the nullifier is
given.  (That was also Hernan's point.)



> Given that one can easily write one's own three line wrapper
> 'reduce_with_attractor()' for this special semantics which hasn't been
> given a use case,
>


My interest in the enhancement is purely performance.  Here's an example
(call it a use case, if you like) that is obviously cooked up to maximize
the benefit of short-circuiting.  In the following ipython session,
`reduce` is the builtin function  (I'm using python 2.7 here), and
`myreduce.reduce` is the python implementation with the nullifier argument:

    In [40]: a = range(100)

    In [41]: %timeit reduce(lambda x, y: x*y, a)
    100000 loops, best of 3: 8.89 ?s per loop

    In [42]: %timeit myreduce.reduce(lambda x, y: x*y, a, nullifier=0)
    1000000 loops, best of 3: 455 ns per loop



> I can't see a point of including the argument in the stdlib.
>
> -1 on proposal.
>
>
>

I think the main objection (here and in other comments) is that the benefit
(performance gain in certain cases) does not outweigh the cost (a more
complicated API for reduce(), and more code and documentation to maintain
in the standard library).  That's a compelling argument!  I think the
enhancement would be useful, but I understand that it might not be useful
enough to accept such a change, especially since the cost of *not* having
it in the library to a user who wants such a feature is pretty low (i.e. it
is easy to "roll your own").

Warren



>
> On Sat, Aug 23, 2014 at 8:30 AM, Warren Weckesser <
> warren.weckesser at gmail.com> wrote:
>
>> I'd like to add an additional optional argument to functools.reduce.
>> The argument is the "nullifier" of the reducing operation.  It is a value
>> such that function(nullifier, anything) returns nullifier.  For example,
>> if
>> function(x, y) computes x*y, the nullifier is 0.  If function(x, y) is
>> the intersection of the sets x and y, the nullifier is the empty set.
>>
>> The argument would allow reduce to "short circuit" its calculation.   When
>> reduce encounters the nullifier, it can return immediately.  This can
>> provide
>> a significant improvement in performance in some cases.
>>
>> The change is simple.  Here, for example, is  the "rough equivalent" for
>> functools.reduce from the docs:
>>
>>     def reduce(function, iterable, initializer=None):
>>         it = iter(iterable)
>>         if initializer is None:
>>             try:
>>                 initializer = next(it)
>>             except StopIteration:
>>                 raise TypeError('reduce() of empty sequence with no
>> initial value')
>>         accum_value = initializer
>>         for x in it:
>>             accum_value = function(accum_value, x)
>>         return accum_value
>>
>> Here's how it looks with the optional nullifier argument; the only
>> change is the new argument and an 'if' statement in the 'for' loop.
>>
>>     def reduce(function, iterable, initializer=None, nullifier=None):
>>         it = iter(iterable)
>>         if initializer is None:
>>             try:
>>                 initializer = next(it)
>>             except StopIteration:
>>                 raise TypeError('reduce() of empty sequence with no
>> initial value')
>>         accum_value = initializer
>>         for x in it:
>>             if nullifier is not None and accum_value == nullifier:
>>                 break
>>             accum_value = function(accum_value, x)
>>         return accum_value
>>
>> (It might be better to use a distinct singleton for the default
>> value of nullifier, to allow None to be a valid nullifier.)
>>
>> The actual implementation is in the extension module _functoolsmodule.c.
>> It looks like the changes to the C code should be straightforward.
>>
>>
>> Warren
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140824/df3e336c/attachment-0001.html>

From warren.weckesser at gmail.com  Sun Aug 24 16:47:02 2014
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sun, 24 Aug 2014 10:47:02 -0400
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CADiSq7ffiRmJK5YFWZHeBrj3Z3ZsGkGtXOFJ5hePmc4tv6BEwQ@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CADiSq7ffiRmJK5YFWZHeBrj3Z3ZsGkGtXOFJ5hePmc4tv6BEwQ@mail.gmail.com>
Message-ID: <CAGzF1udf6uyjs0nXE6VYDayZRXxv=CFZ8fvO3a958nUysJREXQ@mail.gmail.com>

On Sun, Aug 24, 2014 at 12:24 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 24 August 2014 01:30, Warren Weckesser <warren.weckesser at gmail.com>
> wrote:
> > I'd like to add an additional optional argument to functools.reduce.
> > The argument is the "nullifier" of the reducing operation.  It is a value
> > such that function(nullifier, anything) returns nullifier.  For example,
> if
> > function(x, y) computes x*y, the nullifier is 0.  If function(x, y) is
> > the intersection of the sets x and y, the nullifier is the empty set.
>
> When it comes to judging the usefulness of functional programming
> features these days, my first question is generally going to be "Does
> PyToolz offer this?"
>


I took a look, and I couldn't find it there.



>
> Grumblings about the name aside, it's still the solution I recommend
> to folks that wish Python had more functional programming tools in the
> standard library: http://toolz.readthedocs.org/en/latest/api.html
>
> There's even a Cython accelerated version available (Cytoolz).
>
> "pip install toolz" for the pure Python version, "pip install cytoolz"
> for the accelerated one.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140824/ceb1c4fd/attachment.html>

From warren.weckesser at gmail.com  Sun Aug 24 16:56:26 2014
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sun, 24 Aug 2014 10:56:26 -0400
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <20140824130908.GH25957@ando>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <20140823174358.GA25957@ando>
 <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>
 <ltcm7e$fn$1@ger.gmane.org> <20140824130908.GH25957@ando>
Message-ID: <CAGzF1udqr_RvmC1rrTg8THqL3eLAmVHE52e8jNvaJc+WrReR=g@mail.gmail.com>

On Sun, Aug 24, 2014 at 9:09 AM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Sun, Aug 24, 2014 at 02:35:57PM +0200, Peter Otten wrote:
> > David Mertz wrote:
> >
> > >   def reduce_with_attractor(func, it, start=None, end_if=None):
> > >       it = iter(it)
> > >       start = start if start!=None else it.__next__()
> > >       return list(takewhile(lambda x: x!=end_if,
> > >                             accumulate(chain([start],it), func)))[-1]
> >
> > Wouldn't it be better to break this into a function that limits a
> sequence
> > and to combine that with the original reduce()?
>
> In general, if x is an attractor, then we want to stop and return x if
> either of these two scenarios occur:
>
> - we come across x in the input;
>
> - or the sequence of intermediate values reaches x.
>
> You can do the first by filtering the input stream, but not the second.
>
> Here's a concrete example, sticking to product() where 0 is an
> attractor:
>
> # best viewed with a fixed-width font
> product([1, 2, 3, 9, 0, 8, 7, 4])
> .....................^ stop here
>
>
> But multiplication can underflow to zero too, and once it does,
> we likewise want to stop:
>
> product([1e-70, 2e-71, 6e-69, 4e-68, 1e-48, 2e-71, 1e-69, 3e-70])
> .....................................^ stop here
>
>
> Notice that 1e-48 is not only non-zero, but it's considerably bigger
> than the other numbers in the sequence. (About 100000000000000000000
> times bigger, give or take a factor of 10.) Yet it's enough to cause the
> product to underflow to zero, after which the product will never
> shift away from zero. (Ignoring NANs and INFs.)
>
>

An example I gave earlier  is the reduction of a collection of sets using
set intersection.  It is possible that (set1 & set2) gives the empty set,
so the nullifier can occur during the reduction without being in the given
iterable.  E.g.:

    reduce(lambda x, y: x & y, [{1,2}, {3}, {4, 5}, {6}], nullifier={})

That should stop iterating after the first operation.




> I'm still not convinced this belongs in the standard library, but it's a
> nice functional, er, function to add to your private library or as a
> third-party module.
>
>
>
> --
> Steven
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140824/b45f2b72/attachment.html>

From python at mrabarnett.plus.com  Sun Aug 24 17:27:35 2014
From: python at mrabarnett.plus.com (MRAB)
Date: Sun, 24 Aug 2014 16:27:35 +0100
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAGzF1udqr_RvmC1rrTg8THqL3eLAmVHE52e8jNvaJc+WrReR=g@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <20140823174358.GA25957@ando>
 <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>
 <ltcm7e$fn$1@ger.gmane.org> <20140824130908.GH25957@ando>
 <CAGzF1udqr_RvmC1rrTg8THqL3eLAmVHE52e8jNvaJc+WrReR=g@mail.gmail.com>
Message-ID: <53FA0467.2050305@mrabarnett.plus.com>

On 2014-08-24 15:56, Warren Weckesser wrote:
[snip]
>
> An example I gave earlier  is the reduction of a collection of sets
> using set intersection.  It is possible that (set1 & set2) gives the
> empty set, so the nullifier can occur during the reduction without being
> in the given iterable.  E.g.:
>
>      reduce(lambda x, y: x & y, [{1,2}, {3}, {4, 5}, {6}], nullifier={})
>
> That should stop iterating after the first operation.
>
I think it would be more flexible if, instead of a 'nullifier' object,
there were an 'until' or 'while_' predicate:

reduce(lambda x, y: x & y, [{1,2}, {3}, {4, 5}, {6}], while_=bool)


From mertz at gnosis.cx  Sun Aug 24 19:27:13 2014
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 24 Aug 2014 10:27:13 -0700
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAEbHw4Yk4fx=T0+wX2xULYutfNE3AVzSgcD_C-R-zkzS7wQF8A@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <20140823174358.GA25957@ando>
 <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>
 <20140824015325.GB25957@ando>
 <CAEbHw4Yk4fx=T0+wX2xULYutfNE3AVzSgcD_C-R-zkzS7wQF8A@mail.gmail.com>
Message-ID: <CAEbHw4aZrku0-ci_YipkOAetY9x_pws65Unc=XPNySQutyiDYg@mail.gmail.com>

So here's a version of my function that addresses Steven's points, and also
adds what I think is more useful behavior.  A couple points about my
short--but more than 3-line function.

* The notion of "attractor" here still follows the "we're all adults"
philosophy. That is, the 'stop_on' and 'end_if' arguments may very well not
actually specify genuine attractors, but simply "satisfaction" values.  In
the examples below these are completely unlike attractors, but this could
be useful also for something like numeric approximation algorithms where
some non-final reduction value is considered "good enough" anyway.

* To be more like the reduce() builtin, I raise TypeError on an empty
iterable with no 'start' value.

* Although it's not as cute in using itertools as much, but simply a
loop-with-test, I got rid of the "golf" artifact of potentially allocating
a large, useless list.

* Most interesting, I think, I separated a 'stop_on' collection of values
from a 'end_if' predicate.  This lets us generalize attractors, I believe.
 For a simple cyclic attractor, you could list several value in the
'stop_on' collection.  But even for a strange attractor (or simply a
complex one, or e.g. an attractor to a continuous set of values--think an
attractor to a circular, real-valued orbit), the predicate could, in
principle, capture the fact we reached the attractor.

#################################################
% cat reduce_with_attractor.py
#!/usr/bin/env python3
from itertools import *
from operator import add

def reduce_with_attractors(func, it, start=None, stop_on=(), end_if=None):
    it = iter(it)
    try:
        start = start if start!=None else next(it)
    except StopIteration:
        raise TypeError
    for x in accumulate(chain([start],it), func):
        if x in stop_on or (end_if and end_if(x)):
            break
    return x

print(reduce_with_attractors(add, range(100), stop_on=(2, 6, 17)))
print(reduce_with_attractors(add, range(100), stop_on=('foo','bar')))
print(reduce_with_attractors(add, range(100), end_if=lambda x: x>100))
#################################################


On Sat, Aug 23, 2014 at 7:19 PM, David Mertz <mertz at gnosis.cx> wrote:
>
> On Sat, Aug 23, 2014 at 6:53 PM, Steven D'Aprano <steve at pearwood.info>
wrote:
>>
>> On Sat, Aug 23, 2014 at 11:43:42AM -0700, David Mertz wrote:
>>
>> >   def reduce_with_attractor(func, it, start=None, end_if=None):
>> >       it = iter(it)
>> >       start = start if start!=None else it.__next__()
>> >       return list(takewhile(lambda x: x!=end_if,
>> >                             accumulate(chain([start],it), func)))[-1]
>>
>> A couple of points:
>>
>> - Don't use it.__next__(), use the built-in next(it).
>
>
> Yeah, good point.
>
>>
>> - To match the behaviour of reduce, you need to catch the StopIteration
>> and raise TypeError.
>
>
> Oh, OK.  Hadn't thought of that.
>
>>
>> - Your implementation eagerly generates a potentially enormous list of
>> intermediate results, which strikes me as somewhat unfortunate for a
>> functional tool like reduce. In other words, for some inputs, this is
>> going to perform like a dog, generating a HUGE list up front, then
>> throwing it all away except for the final value.
>
>
> I know.  I realized this flaw right away.  I was trying to be cute and
fit it in my promised 3 lines.  It would be better to put it in a loop to
realize the successive values, of course--but would take an extra line or
two.  Maybe there's a way to squeeze it in one line with itertools rather
than a regular loop though.
>
>>
>> > This gives you the accumulation up-to-but-not-including the attractor.
 I
>> > guess the OP wanted to return the attractor itself (although that seems
>> > slightly less useful to me).
>>
>> Not really. His use-case seems to be to short-cut a lot of unnecessary
>> calculations, e.g. suppose you write product() as reduce(operator.mul,
>> iterable). In the event that the product reaches zero, you can[1]
>> short-circuit the rest of the iterable and just return 0:
>>
>> product([1, 2, 3, 0] + [4]*1000000)
>>
>> ought to reduce 0, not 6, and the intent is for it to do so *quickly*,
>> ignoring the 4s at the end of the list.
>
>
> I guess that's true.  Although I can certainly imagine being interested
not only in the final attractor, but *that* it reached an attractor and
ended early.  Not sure how best to signal that.  Actually, now that I think
of it, it would be kinda nice to make the function
'reduce_with_attractorS()' instead, and allow specification of multiple
attractors.
>
> I welcome your improved version of the code :-).  Feel free to take a
whole 10 lines to do it right.
>
>>
>> [1] Actually you can't. 0 is no longer an attractor in the presence of
>> INF or NAN.
>
>
> I was sort of thinking of a "we're all adults here" attitude.  That is,
the "attractor" might not really be a genuine attractor, but we still trust
the caller to say it is.  I.e. my function would accept this call:
>
>     reduce_with_attractor(operator.mul, range(1,1e6), end_if=6))
>
> I'm making the claim that reaching '6' is a stopping point... which, well
it is.  No, it's not an actual attractor, but maybe a caller really does
want to stop iterating if it gets to that value anyway.  Hence 'end_if' is
actually an accurate name.
>
> --
> Keeping medicines from the bloodstreams of the sick; food
> from the bellies of the hungry; books from the hands of the
> uneducated; technology from the underdeveloped; and putting
> advocates of freedom in prisons.  Intellectual property is
> to the 21st century what the slave trade was to the 16th.




--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140824/effb947d/attachment-0001.html>

From jsbfox at gmail.com  Sun Aug 24 19:48:51 2014
From: jsbfox at gmail.com (Thomas Allen)
Date: Sun, 24 Aug 2014 21:48:51 +0400
Subject: [Python-ideas] Negative limit for traceback.*
Message-ID: <CAMPw9HQMQ3msO6fpvmMLwgeeuxuv2oeuSPLQ_uFXK=SCO1gXfQ@mail.gmail.com>

I'd like to propose adding the ability to extract last n stack trace
entries from a traceback using functions from traceback module. Simply
put, passing limit=-n would make functions to return (or print or
yield) last abs(n) entries. No-brainer implementation for extract_tb:

    extracted = list(_extract_tb_iter(tb, limit=(limit, None)[limit < 0]))
    if limit is not None and limit < 0:
        return extracted[limit:]
    return extracted

The motivation: http://stackoverflow.com/q/25472412/2301450

From guido at python.org  Sun Aug 24 19:52:44 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 24 Aug 2014 10:52:44 -0700
Subject: [Python-ideas] Negative limit for traceback.*
In-Reply-To: <CAMPw9HQMQ3msO6fpvmMLwgeeuxuv2oeuSPLQ_uFXK=SCO1gXfQ@mail.gmail.com>
References: <CAMPw9HQMQ3msO6fpvmMLwgeeuxuv2oeuSPLQ_uFXK=SCO1gXfQ@mail.gmail.com>
Message-ID: <CAP7+vJ+itu4nnqBCRodXmOqZD6pXYjS0bDFN8Xk7_CQLzG5eyw@mail.gmail.com>

I like this. It's kind of sad that when the limit causes truncation of the
traceback it drops the most recent frames, which might give a hint as to
what happens, and keeps the oldest frames, which are usually less
interesting (I know my program started at main() :-).


On Sun, Aug 24, 2014 at 10:48 AM, Thomas Allen <jsbfox at gmail.com> wrote:

> I'd like to propose adding the ability to extract last n stack trace
> entries from a traceback using functions from traceback module. Simply
> put, passing limit=-n would make functions to return (or print or
> yield) last abs(n) entries. No-brainer implementation for extract_tb:
>
>     extracted = list(_extract_tb_iter(tb, limit=(limit, None)[limit < 0]))
>     if limit is not None and limit < 0:
>         return extracted[limit:]
>     return extracted
>
> The motivation: http://stackoverflow.com/q/25472412/2301450
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140824/52c43039/attachment.html>

From abarnert at yahoo.com  Sun Aug 24 21:00:01 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 24 Aug 2014 12:00:01 -0700
Subject: [Python-ideas] Optional Static Typing -- the Python Way
In-Reply-To: <20140824115954.GF25957@ando>
References: <53F25EE8.8000900@stoneleaf.us>
 <CADiSq7dbQVQTz9nMjR=ytj6k0R9_NNKPt2xGVDgUe8OSNsNcxA@mail.gmail.com>
 <lt0kc6$s6n$1@ger.gmane.org> <lt0l2r$5ed$1@ger.gmane.org>
 <CAP7+vJ+3+zMwWeCvUQQT3LJytexdjNp=-X4Xw1ByV5=WQZZaNg@mail.gmail.com>
 <lt35je$tp3$1@ger.gmane.org>
 <CAP1=2W70P_Mdeup3bj6om+0N_HrR2cv-QMXHNKw5OiU79Je4Ww@mail.gmail.com>
 <20140824115954.GF25957@ando>
Message-ID: <9214C846-2317-4DD1-82BF-195B91154038@yahoo.com>

On Aug 24, 2014, at 4:59, Steven D'Aprano <steve at pearwood.info> wrote:

> On Thu, Aug 21, 2014 at 02:30:03PM +0000, Brett Cannon wrote:
> 
>> P.S.: Off-topic for this email but something I don't think has been
>> mentioned more than once, type hinting like we are proposing is exactly
>> what Dart does and it is nice. Built-in API documentation of interfaces
>> along with IDE support at the API makes all of this worth it.
> 
> For anyone not familiar with Dart, I think it is worth reading the Dart 
> justification for static type hinting in a dynamic language:
> 
> https://www.dartlang.org/articles/optional-types/
> 
> https://www.dartlang.org/articles/why-dart-types/
> 
> It's not quite the same as Guido's proposal, for example in Dart the 
> type hints have no runtime effect at all, but I think it demonstrates 
> that this is a proven approach and can work well with a dynamic language 
> like Python without compromising the dynamic nature of the language.

Great reference.

It's also worth looking at Boo (http://boo.codehaus.org), which was intentionally designed to read like Python, while also having static typing. So, it was designed in the opposite direction: given the CLR compile-time type system, modify Python syntax and semantics as little as possible to implement it. And it was also designed for completely different goals (mainly performance). But it does show how a type system pretty similar to the one Jukka and Guido are envisioning can fit into a language pretty similar to Python, which makes Boo code a useful example for "how readable could this be in real-life code?" questions.

From abarnert at yahoo.com  Sun Aug 24 22:47:46 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 24 Aug 2014 13:47:46 -0700
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAEbHw4aZrku0-ci_YipkOAetY9x_pws65Unc=XPNySQutyiDYg@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <20140823174358.GA25957@ando>
 <CAEbHw4adU5vKhD-65AuZ5VcDbF8PQdcD-GZnhUj4eT=GVmi8ZA@mail.gmail.com>
 <20140824015325.GB25957@ando>
 <CAEbHw4Yk4fx=T0+wX2xULYutfNE3AVzSgcD_C-R-zkzS7wQF8A@mail.gmail.com>
 <CAEbHw4aZrku0-ci_YipkOAetY9x_pws65Unc=XPNySQutyiDYg@mail.gmail.com>
Message-ID: <1408913266.15289.YahooMailNeo@web181002.mail.ne1.yahoo.com>

I think it would be both simpler and more useful to define accumulate_with_attractors, and then define reduce_with_attractors as a wrapper around that.

Assuming you're doing a lot of this kind of stuff, you're either going to be using a library like more_itertools or pytoolz or your own custom library, so I'll assume you already have a "takewhileplus" that's like takewhile but also yields the first failing value, and an "ilast" that returns the last value in an iterable and raises a TypeError if empty (maybe with an optional default argument, but I'm not going to use it).

_sentinel = object()

def accumulate_with_attractors(iterable, func=operator.add,
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?stop_on=(), end_if=None):
? ? yield from takewhileplus(lambda x: x not in stop_on and not (end_if and end_if(x)),
? ? ? ? ? ? ? ? ? ? ? ? ? ? ?itertools.accumulate(iterable, func))

def reduce_with_attractors(func, iterable, start=_sentinel,
? ? ? ? ? ? ? ? ? ? ? ? ? ?stop_on=(), end_if=None):
? ? if start is not _sentinel:
? ? ? ? iterable = itertools.chain([start], iterable)
? ? return ilast(accumulate_with_attractors(iterable, func, stop_on, end_if))

You can easily optimize this in a few ways. You can use a takeuntilplus so the lambda doesn't have to negate its condition, you can use stop_on.__contains__ if end_if isn't passed, etc.?With those optimizations, using C implementations of takeuntilplus and ilast, this takes about 40% as long as your version.

With no optimizations, using Python takewhileplus and last, it takes about 140% as long. But I think the added simplicity, the fewer edge cases to get wrong, and the fact that you get a usable accumulate_with_attractors out of it makes it a worthwhile tradeoff even at 40% slower.


On Sunday, August 24, 2014 10:27 AM, David Mertz <mertz at gnosis.cx> wrote:


>
>
>So here's a version of my function that addresses Steven's points, and also adds what I think is more useful behavior. ?A couple points about my short--but more than 3-line function.
>
>* The notion of "attractor" here still follows the "we're all adults" philosophy. That is, the 'stop_on' and 'end_if' arguments may very well not actually specify genuine attractors, but simply "satisfaction" values. ?In the examples below these are completely unlike attractors, but this could be useful also for something like numeric approximation algorithms where some non-final reduction value is considered "good enough" anyway.
>
>* To be more like the reduce() builtin, I raise TypeError on an empty iterable with no 'start' value.
>
>* Although it's not as cute in using itertools as much, but simply a loop-with-test, I got rid of the "golf" artifact of potentially allocating a large, useless list.
>
>* Most interesting, I think, I separated a 'stop_on' collection of values from a 'end_if' predicate. ?This lets us generalize attractors, I believe. ?For a simple cyclic attractor, you could list several value in the 'stop_on' collection. ?But even for a strange attractor (or simply a complex one, or e.g. an attractor to a continuous set of values--think an attractor to a circular, real-valued orbit), the predicate could, in principle, capture the fact we reached the attractor.
>
>#################################################
>% cat reduce_with_attractor.py
>#!/usr/bin/env python3
>from itertools import *
>from operator import add
>
>def reduce_with_attractors(func, it, start=None, stop_on=(), end_if=None):
>? ? it = iter(it)
>? ? try:
>? ? ? ? start = start if start!=None else next(it)
>? ? except StopIteration:
>? ? ? ? raise TypeError
>? ? for x in accumulate(chain([start],it), func):
>? ? ? ? if x in stop_on or (end_if and end_if(x)):
>? ? ? ? ? ? break
>? ? return x
>
>print(reduce_with_attractors(add, range(100), stop_on=(2, 6, 17)))
>print(reduce_with_attractors(add, range(100), stop_on=('foo','bar')))
>print(reduce_with_attractors(add, range(100), end_if=lambda x: x>100))
>#################################################
>
>
>On Sat, Aug 23, 2014 at 7:19 PM, David Mertz <mertz at gnosis.cx> wrote:
>>
>> On Sat, Aug 23, 2014 at 6:53 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>>>
>>> On Sat, Aug 23, 2014 at 11:43:42AM -0700, David Mertz wrote:
>>>
>>> > ? def reduce_with_attractor(func, it, start=None, end_if=None):
>>> > ? ? ? it = iter(it)
>>> > ? ? ? start = start if start!=None else it.__next__()
>>> > ? ? ? return list(takewhile(lambda x: x!=end_if,
>>> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? accumulate(chain([start],it), func)))[-1]
>>>
>>> A couple of points:
>>>
>>> - Don't use it.__next__(), use the built-in next(it).
>>
>>
>> Yeah, good point.
>> ?
>>>
>>> - To match the behaviour of reduce, you need to catch the StopIteration
>>> and raise TypeError.
>>
>>
>> Oh, OK. ?Hadn't thought of that.
>> ?
>>>
>>> - Your implementation eagerly generates a potentially enormous list of
>>> intermediate results, which strikes me as somewhat unfortunate for a
>>> functional tool like reduce. In other words, for some inputs, this is
>>> going to perform like a dog, generating a HUGE list up front, then
>>> throwing it all away except for the final value.
>>
>>
>> I know. ?I realized this flaw right away. ?I was trying to be cute and fit it in my promised 3 lines. ?It would be better to put it in a loop to realize the successive values, of course--but would take an extra line or two. ?Maybe there's a way to squeeze it in one line with itertools rather than a regular loop though.
>> ?
>>>
>>> > This gives you the accumulation up-to-but-not-including the attractor. ?I
>>> > guess the OP wanted to return the attractor itself (although that seems
>>> > slightly less useful to me).
>>>
>>> Not really. His use-case seems to be to short-cut a lot of unnecessary
>>> calculations, e.g. suppose you write product() as reduce(operator.mul,
>>> iterable). In the event that the product reaches zero, you can[1]
>>> short-circuit the rest of the iterable and just return 0:
>>>
>>> product([1, 2, 3, 0] + [4]*1000000)
>>>
>>> ought to reduce 0, not 6, and the intent is for it to do so *quickly*,
>>> ignoring the 4s at the end of the list.
>>
>>
>> I guess that's true. ?Although I can certainly imagine being interested not only in the final attractor, but *that* it reached an attractor and ended early. ?Not sure how best to signal that. ?Actually, now that I think of it, it would be kinda nice to make the function 'reduce_with_attractorS()' instead, and allow specification of multiple attractors.
>>
>> I welcome your improved version of the code :-). ?Feel free to take a whole 10 lines to do it right.
>> ?
>>>
>>> [1] Actually you can't. 0 is no longer an attractor in the presence of
>>> INF or NAN.
>>
>>
>> I was sort of thinking of a "we're all adults here" attitude. ?That is, the "attractor" might not really be a genuine attractor, but we still trust the caller to say it is. ?I.e. my function would accept this call:
>>
>> ? ? reduce_with_attractor(operator.mul, range(1,1e6), end_if=6))
>>
>> I'm making the claim that reaching '6' is a stopping point... which, well it is. ?No, it's not an actual attractor, but maybe a caller really does want to stop iterating if it gets to that value anyway. ?Hence 'end_if' is actually an accurate name.
>>
>> --
>> Keeping medicines from the bloodstreams of the sick; food
>> from the bellies of the hungry; books from the hands of the
>> uneducated; technology from the underdeveloped; and putting
>> advocates of freedom in prisons. ?Intellectual property is
>> to the 21st century what the slave trade was to the 16th.
>
>
>
>
>
>--
>Keeping medicines from the bloodstreams of the sick; food
>from the bellies of the hungry; books from the hands of the
>uneducated; technology from the underdeveloped; and putting
>advocates of freedom in prisons. ?Intellectual property is
>to the 21st century what the slave trade was to the 16th.
>
>_______________________________________________
>Python-ideas mailing list
>Python-ideas at python.org
>https://mail.python.org/mailman/listinfo/python-ideas
>Code of Conduct: http://python.org/psf/codeofconduct/
>
>

From ram.rachum at gmail.com  Sun Aug 24 23:41:01 2014
From: ram.rachum at gmail.com (Ram Rachum)
Date: Sun, 24 Aug 2014 14:41:01 -0700 (PDT)
Subject: [Python-ideas] Add an introspection API to Executor
Message-ID: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>

Sometimes I want to take a live executor, like a `ThreadPoolExecutor`, and 
check up on it. I want to know how many threads there are, how many are 
handling tasks and which tasks, how many are free, and which tasks are in 
the queue.

I asked on Stack 
Overflow: http://stackoverflow.com/questions/25474204/checking-up-on-a-concurrent-futures-threadpoolexecutor

There's an answer there, but it uses private variables and it's not part of 
the API.

I suggest it become a part of the API. There should be an API for checking 
on what the executor is currently doing and answering all the questions I 
raised above.


Thanks,
Ram.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140824/a13a72c8/attachment.html>

From greg.ewing at canterbury.ac.nz  Mon Aug 25 00:17:30 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 25 Aug 2014 10:17:30 +1200
Subject: [Python-ideas] Was: Annotations (and static typing),
 Now:Sharing __annotations__
In-Reply-To: <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
 <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
Message-ID: <53FA647A.2030108@canterbury.ac.nz>

> On 24 August 2014 03:18, Cem Karan <cfkaran2 at gmail.com> wrote:
> 
>> Each project (and 'project' is a very
>> loosely defined concept here) chooses a UUID that it uses as key into the
>> dictionary.

Please, not UUIDs. As something meant to be used by
humans, they're horrible.

-- 
Greg

From cfkaran2 at gmail.com  Mon Aug 25 01:17:46 2014
From: cfkaran2 at gmail.com (Cem Karan)
Date: Sun, 24 Aug 2014 19:17:46 -0400
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <53FA647A.2030108@canterbury.ac.nz>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
 <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
 <53FA647A.2030108@canterbury.ac.nz>
Message-ID: <C700203F-B680-427B-A60E-F0A984C71EAD@gmail.com>


On Aug 24, 2014, at 6:17 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

>> On 24 August 2014 03:18, Cem Karan <cfkaran2 at gmail.com> wrote:
>>> Each project (and 'project' is a very
>>> loosely defined concept here) chooses a UUID that it uses as key into the
>>> dictionary.
> 
> Please, not UUIDs. As something meant to be used by
> humans, they're horrible.
> 
> -- 
> Greg

I understand your concerns about them, I really, really do.  But can you think of a better way?  All my prior points still hold.  In my mind, UUIDs are the least bad alternative.  We have to educate people as to what the annotations dictionary standard is, and then encourage libraries to use the standard and to hide the UUIDs from end users.  My code does this already; if someone wants me to extend it in some way, put in a request, and I will.  If the PSF wants it in the standard library, it's theirs!  Please, download it and play with it, it's short to the point of almost being trivial (https://github.com/oranguman/annotizer).  

Don't let the fact that the bare __annotations__ dict is going to look ugly; if we try to get a really powerful typing system in there, people will probably not want to write the annotations by hand anyways for the simple reason that it will be too hard to read.  I mean, which would you rather read?

"""
@doc(a, "docs about a")
@type(a, set(range(10)))
def foo(a):
	pass

def bar(a: {__docs_project_ID: "docs about a", __types_project_ID: set(range(10))}):
	pass
"""

Even if __project_ID was something human readable, that line would be PAINFUL for a human to parse.  UUIDs won't make it much worse, and they do avoid a lot of the other headaches.

Thanks,
Cem Karan

From rosuav at gmail.com  Mon Aug 25 01:22:51 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 25 Aug 2014 09:22:51 +1000
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <C700203F-B680-427B-A60E-F0A984C71EAD@gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
 <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
 <53FA647A.2030108@canterbury.ac.nz>
 <C700203F-B680-427B-A60E-F0A984C71EAD@gmail.com>
Message-ID: <CAPTjJmqznNZo2ikGwaUEW6iiYMVkB0JjbTX77MRMDrovWjGeCg@mail.gmail.com>

On Mon, Aug 25, 2014 at 9:17 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
> @doc(a, "docs about a")
> @type(a, set(range(10)))
> def foo(a):
>         pass

If this is your proposal, then how do UUIDs even help? You already
effectively piggy-back off the standard module import system for
uniqueness. Surely that's going to do everything that UUIDs would?

ChrisA

From cfkaran2 at gmail.com  Mon Aug 25 01:28:36 2014
From: cfkaran2 at gmail.com (Cem Karan)
Date: Sun, 24 Aug 2014 19:28:36 -0400
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <CAPTjJmqznNZo2ikGwaUEW6iiYMVkB0JjbTX77MRMDrovWjGeCg@mail.gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
 <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
 <53FA647A.2030108@canterbury.ac.nz>
 <C700203F-B680-427B-A60E-F0A984C71EAD@gmail.com>
 <CAPTjJmqznNZo2ikGwaUEW6iiYMVkB0JjbTX77MRMDrovWjGeCg@mail.gmail.com>
Message-ID: <A8D6AAC6-CACD-4D08-AC51-3250BEA41AAA@gmail.com>


On Aug 24, 2014, at 7:22 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Mon, Aug 25, 2014 at 9:17 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
>> @doc(a, "docs about a")
>> @type(a, set(range(10)))
>> def foo(a):
>>        pass
> 
> If this is your proposal, then how do UUIDs even help? You already
> effectively piggy-back off the standard module import system for
> uniqueness. Surely that's going to do everything that UUIDs would?

I'm not sure if I follow what you're saying.  Do you mean 'if everyone is using my annotizer library, why bother with UUIDs, because each decorator will have a different meaning enforced by the import system'?

Thanks,
Cem Karan

From rosuav at gmail.com  Mon Aug 25 01:57:37 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 25 Aug 2014 09:57:37 +1000
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <A8D6AAC6-CACD-4D08-AC51-3250BEA41AAA@gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
 <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
 <53FA647A.2030108@canterbury.ac.nz>
 <C700203F-B680-427B-A60E-F0A984C71EAD@gmail.com>
 <CAPTjJmqznNZo2ikGwaUEW6iiYMVkB0JjbTX77MRMDrovWjGeCg@mail.gmail.com>
 <A8D6AAC6-CACD-4D08-AC51-3250BEA41AAA@gmail.com>
Message-ID: <CAPTjJmpzGFyhAumigzjfTTwCFkDQ+up_bjo-zk5bNDyG4GfP-g@mail.gmail.com>

On Mon, Aug 25, 2014 at 9:28 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
> On Aug 24, 2014, at 7:22 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
>> On Mon, Aug 25, 2014 at 9:17 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
>>> @doc(a, "docs about a")
>>> @type(a, set(range(10)))
>>> def foo(a):
>>>        pass
>>
>> If this is your proposal, then how do UUIDs even help? You already
>> effectively piggy-back off the standard module import system for
>> uniqueness. Surely that's going to do everything that UUIDs would?
>
> I'm not sure if I follow what you're saying.  Do you mean 'if everyone is using my annotizer library, why bother with UUIDs, because each decorator will have a different meaning enforced by the import system'?

doc and type can't conflict, because they're separate names. (Though
"type" would need to be renamed to avoid a conflict, unless you're
suggesting that the type type should grow this functionality.) If they
use their own selves as the keys (nobody ever said the keys have to be
strings), there cannot be a conflict. The UUID scheme doesn't add
anything that can't be done with existing objects with no hassle.

ChrisA

From rosuav at gmail.com  Mon Aug 25 02:05:27 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 25 Aug 2014 10:05:27 +1000
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAGzF1ufC_KHkBGytoyR0nLynb_W+zqiX5+j7f4QL2F6vgNZVcg@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAN1F8qUBvptkn52+PWb5sDwbtu5OaALSqE9=UPnt-mS3WQfM4g@mail.gmail.com>
 <CAGzF1ufC_KHkBGytoyR0nLynb_W+zqiX5+j7f4QL2F6vgNZVcg@mail.gmail.com>
Message-ID: <CAPTjJmqZXUgNWzaJFVFC1LizQ1swiUuCTG6xeqZG30hMb79T8A@mail.gmail.com>

On Sun, Aug 24, 2014 at 2:27 AM, Warren Weckesser
<warren.weckesser at gmail.com> wrote:
> That wouldn't help in an example such as
>
>     reduce(lambda x, y: x & y, [{1,2}, {3, 4}, {5, 6}], nullifier={})
>
> The nullifier is the empty set, but the empty set does not occur in the
> iterable.

Caution: Your nullifier is an empty dict, not an empty set, if you
spell it that way.

I think this is a cute concept, but not something that is going to be
needed all that often. Most Python programs don't even use reduce() in
its default form, much less need a specific optimization.

ChrisA

From ethan at stoneleaf.us  Mon Aug 25 01:59:02 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 24 Aug 2014 16:59:02 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <ltb6f6$65d$1@ger.gmane.org>
References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org>
Message-ID: <53FA7C46.1000407@stoneleaf.us>

On 08/23/2014 04:00 PM, Antoine Pitrou wrote:
>
> Le 23/08/2014 13:25, Steven D'Aprano a ?crit :
>>>
>>> It's not a new use. A type class is a class, and calling it is just
>>> instantiating that class. There's nothing new here. If you think that's
>>> a bit "meta", it's no different than e.g. higher-order functions.
>>
>> There's no instantiation during *static* analysis, because the code
>> hasn't run yet.
>
> In your idea of "static analysis", it hasn't. Because you think it should involve some kind of separate syntax analysis
> tool that has nothing to do with regular Python. But Python is powerful enough to let you do that using normal
> introspection of modules.
>
> And it's *exactly* how we are exposing function annotations (and also docstrings, etc.): using runtime-accessible
> introspection information which is gathered by importing modules and therefore actually *executing* toplevel module
> code. Not merely compiling it.

Is this workable?  If I have a nested class in a function:

   def spam():
      class fribble:
         def __init__(self, x:int):
            blahblah

I would expect MyPy to be able to retrieve and use the annotations from fribble without actually calling spam.  If 
static checking is only available at the top level I think a lot of power/usefulness is gone.

--
~Ethan~

From abarnert at yahoo.com  Mon Aug 25 02:21:30 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sun, 24 Aug 2014 17:21:30 -0700
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <C700203F-B680-427B-A60E-F0A984C71EAD@gmail.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
 <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
 <53FA647A.2030108@canterbury.ac.nz>
 <C700203F-B680-427B-A60E-F0A984C71EAD@gmail.com>
Message-ID: <1472A8DB-2427-4ABA-857A-41DBB5707900@yahoo.com>

On Aug 24, 2014, at 16:17, Cem Karan <cfkaran2 at gmail.com> wrote:

> 
> On Aug 24, 2014, at 6:17 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> 
>>> On 24 August 2014 03:18, Cem Karan <cfkaran2 at gmail.com> wrote:
>>>> Each project (and 'project' is a very
>>>> loosely defined concept here) chooses a UUID that it uses as key into the
>>>> dictionary.
>> 
>> Please, not UUIDs. As something meant to be used by
>> humans, they're horrible.
>> 
>> -- 
>> Greg
> 
> I understand your concerns about them, I really, really do.  But can you think of a better way?  All my prior points still hold.  In my mind, UUIDs are the least bad alternative.  We have to educate people as to what the annotations dictionary standard is, and then encourage libraries to use the standard and to hide the UUIDs from end users.  My code does this already; if someone wants me to extend it in some way, put in a request, and I will.  If the PSF wants it in the standard library, it's theirs!  Please, download it and play with it, it's short to the point of almost being trivial (https://github.com/oranguman/annotizer).  
> 
> Don't let the fact that the bare __annotations__ dict is going to look ugly; if we try to get a really powerful typing system in there, people will probably not want to write the annotations by hand anyways for the simple reason that it will be too hard to read.  I mean, which would you rather read?
> 
> """
> @doc(a, "docs about a")
> @type(a, set(range(10)))
> def foo(a):
>    pass
> 
> def bar(a: {__docs_project_ID: "docs about a", __types_project_ID: set(range(10))}):
>    pass
> """

Or:

@doc(a, "docs about a")
def foo(a: set[10]):
    pass

Why does doc have to set an annotation in the first place? Annotations are just syntactic sugar for sticking stuff in a dict attached to the function as an attribute; when that syntactic sugar doesn't make your code more readable, there's no benefit at all to annotations. You could just make the @doc decorator store and find its stuff in a special __docinfo__ attribute instead, and it will look exactly the same to your users.

The only question is: which data should be privileges to use the syntactic sugar? That's up to each project to decide; all Guido is suggesting is that we add a default presumption that it's the MyPy-style static types that get to use annotations.

As a side note, I'm not sure how your decorator syntax is going to work unless you've predefined a global named "a" with some kind of value that can be used to match parameters named "a". While there might be some way to do that dynamically (create a module subclass with a custom __dict__ that implements __missing__, and an import hook that applies that to your modules?), it seems pretty hacky, and doesn't buy you that much.


From antoine at python.org  Mon Aug 25 02:24:27 2014
From: antoine at python.org (Antoine Pitrou)
Date: Sun, 24 Aug 2014 20:24:27 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53FA7C46.1000407@stoneleaf.us>
References: <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <53FA7C46.1000407@stoneleaf.us>
Message-ID: <ltdvns$pnm$1@ger.gmane.org>

Le 24/08/2014 19:59, Ethan Furman a ?crit :
>
> Is this workable?  If I have a nested class in a function:
>
>    def spam():
>       class fribble:
>          def __init__(self, x:int):
>             blahblah
>
> I would expect MyPy to be able to retrieve and use the annotations from
> fribble without actually calling spam.

I don't know. But that's not really the point, because you are still 
able to do, say:

my_type = some_invocation(...)

def spam():
     class fribble:
         def __init__(self, x: my_type):
             blahblah

... as far as the type checker uses module imports, that is. Of course, 
/after/ importing the module it can also walk the bytecode defined in 
that module.

Regards

Antoine.



From steve at pearwood.info  Mon Aug 25 02:51:31 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 25 Aug 2014 10:51:31 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <53FA7C46.1000407@stoneleaf.us>
References: <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <53FA7C46.1000407@stoneleaf.us>
Message-ID: <20140825005131.GJ25957@ando>

On Sun, Aug 24, 2014 at 04:59:02PM -0700, Ethan Furman wrote:

> Is this workable?  If I have a nested class in a function:
> 
>   def spam():
>      class fribble:
>         def __init__(self, x:int):
>            blahblah
> 
> I would expect MyPy to be able to retrieve and use the annotations from 
> fribble without actually calling spam.  If static checking is only 
> available at the top level I think a lot of power/usefulness is gone.

That's entirely up to Mypy or whatever linter, compiler, editor or other 
tool being used, and has nothing to do with this proposal, which is only 
to standardize on the syntax. I would expect different static tools 
would have different levels of ability to deal with dynamically created 
classes like that.



-- 
Steven

From warren.weckesser at gmail.com  Mon Aug 25 03:17:22 2014
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Sun, 24 Aug 2014 21:17:22 -0400
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAPTjJmqZXUgNWzaJFVFC1LizQ1swiUuCTG6xeqZG30hMb79T8A@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAN1F8qUBvptkn52+PWb5sDwbtu5OaALSqE9=UPnt-mS3WQfM4g@mail.gmail.com>
 <CAGzF1ufC_KHkBGytoyR0nLynb_W+zqiX5+j7f4QL2F6vgNZVcg@mail.gmail.com>
 <CAPTjJmqZXUgNWzaJFVFC1LizQ1swiUuCTG6xeqZG30hMb79T8A@mail.gmail.com>
Message-ID: <CAGzF1uevmR0qAqNgt_dQTg0YvV_3+o1pnh=GAtu460QQ=rpJmw@mail.gmail.com>

On Sun, Aug 24, 2014 at 8:05 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Sun, Aug 24, 2014 at 2:27 AM, Warren Weckesser
> <warren.weckesser at gmail.com> wrote:
> > That wouldn't help in an example such as
> >
> >     reduce(lambda x, y: x & y, [{1,2}, {3, 4}, {5, 6}], nullifier={})
> >
> > The nullifier is the empty set, but the empty set does not occur in the
> > iterable.
>
> Caution: Your nullifier is an empty dict, not an empty set, if you
> spell it that way.
>


Ah, right--thanks!  It should be `nullifier=set()`.




>
> I think this is a cute concept, but not something that is going to be
> needed all that often. Most Python programs don't even use reduce() in
> its default form, much less need a specific optimization.
>
> ChrisA
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140824/9e41efe7/attachment-0001.html>

From steve at pearwood.info  Mon Aug 25 03:30:23 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 25 Aug 2014 11:30:23 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <ltb6f6$65d$1@ger.gmane.org>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org>
Message-ID: <20140825013023.GK25957@ando>

On Sat, Aug 23, 2014 at 07:00:54PM -0400, Antoine Pitrou wrote:
> 
> Le 23/08/2014 13:25, Steven D'Aprano a ?crit :
> >>
> >>It's not a new use. A type class is a class, and calling it is just
> >>instantiating that class. There's nothing new here. If you think that's
> >>a bit "meta", it's no different than e.g. higher-order functions.
> >
> >There's no instantiation during *static* analysis, because the code
> >hasn't run yet.
> 
> In your idea of "static analysis", it hasn't. Because you think it 
> should involve some kind of separate syntax analysis tool that has 
> nothing to do with regular Python. 

That's not "my" idea of static analysis, that is the standard definition 
of "static" as happening at compile-time. If it happens at runtime, it's 
not static.


> But Python is powerful enough to let 
> you do that using normal introspection of modules.

This proposal isn't just about the Python interpreter, its also about 
static tools like linters, IDEs and editors.


> And it's *exactly* how we are exposing function annotations (and also 
> docstrings, etc.): using runtime-accessible introspection information 
> which is gathered by importing modules and therefore actually 
> *executing* toplevel module code. Not merely compiling it.

Correct, the annotations will be available at runtime as well as 
compile-time. But tools like linters and editors will rely on static 
analysis, not dynamic run-time checks, and that's Guido's intention. 
Here's his initial post:

https://mail.python.org/pipermail/python-ideas/2014-August/028618.html



-- 
Steven

From cfkaran2 at gmail.com  Mon Aug 25 04:04:18 2014
From: cfkaran2 at gmail.com (Cem Karan)
Date: Sun, 24 Aug 2014 22:04:18 -0400
Subject: [Python-ideas]  Was: Annotations (and static typing),
	Now:Sharing __annotations__
References: <1C09BD80-90E6-4B61-9CA8-850D9F99EE41@gmail.com>
Message-ID: <91DD0A3D-24F1-4169-944B-24A06BA74D57@gmail.com>

On Aug 24, 2014, at 9:22 PM, Chris Angelico <rosuav at gmail.com> wrote:

> On Mon, Aug 25, 2014 at 11:15 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
>> Because of this, using 'annotizer' as a key is meaningless; there could be many instances live at the same time.
> 
> But that's exactly my point! You don't use "annotizer" as the key, you
> use annotizer. You use the object itself, not the string.

And now I feel dumb... You're right of course, by passing in the object itself, there are no namespace conflicts, inter-project coordination is nil, and on top of it all, you can perform introspection of the annotizer object itself to find out more information about it.  UUIDs aren't required at all.  Right-o, with your permission Chris, I'd like to amend my proposal so that instead of using UUIDs, the annotizer object itself is used as the key.  Depending on what work is like this week, I'll modify my code to do this.

Thanks,
Cem Karan


From cfkaran2 at gmail.com  Mon Aug 25 04:11:12 2014
From: cfkaran2 at gmail.com (Cem Karan)
Date: Sun, 24 Aug 2014 22:11:12 -0400
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <1472A8DB-2427-4ABA-857A-41DBB5707900@yahoo.com>
References: <CACac1F_0W6DSZb9G4ihL895syAfW3DdNSpxFY5XPfzJQno47yA@mail.gmail.com>
 <686837EA-F825-4923-BCFB-17A2F53CA754@gmail.com>
 <CABmzr0gP3b6gLbYLkUGbNNJMZpF0jN4Rusz5DREHGN-t5zyBrw@mail.gmail.com>
 <53FA647A.2030108@canterbury.ac.nz>
 <C700203F-B680-427B-A60E-F0A984C71EAD@gmail.com>
 <1472A8DB-2427-4ABA-857A-41DBB5707900@yahoo.com>
Message-ID: <05B80116-58B6-4E33-9EDE-F3D5BC7607FB@gmail.com>


On Aug 24, 2014, at 8:21 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Aug 24, 2014, at 16:17, Cem Karan <cfkaran2 at gmail.com> wrote:
> 
>> 
>> On Aug 24, 2014, at 6:17 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>> 
>>>> On 24 August 2014 03:18, Cem Karan <cfkaran2 at gmail.com> wrote:
>>>>> Each project (and 'project' is a very
>>>>> loosely defined concept here) chooses a UUID that it uses as key into the
>>>>> dictionary.
>>> 
>>> Please, not UUIDs. As something meant to be used by
>>> humans, they're horrible.
>>> 
>>> -- 
>>> Greg
>> 
>> I understand your concerns about them, I really, really do.  But can you think of a better way?  All my prior points still hold.  In my mind, UUIDs are the least bad alternative.  We have to educate people as to what the annotations dictionary standard is, and then encourage libraries to use the standard and to hide the UUIDs from end users.  My code does this already; if someone wants me to extend it in some way, put in a request, and I will.  If the PSF wants it in the standard library, it's theirs!  Please, download it and play with it, it's short to the point of almost being trivial (https://github.com/oranguman/annotizer).  
>> 
>> Don't let the fact that the bare __annotations__ dict is going to look ugly; if we try to get a really powerful typing system in there, people will probably not want to write the annotations by hand anyways for the simple reason that it will be too hard to read.  I mean, which would you rather read?
>> 
>> """
>> @doc(a, "docs about a")
>> @type(a, set(range(10)))
>> def foo(a):
>>   pass
>> 
>> def bar(a: {__docs_project_ID: "docs about a", __types_project_ID: set(range(10))}):
>>   pass
>> """
> 
> Or:
> 
> @doc(a, "docs about a")
> def foo(a: set[10]):
>    pass
> 
> Why does doc have to set an annotation in the first place? Annotations are just syntactic sugar for sticking stuff in a dict attached to the function as an attribute; when that syntactic sugar doesn't make your code more readable, there's no benefit at all to annotations. You could just make the @doc decorator store and find its stuff in a special __docinfo__ attribute instead, and it will look exactly the same to your users.

True, but I was worried about namespace conflicts.  More than one documentation library might decide to use the __docinfo__ attribute, which could lead to problems.  The __annotations__ dictionary seemed like a good place to store it since it didn't have any predefined meaning (except that now its being proposed that it should have a meaning).

> The only question is: which data should be privileges to use the syntactic sugar? That's up to each project to decide; all Guido is suggesting is that we add a default presumption that it's the MyPy-style static types that get to use annotations.
> 
> As a side note, I'm not sure how your decorator syntax is going to work unless you've predefined a global named "a" with some kind of value that can be used to match parameters named "a". While there might be some way to do that dynamically (create a module subclass with a custom __dict__ that implements __missing__, and an import hook that applies that to your modules?), it seems pretty hacky, and doesn't buy you that much.

My code seems to work correctly (see https://github.com/oranguman/annotizer/blob/master/annotizer/annotizer.py) via some hackery involving inspect.signature(), but please feel free to take a look at it and tell me if I should do something different.

Thanks,
Cem Karan

From antoine at python.org  Mon Aug 25 04:11:07 2014
From: antoine at python.org (Antoine Pitrou)
Date: Sun, 24 Aug 2014 22:11:07 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140825013023.GK25957@ando>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
Message-ID: <lte5vu$n94$1@ger.gmane.org>


Le 24/08/2014 21:30, Steven D'Aprano a ?crit :
>>
>> In your idea of "static analysis", it hasn't. Because you think it
>> should involve some kind of separate syntax analysis tool that has
>> nothing to do with regular Python.
>
> That's not "my" idea of static analysis, that is the standard definition
> of "static" as happening at compile-time. If it happens at runtime, it's
> not static

No, that's the standard definition of "static" in a certain category of 
languages such as C. Python has "static methods" and they don't happen 
at compile-time: "staticmethod" is a regular callable which is invoked 
at runtime.

And there are other precedents, such as RPython which is also 
"statically" typed but... using code loaded from imported modules (in 
other words, I'm not inventing anything new here).

> This proposal isn't just about the Python interpreter, its also about
> static tools like linters, IDEs and editors.

Those so-called "static" tools are free to invoke the interpreter and 
use Python's *runtime* introspection facilities. And I'm sure some of 
them do.

Really, it's amusing that I tried several times to explain you that 
"static" analysis didn't need to happen in an entirely separate 
evaluation context, and you still don't get it. You speak like some C 
programmer who has newly discovered Python and doesn't realize the 
magnitude of the differences between the two languages.

Therefore I think the word "static" should be banned from this entire 
discussion, otherwise folks like you will keep associating irrelevant 
concepts with it.

Regards

Antoine.



From rosuav at gmail.com  Mon Aug 25 04:16:12 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 25 Aug 2014 12:16:12 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lte5vu$n94$1@ger.gmane.org>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
Message-ID: <CAPTjJmp1oKagyOsc1dOqF14OFAAxHt3O14sMczTTv+UmVs48Tg@mail.gmail.com>

On Mon, Aug 25, 2014 at 12:11 PM, Antoine Pitrou <antoine at python.org> wrote:
> No, that's the standard definition of "static" in a certain category of
> languages such as C. Python has "static methods" and they don't happen at
> compile-time: "staticmethod" is a regular callable which is invoked at
> runtime.

"Static method" is a quite different meaning of static - they always
execute at run-time.

https://en.wikipedia.org/wiki/Method_(computer_programming)#Static_methods

ChrisA

From rosuav at gmail.com  Mon Aug 25 04:19:44 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 25 Aug 2014 12:19:44 +1000
Subject: [Python-ideas] Was: Annotations (and static typing),
	Now:Sharing __annotations__
In-Reply-To: <91DD0A3D-24F1-4169-944B-24A06BA74D57@gmail.com>
References: <1C09BD80-90E6-4B61-9CA8-850D9F99EE41@gmail.com>
 <91DD0A3D-24F1-4169-944B-24A06BA74D57@gmail.com>
Message-ID: <CAPTjJmq1HHQzrMuKJ1i5zOdFLNTmHET617-Jg94Lgj81jCKygw@mail.gmail.com>

On Mon, Aug 25, 2014 at 12:04 PM, Cem Karan <cfkaran2 at gmail.com> wrote:
> On Aug 24, 2014, at 9:22 PM, Chris Angelico <rosuav at gmail.com> wrote:
>
>> On Mon, Aug 25, 2014 at 11:15 AM, Cem Karan <cfkaran2 at gmail.com> wrote:
>>> Because of this, using 'annotizer' as a key is meaningless; there could be many instances live at the same time.
>>
>> But that's exactly my point! You don't use "annotizer" as the key, you
>> use annotizer. You use the object itself, not the string.
>
> And now I feel dumb...

Ehh, don't feel dumb, my explanation wasn't all that clear - it's hard
to be clear in email. It's like when Rapunzel says, "I like Eugene
Fitzherbert better than Flynn Rider", and the scriptwriters are
(presumably deliberately) ambiguous as to whether she's putting quotes
around those names ("I prefer to call you Eugene than Flynn") or not
("I prefer the persona of Eugene to that of Flynn").

As long as the objects have useful reprs, you can get most of the
benefit of strings anyway - you can print them out and get an idea of
what they're all about, which UUIDs wouldn't give you. You might have
a collision of representation (if you have multiple annotizer
instances), or you might fall back on incorporating the id() in the
repr, but either way can't be worse than nothing.

ChrisA

From guido at python.org  Mon Aug 25 04:30:35 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 24 Aug 2014 19:30:35 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lte5vu$n94$1@ger.gmane.org>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org>
 <20140825013023.GK25957@ando> <lte5vu$n94$1@ger.gmane.org>
Message-ID: <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>

This argument over the word "static" is getting ridiculous. The meaning of
"static analysis" is perfectly well understood (just Google it) and using
Python run-time introspection for the code being analyzed is exactly what
you *don't* want in this case, for a variety of reasons. (And that includes
doing the analysis as part of the byte-code compilation that may happen at
import/load time. Once any part of the program is executing it's simply too
late for the kind of analysis we're interested in.)

People who are talking about what should happen if a decorator changes
__annotations__ are likewise missing the point. (This doesn't mean that
decorators aren't allowed to affect the signature. But the static analyzer
has to use its static powers to figure out what the decorator does, or it
has to be told, e.g. via a stub file.)

Note that I am okay with being somewhat skittish about the term "static
typing", given Python's long struggle for legitimacy in a world where
professors everywhere used to equate static typing and strong typing and
tell their impressionable students that it was the Right Way.

We should probably avoid "static typing" and use "type hinting" instead
(following TypeScript's lead I believe). But the process of looking for
violations against the hinted types before the code run is still called
static analysis.


On Sun, Aug 24, 2014 at 7:11 PM, Antoine Pitrou <antoine at python.org> wrote:

>
> Le 24/08/2014 21:30, Steven D'Aprano a ?crit :
>
>
>>> In your idea of "static analysis", it hasn't. Because you think it
>>> should involve some kind of separate syntax analysis tool that has
>>> nothing to do with regular Python.
>>>
>>
>> That's not "my" idea of static analysis, that is the standard definition
>> of "static" as happening at compile-time. If it happens at runtime, it's
>> not static
>>
>
> No, that's the standard definition of "static" in a certain category of
> languages such as C. Python has "static methods" and they don't happen at
> compile-time: "staticmethod" is a regular callable which is invoked at
> runtime.
>
> And there are other precedents, such as RPython which is also "statically"
> typed but... using code loaded from imported modules (in other words, I'm
> not inventing anything new here).
>
>
>  This proposal isn't just about the Python interpreter, its also about
>> static tools like linters, IDEs and editors.
>>
>
> Those so-called "static" tools are free to invoke the interpreter and use
> Python's *runtime* introspection facilities. And I'm sure some of them do.
>
> Really, it's amusing that I tried several times to explain you that
> "static" analysis didn't need to happen in an entirely separate evaluation
> context, and you still don't get it. You speak like some C programmer who
> has newly discovered Python and doesn't realize the magnitude of the
> differences between the two languages.
>
> Therefore I think the word "static" should be banned from this entire
> discussion, otherwise folks like you will keep associating irrelevant
> concepts with it.
>
> Regards
>
> Antoine.
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140824/983e4e06/attachment.html>

From antoine at python.org  Mon Aug 25 04:35:49 2014
From: antoine at python.org (Antoine Pitrou)
Date: Sun, 24 Aug 2014 22:35:49 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAPTjJmp1oKagyOsc1dOqF14OFAAxHt3O14sMczTTv+UmVs48Tg@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAPTjJmp1oKagyOsc1dOqF14OFAAxHt3O14sMczTTv+UmVs48Tg@mail.gmail.com>
Message-ID: <lte7e6$6ro$1@ger.gmane.org>

Le 24/08/2014 22:16, Chris Angelico a ?crit :
> On Mon, Aug 25, 2014 at 12:11 PM, Antoine Pitrou <antoine at python.org> wrote:
>> No, that's the standard definition of "static" in a certain category of
>> languages such as C. Python has "static methods" and they don't happen at
>> compile-time: "staticmethod" is a regular callable which is invoked at
>> runtime.
>
> "Static method" is a quite different meaning of static - they always
> execute at run-time.

Oh, really, do they?

You missed the entire point. Python's static methods not only execute at 
runtime, they are *defined* at runtime. The compiler doesn't know that 
something will be a "static method", a regular function, a "class 
method"... or whatever. The meaning of "static" is entirely different 
from C's or Java's, where the information about static methods is (and 
has to be) known at compile time.

Of course, this isn't limited to static methods. The same can be said 
about docstrings, which use separate analysis methods in languages such 
as C++ or Java (e.g. doxygen, javadoc...), but regular runtime 
introspection capabilities in Python.

Regards

Antoine.



From antoine at python.org  Mon Aug 25 04:51:15 2014
From: antoine at python.org (Antoine Pitrou)
Date: Sun, 24 Aug 2014 22:51:15 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
Message-ID: <lte8b4$fbu$1@ger.gmane.org>


Le 24/08/2014 22:30, Guido van Rossum a ?crit :
>
> People who are talking about what should happen if a decorator changes
> __annotations__ are likewise missing the point.

But you shouldn't use annotations for something that refuses to use 
runtime introspection abilities. There's no reason to waste annotations 
if the need can be fullfilled by separate description files in a DSL 
that doesn't even need to be Python code (but cutely looks like so).
And it's not surprising that some people may be missing the point when a 
feature pretends to use a runtime facility but interprets it from a 
separate channel which eschews any code execution.

Your proposal seems determined by the fact that mypy has much grander 
ambitions (and therefore requires its insertion into regular Python), 
but it doesn't make use of that power at all, worse, it forbids others 
to use it.

Regards

Antoine.



From antoine at python.org  Mon Aug 25 05:03:46 2014
From: antoine at python.org (Antoine Pitrou)
Date: Sun, 24 Aug 2014 23:03:46 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lte8b4$fbu$1@ger.gmane.org>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
Message-ID: <lte92i$ld4$1@ger.gmane.org>

Le 24/08/2014 22:51, Antoine Pitrou a ?crit :
>
> Le 24/08/2014 22:30, Guido van Rossum a ?crit :
>>
>> People who are talking about what should happen if a decorator changes
>> __annotations__ are likewise missing the point.
>
> But you shouldn't use annotations for something that refuses to use
> runtime introspection abilities. There's no reason to waste annotations
> if the need can be fullfilled by separate description files in a DSL
> that doesn't even need to be Python code (but cutely looks like so).

(*) or a docstring-embedded DSL, as others acutely remarked.

Regards

Antoine.



From guido at python.org  Mon Aug 25 05:12:07 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 24 Aug 2014 20:12:07 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <lte8b4$fbu$1@ger.gmane.org>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org>
 <20140825013023.GK25957@ando> <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
Message-ID: <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>

On Sun, Aug 24, 2014 at 7:51 PM, Antoine Pitrou <antoine at python.org> wrote:

>
> Le 24/08/2014 22:30, Guido van Rossum a ?crit :
>
>  People who are talking about what should happen if a decorator changes
>> __annotations__ are likewise missing the point.
>>
>
> But you shouldn't use annotations for something that refuses to use
> runtime introspection abilities. There's no reason to waste annotations if
> the need can be fullfilled by separate description files in a DSL that
> doesn't even need to be Python code (but cutely looks like so).
>

It doesn't feel like a waste to me -- having it syntactically be part of
the source code to me is a strong advantage over stubs or special comments
or even special syntax in docstrings.

As I have said several times now, it fulfills exactly the original (pre-PEP
3107) goal I had in mind for them.


> And it's not surprising that some people may be missing the point when a
> feature pretends to use a runtime facility but interprets it from a
> separate channel which eschews any code execution.
>

There is no pretense here. It is simply useful to make the type annotations
*also* available at run time.


> Your proposal seems determined by the fact that mypy has much grander
> ambitions (and therefore requires its insertion into regular Python), but
> it doesn't make use of that power at all, worse, it forbids others to use
> it.
>

This remark about mypy's ambitions sounds delusional and paranoid. I
wouldn't care about mypy at all if it wasn't right in line with my
intentions and desires for Python. And while its author *originally*
intended to create a whole new language (since that is what academics do
:-), he gracefully changed his design (over a year ago) when I suggested
that it might be more useful if it fit neatly into Python 3 annotations.

Finally, I would actually be okay if we found a way to let type hints and
other annotations coexist -- I just prefer type hints to be the default
use, since I see them as much more generally useful than all other uses
combined.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140824/8d08ea01/attachment.html>

From guido at python.org  Mon Aug 25 05:29:08 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 24 Aug 2014 20:29:08 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CACwMPm9EkDT98gM-9_fwE_NqEwg5qA7QaR-nVN4WVYjowkc3Eg@mail.gmail.com>
References: <CAP7+vJ+HouBUzaATiFnqGDT4WX99qt4k8X4jaT4T+RLuOO9Deg@mail.gmail.com>
 <CANc-5UyHg_V5dx6qSc0ZXX6y-KeNRXsRK6kYBr3Xa==X88K3JA@mail.gmail.com>
 <CACwMPm9EkDT98gM-9_fwE_NqEwg5qA7QaR-nVN4WVYjowkc3Eg@mail.gmail.com>
Message-ID: <CAP7+vJL7VTBt4r4NBcJOXtBQG_6QmzT-eAgHy5r0dcv9dfXW5Q@mail.gmail.com>

On Fri, Aug 22, 2014 at 7:30 AM, Bob Ippolito <bob at redivi.com> wrote:

>
>
> On Friday, August 22, 2014, Skip Montanaro <skip at pobox.com> wrote:
>
>> There's been a lot to read in this and related threads over the past nine
>> days. (According to Gmail. It qualitatively seems much longer to me.) I
>> think I've followed along reasonably well. Forgive me if I missed the
>> answers to these questions. The proposal on the table is to adopt MyPy's
>> type annotations for Python 3.mumble. I presume MyPy already supports them.
>>
>> 1. Can MyPy be used today as a standalone static type checker for Python
>> 3.x code without actually compiling anything? That is, can I just sprinkle
>> type annotations into my code and run a front-end pass of MyPy much the
>> same way I'd run pylint, vulture, flake8, or other lint-ish program?
>>
>
> Yes. mypy -S performs checking without execution.
>
>  2. Assuming the answer to #1 is "yes," if you start sprinkling type
>> annotations into your code and running "mypy *.py", will it tell you when
>> it needs a missing type annotation to more fully check things, or will it
>> silently process code without annotations and not let you know that it's
>> not really checking much?
>>
>
> No. It happily and silently accepts dynamic code with no annotations. It
> can't do much for you with that code, but it doesn't complain.
>

However, mypy has a flag --html-report which generates a marked-up source
code listing that shows which regions of the code have or haven't been
checked, using similar color-coding as coverage.py.


> 3. Will we get into a phase like the early days of "const" in ANSI C where
>> addition of "const" in one location in your existing code base forced you
>> into a never-ending iterations of adding const all over the place? I forget
>> what that was called ("const propagation"?), but I recall that it generally
>> wasn't a fun activity.
>>
>
> Nope.
>

The Python interpreter won't force you. Of course you can still play this
game (e.g. by insisting that no red zones should exist in the
above-mentioned source listing) but that's no different from insisting on
zero lint warnings -- it is a choice a developer (usually a team) makes,
not something the software enforces.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140824/5e3ee032/attachment.html>

From tjreedy at udel.edu  Mon Aug 25 08:18:12 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 25 Aug 2014 02:18:12 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
Message-ID: <ltekf5$7nj$1@ger.gmane.org>

On 8/24/2014 11:12 PM, Guido van Rossum wrote:

> Finally, I would actually be okay if we found a way to let type hints
> and other annotations coexist -- I just prefer type hints to be the
> default use, since I see them as much more generally useful than all
> other uses combined.

I believe co-existence is possible, but the details will depend on the 
form type hints take. First, other annotations should be easily 
distinguished from type hints. Second, other annotations that would 
interfere with the runtime use of __annotations__ should be pulled out 
by a decorator and possibly moved to another attribute specific to the 
decorator (such as '_deco_name').

-- 
Terry Jan Reedy


From edk141 at gmail.com  Mon Aug 25 12:20:03 2014
From: edk141 at gmail.com (Ed Kellett)
Date: Mon, 25 Aug 2014 11:20:03 +0100
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <ltekf5$7nj$1@ger.gmane.org>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
Message-ID: <CABmzr0grC6qM4swNhQn6J+Fk=miEMdfMh-6Q9c8GGMCW21_8rg@mail.gmail.com>

On 25 August 2014 07:18, Terry Reedy <tjreedy at udel.edu> wrote:
> On 8/24/2014 11:12 PM, Guido van Rossum wrote:
>
>> Finally, I would actually be okay if we found a way to let type hints
>> and other annotations coexist -- I just prefer type hints to be the
>> default use, since I see them as much more generally useful than all
>> other uses combined.
>
>
> I believe co-existence is possible, but the details will depend on the form
> type hints take. First, other annotations should be easily distinguished
> from type hints. Second, other annotations that would interfere with the
> runtime use of __annotations__ should be pulled out by a decorator and
> possibly moved to another attribute specific to the decorator (such as
> '_deco_name').

? or we could have a decorator for type hints, and ascribe no new
meaning at all to __annotations__. Or assume __annotations__ contains
type hints iff all the annotations present are instances of typing.*.
That might be better anyway, since a decorator could then add (or
augment) type information without assigning to __annotations__ (which
would be weird).

Even if conveying type information is the most useful use of
annotations, there's no reason it can't be explicit (and consistent
with other uses of annotations).

Ed Kellett

From cfkaran2 at gmail.com  Mon Aug 25 12:57:23 2014
From: cfkaran2 at gmail.com (Cem Karan)
Date: Mon, 25 Aug 2014 06:57:23 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CABmzr0grC6qM4swNhQn6J+Fk=miEMdfMh-6Q9c8GGMCW21_8rg@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
 <CABmzr0grC6qM4swNhQn6J+Fk=miEMdfMh-6Q9c8GGMCW21_8rg@mail.gmail.com>
Message-ID: <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com>


On Aug 25, 2014, at 6:20 AM, Ed Kellett <edk141 at gmail.com> wrote:

> On 25 August 2014 07:18, Terry Reedy <tjreedy at udel.edu> wrote:
>> On 8/24/2014 11:12 PM, Guido van Rossum wrote:
>> 
>>> Finally, I would actually be okay if we found a way to let type hints
>>> and other annotations coexist -- I just prefer type hints to be the
>>> default use, since I see them as much more generally useful than all
>>> other uses combined.
>> 
>> 
>> I believe co-existence is possible, but the details will depend on the form
>> type hints take. First, other annotations should be easily distinguished
>> from type hints. Second, other annotations that would interfere with the
>> runtime use of __annotations__ should be pulled out by a decorator and
>> possibly moved to another attribute specific to the decorator (such as
>> '_deco_name').
> 
> ? or we could have a decorator for type hints, and ascribe no new
> meaning at all to __annotations__. Or assume __annotations__ contains
> type hints iff all the annotations present are instances of typing.*.
> That might be better anyway, since a decorator could then add (or
> augment) type information without assigning to __annotations__ (which
> would be weird).
> 
> Even if conveying type information is the most useful use of
> annotations, there's no reason it can't be explicit (and consistent
> with other uses of annotations).

I'm going to beat on my drum some more, but if annotations are dictionaries, where the keys are objects known to be associated with the type checker, then we don't have to guess, we'll know.  E.g.:

"""
# type_checker.py

class type_checker(object):
	# decorator magic

TYPE_CHECKER = type_checker()
"""

"""
# Your file
from type_checker import TYPE_CHECKER

@TYPE_CHECKER(a, int)
def foo(a):
	pass
"""

which is morally equivalent to:

"""
from type_checker import TYPE_CHECKER

def foo(a: {TYPE_CHECKER: int}):
	pass
"""

Chris Angelico pointed this trick out to me, and I think it's a good one.  Assuming the type checker has enough brains to check and ensure that TYPE_CHECKER is never reassigned, it is guaranteed by the runtime system that TYPE_CHECKER is unique, which means that it can do the moral equivalent of 'TYPE_CHECKER in foo.__annotations__['a']'.  That gives a VERY simple method of knowing what the annotations are being used for.

Thanks,
Cem Karan

From rosuav at gmail.com  Mon Aug 25 13:03:48 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 25 Aug 2014 21:03:48 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
 <CABmzr0grC6qM4swNhQn6J+Fk=miEMdfMh-6Q9c8GGMCW21_8rg@mail.gmail.com>
 <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com>
Message-ID: <CAPTjJmrAxV1bGy8BU4FXFoumSb_T+3Z831re2MP6fitHr4TjmQ@mail.gmail.com>

On Mon, Aug 25, 2014 at 8:57 PM, Cem Karan <cfkaran2 at gmail.com> wrote:
> from type_checker import TYPE_CHECKER
>
> def foo(a: {TYPE_CHECKER: int}):
>         pass

I still don't think this offers much benefit over

def foo(a: int):
    pass

It's a lot wordier and the flexibility will almost never be needed. So
is that multiplexing really worth it?

ChrisA

From oreilldf at gmail.com  Mon Aug 25 15:44:12 2014
From: oreilldf at gmail.com (Dan O'Reilly)
Date: Mon, 25 Aug 2014 09:44:12 -0400
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
Message-ID: <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>

Adding active/idle/total worker counts for both ThreadPoolExecutor and
ProcessPoolExecutor is pretty straightforward; I just threw a patch
together for both in 30 minutes or so. However, I don't think its possible
to inspect the contents of a ProcessPoolExecutor's queue without actually
consuming items from it. While it *is* possible with ThreadPoolExecutor, I
don't think we should expose it - the queue.Queue() implementation
ThreadPoolExecutor relies on doesn't have a public API for inspecting its
contents, so ThreadPoolExecutor probably shouldn't expose one, either.
Identifying which task each worker is processing is possible, but would
perhaps require more work than its worth, at least for ProcessPoolExecutor.

I do think adding worker count APIs is reasonable, and in-line with a TODO
item in the ThreadPoolExecutor source:

   # TODO(bquinlan): Should avoid creating new threads if there are more
   # idle threads than items in the work queue.

So, at the very least there have been plans to internally keep track
active/idle thread counts. If others agree it's a good idea, I'll open an
issue on the tracker for this and include my patch (which also addresses
that TODO item).



On Sun, Aug 24, 2014 at 5:41 PM, Ram Rachum <ram.rachum at gmail.com> wrote:

> Sometimes I want to take a live executor, like a `ThreadPoolExecutor`, and
> check up on it. I want to know how many threads there are, how many are
> handling tasks and which tasks, how many are free, and which tasks are in
> the queue.
>
> I asked on Stack Overflow:
> http://stackoverflow.com/questions/25474204/checking-up-on-a-concurrent-futures-threadpoolexecutor
>
> There's an answer there, but it uses private variables and it's not part
> of the API.
>
> I suggest it become a part of the API. There should be an API for checking
> on what the executor is currently doing and answering all the questions I
> raised above.
>
>
> Thanks,
> Ram.
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/33f67d4d/attachment.html>

From antoine at python.org  Mon Aug 25 15:57:17 2014
From: antoine at python.org (Antoine Pitrou)
Date: Mon, 25 Aug 2014 09:57:17 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
Message-ID: <ltffbu$h78$1@ger.gmane.org>


Le 24/08/2014 23:12, Guido van Rossum a ?crit :
 >
 > As I have said several times now, it fulfills exactly the original
 > (pre-PEP 3107) goal I had in mind for them.

But were you envisioning back then that said annotations would be
looked up without the regular execution environment?

PEP 3107 doesn't say anything about that (or rather, it says that 
annotations can be looked up on the function, but function objects only 
exist at run-time). Actually, the bytecode isn't very practical to work 
with to extract annotations, it seems:

 >>> def g():
...   def f(a: "foo") -> "bar": pass
...
 >>> dis.dis(g)
   2           0 LOAD_CONST               1 ('foo')
               3 LOAD_CONST               2 ('bar')
               6 LOAD_CONST               3 (('a', 'return'))
               9 LOAD_CONST               4 (<code object f at 
0x7f17339e2580, file "<stdin>", line 2>)
              12 LOAD_CONST               5 ('g.<locals>.f')
              15 EXTENDED_ARG             3
              18 MAKE_FUNCTION        196608
              21 STORE_FAST               0 (f)
              24 LOAD_CONST               0 (None)
              27 RETURN_VALUE

... so I suppose people would want to run an AST pass instead?
(and then evaluate the "annotations" parts of the AST by hand... uh)

 >     Your proposal seems determined by the fact that mypy has much
 >     grander ambitions (and therefore requires its insertion into regular
 >     Python), but it doesn't make use of that power at all, worse, it
 >     forbids others to use it.
 >
 > This remark about mypy's ambitions sounds delusional and paranoid.

It is entirely uncritical about mypy. It's fine to have other Python
(or Python-like) implementations, even with deliberately different
semantics. Such experimentations actually make the community much 
livelier than, say, PHP's or Ruby's.

I don't know at which point mypy changed goals (if it has), but there 
are still signs in the website of the goal of building a separate 
runtime (not necessarily a separate syntax), e.g.

"""Also some language features that are evaluated at runtime in Python
may happen during compilation in mypy when using the native semantics.
For example, mypy base classes may be bound during compilation (or
program loading, before evaluation), unlike Python."""

Also the fact that mypy supports constructs such as "List[int]()". This
is a set of design constraints that you're not bound to.

Regards

Antoine.



From apalala at gmail.com  Mon Aug 25 16:57:45 2014
From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=)
Date: Mon, 25 Aug 2014 10:27:45 -0430
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org>
 <20140825013023.GK25957@ando> <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
Message-ID: <CAN1YFWu5SoUQRf3qFfPQ-uojhfYALRkq_xcJmZaM_qGKWiM51A@mail.gmail.com>

On Sun, Aug 24, 2014 at 10:00 PM, Guido van Rossum <guido at python.org> wrote:

> We should probably avoid "static typing" and use "type hinting" instead
> (following TypeScript's lead I believe). But the process of looking for
> violations against the hinted types before the code run is still called
> static analysis.


+1


-- 
Juancarlo *A?ez*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/ecebfc09/attachment.html>

From guido at python.org  Mon Aug 25 17:12:04 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 25 Aug 2014 08:12:04 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <ltekf5$7nj$1@ger.gmane.org>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org>
 <20140825013023.GK25957@ando> <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
Message-ID: <CAP7+vJKu4b6NcHHaiXO+xvYpK9yUin5giKG7CaQvWvArNhkENQ@mail.gmail.com>

On Sun, Aug 24, 2014 at 11:18 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 8/24/2014 11:12 PM, Guido van Rossum wrote:
>
>  Finally, I would actually be okay if we found a way to let type hints
>> and other annotations coexist -- I just prefer type hints to be the
>> default use, since I see them as much more generally useful than all
>> other uses combined.
>>
>
> I believe co-existence is possible, but the details will depend on the
> form type hints take. First, other annotations should be easily
> distinguished from type hints. Second, other annotations that would
> interfere with the runtime use of __annotations__ should be pulled out by a
> decorator and possibly moved to another attribute specific to the decorator
> (such as '_deco_name').


What exactly do you mean by "other annotations that would interfere with
the runtime use of __annotations__"?

I can only assume that you are thinking of a situation where you are
introspecting some function/method of unknown origin and you are trying to
see if it has any annotations, in which case you are going to use them for
a locally-defined use (e.g. generate an HTML form -- contrived example).

It sounds as if you are worried about being passed a function that in the
past would not have any annotations (so you would just generate a default
form based on the argument names) but which now has been annotated by a
zealous programmer with type hints. And you are worried that those type
hints will confuse (perhaps crash) the form-generation code.

I think initially we will just tell the user "don't use type annotations
for form-handling functions", and also "don't run the type checker on that
code".

The next stage would probably be to create a decorator (say, @form_handler)
and request that all form handlers are thus decorated. The decorator
wouldn't need to do anything -- we could just teach the type checker that
@form_handler means that argument annotations mean something that's
unrelated to type checking, and we would still have to tell the user "don't
use type annotations for @form_handler functions", but at least they could
run the type checker on files that contain form handlers (the handlers
themselves just wouldn't be type-checked).

It should be easy enough to teach the type checker about such decorators;
all you need is some kind of marker on the decorator function (perhaps
itself a decorator :-).

Eventually users might encourage the form library's maintainers to move the
form info elsewhere (e.g. in the decorator arguments) and then the marker
on the decorator can be taken off.

But the initial stage seems a totally fine solution for 3.5, and it doesn't
require anyone to do anything (it just requires some people to refrain from
doing some new things -- which is much easier, because people are naturally
aware that new things can cause new problems :-).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/9650e9b5/attachment.html>

From guido at python.org  Mon Aug 25 17:29:55 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 25 Aug 2014 08:29:55 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CABmzr0grC6qM4swNhQn6J+Fk=miEMdfMh-6Q9c8GGMCW21_8rg@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
 <CABmzr0grC6qM4swNhQn6J+Fk=miEMdfMh-6Q9c8GGMCW21_8rg@mail.gmail.com>
Message-ID: <CAP7+vJJY3rDgS9RL-q=z23J7ipYsj1zrrAHLpJayKvDiTUoKuQ@mail.gmail.com>

On Mon, Aug 25, 2014 at 3:20 AM, Ed Kellett <edk141 at gmail.com> wrote:
>
> On 25 August 2014 07:18, Terry Reedy <tjreedy at udel.edu> wrote:
> > On 8/24/2014 11:12 PM, Guido van Rossum wrote:
> >
> >> Finally, I would actually be okay if we found a way to let type hints
> >> and other annotations coexist -- I just prefer type hints to be the
> >> default use, since I see them as much more generally useful than all
> >> other uses combined.
> >
> >
> > I believe co-existence is possible, but the details will depend on the
form
> > type hints take. First, other annotations should be easily distinguished
> > from type hints. Second, other annotations that would interfere with the
> > runtime use of __annotations__ should be pulled out by a decorator and
> > possibly moved to another attribute specific to the decorator (such as
> > '_deco_name').
>
> ? or we could have a decorator for type hints, and ascribe no new
> meaning at all to __annotations__. Or assume __annotations__ contains
> type hints iff all the annotations present are instances of typing.*.
> That might be better anyway, since a decorator could then add (or
> augment) type information without assigning to __annotations__ (which
> would be weird).
>
> Even if conveying type information is the most useful use of
> annotations, there's no reason it can't be explicit (and consistent
> with other uses of annotations).

All that sounds fine, but you still have to have a way to convey all that
information to the type checker. Remember, the type checker cannot (or
doesn't want to) execute the code and it can only see annotations in their
original syntactic form. (But it can indeed be told about certain
decorators.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/d7c591ac/attachment.html>

From edk141 at gmail.com  Mon Aug 25 18:07:37 2014
From: edk141 at gmail.com (Ed Kellett)
Date: Mon, 25 Aug 2014 17:07:37 +0100
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJJY3rDgS9RL-q=z23J7ipYsj1zrrAHLpJayKvDiTUoKuQ@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
 <CABmzr0grC6qM4swNhQn6J+Fk=miEMdfMh-6Q9c8GGMCW21_8rg@mail.gmail.com>
 <CAP7+vJJY3rDgS9RL-q=z23J7ipYsj1zrrAHLpJayKvDiTUoKuQ@mail.gmail.com>
Message-ID: <CABmzr0hSsX8wweC-F+TWkNNhMHpUtjA60R3SdaKgGgXP0BSuXA@mail.gmail.com>

On 25 Aug 2014 16:29, "Guido van Rossum" <guido at python.org> wrote:
> > ? or we could have a decorator for type hints, and ascribe no new
> > meaning at all to __annotations__. Or assume __annotations__ contains
> > type hints iff all the annotations present are instances of typing.*.
> > That might be better anyway, since a decorator could then add (or
> > augment) type information without assigning to __annotations__ (which
> > would be weird).
> >
> > Even if conveying type information is the most useful use of
> > annotations, there's no reason it can't be explicit (and consistent
> > with other uses of annotations).
>
> All that sounds fine, but you still have to have a way to convey all that
information to the type checker. Remember, the type checker cannot (or
doesn't want to) execute the code and it can only see annotations in their
original syntactic form. (But it can indeed be told about certain
decorators.)

That's reasonable - I was imagining the decorator for type hints wouldn't
do anything (apart from possibly marking the function as having type
hints), it would just be there to tell the static checker "this function
should be type-checked ".

Ed Kellett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/a8a3ef37/attachment-0001.html>

From guido at python.org  Mon Aug 25 19:26:01 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 25 Aug 2014 10:26:01 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <ltffbu$h78$1@ger.gmane.org>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org>
 <20140825013023.GK25957@ando> <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltffbu$h78$1@ger.gmane.org>
Message-ID: <CAP7+vJLDL_fdOevfAWA33Unkh2rcimOkix_hz=FxO2EMzxFW5g@mail.gmail.com>

On Mon, Aug 25, 2014 at 6:57 AM, Antoine Pitrou <antoine at python.org> wrote:

>
> Le 24/08/2014 23:12, Guido van Rossum a ?crit :
>
> > As I have said several times now, it fulfills exactly the original
> > (pre-PEP 3107) goal I had in mind for them.
>
> But were you envisioning back then that said annotations would be
> looked up without the regular execution environment?
>
> PEP 3107 doesn't say anything about that (or rather, it says that
> annotations can be looked up on the function, but function objects only
> exist at run-time). Actually, the bytecode isn't very practical to work
> with to extract annotations, it seems:
>
> >>> def g():
> ...   def f(a: "foo") -> "bar": pass
> ...
> >>> dis.dis(g)
>   2           0 LOAD_CONST               1 ('foo')
>               3 LOAD_CONST               2 ('bar')
>               6 LOAD_CONST               3 (('a', 'return'))
>               9 LOAD_CONST               4 (<code object f at
> 0x7f17339e2580, file "<stdin>", line 2>)
>              12 LOAD_CONST               5 ('g.<locals>.f')
>              15 EXTENDED_ARG             3
>              18 MAKE_FUNCTION        196608
>              21 STORE_FAST               0 (f)
>              24 LOAD_CONST               0 (None)
>              27 RETURN_VALUE
>
> ... so I suppose people would want to run an AST pass instead?
> (and then evaluate the "annotations" parts of the AST by hand... uh)
>

Actually mypy uses an entirely different parser. This is mostly for
historical reasons: it started out as a compiler/checker/runtime for a
different, somewhat-Python-like statically-typed language. But I assume
that static checkers often have no choice in the matter and must write
their own parser -- e.g. PyCharm is itself written in Java and its parser
must work even if the code is not syntactically correct.

Another reason to use a different parser is that mypy needs to see the
comments so it can look for magic comments starting with "#type:". This is
typical for linters.

In general a type checker should not attempt to cope with clever things
(decorators or otherwise) that modify __annotations__; it should take the
expression tree of the annotation at face value. (It should of course know
about certain decorators like @property and other Python ideosyncracies
like self-passing for methods, and I think there should be a way to teach
it about new decorators. But executing Python code should not be part of
that.)


> >     Your proposal seems determined by the fact that mypy has much
> >     grander ambitions (and therefore requires its insertion into regular
> >     Python), but it doesn't make use of that power at all, worse, it
> >     forbids others to use it.
> >
> > This remark about mypy's ambitions sounds delusional and paranoid.
>
> It is entirely uncritical about mypy. It's fine to have other Python
> (or Python-like) implementations, even with deliberately different
> semantics. Such experimentations actually make the community much livelier
> than, say, PHP's or Ruby's.
>

OK, thanks for clarifying that. I took "grand ambitions" as a
(sarcastically) pejorative term, but you're right that it doesn't need to
be read like that.


> I don't know at which point mypy changed goals (if it has), but there are
> still signs in the website of the goal of building a separate runtime (not
> necessarily a separate syntax), e.g.
>
> """Also some language features that are evaluated at runtime in Python
> may happen during compilation in mypy when using the native semantics.
> For example, mypy base classes may be bound during compilation (or
> program loading, before evaluation), unlike Python."""
>

I have talked to Jukka about this, and he is definitely on board with
reducing mypy's functionality to that of a linter; I think there's even an
issue in mypy's tracker about removing the ability to execute the code
(currently there's a flag you must pass to prevent it from trying to run
the code). Updating the website is an ongoing low-priority project (you can
probably send him pull requests for it).


> Also the fact that mypy supports constructs such as "List[int]()". This
> is a set of design constraints that you're not bound to.
>

Yeah, I'm not crazy about that syntax (I think nobody is). It mostly exists
for the common use case where you have a function that is declared to
return e.g. a list of ints and you want to start off with an empty list;
the current mypy implementation complains about such code. Example:

def squares(xs: Sequence[float]) -> List[float]:
    sqs = []
    for x in xs:
        sqs.append(x**2)
    return sqs

mypy complains about the unannotated initialization of sqs. The two ways to
currently address this are

    sqs = []  # type: List[float]

or

    sqs = List[float]()

Neither is very attractive; mypy should just infer the type of sqs.

Nevertheless, I don't want to use call syntax to set parameters for generic
types, since a generic type still "feels" sufficiently like a class that
calling it is easily confused with instantiation -- even though ABCs are
typically not instantiable. (There's no hard rule for that though -- it is
merely the result of typical ABCs having at least one abstract method.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/9fdcc983/attachment.html>

From guido at python.org  Mon Aug 25 19:30:59 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 25 Aug 2014 10:30:59 -0700
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
Message-ID: <CAP7+vJ+Fgr8-yXq1AF-8ExHyCYqgrXzKp=pimbCevgngAcu_+A@mail.gmail.com>

Doesn't queue.Queue also have methods qsize(), empty() and full()? We could
easily wrap those. There's always the caveat that the numbers may be out of
date as soon as you print them.


On Mon, Aug 25, 2014 at 6:44 AM, Dan O'Reilly <oreilldf at gmail.com> wrote:

> Adding active/idle/total worker counts for both ThreadPoolExecutor and
> ProcessPoolExecutor is pretty straightforward; I just threw a patch
> together for both in 30 minutes or so. However, I don't think its possible
> to inspect the contents of a ProcessPoolExecutor's queue without actually
> consuming items from it. While it *is* possible with ThreadPoolExecutor, I
> don't think we should expose it - the queue.Queue() implementation
> ThreadPoolExecutor relies on doesn't have a public API for inspecting its
> contents, so ThreadPoolExecutor probably shouldn't expose one, either.
> Identifying which task each worker is processing is possible, but would
> perhaps require more work than its worth, at least for ProcessPoolExecutor.
>
> I do think adding worker count APIs is reasonable, and in-line with a TODO
> item in the ThreadPoolExecutor source:
>
>    # TODO(bquinlan): Should avoid creating new threads if there are more
>    # idle threads than items in the work queue.
>
> So, at the very least there have been plans to internally keep track
> active/idle thread counts. If others agree it's a good idea, I'll open an
> issue on the tracker for this and include my patch (which also addresses
> that TODO item).
>
>
>
> On Sun, Aug 24, 2014 at 5:41 PM, Ram Rachum <ram.rachum at gmail.com> wrote:
>
>> Sometimes I want to take a live executor, like a `ThreadPoolExecutor`,
>> and check up on it. I want to know how many threads there are, how many are
>> handling tasks and which tasks, how many are free, and which tasks are in
>> the queue.
>>
>> I asked on Stack Overflow:
>> http://stackoverflow.com/questions/25474204/checking-up-on-a-concurrent-futures-threadpoolexecutor
>>
>> There's an answer there, but it uses private variables and it's not part
>> of the API.
>>
>> I suggest it become a part of the API. There should be an API for
>> checking on what the executor is currently doing and answering all the
>> questions I raised above.
>>
>>
>> Thanks,
>> Ram.
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/f6a668d1/attachment-0001.html>

From antoine at python.org  Mon Aug 25 19:34:43 2014
From: antoine at python.org (Antoine Pitrou)
Date: Mon, 25 Aug 2014 13:34:43 -0400
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
Message-ID: <ltfs3j$p99$1@ger.gmane.org>

Le 25/08/2014 09:44, Dan O'Reilly a ?crit :
>
> So, at the very least there have been plans to internally keep track
> active/idle thread counts. If others agree it's a good idea, I'll open
> an issue on the tracker for this and include my patch (which also
> addresses that TODO item).

I agree that basic executor parameters could be reflected, and I also 
agree that some other pieces of runtime state cannot be reliably 
computed and therefore shouldn't be exposed.

Don't hesitate to open an issue with your patch.

Regards

Antoine.



From ram at rachum.com  Mon Aug 25 19:37:25 2014
From: ram at rachum.com (Ram Rachum)
Date: Mon, 25 Aug 2014 20:37:25 +0300
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <ltfs3j$p99$1@ger.gmane.org>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
Message-ID: <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>

"some other pieces of runtime state cannot be reliably computed"

Can you please specify which ones you mean, and why not reliable?


On Mon, Aug 25, 2014 at 8:34 PM, Antoine Pitrou <antoine at python.org> wrote:

> Le 25/08/2014 09:44, Dan O'Reilly a ?crit :
>
>
>> So, at the very least there have been plans to internally keep track
>> active/idle thread counts. If others agree it's a good idea, I'll open
>> an issue on the tracker for this and include my patch (which also
>> addresses that TODO item).
>>
>
> I agree that basic executor parameters could be reflected, and I also
> agree that some other pieces of runtime state cannot be reliably computed
> and therefore shouldn't be exposed.
>
> Don't hesitate to open an issue with your patch.
>
> Regards
>
> Antoine.
>
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> --- You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/python-ideas/pl3r5SsbLLU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/62376a75/attachment.html>

From antoine at python.org  Mon Aug 25 19:57:08 2014
From: antoine at python.org (Antoine Pitrou)
Date: Mon, 25 Aug 2014 13:57:08 -0400
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
Message-ID: <ltftdk$8ct$1@ger.gmane.org>

Le 25/08/2014 13:37, Ram Rachum a ?crit :
> "some other pieces of runtime state cannot be reliably computed"
>
> Can you please specify which ones you mean, and why not reliable?

I cannot say for sure without taking a more detailed look at 
concurrent.futures :-) However, any runtime information such as "the 
tasks current being processes" (as opposed to, say, waiting) may not be 
available to the calling thread or process, or may be unreliable once it 
returns to the function's caller (since the actual state may have 
changed in-between).

In the former case (information not available to the main process), we 
can't expose the information at all; in the latter case, we may still 
choose to expose it with the usual caveats in the documentation (exactly 
like Queue.qsize()).

Regards

Antoine.



From ram at rachum.com  Mon Aug 25 20:16:56 2014
From: ram at rachum.com (Ram Rachum)
Date: Mon, 25 Aug 2014 21:16:56 +0300
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <ltftdk$8ct$1@ger.gmane.org>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
Message-ID: <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>

Maybe I'm missing something, but I don't think that's something that should
block implementation.

Information not available? Change the executor code to make that
information available. Information could have been changed? So what? That
is to be expected. When I read a file in Python, by the time the line
finished someone could have written something to that file so the result of
the read may not be current. Even if I read just a simple variable, by the
next line it might have been changed by another thread. I really don't see
why any of that deserves special consideration.


On Mon, Aug 25, 2014 at 8:57 PM, Antoine Pitrou <antoine at python.org> wrote:

> Le 25/08/2014 13:37, Ram Rachum a ?crit :
>
>  "some other pieces of runtime state cannot be reliably computed"
>>
>> Can you please specify which ones you mean, and why not reliable?
>>
>
> I cannot say for sure without taking a more detailed look at
> concurrent.futures :-) However, any runtime information such as "the tasks
> current being processes" (as opposed to, say, waiting) may not be available
> to the calling thread or process, or may be unreliable once it returns to
> the function's caller (since the actual state may have changed in-between).
>
> In the former case (information not available to the main process), we
> can't expose the information at all; in the latter case, we may still
> choose to expose it with the usual caveats in the documentation (exactly
> like Queue.qsize()).
>
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> --- You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/python-ideas/pl3r5SsbLLU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/aa544a9b/attachment.html>

From antoine at python.org  Mon Aug 25 20:43:50 2014
From: antoine at python.org (Antoine Pitrou)
Date: Mon, 25 Aug 2014 14:43:50 -0400
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
Message-ID: <ltg056$dnd$1@ger.gmane.org>

Le 25/08/2014 14:16, Ram Rachum a ?crit :
> Maybe I'm missing something, but I don't think that's something that
> should block implementation.
>
> Information not available? Change the executor code to make that
> information available.

Not if that would make the implementation much more complicated, or 
significantly slower.

Regards

Antoine.



From guido at python.org  Mon Aug 25 20:54:03 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 25 Aug 2014 11:54:03 -0700
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <ltg056$dnd$1@ger.gmane.org>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
Message-ID: <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>

It might be worth it to make the implementation somewhat more complicated
if it serves a good purpose, for example giving the user of the program
insights into how well the executor is performing. Without such insight you
may be attempting to tune parameters (like the pool size) without being
able to evaluate their effect.


On Mon, Aug 25, 2014 at 11:43 AM, Antoine Pitrou <antoine at python.org> wrote:

> Le 25/08/2014 14:16, Ram Rachum a ?crit :
>
>  Maybe I'm missing something, but I don't think that's something that
>> should block implementation.
>>
>> Information not available? Change the executor code to make that
>> information available.
>>
>
> Not if that would make the implementation much more complicated, or
> significantly slower.
>
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/00533605/attachment.html>

From oreilldf at gmail.com  Mon Aug 25 22:03:47 2014
From: oreilldf at gmail.com (Dan O'Reilly)
Date: Mon, 25 Aug 2014 16:03:47 -0400
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
Message-ID: <CAP3foKKd3s5a7j8WU++6agxuf2XW3fJP20ZaRzADmpC6i2fNXw@mail.gmail.com>

I'll take a look at this again tonight and see if more detailed information
(e.g. which tasks are actually being processed) can be determined without
too much added complexity and/or performance penalties. If I can come up
with something reasonable for both ProcessPool/ThreadPool, I'll add it to
the changes I've already made. Either way, I'll create an issue to track
this.


On Mon, Aug 25, 2014 at 2:54 PM, Guido van Rossum <guido at python.org> wrote:

> It might be worth it to make the implementation somewhat more complicated
> if it serves a good purpose, for example giving the user of the program
> insights into how well the executor is performing. Without such insight you
> may be attempting to tune parameters (like the pool size) without being
> able to evaluate their effect.
>
>
> On Mon, Aug 25, 2014 at 11:43 AM, Antoine Pitrou <antoine at python.org>
> wrote:
>
>> Le 25/08/2014 14:16, Ram Rachum a ?crit :
>>
>>  Maybe I'm missing something, but I don't think that's something that
>>> should block implementation.
>>>
>>> Information not available? Change the executor code to make that
>>> information available.
>>>
>>
>> Not if that would make the implementation much more complicated, or
>> significantly slower.
>>
>>
>> Regards
>>
>> Antoine.
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/49ccc851/attachment.html>

From abarnert at yahoo.com  Mon Aug 25 22:05:18 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 25 Aug 2014 13:05:18 -0700
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
Message-ID: <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>

I don't think there's any issue with letting people introspect the executor. The problem is that the main thing you get is a queue, and there's a limit to how introspectable a queue can be.

In particular, if you want to iterate the waiting tasks, you have to iterate the queue, and there's no safe way to do that.

Since CPython's queue.Queue happens to be just a deque and a mutex, you could make it iterable at the cost of blocking all producers and consumers (which might be fine for many uses, like debugging or exploratory programming), or provide a snapshot API to return a copy of the deque.

But do you want to make that a requirement on all subclasses of Queue, and all other implementations' queue modules? Does ProirityQueue have to nondestructively iterate a heap in order? Does Jython have to use a mutex and a deque instead of a more efficient (and possibly lock-free) queue from the Java stdlib? What does multiprocessing.Queue do on each implementation?

I don't think the costs are worth the benefit. And I assume that's why the existing queue API doesn't provide an iteration or snapshot mechanism.

But there's an option that might be worth doing:

Provide a queue.IntrospectableQueue type that _is_ defined to have such a mechanism, but to otherwise work like a Queue (except maybe less efficiently). Then provide an optional parameter for the Executor that lets you specify an alternate queue constructor in place of the default. So, when exploring or debugging, you could pass queue.IntrospectableQueue (or multiprocessing.IntrospectableQueue for ProcessPoolExecutor).

Whether the interface is "lock_and_return_iterator" or "snapshot", this would be trivial to implement in CPython, and other Pythons could just copy the CPython version instead of extending their native queue types.

Sent from a random iPhone

On Aug 25, 2014, at 11:54, Guido van Rossum <guido at python.org> wrote:

> It might be worth it to make the implementation somewhat more complicated if it serves a good purpose, for example giving the user of the program insights into how well the executor is performing. Without such insight you may be attempting to tune parameters (like the pool size) without being able to evaluate their effect.
> 
> 
> On Mon, Aug 25, 2014 at 11:43 AM, Antoine Pitrou <antoine at python.org> wrote:
>> Le 25/08/2014 14:16, Ram Rachum a ?crit :
>> 
>>> Maybe I'm missing something, but I don't think that's something that
>>> should block implementation.
>>> 
>>> Information not available? Change the executor code to make that
>>> information available.
>> 
>> Not if that would make the implementation much more complicated, or significantly slower.
>> 
>> 
>> Regards
>> 
>> Antoine.
>> 
>> 
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/2a2f55ef/attachment-0001.html>

From ram at rachum.com  Mon Aug 25 23:07:18 2014
From: ram at rachum.com (Ram Rachum)
Date: Tue, 26 Aug 2014 00:07:18 +0300
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
Message-ID: <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>

Sounds good to me. Having to specify `IntrospectableQueue` to the executor
is a bit of a chore, but not too bad to get this functionality. I also bet
that the performance difference wouldn't be an issue for most uses.


On Mon, Aug 25, 2014 at 11:05 PM, 'Andrew Barnert' via python-ideas <
python-ideas at googlegroups.com> wrote:

> I don't think there's any issue with letting people introspect the
> executor. The problem is that the main thing you get is a queue, and
> there's a limit to how introspectable a queue can be.
>
> In particular, if you want to iterate the waiting tasks, you have to
> iterate the queue, and there's no safe way to do that.
>
> Since CPython's queue.Queue happens to be just a deque and a mutex, you
> could make it iterable at the cost of blocking all producers and consumers
> (which might be fine for many uses, like debugging or exploratory
> programming), or provide a snapshot API to return a copy of the deque.
>
> But do you want to make that a requirement on all subclasses of Queue, and
> all other implementations' queue modules? Does ProirityQueue have to
> nondestructively iterate a heap in order? Does Jython have to use a mutex
> and a deque instead of a more efficient (and possibly lock-free) queue from
> the Java stdlib? What does multiprocessing.Queue do on each implementation?
>
> I don't think the costs are worth the benefit. And I assume that's why the
> existing queue API doesn't provide an iteration or snapshot mechanism.
>
> But there's an option that might be worth doing:
>
> Provide a queue.IntrospectableQueue type that _is_ defined to have such a
> mechanism, but to otherwise work like a Queue (except maybe less
> efficiently). Then provide an optional parameter for the Executor that lets
> you specify an alternate queue constructor in place of the default. So,
> when exploring or debugging, you could pass queue.IntrospectableQueue (or
> multiprocessing.IntrospectableQueue for ProcessPoolExecutor).
>
> Whether the interface is "lock_and_return_iterator" or "snapshot", this
> would be trivial to implement in CPython, and other Pythons could just copy
> the CPython version instead of extending their native queue types.
>
> Sent from a random iPhone
>
> On Aug 25, 2014, at 11:54, Guido van Rossum <guido at python.org> wrote:
>
> It might be worth it to make the implementation somewhat more complicated
> if it serves a good purpose, for example giving the user of the program
> insights into how well the executor is performing. Without such insight you
> may be attempting to tune parameters (like the pool size) without being
> able to evaluate their effect.
>
>
> On Mon, Aug 25, 2014 at 11:43 AM, Antoine Pitrou <antoine at python.org>
> wrote:
>
>> Le 25/08/2014 14:16, Ram Rachum a ?crit :
>>
>>  Maybe I'm missing something, but I don't think that's something that
>>> should block implementation.
>>>
>>> Information not available? Change the executor code to make that
>>> information available.
>>>
>>
>> Not if that would make the implementation much more complicated, or
>> significantly slower.
>>
>>
>> Regards
>>
>> Antoine.
>>
>>
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas at python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "python-ideas" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-ideas/pl3r5SsbLLU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-ideas+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140826/44c51d62/attachment.html>

From tjreedy at udel.edu  Tue Aug 26 01:36:53 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 25 Aug 2014 19:36:53 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJKu4b6NcHHaiXO+xvYpK9yUin5giKG7CaQvWvArNhkENQ@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
 <CAP7+vJKu4b6NcHHaiXO+xvYpK9yUin5giKG7CaQvWvArNhkENQ@mail.gmail.com>
Message-ID: <ltghan$s8p$1@ger.gmane.org>

On 8/25/2014 11:12 AM, Guido van Rossum wrote:
> On Sun, Aug 24, 2014 at 11:18 PM, Terry Reedy

>     I believe co-existence is possible, but the details will depend on
>     the form type hints take. First, other annotations should be easily
>     distinguished from type hints. Second, other annotations that would
>     interfere with the runtime use of __annotations__ should be pulled
>     out by a decorator and possibly moved to another attribute specific
>     to the decorator (such as '_deco_name').
>
>
> What exactly do you mean by "other annotations that would interfere with
> the runtime use of __annotations__"?

I am referring to the current stdlib runtime use of .__annotations__ 
directly by inspect.getfullargspec and now inspect.signature and 
indirectly by Idle calltips.

def f(n:[int, 'random non-type info'])->(int, 'more non-type info'): pass
import inspect as ip
print(ip.formatargspec(*ip.getfullargspec(f)))
print(str(ip.signature(f)))

(n: [<class 'int'>, 'random non-type info']) -> (<class 'int'>, 'more 
non-type info')
(n:[<class 'int'>, 'random non-type info']) -> (<class 'int'>, 'more 
non-type info')

(Idle currently displays the first, will change to the second. The 
display difference is more pronounced (now, at least) for C functions 
with .__text_signature__ from Argument Clinic.)

To me, 'argument specification' and especially 'signature' imply 
arguments names and types, but not other stuff. Certainly, name and type 
is all that is wanted for a tip on how to write a call.  Extra stuff 
would typically be noise in this context -- especially for beginners.

> I can only assume that you are thinking of a situation where you are
> introspecting some function/method of unknown origin and you are trying
> to see if it has any annotations, in which case you are going to use
> them for a locally-defined use (e.g. generate an HTML form -- contrived
> example).
>
> It sounds as if you are worried about being passed a function that in
> the past would not have any annotations (so you would just generate a
> default form based on the argument names) but which now has been
> annotated by a zealous programmer with type hints. And you are worried
> that those type hints will confuse (perhaps crash) the form-generation code.

I am worried about annotations other than type hints. Compact type 
hints, especially for type-opaque parameter names, should improve the 
usefulness of call tips. Extra noise detracts.

If type hints were standardized so as to be easily recognizable, 
.signature could be given an option to ignore anything else.

-- 
Terry Jan Reedy


From guido at python.org  Tue Aug 26 01:58:31 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 25 Aug 2014 16:58:31 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <ltghan$s8p$1@ger.gmane.org>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org>
 <20140825013023.GK25957@ando> <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
 <CAP7+vJKu4b6NcHHaiXO+xvYpK9yUin5giKG7CaQvWvArNhkENQ@mail.gmail.com>
 <ltghan$s8p$1@ger.gmane.org>
Message-ID: <CAP7+vJKux+fhktBVDGRuaqcUXDwpP4KDBk3wXeCvwGSTsVcZ5A@mail.gmail.com>

On Mon, Aug 25, 2014 at 4:36 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 8/25/2014 11:12 AM, Guido van Rossum wrote:
>
>> On Sun, Aug 24, 2014 at 11:18 PM, Terry Reedy
>>
>
>      I believe co-existence is possible, but the details will depend on
>>     the form type hints take. First, other annotations should be easily
>>     distinguished from type hints. Second, other annotations that would
>>     interfere with the runtime use of __annotations__ should be pulled
>>     out by a decorator and possibly moved to another attribute specific
>>     to the decorator (such as '_deco_name').
>>
>>
>> What exactly do you mean by "other annotations that would interfere with
>> the runtime use of __annotations__"?
>>
>
> I am referring to the current stdlib runtime use of .__annotations__
> directly by inspect.getfullargspec and now inspect.signature and indirectly
> by Idle calltips.
>
> def f(n:[int, 'random non-type info'])->(int, 'more non-type info'): pass
> import inspect as ip
> print(ip.formatargspec(*ip.getfullargspec(f)))
> print(str(ip.signature(f)))
>
> (n: [<class 'int'>, 'random non-type info']) -> (<class 'int'>, 'more
> non-type info')
> (n:[<class 'int'>, 'random non-type info']) -> (<class 'int'>, 'more
> non-type info')
>
> (Idle currently displays the first, will change to the second. The display
> difference is more pronounced (now, at least) for C functions with
> .__text_signature__ from Argument Clinic.)
>
> To me, 'argument specification' and especially 'signature' imply arguments
> names and types, but not other stuff. Certainly, name and type is all that
> is wanted for a tip on how to write a call.  Extra stuff would typically be
> noise in this context -- especially for beginners.
>

I see two possible concerns here -- can you clarify?

(1) If a function is annotated with multiple *categories* of annotations
(e.g. type hints and html form specifiers) you want to change inspect to
only return the type hints.

(2) If a function is annotated for a non-type-hinting category (e.g. only
html form specifiers) you want to change inspect to return nothing.

I hadn't thought of either of these; I was only concerned about making sure
that a static type checker wouldn't be unduly confused by non-type-hinting
annotations. I assume __annotations__ should always return the full
expression found in the syntactic position of the annotation (though
decorators can change this).


>  I can only assume that you are thinking of a situation where you are
>> introspecting some function/method of unknown origin and you are trying
>> to see if it has any annotations, in which case you are going to use
>> them for a locally-defined use (e.g. generate an HTML form -- contrived
>> example).
>>
>> It sounds as if you are worried about being passed a function that in
>> the past would not have any annotations (so you would just generate a
>> default form based on the argument names) but which now has been
>> annotated by a zealous programmer with type hints. And you are worried
>> that those type hints will confuse (perhaps crash) the form-generation
>> code.
>>
>
> I am worried about annotations other than type hints. Compact type hints,
> especially for type-opaque parameter names, should improve the usefulness
> of call tips. Extra noise detracts.
>
> If type hints were standardized so as to be easily recognizable,
> .signature could be given an option to ignore anything else.
>

So I think you are actually not excited about any mechanism to keep other
uses of annotations alive, and you would just as well only have them used
for type hints? (That's fine with me, but there is the backward
compatibility issue, and perhaps legitimate cool alternate uses of
annotations. We still have the choice to allow only one category of
annotations per function.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/ffe69884/attachment-0001.html>

From steve at pearwood.info  Tue Aug 26 03:48:40 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 26 Aug 2014 11:48:40 +1000
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <ltghan$s8p$1@ger.gmane.org>
References: <20140823172536.GZ25957@ando> <ltb6f6$65d$1@ger.gmane.org>
 <20140825013023.GK25957@ando> <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
 <CAP7+vJKu4b6NcHHaiXO+xvYpK9yUin5giKG7CaQvWvArNhkENQ@mail.gmail.com>
 <ltghan$s8p$1@ger.gmane.org>
Message-ID: <20140826014839.GO25957@ando>

On Mon, Aug 25, 2014 at 07:36:53PM -0400, Terry Reedy wrote:

> I am referring to the current stdlib runtime use of .__annotations__ 
> directly by inspect.getfullargspec and now inspect.signature and 
> indirectly by Idle calltips.
> 
> def f(n:[int, 'random non-type info'])->(int, 'more non-type info'): pass
> import inspect as ip
> print(ip.formatargspec(*ip.getfullargspec(f)))
> print(str(ip.signature(f)))
>
> (n: [<class 'int'>, 'random non-type info']) -> (<class 'int'>, 'more 
> non-type info')
> (n:[<class 'int'>, 'random non-type info']) -> (<class 'int'>, 'more 
> non-type info')
> 
> (Idle currently displays the first, will change to the second. The 
> display difference is more pronounced (now, at least) for C functions 
> with .__text_signature__ from Argument Clinic.)
> 
> To me, 'argument specification' and especially 'signature' imply 
> arguments names and types, but not other stuff.

I don't think that's a useful defintion for Python.

There is no hard definition of "function signature" in computer science, 
since different languages provide different information in the function 
declaration. But here's a definition from the C++ world which seems to 
capture the gist of it to me:

    A function signature consists of the function prototype.  What
    it tells you is the general information about a function, its 
    name, parameters, what scope it is in, and other miscellaneous 
    information.

http://www.cs.unm.edu/~storm/C++/ProgrammingTerms/FunctionSignatures.html

To put it another way, it's anything in the function declaration apart 
from the actual body of the function. In the case of Python, that 
includes annotations, which currently have no official semantics, but 
may include type hints, or documentation, or anything else.


> Certainly, name and type 
> is all that is wanted for a tip on how to write a call.  Extra stuff 
> would typically be noise in this context -- especially for beginners.

I think that it is particularly for beginners that "extra stuff" in the 
form of documentation could be useful. Don't think of "random non-type 
info", as you state above, think of NON-random information about the 
parameter. The most obvious example is documentation about what the 
parameter represents.

Here's a hypothetical example of what I might have done had annotations 
not been prohibited in the standard library:

print(inspect.signature(statistics.pvariance))
=> (data, mu:"pre-calculated mean of the data, if already known"=None)

So I think that there's no good reason to decide, ahead of time, that 
the information in an annotation is "noise" in a tool-tip. I think the 
tool-tip should show whatevr information the author of the function 
thinks is important enough to go in the annotation.

(However, the tool-tip might choose to format it a bit more nicely than 
it currently does. But that's a separate discussion.)


> >I can only assume that you are thinking of a situation where you are
> >introspecting some function/method of unknown origin and you are trying
> >to see if it has any annotations, in which case you are going to use
> >them for a locally-defined use (e.g. generate an HTML form -- contrived
> >example).
> >
> >It sounds as if you are worried about being passed a function that in
> >the past would not have any annotations (so you would just generate a
> >default form based on the argument names) but which now has been
> >annotated by a zealous programmer with type hints. And you are worried
> >that those type hints will confuse (perhaps crash) the form-generation 
> >code.
> 
> I am worried about annotations other than type hints. Compact type 
> hints, especially for type-opaque parameter names, should improve the 
> usefulness of call tips. Extra noise detracts.

Certainly "noise" detracts, but you have no reason to assume that 
annotations can only be two things: type hints, or noise.

In fact, I would argue that for a beginner, type hints will often be 
noise. Do you expect beginners to understand ABCs? If not, what are they 
to make of something like this?

flatten(it:Iterable[Any])->Sequence[Any]



-- 
Steven

From stephen at xemacs.org  Tue Aug 26 03:43:42 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 26 Aug 2014 10:43:42 +0900
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <CAGzF1uf25d-mOC8iuoXnVXQCP4QjHi_21qpogjfGXcxZ_02xEQ@mail.gmail.com>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <CAGzF1uf25d-mOC8iuoXnVXQCP4QjHi_21qpogjfGXcxZ_02xEQ@mail.gmail.com>
Message-ID: <87a96sc9f5.fsf@uwakimon.sk.tsukuba.ac.jp>

Warren Weckesser writes:

 > I don't agree that this change "significantly changes the semantics of
 > reduce".  The nullifier is optional.

Many programs deal with domains that seem to have nullifiers but don't.
See Steven's example of floating point multiplication where the "tail"
might contain Inf or NaN.  You can argue that this is a "consenting
adults" issue, but I consider this possibility an attractive nuisance,
and I'd rather it not be in the stdlib.

IMHO YMMV


From cfkaran2 at gmail.com  Tue Aug 26 04:12:05 2014
From: cfkaran2 at gmail.com (Cem Karan)
Date: Mon, 25 Aug 2014 22:12:05 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAPTjJmrAxV1bGy8BU4FXFoumSb_T+3Z831re2MP6fitHr4TjmQ@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
 <CABmzr0grC6qM4swNhQn6J+Fk=miEMdfMh-6Q9c8GGMCW21_8rg@mail.gmail.com>
 <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com>
 <CAPTjJmrAxV1bGy8BU4FXFoumSb_T+3Z831re2MP6fitHr4TjmQ@mail.gmail.com>
Message-ID: <752DAB4F-C271-4B37-8F6D-7F76D0626086@gmail.com>


On Aug 25, 2014, at 7:03 AM, Chris Angelico <rosuav at gmail.com> wrote:

> On Mon, Aug 25, 2014 at 8:57 PM, Cem Karan <cfkaran2 at gmail.com> wrote:
>> from type_checker import TYPE_CHECKER
>> 
>> def foo(a: {TYPE_CHECKER: int}):
>>        pass
> 
> I still don't think this offers much benefit over
> 
> def foo(a: int):
>    pass
> 
> It's a lot wordier and the flexibility will almost never be needed. So
> is that multiplexing really worth it?

I'm just trying to preserve annotations as they currently are; that is, without a clearly defined meaning.  Right now, I see annotations as a great place to store both documentation and type information, but without some kind of multiplexing standard you can't have both at the same time.  I also suspect that if some kind of standard is in place that allows multiplexing, then other uses will be invented to take advantage of it.  

The other alternative is to define more attributes that functions are supposed to have, but that feels like a hack.  It feels cleaner to me to store information about the arguments and return values with the arguments and return values directly, rather than somewhere else.  I know that with decorators that isn't a problem, but it doesn't feel as clean to me.

Thanks,
Cem Karan

From oreilldf at gmail.com  Tue Aug 26 04:51:18 2014
From: oreilldf at gmail.com (Dan O'Reilly)
Date: Mon, 25 Aug 2014 22:51:18 -0400
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
Message-ID: <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>

The IntrospectableQueue idea seems reasonable to me. I think I would prefer
passing an introspectable (or similar) keyword to the Executor rather than
a queue class, though. Adding support for identifying which tasks are
active introduces some extra overhead, which I think can reasonably be made
optional. If we're going to use a different Queue class to enable
introspection, we might as well disable the other stuff that we're doing to
make introspection work. It also makes it easier to raise an exception if
an API is called that won't work without IntrospectableQueue being used.

>> Does Jython have to use a mutex and a deque instead of a more efficient
(and possibly lock-free) queue from the Java stdlib?

For what it's worth, Jython just uses CPython's queue.Queue implementation,
as far as I can tell.

>> What does multiprocessing.Queue do on each implementation?

In addition to a multiprocessing.Queue, the ProcessPoolExecutor maintains a
dict of all submitted work items, so that can be used instead of trying to
inspect the queue itself.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/a1e739c1/attachment.html>

From ethan at stoneleaf.us  Tue Aug 26 05:02:24 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Aug 2014 20:02:24 -0700
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
Message-ID: <53FBF8C0.3070300@stoneleaf.us>

On 08/25/2014 07:51 PM, Dan O'Reilly wrote:
>
> The IntrospectableQueue idea seems reasonable to me. I think I would prefer passing an introspectable (or similar)
> keyword to the Executor rather than a queue class, though.

Passing the class is the better choice -- it means that future needs can be more easily met by designing the queue 
variant needed and passing it in -- having a keyword to select only one option is unnecessarily limiting.

--
~Ethan~

From oreilldf at gmail.com  Tue Aug 26 05:31:21 2014
From: oreilldf at gmail.com (Dan O'Reilly)
Date: Mon, 25 Aug 2014 23:31:21 -0400
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <53FBF8C0.3070300@stoneleaf.us>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
 <53FBF8C0.3070300@stoneleaf.us>
Message-ID: <CAP3foKJUi3uFanuhHE3kzJogHUp9-j5YkDkdJk+hEpMwuRPEmA@mail.gmail.com>

In that case, what's the best way to disallow use of APIs that require an
introspectable queue implementation? Using isinstance(self._work_queue,
IntrospectableQueue) would work, but seems to be nearly as limiting as
using an introspectable keyword. Perhaps IntrospectableQueue could support
__iter__ as a way of iterating over a snapshot of enqueued items - The
Executor could try iterating over the queue when it needs to inspect its
contents, raising an appropriate exception (something like "Provided queue
class must be introspectable") if that fails. If people prefer __iter__
isn't used for that purpose, we could just do the same thing with whatever
public method ends up being used to get the snapshot instead.


On Mon, Aug 25, 2014 at 11:02 PM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 08/25/2014 07:51 PM, Dan O'Reilly wrote:
>
>>
>> The IntrospectableQueue idea seems reasonable to me. I think I would
>> prefer passing an introspectable (or similar)
>> keyword to the Executor rather than a queue class, though.
>>
>
> Passing the class is the better choice -- it means that future needs can
> be more easily met by designing the queue variant needed and passing it in
> -- having a keyword to select only one option is unnecessarily limiting.
>
> --
> ~Ethan~
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/93966280/attachment-0001.html>

From guido at python.org  Tue Aug 26 05:38:20 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 25 Aug 2014 20:38:20 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <752DAB4F-C271-4B37-8F6D-7F76D0626086@gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org>
 <20140825013023.GK25957@ando> <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
 <CABmzr0grC6qM4swNhQn6J+Fk=miEMdfMh-6Q9c8GGMCW21_8rg@mail.gmail.com>
 <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com>
 <CAPTjJmrAxV1bGy8BU4FXFoumSb_T+3Z831re2MP6fitHr4TjmQ@mail.gmail.com>
 <752DAB4F-C271-4B37-8F6D-7F76D0626086@gmail.com>
Message-ID: <CAP7+vJJUpQo5spKhx0ea9agLRYp4etRsCs64r-MV=+akcA5pLg@mail.gmail.com>

On Mon, Aug 25, 2014 at 7:12 PM, Cem Karan <cfkaran2 at gmail.com> wrote:

>
> On Aug 25, 2014, at 7:03 AM, Chris Angelico <rosuav at gmail.com> wrote:
>
> > On Mon, Aug 25, 2014 at 8:57 PM, Cem Karan <cfkaran2 at gmail.com> wrote:
> >> from type_checker import TYPE_CHECKER
> >>
> >> def foo(a: {TYPE_CHECKER: int}):
> >>        pass
> >
> > I still don't think this offers much benefit over
> >
> > def foo(a: int):
> >    pass
> >
> > It's a lot wordier and the flexibility will almost never be needed. So
> > is that multiplexing really worth it?
>
> I'm just trying to preserve annotations as they currently are; that is,
> without a clearly defined meaning.  Right now, I see annotations as a great
> place to store both documentation and type information, but without some
> kind of multiplexing standard you can't have both at the same time.  I also
> suspect that if some kind of standard is in place that allows multiplexing,
> then other uses will be invented to take advantage of it.
>
> The other alternative is to define more attributes that functions are
> supposed to have, but that feels like a hack.  It feels cleaner to me to
> store information about the arguments and return values with the arguments
> and return values directly, rather than somewhere else.  I know that with
> decorators that isn't a problem, but it doesn't feel as clean to me.
>

I appreciate your efforts, but in my mind this is a lost cause -- whatever
notation you use to put multiple annotations on a single argument is just
too noisy to bear, and I don't think the need is strong enough. Let's focus
instead on coming up with a way to indicate to which category the
annotations for a given function belong. As I've said before, I think a
simple decorator should suffice.

In fact, let me propose a straw-man so we have something to shoot down.

The only thing we need is a way to tell the static type checker (which for
the rest of this message I'm just going to abbreviate as "mypy") "these are
not the annotations you are looking for." The simplest thing that could
possibly work would be a single designated decorator. Let's say we import
it from typing.py:

from typing import no_type_checks

@no_type_checks
def speeder(short: 'r2d2', tall: 'c3po') -> 'luke':
    print('These are not the droids you are looking for')

(Note that mypy uses string literals as forward class references, so
without the decorator mypy would probably issue some confused complaints.)

However, I think we need to be slightly more flexible. Most likely a
function that uses annotations in some pre-mypy way is already decorated by
a decorator that interprets the annotations. It would be annoying to have
to double-decorate such functions, so I think there should be a way to
decorate such decorators. Assuming such a decorator is implemented as a
function, we can just decorate the decorator, as follows:

from typing import no_type_checks

@no_type_checks
def clone_trooper(func):
    print('Looking for', func.__annotations__, 'in', func.__name__)
    func.searched = True
    return func

@clone_trooper
def speeder(short: 'r2d2', tall: 'c3po') -> 'luke':
    print('These are not the droids you are looking for')

This is probably going to put off some people for its lack of purity -- how
does mypy know that the @no_type_checks flag rubs off on anything decorated
with it? In fact, how would it know that in the first example speeder()
isn't a decorator? I personally think decorators are rare enough that no
confusion will happen in practice, but I'll concede that perhaps it would
be better to use two different decorators. Then @no_type_checks could be a
simple no-op decorator which is itself decorated with e.g.
@no_type_checks_decorator; both defined by typing.py.

PS. In case someone asks, mypy is smart enough to follow imports, so
@clone_trooper doesn't need to be defined in the same file where it is used
-- I just did that to simplify the example.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/ac87e1c5/attachment.html>

From antoine at python.org  Tue Aug 26 05:43:04 2014
From: antoine at python.org (Antoine Pitrou)
Date: Mon, 25 Aug 2014 23:43:04 -0400
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <53FBF8C0.3070300@stoneleaf.us>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
 <53FBF8C0.3070300@stoneleaf.us>
Message-ID: <ltgvo8$j3g$1@ger.gmane.org>

Le 25/08/2014 23:02, Ethan Furman a ?crit :
> On 08/25/2014 07:51 PM, Dan O'Reilly wrote:
>>
>> The IntrospectableQueue idea seems reasonable to me. I think I would
>> prefer passing an introspectable (or similar)
>> keyword to the Executor rather than a queue class, though.
>
> Passing the class is the better choice -- it means that future needs can
> be more easily met by designing the queue variant needed and passing it
> in -- having a keyword to select only one option is unnecessarily limiting.

What if an implementation wants to use something other than a queue?
It seems you're breaking the abstraction here.

Regards

Antoine.



From abarnert at yahoo.com  Tue Aug 26 05:48:41 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 25 Aug 2014 20:48:41 -0700
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
Message-ID: <1409024921.35699.YahooMailNeo@web181001.mail.ne1.yahoo.com>

On Monday, August 25, 2014 7:52 PM, Dan O'Reilly <oreilldf at gmail.com> wrote:

>The IntrospectableQueue idea seems reasonable to me. I think I would prefer passing an introspectable (or similar) keyword to the Executor rather than a queue class, though. Adding support for identifying which tasks are active introduces some extra overhead, which I think can reasonably be made optional. If we're going to use a different Queue class to enable introspection, we might as well disable the other stuff that we're doing to make introspection work. It also makes it easier to raise an exception if an API is called that won't work without IntrospectableQueue being used.

Even though this was my suggestion, let me play devil's advocate for a second?

The main reason to use this is for debugging or exploratory programming.?

In the debugger, of course, it's not necessary, because you can just break and suspend all the threads while you do what you want. Would it be reasonable to do the same thing outside the debugger, by providing a threading.Thread.suspend API (and of course the pool and executor APIs have a suspend method that suspends all their threads) so you can safely access the queue's internals?

Obviously suspending threads in general is a bad thing to do unless you're a big fan of deadlocks, but for debugging and exploration it seems reasonable; if a program occasionally deadlocks or crashes while you're screwing with its threads to see what happens, well, you were screwing with its threads to see what happens?

That might be a horrible attractive nuisance, but if you required an extra flag to be passed in at construction time to make these methods available, and documented that it was unsafe and potentially inefficient, it might be acceptable.

On the other hand, it's hard to think of a case where this is a good answer but "just run it in the debugger" isn't a better answer?

>>>?Does Jython have to use a mutex and a deque instead of a more efficient (and possibly lock-free) queue from the Java stdlib?
>
>For what it's worth, Jython just uses CPython's queue.Queue implementation, as far as I can tell.


Now that I think about it, that makes sense; if I really need a lock-free thread pool and queue in Jython, I'm probably going to use the native Java executors, not the Python ones, right?

>>>?What does multiprocessing.Queue do on each implementation?
>
>In addition to a multiprocessing.Queue, the ProcessPoolExecutor maintains a dict of all submitted work items, so that can be used instead of trying to inspect the queue itself.


Interesting. This implies that supplying an inspectable queue class may not be the best answer here; instead, we could have an option for an inspectable work dict, which would just expose the existing one for ProcessPoolExecutor, while it would make ThreadPoolExecutor maintain an equivalent dict as a thread-local in the launching thread.?(I'm assuming you only need to inspect the jobs from the launching process/thread here? I'm not sure if that's sufficient for the OP's intended use or not.)

From abarnert at yahoo.com  Tue Aug 26 05:56:30 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 25 Aug 2014 20:56:30 -0700
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <ltgvo8$j3g$1@ger.gmane.org>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
 <53FBF8C0.3070300@stoneleaf.us> <ltgvo8$j3g$1@ger.gmane.org>
Message-ID: <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com>

On Monday, August 25, 2014 8:44 PM, Antoine Pitrou <antoine at python.org> wrote:

> > Le 25/08/2014 23:02, Ethan Furman a ?crit :
>>  On 08/25/2014 07:51 PM, Dan O'Reilly wrote:
>>> 
>>>  The IntrospectableQueue idea seems reasonable to me. I think I would
>>>  prefer passing an introspectable (or similar)
>>>  keyword to the Executor rather than a queue class, though.
>> 
>>  Passing the class is the better choice -- it means that future needs can
>>  be more easily met by designing the queue variant needed and passing it
>>  in -- having a keyword to select only one option is unnecessarily limiting.
> 
> What if an implementation wants to use something other than a queue?
> It seems you're breaking the abstraction here.


A collection of threads and a shared queue is almost the definition of a thread pool. What else would you use?

Also, this could make it a lot easier to create variations on ThreadPoolExecutor without subclassing or forking it. For example, if you want your tasks to run in priority order, just give it a priority queue keyed on task.priority. If you want a scheduled executor, just give it a priority queue whose get method blocks until the first task's task.timestamp or a new task is added ahead of the first. And so on.

I'm not sure if that's a good idea or not, but it's an interesting possibility at least?

From guido at python.org  Tue Aug 26 06:08:17 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 25 Aug 2014 21:08:17 -0700
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
 <53FBF8C0.3070300@stoneleaf.us> <ltgvo8$j3g$1@ger.gmane.org>
 <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com>
Message-ID: <CAP7+vJKYOrSDCpfnt=xDOxK3dYEqzi=8EwyrhL_K5HaxqYeVSw@mail.gmail.com>

Hm. Maybe we should not complicate the API after all. This seems a lot of
theorizing without enough of a use case.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/95d5068c/attachment.html>

From warren.weckesser at gmail.com  Tue Aug 26 06:09:24 2014
From: warren.weckesser at gmail.com (Warren Weckesser)
Date: Tue, 26 Aug 2014 00:09:24 -0400
Subject: [Python-ideas] Add nullifier argument to functools.reduce?
In-Reply-To: <87a96sc9f5.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAGzF1ucYBTp4=Z9iQGaY5cLF1k1nhxHiRPzyk7xzD-4qjOfKRA@mail.gmail.com>
 <CAEbHw4Z9SRWvTs1+zCduJBPSCXR6sfeX7PyqeQPCGJENoFqYGA@mail.gmail.com>
 <CAGzF1uf25d-mOC8iuoXnVXQCP4QjHi_21qpogjfGXcxZ_02xEQ@mail.gmail.com>
 <87a96sc9f5.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAGzF1ue8TmXX4VM1FHLCqS1moNOaDNb_OYR+5bUjZgQLTsRzNA@mail.gmail.com>

On Mon, Aug 25, 2014 at 9:43 PM, Stephen J. Turnbull <stephen at xemacs.org>
wrote:

> Warren Weckesser writes:
>
>  > I don't agree that this change "significantly changes the semantics of
>  > reduce".  The nullifier is optional.
>
> Many programs deal with domains that seem to have nullifiers but don't.
> See Steven's example of floating point multiplication where the "tail"
> might contain Inf or NaN.  You can argue that this is a "consenting
> adults" issue, but I consider this possibility an attractive nuisance,
> and I'd rather it not be in the stdlib.
>
>

Gotcha.  To spell it out: one might naively think 0 is a nullifier for
floating point multiplication, but 0*inf is nan, not 0.  That means
reduce(lambda x,y: x*y, [1e-50]*10 + [float('inf')], nullifier=0), which
underflows to 0 during the intermediate steps, would incorrectly return 0.
So yeah, consenting adults, know your data, etc.  :)  (nan, on the other
hand, appears to be a true nullifier for the product in IEEE 754 floating
point arithmetic.)

Warren


IMHO YMMV
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140826/e4f996ab/attachment.html>

From antoine at python.org  Tue Aug 26 06:13:53 2014
From: antoine at python.org (Antoine Pitrou)
Date: Tue, 26 Aug 2014 00:13:53 -0400
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
 <53FBF8C0.3070300@stoneleaf.us> <ltgvo8$j3g$1@ger.gmane.org>
 <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com>
Message-ID: <lth1i2$8h2$1@ger.gmane.org>


Le 25/08/2014 23:56, Andrew Barnert a ?crit :
>>
>> What if an implementation wants to use something other than a queue?
>> It seems you're breaking the abstraction here.
>
> A collection of threads and a shared queue is almost the definition of a thread pool. What else would you use?

Definitions don't necessarily have any relationship with the way a 
feature is implemented. Perhaps some version of concurrent.futures would 
like to use some advanced dispatch mechanism provided by the OS (or 
shared memory, or whatever).

(I'll note that such "flexibility" has been chosen for the API of 
threading.Condition and it is making it difficult to write an optimized 
implementation that would you use OS-native facilities, such as pthread 
condition variables)

We have come from a simple proposal to introspect some runtime 
properties of an executor to the idea of swapping out a building block 
with another. That doesn't sound reasonable.

Regards

Antoine.



From ethan at stoneleaf.us  Tue Aug 26 06:22:38 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Aug 2014 21:22:38 -0700
Subject: [Python-ideas] Executor Docs [was: Re: Add an introspection API to
 Executor]
In-Reply-To: <CAP7+vJKYOrSDCpfnt=xDOxK3dYEqzi=8EwyrhL_K5HaxqYeVSw@mail.gmail.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
 <53FBF8C0.3070300@stoneleaf.us> <ltgvo8$j3g$1@ger.gmane.org>
 <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJKYOrSDCpfnt=xDOxK3dYEqzi=8EwyrhL_K5HaxqYeVSw@mail.gmail.com>
Message-ID: <53FC0B8E.1010309@stoneleaf.us>

On 08/25/2014 09:08 PM, Guido van Rossum wrote:
>
> Hm. Maybe we should not complicate the API after all. This seems a lot of theorizing without enough of a use case.

I went in search of docs to see what the API actually was, and while I know the source code is a great place to go look 
for education and finer points, should we have to go looking there just to see what the __init__ parameters are?

I'm going to go out on a limb and say that ThreadPoolExecutor takes a max_workers param, but I only have that because 
it's in the example.

On the up side, having a link to the source is really cool.  Having clicked on that I now know that max_workers is the 
only param taken.  ;)

--
~Ethan~

From ethan at stoneleaf.us  Tue Aug 26 06:27:37 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Aug 2014 21:27:37 -0700
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <CAP7+vJKYOrSDCpfnt=xDOxK3dYEqzi=8EwyrhL_K5HaxqYeVSw@mail.gmail.com>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
 <53FBF8C0.3070300@stoneleaf.us> <ltgvo8$j3g$1@ger.gmane.org>
 <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJKYOrSDCpfnt=xDOxK3dYEqzi=8EwyrhL_K5HaxqYeVSw@mail.gmail.com>
Message-ID: <53FC0CB9.1070202@stoneleaf.us>

On 08/25/2014 09:08 PM, Guido van Rossum wrote:
>
> Hm. Maybe we should not complicate the API after all. This seems a lot of theorizing without enough of a use case.

Introspection (aka debugging) is an important use case.  Having looked at the code, and with Antoine's comments in mind, 
I'd be happy with whatever Dan can get in there without changing the queuing implementation -- if anyone needs that much 
flexibility, they can take the Python code and massage it to their own desires.

--
~Ethan~

From antoine at python.org  Tue Aug 26 06:37:52 2014
From: antoine at python.org (Antoine Pitrou)
Date: Tue, 26 Aug 2014 00:37:52 -0400
Subject: [Python-ideas] Executor Docs [was: Re: Add an introspection API
	to Executor]
In-Reply-To: <53FC0B8E.1010309@stoneleaf.us>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
 <53FBF8C0.3070300@stoneleaf.us> <ltgvo8$j3g$1@ger.gmane.org>
 <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJKYOrSDCpfnt=xDOxK3dYEqzi=8EwyrhL_K5HaxqYeVSw@mail.gmail.com>
 <53FC0B8E.1010309@stoneleaf.us>
Message-ID: <lth2v0$m1r$1@ger.gmane.org>

Le 26/08/2014 00:22, Ethan Furman a ?crit :
>
> I went in search of docs to see what the API actually was, and while I
> know the source code is a great place to go look for education and finer
> points, should we have to go looking there just to see what the __init__
> parameters are?

So, you didn't find the docs?

https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor

"""
class concurrent.futures.ThreadPoolExecutor(max_workers)

     An Executor subclass that uses a pool of at most max_workers 
threads to execute calls asynchronously.
"""

https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor

"""
class concurrent.futures.ProcessPoolExecutor(max_workers=None)

     An Executor subclass that executes calls asynchronously using a 
pool of at most max_workers processes. If max_workers is None or not 
given, it will default to the number of processors on the machine.
"""

Regards

Antoine.



From tjreedy at udel.edu  Tue Aug 26 07:42:31 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 26 Aug 2014 01:42:31 -0400
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
Message-ID: <lth6o9$m5n$1@ger.gmane.org>

On 8/25/2014 7:58 PM, Guido van Rossum wrote:
 > On Mon, Aug 25, 2014 at 4:36 PM, Terry Reedy <tjreedy at udel.edu
 > <mailto:tjreedy at udel.edu>> wrote:

 >     I am referring to the current stdlib runtime use of .__annotations__
 >     directly by inspect.getfullargspec and now inspect.signature and
 >     indirectly by Idle calltips.
 >
 >     def f(n:[int, 'random non-type info'])->(int, 'more non-type info'):
 >     pass
 >     import inspect as ip
 >     print(ip.formatargspec(*ip.__getfullargspec(f)))
 >     print(str(ip.signature(f)))
 >
 >     (n: [<class 'int'>, 'random non-type info']) -> (<class 'int'>,
 >     'more non-type info')
 >     (n:[<class 'int'>, 'random non-type info']) -> (<class 'int'>, 'more
 >     non-type info')
 >
 >     (Idle currently displays the first, will change to the second. The
 >     display difference is more pronounced (now, at least) for C
 >     functions with .__text_signature__ from Argument Clinic.)
 >
 >     To me, 'argument specification' and especially 'signature' imply
 >     arguments names and types, but not other stuff. Certainly, name and
 >     type is all that is wanted for a tip on how to write a call.  Extra
 >     stuff would typically be noise in this context -- especially for
 >     beginners.
 >
 >
 > I see two possible concerns here -- can you clarify?
 >
 > (1) If a function is annotated with multiple *categories* of annotations
 > (e.g. type hints and html form specifiers) you want to change inspect to
 > only return the type hints.
 >
 > (2) If a function is annotated for a non-type-hinting category (e.g.
 > only html form specifiers) you want to change inspect to return nothing.

Both are covered by "I would either like non-type-hinting annotations 
moved elsewhere by decorators (my older idea) *or* I would like 
inspect.signature to at least optionally ignore them (today's idea). The 
latter is more dependable since we control the inspect functions.

I an not too worried that anyone would use the blessed type-hint syntax 
for something completely different. On the other hand, I can imagine 
that there might someday be more that one annotation system in general use.

 > I hadn't thought of either of these; I was only concerned about making
 > sure that a static type checker wouldn't be unduly confused by
 > non-type-hinting annotations.

If, for instance, everything in typing or produced by the function is a 
subclass of BaseTyping or an instance of MetaTyping, then post-compile 
typing objects would be easy to distinguish.  Source annotations are a 
harder problem.

 >I assume __annotations__ should always
 > return the full expression found in the syntactic position of the
 > annotation

I think that should continue.

 > (though decorators can change this).

Because .__annotations__ is a regular dict.

annotation syntax in code --compiler>>
initial .__annotation__ dict --decorators>>
exposed .__annotation__  dict + any other attributes added.

 >     I am worried about annotations other than type hints. Compact type
 >     hints, especially for type-opaque parameter names, should improve
 >     the usefulness of call tips. Extra noise detracts.
 >
 >     If type hints were standardized so as to be easily recognizable,
 >     .signature could be given an option to ignore anything else.
 >
 > So I think you are actually not excited about any mechanism to keep
 > other uses of annotations alive,

I want there to be such a mechanism even though it is does not 'excite' me .

Just as you are focused on type hints for static hints, I, *as an Idle 
maintainer* who just happens to have spent considerable time on the call 
tip feature, am focused on type hints for dynamic documentation.

 > and you would just as well only have them used for type hints?

Too strong.

 > (That's fine with me, but there is the backward compatibility issue,

I said in my first response that I don't think we should break code. 
This is me speaking a as core developer and community member. I do want 
multiple ramifications of annotation use considered, including run-time use.

 > and perhaps legitimate cool alternate uses of annotations.

Most experiments do not amount to much, but a few, such as the metaclass 
hack, certainly have. Python should continue to allow experiments.

 > We still have the choice to allow only one category of
 > annotations per function.)

This seems dubious to me. If strings, tuples, or dicts are allowed as 
annotations, it seems to me that they can be used to combine annotations.
-- 
Terry Jan Reedy


From jlehtosalo at gmail.com  Tue Aug 26 07:47:14 2014
From: jlehtosalo at gmail.com (Jukka Lehtosalo)
Date: Mon, 25 Aug 2014 22:47:14 -0700
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAP7+vJLDL_fdOevfAWA33Unkh2rcimOkix_hz=FxO2EMzxFW5g@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltffbu$h78$1@ger.gmane.org>
 <CAP7+vJLDL_fdOevfAWA33Unkh2rcimOkix_hz=FxO2EMzxFW5g@mail.gmail.com>
Message-ID: <CAA_f+LwzzqrSRtT_w+6osDuYTbp6t=BM=8ZtYOEca_wWCyFxpw@mail.gmail.com>

On Mon, Aug 25, 2014 at 10:26 AM, Guido van Rossum <guido at python.org> wrote:

> On Mon, Aug 25, 2014 at 6:57 AM, Antoine Pitrou <antoine at python.org>
> wrote:
>
>> PEP 3107 doesn't say anything about that (or rather, it says that
>> annotations can be looked up on the function, but function objects only
>> exist at run-time). Actually, the bytecode isn't very practical to work
>> with to extract annotations,
>>
> I don't know at which point mypy changed goals (if it has), but there are
>> still signs in the website of the goal of building a separate runtime (not
>> necessarily a separate syntax), e.g.
>>
>> """Also some language features that are evaluated at runtime in Python
>> may happen during compilation in mypy when using the native semantics.
>> For example, mypy base classes may be bound during compilation (or
>> program loading, before evaluation), unlike Python."""
>>
>
> I have talked to Jukka about this, and he is definitely on board with
> reducing mypy's functionality to that of a linter; I think there's even an
> issue in mypy's tracker about removing the ability to execute the code
> (currently there's a flag you must pass to prevent it from trying to run
> the code). Updating the website is an ongoing low-priority project (you can
> probably send him pull requests for it).
>

Yeah, the website wasn't very up-to-date. The project goals have shifted a
lot in the last 18 months. I just updated the tutorial and the FAQ to not
talk about the native semantics.


> mypy complains about the unannotated initialization of sqs. The two ways
> to currently address this are
>
>     sqs = []  # type: List[float]
>
> or
>
>     sqs = List[float]()
>
> Neither is very attractive; mypy should just infer the type of sqs.
>

Agreed, but this is very difficult to do in all possible contexts. For
example, __init__ methods often use an empty list as an attribute
initializer, and here it's more tricky to infer the item type, as we'd have
to look into other methods to figure out the item type.


>
> Nevertheless, I don't want to use call syntax to set parameters for
> generic types, since a generic type still "feels" sufficiently like a class
> that calling it is easily confused with instantiation -- even though ABCs
> are typically not instantiable. (There's no hard rule for that though -- it
> is merely the result of typical ABCs having at least one abstract method.)
>

Generic types don't need to be abstract, and for those the call syntax
would be ambiguous (and confusing).

Jukka
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140825/bc194125/attachment.html>

From ethan at stoneleaf.us  Tue Aug 26 08:12:08 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 25 Aug 2014 23:12:08 -0700
Subject: [Python-ideas] Executor Docs [was: Re: Add an introspection API
 to Executor]
In-Reply-To: <lth2v0$m1r$1@ger.gmane.org>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
 <53FBF8C0.3070300@stoneleaf.us> <ltgvo8$j3g$1@ger.gmane.org>
 <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJKYOrSDCpfnt=xDOxK3dYEqzi=8EwyrhL_K5HaxqYeVSw@mail.gmail.com>
 <53FC0B8E.1010309@stoneleaf.us> <lth2v0$m1r$1@ger.gmane.org>
Message-ID: <53FC2538.4010109@stoneleaf.us>

On 08/25/2014 09:37 PM, Antoine Pitrou wrote:
> Le 26/08/2014 00:22, Ethan Furman a ?crit :
>>
>> I went in search of docs to see what the API actually was, and while I
>> know the source code is a great place to go look for education and finer
>> points, should we have to go looking there just to see what the __init__
>> parameters are?
>
> So, you didn't find the docs?
>
> https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor
>
> """
> class concurrent.futures.ThreadPoolExecutor(max_workers)
>
>      An Executor subclass that uses a pool of at most max_workers threads to execute calls asynchronously.
> """
>
> https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor
>
> """
> class concurrent.futures.ProcessPoolExecutor(max_workers=None)
>
>      An Executor subclass that executes calls asynchronously using a pool of at most max_workers processes. If
> max_workers is None or not given, it will default to the number of processors on the machine.
> """

I did find the docs, and even with your plain text guide I almost didn't see them when I looked just now.  Too much 
fancy going on there, and all the green examples -- yeah, it's hard to read.

For comparison, here's what help(ThreadPoolExecutor) shows:

class ThreadPoolExecutor(concurrent.futures._base.Executor)
  |  Method resolution order:
  |      ThreadPoolExecutor
  |      concurrent.futures._base.Executor
  |      builtins.object
  |
  |  Methods defined here:
  |
  |  __init__(self, max_workers)
  |      Initializes a new ThreadPoolExecutor instance.
  |
  |      Args:
  |          max_workers: The maximum number of threads that can be used to
  |              execute the given calls.
  |
  |  shutdown(self, wait=True)
  |      Clean-up the resources associated with the Executor.
  |
  |      It is safe to call this method several times. Otherwise, no other
  |      methods can be called after this one.
  |
  |      Args:
  |          wait: If True then shutdown will not return until all running
  |              futures have finished executing and the resources used by the
  |              executor have been reclaimed.
  |
  |  submit(self, fn, *args, **kwargs)
  |      Submits a callable to be executed with the given arguments.
  |
  |      Schedules the callable to be executed as fn(*args, **kwargs) and returns
  |      a Future instance representing the execution of the callable.
  |
  |      Returns:
  |          A Future representing the given call.


Much easier to understand.

Looking at the docs again, I think the biggest hurdle to finding that line and recognizing it for what it is is the fact 
that it comes /after/ all the examples.  That's backwards.  Why would you need examples for something you haven't read yet?

--
~Ethan~

From ncoghlan at gmail.com  Tue Aug 26 11:49:48 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 26 Aug 2014 19:49:48 +1000
Subject: [Python-ideas] Executor Docs [was: Re: Add an introspection API
 to Executor]
In-Reply-To: <53FC2538.4010109@stoneleaf.us>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
 <53FBF8C0.3070300@stoneleaf.us> <ltgvo8$j3g$1@ger.gmane.org>
 <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJKYOrSDCpfnt=xDOxK3dYEqzi=8EwyrhL_K5HaxqYeVSw@mail.gmail.com>
 <53FC0B8E.1010309@stoneleaf.us> <lth2v0$m1r$1@ger.gmane.org>
 <53FC2538.4010109@stoneleaf.us>
Message-ID: <CADiSq7d1R=SM0iZfRtrnvUsVoK=wZxYQ4dOChPcu=VKQsCHyhA@mail.gmail.com>

On 26 Aug 2014 16:12, "Ethan Furman" <ethan at stoneleaf.us> wrote:
> Looking at the docs again, I think the biggest hurdle to finding that
line and recognizing it for what it is is the fact that it comes /after/
all the examples.  That's backwards.  Why would you need examples for
something you haven't read yet?

Many of our module docs serve a dual purpose as a tutorial *and* as an API
reference. That's actually a problem, and often a sign of a separate
"HOWTO" guide trying to get out.

Actually doing the work to split them is rather tedious though, so it tends
not to happen very often.

Cheers,
Nick.

>
> --
> ~Ethan~
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140826/f89e4bac/attachment.html>

From kaiser.yann at gmail.com  Tue Aug 26 12:49:02 2014
From: kaiser.yann at gmail.com (Yann Kaiser)
Date: Tue, 26 Aug 2014 12:49:02 +0200
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <CAA_f+LwzzqrSRtT_w+6osDuYTbp6t=BM=8ZtYOEca_wWCyFxpw@mail.gmail.com>
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org>
 <20140825013023.GK25957@ando> <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltffbu$h78$1@ger.gmane.org>
 <CAP7+vJLDL_fdOevfAWA33Unkh2rcimOkix_hz=FxO2EMzxFW5g@mail.gmail.com>
 <CAA_f+LwzzqrSRtT_w+6osDuYTbp6t=BM=8ZtYOEca_wWCyFxpw@mail.gmail.com>
Message-ID: <CANUJvPVdzgMcE3g7Jhkg2K29v7aEQ02jJ1G8DS+NxFfQAn6DfQ@mail.gmail.com>

Given / assuming that most annotated functions will only concern themselves
with one use of annotations (either typing, web forms, cli parameters),
maybe inspect could be extended to have an annotation-filtering function.
Given a namespace(could be any python object), its protocol regarding
annotations for each parameter could be:

* If the annotation is a mapping(c.abc.Mapping): assume the namespace
object is a key of that mapping. If the key is not present, return None for
that annotation.
* If the annotation is not a mapping, it is always returned whole.

The implications would be:

* A stdlib-approved way of namespacing annotations would exist. Annotations
for different introspectors can coexist.
* inspect.signature's purpose remains unchanged and still tells the whole
truth
* Most annotated functions, typing-related or not are unchanged. (except
those that consist of a mapping ?)

In Python:

    import inspect
    import collections.abc

    def filter_annotation(annotation, namespace):
        if isinstance(annotation, collections.abc.Mapping):
            return annotation.get(namespace, inspect.Signature.empty)
        return annotation

    def filter_annotations(signature, namespace):
        params = [
            p.replace(annotation=filter_annotation(p.annotation, namespace))
            for p in signature.parameters.values()
            ]
        ret = filter_annotation(signature.return_annotation, namespace)
        return signature.replace(parameters=params, return_annotation=ret)


    >>> def classic(param:'annot'): pass
    ...
    >>> cliparser = object()
    >>> typing = object()
    >>> def namespaced(*, param:{typing:int, cliparser:'p'}): pass
    ...
    >>> s_classic = inspect.signature(classic)
    >>> s_namespaced = inspect.signature(namespaced)
    >>> print(filter_annotations(s_classic, cliparser))
    (param:'annot')
    >>> print(filter_annotations(s_classic, typing))
    (param:'annot')
    >>> print(filter_annotations(s_namespaced, cliparser))
    (*, param:'p')
    >>> print(filter_annotations(s_namespaced, typing))
    (*, param:int)





On 26 August 2014 07:47, Jukka Lehtosalo <jlehtosalo at gmail.com> wrote:

> On Mon, Aug 25, 2014 at 10:26 AM, Guido van Rossum <guido at python.org>
> wrote:
>
>> On Mon, Aug 25, 2014 at 6:57 AM, Antoine Pitrou <antoine at python.org>
>> wrote:
>>
>>> PEP 3107 doesn't say anything about that (or rather, it says that
>>> annotations can be looked up on the function, but function objects only
>>> exist at run-time). Actually, the bytecode isn't very practical to work
>>> with to extract annotations,
>>>
>> I don't know at which point mypy changed goals (if it has), but there are
>>> still signs in the website of the goal of building a separate runtime (not
>>> necessarily a separate syntax), e.g.
>>>
>>>
>>> """Also some language features that are evaluated at runtime in Python
>>> may happen during compilation in mypy when using the native semantics.
>>> For example, mypy base classes may be bound during compilation (or
>>> program loading, before evaluation), unlike Python."""
>>>
>>
>> I have talked to Jukka about this, and he is definitely on board with
>> reducing mypy's functionality to that of a linter; I think there's even an
>> issue in mypy's tracker about removing the ability to execute the code
>> (currently there's a flag you must pass to prevent it from trying to run
>> the code). Updating the website is an ongoing low-priority project (you can
>> probably send him pull requests for it).
>>
>
> Yeah, the website wasn't very up-to-date. The project goals have shifted a
> lot in the last 18 months. I just updated the tutorial and the FAQ to not
> talk about the native semantics.
>
>
>> mypy complains about the unannotated initialization of sqs. The two ways
>> to currently address this are
>>
>>     sqs = []  # type: List[float]
>>
>> or
>>
>>     sqs = List[float]()
>>
>> Neither is very attractive; mypy should just infer the type of sqs.
>>
>
> Agreed, but this is very difficult to do in all possible contexts. For
> example, __init__ methods often use an empty list as an attribute
> initializer, and here it's more tricky to infer the item type, as we'd have
> to look into other methods to figure out the item type.
>
>
>>
>> Nevertheless, I don't want to use call syntax to set parameters for
>> generic types, since a generic type still "feels" sufficiently like a class
>> that calling it is easily confused with instantiation -- even though ABCs
>> are typically not instantiable. (There's no hard rule for that though -- it
>> is merely the result of typical ABCs having at least one abstract method.)
>>
>
> Generic types don't need to be abstract, and for those the call syntax
> would be ambiguous (and confusing).
>
> Jukka
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140826/74b2d370/attachment.html>

From oreilldf at gmail.com  Wed Aug 27 03:59:24 2014
From: oreilldf at gmail.com (Dan O'Reilly)
Date: Tue, 26 Aug 2014 21:59:24 -0400
Subject: [Python-ideas] Add an introspection API to Executor
In-Reply-To: <53FC0CB9.1070202@stoneleaf.us>
References: <61fbc488-4b78-4c17-a975-a0a864f5a7f2@googlegroups.com>
 <CAP3foKJd2MJZVXHci3Nh5i1YrkeA0No__kQEmXhd6fvoKYnXVQ@mail.gmail.com>
 <ltfs3j$p99$1@ger.gmane.org>
 <CANXboVaUh-K_YdyN5itxN4VHWS0NhB2Z2tLiwDXp58TKP2mEMA@mail.gmail.com>
 <ltftdk$8ct$1@ger.gmane.org>
 <CANXboVZcE1-WEAr7qjo7SqjPiao4HhKVKWaW7MHqQcjBVLMwbQ@mail.gmail.com>
 <ltg056$dnd$1@ger.gmane.org>
 <CAP7+vJLQeeoOuyT72NNwhnTk-_Ubc5Jf10Z5Pnmf52XqbuRjyg@mail.gmail.com>
 <27CFB108-5385-4E8A-B657-1ED4F246A193@yahoo.com>
 <CANXboVYFG+M9YeWY5BYgQUV9Fdh+neTscL4Jiz+F=UimzPM3rA@mail.gmail.com>
 <CAP3foKJUvUp2OwFajOVrqgGPGbeYVE3QQ9iWbBhqf9Yo4B8ZMg@mail.gmail.com>
 <53FBF8C0.3070300@stoneleaf.us> <ltgvo8$j3g$1@ger.gmane.org>
 <1409025390.52596.YahooMailNeo@web181005.mail.ne1.yahoo.com>
 <CAP7+vJKYOrSDCpfnt=xDOxK3dYEqzi=8EwyrhL_K5HaxqYeVSw@mail.gmail.com>
 <53FC0CB9.1070202@stoneleaf.us>
Message-ID: <CAP3foK+RW2nnQcVwG6FH1SpgwKeo0xo5__kxpfYdcuDdzkoH-g@mail.gmail.com>

As promised, I've opened issue22281 (http://bugs.python.org/issue22281),
and attached a patch that makes an attempt at implementing this. Let's
continue any further discussion on this topic there.


On Tue, Aug 26, 2014 at 12:27 AM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 08/25/2014 09:08 PM, Guido van Rossum wrote:
>
>>
>> Hm. Maybe we should not complicate the API after all. This seems a lot of
>> theorizing without enough of a use case.
>>
>
> Introspection (aka debugging) is an important use case.  Having looked at
> the code, and with Antoine's comments in mind, I'd be happy with whatever
> Dan can get in there without changing the queuing implementation -- if
> anyone needs that much flexibility, they can take the Python code and
> massage it to their own desires.
>
> --
> ~Ethan~
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140826/7beab0d1/attachment.html>

From tony at lownds.com  Wed Aug 27 19:20:18 2014
From: tony at lownds.com (Tony Lownds)
Date: Wed, 27 Aug 2014 17:20:18 +0000 (UTC)
Subject: [Python-ideas]
	=?utf-8?q?Proposal=3A_Use_mypy_syntax_for_function?=
	=?utf-8?q?=09annotations?=
References: <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
 <lt8udm$jbd$1@ger.gmane.org> <20140823051357.GX25957@ando>
 <lta5r3$nl$1@ger.gmane.org> <20140823172536.GZ25957@ando>
 <ltb6f6$65d$1@ger.gmane.org> <20140825013023.GK25957@ando>
 <lte5vu$n94$1@ger.gmane.org>
 <CAP7+vJL0yTY0ChFNQ9ujc2qU-+db_ffCy5RKHbdiHVDhLWpiwQ@mail.gmail.com>
 <lte8b4$fbu$1@ger.gmane.org>
 <CAP7+vJJBvMq_b8b2SC7pmCLooTaxVNaZ_BerC+WOdx5_ksEKxw@mail.gmail.com>
 <ltekf5$7nj$1@ger.gmane.org>
 <CABmzr0grC6qM4swNhQn6J+Fk=miEMdfMh-6Q9c8GGMCW21_8rg@mail.gmail.com>
 <69388FB9-2157-4687-97C4-9AA713BFE242@gmail.com>
 <CAPTjJmrAxV1bGy8BU4FXFoumSb_T+3Z831re2MP6fitHr4TjmQ@mail.gmail.com>
 <752DAB4F-C271-4B37-8F6D-7F76D0626086@gmail.com>
 <CAP7+vJJUpQo5spKhx0ea9agLRYp4etRsCs64r-MV=+akcA5pLg@mail.gmail.com>
Message-ID: <loom.20140827T184955-40@post.gmane.org>

Guido van Rossum <guido at ...> writes:

> from typing import no_type_checks
> 
> @no_type_checks
> def speeder(short: 'r2d2', tall: 'c3po') -> 'luke':
> ??? print('These are not the droids you are looking for')
> 
> @no_type_checks
> def clone_trooper(func):
> ??? print('Looking for', func.__annotations__, 'in', func.__name__)
> ??? func.searched = True
> ??? return func
> 
>  @clone_trooper
> def speeder(short: 'r2d2', tall: 'c3po') -> 'luke':
>??? print('These are not the droids you are looking for')
> 

Can I propose a slight modification -- the designated decorator
takes another decorator, in the decoration line:

@non_typing_annotation(None)
def speeder(short: 'r2d2', tall: 'c3po') -> 'luke':
 ??? print('These are not the droids you are looking for')

@non_typing_annotation(clone_trooper)
def speeder(short: 'r2d2', tall: 'c3po') -> 'luke':
?? print('These are not the droids you are looking for')

This way the library defining `clone_trooper` doesn't have to 
change to support the typing improvements, and the reader always 
knows whether the annotations will be treated as type hints without
needing to consult another libraries' source.

Also, I'd like to suggest that mypy should use the import of 'typing'
as an indicator that the module's annotations are in fact type hints.

-Tony



From burak.arslan at arskom.com.tr  Thu Aug 28 08:36:28 2014
From: burak.arslan at arskom.com.tr (Burak Arslan)
Date: Thu, 28 Aug 2014 09:36:28 +0300
Subject: [Python-ideas] Proposal: Use mypy syntax for function
	annotations
In-Reply-To: <20140823013646.GV25957@ando>
References: <CAFpSVpJw847wJyjmpQTd+K5F2MVnTcpazTa8EBTH4Ju=0-JTSQ@mail.gmail.com>
 <CAP7+vJJeYq=wqW8wEVySBa0dshHJPKw4PYrkxJfpXpoQ982FEQ@mail.gmail.com>
 <CACac1F9__qi-+Z+OFe6TMKwV++X=ueBt7Om6gjZjTC8a-09Arg@mail.gmail.com>
 <CAP1=2W64ZR1+7WZ7yxtfHQXCsUja+Tw0TqQS1OR9--7Dmw+Fow@mail.gmail.com>
 <8290DDD8-FB75-41A9-95EC-0D7D932C0F46@stufft.io>
 <77AC791D-A2D6-460A-9BD3-32BA0A185700@yahoo.com> <lt2sh5$9vi$1@ger.gmane.org>
 <87oaveehyi.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAOhO=aMMZQSExaHTYFmbahAj8BXkLXz+BHkQ=57PtOdSinS53w@mail.gmail.com>
 <lt4rsr$a00$2@ger.gmane.org> <20140823013646.GV25957@ando>
Message-ID: <53FECDEC.4030709@arskom.com.tr>

On 08/23/14 04:36, Steven D'Aprano wrote:
> But another reason for disliking call syntax for this is that it gets
> confusing:
>
> def function(param:Sequence(Spam, Eggs))->Mapping(Cheese, Toast):
>
> suffers compared to this:
>
> def function(param:Sequence[Spam, Eggs])->Mapping[Cheese, Toast]:
>
> In the second case, the type annotations stand out more easily because
> they use square brackets instead of round.
>

You're only giving examples with builtins. What about:

    def function(param:SomeClass(Spam, Eggs))->SomeOtherClass(Cheese,
Toast):

If there are template instantiations  (in the C++ sense) there that's
another meaning for parens which already have two meanings. And now
they're also context-dependent!.. At least they were all the same
everywhere before.

So SomeClass[Spam, Eggs] is a fabulous idea -- in my mind it's just a
special way of looking up a certain type.

But as I said before[1] I'd hate to see annotations only be used for type
checking.

I'd love to be able to do:

    Integer[ge=5]

but this comes with the question about what should actually  passed to
__getitem__

I'd avoid changing its signature to accept **kwargs. So I'd say that

    Unicode[15, pattern='[0-9a-z]+']

should translate to:

    {0: 15, 'pattern': '[0-9a-z]+'}

which is a frozen ordered dict. As that's actually a better tuple that
also happens to accept string keys, I doubt that'd interfere with the
existing usage.

We need this for duck typing to work as well. To my understanding, the
type marker "File" means "only accept what's returned by open()". If
however, we want to accept any file-like object, we could use
Object[write=Callable]. Or maybe more specifically:
Object[write=Callable[Bytes]].



> -1 on re-using call syntax: it is more confusing to read inside a
> parameter list, too flexible, and clashes with existing use calling
> types.
>
> +1 on subscripting syntax: it is more easily distinguishable from
> calling, just flexible enough, and doesn't clash with existing
> functionality.
>

Well, we need that flexibility at the very least for making duck typing
work with type-annotated functions.

best,
burak

[1]: https://mail.python.org/pipermail/python-ideas/2014-August/028988.html
[2]: https://github.com/arskom/spyne/pull/313#issuecomment-29029067


From steve at pearwood.info  Fri Aug 29 07:45:27 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 29 Aug 2014 15:45:27 +1000
Subject: [Python-ideas] Query available codecs and error handlers
Message-ID: <20140829054527.GC23931@ando>

Unless I am badly misinformed, there is no way to programmatically check 
what codecs and error handlers are available. I propose two functions in 
the codecs module:

get_codecs()
get_error_handlers()

which each return a set of the available codec, or error handler, names.


Use-cases
---------

(1) At the interactive interpreter, it is often useful to experiment 
with different codecs. Being able to get a list of available codecs 
avoids the frustration of having to look them up elsewhere, or guess.

(2) Applications which offer to read or write text files may wish to 
allow the user to select an encoding. With no obvious way of finding out 
what encodings are available, the application is limited to hard-coding 
some, and hoping that they don't get renamed or removed.


Why sets, rather than lists?
----------------------------

Sets emphasis that the names are in no particular order. Since the names 
should always be strings and therefore hashable, and the order is 
irrelevant, there is no particular need for a list.


Should they be frozensets rather than sets?
-------------------------------------------

I don't care :-)

Using sets may make it easier for the application to filter the results 
in some way, but I really don't think it matters.


Thoughts or comments?



-- 
Steven

From rosuav at gmail.com  Fri Aug 29 09:27:19 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 29 Aug 2014 17:27:19 +1000
Subject: [Python-ideas] Query available codecs and error handlers
In-Reply-To: <20140829054527.GC23931@ando>
References: <20140829054527.GC23931@ando>
Message-ID: <CAPTjJmr_gGBpidShHsOApY5po+TaMh7H875EF=fFeOwLSTAJHw@mail.gmail.com>

On Fri, Aug 29, 2014 at 3:45 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Unless I am badly misinformed, there is no way to programmatically check
> what codecs and error handlers are available. I propose two functions in
> the codecs module:
>
> get_codecs()
> get_error_handlers()
>
> which each return a set of the available codec, or error handler, names.

A quick look at the codecs module suggests that this may not be as
simple as returning a list/set; when _codecs.lookup() is called, it
does a search. So this is actually like asking how to list importable
modules. It may not be possible, but if it is, I would suggest calling
it "list_codecs" or something, as it's basically going to be tracing
the search path and enumerating every codec it finds.

But it sounds like a quite useful facility. +1.

ChrisA

From mal at egenix.com  Fri Aug 29 09:33:47 2014
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 29 Aug 2014 09:33:47 +0200
Subject: [Python-ideas] Query available codecs and error handlers
In-Reply-To: <20140829054527.GC23931@ando>
References: <20140829054527.GC23931@ando>
Message-ID: <54002CDB.9060704@egenix.com>

On 29.08.2014 07:45, Steven D'Aprano wrote:
> Unless I am badly misinformed, there is no way to programmatically check 
> what codecs and error handlers are available. I propose two functions in 
> the codecs module:
> 
> get_codecs()
> get_error_handlers()
> 
> which each return a set of the available codec, or error handler, names.

Question is: how would you implement these ?

The codec registry uses lookup functions to find codecs, so we'd
have to extend those lookup functions to also support reporting
known installed codecs.

For the stdlib encodings package we could simply put a list into
the package, e.g. encodings.available_codecs returning a dictionary
of mappings from codec name to CodecInfo tuples and then extend
the CodecInfo with some extra information such as supported
error handlers, alternative names and information about the
supported input/output types.

At the moment, the available codecs are documented here:

https://docs.python.org/3.5/library/codecs.html?highlight=codecs#standard-encodings

It's probably a good idea to add information about supported
error handlers to that list.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 29 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2014-08-27: Released eGenix PyRun 2.0.1 ...       http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...                21 days to go
2014-09-27: PyDDF Sprint 2014 ...                          29 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ncoghlan at gmail.com  Fri Aug 29 09:49:07 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 29 Aug 2014 17:49:07 +1000
Subject: [Python-ideas] Query available codecs and error handlers
In-Reply-To: <54002CDB.9060704@egenix.com>
References: <20140829054527.GC23931@ando>
	<54002CDB.9060704@egenix.com>
Message-ID: <CADiSq7e73bgz=TNoQG=+z3hPe4ZPVR7PpsLxKOmV0Qah2=0NtA@mail.gmail.com>

On 29 August 2014 17:33, M.-A. Lemburg <mal at egenix.com> wrote:
> On 29.08.2014 07:45, Steven D'Aprano wrote:
>> Unless I am badly misinformed, there is no way to programmatically check
>> what codecs and error handlers are available. I propose two functions in
>> the codecs module:
>>
>> get_codecs()
>> get_error_handlers()
>>
>> which each return a set of the available codec, or error handler, names.
>
> Question is: how would you implement these ?
>
> The codec registry uses lookup functions to find codecs, so we'd
> have to extend those lookup functions to also support reporting
> known installed codecs.

I'd actually be a fan of a PEP to add such an introspection API that
also made it easier to register new codecs just by adding them to a
suitable namespace package. I believe MvL's original idea was to use
the existing encodings package for that, but that doesn't seem
feasible due to the non-empty __init__

> For the stdlib encodings package we could simply put a list into
> the package, e.g. encodings.available_codecs returning a dictionary
> of mappings from codec name to CodecInfo tuples and then extend
> the CodecInfo with some extra information such as supported
> error handlers, alternative names and information about the
> supported input/output types.
>
> At the moment, the available codecs are documented here:
>
> https://docs.python.org/3.5/library/codecs.html?highlight=codecs#standard-encodings
>
> It's probably a good idea to add information about supported
> error handlers to that list.

Those tables are already pretty busy though - I'm not sure how we
could add supported error handler details without making them hard to
read.

Agreed it would be good to make the info more readily available,
though (I had actually hoped to get some proposed revisions together
for the codecs module docs before 3.4 went out the door, but alas, it
was not to be).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From steve at pearwood.info  Fri Aug 29 09:50:51 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 29 Aug 2014 17:50:51 +1000
Subject: [Python-ideas] Query available codecs and error handlers
In-Reply-To: <CAPTjJmr_gGBpidShHsOApY5po+TaMh7H875EF=fFeOwLSTAJHw@mail.gmail.com>
References: <20140829054527.GC23931@ando>
 <CAPTjJmr_gGBpidShHsOApY5po+TaMh7H875EF=fFeOwLSTAJHw@mail.gmail.com>
Message-ID: <20140829075050.GE23931@ando>

On Fri, Aug 29, 2014 at 05:27:19PM +1000, Chris Angelico wrote:
> On Fri, Aug 29, 2014 at 3:45 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> > Unless I am badly misinformed, there is no way to programmatically check
> > what codecs and error handlers are available. I propose two functions in
> > the codecs module:
> >
> > get_codecs()
> > get_error_handlers()
> >
> > which each return a set of the available codec, or error handler, names.
> 
> A quick look at the codecs module suggests that this may not be as
> simple as returning a list/set; when _codecs.lookup() is called, it
> does a search. 

Sure, but codecs have to be registered before they can be used. The 
register function can cache the names. Perhaps any builtin codecs might 
need to be explicitly added to the cache in order to support this, I 
don't know the details of _codecs. If the codec registry were a dict, we 
might even return a view of the dict.keys().

I'm sure there is some solution which is not quite as difficult as 
enumerating all importable modules. There's a lot fewer codecs, they 
don't have platform-dependent suffixes, and unlike modules in the 
PYTHONPATH, once Python starts up the available codecs won't change 
unless register() is called.


-- 
Steven

From p.f.moore at gmail.com  Fri Aug 29 09:51:00 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 29 Aug 2014 08:51:00 +0100
Subject: [Python-ideas] Subprocess: Add an encoding argument
Message-ID: <CACac1F8Fpn2VwEiBKKbzUgc-+3unOhs4e9j9RGg_+9Lu=HyeXw@mail.gmail.com>

At the moment, subprocess offers two options for handlingthe standard
IO streams of the child. By default, the streams are binary, or you
can set universal_newlines to get text-mode streams with universal
newline handling enabled.

With universal_newlines, the encoding of the streams is the default
value for the environment (whatever locale.getpreferredencoding()
returns). However, there can be cases where you want finer control
over the encoding to use (for example, if you run Python in a
subprocess and set PYTHONIOENCODING).

I propose adding an "encoding" parameter to subprocess.Popen (and the
various wrapper routines) to allow specifying the actual encoding to
use.

Obviously, you can simply wrap the binary streams yourself - the main
use for this facility would be in the higher level functions like
check_output and communicate.

Does this seem like a reasonable suggestion?
Paul

From steve at pearwood.info  Fri Aug 29 10:16:15 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 29 Aug 2014 18:16:15 +1000
Subject: [Python-ideas] Query available codecs and error handlers
In-Reply-To: <54002CDB.9060704@egenix.com>
References: <20140829054527.GC23931@ando> <54002CDB.9060704@egenix.com>
Message-ID: <20140829081615.GF23931@ando>

On Fri, Aug 29, 2014 at 09:33:47AM +0200, M.-A. Lemburg wrote:
> On 29.08.2014 07:45, Steven D'Aprano wrote:
> > Unless I am badly misinformed, there is no way to programmatically check 
> > what codecs and error handlers are available. I propose two functions in 
> > the codecs module:
> > 
> > get_codecs()
> > get_error_handlers()
> > 
> > which each return a set of the available codec, or error handler, names.
> 
> Question is: how would you implement these ?
> 
> The codec registry uses lookup functions to find codecs, so we'd
> have to extend those lookup functions to also support reporting
> known installed codecs.

Arrgggh, you're right -- I've been working on a wrong assumption. I had 
forgotten that the codecs.register function takes a function, not a 
name.

I always forgot that, because it's such a strange and unhelpful API 
compared to, say, a mapping of name:codec. It's probably water under the 
bridge now, but is there any documentation for why this API was used in 
the first place?



-- 
Steven

From mal at egenix.com  Fri Aug 29 10:30:55 2014
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 29 Aug 2014 10:30:55 +0200
Subject: [Python-ideas] Query available codecs and error handlers
In-Reply-To: <20140829081615.GF23931@ando>
References: <20140829054527.GC23931@ando> <54002CDB.9060704@egenix.com>
 <20140829081615.GF23931@ando>
Message-ID: <54003A3F.2050408@egenix.com>

On 29.08.2014 10:16, Steven D'Aprano wrote:
> On Fri, Aug 29, 2014 at 09:33:47AM +0200, M.-A. Lemburg wrote:
>> On 29.08.2014 07:45, Steven D'Aprano wrote:
>>> Unless I am badly misinformed, there is no way to programmatically check 
>>> what codecs and error handlers are available. I propose two functions in 
>>> the codecs module:
>>>
>>> get_codecs()
>>> get_error_handlers()
>>>
>>> which each return a set of the available codec, or error handler, names.
>>
>> Question is: how would you implement these ?
>>
>> The codec registry uses lookup functions to find codecs, so we'd
>> have to extend those lookup functions to also support reporting
>> known installed codecs.
> 
> Arrgggh, you're right -- I've been working on a wrong assumption. I had 
> forgotten that the codecs.register function takes a function, not a 
> name.
> 
> I always forgot that, because it's such a strange and unhelpful API 
> compared to, say, a mapping of name:codec. It's probably water under the 
> bridge now, but is there any documentation for why this API was used in 
> the first place?

Because at the time I designed the API in 1999/2000 it wasn't
clear how people would start writing codecs.

Note that codecs do not use a simple name to codec mapping to
figure out the implementation module name. The name typically
goes through a few layers of normalization and then a alias
dictionary to find the name of the implementation.

The lookup functions were meant to implement these more complex
n-1 mappings.

I also thought that codec implementation might want to tap
into system registries of codecs, use file based tables
as basis for encodings, or even implement load on demand.

Today, it's rather obvious that apparently no one has considered
doing any of this, so it would have been better to design a system
where you explicitly register individual codecs (together
with a set of attributes).

It should be possible to phase out the lookup API and expose
the encodings package lookup mechanism directly in the codecs
module. I can help guide people, if they are willing to do the
work, but don't have time to work on this myself.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 29 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2014-08-27: Released eGenix PyRun 2.0.1 ...       http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...                21 days to go
2014-09-27: PyDDF Sprint 2014 ...                          29 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ncoghlan at gmail.com  Fri Aug 29 10:39:35 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 29 Aug 2014 18:39:35 +1000
Subject: [Python-ideas] Subprocess: Add an encoding argument
In-Reply-To: <CACac1F8Fpn2VwEiBKKbzUgc-+3unOhs4e9j9RGg_+9Lu=HyeXw@mail.gmail.com>
References: <CACac1F8Fpn2VwEiBKKbzUgc-+3unOhs4e9j9RGg_+9Lu=HyeXw@mail.gmail.com>
Message-ID: <CADiSq7fnUuT91wyQskz+aFD23_8noUwvRDmZbebb9ZOLgc5tQw@mail.gmail.com>

On 29 Aug 2014 17:52, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> At the moment, subprocess offers two options for handlingthe standard
> IO streams of the child. By default, the streams are binary, or you
> can set universal_newlines to get text-mode streams with universal
> newline handling enabled.
>
> With universal_newlines, the encoding of the streams is the default
> value for the environment (whatever locale.getpreferredencoding()
> returns). However, there can be cases where you want finer control
> over the encoding to use (for example, if you run Python in a
> subprocess and set PYTHONIOENCODING).
>
> I propose adding an "encoding" parameter to subprocess.Popen (and the
> various wrapper routines) to allow specifying the actual encoding to
> use.
>
> Obviously, you can simply wrap the binary streams yourself - the main
> use for this facility would be in the higher level functions like
> check_output and communicate.
>
> Does this seem like a reasonable suggestion?

This actually gets a little messy once you start digging into it, as you
actually have up to 3 streams to deal with (stdin, stdout, stderr), and may
want to set the error handler in addition to the encoding.

http://bugs.python.org/issue6135 has the many gory details.

It's a problem that definitely needs solving, but it may be better
approached by making it easier to create the pipes *separately*, and then
pass relevant details into the subprocess call.

As with win_unicode_console (and even contextlib2), it's probably worth
experimenting in a PyPI module, as that will make it easier for people to
try out with existing Python 3 releases.

Cheers,
Nick.

> Paul
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140829/c231b055/attachment.html>

From mal at egenix.com  Fri Aug 29 10:44:57 2014
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 29 Aug 2014 10:44:57 +0200
Subject: [Python-ideas] Query available codecs and error handlers
In-Reply-To: <CADiSq7e73bgz=TNoQG=+z3hPe4ZPVR7PpsLxKOmV0Qah2=0NtA@mail.gmail.com>
References: <20140829054527.GC23931@ando>	<54002CDB.9060704@egenix.com>
 <CADiSq7e73bgz=TNoQG=+z3hPe4ZPVR7PpsLxKOmV0Qah2=0NtA@mail.gmail.com>
Message-ID: <54003D89.10606@egenix.com>

On 29.08.2014 09:49, Nick Coghlan wrote:
> On 29 August 2014 17:33, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 29.08.2014 07:45, Steven D'Aprano wrote:
>>> Unless I am badly misinformed, there is no way to programmatically check
>>> what codecs and error handlers are available. I propose two functions in
>>> the codecs module:
>>>
>>> get_codecs()
>>> get_error_handlers()
>>>
>>> which each return a set of the available codec, or error handler, names.
>>
>> Question is: how would you implement these ?
>>
>> The codec registry uses lookup functions to find codecs, so we'd
>> have to extend those lookup functions to also support reporting
>> known installed codecs.
> 
> I'd actually be a fan of a PEP to add such an introspection API that
> also made it easier to register new codecs just by adding them to a
> suitable namespace package. I believe MvL's original idea was to use
> the existing encodings package for that, but that doesn't seem
> feasible due to the non-empty __init__

It's fairly easy to have the lookup function in the encodings
package to also look in say a "siteencodings" package for codecs
it cannot find in the stdlib encodings package.

This new siteencodings package could then be setup as namespace
package to make installation easier.

That doesn't answer the original question, though, since
introspection of available codecs would still not be possible.

>> For the stdlib encodings package we could simply put a list into
>> the package, e.g. encodings.available_codecs returning a dictionary
>> of mappings from codec name to CodecInfo tuples and then extend
>> the CodecInfo with some extra information such as supported
>> error handlers, alternative names and information about the
>> supported input/output types.
>>
>> At the moment, the available codecs are documented here:
>>
>> https://docs.python.org/3.5/library/codecs.html?highlight=codecs#standard-encodings
>>
>> It's probably a good idea to add information about supported
>> error handlers to that list.
> 
> Those tables are already pretty busy though - I'm not sure how we
> could add supported error handler details without making them hard to
> read.

Here's one idea: don't use a table, but instead have one subsection
per codec. There are already a few subsections for specific codecs
on the page.

> Agreed it would be good to make the info more readily available,
> though (I had actually hoped to get some proposed revisions together
> for the codecs module docs before 3.4 went out the door, but alas, it
> was not to be).

Since it's not a feature, the doc change could potentially
be backported.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Aug 29 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2014-08-27: Released eGenix PyRun 2.0.1 ...       http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...                21 days to go
2014-09-27: PyDDF Sprint 2014 ...                          29 days to go

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From ncoghlan at gmail.com  Fri Aug 29 11:23:07 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 29 Aug 2014 19:23:07 +1000
Subject: [Python-ideas] Query available codecs and error handlers
In-Reply-To: <54003D89.10606@egenix.com>
References: <20140829054527.GC23931@ando> <54002CDB.9060704@egenix.com>
 <CADiSq7e73bgz=TNoQG=+z3hPe4ZPVR7PpsLxKOmV0Qah2=0NtA@mail.gmail.com>
 <54003D89.10606@egenix.com>
Message-ID: <CADiSq7e6HM1toh_zp6-o_d=itmuW5zT-HBNKeAz8EzdsDg-aLg@mail.gmail.com>

On 29 August 2014 18:44, M.-A. Lemburg <mal at egenix.com> wrote:
> On 29.08.2014 09:49, Nick Coghlan wrote:
>> Agreed it would be good to make the info more readily available,
>> though (I had actually hoped to get some proposed revisions together
>> for the codecs module docs before 3.4 went out the door, but alas, it
>> was not to be).
>
> Since it's not a feature, the doc change could potentially
> be backported.

Right, I think that's probably worth doing. The tricky part is
figuring out *how* to change them.

At the moment, there isn't an especially clear distinction between the
text encoding specific parts (e.g. several of the error handlers, most
of the encodings) and the underlying general purpose codecs machinery.
However, beyond a vague notion of "that distinction should be made
clearer in the docs now that it affects which codecs can be used with
str.encode, bytes.decode and bytearray.decode, and which can *only* be
used with codecs.encode and codecs.decode", I don't have any specific
suggestions to make.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From zedobject at gmail.com  Fri Aug 29 21:12:56 2014
From: zedobject at gmail.com (Milind Khadilkar)
Date: Sat, 30 Aug 2014 00:42:56 +0530
Subject: [Python-ideas] Indentation,
	again: Problems with moving and combining code across editors.
Message-ID: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>

Hi!
I do appreciate that indentation in Python code shows intention, and I am
all for it, but problems do arise if code is copied from one place to
another, even from specialized Python editors.... and there is little you
can do to recover intentions/indentations lost in the process.
Besides, spaces become difficult to count and distinguish from tabs, adding
to the problem. (Why allow tabs at all?)

Why can't Python be made to accept the following indentation of code (in
*addition* to the current schemes)? There could be some flexibility of
choosing the indentation character. (Pipe was mentioned as an alternative
in another thread)

for i in range(10):
.if i>5:
..print 10-i
.else:
..print i
print "!"

If not in the interpreter for some reason, could it be "advised" as a mode
in a python oriented editor, or a "dictated" mechanism for
copy/paste/transfer of code?

Sorry for posing an elementary question.

Thanks and regards
MK-zedobject
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140830/a0ab4239/attachment.html>

From skip at pobox.com  Fri Aug 29 21:24:59 2014
From: skip at pobox.com (Skip Montanaro)
Date: Fri, 29 Aug 2014 14:24:59 -0500
Subject: [Python-ideas] Indentation,
 again: Problems with moving and combining code across editors.
In-Reply-To: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
References: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
Message-ID: <CANc-5Uwz+y-yqq_goDVWaCjs39J3cuA8pLh3WkdMsxQrw0yoOg@mail.gmail.com>

On Fri, Aug 29, 2014 at 2:12 PM, Milind Khadilkar <zedobject at gmail.com>
wrote:

> I do appreciate that indentation in Python code shows intention, and I am
> all for it, but problems do arise if code is copied from one place to
> another, even from specialized Python editors.... and there is little you
> can do to recover intentions/indentations lost in the process.


It's a known issue. I wouldn't necessarily call it a problem. I don't know
if there are good strategies for tackling that other than "don't do that."
If you find yourself copying around blocks of text which don't correspond
to whole functions or classes, it tells me you might have some refactoring
to do. Tools like pylint can help identify common chunks of code between
two or more files. I don't know what support there is in existing editors
or IDEs for code refactoring. There used to be Bicycle Repair Man, but I
think he's long gone:

http://bicyclerepair.sourceforge.net/

There is also a library called ROPE:

http://rope.sourceforge.net/

Never heard of it before. Don't know if it's currently supported (it at
least has Py3K support).

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140829/5f7eec35/attachment.html>

From breamoreboy at yahoo.co.uk  Fri Aug 29 21:40:02 2014
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Fri, 29 Aug 2014 20:40:02 +0100
Subject: [Python-ideas] Indentation,
 again: Problems with moving and combining code across editors.
In-Reply-To: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
References: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
Message-ID: <ltqkul$27s$1@ger.gmane.org>

On 29/08/2014 20:12, Milind Khadilkar wrote:
> Hi!
> I do appreciate that indentation in Python code shows intention, and I
> am all for it, but problems do arise if code is copied from one place to
> another, even from specialized Python editors.... and there is little
> you can do to recover intentions/indentations lost in the process.
> Besides, spaces become difficult to count and distinguish from tabs,
> adding to the problem. (Why allow tabs at all?)
>
> Why can't Python be made to accept the following indentation of code (in
> *addition* to the current schemes)? There could be some flexibility of
> choosing the indentation character. (Pipe was mentioned as an
> alternative in another thread)
>
> for i in range(10):
> .if i>5:
> ..print 10-i
> .else:
> ..print i
> print "!"
>
> If not in the interpreter for some reason, could it be "advised" as a
> mode in a python oriented editor, or a "dictated" mechanism for
> copy/paste/transfer of code?
>
> Sorry for posing an elementary question.
>
> Thanks and regards
> MK-zedobject
>

It strikes me that these "problems" has existed for 23 years and somehow 
people have survived.  Could it be a classic example of a bad workman 
always blames his tools?  See Skip Montanaro's reply for an explanation 
as to why.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From abarnert at yahoo.com  Fri Aug 29 23:04:24 2014
From: abarnert at yahoo.com (Andrew Barnert)
Date: Fri, 29 Aug 2014 14:04:24 -0700
Subject: [Python-ideas] Indentation,
	again: Problems with moving and combining code across editors.
In-Reply-To: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
References: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
Message-ID: <539EE1FB-7384-451F-8BE1-247FB892D28F@yahoo.com>

On Aug 29, 2014, at 12:12, Milind Khadilkar <zedobject at gmail.com> wrote:

> Hi!
> I do appreciate that indentation in Python code shows intention, and I am all for it, but problems do arise if code is copied from one place to another, even from specialized Python editors.... and there is little you can do to recover intentions/indentations lost in the process.

I copy and paste code between editors all the time--e.g., from emacs to the graphical editors on sites like Github or StackOverflow--and I can't remember the last time I've had this problem, except when copying someone else's code that mixed spaces and tabs (and even in that case, it more often helps me find existing invisible errors than it causes errors).

> Besides, spaces become difficult to count and distinguish from tabs, adding to the problem. (Why allow tabs at all?)

That one I could get behind, but it's been suggested and rejected enough times to have a PEP assigned to it.

> Why can't Python be made to accept the following indentation of code (in addition to the current schemes)? There could be some flexibility of choosing the indentation character. (Pipe was mentioned as an alternative in another thread)
> 
> for i in range(10):
> .if i>5:
> ..print 10-i
> .else:
> ..print i
> print "!"

The first problem is trying to come up with a syntax that isn't ambiguous to the parser or to a human. You haven't succeeded there:

    .2

Is that the int literal `2` indented, or the float literal `.2` in the left column?

Even the pipe character had this problem:

    |2

That pipe could mean the or operator if the previous line ended with a continuation character. But even discounting that, the tokenizer and any human experienced with Python will read it as the or operator and then have to follow some rule saying "an or operator at the start of a line that isn't part of a valid expression counts as a space". And that means guessing that it wasn't intended as an or operator, possibly turning what should have been identified as a simple SyntaxError into something baffling.

I suppose you could suggest bringing back ` for this purpose, but I doubt anyone will like that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140829/01a7c1e8/attachment-0001.html>

From stephen at xemacs.org  Sat Aug 30 07:32:07 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 30 Aug 2014 14:32:07 +0900
Subject: [Python-ideas]  Indentation,
	again: Problems with moving and combining code across editors.
In-Reply-To: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
References: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
Message-ID: <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp>

Milind Khadilkar writes:

 > Besides, spaces become difficult to count and distinguish from
 > tabs, adding to the problem.  (Why allow tabs at all?)

Working code should not be broken gratuitously.  Sure, it was an
unfortunate decision in the first place, but we can't change that
original decision now.

 > Why can't Python be made to accept the following indentation of
 > code (in *addition* to the current schemes)?

Because we *already have it*: the character is ' '.  If *you* follow
that rule, you won't have problems copy/pasting your own code into
well-behaved code.  If you're working with somebody else's code which
mixes tabs and spaces, *using an alternative character doesn't help*,
whether the "bad" code is source or target.







From zedobject at gmail.com  Sat Aug 30 08:19:41 2014
From: zedobject at gmail.com (Milind Khadilkar)
Date: Sat, 30 Aug 2014 11:49:41 +0530
Subject: [Python-ideas] Indentation,
 again: Problems with moving and combining code across editors.
In-Reply-To: <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
 <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CAPDr7zH1WqFo3ei8GwYXdENF-7gHLWZpuMQxqEgNQesSTTF6bQ@mail.gmail.com>

Thanks.
The problem is mostly when code from different people using different tools
needs to be combined, or sample code from the internet (not from the
well-behaved ones like github and stackoverflow) has to be tested and
understood.
Of course it is fun to discover the lost intentions of the original
programmer, but it is not practical in a day job.
QUOTE
Because we *already have it*: the character is ' '.
UNQUOTE
The problem with ' ' is that a sequence of ' 's can't easily be counted,
especially on feature rich editors.
A sequence of '.' or some visible character can be counted.

Of course, with the previous line being continued on the current line, '.'
could present problems, as Andrew B. points out, and they can be messy.
Could the solution lie in providing a "transfer" or "import-export" mode in
editors where the indentation is given in terms of a chosen special
character which is converted to spaces/tabs?
Thanks again

MK-Zedobject

On Sat, Aug 30, 2014 at 11:02 AM, Stephen J. Turnbull <stephen at xemacs.org>
wrote:

> Milind Khadilkar writes:
>
>  > Besides, spaces become difficult to count and distinguish from
>  > tabs, adding to the problem.  (Why allow tabs at all?)
>
> Working code should not be broken gratuitously.  Sure, it was an
> unfortunate decision in the first place, but we can't change that
> original decision now.
>
>  > Why can't Python be made to accept the following indentation of
>  > code (in *addition* to the current schemes)?
>
> Because we *already have it*: the character is ' '.  If *you* follow
> that rule, you won't have problems copy/pasting your own code into
> well-behaved code.  If you're working with somebody else's code which
> mixes tabs and spaces, *using an alternative character doesn't help*,
> whether the "bad" code is source or target.
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140830/78326ba5/attachment.html>

From ben+python at benfinney.id.au  Sat Aug 30 10:02:40 2014
From: ben+python at benfinney.id.au (Ben Finney)
Date: Sat, 30 Aug 2014 18:02:40 +1000
Subject: [Python-ideas] Indentation,
	again: Problems with moving and combining code across editors.
References: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
 <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAPDr7zH1WqFo3ei8GwYXdENF-7gHLWZpuMQxqEgNQesSTTF6bQ@mail.gmail.com>
Message-ID: <85ha0uxv4v.fsf@benfinney.id.au>

Milind Khadilkar <zedobject at gmail.com>
writes:

> The problem with ' ' is that a sequence of ' 's can't easily be
> counted, especially on feature rich editors.

You have a ?feature rich? editor which is unable to count a sequence of
characters? I don't think that qualifies as ?feature rich? for purposes
of programming.

Also, your proposal then only seems to be valid if you have an editor
which:

* is capable of the highly specific task of ?convert Python code between
  ? ? for indentation and ?.? for indentation?; but

* is so feature-poor that it *is not* capable of the generally-useful
  task ?count a sequence of characters?.

I think that's a text editor so bizarre we can discount it as irrelevant
until someone is demonstrated to be using it seriously and can't be
convinced to use a better editor.

-- 
 \     ?If you ever catch on fire, try to avoid seeing yourself in the |
  `\        mirror, because I bet that's what REALLY throws you into a |
_o__)                                             panic.? ?Jack Handey |
Ben Finney


From steve at pearwood.info  Sat Aug 30 11:55:07 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 30 Aug 2014 19:55:07 +1000
Subject: [Python-ideas] Indentation,
	again: Problems with moving and combining code across editors.
In-Reply-To: <CAPDr7zH1WqFo3ei8GwYXdENF-7gHLWZpuMQxqEgNQesSTTF6bQ@mail.gmail.com>
References: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
 <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAPDr7zH1WqFo3ei8GwYXdENF-7gHLWZpuMQxqEgNQesSTTF6bQ@mail.gmail.com>
Message-ID: <20140830095507.GN23931@ando>

On Sat, Aug 30, 2014 at 11:49:41AM +0530, Milind Khadilkar wrote:
> Thanks.
> The problem is mostly when code from different people using different tools
> needs to be combined, or sample code from the internet (not from the
> well-behaved ones like github and stackoverflow) has to be tested and
> understood.
> Of course it is fun to discover the lost intentions of the original
> programmer, but it is not practical in a day job.

Are you suggesting that indentation is the *only* reason to test and 
understand arbitrary code you copy and paste from the Internet?

I don't that's a reasonable argument to make. Whether it is your day 
job, or just a hobby, you should have some understanding of what the 
code is supposed to do before pasting it into your own work. And there 
is so much more than just indentation that you have to be concerned 
about: the imports, the variables it uses, which functions it calls, can 
the code be trusted. Indentation is the least of these.

I accept that there are occasional situations where we might want to 
copy code from a website or email, and the indentation has been lost. 
But that ought to be rare. In my personal experience, I've needed to 
reconstruct the indentation from scratch perhaps three or four times in 
the last ten years. Even if I'm off by a factor of ten, let's call it 
three or four times a year. That's still not very important. Worrying 
that this is "not practical" in a day job seems to be worrying over 
nothing.


> QUOTE
> Because we *already have it*: the character is ' '.
> UNQUOTE
> The problem with ' ' is that a sequence of ' 's can't easily be counted,
> especially on feature rich editors.
> A sequence of '.' or some visible character can be counted.

I don't think that's a reasonable argument. I trust you're not using 
Notepad, are you? In a bare-bones editor, it's hard to count *any* long 
sequence of characters. Quick, how many dots before the X?

   ..................X

If you are a professional programmer, or even just a serious amateur, 
you should be using professional tools. With professional tools, you 
rarely need to *count the spaces*. You just need to notice changes to 
the indent level:

            one level
                another level
        different level

You don't need to care precisely how many spaces (or tabs) are there, 
you just need to ensure things line up. And any decent programming 
editor will give you the tools to increase or decrease indentation over 
a block of lines, without caring about the specific number of spaces or 
counting exactly how many spaces are needed. That includes a wide range 
of IDEs and editors of all sorts of power, on many different platforms: 
Notepad++ or geany on Windows, kate or kwrite on KDE, vim and emacs for 
pretty much any Unix or Linux system, to mention only a few.

Which brings us to another problem with your suggestion. Professional 
editors already know about indentation with tabs, and spaces, but I 
don't know any editor which gives you the ability to indent with 
arbitrary characters. (Although I daresay somebody would soon write a 
macro for Emacs to do that.) Which means you are swapping from a system 
where professional-quality programming tools can do the counting for 
you, to a situation where you actually do need to count the dots 
yourself. And that's a big step backwards.

I don't think this change is practical, or useful. And it's ugly.


-- 
Steven

From solipsis at pitrou.net  Sat Aug 30 13:07:29 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 30 Aug 2014 13:07:29 +0200
Subject: [Python-ideas] Subprocess: Add an encoding argument
References: <CACac1F8Fpn2VwEiBKKbzUgc-+3unOhs4e9j9RGg_+9Lu=HyeXw@mail.gmail.com>
 <CADiSq7fnUuT91wyQskz+aFD23_8noUwvRDmZbebb9ZOLgc5tQw@mail.gmail.com>
Message-ID: <20140830130729.6e8b67ec@fsol>

On Fri, 29 Aug 2014 18:39:35 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> This actually gets a little messy once you start digging into it, as you
> actually have up to 3 streams to deal with (stdin, stdout, stderr), and may
> want to set the error handler in addition to the encoding.

At this point, I'd suggest creating a clean, separate TextPopen class
(or subclass) without any legacy argument baggage.

As for per-stream settings, we could allow passing each of *encoding*
and *errors* in two forms:
- as a string, in which case it applies to all three pipes
- as a dict, in which case it is looked up for each of the "stdin",
  "stdout", "stderr" pipes (if configured as pipes)

We could then deprecate the bogus-named "universal_newlines" in the
main Popen class.

Regards

Antoine.



From p.f.moore at gmail.com  Sat Aug 30 14:05:02 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Sat, 30 Aug 2014 13:05:02 +0100
Subject: [Python-ideas] Subprocess: Add an encoding argument
In-Reply-To: <20140830130729.6e8b67ec@fsol>
References: <CACac1F8Fpn2VwEiBKKbzUgc-+3unOhs4e9j9RGg_+9Lu=HyeXw@mail.gmail.com>
 <CADiSq7fnUuT91wyQskz+aFD23_8noUwvRDmZbebb9ZOLgc5tQw@mail.gmail.com>
 <20140830130729.6e8b67ec@fsol>
Message-ID: <CACac1F9k5qsgzJ=z5VvnZxFg5AK7pHKQaB-uF8kqQP2rhxykiA@mail.gmail.com>

On 30 August 2014 12:07, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Fri, 29 Aug 2014 18:39:35 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> This actually gets a little messy once you start digging into it, as you
>> actually have up to 3 streams to deal with (stdin, stdout, stderr), and may
>> want to set the error handler in addition to the encoding.
>
> At this point, I'd suggest creating a clean, separate TextPopen class
> (or subclass) without any legacy argument baggage.
>
> As for per-stream settings, we could allow passing each of *encoding*
> and *errors* in two forms:
> - as a string, in which case it applies to all three pipes
> - as a dict, in which case it is looked up for each of the "stdin",
>   "stdout", "stderr" pipes (if configured as pipes)
>
> We could then deprecate the bogus-named "universal_newlines" in the
> main Popen class.

Sounds reasonable. I'll look into that (no promises on timescales :-))
In practice, I doubt we'd need per-stram encodings particularly often,
so I like the idea of *not* clutteringthe API to cope with them. I'm
curious, by the way - what arguments do you consider as "legacy
baggage" (a lot of them seem to me to be OS-specific and/or
specialised rather than legacy).

In practice, we'd probably need to do something about the utility
functions like check_output and communicate as well.
Paul

From solipsis at pitrou.net  Sat Aug 30 14:10:52 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 30 Aug 2014 14:10:52 +0200
Subject: [Python-ideas] Subprocess: Add an encoding argument
References: <CACac1F8Fpn2VwEiBKKbzUgc-+3unOhs4e9j9RGg_+9Lu=HyeXw@mail.gmail.com>
 <CADiSq7fnUuT91wyQskz+aFD23_8noUwvRDmZbebb9ZOLgc5tQw@mail.gmail.com>
 <20140830130729.6e8b67ec@fsol>
 <CACac1F9k5qsgzJ=z5VvnZxFg5AK7pHKQaB-uF8kqQP2rhxykiA@mail.gmail.com>
Message-ID: <20140830141052.35226537@fsol>

On Sat, 30 Aug 2014 13:05:02 +0100
Paul Moore <p.f.moore at gmail.com> wrote:
> 
> Sounds reasonable. I'll look into that (no promises on timescales :-))
> In practice, I doubt we'd need per-stram encodings particularly often,
> so I like the idea of *not* clutteringthe API to cope with them. I'm
> curious, by the way - what arguments do you consider as "legacy
> baggage" (a lot of them seem to me to be OS-specific and/or
> specialised rather than legacy).

I was thinking mostly about universal_newlines. Perhaps preexec_fn
applies too, since it's dangerous (read: unstable).

Regards

Antoine.



From moritz.beber at gmail.com  Sat Aug 30 14:37:10 2014
From: moritz.beber at gmail.com (Moritz Beber)
Date: Sat, 30 Aug 2014 14:37:10 +0200
Subject: [Python-ideas] Subprocess: Add an encoding argument
In-Reply-To: <20140830141052.35226537@fsol>
References: <CACac1F8Fpn2VwEiBKKbzUgc-+3unOhs4e9j9RGg_+9Lu=HyeXw@mail.gmail.com>
 <CADiSq7fnUuT91wyQskz+aFD23_8noUwvRDmZbebb9ZOLgc5tQw@mail.gmail.com>
 <20140830130729.6e8b67ec@fsol>
 <CACac1F9k5qsgzJ=z5VvnZxFg5AK7pHKQaB-uF8kqQP2rhxykiA@mail.gmail.com>
 <20140830141052.35226537@fsol>
Message-ID: <CAFOFTpQ2dx=E8BLHu_ystD-WiwdW7eiqJVGbUPm-541Q4wz85A@mail.gmail.com>

Hi,


On Sat, Aug 30, 2014 at 2:10 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Sat, 30 Aug 2014 13:05:02 +0100
> Paul Moore <p.f.moore at gmail.com> wrote:
> >
> > Sounds reasonable. I'll look into that (no promises on timescales :-))
> > In practice, I doubt we'd need per-stram encodings particularly often,
> > so I like the idea of *not* clutteringthe API to cope with them. I'm
> > curious, by the way - what arguments do you consider as "legacy
> > baggage" (a lot of them seem to me to be OS-specific and/or
> > specialised rather than legacy).
>
> I was thinking mostly about universal_newlines. Perhaps preexec_fn
> applies too, since it's dangerous (read: unstable).
>
>
>
preexec_fn is important though if you want to run something with different
uid and gid from a sudo script, for example.

Cheers,
Moritz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140830/9a53aa1c/attachment.html>

From ron3200 at gmail.com  Sat Aug 30 17:36:56 2014
From: ron3200 at gmail.com (Ron Adam)
Date: Sat, 30 Aug 2014 10:36:56 -0500
Subject: [Python-ideas] Indentation,
 again: Problems with moving and combining code across editors.
In-Reply-To: <CAPDr7zH1WqFo3ei8GwYXdENF-7gHLWZpuMQxqEgNQesSTTF6bQ@mail.gmail.com>
References: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
 <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAPDr7zH1WqFo3ei8GwYXdENF-7gHLWZpuMQxqEgNQesSTTF6bQ@mail.gmail.com>
Message-ID: <ltsr2q$v56$1@ger.gmane.org>



On 08/30/2014 01:19 AM, Milind Khadilkar wrote:
> Thanks.
> The problem is mostly when code from different people using different tools
> needs to be combined, or sample code from the internet (not from the
> well-behaved ones like github and stackoverflow) has to be tested and
> understood.
> Of course it is fun to discover the lost intentions of the original
> programmer, but it is not practical in a day job.
> QUOTE
> Because we *already have it*: the character is ' '.
> UNQUOTE
> The problem with ' ' is that a sequence of ' 's can't easily be counted,
> especially on feature rich editors.
> A sequence of '.' or some visible character can be counted.
>
> Of course, with the previous line being continued on the current line, '.'
> could present problems, as Andrew B. points out, and they can be messy.
> Could the solution lie in providing a "transfer" or "import-export" mode in
> editors where the indentation is given in terms of a chosen special
> character which is converted to spaces/tabs?
> Thanks again

Unfortunately there probably isn't much you can do about uses of tabs or 
spaces outside your immediate team.  If they are all are part of your team, 
probably the best approach is to agree to set your editors to hi-light tabs 
in an obvious way, and also set them to use spaces in place of tabs.  Over 
time you should see less uses of tabs and it won't be as much of an issue.

As for cutting and pasting lines, you will still probably need to reindent 
anyway.  For that I use the indent, dedent capability of the editor if it 
has one, or a different editor that has that, if it doesn't.  It's one of 
my must have requirements.

I don't think an explicit indent character would help much, but possibly a 
relative indent marker might be useful for putting multi-line code on a 
single line.  Currently we have the ";" which only keeps the current 
indentation.  It probably wouldn't be that hard to add ";+", and ";-", 
symbols (or something equivalent), but I think it would need it's own pep 
and discussion if anyone cares enough about it.  Probably the biggest issue 
is "is it needed?", so it would need some convincing examples of how it 
would be useful.

-Ron


From tjreedy at udel.edu  Sat Aug 30 21:58:00 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 30 Aug 2014 15:58:00 -0400
Subject: [Python-ideas] Indentation,
 again: Problems with moving and combining code across editors.
In-Reply-To: <ltsr2q$v56$1@ger.gmane.org>
References: <CAPDr7zHU37KbTmaQLM-Xu1jnVnMxCdOB=bKa2xj1Tmzv7vx1-Q@mail.gmail.com>
 <87y4u6a6g8.fsf@uwakimon.sk.tsukuba.ac.jp>
 <CAPDr7zH1WqFo3ei8GwYXdENF-7gHLWZpuMQxqEgNQesSTTF6bQ@mail.gmail.com>
 <ltsr2q$v56$1@ger.gmane.org>
Message-ID: <lttaci$nt1$1@ger.gmane.org>

On 8/30/2014 11:36 AM, Ron Adam wrote:

> Unfortunately there probably isn't much you can do about uses of tabs or
> spaces outside your immediate team.

Except to convert to local standard.

> If they are all are part of your
> team, probably the best approach is to agree to set your editors to
> hi-light tabs in an obvious way,

Hmmm. Idle does not have an option to do that, but it would be useful.

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Sun Aug 31 01:30:32 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 31 Aug 2014 09:30:32 +1000
Subject: [Python-ideas] Subprocess: Add an encoding argument
In-Reply-To: <20140830130729.6e8b67ec@fsol>
References: <CACac1F8Fpn2VwEiBKKbzUgc-+3unOhs4e9j9RGg_+9Lu=HyeXw@mail.gmail.com>
 <CADiSq7fnUuT91wyQskz+aFD23_8noUwvRDmZbebb9ZOLgc5tQw@mail.gmail.com>
 <20140830130729.6e8b67ec@fsol>
Message-ID: <CADiSq7cxWrhXggp4Zz4GSd_rx_C1jcxzX59T2JEW4HQ+AsGo3A@mail.gmail.com>

On 30 Aug 2014 21:08, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> On Fri, 29 Aug 2014 18:39:35 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> >
> > This actually gets a little messy once you start digging into it, as you
> > actually have up to 3 streams to deal with (stdin, stdout, stderr), and
may
> > want to set the error handler in addition to the encoding.
>
> At this point, I'd suggest creating a clean, separate TextPopen class
> (or subclass) without any legacy argument baggage.
>
> As for per-stream settings, we could allow passing each of *encoding*
> and *errors* in two forms:
> - as a string, in which case it applies to all three pipes
> - as a dict, in which case it is looked up for each of the "stdin",
>   "stdout", "stderr" pipes (if configured as pipes)
>
> We could then deprecate the bogus-named "universal_newlines" in the
> main Popen class.

That sounds like a plausible plan to me - splitting the API like that also
reflects what was needed to get IO working sensibly in the first place.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20140831/4015949e/attachment.html>

From jsbfox at gmail.com  Sun Aug 31 18:04:30 2014
From: jsbfox at gmail.com (Thomas Allen)
Date: Sun, 31 Aug 2014 20:04:30 +0400
Subject: [Python-ideas] List available tests without running them
Message-ID: <CAMPw9HQ0dR721TZqO5yWRR4zr0o1Hs0jH6rgFu4PZK-XA75bNg@mail.gmail.com>

I'd like to propose the ability to list all available unit tests
without executing them by providing a specific command line option
(-l/--list for instance).

   python3 -m unittest discover -ls .

would list all tests in current directory.

Motivation & possible implementation:
http://stackoverflow.com/q/24478727/2301450