<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Jan 21, 2016 at 10:14 AM, Agustín Herranz Cecilia <span dir="ltr"><<a href="mailto:agustin.herranz@gmail.com" target="_blank">agustin.herranz@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div><div class="h5">
<div>El 2016/01/21 a las 1:11, Guido van
Rossum escribió:<br>
</div>[...]<br>
</div></div><span class=""><blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>
> - As this is intended to gradual type python2 code
to port it to python 3 I think it's convenient to add
some sort of import that only be used for type checking,
and be only imported by the type analyzer, not the
runtime. This could be achieve by prepending "#type: "
to the normal import statement, something like:<br>
> # type: import module<br>
> # type: from package import module<br>
<br>
</span>That sounds like a bad idea. If the typing module
shadows some global, you won't get any errors, but your
code will be misleading to a reader (and even worse if you
from package.module import t). If the cost of the import
is too high for Python 2, surely it's also too high for
Python 3. And what other reason do you have for skipping
it?<span><br>
</span></blockquote>
<div><br>
</div>
<div>Exactly. Even though (when using Python 2) all type
annotations are in comments, you still must write real
imports. (This causes minor annoyances with linters that
warn about unused imports, but there are ways to teach
them.)<br>
</div>
</div>
</div>
</div>
</blockquote></span>
This type comment 'imports' are not intended to shadow the current
namespace, are intended to tell the analyzer where it can find those
types present in the type comments that are not in the current
namespace without import in it. This surely complicates the analyzer
task but helps avoid namespace pollution and also saves memory on
runtime.<br>
<br>
The typical case I've found is when using a third party library
(that don't have type information) and you creates objects with a
factory. The class of the objects is no needed anywhere so it's not
imported in the current namespace, but it's needed only for type
analysis and autocomplete.<span class=""><br></span></div></blockquote><div><br></div><div>You're describing a case I have also encountered: we have a module with a function foo<br><br></div><div># foo_mod.py<br></div><div>def foo(a):<br></div><div> return a.bar()<br><br></div><div>and the intention is that a is an instance of a class A defined in another module, which is not imported.<br><br></div><div>If we add annotations we have to add an import<br><br></div><div>from a_mod import A<br><br></div><div>def foo(a: A) -> str:<br></div><div> return a.bar()<br></div><div> <br></div><div>But the code that calls foo() is already importing A from a_mod somewhere, so there's not really any time wasted -- the import is just done at a different time.<br><br></div><div>At least, that's the theory.<br><br></div><div>In practice, indeed there are some unpleasant cases. For example, adding the explicit import might create an import cycle, and A may not yet be defined when foo_mod is loaded. We can't use the usual subterfuge, since we really need the definition of A:<br><br></div><div>import a_mod<br><br></div><div>def foo(a: a_mod.A) -> str:<br></div><div> return a.bar()<br><br></div><div>This will still fail if a_mod hasn't defined A yet because we reference a_mod.A at load time (annotations are evaluated when the function definition is executed).<br><br></div><div>So we end up with this:<br><br></div><div>import a_mod<br><br></div><div>def foo(a: 'a_mod.A') -> str:<br></div><div> return a.bar()<br><br></div><div>This is both hard to read and probably wastes a lot of developer time figuring out they have to do this.<br><br></div><div>And there are other issues, e.g. some folks have tricks to work around their start-up time by importing modules late (e.g. do the import inside the function that needs that module).<br></div><div><br></div><div>In mypy there's another hack possible: it doesn't care if an import is inside "if False". So you can write:<br><br></div><div>if False:<br></div><div> from a_mod import A<br><br></div><div>def foo(a: 'A') -> str:<br></div><div> return a.bar()<br><br></div><div>You still have to quote 'A' because A isn't actually defined at run time, but it's the best we can do. When using type comments you can skip the quotes:<br><br></div><div>if False:<br></div><div> from a_mod import A<br><br></div><div>def foo(a):<br></div><div> # type: (A) -> str<br></div><div> return a.bar()<br><br></div><div>All of this is unpleasant but not unbearable -- the big constraint here is that we don't want to add extra syntax (beyond PEP 3107, i.e. function annotations), so that we can use mypy for Python 3.2 and up. And with the type comments we even support 2.7.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span class="">
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>
> - Also there must be addressed how it work on a
python2 to python3 environment as there are types with
the same name, str for example, that works differently
on each python version. If the code is for only one
version uses the type names of that version.<br>
<br>
</span>That's the same problem that exists at runtime, and
people (and tools) already know how to deal with it: use
bytes when you mean bytes, unicode when you mean unicode,
and str when you mean whatever is "native" to the version
you're running under and are willing to deal with it. So
now you just have to do the same thing in type hints that
you're already doing in constructors, isinstance checks,
etc.<br>
</blockquote>
<div><br>
</div>
<div>This is actually still a real problem. But it has no
bearing on the choice of syntax for annotations in Python
2 or straddling code.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br></span>
Yes, this is no related with the choice of syntax for annotations
directly. This is intended to help in the process of porting python2
code to python3, and it's outside of the PEP scope but related to
the original problem. What I have in mind is some type aliases so
you could annotate a version specific type to avoid ambiguousness on
code that it's used on different versions. At the end what I
originally try to said is that it's good to have a convention way to
name this type aliases.<br></div></blockquote><div><br></div><div>Yes, this is a useful thing to discuss.<br><br></div><div>Maybe we can standardize on the types defined by the 'six' package, which is commonly used for 2-3 straddling code:<br><br></div><div>six.text_type (unicode in PY2, str in PY3)<br></div><div>six.binary_type (str in PY2, bytes in PY3)<br><br></div><div>Actually for the latter we might as well use bytes.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">
This are intended to use during the process of porting, to help some
automated tools, in a period of transition between versions. It's a
way to tell the analyzer that a type have a behavior, perhaps
different, than the same type on the running python version.<br>
<br>
For example. You start with some working python2 code that you want
to still be working. A code analysis tool can infer the types and
annotate the code. Also can check which parts are py2/py3 compatible
and which not, and mark those types with the mentioned type aliases.
With this, and test suites, it could be calculated how much code is
needed to be ported. Refactor to adapt the code to python3
maintaining code to still run on python2 (it could be marked for
automate deletion), and when it's done, drop all the python2 code..<span class=""><br></span></div></blockquote><div><br></div><div>Yes, that's the kind of process we're trying to develop. It's still early days though -- people have gotten different workflows already using six and tests and the notion of straddling code, __future__ imports, and PyPI backports of some PY3 stdlib packages (e.g. contextlib2).<br><br></div><div>There's also a healthy set of tools that converts PY2 code to straddling code, approximately (e.g. futurize and modernize). What's missing (as you point out) is tools that help automating a larger part of the conversion once PY2 code has been annotated.<br><br></div><div>But first we need to agree on how to annotate PY2 code.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span class="">
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Of course many people use libraries like six to help them
deal with this, which means that those libraries have to
be type-hinted appropriately for both versions (maybe
using different stubs for py2 and py3, with the right one
selected at pip install time?), but if that's taken care
of, user code should just work.<br>
</blockquote>
<div><br>
</div>
<div>Yeah, we could use help. There are some very
rudimentary stubs for a few things defined by six (<a href="https://github.com/python/typeshed/tree/master/third_party/3/six" target="_blank"></a><a href="https://github.com/python/typeshed/tree/master/third_party/3/six" target="_blank">https://github.com/python/typeshed/tree/master/third_party/3/six</a>,
<a href="https://github.com/python/typeshed/tree/master/third_party/2.7/six" target="_blank">https://github.com/python/typeshed/tree/master/third_party/2.7/six</a>)
but we need more. There's a PR but it's of bewildering
size (<a href="https://github.com/python/typeshed/pull/21" target="_blank">https://github.com/python/typeshed/pull/21</a>).<br>
<br>
</div>
</div>
</div>
</div>
</blockquote></span>
I think the process of porting it's different from the process of
adapting code to work on python 2/3. Code with bytes, unicode, &
str(don't mind) are not python2 code nor python3. Lot's of libraries
that are 2/3 compatibles are just python2 code minimally adapted to
run on python3 with six, and still be developed with a python2
style. When the time of drop python2 arrives the refactor needed
will be huge. There is also an article that recently claims "Stop
writing code that break on Python 4" and show code that treats
python3 as the special case..<span class=""><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>PS. I have a hard time following the rest of Agustin's
comments. The comment-based syntax I proposed for Python
2.7 does support exactly the same functionality as the
official PEP 484 syntax; the only thing it doesn't allow
is selectively leaving out types for some arguments -- you
must use 'Any' to fill those positions. It's not a problem
in practice, and it doesn't reduce functionality (omitted
argument types are assumed to be Any in PEP 484 too). I
should also remark that mypy supports the comment-based
syntax in Python 2 mode as well as in Python 3 mode; but
when writing Python 3 only code, the non-comment version
is strongly preferred. (We plan to eventually produce a
tool that converts the comments to standard PEP 484
syntax).<br>
</div>
<div> </div>
</div>
-- <br>
<div>--Guido van Rossum (<a href="http://python.org/%7Eguido" target="_blank">python.org/~guido</a>)</div>
</div>
</div>
</blockquote>
<br></span>
My original point is that if comment-based function
annotations are going to be added, add it to python 3 too, no only
for the special case of "Python 2.7 and straddling code", even
though, on python 3, type annotations are preferred.<br></div></blockquote><div><br></div><div>The text I added to the end of PEP 484 already says so:<br><br>"""<br>- Tools that support this syntax should support it regardless of the<br> Python version being checked. This is necessary in order to support<br> code that straddles Python 2 and Python 3.<br>"""<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">
<br>
I think that have the alternative to define types of a function as a
type comment is a good thing because annotations could become a
mesh, specially with complex types and default parameters, and I
don't fell that the optional part of gradual typing must include
readability.<br>
Some examples of my own code:<br>
<br>
class Field:<br>
def __init__(self, name: str,<br>
extract: Callable[[str], str],<br>
validate: Callable[[str], bool]=bool_test,<br>
transform: Callable[[str], Any]=identity) ->
'Field':<br>
<br>
class RepeatableField:<br>
def __init__(self,<br>
extract: Callable[[str], str],<br>
size: int,<br>
fields: List[Field],<br>
index_label: str,<br>
index_transform: Callable[[int], str]=lambda x:
str(x)) -> 'RepeatableField':<br>
<br>
def filter_by(field_gen: Iterable[Dict[str, Any]], **kwargs) ->
Generator[Dict[str, Any], Any, Any]:<br>
<br>
<br>
So, for define a comment-based function
annotation it should be accepted two kind of syntax: <br>
- one 'explicit' marking the type of the function according to the
PEP484 syntax:<br>
<pre> def embezzle(self, account, funds=1000000, *fake_receipts):
# type: Callable[[str, int, *str], None]
"""Embezzle funds from account using fake receipts."""
<code goes here>
</pre>
like if was a normal type comment:<br>
<pre> embezzle = get_embezzle_function() # type: Callable[[str, int, *str], None]
</pre>
- and another one that 'implicitly' define the type of the function
as Callable:<br>
<pre> def embezzle(self, account, funds=1000000, *fake_receipts):
# type: (str, int, *str) -> None
"""Embezzle funds from account using fake receipts."""
<code goes here></pre>
Both ways are easily translated back and forth into python3
annotations.<br></div></blockquote><div><br></div><div>I don't see what adding support for<br><br></div><div><div style="margin-left:40px"><span style="font-family:monospace,monospace"># type: Callable[[str, int, *str], None]</span><br></div><br></div><div>adds. It's more verbose, and when the 'implicit' notation is used, the type checker already knows that embezzle is a function with that signature. You can already do this (except for the *str part):<br><br><div style="margin-left:40px"><span style="font-family:monospace,monospace">from typing import Callable</span><br><span style="font-family:monospace,monospace"></span><br><span style="font-family:monospace,monospace">def embezzle(account, funds=1000000):</span><br><span style="font-family:monospace,monospace"> # type: (str, int) -> None</span><br><span style="font-family:monospace,monospace"> """Embezzle funds from account using fake receipts."""</span><br><span style="font-family:monospace,monospace"> pass</span><br><span style="font-family:monospace,monospace"></span><br><span style="font-family:monospace,monospace">f = None # type: Callable[[str, int], None]</span><br><span style="font-family:monospace,monospace"></span><br><span style="font-family:monospace,monospace">f = embezzle</span><br><span style="font-family:monospace,monospace"></span><br><span style="font-family:monospace,monospace">f('a', 42)</span><br></div><br></div><div>However, note that no matter which notation you use, there's no way in PEP 484 to write the type of the original embezzle() function object using Callable -- Callable does not have support for varargs like *fake_receipts. If you want that the best place to bring it up is the typehinting tracker (<a href="https://github.com/ambv/typehinting/issues">https://github.com/ambv/typehinting/issues</a>). But it's going to be a tough nut to crack, and the majority of places where Callable is needed (mostly higher-order functions like filter/map) don't need it -- their function arguments have purely positional arguments.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">
Also, comment-based function
annotations easily goes over one line's characters, so it should be
define which syntax is used to break the line. As it said on
<a href="https://github.com/JukkaL/mypy/issues/1102" target="_blank">https://github.com/JukkaL/mypy/issues/1102</a><br>
<br>
Those things should be on a PEP as a standard way to implement this,
not only for mypy, also for other tools. <br>
Accept comment-based function
annotations in python3 is good for migration python 2/3 code as it
helps on refactor and use (better autocomplete), but makes it a
python2 feature and not python3 increase the gap between versions.<br></div></blockquote><div><br></div><div>Consider it done. The time machine strikes again. :-) <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000">
<br>
<br>
Hope I expressed better, if not, sorry about that.<span class=""><font color="#888888"><br></font></span></div></blockquote><div><br></div><div>It's perfectly fine this time!<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span class=""><font color="#888888">
<br>
Agustín Herranz<br>
<br>
</font></span></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">--Guido van Rossum (<a href="http://python.org/~guido" target="_blank">python.org/~guido</a>)</div>
</div></div>