<div dir="ltr"><div>Following up some of the discussions about the problems of adding keywords and Guido's proposal of making tokenization context-dependent, I wanted to propose an alternate way to go around the problem.</div><div><br></div><div>My proposal essentially boils down to:</div><div><ol><li>The character "$" can be used as a prefix of identifiers. formally, <pre style="overflow-x:auto;overflow-y:hidden;padding:5px;background-color:rgb(238,255,204);color:rgb(51,51,51);line-height:18.528px;border:1px solid rgb(170,204,153);font-family:monospace,sans-serif;font-size:15.44px;border-radius:3px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><strong id="gmail-grammar-token-identifier">identifier </strong> ::= ["$"] <a class="gmail-reference gmail-internal" href="https://docs.python.org/3/reference/lexical_analysis.html#grammar-token-xid_start" style="color:rgb(99,99,187);text-decoration:none"><code class="gmail-xref gmail-docutils gmail-literal gmail-notranslate" style="background-color:transparent;padding:0px 1px;font-size:14.8996px;font-family:monospace,sans-serif;font-weight:normal;border-radius:3px"><span class="gmail-pre" style="hyphens: none;">xid_start</span></code></a> <a class="gmail-reference gmail-internal" href="https://docs.python.org/3/reference/lexical_analysis.html#grammar-token-xid_continue" style="color:rgb(99,99,187);text-decoration:none"><code class="gmail-xref gmail-docutils gmail-literal gmail-notranslate" style="background-color:transparent;padding:0px 1px;font-size:14.8996px;font-family:monospace,sans-serif;font-weight:normal;border-radius:3px"><span class="gmail-pre" style="hyphens: none;">xid_continue</span></code></a>*</pre></li><li>The "$" character is not part of the name. So the program "foo=3;print($foo)" prints 3. So does the program "$foo=3; print(foo)". Both set an entry to globals["foo"] and keep globals["$foo"] unset. </li><li>if "$" appears in a token, it's always an identifier. So "$with", "$if", "$return" are all identifiers.</li></ol><div>If you overcome the "yikes, that looks like awk/bash/perl/php, and I don't like those", and consider it as an escape for "unusual"/"deprecation" situations, I think it's not a bad chose, and allows to a simple solutions to many problems that have been in discussion recently and not so recently. [examples below]</div></div><div><br></div><div>For me the benefits of this approach are:</div><div><ul><li>It's very simple to explain how to use and its semantics</li><li>It (seems to me it) should be easy to explain to a python apprentice what a "$" means in code they read on a book/blogpost/manual</li><li>It's very easy to implement, minimal changes in the tokenizer</li><li>It's also easy to implement/integrate in other tools (editors with syntax highlighters, code formatters, etc)</li><li>It is easy to see that it's 100% backwards compatible (I understand that "$" has never been used in python before)</li><li>It is relatively unsurprising in the sense that other languages are already using $ to label names (there may be some point of confusion to people coming from javascript where "$" is a valid character in names and is not ignored). </li><li>It gives python devs and users a clear, easy and universal upgrade path when keywords are added (language designers: Add a __future__ import to enable keyword in python N+1, add warnings to change kw --> $kw in python N+2, and then turn it on by default in python N+3... ; developers: add the import when they want to upgrade , and fix their code with a search&replace when adding the import or after getting a warning).</li><li>It allows you to use new features even if some libraries were written for older python versions, depending the deprecation period (this could be improved with sth I'll write in another email, but that's the topic for another proposal)</li><li>When clashes occur, which they always do, there's one obvious way to disambiguate (see today the "class_" argument for gettext.translation, the "lambd" argument for random.expovariate, the "class_" filter in libraries like pyquery for CSS class, functions like pyquery, sqlalchemy.sql.operators.as_ , etc. Not counting all the "cls" argument to every classmethod ever)</li><li>If we're worried about over proliferation of "$" in code, I'm quite sure given past experience that just a notice in PEP 8 of "only with $ in names to prevent ambiguity" should be more than enough for the community</li></ul></div><div><br></div><div>What are the drawbacks that you find in this?</div><div>Best,</div><div> Daniel</div><div><br></div><div>[The rest of this post is just examples]</div><div><br></div><div>Example 1:</div><div><br></div><div>Python 3.92 has just added a future import that makes "given" a keyword. Then you can do:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font face="monospace, monospace"># This works because we have no future import</font></div></div><div><div><font face="monospace, monospace"><b>from</b> hypothesis <b>import</b> given, strategies <b>as</b> st</font></div></div><div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">@given(st.integers())</font></div></div><div><font face="monospace, monospace"><b>def</b> foo(i):</font></div><div><font face="monospace, monospace"> x = f(i)**2 + f(i)**3</font></div><div><font face="monospace, monospace"> ....</font></div></blockquote><div><br></div><div>if you want to use the new feature (or upgraded to python 3.93 and started receiving warnings) you can then change it to:</div><div><blockquote style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;margin:0px 0px 0px 40px;border:none;padding:0px"><div style="font-style:normal"><div style="font-weight:400"><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace"><b>from</b> __future__ <b>import</b> given_expression</font></div></div><div style="font-style:normal;font-weight:400"><font face="monospace, monospace"><br></font></div><div style="font-style:normal"><div><font face="monospace, monospace"><b>from</b> hypothesis <b>import</b> $given, strategies <b>as</b> st</font></div></div><div style="font-weight:400"><div style="font-style:normal"><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">@$given(st.integers()) <i># If you forget the $ you get a SyntaxError</i></font></div></div><div style="font-style:normal"><font face="monospace, monospace"><b>def</b> foo(i):</font></div><div style="font-style:normal"><font face="monospace, monospace"> x = z**2 + z**3 <b>given</b> z = f(i)</font></div><div style="font-style:normal;font-weight:400"><font face="monospace, monospace"> ....</font></div></blockquote><div><br></div>And also you could do:</div><div><br></div><div><blockquote style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font face="monospace, monospace"><b>from</b> __future__ <b>import</b> given_expression</font></div></div><div style="font-weight:400"><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace"><b>import</b> hypothesis</font></div></div><div style="font-weight:400"><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">@hypothesis.$given(hypothesis.strategies.integers())</font></div></div><div><font face="monospace, monospace"><b>def</b> foo(i):</font></div><div><font face="monospace, monospace"> x = z**2 + z**3 <b>given</b> z = f(i)</font></div><div style="font-weight:400"><font face="monospace, monospace"> ....</font></div></blockquote><div><br></div></div><div>Or even, if you want to prevent the "$" all over your code:</div><div><br></div><div><blockquote style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);margin:0px 0px 0px 40px;border:none;padding:0px"><span style="font-family:monospace,monospace"><b><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><div><font face="monospace, monospace"><b>from</b><span> </span>__future__<span> </span><b>import</b><span> </span>given_expression</font></div></div>from</b> hypothesis <b>import</b> $given <b>as</b> hgiven, strategies <b>as</b> st</span><font face="monospace, monospace" style="font-weight:400"><br><blockquote style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font face="monospace, monospace"><br></font></div></div></blockquote></font><span style="font-weight:400;font-family:monospace,monospace">@hgiven(st.integers())</span><font face="monospace, monospace" style="font-weight:400"><br></font><span style="font-family:monospace,monospace"><b>def</b></span><span style="font-weight:400;font-family:monospace,monospace"> foo(i):</span><font face="monospace, monospace" style="font-weight:400"><br></font><span style="font-weight:400;font-family:monospace,monospace"> x = z**2 + z**3 </span><span style="font-family:monospace,monospace"><b>given</b></span><span style="font-weight:400;font-family:monospace,monospace"> z = f(i)</span><font face="monospace, monospace" style="font-weight:400"><br></font></blockquote><blockquote style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);margin:0px 0px 0px 40px;border:none;padding:0px"><font face="monospace, monospace"><div><font face="monospace, monospace"> ....</font></div></font></blockquote><blockquote style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);margin:0px 0px 0px 40px;border:none;padding:0px"><font face="monospace, monospace"><br><br class="gmail-Apple-interchange-newline"></font></blockquote>If you have some library which uses a new keyword as a method name (you can't rename those with import ... as ...), it still works perfectly:</div><div><br></div><div><blockquote style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;background-color:rgb(255,255,255);margin:0px 0px 0px 40px;border:none;padding:0px"><span style="font-family:monospace,monospace"><b>from</b> mylib <b>import</b> SomeClass<br><br>instance = SomeClass()<br>instance.$given("foo")<br></span></blockquote><br class="gmail-Apple-interchange-newline">This is also helpful as a universal way to solve name clashes between python keywords and libraries that use some external concept that overlaps with python</div><div><br></div><div>(from <a href="https://pythonhosted.org/pyquery/attributes.html">https://pythonhosted.org/pyquery/attributes.html</a> ):</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><font face="monospace, monospace">>>> <b>import</b> pyquery <b>as</b> pq</font></div><div><font face="monospace, monospace">>>> p = pq(<i>'<p id="hello" class="hello"></p>'</i>)(<i>'p'</i>)</font></div><div><div><font face="monospace, monospace">>>> p.attr(id=<i>'hello'</i>, $class=<i>'hello2'</i>)</font></div></div><div><div><font face="monospace, monospace"><i>[<p#hello.hello2>]</i></font></div></div></blockquote><div><br></div><div>Or even nameclashes within python itself</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><font face="monospace, monospace">@classmethod</font></div><div><font face="monospace, monospace"><b>def </b>new_with_color($class, color): <i># Instead of the usual cls or class_</i></font></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><font face="monospace, monospace"> result = $class()</font></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><font face="monospace, monospace"> result.set_color(color)</font></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><font face="monospace, monospace"> return result</font></div></blockquote><div><br></div><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"></div><div dir="ltr"><table border="0" cellpadding="0" cellspacing="0" style="font-family:"Times New Roman""><tbody><tr valign="top"><td width="10%" style="display:block;width:60px;margin-left:30px"><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td height="150" style="display:block;width:120px;height:50px;margin-top:10px;margin-bottom:30px;margin-left:15px"><a href="https://www.machinalis.co.uk" style="color:rgb(111,111,111);font-size:12px" target="_blank"><img src="http://machinalis.com/images/machinalis.png" style="height: auto;" alt=""></a></td></tr></tbody></table></td><td width="90%"><table border="0" cellpadding="0" cellspacing="0" width="100%"><tbody><tr><td valign="top" style="padding:0px 0px 0px 20px"><table border="0" cellspacing="0" cellpadding="0"><tbody><tr style="font-family:poppins;font-size:14px;font-weight:bold;color:rgb(51,51,51)"><td>Daniel Moisset</td></tr><tr style="font-family:poppins;font-size:12px;text-transform:uppercase;letter-spacing:1px;color:rgb(205,39,66);padding:20px 0px 5px"><td>UK COUNTRY MANAGER</td></tr><tr><td style="padding:6px 0px"><table width="50px" height="1px" border="0" cellspacing="0" cellpadding="0" style="width:300px"><tbody><tr><td bgcolor="#e8e8e8" height="1px" style="font-size:1px;line-height:1px"> </td></tr></tbody></table></td></tr><tr><td style="font-family:poppins"><table border="0" cellspacing="0" cellpadding="0"><tbody><tr><td valign="top"><a href="https://goo.gl/maps/pH9BBLgE8dG2" style="color:rgb(111,111,111);font-size:12px" target="_blank"><span>A: 1 Fore Street, EC2Y 9DT London</span></a></td></tr></tbody></table></td></tr><tr><td style="font-family:poppins"><table border="0" cellspacing="0" cellpadding="0"><tbody><tr><td valign="top"><a href="tel:+44+7398+827139" style="color:rgb(111,111,111);font-size:12px" target="_blank">P: +44 7398 827139</a><span style="color:rgb(111,111,111);font-size:12px"></span> <a style="color:rgb(111,111,111);font-size:12px"></a> </td></tr></tbody></table></td></tr><tr><td style="font-family:poppins"><table border="0" cellspacing="0" cellpadding="0"><tbody><tr><td valign="top"><a href="mailto:dmoisset@machinalis.com" style="color:rgb(111,111,111);font-size:12px" target="_blank">M: dmoisset@machinalis.com</a><span style="color:rgb(111,111,111);font-size:12px"> | </span> <a style="color:rgb(111,111,111);font-size:12px">S: dmoisset</a> </td></tr><tr><td style="padding:10px 0px 0px"><a href="http://www.linkedin.com/company/456525" style="color:rgb(111,111,111);font-size:12px" target="_blank"><img src="https://machinalis.com/images/red_linkedin.png" style="height: auto;"></a> <a href="http://www.twitter.com/machinalis" style="color:rgb(111,111,111);font-size:12px;padding-left:10px" target="_blank"><img src="https://machinalis.com/images/red_twitter.png" style="height: auto;"></a> <a href="http://www.facebook.com/machinalis" style="color:rgb(111,111,111);font-size:12px;padding-left:10px" target="_blank"><img src="https://machinalis.com/images/red_face.png" style="height: auto;"></a> <a href="https://www.instagram.com/machinalis.life/" style="color:rgb(111,111,111);font-size:12px;padding-left:10px" target="_blank"><img src="https://machinalis.com/images/red_insta.png" style="height: auto;"></a></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table></td></tr></tbody></table><span><table width="100%" border="0" cellpadding="0" cellspacing="0" style="font-family:"Times New Roman""><tbody><tr><td><table bgcolor="#ffffff" align="left" cellpadding="0" cellspacing="0" border="0"><tbody></tbody></table></td></tr></tbody></table></span><span style="font-family:Arial;font-size:10.6667px;white-space:pre-wrap">Machinalis Limited is a company registered in England and Wales. Registered number: 10574987.</span></div></div></div></div></div></div></div></div>
</div>