<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Aug 23, 2015 at 9:31 PM, Wes Turner <span dir="ltr"><<a href="mailto:wes.turner@gmail.com" target="_blank">wes.turner@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Sun, Aug 23, 2015 at 8:41 PM, Nick Coghlan <span dir="ltr"><<a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span>On 24 August 2015 at 10:35, Eric V. Smith <<a href="mailto:eric@trueblade.com" target="_blank">eric@trueblade.com</a>> wrote:<br>
> On 08/22/2015 09:37 PM, Nick Coghlan wrote:<br>
</span><span>>> The trick would be to make interpolation lazy *by default* (preserving<br>
>> the triple of the raw template string, the parsed fields, and the<br>
>> expression values), and put the default rendering in the resulting<br>
>> object's *__str__* method.<br>
><br>
> At this point, I think PEPs 498 and 501 have converged, except for the<br>
> delayed string interpolation object (which I realize is important) and<br>
> how expressions are identified in the strings (which I consider less<br>
> important).<br>
><br>
> I think the string interpolation object is interesting. It's basically<br>
> what Petr Viktorin and Chris Angelico discussed and suggested here:<br>
> <a href="https://mail.python.org/pipermail/python-ideas/2015-August/035303.html" rel="noreferrer" target="_blank">https://mail.python.org/pipermail/python-ideas/2015-August/035303.html</a>.<br>
<br>
</span>Aha, I though I'd seen that idea go by in one of the threads, but I<br>
didn't remember where :)<br>
<br>
I'll add Petr and Chris to the acknowledgements section in 501.<br>
<span><br>
> My suggestion would be to add both f-strings (PEP 498) and i-strings (as<br>
> they're currently called in PEP 501), but with the exact same syntax to<br>
> identify and evaluate expressions. I don't particularly care what the<br>
> prefixes are. I'd add the plain f-strings first, then i-strings maybe<br>
> later. There are definitely some issues with delayed interpolation we<br>
> need to think about. An f-string would be shorthand for str(i-string).<br>
<br>
</span>+1, as this is the point of view I've come to as well.<br>
<span><br>
> I think it's hyperbolic to refers f-strings as a new string formatting<br>
> language. With one small difference (detailed in PEP 498, and with zero<br>
> usage I could find in the stdlib outside of tests), f-strings are a<br>
> strict superset of str.format() strings (but not the arguments to<br>
> .format of course). I think f-strings are no more different from<br>
> str.format strings than PEP 501 i-strings are to string.Template strings.<br>
<br>
</span>Yeah, that's a fair criticism of my rhetoric, so I'll stop saying that.<br>
<span><br>
> From what I can tell in the stdlib and in the wild, str.format() has<br>
> hundreds or thousands of times more usage that string.Template. I<br>
> realize that the reasons are not necessarily related to the syntax of<br>
> the replacement strings, but you can't say most people aren't familiar<br>
> with str.format().<br>
<br>
</span>Right, and I think we can actually make an example driven decision on<br>
that front by looking at potential *target* formats for template<br>
rendering. After all, one of the interesting discoveries we made in<br>
having both str.__mod__ and str.format available is that %-formatting<br>
is a great way to template str.format strings, and vice-versa, since<br>
the meta-characters don't conflict, so you can minimise the escaping<br>
needed.<br>
<br>
For use cases like writing object __repr__ methods, I don't think the<br>
choice of $-substitution or {}-substitution matters - neither $ nor {}<br>
are likely to appear in the desired output (except as part of<br>
interpolated values), so escaping shouldn't be common regardless of<br>
which we choose. (Side note: __repr__ and _str__ implementations are<br>
likely worth highlighting as a good use case for the new syntax!)<br>
<br>
I think things get more interesting once we start talking about<br>
interpolation targets other than "human readable text".<br>
<br>
For example, one of the neat (/scary, depending on how you feel about<br>
this kind of feature) things I realised in working on the latest draft<br>
of PEP 501 is that you could use it to template *Python code*,<br>
including eagerly bound references to objects in the current scope.<br>
That is:<br>
<br>
a = b + c<br>
<br>
could instead be written as:<br>
<br>
a = eval(str(i"$b + $c"))<br>
<br>
That's not very interesting if all you do is immediately call eval()<br>
on it, but it's a lot more interesting if you instead want to do<br>
things like extract the AST, dispatch the operation for execution in<br>
another process, etc. For example, you could use this capability to<br>
build eagerly bound closures, which wouldn't see changes in name<br>
bindings, but *would* see state changes in mutable objects.<br>
<br>
With $-substitution, that "just works", as $ generally isn't<br>
syntactically significant in Python code - it can only appear inside<br>
strings (and potentially interpolation templates). With<br>
{}-substitution, you'd have to double all the braces for dictionary<br>
displays, dictionary comprehensions and set comprehensions. In example<br>
form:<br>
<br>
data = {k:v for k, v in source}<br>
<br>
becomes:<br>
<br>
data = eval(str(i"{k:v for k, v in $source}"))<br>
<br>
rather than:<br>
<br>
data = eval(f"{{k:v for k, v in {{source}}}}"))<br>
<br>
You hit a similar problem if you're targeting Django or Jinja2<br>
templates, or any content that involves l20n style JavaScript<br>
translation strings: the use of braces for substitution expressions in<br>
the interpolation template conflicts with their use in the target<br>
format.<br>
<br>
So far, the only target rendering environments I've come up with where<br>
$-substitution would create a conflict are shell commands and<br>
JavaScript localisation using Mozilla's l20n syntax, and in both of<br>
those, I'd actually *want* the Python lookup to take precedence over<br>
the target environment lookup (and doubling the prefix to "$$" for<br>
target environment lookup seems quite reasonable when you actually do<br>
want to do the name lookup in the target environment).<br>
<span><br>
>> That description is probably as clear as mud, though, so back to the<br>
>> PEP I go! :)<br>
><br>
> Thanks for PEP 501. Maybe I'll add delayed interpolation to PEP 498!<br>
><br>
> On a more serious note, I'm thinking of adding i-strings to my f-string<br>
> implementation. I have some ideas that the format_spec (the :.3f stuff)<br>
> could be used by the code that eventually does the string interpolation.<br>
> For example, sql(i-string) might want to interpret this expression using<br>
> __sql__, instead of how str(i-string) would use __format__. Then the<br>
> sql() machinery could look at the format_spec and pass it to the value's<br>
> __sql__ method.<br>
<br>
</span>Yeah, that's the key reason PEP 501 is careful to treat them as opaque<br>
strings that it merely transports through to the renderer. The<br>
*default* renderer would expect them to be str.format format<br>
specifiers, but other renderers may either disallow them entirely, or<br>
expect them to do something different.<br>
<span><br>
> For example:<br>
> sql(i'select {date:as_date} from {tablename}'<br>
><br>
> might call date.__sql__('as_date'), which would know how to cast to the<br>
> write datatype (this happens to me all the time).<br>
><br>
> This is one reason I'm thinking of ditching !s, !r, and !a, at least for<br>
> the first implementation of PEP 498: they're not needed, and are not<br>
> generally applicable if we add the hooks I'm considering into i-strings.<br>
<br>
</span>+1 from me. Given arbitrary expression support, it's both entirely<br>
possible and more explicit to write the builtin calls directly (obj!a,<br>
obj!r, obj!s -> ascii(obj), repr(obj), str(obj))<br></blockquote><div><br></div></div></div><div>IIUC, to do this with SQL, </div><span class=""><div><br></div><div>> sql(i'select {date:as_date} from {tablename}'</div><div><br></div></span><div>needs to be</div><div><br></div><div> ['select ', unescaped(date, 'as_date'), 'from ', unescaped(tablename)]</div><div><br></div><div>so that e.g. sql_92(), sql_2011()</div><div>would know that 'select ' is presumably implicitly escaped</div><div><br></div><div>* <a href="https://en.wikipedia.org/wiki/SQL#Interoperability_and_standardization" target="_blank">https://en.wikipedia.org/wiki/SQL#Interoperability_and_standardization</a></div><div>* <a href="http://docs.sqlalchemy.org/en/rel_1_0/dialects/" target="_blank">http://docs.sqlalchemy.org/en/rel_1_0/dialects/</a> </div><div>* <a href="https://docs.djangoproject.com/en/1.7/ref/models/queries/#f-expressions" target="_blank">https://docs.djangoproject.com/en/1.7/ref/models/queries/#f-expressions</a> "Django F-Expressions"</div><span class=""><div><br></div></span></div></div></div></blockquote><div><br></div><div>For reference, the SQLAlchemy Expression API solves for</div><div>(safer) method-chaining, nesting *Python* expression API; </div><div>or you can reuse a raw SQL connection from a ConnectionPool.</div><div><br></div><div><div>Django F-Objects are relevant because they are deferred</div><div>(and compiled in context to the query context);</div><div>similar to the objectives of a given SQL syntax</div><div>templating, parameterization, and serialization</div><div>library.</div><div><br></div><div>Django Q-Objects are similar,</div><div>in that an f-string is basically</div><div>an iterator of AND-ed expressions</div><div>where AND means string concatenation.</div><div><br></div><div>Personally,</div><div>I'd pretty much always just reflect the tables</div><div>or map them out</div><div>and write SQLAlchemy Python expressions</div><div>which are then compiled to a particular dialect</div><div>(and quoted appropriately, **avoiding CWE-89**</div><div>surviving across table renames,</div><div>managing migrations).</div><div><br></div><div>Is it sometimes faster to write SQL by hand?</div><div><br></div><div>* I'd write the [SQLAlchemy], serialize to SQL, [and modify]</div><div> (because I should have namespaced Python table attrs for those attrs anyway,</div><div> even if it requires table introspection and reflection at (every/pool) instantiation)</div><div>* you can always execute query with a raw connection with an ORM</div><div> (and then **refactor (REF) string-ified table and column names**)</div><div><br></div><div>Each ORM (and DBAPI) have parametrization settings</div><div>(e.g. '%' or '?' or configuration_setting)</div><div>which should not collide with the f-string syntax.</div></div><div><br></div><div>* DBAPI v2.0</div><div> <a href="https://www.python.org/dev/peps/pep-0249/">https://www.python.org/dev/peps/pep-0249/</a></div><div>* SQLite DBAPI</div><div> <a href="https://docs.python.org/2/library/sqlite3.html">https://docs.python.org/2/library/sqlite3.html</a></div><div> <a href="https://docs.python.org/3/library/sqlite3.html">https://docs.python.org/3/library/sqlite3.html</a></div><div><br></div><div><a href="http://docs.sqlalchemy.org/en/rel_1_0/core/tutorial.html#conjunctions">http://docs.sqlalchemy.org/en/rel_1_0/core/tutorial.html#conjunctions</a><br></div><div><br></div><div><pre style="margin-top:5px;margin-bottom:5px;padding:10px;font-size:1.2em;border:1px solid rgb(204,204,204);overflow:auto;line-height:1.3em;color:rgb(51,51,51);background-color:rgb(240,240,240)"><span class="" style="color:rgb(198,93,9);font-weight:bold">>>> </span><span class="">s</span> <span class="" style="color:rgb(102,102,102)">=</span> <span class="">select</span><span class="">([(</span><span class="">users</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">c</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">fullname</span> <span class="" style="color:rgb(102,102,102)">+</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="" style="color:rgb(64,112,160)">", "</span> <span class="" style="color:rgb(102,102,102)">+</span> <span class="">addresses</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">c</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">email_address</span><span class="">)</span><span class="" style="color:rgb(102,102,102)">.</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="">label</span><span class="">(</span><span class="" style="color:rgb(64,112,160)">'title'</span><span class="">)])</span><span class="" style="color:rgb(102,102,102)">.</span>\
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="">where</span><span class="">(</span><span class="">users</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">c</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">id</span> <span class="" style="color:rgb(102,102,102)">==</span> <span class="">addresses</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">c</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">user_id</span><span class="">)</span><span class="" style="color:rgb(102,102,102)">.</span>\
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="">where</span><span class="">(</span><span class="">users</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">c</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">name</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">between</span><span class="">(</span><span class="" style="color:rgb(64,112,160)">'m'</span><span class="">,</span> <span class="" style="color:rgb(64,112,160)">'z'</span><span class="">))</span><span class="" style="color:rgb(102,102,102)">.</span>\
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="">where</span><span class="">(</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="">or_</span><span class="">(</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="">addresses</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">c</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">email_address</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">like</span><span class="">(</span><span class="" style="color:rgb(64,112,160)">'%@<a href="http://aol.com">aol.com</a>'</span><span class="">),</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="">addresses</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">c</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">email_address</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">like</span><span class="">(</span><span class="" style="color:rgb(64,112,160)">'%@<a href="http://msn.com">msn.com</a>'</span><span class="">)</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="">)</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="">)</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">>>> </span><span class="">conn</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">execute</span><span class="">(</span><span class="">s</span><span class="">)</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">fetchall</span><span class="">()</span>
<span class="" style="color:rgb(51,51,51)">SELECT users.fullname || ? || addresses.email_address AS title</span>
<span class="" style="color:rgb(51,51,51)">FROM users, addresses</span>
<span class="" style="color:rgb(51,51,51)">WHERE <a href="http://users.id">users.id</a> = addresses.user_id AND <a href="http://users.name">users.name</a> BETWEEN ? AND ? AND</span>
<span class="" style="color:rgb(51,51,51)">(addresses.email_address LIKE ? OR addresses.email_address LIKE ?)</span>
<span class="" style="color:rgb(51,51,51)">(', ', 'm', 'z', '%@<a href="http://aol.com">aol.com</a>', '%@<a href="http://msn.com">msn.com</a>')</span>
<span class="" style="color:rgb(51,51,51)">[(u'Wendy Williams, <a href="mailto:wendy@aol.com">wendy@aol.com</a>',)]</span></pre></div><div><br></div><div><a href="http://docs.sqlalchemy.org/en/rel_1_0/core/tutorial.html#using-textual-sql">http://docs.sqlalchemy.org/en/rel_1_0/core/tutorial.html#using-textual-sql</a><br></div><div><pre style="margin-top:5px;margin-bottom:5px;padding:10px;font-size:1.2em;border:1px solid rgb(204,204,204);overflow:auto;line-height:1.3em;color:rgb(51,51,51);background-color:rgb(240,240,240)"><span class="" style="color:rgb(198,93,9);font-weight:bold">>>> </span><span class="" style="color:rgb(0,112,32)">from</span> <span class="" style="color:rgb(14,132,181);font-weight:bold">sqlalchemy.sql</span> <span class="" style="color:rgb(0,112,32)">import</span> <span class="">text</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">>>> </span><span class="">s</span> <span class="" style="color:rgb(102,102,102)">=</span> <span class="">text</span><span class="">(</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="" style="color:rgb(64,112,160)">"SELECT users.fullname || ', ' || addresses.email_address AS title "</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="" style="color:rgb(64,112,160)">"FROM users, addresses "</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="" style="color:rgb(64,112,160)">"WHERE <a href="http://users.id">users.id</a> = addresses.user_id "</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="" style="color:rgb(64,112,160)">"AND <a href="http://users.name">users.name</a> BETWEEN :x AND :y "</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="" style="color:rgb(64,112,160)">"AND (addresses.email_address LIKE :e1 "</span>
<span class="" style="color:rgb(198,93,9);font-weight:bold">... </span> <span class="" style="color:rgb(64,112,160)">"OR addresses.email_address LIKE :e2)"</span><span class="">)</span>
<a href="http://docs.sqlalchemy.org/en/rel_1_0/core/tutorial.html#" class="" style="text-decoration:none;color:rgb(153,0,0);padding:1px 2px;font-family:helvetica,arial,sans-serif;font-size:0.9em;text-transform:uppercase;border:1px solid;margin:0px 10px 0px 15px;float:right;line-height:1.2em">SQL</a><span class="" style="color:rgb(198,93,9);font-weight:bold">>>> </span><span class="">conn</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">execute</span><span class="">(</span><span class="">s</span><span class="">,</span> <span class="">x</span><span class="" style="color:rgb(102,102,102)">=</span><span class="" style="color:rgb(64,112,160)">'m'</span><span class="">,</span> <span class="">y</span><span class="" style="color:rgb(102,102,102)">=</span><span class="" style="color:rgb(64,112,160)">'z'</span><span class="">,</span> <span class="">e1</span><span class="" style="color:rgb(102,102,102)">=</span><span class="" style="color:rgb(64,112,160)">'%@<a href="http://aol.com">aol.com</a>'</span><span class="">,</span> <span class="">e2</span><span class="" style="color:rgb(102,102,102)">=</span><span class="" style="color:rgb(64,112,160)">'%@<a href="http://msn.com">msn.com</a>'</span><span class="">)</span><span class="" style="color:rgb(102,102,102)">.</span><span class="">fetchall</span><span class="">()</span>
<span class="" style="color:rgb(51,51,51)">[(u'Wendy Williams, <a href="mailto:wendy@aol.com">wendy@aol.com</a>',)]</span></pre></div><div><br></div><div>SQLAlchemy is not async-compatible</div><div>(besides, most drivers block);</div><div>it's debatable whether async would be faster, anyway:</div><div><a href="https://bitbucket.org/zzzeek/sqlalchemy/issues/3414/asyncio-and-sqlalchemy">https://bitbucket.org/zzzeek/sqlalchemy/issues/3414/asyncio-and-sqlalchemy</a></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class=""><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span><br>
Regards,<br>
Nick.<br>
<br>
--<br>
Nick Coghlan | <a href="mailto:ncoghlan@gmail.com" target="_blank">ncoghlan@gmail.com</a> | Brisbane, Australia<br>
</span><div><div>_______________________________________________<br>
Python-ideas mailing list<br>
<a href="mailto:Python-ideas@python.org" target="_blank">Python-ideas@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-ideas" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/python-ideas</a><br>
Code of Conduct: <a href="http://python.org/psf/codeofconduct/" rel="noreferrer" target="_blank">http://python.org/psf/codeofconduct/</a><br>
</div></div></blockquote></span></div><br></div></div>
</blockquote></div><br></div></div>