On 2019-08-27 00:03, stpasha@gmail.com wrote:

In Python strings are allowed to have a number of special prefixes:

b'', r'', u'', f'' + their combinations.

The proposal is to allow arbitrary (or letter-only) user-defined prefixes as well. Essentially, a string prefix would serve as a decorator for a string, allowing the user to impose a special semantics of their choosing.

There are quite a few situations where this can be used: - Fraction literals: `frac'123/4567'` - Decimals: `dec'5.34'` - Date/time constants: `t'2019-08-26'` - SQL expressions: `sql'SELECT * FROM tbl WHERE a=?'.bind(a=...)` - Regular expressions: `rx'[a-zA-Z]+'` - Version strings: `v'1.13.0a'` - etc.

This proposal has been already discussed before, in 2013: https://mail.python.org/archives/list/python-ideas@python.org/thread/M3OLUUR...

The opinions were divided whether this is a useful addition. The opponents mainly argued that as this only "saves a couple of keystrokes", there is no need to overcomplicate the language. It seems to me that now, 6 years later, that argument can be dismissed by the fact that we had, in fact, added new prefix "f" to the language. Note how the "format strings" would fall squarely within this framework if they were not added by now.

In addition, I believe that "saving a few keystroked" is a worthy goal if it adds considerable clarity to the expression. Readability counts. Compare:

v"1.13.0a" v("1.13.0a")

To me, the former expression is far easier to read. Parentheses, especially as they become deeply nested, are not easy on the eyes. But, even more importantly, the first expression much better conveys the *intent* of a version string. It has a feeling of an immutable object. In the second case the string is passed to the constructor, but the string has no meaning of its own. As such, the second expression feels artificial. Consider this: if the feature already existed, how *would* you prefer to write your code?

The prefixes would also help when writing functions that accept different types of their argument. For example:

collection.select("abc") # find items with name 'abc' collection.select(rx"[abc]+") # find items that match regular expression

I'm not discussing possible implementation of this feature just yet, we can get to that point later when there is a general understanding that this is worth considering.

At what point would backslashes be handled?

Thanks, Andrew, for your feedback. I didn't even think about string **suffixes**, but clearly they can be implemented together with the prefixes for additional flexibility. And your idea that `<string literal> <suffix>` is conceptually no different than `<numeric literal> <suffix>` is absolutely insightful. Speaking of string suffixes, flags on regular expressions immediately come to mind. For example `rx"(abc)"ig` could create a regular expression that performs global case-insensitive search.

I don’t think you can fairly discuss this idea without getting at least a _little_ bit into the implementation details.

Right. So, the first question to answer is what the compiler should do when it sees a prefixed (suffixed) string? That is, what byte-code should be emitted when the compiler sees `lambda: a"bcd"e` ? In one approach, we'd want this expression to be evaluated at compile time, similar to how f-strings work. However, how would the compiler know what prefix "a" means exactly? There has to be some kind of directive to tell the compiler that. For example, imagine the compiler sees near the top of the file #pragma from mymodule import a It would then import the symbol `a`, call `a("bcd", suffix="e")`. This would return an AST tree that will be plugged in place of the original string. This solution allows maximum efficiency, but seems inflexible and deeply invasive. Another approach would defer the construction of objects to compile time. Though not as efficient, it would allow loading prefixes at run-time. In this case `a"bcd"e` can be interpreted by the compiler as if it was a("bcd", suffix="e") where symbol `a` is to be looked up in the local/global scope. One thing I would rather want to avoid though, is the pollution of the variable namespace. For example, I'd like to be able to use variable `t = ...`, without worrying about string prefix `t"..."`. For this approach to work, we'd create a new code op, so that `a"bcd"e` would become 0 LOAD_CONST 1 ('a', 'bcd', 'e') 2 STR_RESOLVE_TAG 0 where `STR_RESOLVE_TAG` would effectively call `__resolve_tag__()` special method. The method would search for `a` in the registry of known string tags, and then pass the tuple to the corresponding constructor. There will, of course, be a method to register new tags. Something like str.___register_tag__('a', MyAObject) As for suffix-only literals, we can treat them as if they begin with an underscore. Thus, `1/3f` would be equivalent to 1/_f(3)

On Aug 26, 2019, at 18:41, stpasha@gmail.com wrote:

Thanks, Andrew, for your feedback. I didn't even think about string **suffixes**, but clearly they can be implemented together with the prefixes for additional flexibility.

What about _instead of_ rather than _together with_? Half of Stephen’s objections are related to the ambiguity (to a human, even if not to the parser) of user prefixes in the (potential) presence of the builtin prefixes. None of those go even arise with suffixes. Anyway, maybe you already have good answers for all of those objections, but if not… Also, there’s at least one mainstream language (C++) that allows user suffixes and has literal syntax otherwise somewhat like Python’s, and the proposals for other languages like Rust generally seem to be generally trying to do “like C++ but minus all the usual C++ over-complexity”. Are there actual examples of languages with user prefixes? The only different designs I know of rely on the static type of the evaluation context. (For example, in Swift, you can just statically type `23 : km` or `"abc]*" : regex`, or even just pass the literal to a function that’s declared or inferred to take a regex if that happens to be readable in your use case, so there’s no need for a suffix syntax.) Which is neat, but obviously not applicable to Python.

And your idea that `<string literal> <suffix>` is conceptually no different than `<numeric literal> <suffix>` is absolutely insightful.

Well, back in 2015 I probably just stole the idea from C++. :) Another question that raises that I just remembered: the word “literal” has three overlapping but distinct meanings in Python. Which one do we actually mean here? In particular, are container displays “literals”? For that matter, is -2 even a literal? Also, from what I remember, either in 2013 or in 2015, the discussion got side-tracked over people not liking the word “literal” to mean “something that’s actually the result of a runtime function call”. That may be less of a problem after f-strings (which are called literals in the PEP; not sure about the language reference), but last time around, bringing up the fact that “-2” is actually a function call didn’t sway anyone. So, maybe I shouldn’t be using the word “literal” this time, and I really hope it doesn’t ruin your proposal…

Speaking of string suffixes, flags on regular expressions immediately come to mind. For example `rx"(abc)"ig` could create a regular expression that performs global case-insensitive search.

That’s an interesting idea. And that’s something you can’t do with a single-affix design; you need prefixes and suffixes, unless you have some kind of separator for chaining, or only allow single characters.

...

I don’t think you can fairly discuss this idea without getting at least a _little_ bit into the implementation details.

Right. So, the first question to answer is what the compiler should do when it sees a prefixed (suffixed) string? That is, what byte-code should be emitted when the compiler sees `lambda: a"bcd"e` ?

In one approach, we'd want this expression to be evaluated at compile time, similar to how f-strings work. However, how would the compiler know what prefix "a" means exactly? There has to be some kind of directive to tell the compiler that. For example, imagine the compiler sees near the top of the file

#pragma from mymodule import a

It would then import the symbol `a`, call `a("bcd", suffix="e")`. This would return an AST tree that will be plugged in place of the original string.

This solution allows maximum efficiency, but seems inflexible and deeply invasive.

Another approach would defer the construction of objects to compile time. Though not as efficient, it would allow loading prefixes at run-time. In this case `a"bcd"e` can be interpreted by the compiler as if it was

a("bcd", suffix="e")

where symbol `a` is to be looked up in the local/global scope.

My hack works basically like this. The compiler just converts it to a function call, which is looked up normally. I think that’s the right tack here. IIRC, my hack translates a D suffix into a call to something like -_user_literal_D, which solves the problem with accidental pollution of the namespace. But this does mean that any code that wants to use the D suffix has to `from decimal_literals import *, or `2.3D` raises a NameError about nothing named _user_literal_D. (Either that, or someone has to inject it into builtins…) I’m not sure whether that’s user-friendly enough. Anyway, I think your registry idea makes more sense. Then `2.3D` effectively just means `__user_literals__['D']('2.3')`, and there’s no namespace pollution at all.

For this approach to work, we'd create a new code op, so that `a"bcd"e` would become

0 LOAD_CONST 1 ('a', 'bcd', 'e') 2 STR_RESOLVE_TAG 0

where `STR_RESOLVE_TAG` would effectively call `__resolve_tag__()` special method. The method would search for `a` in the registry of known string tags, and then pass the tuple to the corresponding constructor.

Do we even need that? It’s true that most things in Python translate reasonably directly to bytecodes, but in this case it might be easier to just compile to existing bytecodes to look up and call the function.

There will, of course, be a method to register new tags. Something like

str.___register_tag__('a', MyAObject)

If the params are (handler, name=None), and None means to use the __name__ of the handler at the tag, then you can use it as a decorator: @__register_tag__ def D(decimal_string): return decimal.Decimal(decimal_string) Although this may not be the best example, because it might actually be clearer (as well as more efficient) to just register the constructor: __register_tag__(decimal.Decimal, 'D') … but I suspect many examples won’t be just a matter of calling a constructor on the string.

As for suffix-only literals, we can treat them as if they begin with an underscore. Thus, `1/3f` would be equivalent to

1/_f(3)

Does that mean you can’t actually register a prefix named `_f`? Or that, if you do, it also registers a suffix named `f`? Also, I think for most non-single-letter suffixes you’d actually want an underscore at the start of the suffix. See C++ for lots of examples, but for a quick illustration,compare these: c = 2.99792458e8mps c = 2.99792458e8_mps c = 299_792_458mps c = 299_792_458_mps The _mps suffix looks a lot better than the mps suffix, doesn’t it? But would you want the function to have to be named __mps with two underscores? It may be worth coming up with the most compelling examples and then working out what feature set would support as many as possible, rather that trying to work out the ultimate feature set first and then see what we can do with it. It’s probably worth stealing liberally from the C++ discussion (and any other languages that have similar features) as well as the 2013 Python discussion, but off the top of my head: * Decimal, Fraction, np.float32, mpz, … * Path objects * Windows native Path objects, possibly with “really raw” processing to allow trailing backslashes * regex, possibly with flags, possibly with “really raw” backslashes * “Really raw” strings in general. * JSON (register the stdlib or simplejson or ujson), XML (register ETree or lxml or bs4 or whatever you want), HTML, etc. * unit suffixes for quantities

Thank you, Steven, for taking the time to write such an elaborate rebuttal. If I understand the heart of your argument correctly, you're concerned that the prefixed strings may add confusion to the code. That nobody knows what `l'abc'` or `czt'xxx'` could possibly mean, while at the same time `v'1.0'` could mean many things, whereas `v'cal-{a}'` would mean nothing at all... These are all valid concerns. The string (or number) prefixes add new power to the language, and with new power comes new responsibility. While the syntax can be used to enhance readability of the code, it can also be abused to make the code more obscure. However, Python does not attempt to be an idiot-proof language. "We are all consenting adults" is one of its guiding principles. If a certain feature can potentially be misused shouldn't deter us from adding it, if the benefits are significant. And the benefits in terms of readability can be significant. Consider the existing python prefixes: `r'...'` is purely for readability, it adds no extra functionality; `f'...'` has a neat compiler support, but even if it didn't (and most python users don't actually realize f-strings get preprocess by the compiler) it would still enhance readability compared to `str.format()`. It's nice to be able to write a complex number as `5 + 3j` instead of `complex(5, 3)`. And so on.

What's v() do? Verbose string? Oh, you intended a version string did you? If only you had written version instead of v I might not have guessed wrong. What were you saying about preferring readability and clarity over brevity?

You're correct that, devoid of context, `v"smth..."` is not very meaningful. The "v" suffix could mean "version", or "verbose", or "volatile", or "vectorized", or "velociraptor", or whatever. Luckily, the code is almost always exists within a specific context. It solves a particular problem, and works within a particular domain, and makes perfect sense for people working within that domain. This isn't much different than, say, `np.` suffix, which means "numpy" in the domain of numerical computations, NP-completeness for some mathematicians, and "no problem" for regular users. From practical perspective, the meaning of each particular symbol will come from the way that it was created or imported. For example, if you script says `from packaging.version import v` then "v" is a version. If, on the other hand, it says `from zoo import velociraptor as v`, then it's an altogether different beast.

In other words, I got all of the meaning from the string part, not the prefix. The prefix on its own, I would have guessed completely wrong.

Exactly. You look at string "1.10a" and you know it must be a version string, because you're a human, you're smart. The compiler is not a human, it has no idea. To the Python interpreter it's just a PyUnicode object of length 5. It's meaningless. But when you combine this string with a prefix into a single object, it gains power. It can have methods or special behaviors. It can have a type, different from `str`, that can be inspected when passing this object to another function. Think of `v"1.10a"` as making a "typed string" (even though it may end up not being a string at all). By writing `v"1.10a"` I convey the intent for this to be a version string.

for rather insignificant gains, the saving of two parentheses.

Two bytes doesn't sound like a lot. I mean, it is quite little on the grand scale of things. However, I don't think the simple byte-count is a proper measure here. There could be benefits to readability even if it was 0 or negative byte difference. I believe a good way to think about this is the following: if the feature was already implemented, would people want to use it, and would it improve readability of their code? I speculate that the answer is true to both of these questions. At least some people. As a practical example, consider function `pandas.read_csv()`. The documentation for its `sep` parameter says "In addition, separators longer than 1 character and different from ``'\s+'`` will be interpreted as regular expressions ...". In this case they wanted the `sep` parameter to handle both simple separators, and the regular expression separators. However, as there is no syntax to create a "regular expression string", they ended up with this dubious heuristic based on the length of the string... Ideally, they should have said that `sep` could be either a string or a regexp-object, but the barrier to write from re import compile as rx rx('...') is just impossibly high for a typical user. Not to mention that such code **would** be actually harder to read, because I'd be inventing my own notation for a function that is commonly known under a different name. My another pet peeve is datetime literals. Or, rather, their absence. I often see, again in pandas, how people create columns of strings ["2010-05-01", "2010-05-02", ...], and then call `parse_datetime()`. It would have been more straightforward if there was a standard syntax for denoting datetime constants, allowing us to create a column of datetime type directly.

On Aug 26, 2019, at 23:43, Serhiy Storchaka wrote:

27.08.19 06:38, Andrew Barnert via Python-ideas пише:

...

* JSON (register the stdlib or simplejson or ujson),

What if the JSON suffix for?

I think you’d mainly want it in combination with percent-, html-, or uu-equals-decoding, which makes it a potential stress test of the “multiple affixes” or “affixes with modifiers” idea. Which I think is important, because I like what the OP came up with for that idea, so I want to push it beyond just the “regex with flags” example to see if it breaks. Maybe URL, which often has the same html and percent encoding issues, would be a better example? I personally don’t need to decode URLs that often in Python (unlike in, say, ObjC, where there’s a smart URL class that you use in place of strings all over the place), but maybe others do?

JSON is virtually a subset of Python except that that it uses true, false and null instead of True, False and None.

Is it _virtually_ a subset, or literally so, modulo those three values? I don’t know off the top of my head. Look at all the trouble caused by Crockford just assuming that the syntax he’d defined was a strict subset of JS when actually it isn’t quite. Actually, now that I think of it, I do know. Python has allow_nan on by default, so you’d need to also `from math import nan as NaN` and `from math import inf as Infinity`. But is that it? I’m not sure. And of course if you’ve done this: jdec = json.JSONDecoder(parse_float=Decimal) __register_prefix__(jdec.decode, 'j') … then even j'1.1' and 1.1 are no longer the same values. Not to mention what you get if you registered Pandas’s JSON reader instead of the stdlib’s.

On Tue, Aug 27, 2019 at 08:22:22AM -0000, stpasha@gmail.com wrote:

The string (or number) prefixes add new power to the language

I don't think they do. It's just syntactic sugar for a function call. There's nothing that czt'...' will do that czt('...') can't already do. If you have a proposal that allows custom string prefixes to do something that a function call cannot do, I've missed it.

If a certain feature can potentially be misused shouldn't deter us from adding it, if the benefits are significant.

Very true, but so far I see nothing in this proposal that suggests that the benefits are more significant than avoiding having to type a pair of parentheses. Every benefit I have seen applies equally to the function call version, but without the added complexity to the language of allowing custom string prefixes.

And the benefits in terms of readability can be significant.

I don't think they will be. I think they will encourage cryptic one-character function names disguised as prefixes: v'...' instead of Version(...) x'...' instead of re.compile(...) to take two examples from your proposal. At least this is somewhat better: sql'...' but that leaves the ambiguity of not knowing whether that's a chained function call s(q(l(...))) or a single sql(...). I believe it will also encourage inefficient and cryptic string parsing instead of more clear use of seperate arguments. Your earlier example: frac'123/4567' The Fraction constructor already accepts such strings, and it is occasionally handy for parsing user-input. But using it to parse string literals gives slow, inefficient code for little or no benefit: [steve@ando cpython]$ ./python -m timeit -s 'from fractions import Fraction' 'Fraction(123, 4567)' 20000 loops, best of 5: 18.9 usec per loop [steve@ando cpython]$ ./python -m timeit -s 'from fractions import Fraction' 'Fraction("123/4567")' 5000 loops, best of 5: 52.9 usec per loop Unless you can suggest a way to parse arbitrary strings in arbitrary ways at compile-time, these custom string prefixes are probably doomed to be slow and inefficient. The best thing I can say about this is that at least frac'123/4567' would probably be easy to understand, since the / syntax for fractions is familiar to most people from school. But the same cannot be said for other custom prefixes: cf'[0; 37, 7, 1, 2, 5]' Perhaps you can guess the meaning of that cf-string. Perhaps you can't. A hint might point you in the right direction: assert cf'[0; 37, 7, 1, 2, 5]' == Fraction(123, 4567) (By the way, the semi-colon is meaningful and not a typo.) To the degree that custom string prefixes will encourage cryptic one and two letter names, I think that this will hurt readability and clarity of code. But if the reader has the domain knowledge to recognise what "cf" stands for, this may be no worse than (say) "re" (regular expression). In conventional code, we might call the cf function like this: cf([0, 37, 7, 1, 2, 5]) # Single list argument. cf(0, 37, 7, 1, 2, 5) # *args version. Either way works for me. But it is your argument that replacing the parentheses with quote marks is "more readable": cf([0, 37, 7, 1, 2, 5]) cf'[0; 37, 7, 1, 2, 5]' not just a little bit more readable, but enough to make up for the inefficiency of having to write your own parser, deal with errors, compile a string literal, parse it at runtime, and only then call the actual cf constructor and return a cf object. Even if I accepted your claim that swapping (...) for '...' was more readable, I am skeptical that the additional work and runtime inefficiency would be worth the supposed benefit. I don't wish to say that parsing strings to extract information is always an anti-pattern: http://cyrille.martraire.com/2010/01/the-string-obsession-anti-pattern/ after all we often need to process data coming from config files or other user-input, where we have no choice but to accept a string. But parsing string *literals* usually is an anti-pattern, especially when there is a trivial transformation from the string to the constructor arguments, e.g. 123/4567 --> Fraction(123, 4567). [...]

Exactly. You look at string "1.10a" and you know it must be a version string, because you're a human, you're smart. The compiler is not a human, it has no idea. To the Python interpreter it's just a PyUnicode object of length 5. It's meaningless. But when you combine this string with a prefix into a single object, it gains power. It can have methods or special behaviors. It can have a type, different from `str`, that can be inspected when passing this object to another function.

Everything you say there applies to ordinary function call syntax too: Version('1.10a') can have methods, special behaviours, a type different from str, etc. Not one of those benefits comes from *custom string prefixes*. They all come from the use of a custom type. In fact, we can can be more explicit and clear with the constructor: Version(major=1, minor=10, stage='a') There is nothing magic about this v-string prefix. You still have to write a Version class with a version-string parser. The compiler can't help you, because it has no knowledge of the format of version strings. All the compiler can do is pass the string '1.10a' to the function v(). [...]

...

for rather insignificant gains, the saving of two parentheses.

Two bytes doesn't sound like a lot. I mean, it is quite little on the grand scale of things. However, I don't think the simple byte-count is a proper measure here. There could be benefits to readability even if it was 0 or negative byte difference.

"There could be..." lots of things, but the onus is on you to prove that there actually *are* such benefits.

I believe a good way to think about this is the following: if the feature was already implemented, would people want to use it, and would it improve readability of their code?

I answered that in my previous post. I would prefer an explicit, clear, self-documenting function call Version() over a terse, unclear syntax that looks like a string but isn't. I don't think that v'1.10a' is clearer or more readable than Version('1.10a'). It is *shorter*, but that's it. The bottom line is, so long as this proposal is for nothing more than mere syntactic sugar allowing you to drop the parentheses from certain function calls (those that take a single string argument), the benefit is tiny, and the added complexity and opportunity for abuse and confusion is large.

As a practical example, consider function `pandas.read_csv()`. The documentation for its `sep` parameter says "In addition, separators longer than 1 character and different from ``'\s+'`` will be interpreted as regular expressions ...". In this case they wanted the `sep` parameter to handle both simple separators, and the regular expression separators. However, as there is no syntax to create a "regular expression string", they ended up with this dubious heuristic based on the length of the string...

I can't help pandas' poor API, and I doubt that your proposal would have prevented it either.

Ideally, they should have said that `sep` could be either a string or a regexp-object, but the barrier to write

from re import compile as rx rx('...')

is just impossibly high for a typical user.

Think about what you are saying about the sophisticated data scientists who are typical pandas users: - they can write "import pandas" - but not "import re" or "from re import compile as rx" - they will be able to import your rx'...' string prefix from wherever it comes from (perhaps "from re import rx"?) - and are capable of writing regular expressions using your custom rx'...' syntax - but adding parentheses is beyond them: rx('...'). I cannot take this argument about sophisticated regex-users who are defeated by function call syntax seriously. -- Steven

On Aug 27, 2019, at 08:52, Steven D'Aprano wrote:

...

On Tue, Aug 27, 2019 at 05:24:19AM -0700, Andrew Barnert via Python-ideas wrote:

There is a possibility in between the two extremes of “useless” and “complete monster”: the prefix accepts exactly one token, but can parse that token however it wants.

How is that different from passing a string argument to a function or class constructor that can parse that token however it wants?

x'...'

x('...')

Unless there is some significant difference between the two, what does this proposal give us?

Before I get into this, let me ask you a question. What does the j suffix give us? You can write complex numbers without it just fine: c = complex c(1, 2) And you can even write a j function trivially: def j(x): return complex(0, x) 1 + j(2) But would anyone ever write that when they can write it like this: 1 + 2j I don’t think so. What does the j suffix give us? The two extra keystrokes are trivial. The visual noise of the parens is a bigger deal. The real issue is that this matches the way we conceptually think of complex numbers, and the way we write them in other contexts. (Well, the way electrical engineers write them; most of the rest of us use i rather than j… but still, having to use j instead of i is less of an impediment to reading 1+2j than having to use function syntax like 1+i(2). And the exact same thing is true in 3D or CUDA code that uses a lot of float32 values. Or code that uses a lot of Decimal values. In those cases, I actually have to go through a string for implementation reasons (because otherwise Python would force me to go through a float64 and distort the values), but conceptually; there are no strings involved when I write this: array([f('0.2'), f('0.3'), f('0.1')]) … and it would be a lot more readable if I could write it the same way I do in other programming languages: array([0.2f, 0.3f, 0.1f]) Again, it’s not about saving 4 keystrokes per number, and the visual noise of the parens is an issue but not the main one (and quotes are barely any noise by comparison); it’s the fact that these numeric values look like numeric values instead of looking like strings The fact that they look the same as the same values in other contexts like a C++ program or a GLSL shader is a pretty large added bonus. But I don’t think that’s essential to the value here. If you forced me to use prefixes instead of suffixes (I don’t think there’s any good reason for that, but who knows how the winds of bikeshedding may blow), I’d still prefer f2.3 to f('2.3'), because it still looks like a number, as it should. I know this is doable, because I’ve written an import hook that does it, plus I have a decade of experience with another popular language (C++) that has essentially the same feature. What about the performance cost of these values not being constants? A decorator that finds np.float32 calls on constants and promoted them to constants by hacking the bytecode is pretty trivial to write, or you can load the whole array in one go from a bytes constant and put the readable version in a comment, or whatever. But anything that’s slow enough to be worth optimizing is doing a huge matmul or pushing zillions of values back and forth to the GPU or something else that swamps the setup cost, even if the setup cost involves a few dozen string parses, so it never matters. At least not for me. —- For a completely different example—but one that I’ve also already given earlier in this thread, so I won’t belabor it too much: path'C:\' bs"this\ space won’t have a backslash before it, also \e[22; is an escape sequence and of course \a is still a bell because I’m using the rules from C/JS/etc." bs37"this\ space has a backslash before it without raising a warning or an error even in Python 3.15 because I’ve implemented the 3.7 rules” … and so on. Some of these _could_ be done with a raw string and a (maybe slightly more complicated) function call, but at least the first one is impossible to do that way. Unlike the numeric suffixes, this one I haven’t actually implemented a hacky version of, and I don’t know of any other languages that have an identical feature, so I can’t promise it’s feasible, but it seems like it should be.

On 27/08/2019 18:07, Andrew Barnert via Python-ideas wrote:

On Aug 27, 2019, at 08:52, Steven D'Aprano wrote:

...

...

On Tue, Aug 27, 2019 at 05:24:19AM -0700, Andrew Barnert via Python-ideas wrote:

There is a possibility in between the two extremes of “useless” and “complete monster”: the prefix accepts exactly one token, but can parse that token however it wants. How is that different from passing a string argument to a function or class constructor that can parse that token however it wants?

x'...'

x('...')

Unless there is some significant difference between the two, what does this proposal give us? Before I get into this, let me ask you a question. What does the j suffix give us? You can write complex numbers without it just fine:

c = complex c(1, 2)

And you can even write a j function trivially:

def j(x): return complex(0, x) 1 + j(2)

But would anyone ever write that when they can write it like this:

1 + 2j

I don’t think so. What does the j suffix give us? The two extra keystrokes are trivial. The visual noise of the parens is a bigger deal. The real issue is that this matches the way we conceptually think of complex numbers, and the way we write them in other contexts. (Well, the way electrical engineers write them; most of the rest of us use i rather than j… but still, having to use j instead of i is less of an impediment to reading 1+2j than having to use function syntax like 1+i(2).

You make the point yourself: this is something we already understand from dealing with complex numbers in other circumstances. That is not true of generic single-character string prefixes. -- Rhodri James *-* Kynesim Ltd

On Wed, Aug 28, 2019 at 3:10 AM Andrew Barnert via Python-ideas wrote:

Before I get into this, let me ask you a question. What does the j suffix give us? You can write complex numbers without it just fine:

c = complex c(1, 2)

And you can even write a j function trivially:

def j(x): return complex(0, x) 1 + j(2)

But would anyone ever write that when they can write it like this:

1 + 2j

I don’t think so. What does the j suffix give us? The two extra keystrokes are trivial. The visual noise of the parens is a bigger deal. The real issue is that this matches the way we conceptually think of complex numbers, and the way we write them in other contexts. (Well, the way electrical engineers write them; most of the rest of us use i rather than j… but still, having to use j instead of i is less of an impediment to reading 1+2j than having to use function syntax like 1+i(2).

And the exact same thing is true in 3D or CUDA code that uses a lot of float32 values. Or code that uses a lot of Decimal values. In those cases, I actually have to go through a string for implementation reasons (because otherwise Python would force me to go through a float64 and distort the values), but conceptually; there are no strings involved when I write this:

array([f('0.2'), f('0.3'), f('0.1')])

… and it would be a lot more readable if I could write it the same way I do in other programming languages:

array([0.2f, 0.3f, 0.1f])

Again, it’s not about saving 4 keystrokes per number, and the visual noise of the parens is an issue but not the main one (and quotes are barely any noise by comparison); it’s the fact that these numeric values look like numeric values instead of looking like strings

If your conclusion here were "and that's why Python needs a proper syntax for Decimal literals", then I would be inclined to agree with you - a Decimal literal would be lossless (as it can entirely encode whatever was in the source file), and you could then create the float32 values from those. But you haven't made the case for generic string prefixes or any sort of "arbitrary literal" that would let you import something that registers something to make your float32 literals. ChrisA

On Tuesday, August 27, 2019, 11:42:23 AM PDT, Serhiy Storchaka wrote:

27.08.19 20:07, Andrew Barnert via Python-ideas пише:

...

Before I get into this, let me ask you a question. What does the j suffix give us? You can write complex numbers without it just fine:

      c = complex       c(1, 2)

And you can even write a j function trivially:

      def j(x): return complex(0, x)       1 + j(2)

But would anyone ever write that when they can write it like this:

      1 + 2j

And it has its limitation. How would you write complex(-0.0, 1.0)?

And yet, despite that limitation, many people find it useful, and use it on a daily basis. Are you suggesting that Python would be better off without the `j` suffix because of that problem? 

But you haven't made the case for generic string prefixes or any sort of "arbitrary literal" that would let you import something that registers something to make your float32 literals.

The case can be made as follows: different people use different parts of the Python language. Andrew would love to see the support for decimals, fractions and float32s (possibly float16s too, and maybe even posit numbers). Myself, I miss datetime and regular expression literals. Other people on the 2013 thread argued at length in favor of supporting sql-literals, which would allow them to be used in a much safer manner. Then there are those who want to write complex numbers in a natural fashion, but they already got their wish granted. In short, the needs vary, and not all of the functionality belongs to the python standard library either.

On Aug 27, 2019, at 14:41, Chris Angelico wrote:

...

On Wed, Aug 28, 2019 at 6:03 AM Andrew Barnert wrote:

...

On Tuesday, August 27, 2019, 11:12:51 AM PDT, Chris Angelico wrote: If your conclusion here were "and that's why Python needs a proper syntax for Decimal literals", then I would be inclined to agree with you - a Decimal literal would be lossless (as it can entirely encode whatever was in the source file), and you could then create the float32 values from those.

I think builtin Decimal literals are a non-starter. The type isn't even builtin.

Not sure that's a total blocker, but in any case, I'm not arguing for that - I'm just saying that everything up to that point in your argument would be better served by a Decimal literal than by any notion of "custom literals".

No, it really couldn’t. A builtin Decimal literal would arguably serve the Decimal use case better (but I’m not even sure about that one; see below), but it doesn’t serve the float32 case that you’re responding to.

...

But they're not. You didn't even attempt to answer the comparison with complex that you quoted. The problem that `j` solves is not that there's no way to create complex values losslessly out of floats, but that there's no way to create them _readably_, in a way that's consistent with the way you read and write them in every other context. Which is exactly the problem that `f` solves. Adding a Decimal literal would not help that at all—letting me write `f(1.23d)` instead of `f('1.23')` does not let me write `1.23f`.

TBH I don't quite understand the problem. Is it only an issue with negative zero? If so, maybe you should say so, because in every other way, building a complex out of a float added to an imaginary is perfectly lossless.

Negative zero is an irrelevant side issue that Serhiy brought up. It means j is not quite perfect—and yet j is still perfectly usable despite that. Ignore negative zero. The problem that j solves is dead simple: 1 + 2j is more readable than complex(1, 2). And it matches what you write and read in other contexts besides Python. That’s the only problem j solves. But it’s a problem worth solving, at least for code that uses a lot of complex numbers. Without it, even if you wanted to pollute the namespace with a single-letter global so you could write c(1, 2) or 1 + j(2), it _still_ wouldn’t be nearly as readable or as familiar. That’s why we have j. There is literally no other benefit, and yet it’s enough. And the problem that f solves would be exactly the same: 1.23f is more readable than calling float32, and it matches what you read and write in other contexts besides Python (like, say, C or shader code). Even if you wanted to pollute the namespace with a single-letter global f, it still wouldn’t be as readable or as familiar. That’s why we should have f. There is literally no other benefit, but I think it’s enough benefit, for enough programs, that we should be allowed to do it. Just like j. Unlike j, however, I don’t think it’s useful in enough programs that it should be builtin. And I think the same is probably true for Decimal. And for most of the other examples that have come up in this thread. Which is why I think we’d be better served with something akin to C++ allowing you to explicitly register affixes for your specific program, than something like C with its too-big-to remember-but-still-not-enough-for-many-uses zoo of builtin affixes.

...

Also, as the OP has pointed out repeatedly and nobody has yet answered, if I want to write `f(1.23d)` or `f('1.23')`, I have to pollute the global namespace with a function named `f` (a very commonly-used name); if I want to write `1.23f`, I don't, since the converter gets stored in some out-of-the-way place like `__user_literals_registry__['f']` rather than `f`. That seems like a serious benefit to me.

Maybe. But far worse is that you have a very confusing situation that this registered value could be different in different programs.

Sure, and the global f could also be different in different programs—or even in different modules in the same program. So what? 1.23f would always have the same meaning everywhere, it’s just that the meaning is something like __user_literals__['f']('1.23') instead of globals()['f']('1.23'). Yes, of course that is something new to be learned, if you’re looking at a program that does a lot of 3D math, or a lot of decimal math, or a lot of Windows path stuff, or whatever, people are likely to have used this feature so you’ll need to know how to look up the f or d or whatever. But that really isn’t a huge hardship, and I think the benefits outweigh the cost.

In contrast, f(1.23d) would have the same meaning everywhere: call a function 'f' with one parameter, the Decimal value 1.23. Allowing language syntax to vary between programs is a mess that needs a LOT more justification than anything I've seen so far.

This doesn’t really allow syntax to vary between programs. It just allows literals to be followed (or maybe preceded) by tags. The rest of the syntax is unchanged.

...

...

But you haven't made the case for generic string prefixes or any sort of "arbitrary literal" that would let you import something that registers something to make your float32 literals.

Sure I did; you just cut off the rest of the email that had other cases.

Which said basically the same as the parts I quoted.

Not even remotely—again, unless you think that Windows paths could somehow be served by builtin decimal literals?

...

And ignored most of what you quoted about the float32 case.

What did I ignore?

That 1.23f is more readable, familiar, etc. in exactly the same way that 2.3j is.

...

And ignored the previous emails by both me and the OP that had other cases. Or can you explain to me how a builtin Decimal literal could solve the problem of Windows paths?

All the examples about Windows paths fall into one of two problematic boxes:

1) Proposals that allow an arbitrary prefix to redefine the entire parser - basically impossible for anything sane

2) Proposals that do not allow the prefix to redefine the parser, and are utterly useless, because the rest of the string still has to be valid.

3) Proposals that do not allow the prefix to redefine the parser for the entire program, but do allow it to manually parse anything the tokenizer can recognize as a single (literal) token. As I said, I haven’t tried to implement this example as I have with the other examples, so I can’t promise that it’s doable (with the current tokenizer, or with a reasonable change to it). But if it is doable, it’s neither insane nor useless. (And evenif it’s not doable, that’s just two examples that affixes can’t solve—Windows paths and general “super-raw strings”. They still solve all of the other examples.)

...

Here's a few more: Numeric types that can't be losslessly converted to and from Decimal, like Fraction.

If you want to push for Fraction literals as well, then sure. But that's still very very different from *arbitrary literal types*.

If we really only ever needed Decimal and Fraction, then yes, I think allowing user-defined literal tags would be better than adding two hard-coded tags that most people will rarely use.

...

I think your reluctance and the OP's excitement here both come from the same source: Any feature that gives you a more convenient way to write and read something is good, because it lets you write things in a way that's consistent with your actual domain, and also bad, because it lets you write things in a way that's not readable to people who aren't steeped in your domain. Those are _always_ both true, so just arguing from first principles is pointless. The question is whether, for this specific feature, there are good uses where the benefit outweighs the cost. And I think there are.

That line of argument is valid for anything that is specifically defined by the language.

Yes, and? “Literal token” is specifically defined by the language. “Literal token with attached tag” will also be specifically defined by the language. The only thing open to customization is what that token gets compiled to. (Or course this is just one suggestion I came up with, not the only way to do things, or what the OP suggested. But it does show that there is at least one possibility besides “insane” and “useless”.) Your argument comes down to the fact that anything that could possibly be construed as affecting syntax, no matter how constrained it is, and no matter how far you have to stretch to see it as affecting syntax in the first place, and even if nobody would ever do that with it, is Inherently so evil that it can’t possibly be allowed. I think that’s a silly argument. Especially in a language that already has a nicely documented feature like, say, import hooks.

...

In fact, if you're already convinced that we need Decimal literals, unless you can come up with a more feasible way to add builtin Decimal literals to Python, Decimal on its own seems like a sufficient use case for the feature.

There are valid use cases for Decimal literals and Fraction literals, but not, IMO, for custom literals. Look at some of the worst abuses of #define in C to get an idea of what syntax customization can do to readability.

Look at the plethora of suffixes C has for number and character literals. Look at how many things people still can’t do with them that they want to. Look at the way user literals work in C++. While technically you can argue that they are “syntax customization”, in practice the customization is highly constrained. Is it _impossible_ to use that feature to write code that can’t be parsed by a human reader? I don’t know if I could prove that it’s impossible. However, I do know that it’s not easy. And that none of the examples, or real-life uses, that I’ve seen have done so. Do you think Python users are incapable of the kind of restraint and taste shown by C++ users, and therefore we can’t trust Python users with a tool that might possibly (but we aren’t sure) if abused badly enough make code harder to visually parse?

Thanks, Andrew, you're able to explain this much better than I do. Just wanted to add that Python *already* has ways to grossly abuse its syntax and create unreadable code. For example, I can write >>> о = 3 >>> o = 5 >>> ο = 6 >>> (о, o, ο) (3, 5, 6) But just because some feature CAN get abused, doesn't mean it ACTUALLY gets abused in practice. People want to write nice, readable code, because they will ultimately be the ones to support it.

On Aug 27, 2019, at 18:59, Steven D'Aprano wrote:

On Tue, Aug 27, 2019 at 10:07:41AM -0700, Andrew Barnert wrote:

...

...

How is that different from passing a string argument to a function or class constructor that can parse that token however it wants?

x'...'

x('...')

Unless there is some significant difference between the two, what does this proposal give us?

...

Before I get into this, let me ask you a question. What does the j suffix give us?

I'm going to answer that question, but before I answer it, I'm going to object that this analogy is a poor one. This proposal is *in no way* a proposal for a new compile-time literal.

Yes, you’re the same person who got hung up on the fact that these affixes don’t really give us “literals” back in either 2013 or 2016, and I don’t want to rehash that argument. I could point out that nobody cares that -1 isn’t really a literal, and almost nobody cares that the CPython optimizer special-cases its way around that, and the whole issue with Python having three different definitions of “literal” that don’t coincide, and so on, but we already had this conversation and I don’t think anyone but the two of us cared. What matters here is not whether things like the OP’s czt'abc' or my 1.23f or 1.23d are literals to the compiler, but whether they’re readable ways to enter constant values to the human reader. If so, they’re useful. Period. Now, it’s possible that even though they’re useful, the feature is still not worth adding because of Chris’s issue that it can be abused, or because there’s an unavoidable performance cost that makes it a bad idea to rely on them, or because they’re not useful in _enough_ code to be worth the effort, or whatever. Those are questions worth discussing. But arguing about whether they meet (one of the three definitions of) “literal” is not relevant.

This proposal is for mere syntactic sugar allowing us to drop the parentheses from a tiny subset of function calls, those which take a single string argument.

And to drop the quotes as well. And to avoid polluting the global namespace with otherwise-unused one-character function names. Can you honestly tell me that you see no significant readability difference between these examples: vec = [1.23f, 2.5f, 1.11f] vec = [f('1.23'), f('2.5'), f('1.11')] I think anyone would agree that the former is a lot more readable. Sure, you have to learn what the f suffix means, but once you do, it means all of the dozens of constants in the module are more readable. (And of course most people reading this code will probably be people who are used to 3D code and already _expect_ that format, since that’s how you write it in C, in shaders, etc.)

And even then, only when the argument is a string literal:

czt'abc' # Okay.

s = 'abc' czt's' # Oops, wrong, doesn't work.

Sure, just like you can’t apply an r or f prefix to a string expression.

But, to answer your question, what does the j suffix give us?

Damn little. Unless there is a large community of Scipy and Numpy users who need complex literals, I suspect that complex literals are one of the least used features in Python.

I do a lot of maths in Python, and aside from experimentation in the interactive interpreter, I think I can safely say that I have used complex literals exactly zero times in code.

I don’t think your experience here is typical. I can’t think of a good way to search GitHub python repos for uses of j, but a hacky search immediately turned up this numpy issue:https://github.com/numpy/numpy/issues/13179:

A fast way to get the inverse of angle, i.e., exp(1j * a) = cos(a) + 1j * sin(a). Note that for large angle arrays, exp(1j*a)needlessly triples memory use…

That doesn’t prove that people actually call it with `1j * a` instead of `complex(0, a)`, but it does seem likely.

...

You can write complex numbers without it just fine: [...]

Indeed. And if we didn't already have complex literals, would we accept a proposal to add them now? I doubt it.

I’m not sure. I assume you’d be against it, but I suspect that most of the people who use it today would be for it. But if we had custom affixes, I think everyone would be happy with “just define a custom j suffix”. Would anyone really argue that they need the performance benefit or compile-time handling? How often do you evaluate zillions of constants in the middle of a tight loop? And what other argument would there be for adding it to the grammar and the compiler and forcing every project to use it? Which is exactly what I think of the Decimal and Fraction suffixes, contrary to what Chris says. There will be a small number of projects than get a lot of readability benefit, but every other project gains nothing, so why add it as a builtin for every project? And I don’t see why float32 is any different from Decimal and Fraction, given that the actual problem is not lossless values but readability, so a builtin Decimal suffix wouldn’t help there.

But if you think we would, how about a proposal to add quaternions?

q = 3 + 4i + 2j - 7k

I’m sure you can guess my answer to that: most projects don’t need it, so there should definitely not be builtin suffixes for it. But if we have custom suffixes, it anyone _does_ need it, they can do it trivially, without having to bother the rest of us asking for it.

...

I don’t think so. What does the j suffix give us? The two extra keystrokes are trivial. The visual noise of the parens is a bigger deal.

I don't think it is.

You’ve cut off my paragraph to make it misleading. The visual noise of the parens _is_ a bigger deal than the two extra keystrokes, but as I said in the very next sentence, it’s still not the point of the feature. The real big deal is that it lets you write complex numbers in a way that looks like complex numbers. I don’t see any benefit in arguing about whether the “bigger deal but still not the point” is actually a bigger deal or not, because who cares?

I think the big deals in this proposal are:

- you have something that looks like a kind of string czt'...' but is really a function call that might return absolutely anything at all;

It doesn’t return “anything at all”, any more than a function returns “anything at all”. It returns something consistent that has a specific meaning in your project. I don’t know what czt means a priori, but if I were reading the OP’s code, I could look it up, and then I would know. And I could assume that, unless the author is an idiot, an affix on a string literal is going to be something stringy, and an affix on a number literal is going to be something numbery. Sure, you _could_ violate that assumption, but that’s no different from the fact that you could write a function called sqrt(n) that returns an iterable of the contents of all the files in the nth sub directory of $HOME. You’re not going to do that. (Or, if you do, I’m going to stop reading your code.)

- you encourage people to write cryptic single-character functions, like v(), x(), instead of meaningful expressions like Version() and re.compile();

Do you honestly not see the readability benefit in a bunch of constants that all look like `2.3d` instead of `Decimal('2.3')`? Do you honestly think that `D('2.3')` is just as good as `2.3d`, and also worth using up a scarce resource (one-letter global names) for? If not, then I don’t see why you’re pretending not to see the benefit of the proposal.

- the OP still hasn't responded to my question about the ambiguity of the proposal (is czt'...' a one three-letter prefix, or three one-letter prefixes?)

Well, this is why I wanted him to get into more details than his initial 10000-feet-vision thing. I honestly see a lot more use for numeric affixes than string ones, and for suffixes rather than prefixes, and for just one suffix per value rather than some rule for combining them. But I know that last time around (or maybe the time before), a sql prefix was the thing that got the most people excited, and I could see wanting to combine that with raw or not, and so on, so I’d like to see a concrete proposal on how all of that works.

all of which *hugely* outweighs the gain of being able to avoid a pair of parentheses.

Which, you’ll note, I already said was not the point of the proposal.

...

And the exact same thing is true in 3D or CUDA code that uses a lot of float32 values

Indeed, but this proposal doesn't help you here. You still have to write strings.

No I don’t. I write `1.23f`, just like I do in C or in GLSL, exactly what I want to write, and read. In my version of the proposal (which I described in my first email in the thread—including a link to a hacky import hook that implements it that I wrote up last time this subject came up in 2015), the parser sees a literal token (any kind of literal, not just string literals) followed by a string of identifier characters (that weren’t munched by that literal), it looks up that string, and calls the looked-up function with the raw text of the literal token. The OP’s second email in the thread incorporated my idea into his existing idea. His version is more complicated than mine because it handles prefixes as well as suffixes, and it doesn’t have a proof of concept to verify that it’s all doable unambiguously, but it still allows me to write `1.23f`.

What you want is a float32 literal, let's say 1.23f

I don’t care whether it’s an actual literal. I do care that I can write it as `1.23f`. And both my proposal and the OP’s allow that, so I’m happy with either.

While I'm sympathetic, and I'd like to see a Decimal literal, I doubt that there's enough use-cases outside of specialists like yourself for 16- or 32-bit floats to justify making them built-ins with literal syntax. Python is not Julia :-)

See, I would _not_ like to see a builtin Decimal literal. Or float16 or float32 or fixed1616, or Fraction, or _any_ other new kinds of literals. As long as I can use `1.23f` to mean `float32('1.23')` in the projects where I have a mess of float32 constants (and maybe copy-paste them into my REPL from a shader or a C debugger session), I’m happy. And I think user-defined affixes are a better way to get that than trying to convince everyone that Python should add a builtin float32 type and all the math that goes with it and then add a suffix for float32 literals. Not because I doubt I could convince anyone of that, not because I’d have to wait 5 years before I could start using it even if I could, but because it’s simply not right in the first place. Python should not have a builtin float32 type. And therefore, Python should not have a builtin `f` suffix. And even if that weren’t an issue, I’d _still_ rather have a custom affix feature rather than a mess of new builtin ones. If I run into an unfamiliar affix in your code, I’d rather look it up in your project that consult a table with a mess of builtin affixes. If I want `d` for Decimal in one project, why should that mean nobody can ever use `d` for something different in a project that doesn’t do any decimal math but does a whole bunch of… something else I can’t think of off the top of my head, but I’m sure there will be at least one suffix that has two good meanings in widely different uses. Just like the namespace for builtin functions, the namespace for builtin affixes is—and should be—a limited resource. And meanwhile, what would I get from `d` being builtin? I can save one import or register call or whatever per project. My program that takes 80 seconds to run starts up 2us faster because a few dozen (or at most hundred) constructed constants can be stored in the .pyc file. I don’t have to watch my speech to carefully avoid using the word “literal” imprecisely. None of those are worth anywhere near as much to me as being able to have the suffixes I want for the project, even if I just thought of them today—and to _not_ have the ones I _don’t_ want, even is some of them have a broader niche.

On Wed, Aug 28, 2019 at 2:40 PM Andrew Barnert wrote:

...

People can be trusted with powerful features that can introduce complexity. There's just not a lot of point introducing a low-value feature that adds a lot of complexity.

But it really doesn’t add a lot of complexity.

If you’re not convinced that really-raw string processing is doable, drop that.

Since the OP hasn’t given a detailed version of his grammar, just take mine: a literal token immediately followed by one or more identifier characters (that couldn’t have been munched by the literal) is a user-suffix literal. This is compiled into code that looks up the suffix in a central registry and calls it with the token’s text. That’s all there is to it.

What is a "literal token", what is an "identifier character", and how does this apply to your example of having digits, a decimal point, and then a suffix? What if you want to have a string, and what if you want to have that string contain backslashes or quotes? If you want to say that this doesn't add complexity, give us some SIMPLE rules that explain this. And make absolutely sure that the rules are identical for EVERY possible custom prefix/suffix, because otherwise you're opening up the problem of custom prefixes changing the parser again.

Compare that adding Decimal (and Fraction, as you said last time) literals when the types aren’t even builtin. That’s more complexity, for less benefit. So why is it better?

Actually no, it's a lot less complexity, because it's all baked into the language. You don't have to have the affix registry to figure out how to parse a script into AST. The definition of a "literal" is given by the tokenizer, and for instance, "-1+2j" is not a literal. How is this going to impact your registry? The distinction doesn't matter to Decimal or Fraction, because you can perform operations on them at compile time and retain the results, so "-1.23d" would syntactically be unary negation on the literal Decimal("1.23"), and -4/5f would be unary negation on the integer 4 and division between that and Fraction(5). But does that work with your proposed registry? What is a "literal token", and would it need to include these kinds of things? What if some registered types need to include them and some don't? ChrisA

On Tue, Aug 27, 2019 at 05:13:41PM -0000, stpasha@gmail.com wrote:

The difference between `x'...'` and `x('...')`, other than visual noise, is the following:

- The first "x" is in its own namespace of string prefixes. The second "x" exists in the global namespace of all other symbols.

Ouch! That's adding a lot of additional complexity to the language. Python's scoping rules are usually described as LEGB: - Local - Enclosing (non-local) - Global (module) - Builtins but that's an over-simplification, dating back to something like Python 1.5 days. Python scope also includes: - class bodies can be the local scope, but they don't work quite the same as function locals); - parts of the body of comprehensions behave as if they were a seperate scope. This proposal adds a completely seperate, parallel set of scoping rules for these string prefixes. How many layers in this parallel scope? The simplest design is to have a single, interpreter wide namespace for prefixes. Then we will have name clashes, especially since you seem to want to encourage single character prefixes like "v" (verbose, version) or "d" (date, datetime, decimal). Worse, defining a new prefix will affect all other modules using the same prefix. So we need a more complex parallel scope. How much more complex? * if I define a string prefix inside a comprehension, function or class body, will that apply across the entire module or just inside that comp/func/class? * how do nested functions interact with prefixes? * do we need a set of parallel keywords equivalent to global and nonlocal for prefixes? If different modules have different registries, then not only do we need to build a parallel set of scoping rules for prefixes into the interpreter, but we need a parallel way to import them from other modules, otherwise they can't be re-used. Does "from module import x" import the regular object x from the module namespace, or the prefix x from the prefix-namespace? So it seems we'll need a parallel import system as well. All this adds more complexity to the language, more things to be coded and tested and documented, more for users to learn, more for other implementations to re-implement, and the benefit is marginal: the ability to drop parentheses from some but not all function calls. Now consider another problem: introspection, or the lack thereof. One of the weaknesses of string prefixes is that it's hard to get help for them. In the REPL, we can easily get help on any class or function: help(function) and that's really, really great. We can use the inspect module or dir() to introspect functions, classes and instances, but we can't do the same for string prefixes. What's the difference between r-strings and u-strings? help() is no help (pun intended), since help sees only the string instance, not the syntax you used to create it. All of these will give precisely the same output: help(str()) help('') help(u'') help(r"") etc. This is a real weakness of the prefix system, and will apply equally to custom prefixes. It is *super easy* to introspect a class or function like Version; it is *really hard* to do the same for a prefix. You want this seperate namespace for prefixes so that you can have an v prefix without "polluting" the module namespace with a v function (or class). But v doesn't write itself! You still have to write a function or class, athough you might give it a better name and then register it with the single letter prefix: @register_prefix('v') class Version: ... (say). This still leaves Version lying around in your global namespace, unless you explicitly delete it: del Version but you probably won't want to do that, since Version will probably be useful for those who want to create Version objects from expressions or variables, not just string literals. So the "pollution" isn't really pollution at all, at least not if you use reasonable names, and the main justification for parallel namespaces seems much weaker. Let me put it another way: parallel namespaces is not a feature of this proposal. It is a point against it.

- Python style discourages too short variable names, especially in libraries, because they have increased chance of clashing with other symbols, and generally may be hard to understand. At the same time, short names for string prefixes could be perfectly fine: there won't be too many of them anyways.

That's an interesting position for the proponent of a new feature to take. "Don't worry about this being confusing, because hardly anyone will use it."

The standard prefixes "b", "r", "u", "f" are all short, and nobody gets confused about them.

Plenty of people get confused about raw strings. There's only four, plus uppercase and combinations, and they are standard across the entire language. If there were dozens of them, coming from lots of different modules and third-party libraries, with lots of conflicts ('v' for version in foolib, but 'v' for verbose in barlib), the situation would be very different. We can't extrapolate from four built-in prefixes being manageable to concluding that dozens of clashing user-defined prefixes will be too.

- Barrier of entry. Today you can write `from re import compile as x` and then write `x('...')` to denote a regular expression (if you don't mind having `x` as a global variable). But this is not the way people usually write code.

I doubt that is true. "from module import foo as bar" is a standard, commonly used Python language feature: https://stackoverflow.com/questions/22245711/from-import-or-import-as-for-mo... in particular this answer here: https://stackoverflow.com/a/29010729 Besides, we don't design the language for the least knowledgable, most ignorant, copy-and-paste coders.

People write the code the way they are taught from examples, and the examples don't speak about regular expression objects. The examples only show regular expressions-as-strings

That's simply wrong. The *very first* example of a regular expression here: https://scotch.io/tutorials/an-introduction-to-regex-in-python uses the compile function. More examples talking about regex objects: https://docs.python.org/3/library/re.html#re.compile https://pymotw.com/2/re/#compiling-expressions https://docs.python.org/3/howto/regex.html#compiling-regular-expressions https://stackoverflow.com/questions/20386207/what-does-pythons-re-compile-do These weren't hard to find. You don't have to dig deep into obscure parts of the WWW to find people talking about regex objects. I think you underestimate the knowledge of the average Python programmer. -- Steven

On 28 Aug 2019, at 14:09, Piotr Duda wrote:

śr., 28 sie 2019 o 13:18 Steven D'Aprano napisał(a):

...

...

On Tue, Aug 27, 2019 at 05:13:41PM -0000, stpasha@gmail.com wrote:

The difference between `x'...'` and `x('...')`, other than visual noise, is the following:

- The first "x" is in its own namespace of string prefixes. The second "x" exists in the global namespace of all other symbols.

Ouch! That's adding a lot of additional complexity to the language.

There is much simpler solution, just make `abc"whatever"` synatctic sugar for `string_literal_abc(r"whatever", closure)` where closure is object that allow read only access to variables in call site.

So to use abc"foo" we must import string_literal_abc? Seems pretty confusing to me!

On Wed, 28 Aug 2019 at 13:15, Anders Hovmöller wrote:

...

On 28 Aug 2019, at 14:09, Piotr Duda wrote:

...

There is much simpler solution, just make `abc"whatever"` synatctic sugar for `string_literal_abc(r"whatever", closure)` where closure is object that allow read only access to variables in call site.

So to use abc"foo" we must import string_literal_abc? Seems pretty confusing to me!

The only sane proposal that I can see (assuming that no-one is proposing to drop the principle that Python shouldn't have mutable syntax) is to modify the definition stringliteral ::= [stringprefix](shortstring | longstring) stringprefix ::= "r" | "u" | "R" | "U" | "f" | "F" | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | "Rf" | "RF" to expand the definition of <stringprefix> to allow any identifier-like token (precise details to be confirmed). Then, if it's one of the values enumerated above (you'd also need some provison for special-casing bytes literals, which are in a different syntax rule), work as at present. For any other identifier-like token, you'd define TOKEN(shortstring|longstring) as being equivalent to TOKEN(r(shortstring|longstring)) I.e., treat the string as a raw string, and TOKEN as a function name, and compile to a function call of the named function with the raw string as argument. That's a well-defined proposal, although whether it's what people want is a different question. Potential issues: 1. It makes a whole class of typos that are currently syntax errors into runtime errors - fru"foo\and {bar}" is now a function call rather than a syntax error (it was never a raw Unicode f-string, even though someone might think it was and be glad to be corrected by the current syntax error...) 2. It begs the question of whether people want raw-string semantics - whilst it's the most flexible option, it does mean that literals wanting to allow escape sequences would need to implement it themselves. 3. It does nothing for the edge case that a trailing \ isn't allowed - p"C:\" wouldn't be a valid Path literal. There are of course other possible proposals, but we'd need more than broad statements to make sense of them (specifically, either "exactly *what* new syntax are you suggesting we allow?", or "how are you proposing to allow users to alter Python syntax on demand?") Paul

On Wed, Aug 28, 2019 at 10:50 PM Rhodri James wrote:

On 28/08/2019 02:38, stpasha@gmail.com wrote:

...

Thanks, Andrew, you're able to explain this much better than I do. Just wanted to add that Python*already* has ways to grossly abuse its syntax and create unreadable code. For example, I can write

>>> о = 3 >>> o = 5 >>> ο = 6 >>> (о, o, ο) (3, 5, 6)

OK, I'll bite: how? If you were using "thing.o" I would believe you were doing something unhelpful with properties, but just "o"?

'\u043e' CYRILLIC SMALL LETTER O 'o' LATIN SMALL LETTER O '\u03bf' GREEK SMALL LETTER OMICRON Virtually indistinguishable in most fonts, but distinct characters. It's the same thing you can do with "I" and "l" in many fonts, or "rn" and "m" in some, but taken to a more untypable level. ChrisA

On Wed, 28 Aug 2019 at 15:55, Mike Miller wrote:

On 2019-08-28 01:05, Paul Moore wrote:

...

However, I know of almost no examples of other languages that have added*user-definable* literal types (with the notable exception of

Believe there is such a feature in modern JavaScript:

https://developers.google.com/web/updates/2015/01/ES6-Template-Strings#tagge...

Interesting - thanks for the pointer! Paul

On Aug 28, 2019, at 00:40, Chris Angelico wrote:

On Wed, Aug 28, 2019 at 2:40 PM Andrew Barnert wrote:

...

...

People can be trusted with powerful features that can introduce complexity. There's just not a lot of point introducing a low-value feature that adds a lot of complexity.

But it really doesn’t add a lot of complexity.

If you’re not convinced that really-raw string processing is doable, drop that.

Since the OP hasn’t given a detailed version of his grammar, just take mine: a literal token immediately followed by one or more identifier characters (that couldn’t have been munched by the literal) is a user-suffix literal. This is compiled into code that looks up the suffix in a central registry and calls it with the token’s text. That’s all there is to it.

What is a "literal token", what is an "identifier character",

Literals and identifier characters are already defined today, so I don’t need new definitions for them. The existing tokens are already implemented in the tokenizer and in the tokenize module, which is why I was able to slap together multiple variations on a proof of concept 4 years ago in a few minutes as a token-stream-processing import hook. My import hook version is a hack, of course, but it serves as a counter to your argument that there’s no simple thing that could work by being a dead simple thing that does work. And there’s no reason to believe a real version wouldn’t be at least as simple.

and how does this apply to your example of having digits, a decimal point, and then a suffix

We add a`suffixedfloatnumber` production defined as `floatnumber identifier`. So, the `2.34` parses as a `floatnumber` the same as always. That `d` can’t be part of a `floatnumber`, but it can be the start of an `idenfifier`, and those two nodes together can make up a `suffixedfloatnumber`. No need for any new lookahead or other context. And for the concrete implementation in CPython, it should be obvious that the suffix can be pushed down into the tokenizer, at which point the parse becomes trivial. If you’re asking how my hacky version works, you could just read the code, which is simpler than an explanation, but here goes (from memory, because I’m on my phone): To the existing tokenizer, `d` isn’t a delimiter character, so it tries to match the whole `2.34d`. That doesn’t match anything. But `2.34` does match something, etc., so ultimately it emits two tokens, `floatnumber('2.34'), error('d')`. My import hook reads the stream of tokens. When it sees a `floatnumber` followed by an `error`, it checks whether the error body could be an identifier token. If so, it replaces those two tokens in the steam with… I forget, but probably I just hand-parsed the lookup and call and emit the tokens for that. I can’t _guarantee_ that the real version would be simpler until I try it. And I don’t want to hijack the OP’s thread and replace his proposal (which does give me what I want) with mine (which doesn’t give him what he wants), unless he abandons the idea of attempting to implement his version. But I’m pretty confident it would be as simple as it sounds, which is even simpler than the hacky version (which, again, is dead simple and works today). And most variations on the idea you could design would be just as simple. Maybe the OP will perversely design one that isn’t. If so, it’s his job to show that it can be implemented. And if he gives up, then I’ll argue for something that I can implement simply. But I don’t think that’s even going to come up.

What if you want to have a string, and what if you want to have that string contain backslashes or quotes? If you want to say that this doesn't add complexity, give us some SIMPLE rules that explain this.

Well, that works exactly the same way a string does today (including the optional r prefix). The closing quote can now be followed by a string of identifier characters, but everything up to there is exactly the same as today. So, it doesn’t add any complexity, because it uses the same rules as today. I did suggest, as a throwaway addon to the OP’s proposal, that you could instead do raw strings or even really-raw (the string ends at the first matching quote; backslashes mean nothing). I don’t know if he wants either of those, but if he does, raw string literals are already defined in the grammar and implemented in the tokenizer, and really-raw is an even simpler grammar (identical to the existing grammar except that instead of `longstringchar | stringescapeseq` there’s a `<any source character except the quote>` node, and the same for `shortstringitem`).

And make absolutely sure that the rules are identical for EVERY possible custom prefix/suffix,

Well, in my version, since the rule for suffixedstringliteral is just `stringliteral identifier`, of course it’s the same for every possible suffix; there’s no conceivable way it could be different. If the OP wants to propose something more complicated that provides some way of selecting different rules, he could, but I don’t think he has, and if he doesn’t, then the issue will be equally nonexistent. I don’t know whether he wants to interact with the existing string prefixes (or, if so, how that works), or always do normal strings, or always do really-raw strings, or what, but there are multiple plausible designs, most of which are not impossible or even complicated, so the fact that you can imagine that there might be a design that would be impossible really isn’t relevant. Just to show how easy it is to come up with something (but which, again, may not be what the OP actually wants here): a stringliteral is now a stringprefix followed by shortstring or longstring (as today) or an identifier followed by rrshortstring or rrlongstring. The rr tokens are defined as I described above: they end at the first matching quote, no backslashing. This option would have some limitations— people can’t use \” to escape quotes in prefixed strings, there’s no way to get prefixed bytes, you probably can’t call a prefix “bub”… does that make some of the OP’s desired use cases or some of the 2013 use cases no longer viable? I don’t know. If so, the OP presumably won’t use this option and will use a different one. Any option will have some limitations, and I don’t know which one he wants, but there are a huge number of simple, and nonmagical, options that he could pick.

...

Compare that adding Decimal (and Fraction, as you said last time) literals when the types aren’t even builtin. That’s more complexity, for less benefit. So why is it better?

Actually no, it's a lot less complexity, because it's all baked into the language.

Making the language definition and the interpreter and the compiler more complicated doesn’t eliminate the complexity, it just moves it somewhere else.

You don't have to have the affix registry to figure out how to parse a script into AST.

You don’t need the registry to parse to an AST for my proposal either; it’s only used at runtime. And, while the OP didn’t give us a grammar, he did give us proposed bytecode output of (one version of) his idea, and it’s pretty obvious that the registry isn’t getting involved until the interpreter eval loop processes the new registry-lookup opcode, so it clearly isn’t involved in parsing. And why would it get involved in parsing? It’s not like someone is proposing Rust or Dylan macros here.

The definition of a "literal" is given by the tokenizer, and for instance, "-1+2j" is not a literal. How is this going to impact your registry?

Not at all. Why would it?

The distinction doesn't matter to Decimal or Fraction, because you can perform operations on them at compile time and retain the results, so "-1.23d" would syntactically be unary negation on the literal Decimal("1.23"), and -4/5f would be unary negation on the integer 4 and division between that and Fraction(5). But does that work with your proposed registry?

Yes, of course it does. That should be obvious from the fact that I said that `1/2F` would end up equivalent to `1/Fraction(2)`. Concretely, it ends up as something like `1/sys.__user_suffixes__['F']('2')` Except probably with nicer error handling, so you don’t get a KeyError for an unknown suffix. Notice that the only thing looked up in the registry is the function to process the text, and this doesn’t need to happen until runtime, long after the code has been not just parsed, but compiled. Of course the OP’s version will be a little different. He wants to handle both prefixes and suffixes by looking up the prefix and passing the suffix as an second argument. And I’m not sure what exactly he wants as the main argument. But I still don’t see any reason it would need to look in the registry at tokenizer or parse or compile time. And again, his proposed bytecode translation implies that it doesn’t do so. So why imagine that it has to when there’s no visible reason for it?

What is a "literal token", and would it need to include these kinds of things?

How could this not be obvious? I deliberately chose the phrase “literal token”, and you clearly understand what this means because you invoked that meaning just one paragraph above. I also provided a link to a hacky implementation that blatantly relies on the tokenizer’s current processing of literals. And I gave examples that make it clear that `2` is a literal token and `1/2` is not. So why do you even need to ask whether `-4/5` is one? How could `-4/5` possibly be a literal token it '1/2` is not, or if it isn’t a token at all?

What if some registered types need to include them and some don't?

They can’t. The simple rule works for every numeric example everyone has come up with so far, even Steven’s facetious quaternion example that he proposed as too ridiculous for anyone to actually want. Is it a flaw that there may or may not be some examples that nobody has been able to think of that might work with a much more complicated feature but won’t work with this feature? Of course not. That’s true for every feature ever. There’s no reason to ignore the obvious simple design and try to imagine more complicated designs that may or may not solve additional problems that nobody’s even imagined just so you can dismiss the idea as too complicated.

In addition, there is the question of how user-defined literals would get turned into constants within the code.

So, I'm just brainstorming here, but how about the following approach: - Whenever a compiler sees `abc"def"`, it creates a constant of the type `ud_literal` with fields `.prefix="abc"`, `.content="def"`. - When it compiles a function then instead of `LOAD_CONST n` op it would emit `LOAD_UD_CONST n` op. - This new op first checks whether its argument is a "ud_literal", and if so calls the '.resolve()` method on that argument. The method should call the prefix with the content, producing an object that the LOAD_UD_CONST op stores back in the `co_consts` storage of the function. It is a TypeError for the resolve method to return another ud_literal. - Subsequent calls to the LOAD_UD_CONST op will see that the argument is no longer a ud-literal, and will return it as-is. This system would allow each constant to be evaluated only once and subsequently memoized, and only compute those constants that will actually be used.

all of which hugely outweighs the gain of being able to avoid a pair of parentheses.

Thank you for summarizing the main objections so succinctly, otherwise it becomes too easy to get lost in the discussion. Let me try to answer them as best as I can:

you have something that looks like a kind of string czt'...' but is really a function call that might return absolutely anything at all;

This is kinda the whole point. I understand, of course, how the idea of a string-that-is-not-a-string may sound blasphemous, however I invite you to look at this from a different perspective. Today's date is 2019-08-28. The date is a moment in time, or perhaps a point in the calendar, but it is certainly not a string. How do we write this date in Python? As `datetime("2019-08-28")`. We are forced to put the date into a string and pass that string into a function to create an actual datetime object. With this proposal the code would look something like `dt"2019-08-28"`. You're right, it's not a string anymore. But it *should not* have been a string to begin with, we only used a string there because Python didn't offer us any other way. Now with prefixed strings the justice is finally done: we are able to express the notion of <a specific date> directly. And the fact that it may still use strings under the hood to achieve the desired result is really an implementation detail, that may even change at some point in the future.

you have a redundant special case for calling functions that take a single argument, but only if that argument is a string literal;

There are many things in python that are in fact function calls in disguise. Decorators? function calls. Imports? function calls. Class definition? function call. Getters/setters? function calls. Attribute access? function calls. Even a function call is a function call via `__call__()`. I may be oversimplifying a bit, but the point is that just because something can be written as a function call doesn't mean it's the most natural way of doing it. Besides, there are use cases (such as `sql'...'`) where people do actually want to have a function that is constrained to string literals only. Having said that, prefixed (suffixed) strings (numbers) are not *exactly* equivalent to function calls. The points of difference are: - prefixes/suffixes are namespaced separately from regular variable names. - their results can be automatically memoized, bringing them closer to builtin literals.

you encourage people to write cryptic single-character functions, like v(), x(), instead of meaningful expressions like Version() and re.compile();

Which is why I suggested to put them in a separate namespace. You're right that function `v()` is cryptic and should be avoided. But a prefix `v"..."` is neither a function nor a variable, it's ok for it to be short. The existing string prefixes are all short after all.

you encourage people to defer parsing that could be efficiently done in your head at edit time into slow and likely inefficient string parsing done at runtime;

I don't encourage such thing, it's just that most often there is no other way. For example, consider regular expression `[0-9]+`. I can "parse it in my head" to understand that it means a sequence of digits, but how exactly am I supposed to convey this understanding to Python? Or perhaps I can parse "2019-08-28" in my head, and write in Python `datetime(year=2019, month=8, day=28)`. However, such form would greatly reduce readability of the code from humans' perspective. And human readability matters more than computer readability, for now. In fact, purely from the efficiency perspective, the prefixed strings can potentially have better performance because they are auto-memoized, while `datetime("2019-08-28")` needs to re-parse its input string every time (or add its own internal memoization, but even that would be less efficient because it doesn't know the input is a literal string).

the OP still hasn't responded to my question about the ambiguity of the proposal (is czt'...' a one three-letter prefix, or three one-letter prefixes?)

Sorry, I thought this part was obvious. It's a single three-letter prefix.

On Aug 28, 2019, at 01:05, Paul Moore wrote:

On Wed, 28 Aug 2019 at 05:04, Andrew Barnert via Python-ideas wrote:

...

What matters here is not whether things like the OP’s czt'abc' or my 1.23f or 1.23d are literals to the compiler, but whether they’re readable ways to enter constant values to the human reader.

If so, they’re useful. Period.

Now, it’s possible that even though they’re useful, the feature is still not worth adding because of Chris’s issue that it can be abused, or because there’s an unavoidable performance cost that makes it a bad idea to rely on them, or because they’re not useful in _enough_ code to be worth the effort, or whatever. Those are questions worth discussing. But arguing about whether they meet (one of the three definitions of) “literal” is not relevant.

Extended (I'm avoiding the term "custom" for now) literals like 0.2f, 3.14D, re/^hello.*/ or qw{a b c} have a fairly solid track record in other languages, and I think in general have proved both useful and straightforward in those languages. And even in Python, constructs like f-strings and complex numbers are examples of such things. However, I know of almost no examples of other languages that have added *user-definable* literal types (with the notable exception of C++, and I don't believe I've seen use of that feature in user code - which is not to say that it's not used). That to me says that there are complexities in extending the question to user-defined literals that we need to be careful of.

Agreed 100%. That’s why I think we need a more concrete proposal, that includes at least some thought on implementation, before we can go any farther, as I said in my first reply. The OP wanted to get some feeling of whether at least some people might find some version of this useful before going further. I think we’ve got that now (the fact that not 100% of the responders agree doesn’t change that), so we need to get more detailed now. My own proposal was just to answer the charge that any design will inherently be impossible or magical or complicated by giving a design that is none of those. It shouldn’t be taken as any more than that. If there are good use cases for prefixes, prefixes plus suffixes, etc., then my proposal can’t get you there, so let’s wait for the OP’s

Some specific questions which would need to be dealt with:

1. What is valid in the "literal" part of the construct (this is the p"C:\" question)?

I think this pretty much has to be either (a) exactly what’s valid in the equivalent literals today, or (b) something equally simple to describe, and parse, even if it’s different (like really-raw strings, or perlesque regex with delimiters other than quotes, or whatever). Either way, I think you want to use the same rule for all affixed literals, not allow a choice of different ones like C++ does.

2. How do definitions of literal syntax get brought into scope in time for the parser to act on them (this is about "import xyz_literal" making xyz"a string" valid but leaving abc"a string" as a syntax error)?

I don’t know that this is actually necessary. If `abc"a string"` raises an error at execution time rather than compile time, yes, that’s different from how most syntax errors work today, but is it really unacceptable? (Notice that in the most typical case, the error still gets raised from importing the module or from the top level of the script—but that’s just the most typical case, not all cases—you could get those errors from, say, calling a method, which you don’t normally expect.) There’s clearly a trade off here, because the only other alternative (at least that I’ve thought of or seen from anyone else; I’d love to be wrong) is that what you’ve imported and/or registers affects how later imports work (and doesn’t that mean some kind of registry hash needs to get encoded in .pyc files or something too?). While that is normal for people who use import hooks, most people don’t use import hooks most of the time, and I suspect that weirdness would be more off-putting than the late errors. Another big one: How do custom prefixes interact with builtin string prefixes? For suffixes, there’s no problem suffixing, say, a b-string, but for prefixes, there is. If this is going to be allowed, there are multiple ways it could be designed, but someone has to pick one and specify it. (Actually, for suffixes, there _is_ a similar issue: is `1.2jd` a `d` suffix on the literal `1.2j`, or a `jd` suffix on `1.2`? I think the former, because it’s a trivially simple rule that doesn’t need to touch any of the rest of the grammar. Plus, not only is it likely to never matter, but on the rare cases where it does matter, I think it’s the rule you’d want. For example, if I created my own ComplexDecimal class and wanted to use a suffix for it, why would I want to define both `d` and `jd` instead of just defining `d` and having it work with imaginary literals?)

These questions also fundamentally affect other tools like IDEs, linters, code formatters, etc.

Good point. I was thinking that any rule that’s simple enough for Python and humans to parse will probably be reasonably simple for other tools, and any rule that isn’t simple enough for Python and humans is probably a non-starter anyway. But the “lookup affixes at compile time” idea is an example of something that would be easy for Python and for humans but difficult for single-file-at-a-time tools, so this can be important.

In addition, there is the question of how user-defined literals would get turned into constants within the code. In common with list expressions, tuples, etc, user-defined literals would need to be handled as translating into runtime instructions for constructing the value (i.e., a function call). But people typically don't expect values that take the form of a literal like this to be "just" syntax sugar for a function call. So there's an education issue here. Code will get errors at runtime that the users might have expected to happen at compile time, or in the linter.

I really don’t think this one is a serious issue. Many people never need to learn that -2, 1+2j, (1,2), etc. are not literals, or which of those get optimized by CPython and packed into co_consts anyway, or which things that don’t even look like literals get similarly optimized. So how often will they need to know whether 1.23f is a literal, not a literal but optimized into a const, or neither?

Also, it's worth noting that the benefits of *user-defined* literals are *not* the same as the benefits of things like 0.2f, or 3.14d, or even re/^hello.*/. Those things may well be useful. But the benefit you gain from *user-defined* literals is that of letting the end user make the design decisions, rather than the language designer. And that's a subtly different thing.

That’s a good point, but I think you’re missing something big here. Think about it this way; assuming f and frac and dec and re and sql and so on are useful, out options are: 1) people don’t get a useful feature 2) we add user-defined affixes 3) we add all of these as builtin affixes While #3 theoretically isn’t impossible, it’s wildly implausible, and probably a bad idea to boot, so the realistic choice is between 1 and 2. Now, you’re right that choice 2 inherently means that we’re putting a new design decision on the end user (or library designer). That is definitely a factor to be weighed on the decision. But I don’t think it’s an immediate disqualifying factor. And in fact, if the feature is properly designed to be restrictive enough (but not too restrictive) I don’t think it will even end up being that big of a deal. There are all kinds of things that we leave up to the user, from the trivial (e.g., in Haskell, a capital letter means a type rather than a value; in Python it’s entirely up to each project whether it means anything at all) to the drastic but rarely used (import hooks probably being the most extreme). This one isn’t going to be trivial, but I think it will fall much closer to the less-disruptive side than many people are assuming. It’s only going to touch a small part of the grammar, and the language in general. (And if that turns out not to be true of the actual proposal, then I probably won’t support the actual proposal.)

So, to summarise, the real problem with user defined literal proposals is that the benefit they give hasn't yet proven sufficient to push anyone to properly address all of the design-time details. We keep having high-level "would this be useful" debates, but never really focus on the key question, of what, in precise detail, is the "this" that we're talking about - so people are continually making arguments based on how they conceive such a feature might work. A really good example here is the p"C:\" question. Is the proposal that the "string part" of the literal is just a normal string? If so, then how do you address this genuine issue that not all paths are valid? What about backslash-escapes (p"C:\temp")? Is the string a raw string or not? If the proposal is that the path-literal code can define how the string is parsed, then *how does that work*?

The OP even made this point explicitly:

...

I'm not discussing possible implementation of this feature just yet, we can get to that point later when there is a general understanding that this is worth considering.

I don't think we *can* agree on much without the implementation details (well, other than "yes, it's worth discussing, but only if someone proposes a properly specified design"

Again, agreed.

On Thu, 29 Aug 2019 at 01:18, Andrew Barnert wrote:

...

Also, it's worth noting that the benefits of *user-defined* literals are *not* the same as the benefits of things like 0.2f, or 3.14d, or even re/^hello.*/. Those things may well be useful. But the benefit you gain from *user-defined* literals is that of letting the end user make the design decisions, rather than the language designer. And that's a subtly different thing.

That’s a good point, but I think you’re missing something big here.

Think about it this way; assuming f and frac and dec and re and sql and so on are useful, out options are:

1) people don’t get a useful feature 2) we add user-defined affixes 3) we add all of these as builtin affixes

While #3 theoretically isn’t impossible, it’s wildly implausible, and probably a bad idea to boot, so the realistic choice is between 1 and 2.

That's a completely different point. Built in affixes are defined by the language, user defined affixes are defined by the user (obviously!) That includes all aspects of design - both how a given affix works, and whether it's justified to have an affix at all for a given use case. The argument is identical to that of user-defined operators vs built in operators. If you can use this argument to justify user-defined affixes, it applies equally to user-defined operators, which is something that has been asked for far more often, with much more widespread precedents in other languages, and been rejected every time. Regarding your cases #1, #2, and #3, this is the fundamental point of language design - you have to choose whether a feature is worthwhile (in the face of people saying "well *I* would find it useful), and whether to provide a general mechanism or make a judgement on which (if any) use cases warrant a special-case language builtin. If you assume everything should be handled by general mechanisms, you end up at the Lisp/Haskell end of the spectrum. If you decide that the language defines the limits, you are at the C end. Traditionally, Python has been a lot closer to the "language defined" end of the scale than the "general mechanisms" end. You can argue whether that's good or bad, or even whether things should change because people have different expectations nowadays, but it's a fairly pervasive design principle, and should be treated as such. This actually goes back to the OP's point:

we can get to that point later when there is a general understanding that this is worth considering

The biggest roadblock to a "general understanding that this is worth considering" is precisely that Python has traditionally avoided (over-) general mechanisms for things like this. The obvious other example, as I mentioned above, being user defined operators. I've been very careful *not* to use the term "Pythonic" here, as it's too easy for that to be a way of just saying "my opinion is more correct than yours" without a real justification, but the real stumbling block for proposals like this tends to be far less about the technical issues, and far *more* about "does this fit into the philosophy of Python as a language, that has made it as successful as it is?" My instinct is that it doesn't fit well with Python's general philosophy. Paul

On Wed, Aug 28, 2019 at 10:01:25PM -0000, stpasha@gmail.com wrote:

...

you have something that looks like a kind of string czt'...' but is really a function call that might return absolutely anything at all;

This is kinda the whole point.

Yes, I understand that. And that's one of the reasons why I think that this is a bad idea. Since Python is limited to ASCII syntax, we only have a small number of symbols suitable for delimiters. With such a small number available, - parentheses () are used for grouping and function calls; - square brackets [] are used for lists and subscripting; - curly brackets {} are used for dicts and sets; - quote marks are used for bytes and strings; And with your proposal: - quote marks are also used for function calls, but only a limited subset of function calls (those which take a single string literal argument). Across a large majority of languages, it is traditional and common to use round brackets for grouping and function calls, and square and curly brackets for collections. There are a handful of languages, like Mathematica, which use [] for function calls.

I understand, of course, how the idea of a string-that-is-not-a-string may sound blasphemous,

Its not a matter of blasphemy. It's a matter of readability and clarity.

however I invite you to look at this from a different perspective.

Today's date is 2019-08-28. The date is a moment in time, or perhaps a point in the calendar, but it is certainly not a string. How do we write this date in Python? As `datetime("2019-08-28")`. We are forced to put the date into a string and pass that string into a function to create an actual datetime object.

We are "forced" to write that are we? Have you ever tried it? py> from datetime import datetime py> datetime("2019-08-28") Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: an integer is required (got type str)

With this proposal the code would look something like `dt"2019-08-28"`. You're right, it's not a string anymore. But it *should not* have been a string to begin with, we only used a string there because Python didn't offer us any other way.

py> datetime(2019, 8, 28) datetime.datetime(2019, 8, 28, 0, 0) It is difficult to take your argument seriously when so much of it rests on things which aren't true. -- Steven

On Aug 29, 2019, at 00:58, Paul Moore wrote:

If you assume everything should be handled by general mechanisms, you end up at the Lisp/Haskell end of the spectrum. If you decide that the language defines the limits, you are at the C end.

And if you don’t make either assumption, but instead judge each case on its own merits, you end up with a language which is better than languages at either extreme. There are plenty of cases where Python generalizes beyond most languages (how many languages use the same feature for async functions and sequence iteration? or get metaclasses for free by having only one “kind” and then defining both construction and class definitions as type calls?), and plenty where It doesn’t generalize as much as most languages, and its best features are found all across that spectrum. You can’t avoid tradeoffs by trying to come up with a rule that makes language decisions automatically. (If you could, why would this list even exist?) The closest thing you can get to that is the vague and self-contradictory and facetious but still useful Zen. If you really did try to zealously pick one side or the other, always avoiding general solutions whenever a hardcoded solution is simpler no matter what, the best-case scenario would be something like Go, where a big ecosystem of codegen tools defeats your attempt to be zealous and makes your language actually usable despite your own efforts until soon you start using those tools even in the stdlib. Also, I’m not sure the spectrum is nearly as well defined as you imply in the first place. It’s hard to find a large C project that doesn’t use the hell out of preprocessor macros to effectively create custom syntax for things like error handling and looping over collections (not to mention M4 macros to autoconf the code so it’s actually portable instead of just theoretically portable), and meanwhile Haskell’s syntax is chock full of special-purpose features you couldn’t build yourself (would anyone even use the language without, say, do blocks?).

On Thu, 29 Aug 2019 at 14:21, Andrew Barnert wrote:

You can’t avoid tradeoffs by trying to come up with a rule that makes language decisions automatically. (If you could, why would this list even exist?) The closest thing you can get to that is the vague and self-contradictory and facetious but still useful Zen.

Sorry, I wasn't trying to imply that you could. Just that choosing to implement some, but not all, possible literal affixes on a case by case basis was a valid language design option, and one that is taken in many cases. Your statement

Think about it this way; assuming f and frac and dec and re and sql and so on are useful, out options are:

1) people don’t get a useful feature 2) we add user-defined affixes 3) we add all of these as builtin affixes

While #3 theoretically isn’t impossible, it’s wildly implausible, and probably a bad idea to boot, so the realistic choice is between 1 and 2.

seemed to imply that you thought it was an "all or nothing" choice. My apologies if I misunderstood your point. Paul

On 29/08/2019 00:24, Andrew Barnert wrote:

On Aug 27, 2019, at 10:21, Rhodri James wrote:

...

You make the point yourself: this is something we already understand from dealing with complex numbers in other circumstances. That is not true of generic single-character string prefixes.

It certainly is true for 1.23f.

I would contend that (and anyway 1.23f is redundant; 1.23 is already a float literal). But anyway I said "generic single-character string prefixes", because that's what the original proposal was. You seem to be going off on creating literal syntax for standard library types (which, for the record, I think is a good idea and deserves its own thread), but that's not what the OP seems to be going for. -- Rhodri James *-* Kynesim Ltd

On 29/08/2019 14:40, Rhodri James wrote:

Pace Stephen's point

My apologies, it was Steven's point. -- Rhodri James *-* Kynesim Ltd

On Thu, 29 Aug 2019 at 15:54, Steven D'Aprano wrote:

Let me suggest some design principles that should hold for languages with more-or-less "conventional" syntax. Languages like APL or Forth excluded.

This will degenerate into nitpicking very fast, so let me just say that I understand the general idea that you're trying to express here. I don't entirely agree with it, though, and I think there are some fairly common violations of your suggestion below that make your arguments less persuasive than maybe you'd like.

- anything using ' or " quotation marks as delimiters (with or without affixes) ought to return a string, and nothing but a string;

In C, Java and C++, 'x' is an integer (char). In SQL (some dialects, at least) TIMESTAMP'2019-08-22 11:32:12' is a TIMESTAMP value. In Python, b'123' is a bytes object (which maybe you're willing to classify as "a string", but the line blurs quite fast). Paul

My understanding is that for a sql prefix the most valuable part is to be able to know that it was created from a literal. No other magic, definitely not auto-executing. Then it would be legal to write result = conn.execute(sql"SELECT * FROM people WHERE id=?", user_id) but not result = conn.execute(f"SELECT * FROM people WHERE id={user_id}") In order to achieve this, the `execute()` method only has to look at the type of its argument, and throw an error if it's a plain string. Perhaps with some more imagination we can make result = conn.execute(sql"SELECT * FROM people WHERE id={user_id}") work too, but in this case the `sql"..."` token would only create an `UnpreparedStatement` object, which expects a variable named "user_id", and then the `conn.execute()` method would pass locals()/globals() into the `.prepare()` method of that statement, binding those values to the placeholders. Crucially, the `.prepare()` method shouldn't modify the object, but return a new PreparedStatement, which then gets executed by the `conn.execute()`.

How does one get a value into locals()["re~"]?

You're right, I didn't think about that. I agree with Steven's interpretation that the user is not expected to modify locals herself, still the immutable nature of locals presents a considerable challenge. So I'm thinking that perhaps we could change that to `globals()["re~"]`, where globals are in fact mutable and can even be modified by the user. This would make it so that affixes can only be declared at a module level, similar to how `from library import *` is not allowed in a function either. This is probably a saner approach anyways -- if affixes could mean different things in different functions, that could be quite confusing...

On 8/29/19 11:14 AM, Chris Angelico wrote:

...

My understanding is that for a sql prefix the most valuable part is to be able to know that it was created from a literal. No other magic, definitely not auto-executing. Then it would be legal to write

result = conn.execute(sql"SELECT * FROM people WHERE id=?", user_id)

but not

result = conn.execute(f"SELECT * FROM people WHERE id={user_id}")

In order to achieve this, the `execute()` method only has to look at the type of its argument, and throw an error if it's a plain string. There's no such thing, though, any more than there's such a thing as a "raw string". There are only two types of string in Python - text and bytes. You can't behave differently based on whether you were given a

On Fri, Aug 30, 2019 at 3:51 AM Pasha Stetsenko wrote: triple-quoted, raw, or other string literal.

But isn't the idea of the sql" (or other) prefix was that the 'plain string' was put through a special function that processes it, and that function could return an object of some other type, so it could detect the difference.

...

Perhaps with some more imagination we can make

result = conn.execute(sql"SELECT * FROM people WHERE id={user_id}")

work too, but in this case the `sql"..."` token would only create an `UnpreparedStatement` object, which expects a variable named "user_id", and then the `conn.execute()` method would pass locals()/globals() into the `.prepare()` method of that statement, binding those values to the placeholders. Crucially, the `.prepare()` method shouldn't modify the object, but return a new PreparedStatement, which then gets executed by the `conn.execute()`. One way to handle this particular case would be to do it as a variant of f-string that doesn't join its arguments, but passes the list to some other function. Just replace the final step BUILD_STRING step with BUILD_LIST, then call the function. There'd need to be some way to recognize which sections were in the literal and which came from interpolations (one option is to simply include empty strings where necessary such that it always starts with a literal and then alternates), but otherwise, the "sql" manager could do all the escaping it wants. However, this wouldn't be enough to truly parameterize a query; it would only do escaping into the string itself.

Another option would be to have a single variant of f-string that, instead of creating a string, creates a "string with formatted values". That would then be a single object that can be passed around as normal, and if conn.execute() received such a string, it could do the proper parameterization.

Not sure either of them would be worth the hassle, though.

ChrisA

= -- Richard Damon

On Aug 29, 2019, at 06:40, Rhodri James wrote:

However, it sounds like what you really want is something I've often really wanted to -- a way to get the compiler to pre-create "constant" objects for me.

People often say they want this, but does anyone actually ever have a good reason for it? I was taken in by the lure of this idea myself—all those wasted frozenset constructor calls! (This was before the peephole optimizer understood frozensets.) Of course I hadn’t even bothered to construct the frozensets from tuples instead of lists, which should have been a hint that I was in premature optimization mode, and should have been the first thing I tried before going off the deep end. But hacking bytecode is fun, so I sat down and wrote a bytecode processor that let me replace any expression with a LOAD_CONST, much as the builtin optimizer does for things like simple arithmetic. It’s easy to hook it up to a decorator to call on a function, or to an import hook to call at module compile time. And then, finally, it’s time to benchmark and discover that it makes no difference. Stripping things down to something trivial enough to be tested… aha, I really was saving 13us, it’s just that 13us is not measurable in code that takes seconds to run. Maybe someone has a real use case where it matters. But I’ve never seen one. I tried to find good nails for my shiny new hammer and never found one, and eventually just stopped maintaining it. And then I revived it when I wrote my decimal literal hack (the predecessor to the more general user literal hack I linked earlier in the thread) back during the 2015 iteration of this discussion, but again couldn’t come up with a plausible example where those 2.3d pseudo-literals were measurably affecting performance and needed constifying; I don’t think I even bothered mentioning it in that thread. Also, even if you find a problem, it‘s almost always easy to work around today. If the constant is constructed inside a loop, just manually lift it out of the loop. If it’s in a function body, this is effectively the same problem as global or builtin lookups being too slow inside a function body, and can be solved the same way, with a keyword parameter with a default value. And if the Python community thinks that _sin=sin is good enough for the uncommon problem of lookups significantly affecting performance, surely of _vals=frozenset((1,2,3)) is also good enough for that far more uncommon problem, and therefore _limit=1e1000dec would also be good enough for the new but probably even more uncommon one. (Also, notice that the param default can be used with mutable values, it’s just up to you to make sure you don’t accidentally mutate them; an invisible compiler optimization couldn’t do that, at least not without something like Victor Stinner’s FAT guards.) For what it’s worth, I actually found my @constify decorator more readable than the param default, especially for global functions—but not nearly enough so that it’s worth using a hacky, CPython-specific module that I have to maintain across Python versions (and byteplay to byteplay3 to bytecode) and that nobody else is using. Or to propose for a builtin (or stdlib but magic) feature. What this all comes down to is that, despite my initial impression, I really don’t care whether Python thinks 1.23d is a constant value or not; I only care whether the human reader thinks it is one. Think about it this way: do you know off the top of your head whether (1, (2,3)) gets optimized to a const the same way (1,2) does in CPython? Has it ever occurred to you to check before I asked? And this is actually something that changed relatively recently. Why would someone who doesn’t even think about when tuples are constified want to talk about how to force Python to constify other types? Because even years of Python experience hasn’t cured us of premature-optimization-itis.

On Aug 29, 2019, at 07:52, Steven D'Aprano wrote:

...

On Thu, Aug 29, 2019 at 05:30:39AM -0700, Andrew Barnert wrote:

...

On Aug 29, 2019, at 04:58, Steven D'Aprano wrote:

- quote marks are also used for function calls, but only a limited subset of function calls (those which take a single string literal argument).

This is a disingenuous argument.

When you read spam.eggs, of course you know that that means to call the __getattr__('eggs') method on spam. But do you actually read it as a special method calling syntax that’s restricted to taking a single string that must be an identifier as an argument

You make a good point about abstractions, but you are missing the critical point that spam.eggs *doesn't look like a string*. Things that look similar should be similar; things which are different should not look similar.

Which is exactly why you’d read 1.23dec or 1.23f as a number, because it looks like a number and also acts like a number, rather than as a function call that takes the string '1.23', even if you know that’s how it’s implemented. And most of the string affixes people have suggested are for string-ish things. I’m not sure what a “version string” is, but I might design that as an actual subclass of str that adds extractor methods and overrides comparison. A compiled regex isn’t literally a string, but neither is a bytes; it’s still clearly _similar_ to a string, in important ways. And so is a path, or a URL (although I don’t know what you’d use the url prefix for in Python, given that we don’t have a string-ish type like ObjC’s NSURL to return and I don’t think we need one, but presumably whoever wrote the url affix would be someone who disagreed and packaged the prefix with such a class). And versions of the proposal that allow delimiters other than quotes so you can write things like regex/a.*b/, well, I’d need to see a specific proposal to be sure, but that seems even less objectionable in this regard. That looks like nothing else in Python, but it looks like a regex in awk or sed or perl, so I’d probably read it as a regex object.

I acknowledge your point (and the OP's) that many things in Python are ultimately implemented as function calls. But none of those things look like strings:

- The argument to the import statement looks like an identifier (since it is an identifier, not an arbitrary string);

- The argument to __getattr__ etc looks like an identifier (since it is an identifier, not an arbitrary string);

- The argument to __getitem__ is an arbitrary expression, not just a string.

The arguments to the dec and f affix handlers look like numeric literals, not arbitrary strings. The arguments to path and version are… probably string literal representations (with the quotes and all), not arbitrary strings. Although that does depends on the details of the specific proposal, if _any_ of your killer uses needs uncooked strings, then either you rcome up with something over complicated like C++ where you can register three different kinds of affixes, or you just always pass uncooked strings (because it’s trivial to cook on demand but impossible to de-cook). And the arguments to regex may be some _other_ kind of restricted special string that… I don’t think anyone has tried to define yet, but you can vaguely imagine what it would have to be like, and it certainly won’t be any arbitrary string.

Let me suggest some design principles that should hold for languages with more-or-less "conventional" syntax. Languages like APL or Forth excluded.

- anything using ' or " quotation marks as delimiters (with or without affixes) ought to return a string, and nothing but a string;

So b"abc" should not be allowed? Let’s say I created a native-UTF16-string type to deal with some horrible Windows or Java stuff. Why would this principle of yours suggest that I shouldn’t be allowed to use u16"" just like b””? This is a design guideline for affixes, custom or otherwise. Which could be useful as a filter on the list of proposed uses, to see if any good ones remain (and if no string affix uses remain, then of course the proposal is either useless or should be restricted to just numbers or whatever), but it can’t be an argument against all affixes, or against custom affixes, or anything else generic like that.

- as a strong preference, anything using quotation marks as delimiters ought to be processed at compile-time (f-strings are a conspicuous exception to that principle);

I don’t see why you should even want to _know_ whether it’s true, much less have a strong preference. Here are things you probably really do care about: (a) they act like strings, (b) they act like constants, (c) if there are potential issues parsing them, you see those issues as soon as possible, (d) working with them is more than fast enough. Compile time is neither necessary (Haskell) nor sufficient (Tcl) for any of that. So why insist on compile-time instead of insisting on a-d?

No I'm not. I'm going to think of it as a *string*, because it looks like a string.

Well, yes. It’s a path string, or a regex string, or a version string, or whatever, which is loosely a kind of string but not literally one. Like bytes. Or it’s a sql cursor, in which case it was probably a misuse of the feature.

Particularly given the OP's preference for single-letter prefixes.

OK, I will agree with you there that the overuse of single-letter prefixes in the motivating examples is a worrying sign. In principle there’s nothing wrong with single letters (and I think I can make a good case for the f suffix as a good use in 3D-math code). And a program that used a whole ton of version strings and version string constants might find it useful to use v instead of ver. But I’m having a hard time imagining such a program existing. (Even something like pip or the PyPI backend might have lots of version strings, but why would it have lots of version string constants?) So, maybe that’s a sign that the OP’s eventual detailed set of use cases is not going to make me happy. Of course the burden is on the proposer, and if every proposed string affix use case ends up looking bad, then I’d either oppose the proposal or suggest that it be restricted to numeric affixes or something. But that’s not a reason to reject the proposal before seeing it, or to argue that whatever it is can’t conceivably be good because of [some posited universal principle that doesn’t even hold in Python today].

1.23f doesn't look like a string, it looks like a number. I have no objection to that in principle, although of course there is a question whether float32 is important enough to justify either builtin syntax or custom, user-defined syntax.

As I’ve said before, I believe that anything that doesn’t have a builtin type does not deserve builtin syntax. And I don’t understand why that isn’t a near-ubiquitous viewpoint. But it’s not just you; at least three people (all of whom dislike the whole concept of custom affixes) seem at least in principle open to the idea of adding builtin affixes for types that don’t exist. Which makes me think it’s almost certainly not that you’re all crazy, but that I’m missing something important. Can you explain it to me?

Rhodri James wrote:

Suppose that we did have some funky mechanism to get the compiler to create objects at compile time

It doesn't necessarily have to be at compile time. It can be at run time, as long as it only happens once.

So we use "start_date" somewhere, and mutate it because the start date for some purpose was different. Then we use it somewhere else, and it's not the start date we thought it was. This is essentially the mutable default argument gotcha, just writ globally.

I don't think this is as much of a problem as it seems. We often assign things to globals that are intended to be treated as constants, with the understanding that it's our responsibility to refrain from mutating them. -- Greg

On Aug 29, 2019, at 16:58, Greg Ewing wrote:

Steven D'Aprano wrote:

...

I don't think that stpasha@gmail.com means that the user literally assigns to locals() themselves. I read his proposal as having the compiler automatical mangle the names in some way, similar to name mangling inside classes.

Yes, but at some point you have to define a function to handle your string prefix. If it's at the module level then it's no problem, because you can do something like

globals()["~f"] = lambda: ...

What happens if you do this, and then include "~f" in __all__, and then import * from that module? I personally would rather have my prefixes or suffixes available in every module that imports them, without needing to manually register them each time. Not a huge deal, and if nobody else agrees, fine. But if I could __all__ it, I could get what I want anyway. :)

On Thu, Aug 29, 2019 at 02:10:21PM -0700, Andrew Barnert wrote: [...]

And most of the string affixes people have suggested are for string-ish things.

I don't think that's correct. Looking back at the original post in this thread, here are the motivating examples: [quote] There are quite a few situations where this can be used: - Fraction literals: `frac'123/4567'` - Decimals: `dec'5.34'` - Date/time constants: `t'2019-08-26'` - SQL expressions: `sql'SELECT * FROM tbl WHERE a=?'.bind(a=...)` - Regular expressions: `rx'[a-zA-Z]+'` - Version strings: `v'1.13.0a'` - etc. [/quote] By my count, that's zero out of six string-ish things. There may have been other proposals, but I haven't trolled through the entire thread to find them.

I’m not sure what a “version string” is, but I might design that as an actual subclass of str that adds extractor methods and overrides comparison.

A version object is a record with fields, most of which are numeric. For an existing example, see sys.version_info which is a kind of named tuple, not a string. The version *string* is just a nice human-readable representation. It doesn't make sense to implement string methods on a Version object. Why would you offer expandtabs(), find(), splitlines(), translate(), isspace(), capitalise(), etc methods? Or * and + (repetition and concatenation) operators? I cannot think of a single string method/operator that a Version object should implement.

A compiled regex isn’t literally a string, but neither is a bytes; it’s still clearly _similar_ to a string, in important ways.

It isn't clear to me how a compiled regex object is "similar" to a string. The set of methods offered by both regexes and strings is pretty small, by my generous count it is just two methods: - str.split and SRE_Pattern.split; - str.replace and SRE_Pattern.sub neither of which use the same API or have the same semantics. Compiled regex objects don't offer string methods like translate, isdigits, upper, encode, etc. I would say that they are clearly *not* strings. [...]

And versions of the proposal that allow delimiters other than quotes so you can write things like regex/a.*b/, well, I’d need to see a specific proposal to be sure, but that seems even less objectionable in this regard. That looks like nothing else in Python, but it looks like a regex in awk or sed or perl, so I’d probably read it as a regex object.

Why do you need the "regex" prefix? Assuming the parser and the human reader can cope with using / as both a delimiter and a operator (which isn't a given!) /.../ for a regex object seems fine to me. I suspect that this is going to be ambiguous though: target = regex/a*b/ +x could be: target = ((regex / a) * b) / ( unary-plus x) or target = (regex object) + x so maybe we do need a prefix.

...

Let me suggest some design principles that should hold for languages with more-or-less "conventional" syntax. Languages like APL or Forth excluded.

- anything using ' or " quotation marks as delimiters (with or without affixes) ought to return a string, and nothing but a string;

So b"abc" should not be allowed?

In what way are byte-STRINGS not strings? Unicode-strings and byte-strings share a significant fraction of their APIs, and are so similar that back in Python 2.2 the devs thought it was a good idea to try automagically coercing from one to the other. I was careful to write *string* rather than *str*. Sorry if that wasn't clear enough.

Let’s say I created a native-UTF16-string type to deal with some horrible Windows or Java stuff. Why would this principle of yours suggest that I shouldn’t be allowed to use u16"" just like b””?

It is a utf16 STRING so making it look like a STRING is perfectly fine. [...]

...

- as a strong preference, anything using quotation marks as delimiters ought to be processed at compile-time (f-strings are a conspicuous exception to that principle);

I don’t see why you should even want to _know_ whether it’s true, much less have a strong preference.

Because I care about performance, at least a bit. Because I don't want to write code that is unnecessarily slow, for some definition of "unnecessary". Because I want to be able to reason (at least in broad terms) about the cost of certain operations. Because I want to be able to reason about the semantics of my code. Why do I write 1234 instead of int("1234")? The second is longer, but it is more explicit and it is self-documenting: the reader knows that its an int because it says so right there in the code, even if they come from Javascript where 1234 is an IEEE-754 float. Assuming the builtin int() hasn't be shadowed. But it's also wastefully slow. If we are genuinely indifferent to the difference, then we should be equally indifferent to a proposal to replace the LOAD_CONST byte-code for ints as follows: dis("1234") # in current Python LOAD_CONST 0 (1234) # In the future: LOAD_NAME 0 (int) LOAD_CONST 0 ('1234') CALL_FUNCTION 1 (1 positional, 0 keyword pair) If you were asked to vote +1 or -1 on this proposal (sitting on the fence not allowed), which would you vote? I would vote -1. Aside from the performance hit, it's also a semantic change: what was a compile-time literal is now a runtime function call which can be shadowed. It is nice to know that when I say ``n = 1234`` that the value of n is guaranteed to be 1234 no matter what odd things are going on. (Short of running a modified interpreter.) String literals (byte- or unicode, raw or cooked, triple- or single-quoted) are, with the exception of f-strings, LOAD_CONST calls like ints. I think that's a valuable, useful thing to know, and not something we should lightly give up.

Here are things you probably really do care about: (a) they act like strings, (b) they act like constants,

Don't confuse immutability with constant-ness. Python doesn't have constants, except by convention. There's no easy way to prevent a simple name from being rebound.

(c) if there are potential issues parsing them, you see those issues as soon as possible,

Like at compile-time? Consider the difference between the compile-time syntax error you get here: x = 123w456 versus the run-time error you get here: x = int("123w456") I can understand saying "we have no choice but to make this a runtime operation", or even "on the balance of pros and cons, it isn't worth the extra work to make this happen at compile-time". I don't like it that we have to write Decimal("123.456"), but I understand the reasons why we have to and can accept that it is a necessary evil. (To clarify: of course it is a feature that we *can* pass strings to the Decimal constructor, when such strings come from user-input or are read from data files etc.) But I don't think that it is a feature that there is no alternative but to pass a string, even when the value is known at edit-time. And I don't understand your position that I shouldn't care about the difference.

(d) working with them is more than fast enough.

You are right that Python is usually "fast enough" (except when it isn't), and that the one-off cost of creating a few pseudo-constants is generally only a small fraction of the cost of most programs. But Python isn't quote-unquote "slow" because of any one single thing, it is more of a death by a thousand cuts, lots of *small* inefficiences which individually don't matter but collectively add up to making Python up to a hundred times slower than C. When performance matters, which would you rather write? for item in huge_sequence: value = item + 1234 value = item + int("1234") I know that when I use a literal, it will be as fast as it possibly can be in Python, or at least there's nothing *I* can do to make it faster. But when I have to use a call like Decimal("123.45"), that's one more thing for me to have to worry about: is it fast enough? Can I make it faster? Should I make it faster? We should be wary about hiding potentially slow code in something that looks like fast code. (Yes, that's a criticism of properties too, but in the case of properties we know that the benefits outweigh the risk. It's not clear that this is the case here.)

Compile time is neither necessary (Haskell) nor sufficient (Tcl) for any of that. So why insist on compile-time instead of insisting on a-d?

I think you will find that I said this should be "a strong preference", which is hardly *insisting*.

...

No I'm not. I'm going to think of it as a *string*, because it looks like a string.

Well, yes. It’s a path string, or a regex string, or a version string,

Actually, no, it will be a Path object, a compiled regex SRE_pattern object, or a Version object, not a string at all.

or whatever, which is loosely a kind of string but not literally one. Like bytes.

Bytes literally are strings. They just aren't strings of Unicode characters.

Or it’s a sql cursor, in which case it was probably a misuse of the feature.

That's one of the motivating examples. I agree it is a misuse of the proposed feature.

...

Particularly given the OP's preference for single-letter prefixes.

OK, I will agree with you there that the overuse of single-letter prefixes in the motivating examples is a worrying sign. In principle there’s nothing wrong with single letters (and I think I can make a good case for the f suffix as a good use in 3D-math code).

I can concur with all of that. [...]

As I’ve said before, I believe that anything that doesn’t have a builtin type does not deserve builtin syntax.

Agreed. Although there's a bit of fuzziness over the concept of "builtin". Not all built-in objects are available in the ``builtins`` module, e.g. NoneType, or FunctionType.

And I don’t understand why that isn’t a near-ubiquitous viewpoint. But it’s not just you; at least three people (all of whom dislike the whole concept of custom affixes) seem at least in principle open to the idea of adding builtin affixes for types that don’t exist. Which makes me think it’s almost certainly not that you’re all crazy, but that I’m missing something important. Can you explain it to me?

I thought it went without saying that a necessary pre-condition for adding builtin syntax for a type was for the type to become built-in first. Sorry if it wasn't as clear or obvious as I thought. -- Steven

On Sat, Aug 31, 2019 at 09:31:15PM +1000, Chris Angelico wrote:

On Sat, Aug 31, 2019 at 8:44 PM Steven D'Aprano wrote:

...

...

So b"abc" should not be allowed?

In what way are byte-STRINGS not strings? Unicode-strings and byte-strings share a significant fraction of their APIs, and are so similar that back in Python 2.2 the devs thought it was a good idea to try automagically coercing from one to the other.

I was careful to write *string* rather than *str*. Sorry if that wasn't clear enough.

We call it a string, but a bytes object has as much in common with bytearray and with a list of integers as it does with a text string.

I don't think that's true. py> b'abc'.upper() b'ABC' py> [1, 2, 3].upper() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'list' object has no attribute 'upper' Shall I beat this dead horse some more by listing the other 33 methods that byte-strings share with Unicode-strings but not lists? Compared to just two methods shared by all three of bytes, str and list, (namely count() and index()), and *zero* methods shared by bytes and list but not str. In Python2, byte-strings and Unicode strings were both subclasses of type basestring. Although we have moved away from that shared base class in Python3, it does demonstrate that conceptually bytes and str are closely related to each other.

Is the contents of a MIDI file a "string"? I would say no, it's not - but it can *contain* strings, eg for metadata and lyrics.

Don't confuse *human-readable native language strings* for generic strings. "Hello world!" is a string, but so are '&w-8\x02^xs\0' and b'DEADBEEF'.

You can't upper-case the variable-length-integer b"\xe7\x61" any more than you can upper-case the integer 13281.

Of course you can. py> b"\xe7\x61".upper() b'\xe7A' Whether it is *meaningful* to do so is another question. But the same applies to str.upper: just because you can call the method doesn't mean that the result will be semantically valid. source = "def spam():\n\tpass\n" source = source.upper() # no longer valid Python source code.

Those common methods are mostly built on the assumption that the string contains ASCII text.

As they often do. If they don't, then don't call the text methods which don't make sense in context. Just as there are cases where text methods don't make sense on Unicode strings. You wouldn't want to call .casefold() on a password, or .lstrip() on a line of Python source code. [...]

Bytes and text have a long relationship, and as such, there are special similarities. That doesn't mean that bytes ARE text,

I didn't say that bytes are (human-readable) text. Although they can be: not every application needs Unicode strings, ASCII strings are still special, and there are still applications where once has to mix binary and ASCII text data. I said they were *strings*. Strings are not necessarily text, although they often are. Formally, a string is a finite sequence of symbols that are chosen from a set called an alphabet. See: https://en.wikipedia.org/wiki/String_%28computer_science%29

I don't think it's necessary to be too adamant about "must be some sort of thing-we-call-string" here. Let practicality rule, since purity has already waved a white flag at us.

It is because of *practicality* that we should prefer that things that look similar should be similar. Code is read far more often that it is written, and if you read two pieces of code that look similar, we should strongly prefer that they should actually be similar. Would you be happy with a Pythonesque language that used prefixed strings as the delimiter for arbitrary data types? mylist = L"1, 2, None, {}, L"", 99.5" mydict = D"key: value, None: L"", "abc": "xyz"" myset = S"1, 2, None" That's what this proposal wants: string syntax that can return arbitrary data types. How about using quotes for function calls? assert chr"9" == "\t" assert ord"9" == 57 That's what this proposal wants: string syntax for a subset of function calls. Don't say that this proposal won't be abused. Every one of the OP's motivating examples is an abuse of the syntax, returning non-strings from something that looks like a string. -- Steven

On Sun, Sep 1, 2019 at 10:47 AM Steven D'Aprano wrote:

On Sat, Aug 31, 2019 at 09:31:15PM +1000, Chris Angelico wrote:

...

On Sat, Aug 31, 2019 at 8:44 PM Steven D'Aprano wrote:

...

...

So b"abc" should not be allowed?

In what way are byte-STRINGS not strings? Unicode-strings and byte-strings share a significant fraction of their APIs, and are so similar that back in Python 2.2 the devs thought it was a good idea to try automagically coercing from one to the other.

I was careful to write *string* rather than *str*. Sorry if that wasn't clear enough.

We call it a string, but a bytes object has as much in common with bytearray and with a list of integers as it does with a text string.

I don't think that's true.

Older versions of Python had text and bytes be the same things. That means that, for backward compatibility, they have some common methods. But does that really mean that bytes can be uppercased? Or is it that we allow bytes to be treated as ASCII-encoded text, which is then uppercased, and then returned to being bytes?

py> b'abc'.upper() b'ABC'

py> [1, 2, 3].upper() Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'list' object has no attribute 'upper'

In Python2, byte-strings and Unicode strings were both subclasses of type basestring. Although we have moved away from that shared base class in Python3, it does demonstrate that conceptually bytes and str are closely related to each other.

Or does it actually demonstrate that Python 3 maintains backward compatibility with Python 2?

...

Is the contents of a MIDI file a "string"? I would say no, it's not - but it can *contain* strings, eg for metadata and lyrics. You can't upper-case the variable-length-integer b"\xe7\x61" any more than you can upper-case the integer 13281.

Of course you can.

py> b"\xe7\x61".upper() b'\xe7A'

Whether it is *meaningful* to do so is another question. But the same applies to str.upper: just because you can call the method doesn't mean that the result will be semantically valid.

So what did you actually do here? You took some bytes that represent an integer, and you called a method on it that makes no sense whatsoever, because now it represents a different integer. There's no sense in which your new bytes object represents an "upper-cased version of" the integer 13281. If I were to decode that string to text and THEN uppercase it, it might give a quite different result:

...

...

b"\xe7\x61".decode("Latin-1").upper().encode("Latin-1") b'\xc7A'

And if you choose some other encoding than Latin-1, you might get different results again. I put it to you that bytes.upper() exists more for backward compatibility with Python 2 than because a bytes object is, in some way, uppercaseable.

source = "def spam():\n\tpass\n" source = source.upper() # no longer valid Python source code.

But it started out as text, and it is now uppercase text. When you do that with bytes, you have to first layer in "this is actually encoded text", and you are then able to destroy that.

...

Bytes and text have a long relationship, and as such, there are special similarities. That doesn't mean that bytes ARE text,

I didn't say that bytes are (human-readable) text. Although they can be: not every application needs Unicode strings, ASCII strings are still special, and there are still applications where once has to mix binary and ASCII text data.

I said they were *strings*. Strings are not necessarily text, although they often are. Formally, a string is a finite sequence of symbols that are chosen from a set called an alphabet. See:

https://en.wikipedia.org/wiki/String_%28computer_science%29

A finite sequence of symbols... you mean like a list of integers within the range [0, 255]? Nothing in that formal definition says that a "string" of anything other than characters should be meaningfully treated as text.

...

I don't think it's necessary to be too adamant about "must be some sort of thing-we-call-string" here. Let practicality rule, since purity has already waved a white flag at us.

It is because of *practicality* that we should prefer that things that look similar should be similar. Code is read far more often that it is written, and if you read two pieces of code that look similar, we should strongly prefer that they should actually be similar.

And you have yet to prove that this similarity is actually a thing.

Would you be happy with a Pythonesque language that used prefixed strings as the delimiter for arbitrary data types?

mylist = L"1, 2, None, {}, L"", 99.5"

mydict = D"key: value, None: L"", "abc": "xyz""

myset = S"1, 2, None"

At some point it's meaningless to call it a "Pythonesque" language, but I've worked with plenty of languages that simply do not have data types this rich, and so everything is manipulated the exact same way. When a list of values is represented as ";item 1;item 2;item 3" (actually as a string), or when you unpack a URL to find that it has JSON embedded inside it, the idea of a "prefixed string" that tells you exactly what data type is coming would be a luxury.

That's what this proposal wants: string syntax that can return arbitrary data types.

How about using quotes for function calls?

assert chr"9" == "\t"

assert ord"9" == 57

That's what this proposal wants: string syntax for a subset of function calls.

Don't say that this proposal won't be abused. Every one of the OP's motivating examples is an abuse of the syntax, returning non-strings from something that looks like a string.

Let's look at regular expressions. JavaScript has a syntax for them involving leading and trailing slashes, borrowed from Perl, but I can't figure out whether a regex is a first-class object in Perl. So you can do something like this:

findme = /spa*m/ "This has spaaaaaam in it".match(findme) [ 'spaaaaaam', index: 9, input: 'This has spaaaaaam in it' ]

In Python, I can do the exact same thing, only using double quotes as the delimiter.

...

...

re.search("spa*m", "This has spaaaaam in it")

So what do you mean by "non-string" exactly? In what way is a regular expression "not a string", yet the byte-encoded form of an integer somehow is? It makes absolutely no sense to uppercase an integer, yet you could uppercase a regex (since all regex special characters are non-letters, this will make it match uppercase strings). Yet when you encode the string as bytes, it gains an upper() method, and when you encode a regex as a compiled regex object, it loses one. Why do you insist that a regex is somehow not a string, but b"\xe7\x61" is? ChrisA

Chris Angelico writes:

Older versions of Python had text and bytes be the same things. That means that, for backward compatibility, they have some common methods. But does that really mean that bytes can be uppercased? Or is it that we allow bytes to be treated as ASCII-encoded text, which is then uppercased, and then returned to being bytes?

Not just older versions. There have been several, more or less hotly contested, changes post-2/3 fork that basically come down to "bytes are frequently the wire format of ASCII-compatibly-encoded text, so we're going to add text methods for the convenience of people who work with those wire formats but do not need to (and sometimes cannot) decode to Unicode." For example, RFC 5322 header field tags are defined to be case- insensitive ASCII, and therefore it's useful to match them by upper- or lowercasing the tag, then matching against fixed strings. Could you convert to text and do the work? Not usefully: you need to parse the bytes to determine which text encoding is in use. (And ironically enough, if the message is RFC 5322 + RFC 2045-conformant, the hacky iso-8859-1 "conversion" will be allocation of a str object and then a memcpy of the bytes. I don't think that's a rebuttal to your argument, of course, it's just amusing.) That doesn't mean that bytes ARE text that happens to fit in 8-bit code units (PEP 393). It does mean that the similarities of the APIs are neither random accidents nor historical artifact. They're intentional. I don't think this has anything whatsoever to do with whether the "custom string prefix" proposal is a good idea or not. (other) Steve

On Sun, Sep 01, 2019 at 12:24:24PM +1000, Chris Angelico wrote:

Older versions of Python had text and bytes be the same things.

Whether a string object is *text* is a semantic question, and independent of what data format you use. 'Hello world!' is text, whether you are using Python 1.5 or Python 3.8. '\x01\x06\x13\0' is not text, whether you are using Python 1.5 or Python 3.8.

That means that, for backward compatibility, they have some common methods. But does that really mean that bytes can be uppercased?

I'm curious what you think that b'chris angelico'.upper() is doing, if it is not uppercasing the byte-string b'chris angelico'. Is it a mere accident that the result happens to be b'CHRIS ANGELICO'? Unicode strings are sequences of code-points, abstract integers between 0 and 1114111 inclusive. When you uppercase the Unicode string 'chris angelico', you're transforming the sequence of integers: U+0063,0068,0072,0069,0073,0020,0061,006e,0067,0065,006c,0069,0063,006f to this sequence of integers: U+0043,0048,0052,0049,0053,0020,0041,004e,0047,0045,004c,0049,0043,004f If you are prepared to call that "uppercasing", you should be prepared to do the same for the byte-string equivalent. (For the avoidance of doubt: this is independent of the encoding used to store those code points in memory or on disk. Encodings have nothing to do with this.) [...]

Or is it that we allow bytes to be treated as ASCII-encoded text, which is then uppercased, and then returned to being bytes?

I'm fairly confident that bytes methods aren't implemented by decoding to Unicode, applying the method, then re-encoding back to bytes. But even if they were, that's just an implementation detail. Imagine a float method that internally converted the float to a pair of integers (numerator/denominator), operated on that fraction, and then re-converted back to a float. I'm sure you wouldn't want to say that this proves that floats aren't numbers. The same applies to byte-strings. In the unlikely case that byte methods delegate to str methods, that doesn't mean byte-strings aren't strings. It just means that two sorts of strings can share a single implementation for their methods. Code reuse for the win! [...]

...

py> b"\xe7\x61".upper() b'\xe7A'

Whether it is *meaningful* to do so is another question. But the same applies to str.upper: just because you can call the method doesn't mean that the result will be semantically valid.

So what did you actually do here? You took some bytes that represent an integer,

For the sake of the argument I'll accept that *this particular* byte string represents an integer rather than a series of mixed binary data and ASCII text, or text in some unusual encoding, or pixels in an image, or any of a million other things it could represent. That's absolutely fine: if it doesn't make sense to call .upper() on your bytes, then don't call .upper() on them. Precisely as you wouldn't call .upper() on a str object, if it didn't make sense to do so.

and you called a method on it that makes no sense whatsoever, because now it represents a different integer.

The same applies to Unicode strings too. Any Unicode string method that transforms the input returns something that represents a different sequence of code-points, hence a different sequence of integers. Shall we agree that neither bytes nor Unicode are strings? No, I don't think so either :-)

If I were to decode that string to text and THEN uppercase it, it might give a quite different result:

Sure. If you perform *any* transformation on the data first, it might give a different result on uppercasing: - if you reverse the bytes, uppercasing gives a different result; - if you replace b'a' with b'e', uppercasing gives a different result etc. And exactly the same observation applies to str objects: - if you reverse the characters, uppercasing gives a different result; - if you replace 'a' with 'e', uppercasing gives a different result.

And if you choose some other encoding than Latin-1, you might get different results again.

Sure. The bytes methods like .upper() etc are predicated on the assumption that your bytes represent ASCII text. If your bytes represent something else, then calling the .upper() method may not be meaningful or useful. In other words... if your bytes string came from an ASCII text file, it's probably safe to uppercase it. If your bytes string came from a JPEG, then uppercasing them will probably make a mess of the image, if not corrupt the file. So don't do that :-) Analogy: ints support the unary minus operator. But if your int represents a mass, then negating it isn't meaningful. There's no such thing as -5 kg. Should we conclude from this that the int type in Python doesn't represent a number, and that the support of numeric operators and methods is merely for backwards compatibility? I think not. The formal definition of a string is a sequence of symbols from an alphabet. That is precisely what bytes objects are: the alphabet in this case is the 8-bit numbers 0 to 255 inclusive, which for usefulness, convenience and backwards compatibility can be optionally interpreted as the 7-bit ASCII character set plus another 128 abstract "characters".

...

I said they were *strings*. Strings are not necessarily text, although they often are. Formally, a string is a finite sequence of symbols that are chosen from a set called an alphabet. See:

https://en.wikipedia.org/wiki/String_%28computer_science%29

A finite sequence of symbols... you mean like a list of integers within the range [0, 255]? Nothing in that formal definition says that a "string" of anything other than characters should be meaningfully treated as text.

Sure. If your bytes don't represent text, then methods like upper() probably won't do anything meaningful. It's still a string though.

...

...

I don't think it's necessary to be too adamant about "must be some sort of thing-we-call-string" here. Let practicality rule, since purity has already waved a white flag at us.

It is because of *practicality* that we should prefer that things that look similar should be similar. Code is read far more often that it is written, and if you read two pieces of code that look similar, we should strongly prefer that they should actually be similar.

And you have yet to prove that this similarity is actually a thing.

I'm not sure the onus is on me to prove this. "Status quo wins a stalemate." And surely the onus is on those proposing the new syntax to demonstrate that it will be fine to use string delimiters as function calls. You could make a good start by finding other languages, reasonably conventional languages with syntax based on the Algol or C tradition, that use quotes '' or "" to return arbitrary types. Even languages with unconventional syntax like Forth or APL would be a good start. Maybe I'm wrong. Maybe quotation marks are widely used for purposes other than delimiting strings, and I'm just too ignorant of other languages to know it. Maybe Python is in the minority here. Anyway, the bottom line is this: I have no objection to using prefixed quotes to represent Unicode strings, or byte strings, or Andrew's hypothetical UTF-16 strings, or EBCDIC strings, or TRON strings. https://en.wikipedia.org/wiki/TRON_(encoding) But I think that any API that would allow z"..." to represent (let's say) a socket, or a float, or a HTTP_Server instance, or a list, would be a deeply flawed API.

Let's look at regular expressions. JavaScript has a syntax for them involving leading and trailing slashes, borrowed from Perl, but I can't figure out whether a regex is a first-class object in Perl. So you can do something like this:

...

findme = /spa*m/ "This has spaaaaaam in it".match(findme) [ 'spaaaaaam', index: 9, input: 'This has spaaaaaam in it' ]

In Python, I can do the exact same thing, only using double quotes as the delimiter.

...

...

...

re.search("spa*m", "This has spaaaaam in it")

Sure. As a convenience, the re module has functions which accepts regular expression patterns as well as compiled regular expression objects.

So what do you mean by "non-string" exactly? In what way is a regular expression "not a string",

That question is ambiguous. Are you asking about regular expression patterns, or regular expression objects? Regular expression *patterns* are clearly strings: pattern = r'...' We type them with string delimiters, if you call type(pattern) it will return str, you can slice the pattern or uppercase it. Regular expression *objects* are just as clearly not strings: rx = re.compile(pattern) You can't slice them, they aren't sequences of symbols, they have attributes like rx.flags which have no meaning in a string, they lack string methods like upper, and those methods they have operate very differently from their equivalent string methods: pattern.find("X") # search pattern for "X" rx.search("X") # search "X" for pattern Regex objects are far more than just the regex pattern.

yet the byte-encoded form of an integer somehow is?

If your bytes represent an integer, then uppercasing them isn't meaningful. If your bytes represent ASCII text then uppercasing them may be meaningful.

It makes absolutely no sense to uppercase an integer, yet you could uppercase a regex (since all regex special characters are non-letters, this will make it match uppercase strings).

In general, you can't uppercase regex patterns without radically changing the meaning of them. Consider r'\d' and r'\D'.

Yet when you encode the string as bytes, it gains an upper() method, and when you encode a regex as a compiled regex object, it loses one. Why do you insist that a regex is somehow not a string, but b"\xe7\x61" is?

Because a byte-string matches the definition of strings, while compiled regex objects do not. -- Steven

On 31/08/2019 12:31, Chris Angelico wrote:

We call it a string, but a bytes object has as much in common with bytearray and with a list of integers as it does with a text string.

You say that as if text strings aren't sequences of bytes. Complicated and restricted sequences, I grant you, but no more so than a packet for a given network protocol. -- Rhodri James *-* Kynesim Ltd

On 03/09/2019 13:31, Chris Angelico wrote:

On Tue, Sep 3, 2019 at 10:27 PM Rhodri James wrote:

...

On 31/08/2019 12:31, Chris Angelico wrote:

...

We call it a string, but a bytes object has as much in common with bytearray and with a list of integers as it does with a text string.

You say that as if text strings aren't sequences of bytes. Complicated and restricted sequences, I grant you, but no more so than a packet for a given network protocol.

A text string is a sequence of characters. By "byte", I really mean "octet", but Python prefers to say "byte".

And a character is a byte or sequence of bytes. (Odd-sized bytes are pretty much history now, so for non-pendantic usages "byte" is good enough.) -- Rhodri James *-* Kynesim Ltd

On 03/09/2019 15:27, Chris Angelico wrote:

On Tue, Sep 3, 2019 at 11:19 PM Rhodri James wrote:

...

On 03/09/2019 13:31, Chris Angelico wrote:

...

On Tue, Sep 3, 2019 at 10:27 PM Rhodri James wrote:

...

On 31/08/2019 12:31, Chris Angelico wrote:

...

We call it a string, but a bytes object has as much in common with bytearray and with a list of integers as it does with a text string.

You say that as if text strings aren't sequences of bytes. Complicated and restricted sequences, I grant you, but no more so than a packet for a given network protocol.

A text string is a sequence of characters. By "byte", I really mean "octet", but Python prefers to say "byte".

And a character is a byte or sequence of bytes. (Odd-sized bytes are pretty much history now, so for non-pendantic usages "byte" is good enough.)

But a character is not an octet.

I get that you're distinguishing between the thing and its representation, but I'm coming at this as an embedded systems engineer. For me, it's turtles^Woctets all the way down. -- Rhodri James *-* Kynesim Ltd

On Sep 3, 2019, at 06:17, Rhodri James wrote:

...

On 03/09/2019 13:31, Chris Angelico wrote:

...

On Tue, Sep 3, 2019 at 10:27 PM Rhodri James wrote:

...

On 31/08/2019 12:31, Chris Angelico wrote: We call it a string, but a bytes object has as much in common with bytearray and with a list of integers as it does with a text string.

You say that as if text strings aren't sequences of bytes. Complicated and restricted sequences, I grant you, but no more so than a packet for a given network protocol.

A text string is a sequence of characters. By "byte", I really mean "octet", but Python prefers to say "byte".

And a character is a byte or sequence of bytes. (Odd-sized bytes are pretty much history now, so for non-pendantic usages "byte" is good enough.)

Forget about bytes vs. octets; this still isn’t a useful perspective. A character is a grapheme cluster, a sequence or one or more code points. A code point is an integer between 0 and 1.1M. A string is a flattened sequence of grapheme clusters—that is, a sequence of code points. (Python ignores the cluster part, pretending code points are characters, at the cost of requiring every application to handle normalization manually. Which is normally a good tradeoff, but it does mean that you can’t even say whether two sequences of code points are the same string without calling a function.) Meanwhile, there are multiple ways to store those code points as bytes. Python does whatever it wants under the covers, hiding it from the user. Obviously there is _some_ array of bytes somewhere in memory that represents the characters of the string in some way (I say “obviously”, but that isn’t always true in Swift, and isn’t even frequently true in Haskell…), but you don’t have access to that. If you want a sequence of bytes, you have to ask for a sequence in some specific representation, like UTF-8 or UTF-16-BE or Shift-JIS, which it creates for you on the fly (albeit cached in a few special cases). So, from your system programmer’s perspective, in what useful sense is a character, or a string, a sequence of bytes? And this is all still ignoring the fact that in Python, all values are “boxed” in an opaque structure that you can’t access from within the language, and even from the C API of CPython the box structure isn’t part of the API, so even something simpler like, say, an int isn’t usefully a sequence of 30-bit digits from the system programmer’s perspective, it’s an opaque handle that you can pass to functions to _obtain_ a sequence of 30-bit digits. (In the case of strings, you have to first pass to opaque handle to one function to see what format to ask for, then pass it to another to obtain a sequence of 1, 2, or 4-byte integers representing the code points in native-endian ASCII, UCS2, or UCS4. Which normally you don’t do—you ask for a UTF-8 string or a UTF-32 string that may get constructed on the fly—but if you really do want the actual storage, this is the way to get it.) And most of this is not peculiar to Python. In Swift, a string is a sequence of grapheme clusters. In Java, it’s a sequence of UTF-16 code units. In Go, it’s a sequence of UTF-8 code units. In Haskell, it’s a lazy linked list of code points. And so on. In some of those cases, a character does happen to be represented as a string of bytes within a larger representation, but even when it is, that still doesn’t mean you can usefully access it that way. Of course a text file on disk is a sequence or bytes, and (if you know the encoding and normalization) you could operate directly on those. But you don’t; you pass the byte strings to a function that decodes them (and then sometimes to a second function that normalizes them into a canonical form) and then use your language’s string functions on the result. In fact, you probably don’t even do that; you let the file object buffer the byte strings however it wants to and just hand you decoded text objects, so you don’t even know which byte substrings exist in memory at any given time.(Languages with powerful optimizers or macro systems like Haskell or Rust might actually do that by translating all your string-function calls into calls directly on the steam of bytes, but from your perspective that’s entirely under the covers, and you’re doing the same thing you do in Python.)