On Jan 20, 2015, at 23:48, Stephen Hansen wrote:

...

What do people do now? or strings

df.sort('name')

...

...

strings work but feel unpleasant

Why?

What does :foo get you that 'foo' doesn't?

One character being saved doesn't seem like a good reason to add syntax.

Mainly they're used to fill in gaps where a language isn't dynamic enough to just use strings, which Python doesn't need. (Sometimes this is claimed to be a strength rather than a weakness. Arguably, using auto-interned symbols instead of strings for things like attribute or global lookup could be more efficient, but I don't think languages like Ruby have shown any demonstrable gain. And we just had a thread from someone who considered it a "bug" that Python allows setattr(spam, '3', 3) when 3 isn't a valid identifier, which symbols "fix" automatically. But even if these were real advantages, Python obviously isn't going to make such a radical change as banning strings for dynamic access.) A language can also add things on top of the symbol/string distinction (besides auto-interning)--e.g., you can add syntactic sugar for dynamic lookup ("foo.:bar" means "getattr(foo, bar)"), or make undefined variables have the value of their symbol so you can LBYL test them, or all kinds of other things--but again, they're all things I don't think Python wants or needs. And without a vague idea for why Python would want to do any of the things that other languages do with their symbol-vs.-string distinctions, this really does seem to come down to just saving one character.

Yeah, basically this was a fork in the road roughly 25 years ago that Python didn't take. (I was aware of the idea from Lisp, and decided Python could do with just strings instead.) There's no way adding something like this today will suddenly change everything else that is part of having "symbols"; the proposed feature would be lacking the integration with the internals that makes it work well in some other languages. On Wed, Jan 21, 2015 at 1:13 AM, Andrew Barnert < abarnert@yahoo.com.dmarc.invalid> wrote:

On Jan 20, 2015, at 23:48, Stephen Hansen wrote:

*What do people do now?*

...

or strings

df.sort('name')

...

...

strings work but feel unpleasant

Why?

What does :foo get you that 'foo' doesn't?

One character being saved doesn't seem like a good reason to add syntax.

Mainly they're used to fill in gaps where a language isn't dynamic enough to just use strings, which Python doesn't need.

(Sometimes this is claimed to be a strength rather than a weakness. Arguably, using auto-interned symbols instead of strings for things like attribute or global lookup could be more efficient, but I don't think languages like Ruby have shown any demonstrable gain. And we just had a thread from someone who considered it a "bug" that Python allows setattr(spam, '3', 3) when 3 isn't a valid identifier, which symbols "fix" automatically. But even if these were real advantages, Python obviously isn't going to make such a radical change as banning strings for dynamic access.)

A language can also add things on top of the symbol/string distinction (besides auto-interning)--e.g., you can add syntactic sugar for dynamic lookup ("foo.:bar" means "getattr(foo, bar)"), or make undefined variables have the value of their symbol so you can LBYL test them, or all kinds of other things--but again, they're all things I don't think Python wants or needs.

And without a vague idea for why Python would want to do any of the things that other languages do with their symbol-vs.-string distinctions, this really does seem to come down to just saving one character.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

On Wed, Jan 21, 2015 at 11:11 AM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

On Wed, Jan 21, 2015 at 12:42 PM, Guido van Rossum wrote:

...

There's no way adding something like this today will suddenly change everything else that is part of having "symbols"; the proposed feature would be lacking the integration with the internals that makes it work well in some other languages.

Wouldn't it be possible to simply have

def Symbol(str): def __new__(cls_, *args): istr = sys.intern(str(*args)) return str.__new__(cls, istr)

and have sys.intern() return Symbol instances at some point? I don't see any obvious compatibility problems.

I don't see any obvious advantages either.

How would this be different from adding bool as a subclass of int some time ago?

Lack of a use case?

BTW, I think Pandas found [1] the way to have memory-saving string vectors without having a symbol type in the core.

[1] http://pandas-docs.github.io/pandas-docs-travis/categorical.html

Great. -- --Guido van Rossum (python.org/~guido)

On Wed, Jan 21, 2015 at 11:38 AM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

On Wed, Jan 21, 2015 at 2:13 PM, Guido van Rossum wrote:

...

Lack of a use case?

My use case comes from interoperability with other languages. I maintain a package which provides an interface [1] between Python and the language [2] called Q. Q has 3 scalar types to represent text data (char, byte and symbol) and the corresponding vector types. When I pass Python strings to Q, I convert them to symbols because that is what users expect most of the time, but as a result, working with char vectors is unnatural.

couldn't you create a subclass of str that was exactly the same except that it was defined as mapping to a Q symbol? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Hi, On 2015-01-21 14:38, Alexander Belopolsky wrote:

[...] My use case comes from interoperability with other languages. I maintain a package which provides an interface [1] between Python and the language [2] called Q. Q has 3 scalar types to represent text data (char, byte and symbol) and the corresponding vector types. When I pass Python strings to Q, I convert them to symbols because that is what users expect most of the time, but as a result, working with char vectors is unnatural. I currently use bytes type for that, but it is really an abuse because bytes should map to byte vectors.

Hi, after thinking about it some more I've come to believe new, empty classes are the best way to model symbols in Python (for now at least). My reasoning is: - Symbols are tags (i.e. unique names for things). - In a typed language which has enums or discriminated unions (aka tagged unions), tags are often created from enums. - Enums and discriminated unions are isomorphic to base and inheriting classes. - Therefore, symbols are classes which could optionally inherit from a base class. In your case, you could define a function which dynamically creates these 'symbols' for you and then stashes them somewhere for re-use. Below, I've used the calling module's `globals()` dict, but you could store them away very neatly in some other dict to avoid any possible name conflicts. # In called module: class sym(object): pass def get_sym(sym_name, d): if s in d: return d[sym_name] new_sym = type(sym_name, (sym,), {}) d[sym_name] = new_sym return new_sym # In calling module: def char_to_sym(c): "Map a 'character' to a symbol." assert len(c) == 1 return get_sym(c, globals()) # This could be any dict. def str_to_syms(s): return map(char_to_sym, s) print str_to_syms("Hello, World!") # ==> [, ...] print char_to_sym("!") == char_to_sym("!") # ==> True Regards, Yawar

On Wed, Jan 21, 2015 at 2:11 PM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

Wouldn't it be possible to simply have

def Symbol(str):

Hmm, I should have tested the code before posting. That should have been a class and the logic would not work without modifying sys.intern first, but it does not kill the idea. Why can't we make interned strings a subclass of strings like bool is a subclass of int?

On Wed, Jan 21, 2015 at 12:58 PM, Chris Angelico wrote:

On Thu, Jan 22, 2015 at 6:18 AM, Alexander Belopolsky wrote:

...

Why can't we make interned strings a subclass of strings like bool is a subclass of int?

My gut feeling is that an interned string is exactly the same thing as a non-interned string, and that it should be possible for sys.intern() to return the original string unchanged.

I would tend to agree that there is no real difference between an interned and non-interned string in Python. If Python had mutable strings as the base type, that would be different; then having an immutable version of string would be helpful for various reasons.

But on the flip side, if you *know* that two strings are interned, you can compare their identities instead of their values; currently, identical interned strings compare equal quickly, but distinct ones still have to be compared character-for-character:

...

...

...

timeit.repeat("x==y","x='a'*65536+'q'; y='a'*65536+'q'") [2.535494999960065, 2.2198410001583397, 2.2210810002870858] timeit.repeat("x==y","import sys; x=sys.intern('a'*65536+'q'); y=sys.intern('a'*65536+'q')") [0.056215000338852406, 0.05387900024652481, 0.04834900004789233] timeit.repeat("x==y","x='a'*65536+'x'; y='a'*65536+'y'") [2.53101500030607, 2.2251750002615154, 2.228022000286728] timeit.repeat("x==y","import sys; x=sys.intern('a'*65536+'x'); y=sys.intern('a'*65536+'y')") [2.583013000432402, 2.2194420001469553, 2.224679999984801]

If, in the fourth example, the interpreter knew that both those string objects were interned, it could know that they can't possibly be the same string. So is that distinction worth coding into the class hierarchy?

If such an optimization were desirable, it would be better implemented as a flag that is accessible on the string object, but would only be writable internally (no public API for setting*). This would then allow the string compare to add a check to see if both are interned after checking the pointer values and early exit with a false result. There is not real need (that I know of) to have a class distinction to indicate interning vs a non-mutable string. Chris K *Note that hacks similar to the ones that can be used to change the value of small integers in Python may still function. Depending on internal implementation, it could also be viable to instead have the flag actually perform a lookup into the intern table, but that may negative (much of) the performance benefits.

On Thu, Jan 22, 2015 at 9:24 AM, Greg Ewing wrote:

Chris Angelico wrote:

...

If, in the fourth example, the interpreter knew that both those string objects were interned, it could know that they can't possibly be the same string. So is that distinction worth coding into the class hierarchy?

I believe there is a flag in the string object indicating that it is interned, so the interpreter *does* know this. It's just not visible to Python code.

Huh. Okay. It's not made use of for optimizing the == operator, which ought to be easy to do. Or is the cost of checking not worth it? ChrisA

Alexander Belopolsky wrote:

Why can't we make interned strings a subclass of strings like bool is a subclass of int?

Because a string can change from being non-interned to being interned at some point in its life. I suppose its tp_class could be changed at that point, but that would be surprising for an otherwise immutable type. -- Greg

Sorry, I misread what you wrote. I thought you suggested that an interned string can get "uninterned" somehow. Yes, sys.intern(x) will change x's status, but as you suggested, it can change x.__class__ as well. However, with a Symbol class in-place, I don't think there will be a need for such transitions. Even now, I don't see sys.intern() in user code too often. On Wed, Jan 21, 2015 at 9:08 PM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

On Wed, Jan 21, 2015 at 5:18 PM, Greg Ewing wrote:

...

Alexander Belopolsky wrote:

...

Why can't we make interned strings a subclass of strings like bool is a subclass of int?

Because a string can change from being non-interned to being interned at some point in its life.

I did not know that. Can you show the code that would cause such a change?

On 21.01.15 21:18, Alexander Belopolsky wrote:

Why can't we make interned strings a subclass of strings like bool is a subclass of int?

Some code (e.g. pickle, copy, lry_cache, re, pprint) relies on exact type and wouldn't work with str subtypes as good as with str.

Steven D'Aprano wrote:

On Tue, Jan 20, 2015 at 11:48:58PM -0800, Stephen Hansen wrote:

...

What does :foo get you that 'foo' doesn't?

It gets you an atomic data type which isn't a string and isn't an int.

If you want that, you're better off creating your own class. Otherwise, there's nothing to distinguish symbols used for one purpose from symbols used for another, which is the same problem recurring at another level.

Python already has a number of symbol-like objects in the language:

None True and False NotImplemented Ellipsis

And it's not hard to create more, especially now that we have Enums as a provided battery. -- Greg

On Wed Jan 21 2015 at 07:59:41 Todd wrote:

At best this saves one character, may reduce readability, and would require a huge amount of software to support another code path. Is there any advantage to this besides the one character saved?

If symbols are [a subclass of] str, I don't see why a huge amount of software - or indeed any software - would need to do anything special to support it. (Other than the implementation of the feature itself, obviously). That said, I still don't think it's very useful.