Mailman 3 Change magic strings to enums - Python-ideas

Change magic strings to enums

Jacco van Dorp

24 Apr 2018 24 Apr '18

12:52 p.m.

A bit ago I was reading some of the python docs ( https://docs.python.org/3.6/library/warnings.html ), the warning module, and I noticed a table of magic strings. I can think of a few other places where magic strings are used - for example, string encoding/decoding locales and strictness, and probably a number of other places. Since Python 3.4, We've been having Enums. Wouldn't it be cleaner to use enums by default instead of those magic strings ? for example, for warnings filter actions, (section 29.5.2), quite near the top of the page. You could declare in the warnings module: class FilterAction(Enum): Error = 'error' Ignore = 'ignore' Always = 'always' Default = 'default' Module = 'module' Once = 'once' And put in the docs that people should use enums. For as long as a transition period would last, any entrypoint into the module where currently the magic string is used, you could transform it with the single line: action = FilterAction(action) # transforms from old magic string to shiny new enum member Then whenever enough people have shifted/4.0 comes around, the string argument version can be deprecated. Pro's - no magic strings - more clarity - easier to get all possible values with for example type checking editors (because of the type annotation) Con's : - implementation effort (as long as the modules are in pure Python, I could perhaps help. Im not to confident about my C skills tho) Backwards compatibility wouldn't be an issue because of the easy transformation from the old string as long as we use those string as the enum values. ofc, precise names of enums/members is up for debate. I personally prefer the above version to ALLCAPS, but that might be my comparitive lack of C experience. These named constants are generally done in all caps, but they're also often because of lack of a simple enum class. I tried to search for this, but couldn't find any discussion about it. Apologies if this has been rejected before. Jacco

Show replies by date

Nick Coghlan

24 Apr 24 Apr

1:58 p.m.

On 24 April 2018 at 22:52, Jacco van Dorp wrote:

...

A bit ago I was reading some of the python docs ( https://docs.python.org/3.6/library/warnings.html ), the warning module, and I noticed a table of magic strings.

I can think of a few other places where magic strings are used - for example, string encoding/decoding locales and strictness, and probably a number of other places.

Since Python 3.4, We've been having Enums.

Wouldn't it be cleaner to use enums by default instead of those magic strings ? for example, for warnings filter actions, (section 29.5.2), quite near the top of the page.

"It's cleaner" isn't a user problem though. The main justification for using enums is that they're easier to interpret in log messages and expection tracebacks than opaque constants, and that argument is much weaker for well-chosen string constants than it is for other constants (like the numeric constants in the socket and errno modules). For backwards compatibility reasons, we'd want to keep accepting the plain string versions anyway (implicitly converting them to their enum counterparts). At a less philosophical level, many of the cases where we use magic strings are in code that has to work even when the import system isn't working yet - that's relatively straightforward to achieve when the code is only relying on strings with particular contents, but *much* harder if they're relying on a higher level type like enum objects. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Jacco van Dorp

3:06 p.m.

I 2018-04-24 15:58 GMT+02:00 Nick Coghlan :

...

On 24 April 2018 at 22:52, Jacco van Dorp wrote:

...
Wouldn't it be cleaner to use enums by default instead of those magic strings ? for example, for warnings filter actions, (section 29.5.2), quite near the top of the page.

"It's cleaner" isn't a user problem though. The main justification for using enums is that they're easier to interpret in log messages and expection tracebacks than opaque constants, and that argument is much weaker for well-chosen string constants than it is for other constants (like the numeric constants in the socket and errno modules).

For backwards compatibility reasons, we'd want to keep accepting the plain string versions anyway (implicitly converting them to their enum counterparts).

At a less philosophical level, many of the cases where we use magic strings are in code that has to work even when the import system isn't working yet - that's relatively straightforward to achieve when the code is only relying on strings with particular contents, but *much* harder if they're relying on a higher level type like enum objects.

Cheers, Nick.

I guess we could add inconsistency as a con, then, since if the import system isn't working at places where you'd like to use the Enums (or even executing python code ?). This would mean that to the casual observer, it'd be arbitrary where they could be used instead. I wonder how many of these would be in places used by most people, though. I don't mind putting in some time to figure it out, but I have no idea where to start. Is there any easily searchable place where I could scan the standard library for occurences of magic strings ?

Chris Angelico

3:18 p.m.

On Wed, Apr 25, 2018 at 1:06 AM, Jacco van Dorp wrote:

...

I

2018-04-24 15:58 GMT+02:00 Nick Coghlan :

...
On 24 April 2018 at 22:52, Jacco van Dorp wrote:

...
Wouldn't it be cleaner to use enums by default instead of those magic strings ? for example, for warnings filter actions, (section 29.5.2), quite near the top of the page.

"It's cleaner" isn't a user problem though. The main justification for using enums is that they're easier to interpret in log messages and expection tracebacks than opaque constants, and that argument is much weaker for well-chosen string constants than it is for other constants (like the numeric constants in the socket and errno modules).

For backwards compatibility reasons, we'd want to keep accepting the plain string versions anyway (implicitly converting them to their enum counterparts).

At a less philosophical level, many of the cases where we use magic strings are in code that has to work even when the import system isn't working yet - that's relatively straightforward to achieve when the code is only relying on strings with particular contents, but *much* harder if they're relying on a higher level type like enum objects.

Cheers, Nick.

I guess we could add inconsistency as a con, then, since if the import system isn't working at places where you'd like to use the Enums (or even executing python code ?). This would mean that to the casual observer, it'd be arbitrary where they could be used instead.

I wonder how many of these would be in places used by most people, though. I don't mind putting in some time to figure it out, but I have no idea where to start. Is there any easily searchable place where I could scan the standard library for occurences of magic strings ?

First, though, can you enumerate (pun intended) the problems with magic strings? You list "no magic strings" as a benefit, as if it's self-evident; I'm not sure that it is. ChrisA

Steven D'Aprano

5:19 p.m.

On Wed, Apr 25, 2018 at 01:18:10AM +1000, Chris Angelico wrote:

...

First, though, can you enumerate (pun intended) the problems with magic strings? You list "no magic strings" as a benefit, as if it's self-evident; I'm not sure that it is.

It shouldn't be self-evident, because the use of strings in the warnings module doesn't match the most common accepted meaning of magic strings. https://en.wikipedia.org/wiki/Magic_string Possibly Jacco was thinking of "magic constants": https://en.wikipedia.org/wiki/Magic_number_%28programming%29#Unnamed_numeric... (although in this case, they're text constants, not numerical). But this seems like a fairly benign example to my eyes: the strings aren't likely to change their values. As discussed here: https://softwareengineering.stackexchange.com/questions/221034/usage-of-magi... not all uses of literals are harmful. A fairly lightweight change would be to add named constants to the warning module: ERROR = 'error' etc, and refactor the module to use the named constants instead of hard-coded strings. I'm surprised that it doesn't already do that. That would be 100% backwards compatible, without the cost of importing and creating enums, but allow consumers of the warnings module to inspect it and import symbolic names if they so choose: from warnings import ERROR By the way, I notice that the warnings module makes heavy use of assert to check user-supplied input. That's dangerous since asserts can be disabled, and also poor API design: it means that even if the asserts trigger on error (which isn't guaranteed), they raise the wrong kind of exception: AssertionError instead of TypeError for bad types or ValueError for bad values. -- Steve

Chris Angelico

5:42 p.m.

On Wed, Apr 25, 2018 at 3:19 AM, Steven D'Aprano wrote:

...

On Wed, Apr 25, 2018 at 01:18:10AM +1000, Chris Angelico wrote:

...
First, though, can you enumerate (pun intended) the problems with magic strings? You list "no magic strings" as a benefit, as if it's self-evident; I'm not sure that it is.

It shouldn't be self-evident, because the use of strings in the warnings module doesn't match the most common accepted meaning of magic strings.

https://en.wikipedia.org/wiki/Magic_string

Possibly Jacco was thinking of "magic constants":

https://en.wikipedia.org/wiki/Magic_number_%28programming%29#Unnamed_numeric...

(although in this case, they're text constants, not numerical).

I assumed this to be the case, yes. Many people decry "magic numbers" where the number 7 means one thing, and the number 83 means something else; but I'm less convinced that the textual equivalent is as problematic. (If not "magic strings", what would you call them?)

...

A fairly lightweight change would be to add named constants to the warning module:

ERROR = 'error'

etc, and refactor the module to use the named constants instead of hard-coded strings. I'm surprised that it doesn't already do that.

That would be 100% backwards compatible, without the cost of importing and creating enums, but allow consumers of the warnings module to inspect it and import symbolic names if they so choose:

from warnings import ERROR

Yeah, that would be one way to do it. But I'd still like to know what problems are being solved by this, as a means of determining whether they're being solved adequately. Is it the risk of misspellings? Because that can happen just as easily with the imported name as with a string literal. (The from-import pollutes your namespace, and "warnings.EROR" is a run-time failure just as much as a misspelled string literal would be.) Is it the need to list all the possible strings? That could be done with something like __future__.all_feature_names or the way the logging module translates level names into numbers. Something else? ChrisA

Nick Coghlan

3:49 p.m.

On 25 April 2018 at 01:06, Jacco van Dorp wrote:

...

I guess we could add inconsistency as a con, then, since if the import system isn't working at places where you'd like to use the Enums (or even executing python code ?). This would mean that to the casual observer, it'd be arbitrary where they could be used instead.

Running './python -X importtime -Wd -c "pass"' with Python 3.7 gives a pretty decent list of the parts of the standard library that constitute the low level core that we try to keep independent of everything else (there's a slightly smaller core that omits the warning module and it's dependencies - leaving "-Wd" off the command line will give that list).

...

I wonder how many of these would be in places used by most people, though.

...

I don't mind putting in some time to figure it out, but I have no idea where to start. Is there any easily searchable place where I could scan the standard library for occurences of magic strings ?

Searching the documentation for :data: fields, and then checking those to see which ones had already been converted to enums would likely be your best bet. You wouldn't be able to get a blanket approval for "Let's convert all the magic strings to Enums" though - you'd need to make the case that each addition of a new Enum provided a genuine API improvement for the affected module (e.g. I suspect a plausible case could potentially be made for converting some of the inspect module state introspection APIs over to StringEnum, so it was easier to iterate over the valid states in a consistent way, but even there I'd need to see a concrete proposal before I made up my mind). Making the case for IntEnum usage tends to be much easier, simply due to the runime introspection benefits that it brings. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Antoine Pitrou

5:32 p.m.

On Tue, 24 Apr 2018 23:58:19 +1000 Nick Coghlan wrote:

...

On 24 April 2018 at 22:52, Jacco van Dorp wrote:

...
A bit ago I was reading some of the python docs ( https://docs.python.org/3.6/library/warnings.html ), the warning module, and I noticed a table of magic strings.

I can think of a few other places where magic strings are used - for example, string encoding/decoding locales and strictness, and probably a number of other places.

Since Python 3.4, We've been having Enums.

Wouldn't it be cleaner to use enums by default instead of those magic strings ? for example, for warnings filter actions, (section 29.5.2), quite near the top of the page.

"It's cleaner" isn't a user problem though. The main justification for using enums is that they're easier to interpret in log messages and expection tracebacks than opaque constants, and that argument is much weaker for well-chosen string constants than it is for other constants (like the numeric constants in the socket and errno modules).

Also beware the import time cost of having a widely-used module like "warnings" depend on the "enum" module and its own dependencies. Regards Antoine.

Ethan Furman

6:56 p.m.

On 04/24/2018 10:32 AM, Antoine Pitrou wrote:

...

Also beware the import time cost of having a widely-used module like "warnings" depend on the "enum" module and its own dependencies.

With all the recent changes to Python, I should go through and see which dependencies are no longer needed. -- ~Ethan~

Nick Coghlan

25 Apr 25 Apr

3:04 a.m.

On 25 April 2018 at 04:56, Ethan Furman wrote:

...

On 04/24/2018 10:32 AM, Antoine Pitrou wrote:

...
Also beware the import time cost of having a widely-used module like "warnings" depend on the "enum" module and its own dependencies.

With all the recent changes to Python, I should go through and see which dependencies are no longer needed.

I was checking this with "./python -X importtime -c 'import enum'", and the overall import time was around 9 ms with a cold disk cache, and 2 ms with a warm one. In both cases, importing "types" and "_collections" accounted for around a 3rd of the time, with the bulk of the execution time being enum's own module level code. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

INADA Naoki

5:03 a.m.

On Wed, Apr 25, 2018 at 12:04 PM, Nick Coghlan wrote:

...

On 25 April 2018 at 04:56, Ethan Furman wrote:

...
On 04/24/2018 10:32 AM, Antoine Pitrou wrote:

...
Also beware the import time cost of having a widely-used module like "warnings" depend on the "enum" module and its own dependencies.

With all the recent changes to Python, I should go through and see which dependencies are no longer needed.

I was checking this with "./python -X importtime -c 'import enum'", and the overall import time was around 9 ms with a cold disk cache, and 2 ms with a warm one. In both cases, importing "types" and "_collections" accounted for around a 3rd of the time, with the bulk of the execution time being enum's own module level code.

enum class&member creation cost is much heavier than "import enum" cost. Especially, "import socket, ssl" is much slower than before... -- INADA Naoki

Ivan Levkivskyi

7:57 a.m.

On 25 April 2018 at 06:03, INADA Naoki wrote:

...

On Wed, Apr 25, 2018 at 12:04 PM, Nick Coghlan wrote:

...
On 25 April 2018 at 04:56, Ethan Furman wrote:

...
On 04/24/2018 10:32 AM, Antoine Pitrou wrote:

...
Also beware the import time cost of having a widely-used module like "warnings" depend on the "enum" module and its own dependencies.

With all the recent changes to Python, I should go through and see which dependencies are no longer needed.

I was checking this with "./python -X importtime -c 'import enum'", and the overall import time was around 9 ms with a cold disk cache, and 2 ms with a warm one. In both cases, importing "types" and "_collections" accounted for around a 3rd of the time, with the bulk of the execution time being enum's own module level code.

enum class&member creation cost is much heavier than "import enum" cost. Especially, "import socket, ssl" is much slower than before...

Is it slow simply because we are creating new class objects or EnumMeta.__new__ does some extensive calculations? In the latter case rewriting EnumMeta in C might help. -- Ivan

Jacco van Dorp

8:06 a.m.

...

First, though, can you enumerate (pun intended) the problems with magic strings? You list "no magic strings" as a benefit, as if it's self-evident; I'm not sure that it is.

ChrisA

One of my main reasons would be the type-checking from tools like Pycharm, which is the one I use. If I don't remember the exact strings, I won't have to switch to my browser to look up the documentation, but instead I type the enum name, and the typechecker will give me the members with correct spelling - all I need to remember is a vague idea of what option did what. The option names will be reminders instead of the thing to remember. Perhaps the string encode/decode would be a better case, tho. Is it latin 1 or latin-1 ? utf-8 or UTF-8 ? They might be fast to look up if you know where to look (probably the top result of googling "python string encoding utf 8", and it's the second and first option respectively IIRC. But I shouldn't -have- to recall correctly), but it's still a lot faster if you can type "Encoding.U" and it gives you the option. I'll go and see if I can make a small list of modules using these kind of strings that aren't of the essential core when I get home this evening. My apologies if magic strings isn't the correct word. Despite that, I believe everyone knows what I intend to say.

Chris Angelico

8:30 a.m.

On Wed, Apr 25, 2018 at 6:06 PM, Jacco van Dorp wrote:

...

...
First, though, can you enumerate (pun intended) the problems with magic strings? You list "no magic strings" as a benefit, as if it's self-evident; I'm not sure that it is.

ChrisA

One of my main reasons would be the type-checking from tools like Pycharm, which is the one I use. If I don't remember the exact strings, I won't have to switch to my browser to look up the documentation, but instead I type the enum name, and the typechecker will give me the members with correct spelling - all I need to remember is a vague idea of what option did what. The option names will be reminders instead of the thing to remember.

Perhaps the string encode/decode would be a better case, tho. Is it latin 1 or latin-1 ? utf-8 or UTF-8 ? They might be fast to look up if you know where to look (probably the top result of googling "python string encoding utf 8", and it's the second and first option respectively IIRC. But I shouldn't -have- to recall correctly), but it's still a lot faster if you can type "Encoding.U" and it gives you the option.

There are so many encodings that I don't think an enum would be practical. Also, their canonical names are not valid identifiers, so you would have to futz around just as much - is it Encoding.ISO_8859_1 or Encoding.ISO88591 or something else? Perhaps an alternative tool in PyCharm is the solution. There's no reason that you can't have tab completion inside string literals; imagine, for instance, if >> open("/usr/lo << could tab-complete "/usr/local" and let you fill in a valid path name from your file system. Tab-completing a set of common encodings would be reasonably easy. Tab-completing a set of constant strings for the warnings module, even easier. Maybe there could be a way to store this info on a function object, and then PyCharm just has to follow instructions ("this arg of this function uses these strings")? Possibly as a type annotation, even - instead of saying "this takes a string", it can say "this takes a string drawn from these options"? The strings themselves don't have to be any different. ChrisA

Jacco van Dorp

9:12 a.m.

2018-04-25 10:30 GMT+02:00 Chris Angelico :

...

On Wed, Apr 25, 2018 at 6:06 PM, Jacco van Dorp wrote:

...
...
First, though, can you enumerate (pun intended) the problems with magic strings? You list "no magic strings" as a benefit, as if it's self-evident; I'm not sure that it is.

ChrisA

One of my main reasons would be the type-checking from tools like Pycharm, which is the one I use. If I don't remember the exact strings, I won't have to switch to my browser to look up the documentation, but instead I type the enum name, and the typechecker will give me the members with correct spelling - all I need to remember is a vague idea of what option did what. The option names will be reminders instead of the thing to remember.

Perhaps the string encode/decode would be a better case, tho. Is it latin 1 or latin-1 ? utf-8 or UTF-8 ? They might be fast to look up if you know where to look (probably the top result of googling "python string encoding utf 8", and it's the second and first option respectively IIRC. But I shouldn't -have- to recall correctly), but it's still a lot faster if you can type "Encoding.U" and it gives you the option.

There are so many encodings that I don't think an enum would be practical. Also, their canonical names are not valid identifiers, so you would have to futz around just as much - is it Encoding.ISO_8859_1 or Encoding.ISO88591 or something else?

Which is where the auto-completion comes in. type Encoding.IS - and you'd have a list of options. If the function is annotated with the type of the enum, it'll even suggest you that. I'll freely admit that the amount of encodings might make this a bad idea. On the other hand....it might make it a good idea as well. See a list of possibilities, and the IDE can filter it as you type. IIRC tho, you can add encodings at runtime, while you can't add Enum members. If you actually can, than this might be unsuitable for Enum solution. (looking at the docs 7.2:codecs though, the error handling strings look...tasty ;p)

...

Perhaps an alternative tool in PyCharm is the solution. There's no reason that you can't have tab completion inside string literals; imagine, for instance, if >> open("/usr/lo << could tab-complete "/usr/local" and let you fill in a valid path name from your file system. Tab-completing a set of common encodings would be reasonably easy. Tab-completing a set of constant strings for the warnings module, even easier.

But where would you get a list of these strings, and how'd you define that list of strings ? It's currently commonly accepted that annotations are for types, and the later PEP's about the subject seem to assume this without question. Another annotation-like functionality ? hardcode the list of possibilities inside pycharm ? Also, perhaps I want to select the string first, store in into a variable, then pass that variable into a function. How would any checker ever know what im going to feed that string into ? But an Enum, I can use that as type annotation myself and then it'll know without question which are legal arguments.

...

Maybe there could be a way to store this info on a function object, and then PyCharm just has to follow instructions ("this arg of this function uses these strings")? Possibly as a type annotation, even - instead of saying "this takes a string", it can say "this takes a string drawn from these options"? The strings themselves don't have to be any different.

ChrisA

Pycharm doesn't execute your code - it scans it. It wont know what you store on a function object. As whether type annotations can mean "a string from either of these options" would require a currently not known to me interpretation of type annotations - and probably incompatible with what most people currenlty use them for. Forgive me if I misunderstand you, but aren't you really just trying to use those strings as enum members when you define a function like "takes one of these strings as argument" ? Because as far as I know, except from some fluff, that's exactly what enums are and are intended for - a unique set of keys that all have special meaning.

Chris Angelico

9:46 a.m.

On Wed, Apr 25, 2018 at 7:12 PM, Jacco van Dorp wrote:

...

Pycharm doesn't execute your code - it scans it. It wont know what you store on a function object.

How does it currently know what type something is? If you define something using an enum, how is PyCharm going to know what the members of that enum are?

...

Forgive me if I misunderstand you, but aren't you really just trying to use those strings as enum members when you define a function like "takes one of these strings as argument" ? Because as far as I know, except from some fluff, that's exactly what enums are and are intended for - a unique set of keys that all have special meaning.

That's *one of* the things you can do with an enum. There are many other features of enumerations, and I'm trying to boil your idea down to its most compact form. You don't need all the power of Enum - you just want to be able to list off the valid options. ChrisA

Guido van Rossum

3:05 p.m.

On Wed, Apr 25, 2018, 02:13 Jacco van Dorp wrote:

...

... Which is where the auto-completion comes in. ...

Designing the language with auto-complete in mind feels wrong to me. It assumes a very sophisticated IDE and may lead to lazy design compromises. --Guido

...

Soni L.

11:11 p.m.

On 2018-04-25 12:05 PM, Guido van Rossum wrote:

...

On Wed, Apr 25, 2018, 02:13 Jacco van Dorp mailto:j.van.dorp@deonet.nl> wrote:

... Which is where the auto-completion comes in. ...

Designing the language with auto-complete in mind feels wrong to me. It assumes a very sophisticated IDE and may lead to lazy design compromises.

You can tab-complete enums (in the REPL), but not strings. Tab-complete is not an IDE thing, it's a CPython REPL thing. It seems reasonable to design for it. Any IDE worth my time would support string autocompletion, anyway. ;)

...

--Guido

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Jacco van Dorp

26 Apr 26 Apr

6:55 a.m.

Even if not just for the autocompletion, it would be more explicit that it's not just a random string like you'd pass to print(), but it has a specific meaning. Something in PEP 20 about explicit and implicit ? Autocompletion might be a good advantage, but 1) the IDE would need to know what to autocomplete it to, and you probably shouldn't special-case the stdlib like you'd need to with strings, and 2) enums -are- more explicit. When there's a distinct and limited set of options, they're just the tool for the job. (or at least, a much better tool for this job than to remember colors, which is used all over their documentation). Im naming auto-completion here as a solid reason, but when clean code itself can be considered a solid reason, I think that's probably the real reason. 2018-04-26 1:11 GMT+02:00 Soni L. :

...

On 2018-04-25 12:05 PM, Guido van Rossum wrote:

On Wed, Apr 25, 2018, 02:13 Jacco van Dorp wrote:

...
... Which is where the auto-completion comes in. ...

Designing the language with auto-complete in mind feels wrong to me. It assumes a very sophisticated IDE and may lead to lazy design compromises.

You can tab-complete enums (in the REPL), but not strings. Tab-complete is not an IDE thing, it's a CPython REPL thing. It seems reasonable to design for it.

Any IDE worth my time would support string autocompletion, anyway. ;)

--Guido

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Steven D'Aprano

25 Apr 25 Apr

10:21 a.m.

On Wed, Apr 25, 2018 at 10:06:56AM +0200, Jacco van Dorp wrote:

...

Perhaps the string encode/decode would be a better case, tho. Is it latin 1 or latin-1 ? utf-8 or UTF-8 ?

py> 'abc'.encode('latin 1') == 'abc'.encode('LATIN-1') True py> 'abc'.encode('utf8') == 'abc'.encode('UTF 8') == 'abc'.encode('UtF_8') True Encoding names are normalised before being used.

...

They might be fast to look up if you know where to look (probably the top result of googling "python string encoding utf 8", and it's the second and first option respectively IIRC. But I shouldn't -have- to recall correctly), but it's still a lot faster if you can type "Encoding.U" and it gives you the option.

If you did this with Encodings.ISO you would get a couple of dozen possibilities. ISO-8859-1 ISO-8859-7 ISO-8859-14 ISO-8859-15 etc, just to pick a few at random. How do you know which one you want? In general, there's not really much *practical* use-case for code completion on encodings, aside from just exploratory mucking about in the interactive interpreter. There are too many codecs (multiple dozen), the names are too similar and not self-explanatory, and they can have aliases. It would be like doing code-completion on an object and getting a couple of dozen methods looking like method1245 method1246 method1247 method2390 method2395 Besides, aside from UTF-16, UTF-8 and ASCII, we shouldn't encourage the use of most codecs except for legacy data. And when working with legacy data, we really need to know ahead of time what the encoding is, and declare it as constant or application option. (Or, worst case, we've used chardet or another encoding guesser, and stored the name of the encoding in a variable.) I don't really see a big advantage aside from laziness for completing on encodings. And while laziness is a virtue in programmers, that only goes so far before it becomes silly. Having to type import encodings enc <tab> .Enc <tab> .u <tab> arrow arrow arrow arrow arrow arrow enter (19 key presses, plus the import) to save from having to type 'utf8' (six keypresses) is not what I would call efficient use of programmer time and effort. (Why so many arrows? Since you'll have to tab past at least utf16 utf16be utf16le utf32 utf32be utf32le utf7 before you get to utf8.) But the biggest problem is that they aren't currently available for introspection anywhere. You can register new codecs, but there's no API for querying the list of currently registered codecs or their aliases. I think that problem would need to be solved first, in which case code completion will then be either easy, or irrelevant. (I'd be perfectly satisfied with an API I could call from the interactive interpreter.) -- Steve

Serhiy Storchaka

10:03 a.m.

25.04.18 10:57, Ivan Levkivskyi пише:

...

On 25 April 2018 at 06:03, INADA Naoki mailto:songofacandy@gmail.com> wrote: enum class&member creation cost is much heavier than "import enum" cost. Especially, "import socket, ssl" is much slower than before...

Is it slow simply because we are creating new class objects or EnumMeta.__new__ does some extensive calculations? In the latter case rewriting EnumMeta in C might help.

Creating a new function is very cheap -- just around 50 ns on my computer. Creating a new class is over two orders costly -- around 7 us for an empty class on my computer. Creating a new Enum class is much more costly -- around 40 us for an empty class (or 50 us for IntEnum) plus 7 us per member. Creating a new namedtuple type has the same cost as creating Enum class. It was much costly before 3.7. Thus creating a typical Enum class with 3-5 members is like creating 10 normal classes. Not much modules have 10 classes.

Ivan Levkivskyi

10:15 a.m.

On 25 April 2018 at 11:03, Serhiy Storchaka wrote:

...

25.04.18 10:57, Ivan Levkivskyi пише:

...
On 25 April 2018 at 06:03, INADA Naoki > wrote: enum class&member creation cost is much heavier than "import enum" cost. Especially, "import socket, ssl" is much slower than before...

Is it slow simply because we are creating new class objects or EnumMeta.__new__ does some extensive calculations? In the latter case rewriting EnumMeta in C might help.

Creating a new function is very cheap -- just around 50 ns on my computer.

Creating a new class is over two orders costly -- around 7 us for an empty class on my computer.

Creating a new Enum class is much more costly -- around 40 us for an empty class (or 50 us for IntEnum) plus 7 us per member.

Hm, this is what I wanted to know. I think by rewriting EnumMeta in C we can reduce the creation time of an Enum class (almost) down to the creation time of a normal class, which may be a 4-5x speed-up. What do you think? -- Ivan

Serhiy Storchaka

11:01 a.m.

25.04.18 13:15, Ivan Levkivskyi пише:

...

Hm, this is what I wanted to know. I think by rewriting EnumMeta in C we can reduce the creation time of an Enum class (almost) down to the creation time of a normal class, which may be a 4-5x speed-up. What do you think?

It could be great. But I afraid this may add too much complexity in C code. Maybe try to implement a simple and fast Enum for using it in the stdlib and extend it with a richer interface in the enum module?

Antoine Pitrou

10:03 p.m.

On Wed, 25 Apr 2018 14:01:43 +0300 Serhiy Storchaka wrote:

...

25.04.18 13:15, Ivan Levkivskyi пише:

...
Hm, this is what I wanted to know. I think by rewriting EnumMeta in C we can reduce the creation time of an Enum class (almost) down to the creation time of a normal class, which may be a 4-5x speed-up. What do you think?

It could be great. But I afraid this may add too much complexity in C code. Maybe try to implement a simple and fast Enum for using it in the stdlib and extend it with a richer interface in the enum module?

Or perhaps we want a way to eschew the current complexity of the EnumMeta constructor, for example by passing some pre-computed values. Regards Antoine.

Ivan Levkivskyi

26 Apr 26 Apr

7:38 a.m.

On 25 April 2018 at 12:01, Serhiy Storchaka wrote:

...

25.04.18 13:15, Ivan Levkivskyi пише:

...
Hm, this is what I wanted to know. I think by rewriting EnumMeta in C we can reduce the creation time of an Enum class (almost) down to the creation time of a normal class, which may be a 4-5x speed-up. What do you think?

It could be great. But I afraid this may add too much complexity in C code. Maybe try to implement a simple and fast Enum for using it in the stdlib and extend it with a richer interface in the enum module?

I think we can do something similar to ABCMeta, i.e. the metaclass itself will stay defined in Python, but the "hottest" parts of its methods will be replaced with helper functions written in C. This way we can limit complexity of the C code while still getting almost all the performance benefits. -- Ivan

Antoine Pitrou

8:42 a.m.

On Thu, 26 Apr 2018 08:38:43 +0100 Ivan Levkivskyi wrote:

...

On 25 April 2018 at 12:01, Serhiy Storchaka wrote:

...
25.04.18 13:15, Ivan Levkivskyi пише:

...
Hm, this is what I wanted to know. I think by rewriting EnumMeta in C we can reduce the creation time of an Enum class (almost) down to the creation time of a normal class, which may be a 4-5x speed-up. What do you think?

It could be great. But I afraid this may add too much complexity in C code. Maybe try to implement a simple and fast Enum for using it in the stdlib and extend it with a richer interface in the enum module?

I think we can do something similar to ABCMeta, i.e. the metaclass itself will stay defined in Python, but the "hottest" parts of its methods will be replaced with helper functions written in C.

It's still a PITA to maintain, and it's not nice to Ethan (the main enum maintainer) if suddenly he has to act on C code when he wants to fix a bug or add a feature. Regards Antoine.

INADA Naoki

8:52 a.m.

...

...
It could be great. But I afraid this may add too much complexity in C code. Maybe try to implement a simple and fast Enum for using it in the stdlib and extend it with a richer interface in the enum module?

...

I think we can do something similar to ABCMeta, i.e. the metaclass itself will stay defined in Python, but the "hottest" parts of its methods will be replaced with helper functions written in C. This way we can limit complexity of the C code while still getting almost all the performance benefits.

...

-- Ivan

Adding C speedup module has high bar, especially performance gain is only on startup time. Personally speaking, I want speedup module for `re.compile` than enum. In case of enum, I feel CONSTANT = "magic" is enough for most cases. re doesn't have such workaround, and using third party regular expression library has very high bar, especially when using it in stdlib. But there were some -1 on adding speedup module for re. I think it's same to enum. So I think we should: * Don't use enum blindly; use it only when it's very far better than CONST = "magic". * Add faster API which bypass some slow part, especially for IntEnum.convert() or IntFlag.convert() in socket module. Regards, -- INADA Naoki

Jacco van Dorp

9:37 a.m.

I'm kind of curious why everyone here seems to want to use IntFlags and other mixins. The docs themselves say that their use should be minimized, and tbh I agree with them. Backwards compatiblity can be maintained by allowing the old value and internally converting it to the enum. Combinability is inherent to enum.Flags. There'd be no real reason to use mixins as far as I can see ? 2018-04-26 10:52 GMT+02:00 INADA Naoki :

...

...
...
It could be great. But I afraid this may add too much complexity in C code. Maybe try to implement a simple and fast Enum for using it in the stdlib and extend it with a richer interface in the enum module?

...
I think we can do something similar to ABCMeta, i.e. the metaclass itself will stay defined in Python, but the "hottest" parts of its methods will be replaced with helper functions written in C. This way we can limit complexity of the C code while still getting almost all the performance benefits.

...
-- Ivan

Adding C speedup module has high bar, especially performance gain is only on startup time.

Personally speaking, I want speedup module for `re.compile` than enum. In case of enum, I feel CONSTANT = "magic" is enough for most cases. re doesn't have such workaround, and using third party regular expression library has very high bar, especially when using it in stdlib.

But there were some -1 on adding speedup module for re. I think it's same to enum.

So I think we should:

* Don't use enum blindly; use it only when it's very far better than CONST = "magic". * Add faster API which bypass some slow part, especially for IntEnum.convert() or IntFlag.convert() in socket module.

Regards,

-- INADA Naoki _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Nick Coghlan

1:26 p.m.

On 26 April 2018 at 19:37, Jacco van Dorp wrote:

...

I'm kind of curious why everyone here seems to want to use IntFlags and other mixins. The docs themselves say that their use should be minimized, and tbh I agree with them. Backwards compatiblity can be maintained by allowing the old value and internally converting it to the enum. Combinability is inherent to enum.Flags. There'd be no real reason to use mixins as far as I can see ?

Serialisation formats are a good concrete example of how problems can arise by switching out concrete types on people:

...

...
...
import enum, json a = "A" class Enum(enum.Enum): ... a = "A" ... class StrEnum(str, enum.Enum): ... a = "A" ... json.dumps(a) '"A"' json.dumps(StrEnum.a) '"A"' json.dumps(Enum.a) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python3.6/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib64/python3.6/json/encoder.py", line 180, in default o.__class__.__name__) TypeError: Object of type 'Enum' is not JSON serializable

The mixin variants basically say "If you run into code that doesn't natively understand enums, act like an instance of this type". Since most of the standard library has been around for years, and sometimes even decades, we tend to face a *lot* of backwards compatibility requirements along those lines. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Jacco van Dorp

30 Apr 30 Apr

7:04 a.m.

2018-04-26 15:26 GMT+02:00 Nick Coghlan :

...

On 26 April 2018 at 19:37, Jacco van Dorp wrote:

...
I'm kind of curious why everyone here seems to want to use IntFlags and other mixins. The docs themselves say that their use should be minimized, and tbh I agree with them. Backwards compatiblity can be maintained by allowing the old value and internally converting it to the enum. Combinability is inherent to enum.Flags. There'd be no real reason to use mixins as far as I can see ?

Serialisation formats are a good concrete example of how problems can arise by switching out concrete types on people:

...
...
...
import enum, json a = "A" class Enum(enum.Enum): ... a = "A" ... class StrEnum(str, enum.Enum): ... a = "A" ... json.dumps(a) '"A"' json.dumps(StrEnum.a) '"A"' json.dumps(Enum.a) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python3.6/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib64/python3.6/json/encoder.py", line 180, in default o.__class__.__name__) TypeError: Object of type 'Enum' is not JSON serializable

The mixin variants basically say "If you run into code that doesn't natively understand enums, act like an instance of this type".

Since most of the standard library has been around for years, and sometimes even decades, we tend to face a *lot* of backwards compatibility requirements along those lines.

Cheers, Nick.

However, as the docs not, they will be comparable with each other, which should throw an error. Since this is only a problem for this case when serializing (since the functions would allow the old str arguments for probably ever), shouldn't this be something caught when you upgrade the version you run your script under ? It's also a rather simple fix: json.dumps(Enum.a.value) would work just fine. Now im aware that most people don't have 100% test coverage and such. I also rather lack the amount of experience you guys have. Guess im just a bit behind on the practicality beats purity here :) Jacco

Ethan Furman

25 Apr 25 Apr

4:20 p.m.

On 04/25/2018 03:15 AM, Ivan Levkivskyi wrote:

...

On 25 April 2018 at 11:03, Serhiy Storchaka wrote:

...

...
Creating a new function is very cheap -- just around 50 ns on my computer.

Creating a new class is over two orders costly -- around 7 us for an empty class on my computer.

Creating a new Enum class is much more costly -- around 40 us for an empty class (or 50 us for IntEnum) plus 7 us per member.

Hm, this is what I wanted to know. I think by rewriting EnumMeta in C we can reduce the creation time of an Enum class (almost) down to the creation time of a normal class, which may be a 4-5x speed-up. What do you think?

Someone else would have to take that on (a C version of EnumMeta) -- my C skills are not up to that task and I do not currently possess the time to get them there. -- ~Ethan~

2186

Age (days ago)

2192

Last active (days ago)

List overview

Download

30 comments

11 participants

participants (11)

Antoine Pitrou
Chris Angelico
Ethan Furman
Guido van Rossum
INADA Naoki
Ivan Levkivskyi
Jacco van Dorp
Nick Coghlan
Serhiy Storchaka
Soni L.
Steven D'Aprano

Change magic strings to enums

Jacco van Dorp

Jacco van Dorp

Jacco van Dorp

Jacco van Dorp

Jacco van Dorp

Jacco van Dorp

Jacco van Dorp

tags

participants (11)