a set of enum.Enum values rather than the construction of bit-sets as the "norm"?
Maybe it is time to deemphasize the creation and passing of what is, in effect, bit-sets <https://en.wikipedia.org/wiki/Bit_array>as a single argument, and instead promote the passing of a set of members of an enum.Enum <https://docs.python.org/3/library/enum.html> constants. Thi comes about because someone wrote a description, (since deleted), of constructing bit-sets to use in passing flags to, for example, the re.compile <https://docs.python.org/3/library/re.html#re.compile>function. The use of individual bits in a bit-array/bit-set to pass multiple flags is an implementation detail. Should we not *first *teach the passing of a set of enum.Enum constant values in one argument as the *pythonic *way; and leave bit-sets and other types of enum's as a performance or interoperability detail? Comments please, (And a happy new year to you :-)
On Tue, Dec 26, 2017 at 12:07:52PM -0800, Paddy3118 wrote:
That implies that we are promoting bit sets (not merely using them). Where are we doing that?
Thi comes about because someone wrote a description, (since deleted),
"Someone"? One of the Python core developers? If not, well, the Python devs cannot be held responsible for what random people on the internet write.
It certainly is not an implementation detail -- it is a part of the public, published interface to the re module. Of course had history been different, re.compile *could* have taken a set of Enums instead. But that doesn't make it an implementation detail. That would be like saying that: re.search(pattern, text, flags=0) is an implementation detail, and we should feel free to change it to re.examine(text, pattern, set_of_enums=frozenset()) We can't change a public interface documented as taking an integer to one taking a set of Enums without going through a deprecation period. However, we could *add* an additional interface, where the re functions that currently accept an integer flag *also* accepts a set of Enums.
Should we not *first *teach the passing of a set of enum.Enum constant values in one argument as the *pythonic *way;
What makes you say that is the Pythonic way? (That's not a rhetorical question.) -- Steve
Hi Steve, I did not write an attack on the "Python devs". Re-read my original with a little less hostility and there should be room for an interpretation, (which I meant), that does not warrant such a hostile reply. The original is written in the hope of furthering discussion on the need for what is deemed pythonic , and on what Python is taught , as the language itself changes. We now have enums if you want to pass a set of flags to a function then you could have them as separate arguments - but that leads to long and cumbersome parameter lists; you could, and many do, have flags that are individual powers of two and then or them together and pass the result - creating a bitset; but we can now have the flags as separate enum.Enums and pass a set of values to a function as the flag set. This new way means that people being taught the method can use a knowledge of sets and enums - no need to know about powers of two,, what happens when they are bit-or'd together; and bitsets. We have gone to a higher level description of what we are doing; no need to mired in the details of the how of what one wants to achieve. bitsets can be taught as an optimisation. As for re, and otheralready written libraries, their is no need to change them, but other Pythoneers might well opt to not use bitsets, but rather sets of enum values. On Wednesday, 27 December 2017 05:58:00 UTC, Steven D'Aprano wrote:
On Thu, Dec 28, 2017 at 12:23:53PM -0800, Paddy3118 wrote:
Hi Steve, I did not write an attack on the "Python devs".
I didn't say you attacked anyone and your implication that I did is unfair.
Please re-read my response without the hair-trigger defensiveness. Disagreement is not hostility. Questioning your statements is not hostility. None of us are entitled to only positive responses that agree with what we suggest, but we are all entitled to the presumption that we are arguing in good faith. See also: https://mail.python.org/pipermail/python-ideas/2017-December/048448.html and: https://mail.python.org/pipermail/python-ideas/2017-December/048449.html (I don't agree with *everything* Brett says, but its a good starting point.)
Right -- and I responded to that discussion. I asked some genuine questions which you haven't responded to. Let me rephrase them: - Are we currently promoting ints as bitsets? - How is it relevant that "someone" (who?) wrote a description of using ints as bitsets and then deleted it? - Why is a set of Enums the Pythonic way?
And no one is stopping anyone from writing code that does so.
Personally, I think that sets and Enums are no easier to learn than bit twiddling. But YMMV.
We have gone to a higher level description of what we are doing; no need to mired in the details of the how of what one wants to achieve.
Whether we use an int or a set of Enums, we still need to understand the details of how to set and clear flags, and test for them. THE_FLAG = 8 the_flags = THE_FLAG | ANOTHER_FLAG if the_flags & THE_FLAG: print("flag is set") versus: class MyFlags(Enum): THE_FLAG = "whatever" the flags = {THE_FLAG, ANOTHER_FLAG} if MyFlags.THE_FLAG in the_flags: print("flag is set") Honestly, I'm not seeing a *lot* of difference here. [...]
As for re, and otheralready written libraries, their is no need to change them,
I see. It wasn't clear to me. It seemed to me that you were suggesting changing the re module, since that was just an "implementation" detail.
but other Pythoneers might well opt to not use bitsets, but rather sets of enum values.
Since ints don't provide a set-like interface, they aren't strictly speaking bitsets. But in any case, nobody is stopping people from using sets of enum values. If you have a concrete proposal, then please state it explicitly. I thought I understood your proposal: change the implementation of re to use sets of Enums in order to promote their use and act as an example of best practice and Pythonic style but apparently I misunderstood. Sorry. So now I'm left puzzled as to what you actually want. Can you be explicit and explain what you expect us to do when you say we should "teach" and "promote" (your words) Enums instead of using ints? Please be concrete. - Should we change the standard library, and if so, in what ways? - Should we change the documentation? Which parts? - Insert something in PEP 8 about using sets of Enums? - Re-write the tutorial? Where? - Something else? Without a concrete proposal, I don't think this discussion is going anywhere. -- Steve
On Fri, Dec 29, 2017 at 7:18 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I'm not sure what "set-like interface" you'd be looking for, but the built-in set type has a lot of the same operations as an integer does, and the semantics are virtually identical to a set of bits. The only one you really lack is __contains__, which could easily be added: class BitSet(int): def __contains__(self, bit): return (self & bit) == bit
Set union, intersection, etc are all provided using the same operators in sets and ints. ChrisA
On Fri, Dec 29, 2017 at 10:26:22PM +1100, Chris Angelico wrote:
The lack of support for the `in` operator is a major difference, but there's also `len` (equivalent to "count the one bits"), superset and subset testing, various in-place mutator methods, etc. Java has a BitSet class, and you can see the typical sorts of operations commonly required: https://docs.oracle.com/javase/8/docs/api/java/util/BitSet.html Of course we can emulate set-like operations using ints, but the interfaces are different, which is my point. Here's how to clear all the flags of a set or int: the_flags.clear() the_flags = 0 # clear all the bits in an int Setting a flag is *almost* the same between the two: the_flags |= {flag} # set the_flags |= flag # int although for sets, there are two other ways to set a flag which aren't supported by ints: the_flags.add(flag) the_flags.update({flag}) Similarly for clearing flags: the_flags.discard(flag) the_flags & ~flag -- Steve
On Sat, Dec 30, 2017 at 2:38 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Okay. A subclass of int could easily add a few more. Counting the 1 bits isn't difficult; superset and subset testing are actually the same as 'contains' but with more than one bit at a time. (In fact, checking if a set contains a subset is *easier* with ints than with actual sets!) Are in-place mutators that big a deal? I'm sure there are sets in languages with no mutables.
That's a consequence of Python's mutability distinction. I don't think it's a fundamental difference. You could just as easily use "the_flags = set()" if it weren't for aliasing.
That's because you can implicitly upcast a bitflag to a bitset. Effectively, ints give you a short-hand that sets can't. But if you explicitly call BitSet(flag) to create a set containing one flag, it would have the same effect.
Mutability again. If you were to create an ImmutableSet type in Python, what would its API look like? My suspicion is that it'd largely use operators, and that it'd end up looking a lot like the integer API. An integer, at its lowest level, is represented as a set of bits. It's no more crazy to use an int as a set of bits than to use a string as a set of characters: https://docs.python.org/3/library/stdtypes.html#str.strip ChrisA
On Sat, Dec 30, 2017 at 3:56 AM, Stephan Hoyer <shoyer@gmail.com> wrote:
We already have a built-in immutable set for Python. It's called frozenset.
This is true, but AIUI its API is based primarily on that of the (mutable) set. If you were creating a greenfield ImmutableSet class, what would its API look like? ChrisA
On Sat, Dec 30, 2017 at 02:56:46AM +1100, Chris Angelico wrote:
We seem to be talking at cross-purposes. Obviously we can and do already use ints as if they were set-like data structures. For example, the re module already does so. If you want to call that a kind of "bit set", I'm okay with that, but Wikipedia suggests that "bit array" is the canonical name: "bit array (also known as bit map , bit set, bit string, or bit vector)" https://en.wikipedia.org/wiki/Bit_array The obvious reason why is that sets are unordered but arrays of bits are not: 0b1000 is not the same "set" as 0b0010. However, the point I was making was that ints don't provide the same interface as sets. I don't deny that you can use an int to provide set-like functionality, or that with sufficient effort you could subclass int to do so, but what you cannot do is seemlessly interchange an int for a set and visa versa and expect the code to work without modification. Even if the function is limited to using the set-like functionality. I think I have beaten this dead horse enough. This was a minor point about the terminology being used, so I think we're now just waiting on Paddy to clarify what his proposal means in concrete terms. -- Steve
On Fri, Dec 29, 2017 at 11:25 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I think "bit-set" was used because it has semantic meaning in this context. In your example, it is not the bits that are ordered, but the values, which have a canonical order (or, more generally, a specified order). 0b1000 represents the set {3}, while 0b0010 represents the set {1}. A bit set representation is, in fact, unordered, since {1,3} and {3,1} are both represented by the same int. The values of a bit array are the bits themselves, but the values of a bitset are the indices which have a 1.
Paddy might want something like this: - For existing APIs which take int or IntFlag flags, allow them to also take a set (or perhaps any collection) of flags. - In new APIs, take sets of Enum flags, and don't make them IntFlag. - Documentation should show preference toward using sets of Enum flags. Tutorials should pass sets.
On Sat, Dec 30, 2017 at 8:50 PM, Franklin? Lee < leewangzhong+python@gmail.com> wrote:
I'm not keen on this recommendation. An argument that takes a Set[Foo] would mean that in order to specify: - no flags: you'd have to pass set() -- you can't use {} since that's an empty dict, not an empty set - one flag: you'd have to pass {Foo.BAR} rather than just Foo.BAR - two flags: you'd have to pass {Foo.BAR, Foo.BAZ} rather than Foo.BAR | Foo.BAZ I think for each of these the proposal would be strictly worse than the current convention. -- --Guido van Rossum (python.org/~guido)
I should probably clarify that for this to work, Foo must derive from enum.Flags. See https://docs.python.org/3/library/enum.html#flag. (Or enum.IntFlag, https://docs.python.org/3/library/enum.html#intflag.) Note that when using Flag, you can name the "zero" value (Color.BLACK in the 3rd example). When using IntFlag, you just use 0. On Sat, Dec 30, 2017 at 9:33 PM, Guido van Rossum <guido@python.org> wrote:
-- --Guido van Rossum (python.org/~guido)
Hmm, yea I had not thought of how it would look - I had thought formost of not needing to necessarily learn about bitsets.when learning about passing a large number of optional flags to a function. Although the default could be None, interpreted as an empty set of zero values.; a set of one or more enums does use more characters compared to or-ing flags... On Sunday, 31 December 2017 05:34:23 UTC, Guido van Rossum wrote:
I'm guessing that what this thread is about is coming up with an API rule that makes providing a set of boolean options available to a function or class in the least error prone way. Its the error prone nature of integer bit masks that is behind the enum suggestion I assume. From the C tradition we have the integer bit mask which is error prone as there is no type checking that the masks belong to the option flags. Some APIs use calls with lots of keyword args that you set true or false and even none to mean default. The suggestion for a set of enums from this thread. You would need a class to represent a set of a particular enum to get type safety. List of strings or enums. You could even use a class that represents the options and set up an instance and pass it in. Hard to get wrong. I can see that all these styles have their place and each designer will pick the style they think fits the API they are designing. Barry
On Tue, Dec 26, 2017 at 12:07:52PM -0800, Paddy3118 wrote:
That implies that we are promoting bit sets (not merely using them). Where are we doing that?
Thi comes about because someone wrote a description, (since deleted),
"Someone"? One of the Python core developers? If not, well, the Python devs cannot be held responsible for what random people on the internet write.
It certainly is not an implementation detail -- it is a part of the public, published interface to the re module. Of course had history been different, re.compile *could* have taken a set of Enums instead. But that doesn't make it an implementation detail. That would be like saying that: re.search(pattern, text, flags=0) is an implementation detail, and we should feel free to change it to re.examine(text, pattern, set_of_enums=frozenset()) We can't change a public interface documented as taking an integer to one taking a set of Enums without going through a deprecation period. However, we could *add* an additional interface, where the re functions that currently accept an integer flag *also* accepts a set of Enums.
Should we not *first *teach the passing of a set of enum.Enum constant values in one argument as the *pythonic *way;
What makes you say that is the Pythonic way? (That's not a rhetorical question.) -- Steve
Hi Steve, I did not write an attack on the "Python devs". Re-read my original with a little less hostility and there should be room for an interpretation, (which I meant), that does not warrant such a hostile reply. The original is written in the hope of furthering discussion on the need for what is deemed pythonic , and on what Python is taught , as the language itself changes. We now have enums if you want to pass a set of flags to a function then you could have them as separate arguments - but that leads to long and cumbersome parameter lists; you could, and many do, have flags that are individual powers of two and then or them together and pass the result - creating a bitset; but we can now have the flags as separate enum.Enums and pass a set of values to a function as the flag set. This new way means that people being taught the method can use a knowledge of sets and enums - no need to know about powers of two,, what happens when they are bit-or'd together; and bitsets. We have gone to a higher level description of what we are doing; no need to mired in the details of the how of what one wants to achieve. bitsets can be taught as an optimisation. As for re, and otheralready written libraries, their is no need to change them, but other Pythoneers might well opt to not use bitsets, but rather sets of enum values. On Wednesday, 27 December 2017 05:58:00 UTC, Steven D'Aprano wrote:
On Thu, Dec 28, 2017 at 12:23:53PM -0800, Paddy3118 wrote:
Hi Steve, I did not write an attack on the "Python devs".
I didn't say you attacked anyone and your implication that I did is unfair.
Please re-read my response without the hair-trigger defensiveness. Disagreement is not hostility. Questioning your statements is not hostility. None of us are entitled to only positive responses that agree with what we suggest, but we are all entitled to the presumption that we are arguing in good faith. See also: https://mail.python.org/pipermail/python-ideas/2017-December/048448.html and: https://mail.python.org/pipermail/python-ideas/2017-December/048449.html (I don't agree with *everything* Brett says, but its a good starting point.)
Right -- and I responded to that discussion. I asked some genuine questions which you haven't responded to. Let me rephrase them: - Are we currently promoting ints as bitsets? - How is it relevant that "someone" (who?) wrote a description of using ints as bitsets and then deleted it? - Why is a set of Enums the Pythonic way?
And no one is stopping anyone from writing code that does so.
Personally, I think that sets and Enums are no easier to learn than bit twiddling. But YMMV.
We have gone to a higher level description of what we are doing; no need to mired in the details of the how of what one wants to achieve.
Whether we use an int or a set of Enums, we still need to understand the details of how to set and clear flags, and test for them. THE_FLAG = 8 the_flags = THE_FLAG | ANOTHER_FLAG if the_flags & THE_FLAG: print("flag is set") versus: class MyFlags(Enum): THE_FLAG = "whatever" the flags = {THE_FLAG, ANOTHER_FLAG} if MyFlags.THE_FLAG in the_flags: print("flag is set") Honestly, I'm not seeing a *lot* of difference here. [...]
As for re, and otheralready written libraries, their is no need to change them,
I see. It wasn't clear to me. It seemed to me that you were suggesting changing the re module, since that was just an "implementation" detail.
but other Pythoneers might well opt to not use bitsets, but rather sets of enum values.
Since ints don't provide a set-like interface, they aren't strictly speaking bitsets. But in any case, nobody is stopping people from using sets of enum values. If you have a concrete proposal, then please state it explicitly. I thought I understood your proposal: change the implementation of re to use sets of Enums in order to promote their use and act as an example of best practice and Pythonic style but apparently I misunderstood. Sorry. So now I'm left puzzled as to what you actually want. Can you be explicit and explain what you expect us to do when you say we should "teach" and "promote" (your words) Enums instead of using ints? Please be concrete. - Should we change the standard library, and if so, in what ways? - Should we change the documentation? Which parts? - Insert something in PEP 8 about using sets of Enums? - Re-write the tutorial? Where? - Something else? Without a concrete proposal, I don't think this discussion is going anywhere. -- Steve
On Fri, Dec 29, 2017 at 7:18 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I'm not sure what "set-like interface" you'd be looking for, but the built-in set type has a lot of the same operations as an integer does, and the semantics are virtually identical to a set of bits. The only one you really lack is __contains__, which could easily be added: class BitSet(int): def __contains__(self, bit): return (self & bit) == bit
Set union, intersection, etc are all provided using the same operators in sets and ints. ChrisA
On Fri, Dec 29, 2017 at 10:26:22PM +1100, Chris Angelico wrote:
The lack of support for the `in` operator is a major difference, but there's also `len` (equivalent to "count the one bits"), superset and subset testing, various in-place mutator methods, etc. Java has a BitSet class, and you can see the typical sorts of operations commonly required: https://docs.oracle.com/javase/8/docs/api/java/util/BitSet.html Of course we can emulate set-like operations using ints, but the interfaces are different, which is my point. Here's how to clear all the flags of a set or int: the_flags.clear() the_flags = 0 # clear all the bits in an int Setting a flag is *almost* the same between the two: the_flags |= {flag} # set the_flags |= flag # int although for sets, there are two other ways to set a flag which aren't supported by ints: the_flags.add(flag) the_flags.update({flag}) Similarly for clearing flags: the_flags.discard(flag) the_flags & ~flag -- Steve
On Sat, Dec 30, 2017 at 2:38 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Okay. A subclass of int could easily add a few more. Counting the 1 bits isn't difficult; superset and subset testing are actually the same as 'contains' but with more than one bit at a time. (In fact, checking if a set contains a subset is *easier* with ints than with actual sets!) Are in-place mutators that big a deal? I'm sure there are sets in languages with no mutables.
That's a consequence of Python's mutability distinction. I don't think it's a fundamental difference. You could just as easily use "the_flags = set()" if it weren't for aliasing.
That's because you can implicitly upcast a bitflag to a bitset. Effectively, ints give you a short-hand that sets can't. But if you explicitly call BitSet(flag) to create a set containing one flag, it would have the same effect.
Mutability again. If you were to create an ImmutableSet type in Python, what would its API look like? My suspicion is that it'd largely use operators, and that it'd end up looking a lot like the integer API. An integer, at its lowest level, is represented as a set of bits. It's no more crazy to use an int as a set of bits than to use a string as a set of characters: https://docs.python.org/3/library/stdtypes.html#str.strip ChrisA
On Sat, Dec 30, 2017 at 3:56 AM, Stephan Hoyer <shoyer@gmail.com> wrote:
We already have a built-in immutable set for Python. It's called frozenset.
This is true, but AIUI its API is based primarily on that of the (mutable) set. If you were creating a greenfield ImmutableSet class, what would its API look like? ChrisA
On Sat, Dec 30, 2017 at 02:56:46AM +1100, Chris Angelico wrote:
We seem to be talking at cross-purposes. Obviously we can and do already use ints as if they were set-like data structures. For example, the re module already does so. If you want to call that a kind of "bit set", I'm okay with that, but Wikipedia suggests that "bit array" is the canonical name: "bit array (also known as bit map , bit set, bit string, or bit vector)" https://en.wikipedia.org/wiki/Bit_array The obvious reason why is that sets are unordered but arrays of bits are not: 0b1000 is not the same "set" as 0b0010. However, the point I was making was that ints don't provide the same interface as sets. I don't deny that you can use an int to provide set-like functionality, or that with sufficient effort you could subclass int to do so, but what you cannot do is seemlessly interchange an int for a set and visa versa and expect the code to work without modification. Even if the function is limited to using the set-like functionality. I think I have beaten this dead horse enough. This was a minor point about the terminology being used, so I think we're now just waiting on Paddy to clarify what his proposal means in concrete terms. -- Steve
On Fri, Dec 29, 2017 at 11:25 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I think "bit-set" was used because it has semantic meaning in this context. In your example, it is not the bits that are ordered, but the values, which have a canonical order (or, more generally, a specified order). 0b1000 represents the set {3}, while 0b0010 represents the set {1}. A bit set representation is, in fact, unordered, since {1,3} and {3,1} are both represented by the same int. The values of a bit array are the bits themselves, but the values of a bitset are the indices which have a 1.
Paddy might want something like this: - For existing APIs which take int or IntFlag flags, allow them to also take a set (or perhaps any collection) of flags. - In new APIs, take sets of Enum flags, and don't make them IntFlag. - Documentation should show preference toward using sets of Enum flags. Tutorials should pass sets.
On Sat, Dec 30, 2017 at 8:50 PM, Franklin? Lee < leewangzhong+python@gmail.com> wrote:
I'm not keen on this recommendation. An argument that takes a Set[Foo] would mean that in order to specify: - no flags: you'd have to pass set() -- you can't use {} since that's an empty dict, not an empty set - one flag: you'd have to pass {Foo.BAR} rather than just Foo.BAR - two flags: you'd have to pass {Foo.BAR, Foo.BAZ} rather than Foo.BAR | Foo.BAZ I think for each of these the proposal would be strictly worse than the current convention. -- --Guido van Rossum (python.org/~guido)
I should probably clarify that for this to work, Foo must derive from enum.Flags. See https://docs.python.org/3/library/enum.html#flag. (Or enum.IntFlag, https://docs.python.org/3/library/enum.html#intflag.) Note that when using Flag, you can name the "zero" value (Color.BLACK in the 3rd example). When using IntFlag, you just use 0. On Sat, Dec 30, 2017 at 9:33 PM, Guido van Rossum <guido@python.org> wrote:
-- --Guido van Rossum (python.org/~guido)
Hmm, yea I had not thought of how it would look - I had thought formost of not needing to necessarily learn about bitsets.when learning about passing a large number of optional flags to a function. Although the default could be None, interpreted as an empty set of zero values.; a set of one or more enums does use more characters compared to or-ing flags... On Sunday, 31 December 2017 05:34:23 UTC, Guido van Rossum wrote:
I'm guessing that what this thread is about is coming up with an API rule that makes providing a set of boolean options available to a function or class in the least error prone way. Its the error prone nature of integer bit masks that is behind the enum suggestion I assume. From the C tradition we have the integer bit mask which is error prone as there is no type checking that the masks belong to the option flags. Some APIs use calls with lots of keyword args that you set true or false and even none to mean default. The suggestion for a set of enums from this thread. You would need a class to represent a set of a particular enum to get type safety. List of strings or enums. You could even use a class that represents the options and set up an instance and pass it in. Hard to get wrong. I can see that all these styles have their place and each designer will pick the style they think fits the API they are designing. Barry
participants (10)
-
Barry Scott -
Chris Angelico -
Franklin? Lee -
Guido van Rossum -
MRAB -
Neil Girdhar -
Paddy3118 -
Random832 -
Stephan Hoyer -
Steven D'Aprano