Mailman 3 divmod(): fallback to __floordiv__ and __mod__? - Python-ideas - python.org

newer
___repr__ on collections.UserDict,...

divmod(): fallback to floordiv and mod?

older
(Windows-only - calling Steve...

Spencer Brown

17 Sep 2016 17 Sep '16

9:01 a.m.

Currently, calling divmod() on a class with __floordiv__ and __mod__ defined, but not __divmod__ raises a TypeError. Is there any reason why it doesn't fallback to (self // x, self % x)? The invariants described in the function documentation should still apply (with well behaved % and //). If divmod() doesn't make sense setting it to None should be permitted. - Spencer

Attachments:

attachment.htm (text/html — 606 bytes)

Reply

Sign in to reply online Use email software

Show replies by date

Mark Dickinson

17 Sep 17 Sep

10:56 a.m.

On Sat, Sep 17, 2016 at 10:01 AM, Spencer Brown wrote:

Currently, calling divmod() on a class with __floordiv__ and __mod__ defined, but not __divmod__ raises a TypeError. Is there any reason why it doesn't fallback to (self // x, self % x)?

It's an interesting idea. I wonder whether the falling back shouldn't be in the other direction, though: that is, if a class defines `__divmod__` but not `__floordiv__` or `__mod__`, perhaps the `//` and `%` operations should use `__divmod__`? That way, if you're writing a class that intends to support all three operations, you only have to write one method. And it might make sense from an efficiency perspective, too; it's common for a `divmod` computation to be cheaper than doing separate computations of the quotient and remainder. For the builtin int type, for example, in nontrivial cases Python computes both the quotient and remainder when asked to do a % or // operation, and then discards whichever part isn't needed. So in that case it would be wasteful to build up the divmod result from two separate % and // calls. -- Mark

Reply

Sign in to reply online Use email software

David Mertz

5:34 p.m.

It seems like this could be something similar to `functools.total_ordering` and decorate a class. In principle that transformation could go in either direction, but only if the decorator is used. On Sat, Sep 17, 2016 at 3:56 AM, Mark Dickinson wrote:

On Sat, Sep 17, 2016 at 10:01 AM, Spencer Brown wrote:

...
Currently, calling divmod() on a class with __floordiv__ and __mod__ defined, but not __divmod__ raises a TypeError. Is there any reason why it doesn't fallback to (self // x, self % x)?

It's an interesting idea. I wonder whether the falling back shouldn't be in the other direction, though: that is, if a class defines `__divmod__` but not `__floordiv__` or `__mod__`, perhaps the `//` and `%` operations should use `__divmod__`? That way, if you're writing a class that intends to support all three operations, you only have to write one method. And it might make sense from an efficiency perspective, too; it's common for a `divmod` computation to be cheaper than doing separate computations of the quotient and remainder.

For the builtin int type, for example, in nontrivial cases Python computes both the quotient and remainder when asked to do a % or // operation, and then discards whichever part isn't needed. So in that case it would be wasteful to build up the divmod result from two separate % and // calls.

-- Mark _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

Reply

Sign in to reply online Use email software

Ethan Furman

5:42 p.m.

On 09/17/2016 10:34 AM, David Mertz wrote:

On Sat, Sep 17, 2016 at 3:56 AM, Mark Dickinson wrote:

...
On Sat, Sep 17, 2016 at 10:01 AM, Spencer Brown wrote:

...
...
Currently, calling divmod() on a class with __floordiv__ and __mod__ defined, but not __divmod__ raises a TypeError. Is there any reason why it doesn't fallback to (self // x, self % x)?

It's an interesting idea. I wonder whether the falling back shouldn't be in the other direction, though: that is, if a class defines `__divmod__` but not `__floordiv__` or `__mod__`, perhaps the `//` and `%` operations should use `__divmod__`? That way, if you're writing a

It seems like this could be something similar to `functools.total_ordering` and decorate a class. In principle that transformation could go in either direction, but only if the decorator is used.

Not at all. Currently Python will fallback to `not ==` if a class does not define `!=`. Having `//` and `%` fall back to `__divmod__` would not be out of place. -- ~Ethan~

Reply

Sign in to reply online Use email software

David Mertz

5:57 p.m.

The fallback you described would be a change in the behavior of some working programs. Moreover, it would only affect custom classes where adding a decorator is an option (even in external code, you can use `MyThing = total_divmod(library.MyThing)` under this option. Showing that a recipe for a decorator is of wide use feels like a necessary step towards any future semantic change in the language to me. For example, '%' is fairly widely (ab)used for meanings other than modulo. E.g. string formatting. Probably not that many classes that respond to '%' to do something non-modulo simultaneously implement `.__divmod__()` ... but maybe some use case is not obvious to me. If those exist, your change would break that (well, depending whether methods of ancestors are used or not). On Sat, Sep 17, 2016 at 10:42 AM, Ethan Furman wrote:

On 09/17/2016 10:34 AM, David Mertz wrote:

...
On Sat, Sep 17, 2016 at 3:56 AM, Mark Dickinson wrote:

...
On Sat, Sep 17, 2016 at 10:01 AM, Spencer Brown wrote:

Currently, calling divmod() on a class with __floordiv__ and __mod__

...
...
...
defined, but not __divmod__ raises a TypeError. Is there any reason why it doesn't fallback to (self // x, self % x)?

It's an interesting idea. I wonder whether the falling back shouldn't be in the other direction, though: that is, if a class defines `__divmod__` but not `__floordiv__` or `__mod__`, perhaps the `//` and `%` operations should use `__divmod__`? That way, if you're writing a

It seems like this could be something similar to `functools.total_ordering` and decorate a class. In principle that transformation could go in either direction, but only if the decorator is used.

Not at all. Currently Python will fallback to `not ==` if a class does not define `!=`. Having `//` and `%` fall back to `__divmod__` would not be out of place.

-- ~Ethan~

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

Reply

Sign in to reply online Use email software

Chris Angelico

6:05 p.m.

On Sun, Sep 18, 2016 at 3:57 AM, David Mertz wrote:

For example, '%' is fairly widely (ab)used for meanings other than modulo. E.g. string formatting. Probably not that many classes that respond to '%' to do something non-modulo simultaneously implement `.__divmod__()` ... but maybe some use case is not obvious to me. If those exist, your change would break that (well, depending whether methods of ancestors are used or not).

So there'll be classes that define __mod__ but not __divmod__ - that's fine. Are there any that go the other way around? It might technically be a breaking change, but it's unlikely to cause very much breakage. So it'd be inappropriate for 3.6.1, but okay for 3.7. +1 on the fallback exactly as Ethan described. The One Obvious Way to implement arithmetic division would be to define __divmod__. Defining __truediv__ or __mod__ would be for times when you can implement it more efficiently by not calculating the other half, and of course the times when they're not implementing division (cf pathlib.Path and str, for / and % respectively). ChrisA

Reply

Sign in to reply online Use email software

Ethan Furman

10:06 p.m.

On 09/17/2016 10:57 AM, David Mertz wrote:

The fallback you described would be a change in the behavior of some working programs. Moreover, it would only affect custom classes where adding a decorator is an option (even in external code, you can use `MyThing = total_divmod(library.MyThing)` under this option.

I'm really not sure what you're saying here, but it sounds like you're concerned about a __divmod__ overriding existing __mod__ and __floordiv__? That is not the case. Just like Python will use the defined __ne__ if it's present, or fall back to negating the result of __eq__ if __ne__ is not present, I see __divmod__ working the same way: - is __mod__ present? use it - is __floordiv__ present? use it - otherwise, use __divmod__ and return the needed piece I'm pretty sure __div__ should not fall back to __divmod__. -- ~Ethan~

Reply

Sign in to reply online Use email software

Chris Angelico

10:14 p.m.

On Sun, Sep 18, 2016 at 8:06 AM, Ethan Furman wrote:

Just like Python will use the defined __ne__ if it's present, or fall back to negating the result of __eq__ if __ne__ is not present, I see __divmod__ working the same way:

- is __mod__ present? use it - is __floordiv__ present? use it - otherwise, use __divmod__ and return the needed piece

I'm pretty sure __div__ should not fall back to __divmod__.

How does __mod__ fall back to __floordiv__? I'm lost. ChrisA

Reply

Sign in to reply online Use email software

Ethan Furman

11:35 p.m.

On 09/17/2016 03:14 PM, Chris Angelico wrote:

On Sun, Sep 18, 2016 at 8:06 AM, Ethan Furman wrote:

...
Just like Python will use the defined __ne__ if it's present, or fall back to negating the result of __eq__ if __ne__ is not present, I see __divmod__ working the same way:

- is __mod__ present? use it - is __floordiv__ present? use it - otherwise, use __divmod__ and return the needed piece

I'm pretty sure __div__ should not fall back to __divmod__.

How does __mod__ fall back to __floordiv__? I'm lost.

Oops, sorry. Got my directions reversed when thinking about how __div__ should fit in. Bird's eye view: if the exact method needed is present, use it; otherwise if a fallback method is available, use that. Currently this is done for __ne__ --> not __eq__, and I seem to remember another case or two that was talked about but I don't remember what they were and I'm not sure if they got implemented to follow the fallback pattern. -- ~Ethan~

Reply

Sign in to reply online Use email software

Steven D'Aprano

19 Sep 19 Sep

2:04 a.m.

On Sat, Sep 17, 2016 at 09:01:53AM +0000, Spencer Brown wrote:

Currently, calling divmod() on a class with __floordiv__ and __mod__ defined, but not __divmod__ raises a TypeError. Is there any reason why it doesn't fallback to (self // x, self % x)?

Because things get really complex, fast. Does the added complexity pay its own way? I may have some of the details wrong, but I understand the implementation of divmod() is something like this: # pseudo-code for divmod def divmod(a, b): A, B = type(a), type(b) if issubclass(B, A): # Give priority to the reverse dunder version if hasattr(B, '__rdivmod__'): result = B.__rdivmod__(b, a) if result is not NotImplemented: return result if hasattr(A, '__divmod__'): result = A.__divmod__(a, b) if result is not NotImplemented: return result raise TypeError if hasattr(A, '__divmod__'): result = A.__divmod__(a, b) if result is not NotImplemented: return result if hasattr(B, '__rdivmod__'): result = B.__rdivmod__(b, a) if result is not NotImplemented: return result raise TypeError only it will be in C, so probably three or five times as much code :-) Now consider adding a fallback to __floordiv__ and __mod__ as suggested. That adds a lot more complexity: - what if the object has __floordiv__, but not __mod__? Do you try the reversed method, __rmod__, instead? - likewise for NotImplemented? - what if they're None? So let's look at the complexity of the fallback: # reverse dunder version, for when b is a subclass of a rdiv = getattr(B, '__rfloordiv__, None) rmod = getattr(B, '__rmod__, None) if rdiv is not None and rmod is not None: x, y = rdiv(b, a), rmod(b, a) if x is NotImplemented: div = getattr(A, '__floordiv__', None) if div is not None: x = div(a, b) if x is NotImplemented: raise TypeError if y is NotImplemented: mod = getattr(A, '__mod__', None) if mod is not None: y = mod(a, b) if y is NotImplemented: raise TypeError assert NotImplemented not in (x, y) else: # try the __floordiv__ and __mod__ versions, without falling # back to reversed versions ... return (x, y) And then more or less the same for the "standard" case where a has priority over b (i.e. isn't a subclass). So, my estimate is that this will roughly triple the complexity of the divmod() function. Perhaps worse. Oh, and I forgot the case where __divmod__ exists but is None, but I'm not going back to add it in because I'm not even sure where that ought to be tested. Now I daresay we can refactor my naive pseudo-code above, but however you look at it, there's going to be a big jump in complexity for not a lot of benefit. In the case of __ne__ falling back on not __eq__, there is at least an utterly clear and obvious user-expectation that the two methods are linked, and there are no reversed __req__ and __rne__ methods to care about. That's not the case here. Going the other way (fall back to __divmod__ if __floordiv__ or __mod__ is not defined) will probably be not quite as complex, but it will still add a fair amount of complexity. So, *emotionally* I actually do quite like the idea of having divmod fall back to // and %, *and* the other way, but *intellectually* when I think about the complexity and the tests that will be needed to cover all the cases, I have to say "Uh uh, no way!" It may be that the actual C implementation is simpler than I think, in which case maybe this is a reasonable idea, but on the face of it, I think that it will be really hard to get right, add a lot of code, and provide very little benefit. -- Steve

Reply

Sign in to reply online Use email software

2773

Age (days ago)

2775

Last active (days ago)

Download

9 comments

6 participants

tags

participants (6)

Chris Angelico
David Mertz
Ethan Furman
Mark Dickinson
Spencer Brown
Steven D'Aprano