Hi all, I curate the multipledispatch library on PyPI. I believe that it is a natural continuation of singledispatch outlined in PEP 443 and included in functools 3.4. For those unaware, dispatching correctly and unambiguously on multiple inputs is a somewhat more complex problem than on a single input. I believe the approach taken in multipledispatch is fairly robust. I wrote a blogpost about MD a while ago: http://matthewrocklin.com/blog/work/2014/02/25/Multiple-Dispatch/ The docs pages live here: http://multiple-dispatch.readthedocs.org/en/latest/ And the github page lives here: https://github.com/mrocklin/multipledispatch Recommendations for improvement welcome. Thoughts on whether or not this is appropriate to include in the standard library also welcome. Best, -Matthew Rocklin
Have you written anything significant that you're distributing with it? Has anyone else? Are there good uses of singledispatch in the wild even? On Fri, Aug 15, 2014 at 8:33 AM, Matthew Rocklin <mrocklin@gmail.com> wrote:
Hi all,
I curate the multipledispatch library on PyPI. I believe that it is a natural continuation of singledispatch outlined in PEP 443 and included in functools 3.4.
For those unaware, dispatching correctly and unambiguously on multiple inputs is a somewhat more complex problem than on a single input. I believe the approach taken in multipledispatch is fairly robust.
I wrote a blogpost about MD a while ago: http://matthewrocklin.com/blog/work/2014/02/25/Multiple-Dispatch/
The docs pages live here: http://multiple-dispatch.readthedocs.org/en/latest/
And the github page lives here: https://github.com/mrocklin/multipledispatch
Recommendations for improvement welcome. Thoughts on whether or not this is appropriate to include in the standard library also welcome.
Best, -Matthew Rocklin
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido)
On 08/15/2014 10:28 AM, Guido van Rossum wrote:
Are there good uses of singledispatch in the wild even?
Well, I don't know if it's a "good use", but I have used singledispatch to override 'float' in certain modules so that I can say, for example, float(DateTime.Time) or float("time string"). It's really quite handy. -- ~Ethan~
On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido@python.org> wrote:
Are there good uses of singledispatch in the wild even?
In one of my projects, I have a todo item to convert a homegrown {type:func} dictionary based dispatch to singledispatch. It has been open for 5 months already. I wouldn't be surprised if others had something similar: a good-enough (and probably buggy) solution that is not bad enough to justify adding extra dependency in 2.x or replacing with a 3.x only solution. Multiple dispatch is much harder to get right or even "good enough," so I don't think singledispatch popularity or lack thereof is a good predictor for multipledispatch.
I've definitely seen homegrown solutions like Alexander talks about in a variety of projects. It has popped up a few times in SymPy. My implementation was influenced somewhat by Julia's solution which seems fairly sober. Multiple dispatch has demonstrated value in that community. I used the multipledispatch library in a few of my projects. I can provide examples of where it's been helpful if desired. On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido@python.org> wrote:
Are there good uses of singledispatch in the wild even?
In one of my projects, I have a todo item to convert a homegrown {type:func} dictionary based dispatch to singledispatch. It has been open for 5 months already. I wouldn't be surprised if others had something similar: a good-enough (and probably buggy) solution that is not bad enough to justify adding extra dependency in 2.x or replacing with a 3.x only solution.
Multiple dispatch is much harder to get right or even "good enough," so I don't think singledispatch popularity or lack thereof is a good predictor for multipledispatch.
Please do write about non-toy examples! On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin <mrocklin@gmail.com> wrote:
I've definitely seen homegrown solutions like Alexander talks about in a variety of projects. It has popped up a few times in SymPy.
My implementation was influenced somewhat by Julia's solution which seems fairly sober. Multiple dispatch has demonstrated value in that community.
I used the multipledispatch library in a few of my projects. I can provide examples of where it's been helpful if desired.
On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido@python.org> wrote:
Are there good uses of singledispatch in the wild even?
In one of my projects, I have a todo item to convert a homegrown {type:func} dictionary based dispatch to singledispatch. It has been open for 5 months already. I wouldn't be surprised if others had something similar: a good-enough (and probably buggy) solution that is not bad enough to justify adding extra dependency in 2.x or replacing with a 3.x only solution.
Multiple dispatch is much harder to get right or even "good enough," so I don't think singledispatch popularity or lack thereof is a good predictor for multipledispatch.
-- --Guido van Rossum (python.org/~guido)
It has over 25k downloads from PyPI per month. It’s used in OpenStack’s Nova. It’s used at Facebook as part of a deadline based, resource aware, dependency based, fault tolerant scheduler. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev On Aug 15, 2014, at 11:01 AM, Guido van Rossum <guido@python.org> wrote:
Please do write about non-toy examples!
On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin <mrocklin@gmail.com> wrote: I've definitely seen homegrown solutions like Alexander talks about in a variety of projects. It has popped up a few times in SymPy.
My implementation was influenced somewhat by Julia's solution which seems fairly sober. Multiple dispatch has demonstrated value in that community.
I used the multipledispatch library in a few of my projects. I can provide examples of where it's been helpful if desired.
On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido@python.org> wrote: Are there good uses of singledispatch in the wild even?
In one of my projects, I have a todo item to convert a homegrown {type:func} dictionary based dispatch to singledispatch. It has been open for 5 months already. I wouldn't be surprised if others had something similar: a good-enough (and probably buggy) solution that is not bad enough to justify adding extra dependency in 2.x or replacing with a 3.x only solution.
Multiple dispatch is much harder to get right or even "good enough," so I don't think singledispatch popularity or lack thereof is a good predictor for multipledispatch.
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
I assume that Lukasz is referrring to singledispatch? multipledispatch doesn't yet have that level of adoption as far as I'm aware. On Fri, Aug 15, 2014 at 11:25 AM, Łukasz Langa <lukasz@langa.pl> wrote:
It has over 25k downloads from PyPI per month. It’s used in OpenStack’s Nova. It’s used at Facebook as part of a deadline based, resource aware, dependency based, fault tolerant scheduler.
-- Best regards, Łukasz Langa
WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
On Aug 15, 2014, at 11:01 AM, Guido van Rossum <guido@python.org> wrote:
Please do write about non-toy examples!
On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin <mrocklin@gmail.com> wrote:
I've definitely seen homegrown solutions like Alexander talks about in a variety of projects. It has popped up a few times in SymPy.
My implementation was influenced somewhat by Julia's solution which seems fairly sober. Multiple dispatch has demonstrated value in that community.
I used the multipledispatch library in a few of my projects. I can provide examples of where it's been helpful if desired.
On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:
On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido@python.org> wrote:
Are there good uses of singledispatch in the wild even?
In one of my projects, I have a todo item to convert a homegrown {type:func} dictionary based dispatch to singledispatch. It has been open for 5 months already. I wouldn't be surprised if others had something similar: a good-enough (and probably buggy) solution that is not bad enough to justify adding extra dependency in 2.x or replacing with a 3.x only solution.
Multiple dispatch is much harder to get right or even "good enough," so I don't think singledispatch popularity or lack thereof is a good predictor for multipledispatch.
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Ah, that’s right, I meant singledispatch. Sorry for the confusion! :) -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev On Aug 15, 2014, at 11:33 AM, Matthew Rocklin <mrocklin@gmail.com> wrote:
I assume that Lukasz is referrring to singledispatch? multipledispatch doesn't yet have that level of adoption as far as I'm aware.
On Fri, Aug 15, 2014 at 11:25 AM, Łukasz Langa <lukasz@langa.pl> wrote: It has over 25k downloads from PyPI per month. It’s used in OpenStack’s Nova. It’s used at Facebook as part of a deadline based, resource aware, dependency based, fault tolerant scheduler.
-- Best regards, Łukasz Langa
WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
On Aug 15, 2014, at 11:01 AM, Guido van Rossum <guido@python.org> wrote:
Please do write about non-toy examples!
On Fri, Aug 15, 2014 at 10:55 AM, Matthew Rocklin <mrocklin@gmail.com> wrote: I've definitely seen homegrown solutions like Alexander talks about in a variety of projects. It has popped up a few times in SymPy.
My implementation was influenced somewhat by Julia's solution which seems fairly sober. Multiple dispatch has demonstrated value in that community.
I used the multipledispatch library in a few of my projects. I can provide examples of where it's been helpful if desired.
On Fri, Aug 15, 2014 at 10:47 AM, Alexander Belopolsky <alexander.belopolsky@gmail.com> wrote:
On Fri, Aug 15, 2014 at 1:28 PM, Guido van Rossum <guido@python.org> wrote: Are there good uses of singledispatch in the wild even?
In one of my projects, I have a todo item to convert a homegrown {type:func} dictionary based dispatch to singledispatch. It has been open for 5 months already. I wouldn't be surprised if others had something similar: a good-enough (and probably buggy) solution that is not bad enough to justify adding extra dependency in 2.x or replacing with a 3.x only solution.
Multiple dispatch is much harder to get right or even "good enough," so I don't think singledispatch popularity or lack thereof is a good predictor for multipledispatch.
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On 16 August 2014 04:38, Łukasz Langa <lukasz@langa.pl> wrote:
Ah, that’s right, I meant singledispatch. Sorry for the confusion! :)
I think "singledispatch" is still potentially relevant to Matthew's idea of a library specifically devoted to dispatch and signature matching. My rationale is that "functools.singledispatch" actually has some pretty neat funcationality behind it. I'm talking things like functools._c3_merge, functools._compose_mro, functools._find_impl - all the machinery that is needed to take the complexity of ABC registration and turn it into something that can be checked quickly at runtime. Even the trick with using abc.get_cache_token() to invalidate the dispatch caches whenever the object graph changes via explicit ABC registration is worth illuminating further. Decoupling the underlying dispatch machinery from the specific functools use case also opens up additional possibilities for type based dispatch. The way Julia uses multiple dispatch for binary operators is interesting (http://julia.readthedocs.org/en/latest/manual/methods/), although not directly applicable to Python's binary operators, since we use the "return NotImplemented" dance to decide which implementation to use. That's where I can see a possible fit for something like multiple dispatch support in the standard library: making it easier to write binary operator overloads correctly. A *lot* of folks make the mistake of raising TypeError or NotImplementedError directly in their operator overload implementations, rather than returning the NotImplemented singleton that tells the interpreter to try the other type. Even some of the CPython builtins get that wrong, since the sq_concat and sq_repeat slots in C don't properly support the type coercion dance, so you *have* to raise the exception yourself if you're only implementing those without implementing nb_add and nb_mul (types defined in Python automatically populate both sets of C level slots if you define __add__ or __mul__). Dealing with the "NotImplemented dance" properly is also why functools.total_ordering got substantially slower in Python 3.4 - it isn't at risk of blowing up with RecursionError in some cases any more, but it paid a hefty price in speed to get there. Notation wise, I strongly encourage going with the format defined in PEP 443: a "default implementation" that defines the namespace all the other implementations will hook into, along with explicit registration of additional overloads. If there's no sensible default implementation, it can be written to raise an appropriate exception (a dispatch library could even help with raising an appropriately formatted type error). For example, here's how a PEP 443 inspired notation would handle the task of defining a stricter version of sequence repetition than the default "*" binary operator: @functools.multidispatch(2) # Multiple dispatch on the first 2 positional arguments def repeat(lhs, rhs): raise BinaryDispatchError('repeat', lhs, rhs) # See below for possible definition @example.register(numbers.Integral, collections.abc.Sequence): @example.register(collections.abc.Sequence, numbers.Integral): def repeat(lhs, rhs): return lhs * rhs # Assume this useful helper class is defined somewhere... class BinaryDispatchError(TypeError): def __init__(self, fname, lhs, rhs): self.fname = fname self.lhs_kind = lhs_kind = lhs.__class__.__name__ self.rhs_kind = rhs_kind = rhs.__class__.__name__ msg = "Unsupported operand type(s) for {!r}: {!r} and {!r}".format(fname, lhs_kind, rhs_kind) super().__init__(msg) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 15 Aug 2014, at 20:01, Guido van Rossum <guido@python.org> wrote:
Please do write about non-toy examples!
The entire Zope stack has been build around multiple-dispatch for the last 8 years or so. Of course that uses zope.interface/zope.component, which predate singledispatch by about 10 years as far as I can tell. Wichert.
Le 15/08/2014 14:01, Guido van Rossum a écrit :
Please do write about non-toy examples!
Are you looking for examples using the multipledispatch library, or multiple dispatch in general? As for multiple dispatch in general, Numba uses something which is morally one in order to select the right specialization of, say, an operator (for example to choose amongst '+ between int and int', '+ between numpy.datetime64 and numpy.timedelta64', '+ between numpy.timedelta64 and numpy.timedelta64', etc.). Regards Antoine.
Here is a non-trivial example of multiple dispatch. I want to convert data between container types, i.e. given into(a, b) I want to return something with the information content of b in a container like a, e.g. In [24]: into([], (1, 2, 3)) Out[24]: [1, 2, 3] We use this abstraction pretty heavily in Blaze, a project that tries to map relational algebra onto a variety of projects that might possibly be used to do relational-algebra-like tasks. Projects in this scope include sqlalchemy, pandas, numpy, pyspark, pytables, etc.. In [26]: from blaze import into A dataframe with some test data In [25]: df = DataFrame([[1, 'Alice', 100], [2, 'Bob', -200], [3, 'Charlie', 300], [4, 'Dennis', 400], [5, 'Edith', -500]], columns=['id', 'name', 'amount']) migrate list <- DataFrame In [27]: into([], df) Out[27]: [[1, 'Alice', 100], [2, 'Bob', -200], [3, 'Charlie', 300], [4, 'Dennis', 400], [5, 'Edith', -500]] migrate numpy array <- DataFrame In [28]: into(np.ndarray(0), df) Out[28]: rec.array([(1, 'Alice', 100), (2, 'Bob', -200), (3, 'Charlie', 300), (4, 'Dennis', 400), (5, 'Edith', -500)], dtype=[('id', '<i8'), ('name', 'O'), ('amount', '<i8')]) In [29]: x = into(np.ndarray(0), df) # store for later connect to local pymongo database In [30]: import pymongo In [31]: db = pymongo.MongoClient().db In [34]: into(db.my_collection, df) # migrate mongo <- pandas Out[34]: Collection(Database(MongoClient('localhost', 27017), u'db'), u'my_collection') In [35]: into(db.my_collection2, x) # migrate mongo <- numpy Out[35]: Collection(Database(MongoClient('localhost', 27017), u'db'), u'my_collection2') In [36]: list(db.my_collection2.find()) # verify that things transferred well Out[36]: [{u'_id': ObjectId('53ef6167fb5d1b34b9fd00e2'), u'amount': 100, u'id': 1, u'name': u'Alice'}, {u'_id': ObjectId('53ef6167fb5d1b34b9fd00e3'), u'amount': -200, u'id': 2, u'name': u'Bob'}, {u'_id': ObjectId('53ef6167fb5d1b34b9fd00e4'), u'amount': 300, u'id': 3, u'name': u'Charlie'}, {u'_id': ObjectId('53ef6167fb5d1b34b9fd00e5'), u'amount': 400, u'id': 4, u'name': u'Dennis'}, {u'_id': ObjectId('53ef6167fb5d1b34b9fd00e6'), u'amount': -500, u'id': 5, u'name': u'Edith'}] migrate bcolz <- mongo In [37]: into(bcolz.ctable(), db.my_collection) Out[37]: ctable((5,), [('amount', '<i8'), ('id', '<i8'), ('name', '<U7')]) nbytes: 220; cbytes: 63.99 KB; ratio: 0.00 cparams := cparams(clevel=5, shuffle=True, cname='blosclz') [(100, 1, u'Alice') (-200, 2, u'Bob') (300, 3, u'Charlie') (400, 4, u'Dennis') (-500, 5, u'Edith')] Note in this last case that the two libraries, bcolz (a compressed on-disk storage library) and pymongo know absolutely nothing about each other. Many of these into definitions are very simple @dispatch(np.ndarray, DataFrame) def into(a, df): return df.to_records(index=False) While some of them rely on others, or on inheritance @dispatch(Collection, np.ndarray) def into(coll, x, **kwargs): return into(coll, into(DataFrame(), x), **kwargs) But remembering all of the appropriate .to_foo and .from_bar methods can be a real pain. Collecting them all into a single abstraction cuts down significantly on the administrative burden of data migrations. On Sat, Aug 16, 2014 at 6:34 AM, Antoine Pitrou <antoine@python.org> wrote:
Le 15/08/2014 14:01, Guido van Rossum a écrit :
Please do write about non-toy examples!
Are you looking for examples using the multipledispatch library, or multiple dispatch in general?
As for multiple dispatch in general, Numba uses something which is morally one in order to select the right specialization of, say, an operator (for example to choose amongst '+ between int and int', '+ between numpy.datetime64 and numpy.timedelta64', '+ between numpy.timedelta64 and numpy.timedelta64', etc.).
Regards
Antoine.
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Antoine Pitrou schrieb am 16.08.2014 um 15:34:
Le 15/08/2014 14:01, Guido van Rossum a écrit :
Please do write about non-toy examples!
Are you looking for examples using the multipledispatch library, or multiple dispatch in general?
As for multiple dispatch in general, Numba uses something which is morally one in order to select the right specialization of, say, an operator (for example to choose amongst '+ between int and int', '+ between numpy.datetime64 and numpy.timedelta64', '+ between numpy.timedelta64 and numpy.timedelta64', etc.).
Similar for Cython's "fused types", i.e. compile time generics. It generates specialised function implementations for a cross product of the type sets used in the signature and then calls the right specialisation based on the input types at runtime, when called from Python code. Understands input buffer types and C types in signatures, though, so it's a bit more complex than the "average Python generic function dispatch" (same applies to Numba, I guess). Stefan
participants (9)
-
Alexander Belopolsky -
Antoine Pitrou -
Ethan Furman -
Guido van Rossum -
Matthew Rocklin -
Nick Coghlan -
Stefan Behnel -
Wichert Akkerman -
Łukasz Langa