Dictionary destructing and unpacking.
In python, we have beautiful unpacking: a, b, c = [1,2,3] and even a, b, *c = [1,2,3,4,5] We also have dictionary destructing for purposes of keywords: myfunc(**mydict) You can currently unpack a dictionary, but its almost certainly not what you would intend. a, b, c = {'a': 1, 'c': 3, 'b': 2}.values() In python 3.6+ this is better since the dictionary is insertion-ordered, but is still not really what one would probably want. It would be cool to have a syntax that would unpack the dictionary to values based on the names of the variables. Something perhaps like: a, b, c = **mydict which would assign the values of the keys 'a', 'b', 'c' to the variables. The problem with this approach is that it only works if the key is also a valid variable name. Another syntax could potentially be used to specify the keys you care about (and the order). Perhaps: a, b, c = **mydict('a', 'b', 'c') I dont really like that syntax, but it gives a good idea. One way to possibly achieve this today without adding syntax support could be simply adding a builtin method to the dict class: a, b, c = mydict.unpack('a', 'b', 'c') The real goal of this is to easily get multiple values from a dictionary. The current ways of doing this are: a, b, c, = mydict['a'], mydict['b'], mydict['c'] or a = mydict['a'] b = mydict['b'] c = mydict['c'] The later seams to be more common. Both are overly verbose in my mind. One thing to consider however is the getitem vs get behavior. mydict['a'] would raise a KeyError if 'a' wasnt in the dict, whereas mydict.get('a') would return a "default" (None if not specified). Which behavior is chosen? Maybe there is no clean solutions, but those are my thoughts. Anyone have feedback/ideas on this? Nick
On 07/06/17 19:14, Nick Humrich wrote:
a, b, c = mydict.unpack('a', 'b', 'c')
def retrieve(mapping, *keys): return (mapping[key] for key in keys) $ python3 Python 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information.
E.
On Wed, Jun 7, 2017 at 3:11 PM, Erik <python@lucidity.plus.com> wrote:
Or even: from operator import itemgetter retrieve = itemgetter('a', 'b', 'c') a, b, c = retrieve(dictionary)
-- Matt Gilson | Pattern Software Engineer getpattern.com <https://www.getpattern.com?utm_source=email&utm_medium=email&utm_campaign=signature-matt>
On Jun 7, 2017 5:15 PM, "Matt Gilson" <matt@getpattern.com> wrote: On Wed, Jun 7, 2017 at 3:11 PM, Erik <python@lucidity.plus.com> wrote:
Or even: from operator import itemgetter retrieve = itemgetter('a', 'b', 'c') a, b, c = retrieve(dictionary) Neither of these are really comparable to destructuring. If you take a look at how Erlang and Elixir do it, and any related code, you'll find it used constantly, all over the place. Recent ECMAScript is very similar, allowing both destructuring into vars matching the key names, or arbitrary var names. They both allow destructuring in the function header (IIRC python can do this with at least tuples). Erlang/Elixir goes beyond this by using the pattern matching to select the appropriate function clause within a function definition, but that's less relevant to Python. This feature has been requested before. It's easily one of the most, if not the top, feature I personally wish Python had. Incredibly useful and intuitive, and for me again, way more generally applicable than iterable unpacking. Maps are ubiquitous. -- C Anthony [mobile]
On 07/06/17 23:42, C Anthony Risinger wrote:
Neither of these are really comparable to destructuring.
No, but they are comparable to the OP's suggested new built-in method (without requiring each mapping type - not just dicts - to implement it). That was what _I_ was responding to. E.
On Jun 7, 2017 5:54 PM, "Erik" <python@lucidity.plus.com> wrote: On 07/06/17 23:42, C Anthony Risinger wrote:
Neither of these are really comparable to destructuring.
No, but they are comparable to the OP's suggested new built-in method (without requiring each mapping type - not just dicts - to implement it). That was what _I_ was responding to. No worries, I only meant to emphasize that destructuring is much much more powerful and less verbose/duplicative than anything based on functions. It could readily apply/fallback against any object's __dict__ because maps underpin the entire Python object system. -- C Anthony [mobile]
Maps with a known, fixed set of keys are relatively uncommon in Python, though.
This is false in interacting with HTTP services, where frequently you're working with deserialized JSON dictionaries you expect to be in a precise format (and fail if not). On Wed, Jun 7, 2017 at 11:32 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Hi Lucas, I would consider converting the dict into a namedtuple then. Essentially the namedtuple acts as a specification for expected fielsds: abc = namedtuple("ABC", "a b c") d = {"a":1, "b": 2, "c":3} # presumably you got this from reading some JSON abc(**d) # returns: ABC(a=1, b=2, c=3) Stephan 2017-06-08 8:53 GMT+02:00 Lucas Wiman <lucas.wiman@gmail.com>:
Lucas Wiman writes:
It's still true. In Python, if I need those things in variables *frequently*, I write a destructuring function that returns a sequence and use sequence unpacking. If I don't need the values in a particular use case, I use the "a, _, c = second_item_ignorable()" idiom. Or, in a larger or more permanent program, I write a class that is initialized with such a dictionary. If you like this feature, and wish it were in Python, I genuinely wish you good luck getting it in. My point is just that in precisely that use case I wouldn't be passing dictionaries that need destructuring around. I believe that to be the case for most Pythonistas. (Although several have posted in favor of some way to destructure dictionaries, typically those in favor of the status quo don't speak up until it looks like there will be a change.)
Yes, you frequently need to write destructuring functions. Of course there are ways to write them within Python, but they're often sort of ugly to write, and this syntax would make writing them nicer. I was disagreeing with your assertion that this is an uncommon use case: it's not, by your own admission. Of course, it is business logic which can be factored into a separate method, it's just that it would often be nice to not need to write such a method or define a NamedTuple class. If this feature does get added, I think it would make more sense to add it as part of a general framework of unification and pattern matching. This was previously discussed in this thread: https://mail.python.org/pipermail/python-ideas/2015-April/thread.html#32907 - Lucas On Thu, Jun 8, 2017 at 12:15 AM, Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
On 8 June 2017 at 08:15, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
The most common use case I find for this is when dealing with JSON (as someone else pointed out). But that's a definite case of dealing with data in a format that's "unnatural" for Python (by definition, JSON is "natural" for JavaScript). While having better support for working with JSON would be nice, I typically find myself wishing for better JSON handling libraries (ones that deal better with mappings with known keys) than for language features. But of course, I could write such a library myself, if it mattered sufficiently to me - and it never seems *that* important :-) Paul
On 8 June 2017 at 17:49, Paul Moore <p.f.moore@gmail.com> wrote:
Aye, I've had good experiences with using JSL to define JSON schemas for ad hoc JSON data structures that didn't already have them: https://jsl.readthedocs.io/en/latest/ And then, if you really wanted to, something like JSON Schema Objects provides automated destructuring and validation based on those schemas: https://python-jsonschema-objects.readthedocs.io/en/latest/Introduction.html However, it really isn't an ad hoc scripting friendly way to go - it's an "I'm writing a tested-and-formally-released application and want to strictly manage the data processing boundaries between components" style solution. pandas.read_json is pretty nice (https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_json.html), but would be a heavy dependency to bring in *just* for JSON -> DataFrame conversions. For myself, the things I mainly miss are: * getitem/setitem/delitem counterparts to getattr/setattr/delattr * getattrs and getitems builtins for retrieving multiple attributes or items in a single call (with the default value for missing results moved to a keyword-only argument) Now, these aren't hard to write yourself (and you can even use operator.attrgetter and operator.itemgetter as part of building them), but it's a sufficiently irritating niggle not to have them at my fingertips whenever they'd be convenient that I'll often end up writing out the long form equivalents instead. Are these necessary? Clearly not (although we did decide operator.itemgetter and operator.attrgetter were important enough to add for use with the map() and filter() builtins and other itertools). Is it a source of irritation that they're not there? Absolutely, at least for me. Cheers, Nick. P.S. Just clearly not irritating enough for me to actually put a patch together and push for a final decision one way or the other regarding adding them ;) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Well, it's not deliberately not destructive, but I'd be more in favor of dict unpacking-assignment if it were spelled more like this: >>> foo = {'a': 1, 'b': 2, 'c': 3, 'd': 4} >>> {'a': bar, 'b': baz, **rest} = foo >>> bar 1 >>> baz 2 >>> rest {'c': 3, 'd': 4} >>> foo {'a': 1, 'b': 2, 'c': 3, 'd': 4} That also takes care of the ordering issue, and any ambiguity about "am I unpacking the keys, the values, or both?", at the cost of a bit more typing. However, I'm a bit on the fence about this syntax as well: it's pretty easily confused with dictionary creation. Maybe the same thing but without the brackets? Just a thought I had this morning. Nick Nick Badger https://www.nickbadger.com 2017-06-08 7:00 GMT-07:00 Nick Coghlan <ncoghlan@gmail.com>:
While I don't like that syntax, we do know that sets are unhashable, so we can be certain that that would be a TypeError if it was meant to construct a set containing a set. (ie. {{foo}} will always result in a TypeError in python). On Thu, Jun 8, 2017 at 1:40 PM Chris Angelico <rosuav@gmail.com> wrote:
I ha d not read all the e-mails in the thread - but just to announce to people eager for a similar feature - I have a package on pypi that enables one to do: In [74]: with extradict.MapGetter({'a': 1}) as d: ...: from d import a ...: In [75]: a Out[75]: 1 (just pip install extradict ) On 8 June 2017 at 23:00, Joshua Morton <joshua.morton13@gmail.com> wrote:
By the way, this is already in the CPython source code if I remember from when I worked on 448. The code for dict unpacking is merely blocked. I like this syntax from a purity standpoint, but I don't think I would use it that much. On Thursday, June 8, 2017 at 3:18:21 PM UTC-4, Nick Badger wrote:
On Jun 8, 2017 1:35 AM, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote: C Anthony Risinger wrote:
Incredibly useful and intuitive, and for me again, way more generally applicable than iterable unpacking. Maps are ubiquitous.
Maps with a known, fixed set of keys are relatively uncommon in Python, though. Such an object is more likely to be an object with named attributes. I would generally agree, but in the 3 languages I mentioned at least, map destructuring does not require the LHS to exactly match the RHS (a complete match is required for lists and tuples though). Because pattern matching is central to the language, Elixir takes it even further, providing syntax that allows you to choose whether a variable on the LHS is treated as a match (similar to the None constant in my example) or normal variable binding. In all cases though, the LHS need only include the attributes you are actually interested in matching and/or binding. I need to review the linked thread still, but the way ECMAScript does it: const {one, two} = {one: 1, two: 2}; I think could also be useful in Python, especially if we defined some default handling of objects and dicts via __getattr__ and/or __getitem__, because people could use this to destructure `self` (I've seen this requested a couple times too): self.one = 1 self.two = 2 self.three = 3 {one, two} = self Or maybe even: def fun({one, two}, ...): Which is also supported (and common) in those 3 langs, but probably less pretty to Python eyes (and I think a bit less useful without function clauses). -- C Anthony [mobile]
On Wed, Jun 07, 2017 at 06:14:08PM +0000, Nick Humrich wrote:
This was discussed (briefly, to very little interest) in March/April 2008: https://mail.python.org/pipermail/python-ideas/2008-March/001511.html https://mail.python.org/pipermail/python-ideas/2008-April/001513.html and then again in 2016, when it spawned a very large thread starting here: https://mail.python.org/pipermail/python-ideas/2016-May/040430.html I know there's a lot of messages, but I STRONGLY encourage anyone, whether you are for or against this idea, to read the previous discussion before continuing it here. Guido was luke-warm about the **mapping syntax: https://mail.python.org/pipermail/python-ideas/2016-May/040466.html Nathan Schneider proposed making dict.values() take optional key names: https://mail.python.org/pipermail/python-ideas/2016-May/040517.html Guido suggested that this should be a different method: https://mail.python.org/pipermail/python-ideas/2016-May/040518.html My recollection is that the discussion evertually petered out with a more-or-less consensus that having a dict method (perhaps "getvalues"?) plus regular item unpacking is sufficient for the common use-case of unpacking a subset of keys: prefs = {'width': 80, 'height': 200, 'verbose': False, 'mode': PLAIN, 'name': 'Fnord', 'flags': spam|eggs|cheese, ... } # dict includes many more items width, height, size = prefs.getvalues( 'width', 'height', 'papersize', ) This trivially supports the cases where keys are not strings or valid identifiers: class_, spam, eggs = mapping.getvalues('class', 42, '~') It easily supports assignment targets which aren't simple variable names: obj.attribute[index], spam().attr = mapping.getvalues('foo', 'bar') An optional (defaults to False) "pop" keyword argument supports extracting and removing values from the dict in one call, which is commonly needed inside __init__ methods with **kwargs: class K(parent): def __init__(self, a, b, c, **kwargs): self.spam = kwargs.pop('spam') self.eggs = kwargs.pop('eggs') self.cheese = kwargs.pop('cheese') super().__init__(a, b, c, **kwargs) becomes: self.spam, self.eggs, self.cheese = kwargs.getvalues( 'spam eggs cheese'.split(), pop=True ) I don't recall this being proposed at the time, but we could support keyword arguments for missing or default values: DEFAULTS = {'height': 100, 'width': 50} prefs = get_prefs() # returns a dict height, width, size = prefs.getvalues( 'height', 'width', 'papersize', defaults=DEFAULTS, missing=None ) A basic implementation might be: # Untested. def getvalues(self, *keys, pop=False, defaults=None, missing=SENTINEL): values = [] for key in keys: try: x = self[key] except KeyError: if defaults is not None: x = defaults.get(key, SENTINEL) if x is SENTINEL: x = missing if x is SENTINEL: raise KeyError('missing key %r' % key) if pop: del self[key] values.append(x) return tuple(values) It's a bit repetitive for the common case where keys are the same as the assignment targets, but that's a hard problem to solve, and besides, "explicit is better than implicit". It also doesn't really work well for the case where you want to blindly create new assignment targets for *every* key, but: - my recollection is that nobody really came up with a convincing use-case for this (apologies if I missed any); - and if you really need this, you can do: locals().update(mapping) inside a class body or at the top-level of the module (but not inside a function). Please, let's save a lot of discussion here and now, and just read the 2016 thread: it is extremely comprehensive. -- Steve
Thank you! This overview really helps! On Thu, Jun 08, 2017 at 11:18:06AM +1000, Steven D'Aprano <steve@pearwood.info> wrote:
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
In python 3.6+ this is better since the dictionary is insertion-ordered, but is still not really what one would probably want.
Be careful: ordered dict is an implementation detail. You must use explicitly collections.OrderedDict() to avoid bad surprises. In CPython 3.7, dict might change again. Victor
One existing way to do this: a, b, c = (mydict[k] for k in ('a', 'b', 'c')) -- Greg
a few notes:
From the OP:
It would be cool to have a syntax that would unpack the dictionary to
a number of alternatives have been brought up, but I think it's important to note that this doesn't require naming the variables twice -- I think that's key to the proposal. Personally, I don't think that use case is common enough that it's something we should support with syntax. And while the OP drew a parallel with sequence unpacking, THAT doesn't make any assumptions about variable names on either side of the equals... The solutions that work with locals() being the only ones that do satisfy this, maybe a utility function that lets you specify which names you want unpacked would be handy. (a bit tricky to get the right locals() dict, though. Also: Paul Moore wrote:
I suppose so - what is a dict in python (data structure) is an object in Javascript (code). I often find teasing out what is data and what is code to be a tricky code structure issue. However, most of the time when you pass JSON around it really is "data", rather than code, so the Python interpretation is the more appropriate one.
exactly -- after all, under the hood, python objects have a _dict__ -- s mapping a JSON "object" to a Python object is really easy code to write: class JSObject: def __init__(self, js): self.__dict__.update(json.loads(js)) Of course, this doesn't deal with nested objects, but you can do that pretty easily, too: class JSObject: def __init__(self, js): """ initialized an object that matches the JSON passed in js is a JSON compatible python object tree (as produced by json.load*) """ for key, val in js.items(): if type(val) is dict: self.__dict__[key] = JSObject(val) else: self.__dict__[key] = val which won't deal with objects nested inside arrays. So, as I am hainvg fun: class JSObject: def __new__(cls, js_obj): """ create an object that matches the JSON passed in js is a JSON compatible python object tree (as produced by json.load*) """ if type(js_obj) is dict: self = super().__new__(cls) for key, val in js_obj.items(): self.__dict__[key] = JSObject(val) return self elif type(js_obj) is list: return [JSObject(item) for item in js_obj] else: return js_obj def __repr__(self): # note -- this does not do indentation... s = ['{\n'] for key, val in self.__dict__.items(): s.append('"%s": %s\n' % (key, str(val))) s.append('}') return "".join(s) I"m sure there are all sorts of edge cases this doesn't handle, but it shows it's pretty easy. However, I haven't seen a lib like this (though it may exist). I think the reason is that if you really want to convert JSON to Python objects, you probably want to use a schema and do validation, etc. And there are packages that do that. Also -- the Python data model is richer than the javascript one (dict keys that aren't identifiers, tuples vs lists, float vs int, ...) so another reason you probably don't want to unpack JSON into python Objects rather than dicts. This has gotten pretty off topic, though I think it shows that having a sample way to unpack a dict is probably not really helpful to the language - when you need to do that, you really need to have a bunch of extra logic in there anyway.
So here it is :-) Not that I seems important to me -- just fun. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 07/06/17 19:14, Nick Humrich wrote:
a, b, c = mydict.unpack('a', 'b', 'c')
def retrieve(mapping, *keys): return (mapping[key] for key in keys) $ python3 Python 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information.
E.
On Wed, Jun 7, 2017 at 3:11 PM, Erik <python@lucidity.plus.com> wrote:
Or even: from operator import itemgetter retrieve = itemgetter('a', 'b', 'c') a, b, c = retrieve(dictionary)
-- Matt Gilson | Pattern Software Engineer getpattern.com <https://www.getpattern.com?utm_source=email&utm_medium=email&utm_campaign=signature-matt>
On Jun 7, 2017 5:15 PM, "Matt Gilson" <matt@getpattern.com> wrote: On Wed, Jun 7, 2017 at 3:11 PM, Erik <python@lucidity.plus.com> wrote:
Or even: from operator import itemgetter retrieve = itemgetter('a', 'b', 'c') a, b, c = retrieve(dictionary) Neither of these are really comparable to destructuring. If you take a look at how Erlang and Elixir do it, and any related code, you'll find it used constantly, all over the place. Recent ECMAScript is very similar, allowing both destructuring into vars matching the key names, or arbitrary var names. They both allow destructuring in the function header (IIRC python can do this with at least tuples). Erlang/Elixir goes beyond this by using the pattern matching to select the appropriate function clause within a function definition, but that's less relevant to Python. This feature has been requested before. It's easily one of the most, if not the top, feature I personally wish Python had. Incredibly useful and intuitive, and for me again, way more generally applicable than iterable unpacking. Maps are ubiquitous. -- C Anthony [mobile]
On 07/06/17 23:42, C Anthony Risinger wrote:
Neither of these are really comparable to destructuring.
No, but they are comparable to the OP's suggested new built-in method (without requiring each mapping type - not just dicts - to implement it). That was what _I_ was responding to. E.
On Jun 7, 2017 5:54 PM, "Erik" <python@lucidity.plus.com> wrote: On 07/06/17 23:42, C Anthony Risinger wrote:
Neither of these are really comparable to destructuring.
No, but they are comparable to the OP's suggested new built-in method (without requiring each mapping type - not just dicts - to implement it). That was what _I_ was responding to. No worries, I only meant to emphasize that destructuring is much much more powerful and less verbose/duplicative than anything based on functions. It could readily apply/fallback against any object's __dict__ because maps underpin the entire Python object system. -- C Anthony [mobile]
Maps with a known, fixed set of keys are relatively uncommon in Python, though.
This is false in interacting with HTTP services, where frequently you're working with deserialized JSON dictionaries you expect to be in a precise format (and fail if not). On Wed, Jun 7, 2017 at 11:32 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Hi Lucas, I would consider converting the dict into a namedtuple then. Essentially the namedtuple acts as a specification for expected fielsds: abc = namedtuple("ABC", "a b c") d = {"a":1, "b": 2, "c":3} # presumably you got this from reading some JSON abc(**d) # returns: ABC(a=1, b=2, c=3) Stephan 2017-06-08 8:53 GMT+02:00 Lucas Wiman <lucas.wiman@gmail.com>:
Lucas Wiman writes:
It's still true. In Python, if I need those things in variables *frequently*, I write a destructuring function that returns a sequence and use sequence unpacking. If I don't need the values in a particular use case, I use the "a, _, c = second_item_ignorable()" idiom. Or, in a larger or more permanent program, I write a class that is initialized with such a dictionary. If you like this feature, and wish it were in Python, I genuinely wish you good luck getting it in. My point is just that in precisely that use case I wouldn't be passing dictionaries that need destructuring around. I believe that to be the case for most Pythonistas. (Although several have posted in favor of some way to destructure dictionaries, typically those in favor of the status quo don't speak up until it looks like there will be a change.)
Yes, you frequently need to write destructuring functions. Of course there are ways to write them within Python, but they're often sort of ugly to write, and this syntax would make writing them nicer. I was disagreeing with your assertion that this is an uncommon use case: it's not, by your own admission. Of course, it is business logic which can be factored into a separate method, it's just that it would often be nice to not need to write such a method or define a NamedTuple class. If this feature does get added, I think it would make more sense to add it as part of a general framework of unification and pattern matching. This was previously discussed in this thread: https://mail.python.org/pipermail/python-ideas/2015-April/thread.html#32907 - Lucas On Thu, Jun 8, 2017 at 12:15 AM, Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
On 8 June 2017 at 08:15, Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
The most common use case I find for this is when dealing with JSON (as someone else pointed out). But that's a definite case of dealing with data in a format that's "unnatural" for Python (by definition, JSON is "natural" for JavaScript). While having better support for working with JSON would be nice, I typically find myself wishing for better JSON handling libraries (ones that deal better with mappings with known keys) than for language features. But of course, I could write such a library myself, if it mattered sufficiently to me - and it never seems *that* important :-) Paul
On 8 June 2017 at 17:49, Paul Moore <p.f.moore@gmail.com> wrote:
Aye, I've had good experiences with using JSL to define JSON schemas for ad hoc JSON data structures that didn't already have them: https://jsl.readthedocs.io/en/latest/ And then, if you really wanted to, something like JSON Schema Objects provides automated destructuring and validation based on those schemas: https://python-jsonschema-objects.readthedocs.io/en/latest/Introduction.html However, it really isn't an ad hoc scripting friendly way to go - it's an "I'm writing a tested-and-formally-released application and want to strictly manage the data processing boundaries between components" style solution. pandas.read_json is pretty nice (https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_json.html), but would be a heavy dependency to bring in *just* for JSON -> DataFrame conversions. For myself, the things I mainly miss are: * getitem/setitem/delitem counterparts to getattr/setattr/delattr * getattrs and getitems builtins for retrieving multiple attributes or items in a single call (with the default value for missing results moved to a keyword-only argument) Now, these aren't hard to write yourself (and you can even use operator.attrgetter and operator.itemgetter as part of building them), but it's a sufficiently irritating niggle not to have them at my fingertips whenever they'd be convenient that I'll often end up writing out the long form equivalents instead. Are these necessary? Clearly not (although we did decide operator.itemgetter and operator.attrgetter were important enough to add for use with the map() and filter() builtins and other itertools). Is it a source of irritation that they're not there? Absolutely, at least for me. Cheers, Nick. P.S. Just clearly not irritating enough for me to actually put a patch together and push for a final decision one way or the other regarding adding them ;) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Well, it's not deliberately not destructive, but I'd be more in favor of dict unpacking-assignment if it were spelled more like this: >>> foo = {'a': 1, 'b': 2, 'c': 3, 'd': 4} >>> {'a': bar, 'b': baz, **rest} = foo >>> bar 1 >>> baz 2 >>> rest {'c': 3, 'd': 4} >>> foo {'a': 1, 'b': 2, 'c': 3, 'd': 4} That also takes care of the ordering issue, and any ambiguity about "am I unpacking the keys, the values, or both?", at the cost of a bit more typing. However, I'm a bit on the fence about this syntax as well: it's pretty easily confused with dictionary creation. Maybe the same thing but without the brackets? Just a thought I had this morning. Nick Nick Badger https://www.nickbadger.com 2017-06-08 7:00 GMT-07:00 Nick Coghlan <ncoghlan@gmail.com>:
While I don't like that syntax, we do know that sets are unhashable, so we can be certain that that would be a TypeError if it was meant to construct a set containing a set. (ie. {{foo}} will always result in a TypeError in python). On Thu, Jun 8, 2017 at 1:40 PM Chris Angelico <rosuav@gmail.com> wrote:
I ha d not read all the e-mails in the thread - but just to announce to people eager for a similar feature - I have a package on pypi that enables one to do: In [74]: with extradict.MapGetter({'a': 1}) as d: ...: from d import a ...: In [75]: a Out[75]: 1 (just pip install extradict ) On 8 June 2017 at 23:00, Joshua Morton <joshua.morton13@gmail.com> wrote:
By the way, this is already in the CPython source code if I remember from when I worked on 448. The code for dict unpacking is merely blocked. I like this syntax from a purity standpoint, but I don't think I would use it that much. On Thursday, June 8, 2017 at 3:18:21 PM UTC-4, Nick Badger wrote:
On Jun 8, 2017 1:35 AM, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote: C Anthony Risinger wrote:
Incredibly useful and intuitive, and for me again, way more generally applicable than iterable unpacking. Maps are ubiquitous.
Maps with a known, fixed set of keys are relatively uncommon in Python, though. Such an object is more likely to be an object with named attributes. I would generally agree, but in the 3 languages I mentioned at least, map destructuring does not require the LHS to exactly match the RHS (a complete match is required for lists and tuples though). Because pattern matching is central to the language, Elixir takes it even further, providing syntax that allows you to choose whether a variable on the LHS is treated as a match (similar to the None constant in my example) or normal variable binding. In all cases though, the LHS need only include the attributes you are actually interested in matching and/or binding. I need to review the linked thread still, but the way ECMAScript does it: const {one, two} = {one: 1, two: 2}; I think could also be useful in Python, especially if we defined some default handling of objects and dicts via __getattr__ and/or __getitem__, because people could use this to destructure `self` (I've seen this requested a couple times too): self.one = 1 self.two = 2 self.three = 3 {one, two} = self Or maybe even: def fun({one, two}, ...): Which is also supported (and common) in those 3 langs, but probably less pretty to Python eyes (and I think a bit less useful without function clauses). -- C Anthony [mobile]
On Wed, Jun 07, 2017 at 06:14:08PM +0000, Nick Humrich wrote:
This was discussed (briefly, to very little interest) in March/April 2008: https://mail.python.org/pipermail/python-ideas/2008-March/001511.html https://mail.python.org/pipermail/python-ideas/2008-April/001513.html and then again in 2016, when it spawned a very large thread starting here: https://mail.python.org/pipermail/python-ideas/2016-May/040430.html I know there's a lot of messages, but I STRONGLY encourage anyone, whether you are for or against this idea, to read the previous discussion before continuing it here. Guido was luke-warm about the **mapping syntax: https://mail.python.org/pipermail/python-ideas/2016-May/040466.html Nathan Schneider proposed making dict.values() take optional key names: https://mail.python.org/pipermail/python-ideas/2016-May/040517.html Guido suggested that this should be a different method: https://mail.python.org/pipermail/python-ideas/2016-May/040518.html My recollection is that the discussion evertually petered out with a more-or-less consensus that having a dict method (perhaps "getvalues"?) plus regular item unpacking is sufficient for the common use-case of unpacking a subset of keys: prefs = {'width': 80, 'height': 200, 'verbose': False, 'mode': PLAIN, 'name': 'Fnord', 'flags': spam|eggs|cheese, ... } # dict includes many more items width, height, size = prefs.getvalues( 'width', 'height', 'papersize', ) This trivially supports the cases where keys are not strings or valid identifiers: class_, spam, eggs = mapping.getvalues('class', 42, '~') It easily supports assignment targets which aren't simple variable names: obj.attribute[index], spam().attr = mapping.getvalues('foo', 'bar') An optional (defaults to False) "pop" keyword argument supports extracting and removing values from the dict in one call, which is commonly needed inside __init__ methods with **kwargs: class K(parent): def __init__(self, a, b, c, **kwargs): self.spam = kwargs.pop('spam') self.eggs = kwargs.pop('eggs') self.cheese = kwargs.pop('cheese') super().__init__(a, b, c, **kwargs) becomes: self.spam, self.eggs, self.cheese = kwargs.getvalues( 'spam eggs cheese'.split(), pop=True ) I don't recall this being proposed at the time, but we could support keyword arguments for missing or default values: DEFAULTS = {'height': 100, 'width': 50} prefs = get_prefs() # returns a dict height, width, size = prefs.getvalues( 'height', 'width', 'papersize', defaults=DEFAULTS, missing=None ) A basic implementation might be: # Untested. def getvalues(self, *keys, pop=False, defaults=None, missing=SENTINEL): values = [] for key in keys: try: x = self[key] except KeyError: if defaults is not None: x = defaults.get(key, SENTINEL) if x is SENTINEL: x = missing if x is SENTINEL: raise KeyError('missing key %r' % key) if pop: del self[key] values.append(x) return tuple(values) It's a bit repetitive for the common case where keys are the same as the assignment targets, but that's a hard problem to solve, and besides, "explicit is better than implicit". It also doesn't really work well for the case where you want to blindly create new assignment targets for *every* key, but: - my recollection is that nobody really came up with a convincing use-case for this (apologies if I missed any); - and if you really need this, you can do: locals().update(mapping) inside a class body or at the top-level of the module (but not inside a function). Please, let's save a lot of discussion here and now, and just read the 2016 thread: it is extremely comprehensive. -- Steve
Thank you! This overview really helps! On Thu, Jun 08, 2017 at 11:18:06AM +1000, Steven D'Aprano <steve@pearwood.info> wrote:
Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
In python 3.6+ this is better since the dictionary is insertion-ordered, but is still not really what one would probably want.
Be careful: ordered dict is an implementation detail. You must use explicitly collections.OrderedDict() to avoid bad surprises. In CPython 3.7, dict might change again. Victor
One existing way to do this: a, b, c = (mydict[k] for k in ('a', 'b', 'c')) -- Greg
a few notes:
From the OP:
It would be cool to have a syntax that would unpack the dictionary to
a number of alternatives have been brought up, but I think it's important to note that this doesn't require naming the variables twice -- I think that's key to the proposal. Personally, I don't think that use case is common enough that it's something we should support with syntax. And while the OP drew a parallel with sequence unpacking, THAT doesn't make any assumptions about variable names on either side of the equals... The solutions that work with locals() being the only ones that do satisfy this, maybe a utility function that lets you specify which names you want unpacked would be handy. (a bit tricky to get the right locals() dict, though. Also: Paul Moore wrote:
I suppose so - what is a dict in python (data structure) is an object in Javascript (code). I often find teasing out what is data and what is code to be a tricky code structure issue. However, most of the time when you pass JSON around it really is "data", rather than code, so the Python interpretation is the more appropriate one.
exactly -- after all, under the hood, python objects have a _dict__ -- s mapping a JSON "object" to a Python object is really easy code to write: class JSObject: def __init__(self, js): self.__dict__.update(json.loads(js)) Of course, this doesn't deal with nested objects, but you can do that pretty easily, too: class JSObject: def __init__(self, js): """ initialized an object that matches the JSON passed in js is a JSON compatible python object tree (as produced by json.load*) """ for key, val in js.items(): if type(val) is dict: self.__dict__[key] = JSObject(val) else: self.__dict__[key] = val which won't deal with objects nested inside arrays. So, as I am hainvg fun: class JSObject: def __new__(cls, js_obj): """ create an object that matches the JSON passed in js is a JSON compatible python object tree (as produced by json.load*) """ if type(js_obj) is dict: self = super().__new__(cls) for key, val in js_obj.items(): self.__dict__[key] = JSObject(val) return self elif type(js_obj) is list: return [JSObject(item) for item in js_obj] else: return js_obj def __repr__(self): # note -- this does not do indentation... s = ['{\n'] for key, val in self.__dict__.items(): s.append('"%s": %s\n' % (key, str(val))) s.append('}') return "".join(s) I"m sure there are all sorts of edge cases this doesn't handle, but it shows it's pretty easy. However, I haven't seen a lib like this (though it may exist). I think the reason is that if you really want to convert JSON to Python objects, you probably want to use a schema and do validation, etc. And there are packages that do that. Also -- the Python data model is richer than the javascript one (dict keys that aren't identifiers, tuples vs lists, float vs int, ...) so another reason you probably don't want to unpack JSON into python Objects rather than dicts. This has gotten pretty off topic, though I think it shows that having a sample way to unpack a dict is probably not really helpful to the language - when you need to do that, you really need to have a bunch of extra logic in there anyway.
So here it is :-) Not that I seems important to me -- just fun. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
participants (21)
-
Antoine Rozo -
C Anthony Risinger -
Chris Angelico -
Chris Barker -
Erik -
Greg Ewing -
Joao S. O. Bueno -
Joshua Morton -
Lucas Wiman -
Matt Gilson -
MRAB -
Neil Girdhar -
Nick Badger -
Nick Coghlan -
Nick Humrich -
Oleg Broytman -
Paul Moore -
Stephan Houben -
Stephen J. Turnbull -
Steven D'Aprano -
Victor Stinner